Linux Host housing Metadata Tier got corrupted. Any mechanism to fix this without impacting SAS?

[SangramjitPanda]

Hi All,

We have SAS 9.4 M3 Servers installed on 3 Linux Boxes:

LXXXXX1 -> Metadata Tier, LXXXX1   -> Compute Tier and LXXXXXX1 -> Web Tier

Few days back got a mail from Unix team stating that the Linux Host housing Metadata Server got corrupted as Unix team does patching using IBM BIGFIX tool each month and somehow the rpm and yum utilities are not working on this host and there is no good backup of this Server too from where they can restore it.

So they are suggesting to create a fresh server from scratch with same hostname and if there is any possibility to install the SAS Metadata Tier again?

Can anyone please suggest if it is recommended to go for a fresh installation and if we do will it impact other tiers like Compute and Web tier?

Can this be solved without Installing the Metadata Tier? like moving the SAS Deployedd Folders(Install and Config) to new host(same name which was hosting Metadata) with same permission and path name ? Will this work?

Any suggestions to handle this without much impact to SAS Application?

Thanks,

Sangramjit

[boemskats]

I think it's easier to perform the binaries install than to try to copy the files over to the new host, but I think porting the config over should be ok.

Run the 'Install software' step on the new host, make sure your SASHome path matches, perform all the setuid steps & vfabric licences, ensure the credentials for the host users match, and then move your config directory over to the same location as on the previous host. I can't think of any reason why this wouldn't work, unless your metadata repository or part of that filesystem is also corrupted.

[SangramjitPanda]

Thanks @boemskats for the quick response and update.

Metadata Repository and other SAS Related files and folder on Linux Host are fine. Only the OS related packages like yum and rpm utilities are not working due to which patching is getting failed.

Let me confirm my understanding on this. Please correct me if I misunderstood.

You mean to say, I can go-ahead and Use Deployment Wizard to only Install the SAS Metadata Tier (no Configuration required) on new host.

Will make sure that SASHome path matches, perform all the setuid steps & vfabric licences, ensure the credentials for the host users matches.

Once Installation is done, I need to move the Config Folder only to the same location as it was on previous path.

If possible can you please suggest on the below items:

1. If we follow the above approach, Will it impact to other SAS tiers like Compute and Web which are installed on other Linux Boxes?

2. Any pre-requisites to follow for this approach? 

3. If we follow the approach, will the SAS Application and Metadata will work fine as it is working now? Any impact 

4. Only copying the SAS Install and Config Folder to new host won't work ? Installation is the only way?

[boemskats]

1. You'll need to make sure routing works and you'll obviously need to restart the relevant components, but I think that'll be all.

2. Don't overwrite your original disk to do this, make sure you keep the approach non-destructive.

3. Only way to find out is try, but I'm pretty confident it will.

4. An SDW based Installation will alert you to any prerequisites you might have missed in building the server, and make sure that the necessary prereqs are in place for a configuration to be carried out. Just copying the filesystem over won't do that. If I was doing this I'd definitely perform the installation step using the SDW, and then add the config (which is arguably what the config step of the SDW would do anyway). Only thing I'm not sure about is how this will impact Environment Manager, as I recall it being the only component that isn't strictly confined to the Config directory. 

Nik

[SangramjitPanda]

Thanks Nik for the update. It seems this is the only option worth to try. 

[SangramjitPanda]

I was going through the SAS Disaster Recover Policy Note and found out that:

• Copying only the SAS deployment (the SASHome or SAS Configuration directories) from one system to another is not supported. In this scenario, SAS Deployment Manager tasks fail to be able to properly manage and maintain the deployment and the SAS Deployment Agent (necessary to perform future backups and middle-tier clustering). Copying the SAS deployment as part of Option A is supported as long as the disaster-recovery system remains a clone of the production system.

Ref: https://support.sas.com/en/technical-support/services-policies/disaster-recovery-policy.html

As we don't have good backup nor do we have any system cloned for this affected host or else we could have copied the SAS Config Folder to the cloned environment.

I am confused now. So fresh installation & configuration of SAS Components (Metadata, Compute and Web Tier) on Linux Host from scratch and then migrate the SAS Metadata Content is the only approach or is there any other approach?

[SASKiwi]

I hope you are making sure that this server gets backed up properly in the future.

[SangramjitPanda]

Yes Unix team do perform full backups of the servers but somehow one of them got corrupted and we don't have any good backup to restore.

[dougunixts]

Unfortunately the suggested approach of reinstalling the metadata server SASHOME and copying the configuration directory over to the new machine is not supported.  Replication of SAS environments or parts of SAS environments must conform to the same standards documented in the SAS Disaster Recovery Policy and Update Host Name References documentation.

https://support.sas.com/en/technical-support/services-policies/disaster-recovery-policy.html

http://go.documentation.sas.com/?docsetId=bisag&docsetTarget=n0zdcqxsmd21wtn17vt0w2prek90.htm&docset...

While the plan to reinstall SASHOME is good in that the SAS Deployment Wizard will be run to make sure some of the checks are done it still fails to provide a SAS environment which will not only run but is supportable and maintainable.  Tasks such as applying hotfixes, applying maintenance and being able to migrate the environment often fail in these scenarios and often there is no method to recover from those failures.  Even from the perspective of running, the SAS Backup and Recovery Tool and SAS Environment Manager fail to function in environments repaired using techniques such as this.

Since unfortunately there are no backups of the metadata server the two supported choices here are to perform a complete new deployment using either a SAS Migration Utility package or promotion after the new environment is installed.

Here is what I encourage customer to do before performing updates of their Unix SAS environment.  These suggestions can also be incorporated into your backup and recover plan for your SAS environment.  It is imperative that you have operating system level backups done using a method that preserves the original date/time stamps, ownership and permissions of the files/directories in your SAS environment.  You need a backup of every SASHOME and SAS Configuration directory for every tier of your environment which were taken during the same outage window with all the SAS servers and processes stopped. The backups must be in sync because you will possibly need to restore all the tiers if you have unresolvable problems during your deployment. If you do not have these backups then a reinstall of the environment will likely have to be done.

Thanks,

Doug

SAS Unix Systems Technical Support