Architecting, installing and maintaining your SAS environment

How to automatically compare the configurations/metadata/midtier of two SAS servers

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 6
Accepted Solution

How to automatically compare the configurations/metadata/midtier of two SAS servers

Hi Guys,

 

Happy Holidays !

Currently our production SAS server is SAS9.3M2 on Unix. Recently we installed and configured a fresh new SAS 9.4M4 server (on Unix as well), and the new server is still at testing stage.
We partially promoted metadata & midtier from the SAS9.3 server to SAS9.4.
Since SAS9.3 currently is our production server, and hence its configurations, metadata, and midtier are updated constantly, but we
might not promote the new information to the testing SAS9.4 promptly.
I want to write a program (either Unix shell script or SAS program, or the combination of the two), so that I could automatically
compare the configurations, metadata, and midtier of the production SAS9.3 server v.s. the testing SAS9.4 server, and find their
differences or inconsistency between the two servers.

I have two questions regarding this issue:
1. Do you have some sample programs (e.g., unix scripts, and/or SAS programs) to achieve my goal mentioned above ? Or any suggestions on this topic ?
2. Could you recommend any SAS documents, papers, SAS Notes, etc., regarding this topic ?

 

Thanks in advance,

 

Alex


Accepted Solutions
Solution
‎01-18-2018 10:09 AM
Frequent Contributor
Posts: 145

Re: How to automatically compare the configurations/metadata/midtier of two SAS servers

Posted in reply to BigSASFan

Happy New Year !

 

 As @jklaverstijn said previously, the lack of answers to your post does not necessarily imply lack of contributors - here we are - but rather scanty solutions if any at all 100% suited to your goal. 

 

SAS 9 Intelligence platforms based on a central Metadata repository do not come with an fully integrated versioning tool or lifecycle framework, nor do they come with some kind of interface with an external versioning database like CVS or GitHub, save for the Data Integration studio ETL objects (rarely used, in practice). This doesn't mean, however, that there aren't any tools provided by SAS to generate the corresponding information. On the contrary, there are many useful tools to gather this information and each version up to 9.4 has significantly added or extended this collection.

 

here is, for instance, an example of an end-to-end (business) promotion process that can be achieved using only SAS tools :

 

https://support.sas.com/resources/papers/proceedings12/019-2012.pdf

 

As you see, some custom developments are needed to integrate the whole bunch of components.

 

But you won't have out of the box the necessary features, for instance, to compare two metadata objects based upon their inner content, only partially upon some of their (visible) attributes. If we draw some comparison with shell commands, then this means you have the equivalent of  "find", "ls" or "stat" (istat) even some kind of "ldd" perhaps without "diff", "cmp" or "md5sum". 

Furthermore, since the reporting tools have significantly changed from 9.3 to 9.4, sometimes you won't be able to use the same features in order to compare your old 9.3 metadata with your brand new 9.4. There is no all-round solution to your question, in short, only partial workarounds. This being said, I can suggest you the following tricks according to a somewhat naive approach :

 

1. Your SAS platforms are usually installed with SAS Deployment Wizard (wicked) which rely on a "response.properties" script to store the parameters values : you might find useful to simply compare the two files, 9.3 & 9.4 with "diff" to highlight the major differences (if any). 

 

2. You can easily extract the full list of your main objects stored in metadata, which means those that can be promoted, each with three attributes : Type / Folder Path / Name

 

#/<SASHome>/SASPlatformObjectFramework/<9.3 or 9.4>/ExportPackage -host <metadata server> -port 8561 -user ******* -password ****** -package ~/test.spk -objects "/" -noexecute

 

#noexecute switch translates to No package file created

 

3. Depending upon your installation (the 9.3 M2 I am currently using wasn't shipped with any SMC BI Lineage Plugin ...) , you can also analyse the relationships between some types of metadata objects with

 

9.3 & 9.4 : SAS BI Lineage Plugins (this can even generates SAS Tables storing the objects)

 

9.4  sas-relationship-reporter

 

4. You can enhance your list by adding the security attributes assigned to the metadata objects :

 

https://support.sas.com/documentation/cdl/en/bisecag/63082/HTML/default/viewer.htm#p0lz8x8q07wo67n1v...

 

See also Metacoda Security Plugins  (additional licence required)

 

5. In the particular case of security objects like Users, Groups, Roles etc. , Metacoda provides you with a highly customisable tool called "Security Testing Framework". This enables some inner comparisons for these objects ; for instance, you can then compare two sets of user groups (9.3 vs 9.4) based on their inner subgroups, attributes etc.

 

6. Then there are the main general APIs to devise your own requests in metadata : 

 

https://support.sas.com/documentation/cdl/en/lrmeta/63180/HTML/default/viewer.htm#titlepage.htm

 

Lastly, you can even try to combine the latter with SAS Base functions like md5 (exists for 9.3 & 9.4) to extract and compare some objects like Stored Process (metadata type) within SAS.

 

HTH

Ronan

 

View solution in original post


All Replies
Occasional Contributor
Posts: 6

Re: How to automatically compare the configurations/metadata/midtier of two SAS servers

Posted in reply to BigSASFan

It would be highly appreciated if you have some idea and could share your thoughts.

 

Thanks in advance,

 

Alex

Super Contributor
Posts: 493

Re: How to automatically compare the configurations/metadata/midtier of two SAS servers

Posted in reply to BigSASFan

Hi Alex,

 

I think the lacklustre response to this question is because there is actually no such thing. Someone may have devised something as a personal tool for parts of it but nothing as wide as you're looking, no matter how interesting it would be, has come up. At least not to my knowledge.Especially the comparison of metadata configuration can be complicated. Comparing config files is much easier.

 

We are facing similar challenges as we have multiple grid environments that all have components that should be configured alike and follow the same life cycle. We are looking into desired-state tooling and a central repository for config files such as git or svn and Ansible. But our use case is not entirely the same as yours.

 

Regards,

- Jan.

Solution
‎01-18-2018 10:09 AM
Frequent Contributor
Posts: 145

Re: How to automatically compare the configurations/metadata/midtier of two SAS servers

Posted in reply to BigSASFan

Happy New Year !

 

 As @jklaverstijn said previously, the lack of answers to your post does not necessarily imply lack of contributors - here we are - but rather scanty solutions if any at all 100% suited to your goal. 

 

SAS 9 Intelligence platforms based on a central Metadata repository do not come with an fully integrated versioning tool or lifecycle framework, nor do they come with some kind of interface with an external versioning database like CVS or GitHub, save for the Data Integration studio ETL objects (rarely used, in practice). This doesn't mean, however, that there aren't any tools provided by SAS to generate the corresponding information. On the contrary, there are many useful tools to gather this information and each version up to 9.4 has significantly added or extended this collection.

 

here is, for instance, an example of an end-to-end (business) promotion process that can be achieved using only SAS tools :

 

https://support.sas.com/resources/papers/proceedings12/019-2012.pdf

 

As you see, some custom developments are needed to integrate the whole bunch of components.

 

But you won't have out of the box the necessary features, for instance, to compare two metadata objects based upon their inner content, only partially upon some of their (visible) attributes. If we draw some comparison with shell commands, then this means you have the equivalent of  "find", "ls" or "stat" (istat) even some kind of "ldd" perhaps without "diff", "cmp" or "md5sum". 

Furthermore, since the reporting tools have significantly changed from 9.3 to 9.4, sometimes you won't be able to use the same features in order to compare your old 9.3 metadata with your brand new 9.4. There is no all-round solution to your question, in short, only partial workarounds. This being said, I can suggest you the following tricks according to a somewhat naive approach :

 

1. Your SAS platforms are usually installed with SAS Deployment Wizard (wicked) which rely on a "response.properties" script to store the parameters values : you might find useful to simply compare the two files, 9.3 & 9.4 with "diff" to highlight the major differences (if any). 

 

2. You can easily extract the full list of your main objects stored in metadata, which means those that can be promoted, each with three attributes : Type / Folder Path / Name

 

#/<SASHome>/SASPlatformObjectFramework/<9.3 or 9.4>/ExportPackage -host <metadata server> -port 8561 -user ******* -password ****** -package ~/test.spk -objects "/" -noexecute

 

#noexecute switch translates to No package file created

 

3. Depending upon your installation (the 9.3 M2 I am currently using wasn't shipped with any SMC BI Lineage Plugin ...) , you can also analyse the relationships between some types of metadata objects with

 

9.3 & 9.4 : SAS BI Lineage Plugins (this can even generates SAS Tables storing the objects)

 

9.4  sas-relationship-reporter

 

4. You can enhance your list by adding the security attributes assigned to the metadata objects :

 

https://support.sas.com/documentation/cdl/en/bisecag/63082/HTML/default/viewer.htm#p0lz8x8q07wo67n1v...

 

See also Metacoda Security Plugins  (additional licence required)

 

5. In the particular case of security objects like Users, Groups, Roles etc. , Metacoda provides you with a highly customisable tool called "Security Testing Framework". This enables some inner comparisons for these objects ; for instance, you can then compare two sets of user groups (9.3 vs 9.4) based on their inner subgroups, attributes etc.

 

6. Then there are the main general APIs to devise your own requests in metadata : 

 

https://support.sas.com/documentation/cdl/en/lrmeta/63180/HTML/default/viewer.htm#titlepage.htm

 

Lastly, you can even try to combine the latter with SAS Base functions like md5 (exists for 9.3 & 9.4) to extract and compare some objects like Stored Process (metadata type) within SAS.

 

HTH

Ronan

 

Trusted Advisor
Posts: 1,571

Re: How to automatically compare the configurations/metadata/midtier of two SAS servers

[ Edited ]
Posted in reply to BigSASFan

Hello @lazhang, Alex,

 

even with the high chance nothing I would say here would be new:

 

There is a tool that can give you some of the information: http://support.sas.com/kb/20/390.html

 

But in general, to fully compare 2 environments it is a gigantic task, specially if we talk about config files between 2 SAS versions as different as 9.3 and 9.4, middle-tier wise. Basically your best option would be a deployment expert/consultant on site and trust on her/his expert eye.

 

If you insist on doing a programmatic check, I truly think your best options would be:

 

1- Metadata an Compute configuration files: a grep tool and ad your own comparison rules.

 

2- Metadata items (SAS Folders, Security): SPK packages and relationship tool

 

3- Other metadata items (middle tier - Configuration Manager-, Servers, System, etc): theory says the SAS Metadata Interface could help you to script it. Practice is not that simple. Best you can do is to do a manual check.

 

4- Middle tier configuration files: since 9.3 is based on Jboss/WebLogic/WebSphere but 9.5 has its own Apache/vFabric servers, configuration files are quite different in name and content, so not comparable. Best is to have a good log/document with your customisations and try to reapply them again in 9.4. Perhaps in some cases you would need to assume some customisations won;t be even noticeable so won't need to be reapplied. That is why you have a good test environment, to test and accept the environment.

 

5- Content Server: Perhaps you can compare with the DAV Tree tools, but again, if you cannot recall on especific customizations, and are not documented, trust on your SPK import.

 

6- Shared Services database: same as above. 

 

To finalise my comments, a tip: many experts do migrate 9.X environments to 9.4 in 2 steps, meaning: SAS Migration Utility package migration to a temporary Lev9 (so SCS, Shared Services db and others will be fully migrated as best as SAS can do) and then do a second migration with Partial Promotion from that temporary Lev9 to the real Lev1 environment. Once finished, Lev9 can be decommissioned. Before decommission, you can then better compare your temporary Lev9 and your final Lev1 environments with SPK and grep tools with much less differences because now you compare two 9.4 environments.

 

Hope it helps,

 

Kind regards,

Juan

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 4 replies
  • 498 views
  • 11 likes
  • 4 in conversation