System Evaluation Tool Helps with SAS 9.4 Upgrade-in-Place Planning
With the impending end of Adobe Flash a large number of SAS customers will be upgrading to SAS 9.4 M6 or later over the next few months. While there are several upgrade paths, upgrade-in-place (UIP) is often an enticing option as it minimizes hardware demands and data movement. UIP is, however, considered a somewhat risky option as customers have had little help in identifying potential issues that might cause problems for them before they embark on the upgrade. Because of the number of customers who will be upgrading this year, SAS is trying to do everything possible to make upgrade-in-place a more palatable option for as many of those customers as possible. As part of that effort, the System Evaluation tool has been created to scan a customer's deployment for potential issues that might impede an upgrade-in-place.
The System Evaluation tool is included in the suite of SAS Content Assessment tools and is available on SAS Downloads for the benefit of all SAS customers and partners. The version just released in August is the latest of several phased releases planned to bring additional issue identification onboard.
This article focuses on the System Evaluation tool but if you would like to know more about the other tools in the SAS Content Assessment package, you should read Brian Kinnebrew's Communities article, The SAS 9 Content Assessment Tool.
What is the System Evaluation Tool?
The System Evaluation tool is run against each SAS configuration directory in a SAS 9 deployment and scans the configuration directory and metadata for issues that are known to potentially impact a successful UIP such as
- A hostname value for the configuration that does not match the current hostname
- BUILD FAILED, ReportFailures or CheckForErrors messages appearing in configuration logs
- Anomalies in status.xml such as multiple order numbers
- Duplicate web application servers associated to the configuration
- A missing or unwritable configuration.properties file
- Inability to access the SAS Content Server
- Insufficient MAX_CONNECTIONS for Postgres
- Missing gemfire.member-timeout property on managed web application servers
- Missing or corrupt deployment registry information
- Anomalies in the metadata registration of web tier components
As we will see later, if any of the issues above are detected the System Evaluation tool collects information to help assess the seriousness of the situation including, but not limited to
- a .CSV file mapping the identified issues to a specific log or xml file for each issue
- any supporting files (.log, xml, .properties files)
- status.xml
- registry.xml
- SAS Deployment Wizard response files and logs
- Extensive logs from the configuration directory's Web subdirectory
- Details for each of the managed web application servers (logs, server.xml, setenv.sh, gemfire logs, etc.)
- Summary reports detailing the issues found by the tool and suggested fixes for some issues
and zips them into an archive file for that particular machine. You can then partner with Tech Support to examine the reported issues and ensure the correct assumptions and interpretation of results occurs.
When do I use it?
The System Evaluation tool should be one of the tasks you perform during the planning phase of your upgrade project. It is intended to provide insight into potential problems that may take time to work through and if necessary, correct. Do not wait to run the tool just before you start the UIP!
The remainder of this article steps through using the System Evaluation tool. If videos are more your thing, you can watch Gerry Nelson demonstrate the tool in his Update SAS 9.4 with System Evaluation YouTube tutorial, part of the SAS Technical Insights channel.
How do I use it?
Step 1: Download the tool
The first step is to download the SAS Content Assessment package from SAS Downloads. There is a version for Linux and a separate version for Windows.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
The deployment I have includes a Windows metadata tier, a Linux compute tier, and a Linux middle tier so I downloaded the appropriate version to each of the three machines.
On the Windows machine I used the File Explorer to unzip the SAS Content Assessment package. On each of the Linux machines I extracted the archive using this command:
tar zxf /tmp/cat/SASContentAssessment.v1.5.lax.tgz
which resulted in the following structure:
-rwxr-xr-x. 1 sasinst sas 7266304 Aug 18 23:20 codeCheck
drwxr-xr-x. 5 sasinst sas 4096 Aug 18 23:20 codechk
drwxr-xr-x. 3 sasinst sas 4096 Aug 18 23:20 data
drwxr-xr-x. 2 sasinst sas 4096 Aug 18 23:20 doc
-rwxr-xr-x. 1 sasinst sas 7733248 Aug 18 23:20 inventoryContent
drwxr-xr-x. 2 sasinst sas 4096 Aug 18 23:20 maps
-rw-------. 1 sasinst sas 897 Aug 18 23:20 metaparms.sas
drwxr-xr-x. 5 sasinst sas 4096 Aug 18 23:20 packages
-rw-r--r--. 1 sasinst sas 630 Aug 18 23:20 pathslist.txt
drwxr-xr-x. 11 sasinst sas 4096 Aug 18 23:20 products
-rwxr-xr-x. 1 sasinst sas 7749632 Aug 18 23:20 profileContent
-rwxr-xr-x. 1 sasinst sas 7385088 Aug 18 23:20 publishAssessedContent
-rw-r--r--. 1 sasinst sas 810 Aug 18 23:20 repos_exclusion_list.sas
-rw-r--r--. 1 sasinst sas 3482 Aug 18 23:20 setenv.yaml
-rwxr-xr-x. 1 sasinst sas 9814016 Aug 18 23:20 systemEvaluation
drwxr-xr-x. 3 sasinst sas 4096 Aug 18 23:20 utilities
As mentioned earlier, the systemEvaluation tool is delivered alongside other tools in the SAS Content Assessment package.
The doc subdirectory contains two documents you should refer to for version specific details before you run any of the Content Assessment tools
- SASContentAssessment.pdf which has detailed information on the Code Check, Inventory, and Profiling tools which are useful for detailing existing content in the SAS 9 deployment.
- SASContentAssessment-SystemEvaluation.pdf which has details for just the System Evaluation tool.
Step 2: Modify two configuration files
The second step is to edit metaparms.sas and setenv.yaml and provide deployment specific values.
Metaparms.sas contains (surprise, surprise!) parameters to connect to the metadata server as well as a few values to help identify and label the results of the scan. The only parameter that changes from machine to machine is the METADATALABEL. While this can be any text you like, the value will be used to identify the results so I chose to label with the short name of the host on which I ran the test. Using this approach, it will be easy to know which scan results to associate with each of the three machines in my deployment. I provided the values marked with the blue bubble.
The parameters in setenv.yaml specify the location of SASHome and the SAS Configuration directories on the current machine as well as directories for results. Most parameters for results should usually be ok to leave as the defaults which is to collect results in subdirectories of where the systemEvaluation tool is located. The only parameters I supplied are marked with a blue bubble.
With v1.5, you will notice a new ASSESSMENT_SWEEPER_FULL_MODE parameter at the bottom of setenv.yaml. The sweeper is a super-powered collection mechanism that combs the SAS Configuration directories for supplemental information that could be helpful to Technical Support to have in the results package. When set to true, the sweeper will augment the results package with
- response files and logs from the SAS Deployment Wizard
- additional configuration files and logs from the Web directory.
Depending on whether or not the logs in your SAS 9 deployment have been properly managed, it is certainly possible that including extensive logs from the /Web directory will inflate the size of your result package. While everything in the result package is compressed, it is still possible that the size of the results could tax the available disk storage you have available. With my deployment, setting ASSESSMENT_SWEEPER_FULL_MODE to false reduced the size of my result package from 23Mb to 13Mb. Unless you are extremely limited on disk space, I recommend you keep the sweeper set to full mode so you can provide Technical Support with all they may need to assess the reported issues.
Step 3: Run the tool on each machine
Now all I have to do is run the systemEvaluation tool. For example, on my Linux-based midtier machine I ran
./systemEvaluation
At this point, the Linux console displays informational message to let me know what is being done and notifies me of any issue found.
The start of the log shows the success of several setup steps and a self-check on the tool. Depending on the machine being scanned, you may see other tests such as validation of the SAS license or connections to the metadata server. You are also provided with information about the physical machine such as the number of CPU cores, Operating System information, and memory.
Beginning the SAS System Evaluation process: [20Aug20:15:35:13]
Validating required user input.
Validating required user input: SUCCESS
Initializing evaluation directories.
Initializing evaluation directories: SUCCESS
Determining system characteristics.
SASHOME: /opt/sas/SASHome
SAS Config Directory: /opt/sas/config/Lev1
Files in SAS Config Directory: 306612
Operating System: Linux
CPU(s): 2
Thread(s) per core: 1
Core(s) per socket: 1
CPU Socket(s):
NUMA Node(s): 0,1
Kernel Name: Linux
Kernel Release: 2.6.32-504.16.2.el6.x86_64
Kernel Version: #1 SMP Tue Mar 10 17:01:00 EDT 2015
System Locale: en_US.UTF-8
Total System Memory: 16334400 kB
Available System Memory: 4414072 kB
Determining system characteristics: SUCCESS
Verifying that /tmp is not mounted with -noexec.
Verifying that /tmp is not mounted with -noexec: SUCCESS
Verifying that there are no incomplete SAS deployments on this system.
Verifying that there are no incomplete SAS deployments on this system: SUCCESS
Processing the SAS Deployment Registry.
Processing the SAS Deployment Registry: SUCCESS
Initializing the SAS 9 Content Assessment Framework.
Initializing the SAS 9 Content Assessment Framework: SUCCESS
Validating the SAS 9 Content Assessment Framework.
....................
Validating the SAS 9 Content Assessment Framework: SUCCESS
With the framework in place, the tests begin. In this next section of log you can see that two issues were flagged for my middle tier machine
- status.xml contains references to multiple orders
- gemfire.member-timeout property was not set for any of my managed servers
- Configuration logs show evidence of ReportFailures, CheckForErrors, or BUILD FAILED messages.
Determining the SAS mid-tier connection information.
Determining the SAS mid-tier connection information: SUCCESS
Processing configuration.properties for the SAS configuration directory /opt/sas/config/Lev1.
Processing configuration.properties for the SAS configuration directory /opt/sas/config/Lev1: SUCCESS
Processing metadata.connection.properties for the SAS configuration directory /opt/sas/config/Lev1.
Processing metadata.connection.properties for the SAS configuration directory /opt/sas/config/Lev1: SUCCESS
Determining and validating the last SAS Software order applied to the specified SAS configuration directory /opt/sas/config/Lev1.
Determining and validating the last SAS Software order applied to the specified SAS configuration directory /opt/sas/config/Lev1: SUCCESS
Validating all SAS software orders applied to the specified SAS configuration directory /opt/sas/config/Lev1.
Validating all SAS software orders applied to the specified SAS configuration directory /opt/sas/config/Lev1: SUCCESS
Processing status.xml for the SAS configuration directory /opt/sas/config/Lev1.
There appear to be multiple orders in the status.xml file for the specified SAS configuration directory.
Processing status.xml for the SAS configuration directory /opt/sas/config/Lev1: WARNINGS
Validating that the specified configuration directory is not shared improperly.
Validating that the specified configuration directory is not shared improperly: SUCCESS
Validating the host name associated with this configuration directory.
Validating the host name associated with this configuration directory: SUCCESS
Validating the dependent SAS metadata components for the SAS configuration directory /opt/sas/config/Lev1.
Validating the dependent SAS metadata components for the SAS configuration directory /opt/sas/config/Lev1: SUCCESS
Validating the SAS web application servers associated with this configuration directory.
Validating the SAS web application servers associated with this configuration directory: SUCCESS
Evaluating the SAS Content Server.
Evaluating the SAS Content Server: SUCCESS
Evaluating SAS web applications.
Creating CSVs for SAS web applications evaluations.
Evaluating SAS web applications: SUCCESS
Determining SASLogon externalization status.
Determining SASLogon externalization status: SUCCESS
Validating cache locator timeout settings.
Validating settings in: /opt/sas/config/Lev1/Web/WebAppServer.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer12_1/Backup/SASServer12_1.2015-09-10-19.20.27/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer12_1/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer13_1/Backup/SASServer13_1.2015-09-10-19.20.31/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer13_1/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/Backup/SASServer1_1.2015-09-10-19.20.21/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer1_1/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer2_1/Backup/SASServer2_1.2015-09-10-19.20.36/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/WebAppServer/SASServer2_1/bin/setenv.sh. The gemfire.member-timeout property is recommended to be set to 600000.
Validating settings in: /opt/sas/config/Lev1/Web/gemfire/instances.
The gemfire.member-timeout property was not set in /opt/sas/config/Lev1/Web/gemfire/instances/ins_41415/gemfire-start-locator-sas.sh. The gemfire.member-timeout property is recommended to be set to 600000.
Validating cache locator timeout settings: WARNINGS
Evaluating the configuration logs for the SAS configuration directory /opt/sas/config/Lev1.
The existence of "ReportFailures", "CheckForErrors", or "BUILD FAILED" in logs indicate the following applications may not be configured properly:
vfabrchyperc (/opt/sas/config/Lev1/Logs/Configure/vfabrchyperc_postConfigure_2015-06-15-15.33.57.log)
vfabrictcsvr (/opt/sas/config/Lev1/Logs/Configure/vfabrictcsvr_checkForErrors_2015-06-15-12.22.52.log)
vfabrictcsvr (/opt/sas/config/Lev1/Logs/Configure/vfabrictcsvr_configure_2015-06-15-12.22.44.log)
Evaluating the configuration logs for the SAS configuration directory /opt/sas/config/Lev1: WARNINGS
Stopping the SAS 9 Content Assessment Framework.
Stopping the SAS 9 Content Assessment Framework: SUCCESS
The next section of logging tells me that the tool is collecting supporting information to investigate the issues and creating an archive for me for easy sharing with Tech Support. As you can see, with the sweeper set to full mode the tool collected 576 files from various locations for my results. Again, this minimizes the need for Tech Support to come back to you for more information in the middle of an assessment.
Most importantly, note that at the end of the messages there is a reference to a Summary Report that should be reviewed for next steps. We will look at that later.
Issues were encountered in the evaluation of the SAS configuration directory /opt/sas/config/Lev1 and can potentially impact upgrading to new versions of SAS software.
Additional information will now be collected and collated.
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
..............
Adding all collected information for the SAS configuration directory /opt/sas/config/Lev1 to CollectedResults_midtier01_Lev1.tgz
Archiving all files (576) in the results directory: /tmp/assessmentTool/results/evaluation/midtier01.
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................................................................................
................
Archiving all files (576) in the results directory /tmp/assessmentTool/results/evaluation/midtier01: SUCCESS
All collected information has been archived in /tmp/assessmentTool/results/evaluation/midtier01/CollectedResults_midtier01_Lev1.tgz.
Please contact SAS Technical Support for further assistance.
Finalizing the evaluation data mart
Finalizing the evaluation data mart: SUCCESS
Recording status and results.
Recording status and results: SUCCESS
Creating the HTML reports.
Creating HTML reports: SUCCESS
Reports can be found here:
/tmp/assessmentTool/results/evaluation/midtier01/SummaryReports/SummaryReport_midtier01_Lev1.html
Logging information for this system evaluation execution can be found here: /tmp/assessmentTool/logs/evaluation/midtier01/systemEvaluation_midtier01_Lev1_2020-08-20-15.35.13.log
SAS System Evaluation process complete: [20Aug20:15:36:03]
The results of the scan are collected in assessmentTool/results/evaluation/midtier01. One of the recent improvements of System Evaluation is a cleaner results directory which helps reduce confusion for what needs to be sent to Technical Support. After the run above, I see the following files listed.
-rwx------. 1 sasinst sas 23695385 Aug 20 15:36 CollectedResults_midtier01_Lev1.tgz
drwxr-xr-x. 4 sasinst sas 4096 Aug 20 15:35 SummaryReports
-rw-r--r--. 1 sasinst sas 56321 Aug 20 15:35 systemEvaluation_midtier01_Lev1_2020-08-20-15.35.13.log
-rw-r--r--. 1 sasinst sas 778 Aug 20 15:35 SystemEvaluationResults.properties
The CollectedResults_midtier01_Lev1.tgz archive contains everything swept up from the configuration directory in support of detected issues. You can see that the archive is fairly sizable but it contains a lot of supporting information that Technical Support will need to assess reported issues. If issues are identified, this .tgz file is the payload from the System Evaluation tool that is to be shared with Tech Support.
The only other items in the results directory are the Summary reports and the properties file and log file from the System Evaluation tool itself.
Step 4: Review the Summary Report
As noted earlier, the SummaryReports directory in the results directory contains an attractive HTML report that recaps all of the tests that were performed, any issues identified, and in some cases, links to resources that can be used to resolve issues with known solutions. You should carefully review the results of the SummaryReport and try to resolve any issues you can before contacting Technical Support.
The top portion of the report lists the number of issues identified, captures physical metrics of the machine, and provides links to SAS documentation that can be used to fix some of the known issues.
The remainder of the report will list the issues found by the tool. You will see two basic types of issues:
- Those that have known solutions and can be handled without involving Technical Support
- Issues that require a review by Technical Support to assess.
For example, one of the issues identified in my deployment was for the gemfire.member-timeout issue. The summary report notes that there is a link at the top of the report to a SAS Note that will help me resolve this issue on my own so this is one I can take care of myself.
In comparison, the issue regarding errors found in the configuration logs directs me to consult with Technical Support to assess the seriousness of the problem.
For a complete evaluation, I need to repeat steps 1 through 3 on both the Windows metadata tier and the Linux compute tier. At the completion of my scans I will have an archive file for each host for which at least one issue has been identified. If the scan does not find anything amiss on a host there is no .tgz archive. Instead, there is simply a message stating that no issues were found. In my case, all three hosts displayed at least the double order issue so I have these three result archives that comprise my complete result set:
- CollectedResults_meta01_Lev1.tgz
- CollectedResults_compute01_Lev1.tgz
- CollectedResults_midtier01_Lev1.tgz.
The next to last message in the console log informs me that a more detailed log of what the tool did has been collected for me in /tmp/cat/assessmentTool/logs/evaluation/midtier01/systemEvaluation-2020-08-20-15.35.13.log. I will not drag you through the details of this log but I do want to point out that at the very end of this log, there is a summary of the errors found and each error is tagged with a specific symbolic value (e.g., SASCA9SE-STATUSXML, SASCA9SE-LOGCHCK). These codes will help Technical Support identify the specific error test that emitted the issue.
Step 5: Partner with Tech Support to evaluate the results
At this point, I would collect the archive file from each of my hosts and bundle them off to Tech Support. An analyst will partner with you to interpret the impact and severity of each issue and work with you to rectify the problems if need be. When you open a track with Technical Support, please provide the following information:
- Your site number
- Your name and contact information
- Version of SAS
- Operating System
- Reason for the track (Typically, "we are upgrading from SAS 9.4mX to SAS 9.4mY")
The important point here, though, is the System Evaluation tool has identified a handful of issues that might negatively impact an UIP on my deployment. I can proactively work with Tech Support to address the serious issues before I do the UIP and hopefully improve my chances of success.
Wrapping up
So now you know about the System Evaluation tool. As you are planning your next SAS 9 upgrade, please take advantage of this new tool to be aware of issues that might impact an UIP approach.
A: Unfortunately, no, there is little chance of a tool catching every possible issue. The point is that you now have a way to identify and remediate some of the more common issues before attempting an UIP.
A: No, but at least you will be forewarned of issues before you start the process. Over time, as tests are added for additional issues, the hope is that more UIP attempts will go smoother.
Give it a try and feel free to share your experiences in the comments.
Hi All,
The SAS EBI server with 9.4 M7 on AIX LPAR platform.
I am trying to run the system evaluation tool in SAS platform, able to run successfully in meta server got reports, but not in App server. getting following error:
I see below errors in "/opt/sas/assessment/logs/evaluation/prodmeta/init.log":
94 /* Define a libname to parse the repositories xml file */
95 libname datafile xmlv2 xmlmap=mapfile automap=replace;
ERROR: The creation of the XML Mapper file failed.
ERROR: Error in the LIBNAME statement.
96
97 /* Copy our repository datasets to the OUTFILES library */
98 libname outfiles "&outfilesdir";
NOTE: Libref OUTFILES was successfully assigned as follows:
Engine: V9
Physical Name: /opt/sas/assessment/assessment/outfiles/evaluation/NP2
99 proc copy in=datafile out=outfiles;
ERROR: /opt/sas/assessment/assessment/maps/repos.map (A file or directory in the path name does not exist.)
100 run;
ERROR: Libref DATAFILE is not assigned.
NOTE: Statements not processed because of errors noted above.
NOTE: PROCEDURE COPY used (Total process time):
real time 0.04 seconds
cpu time 0.01 seconds
NOTE: The SAS System stopped processing this step because of errors.
101
102 /* Add repository ID reference for later DIS processing */
103 data outfiles.repository;
104 set outfiles.repository;
3 The SAS System Wednesday, September 27, 2023 02:18:00 PM
ERROR: File OUTFILES.REPOSITORY.DATA does not exist.
105 length REPOS_ID_LEVEL2 $8;
106 REPOS_ID_LEVEL2 = kscan(Repository_Id, 2, '.');
107 run;
NOTE: Numeric values have been converted to character values at the places given by: (Line):(Column).
106:27
NOTE: Compression was disabled for data set OUTFILES.REPOSITORY because compression overhead would increase the size of the data
set.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: Due to ERROR(s) above, SAS set option OBS=0, enabling syntax check mode.
This prevents execution of subsequent data modification statements.
WARNING: The data set OUTFILES.REPOSITORY may be incomplete. When this step was stopped there were 0 observations and 2 variables.
NOTE: DATA statement used (Total process time):
real time 0.00 seconds
cpu time 0.00 seconds
Can someone check and suggest why do I get above error which stopping to generate reports ?
