Problem migrating to VA 6.4

[snlbse]

Hello,

I’ve been trying to get a customer’s 6-blade environment upgraded from VA 6.2 to VA 6.4 and failing. I was hoping some of you could take a look at this, and give me your thoughts on it. I installed a clean 6.4 environment, using an SMU package to transfer all metadata.

Customer uses an SSL reverse proxy, so I updated:

• All connection properties in SMC
• Server.xml files for the 3 webapp servers
• Copied the cacerts file with the SSL certificate from the old 6.2 deployment

After installing, I logged on to the VA Hub and everything appeared to be running smoothly, however, as soon as I went to the VA Admin page and ‘Manage Servers’ (or any of the other options there), it failed to load or display anything and just gave this pop-up:

Going through the log files, I see this one popping up quite a few times:
      [VIEWER   ] ERROR [tomcat-http--36] 2014-04-01 11:19:15,801 [ST-32-7uZmnfxNm1P0qKoWtPnN-cas:<username>] mobiledesigner.data.query.LasrQueryRunner - Error of type           ServerNotFound.Connection refused
     Exception caught in LASRClient (lasr=null, host=<hostname>, port=10010)
     java.net.ConnectException: Connection refused
     action=ServerVersionInfo
     Exception caught in LASRClient (lasr=null, host=<hostname>, port=10010)
     java.net.ConnectException: Connection refused
     action=ServerInfo
      [VIEWER   ] ERROR [tomcat-http--36] 2014-04-01 11:19:15,822 [ST-32-7uZmnfxNm1P0qKoWtPnN-cas:<username>] sas.bidata.error.DataErrorHandler - ServerNotFound
     java.net.ConnectException: Connection refused

At first glance I figured there was either something wrong with the LASR license, or with the password used for the sassrv account. Re-applying the license and using the SDM to update the sassrv password didn’t solve anything though.

In the end I reverted the environment back to 6.2 so that the customer could at least go on with their normal production. VA 6.2 is running normally, so the problem must be somewhere in 6.4.

Anyway, that’s the situation. I am very interested in hearing your suggestions or pointers for troubleshooting this issue. Maybe you’ve seen something like this before.
To make your life easier, this is a list of what I have already checked:

• LASR license appears to be correct. I applied it manually using the applylasrlic tool and also by simply running the sassetup tool to update the entire environment.
• The password for the sassrv account was updated and verified to be correct. Used the SDM for this.
• File permissions were set correctly. The entire config and sashome directories are owned by the SAS Installer account, except for the files that need root ownership.
• Metadata permissions are set correctly, for as far as I could check.
• System settings were functioning correctly for VA 6.2, so I assumed this should work in VA 6.4 as well. However, comparing the system settings to the prerequisites mentioned for 6.4, these are the differences:
  • /etc/security/limits.conf uses the old requirements of 65536 and 350000 instead of 100000 and 350000. Also, no limits are set for stack.
  • /etc/security/limits.d/90-nproc.conf has a value of 65536 instead of 100000.
  • /etc/ssh/sshd_config has MaxSessions of 1000, but MaxStartups is not set. Works fine in 6.2 though.
• IPTables is running, but was configured to allow traffic over all the necessary ports. Unless new ports are added, that shouldn’t be the problem.
• SELinux is set to permissive.
• Running gridmon.sh showed all 6 nodes; no problems there.
• Password-less SSH has been correctly set up for the sasinst, lasradm and hadoop accounts.
• Submitting SAS code manually to start a LASR also resulted in an error, saying that a connection to LASR could not be made.

And these are my questions:

• I haven’t updated Hadoop, as I figured it should run just as well on the version shipped with 6.2. Also, the customer hardly uses Hadoop anyway, but instead loads everything directly into memory (I know, I know …). Is there some hard dependency between TKGrid/SAS and Hadoop?
• I haven’t disabled the IPTables service, as it was configured to not be a problem. Besides, I expect other error messages if this is the problem. Am I right in assuming this?
• I think the changes in system settings between 6.2 and 6.4 are not the cause of the problem. Can anyone verify that?
• I’ve not changed the setenv.sh files for the 3 webapp servers, apart from increasing the memory settings in the JVM_OPTS. Should I have made changes there to reflect the SSL proxied address? Documentation doesn’t say so and I was able to log on just fine.
• After seeing a track about this, I have added the sasinst and lasradm accounts to the SAS General Servers group in SMC. Do you see any relation between this and the error?

I would really appreciate some help and/or your thoughts on this.

Kind regards and thanks in advance.

Bram Seldenthuis
Technical Administration Consultant, SAS Application Management

SAS Netherlands

[AndiWindisch]

Hey Bram,

we have some similar problems with our Update 6.2 to 6.3 .

As is can see there is a major change between TKGRid and Hadoop. In the 6.2 Installation thers is a TKGrid_2.1 and with 6.3 came up with TKGrid_2.4 and a new directory TKTGDat_2.4 (for text analysis). There are also changes on the hadoop installation between 6.2 and 6.3.

From my point of view you will have to ensure, that your installation is running quite all correct "versions", in all other cases you will have errors. I have just a short look at 6.4 but i think it comes with TKGrid_2.5.

Hope this helps...

Best regards,
Andreas

Nachricht wurde geändert durch: Andreas Windisch Have a look at <http://support.sas.com/documentation/solutions/hpainfrastructure/25/hpaicg25.pdf>

[snlbse]

Hello Andreas,

Thanks for the answer, but I did actually install TKGrid_2.5 prior to migrating the environment to VA 6.4. The one thing that I didn't do here, was update the Hadoop installation.

If anyone can verify that an updated Hadoop is necessary for the migration or that not doing so can cause the issue I mentioned, I would love to know.

Kind regards,

Bram Seldenthuis

[justin_sas]

We asked Bram to open a Tech Support Track for this.