Evironment Manager startup after certificates and passwords update

[PriitL]

Hi!

I need to set up a test environment as similar as possible to current live system. So my approach was to clone the current quite new system (to avoid all the configuration from zero), replace hostnames (+certificates) and changing passwords and cleaning everything up from real data.

* cloning - success

* replacing hostnames - success

* replacing passwords - success (with wizard + manually for cpmowner and cpmuser - not mentioned in manuals, got 404 when starting CPM as it couldn't log to database)

* replaced certificates - half success (because of java keystorage)

* cleaning from real data - half success (because of auto backups)

When cleaning from real data i also need to clear the data from automatic backups (from SASApp and SASApp_VA) stored at config/Lev(n)/Backup/Vault/. As i found out the only way to do that is to set the automatic backups as short as needed (for a day for example) and then it removes those backups by itself. As i understand there are two ways to modify backup- from Management Console or from Enironment Manager. Either way the connection between Environment Manager server (hq-server) and agent (hq-agent) must work or the settings won't reach to SASApp and SASApp_VA servers.

Now the problem is - as the hostnames and passwords has been changed - that Environment Manager connections between hq-server and hq-agents do not work. So to update the certificates (the date for config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/conf/hyperic.keystore file was the same as installation for the original setup, this means the Deployment Manager wizard is not updating all the certificates - should it?) i followed the manual <https://documentation.sas.com/?docsetId=bimtag&docsetTarget=p1fpnnm9hxkhlzn1x5tkqs1caeg5.htm&docsetV... steps "Generate the SAS Environment Manager Keystore with a Self-Signed Certificate" and "Update Certificates for SAS Environment Manager". After several tries it seems that the hq-server at least starts up when i won't delete eam_keystore entries (previous link, title "Delete the Keystore From the Database"). So this means that it still use some records from old setup and probably maybe even password and/or certificates or some salt from those. Or i still haven't replaced all the certificates/passwords.

Anyway, even with this old eam_keystore entry the hq-server eventually starts up and the webpage (7443) asks for credentials. I still get "SAS Environment Manager Login Error" but i could live with that at the moment as i need to set up the connection between hq-server and hq-agents to Apply automatic backup settings to the agents.

After flushing the data/ and log/ directory contents and staring the agent:

Starting HQ Agent...... running (15079).
[ Running agent setup ]
Should Agent communications to HQ be unidirectional [default=no]: yes
What is the HQ server IP address: <hostname>
Should Agent communications to HQ always be secure [default=yes]: Yes
What is the HQ server SSL port [default=7443]: 7443
- Testing secure connection ... Success
What is your HQ login [default=hqadmin]: sasevs@saspw
What is your HQ password: **Not echoing value**
What is the agent IP address [default=x.x.x.x]: <hostname>
- Received temporary auth token from agent
- Registering agent with HQ
- Unable to register agent: Permission denied

and log:

21-09-2020 13:00:09,683 EEST ERROR [autoinventory-scanner] [AutoinventoryCommandsServer@280] Unable to send autoinventory platform data to server, sleeping for 75 secs before retrying. Error: Unable to communicate with server -- provider not yet setup

and hq-server's server.log:

21-09-2020 12:57:46,354 EEST INFO [tomcat-http--30] [com.sas.hyperic.security.CasIdentityRetrievingTicketValidator@61] The logon username return by SSO, before getting current user. is: sasevs@saspw
21-09-2020 12:57:46,356 EEST INFO [tomcat-http--30] [org.jasig.cas.client.proxy.ProxyGrantingTicketStorageImpl@76] No Proxy Ticket found for [].
21-09-2020 12:57:46,359 EEST INFO [tomcat-http--30] [com.sas.hyperic.security.SasSimpleUserDetailsService@97] Userdetailservice provider is:com.sas.hyperic.security.sasserver.SASIdentityServiceProvider@575c6304
21-09-2020 12:57:46,360 EEST INFO [tomcat-http--30] [com.sas.hyperic.security.SasSimpleUserDetailsService@110] User [sasevs@saspw] does not exist in EVM local database.
21-09-2020 12:57:46,360 EEST ERROR [tomcat-http--30] [com.sas.hyperic.security.SasSimpleUserDetailsService@136] java.lang.NullPointerException
21-09-2020 12:57:46,375 EEST INFO [tomcat-http--30] [org.hyperic.hq.ui.security.BaseSessionInitializationStrategy@180] User subject is null, don't auto-create user, program will return without executing afterward logic.

So my first problem is to get rid of those SASApp  and SASApp_VA backups located in Vault directory. Maybe i could just remove those with rm? What metadata thinks about this?

And the second goal would be to get up the connection between Environment Manager server and agents.

I'm i missing something? I know my weak spot could be generating certificates, especially Java keystore ones but maybe there's some hint i just haven't come to.

Yes, the new jks certificate is generated from 1024b RSA key. For testing purposes hq-server.conf server.keystore.password parameter is plain text (to avoid ENC() encoded password miscofiguration. By the way- how do i encrypt the password to ENC()? Tried like described https://docs.oracle.com/cd/E35976_01/server.740/es_admin/src/tadm_ssl_convert_pem_to_jks.html).

Thanks!

PL

[Anand_V]

Try replacing the data directory of EVM agent and then restarting it.

mv SAS-configuration-directory/Web/SASEnvironmentManager/agent-version-EE/data SAS-configuration-directory/Web/SASEnvironmentManager/agent-version-EE/data.old

[PriitL]

Thanks for the reply! I have flushed everything from data/ and log/ after every restart of EVM agent. Still no luck.

[Anand_V]

My bad, I didn't notice you mentioned that just before the log snap. Can you try the steps provided in this KB link?
https://support.sas.com/kb/58/621.html

[PriitL]

Unfortunately not. I can't log in to EVM, it authenticates but shows Login Error.
SAS Environment Manager Login Error
One of these conditions might have occurred:

- You are not a member of a SAS Environment Manager group that has access to the application.
- The SAS identity service address is not correctly set to either the SAS Web Server address or the proxy address.
- Initial data for users or roles is not present in the SAS Environment Manager database.
- SSL is configured for SAS Web Server and SAS Environment Manager, but SSL is configured incorrectly or the
certificates and keystore are not consistent.

If you continue to receive this error, contact SAS Technical Support.

Sign Out

As i understand it is also irrelevant because automatic backups are configurable from Management Console too. And after all- EVM was working before clone, so the accounts, roles and groups should be fine.

[Anand_V]

Was there any alias or load balancer used for web applications in the initial configuration?

If yes, you will have to update those entries manually.

For hyperic.keystore, yes it has to be generated again with new certificates. I think the steps are documented in the guide you have shared.

As part of updating hostnames there is a file generated for some manual changes which has some steps to be done in the agent configuration, have you performed them?

There is a manual step on the compute server where EVM datamart is hosted to change the hostname manually in the sasev.properties file. Have you performed that as well?

I had a similar issue after cloning of environment and it got resolved after all the required manual steps and flushing data folders.

[PriitL · Accepted solution]

I got the hq-server up and running after following the step-by-step how-to i got form our local support guy.

The problem was badly made keystore certificates, probably. I'll paste it here also, maybe helps someone.

Please follow the steps below to update the SAS Environment Manager certificate (hyperic.keystore) 

1. a) Stop SAS Environment Manager Agent and Server on middle tier.

<SAS_CONFIG>/Lev2/Web/SASEnvironmentManager/server-5.8.0-EE/bin/hq-server.sh stop

<SAS_CONFIG>/Lev2/Web/SASEnvironmentManager/agent-5.8.0-EE/bin/hq-agent.sh stop

1. b) Create a backup copy hypreic.keystore from the location:

mv <SAS_CONFIG>/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/conf/hyperic.keystore <SAS_CONFIG>/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/conf/hyperic.keystore.bkup

1. c) Run the following command to create a PKCS12 keystore :

cd <SASHome>\SASWebServer\9.4\httpd-2.X.X.X-64\bin>

openssl pkcs12 -export -chain -inkey <saswebserver.key> -in <saswebserver.crt> -name hq -CAfile <Chaincerts.txt> -out /tmp/hyperic.p12

Note: saswebserver.key , saswebserver.crt and Chaincerts.txt need to be replaced with the actual files which represent your SAS Web Server Key, SAS Web Server certificate and chain(if it exists) respectively

Chaincerts.txt – is a file created by concatenating the contents of the root, intermediate and server certificates in a single file

When prompted for a password, enter hyperic

1. d) Convert the PKCS12 keystore to Java Keystore to be used by SAS Environment Manager:

cd <SASHome> \ SASPrivateJavaRuntimeEnvironment \ 9.4 \ jre \ bin>

keytool -importkeystore -deststorepass hyperic -destkeypass hyperic -destkeystore <SASConfig>/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/conf/hyperic.keystore  -srckeystore /tmp/hyperic.p12 -srcstoretype PKCS12 -srcstorepass hyperic -alias hq

Note: SASConfig needs to be replaced with the actual path to the configuration folder.

Ignore any warning message that you get, so long as the hyperic.keystore file is generated correctly , you are good

1. e) Login to machine where you have SAS Web Infrastructure Platform Database Server (usually compute tier) and set the following path variables :

POSTGRES_HOME=<SAS_HOME>/SASHome/SASWebInfrastructurePlatformDataServer/9.4

export PATH=${POSTGRES_HOME}/bin:$PATH

export LD_LIBRARY_PATH=${POSTGRES_HOME}/lib:$LD_LIBRARY_PATH

where SAS_HOME is the actual folder path of SAS Installation Folder.

Run the following PSQL command

• psql -h localhost -p 9432 -U EVManager -c "select * from public.eam_keystore;"

That table should list one entry/row.

If that row exists, then issue the following psql command to delete that row:

• psql -h localhost -p 9432 -U EVManager -c "delete from eam_keystore;"

1. f) Return back to the machine where you have SAS Middle Tier machine and restart SAS Environment Manager Server and agent

<SAS_CONFIG>/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/bin/hq-server.sh start

<SAS_CONFIG>/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/bin/hq-agent.sh start

1. g) Please access SAS Environment Manager URL https://<hostname>:7443 on a browser and login using sasadm@saspw account and let us know if you are able to login.

1. h) If you are unable to log in , provide the following:

• Date and time of reproducing the error
• Screenshot of error on the browser
• <SAS_CONFIG>/Lev1/Web/Logs/SASServer1_1/SASLogon9.4.log
• <SAS_CONFIG>/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/logs/server.log
• <SAS_CONFIG>/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/logs/bootstrap.log
• <SAS_CONFIG>/Lev1/SASEnvironmentManager/server-5.8.0-EE/conf/hyperic.keystore
• <SAS_CONFIG>/Lev1/SASEnvironmentManager/server-5.8.0-EE/conf/hq-server.conf
• ls -altr <SAS_CONFIG>/Lev1/SASEnvironmentManager/server-5.8.0-EE/conf