After restarting aws instances SAS Viya fails to start

BeNur — Sat, 25 Aug 2018 20:10:34 GMT

Hi,

I have a problem with starting SAS Viya 3.3. We installed it from quickstart ( https://aws-quickstart.s3.amazonaws.com/quickstart-sas-viya/templates/sas-viya-master.template.yaml)

Step to reproduce:

1. Stop aws instances (visual.viya.sas, prog.viya.sas, stateful.viya.sas, controller.viya.sas and ansible)

2. Start all aws instances

3. Try to connect to SAS Viya through https://<DOMEN>/SASHome

getting 404 error: The requested URL /SASHome was not found on this server.

4. login to visual.viya: sudo /etc/init.d/sas-viya-all-services status

see that some of the services didn't start properly

5. trying to restart sas-viya-all-services
see log

Service start time summary:
SERVICE ELAPSED (+/-10)
sas-viya-consul-default 00:00:20
sas-viya-sasdatasvrc-postgres-node2-ct-pg_hba 00:00:20
sas-viya-sasdatasvrc-postgres-node2-ct-postgresql 00:00:20
sas-viya-vault-default 00:00:20
sas-viya-sasdatasvrc-postgres-node2 00:00:20
sas-viya-httpproxy-default 00:00:21
sas-viya-alert-track-default 00:00:21
sas-viya-authorization-default Failed!
sas-viya-cachelocator-default 00:00:44
sas-viya-cacheserver-default 00:01:07
sas-viya-identities-default Failed!
sas-viya-ops-agent-default 00:00:22
sas-viya-saslogon-default Failed!
sas-viya-watch-log-default 00:00:23
sas-viya-cas-management-default 00:01:14
sas-viya-files-default Failed!
sas-viya-folders-default Failed!
sas-viya-preferences-default Failed!
sas-viya-relationships-default Failed!
sas-viya-transformations-default Failed!
sas-viya-types-default Failed!
sas-viya-appregistry-default Failed!
sas-viya-audit-default Failed!
sas-viya-cas-access-management-default 00:01:41
sas-viya-cas-formats-default 00:01:14
sas-viya-cas-license-default 00:01:41
sas-viya-casproxy-default 00:01:41
sas-viya-casrowsets-default 00:01:15
sas-viya-collections-default Failed!
sas-viya-comments-default Failed!
sas-viya-credentials-default Failed!
sas-viya-crossdomainproxy-default 00:00:58
sas-viya-dataflowexecutionprovider-default 00:00:58
sas-viya-dataplans-default Failed!
sas-viya-datasources-default Failed!
sas-viya-datatables-default 00:00:57
sas-viya-deploymentBackup-default Failed!
sas-viya-analytics-components-default Failed!
sas-viya-analytics-flows-default Failed!
sas-viya-analytics-gateway-default 00:01:06
sas-viya-analytics-localization-default Failed!
sas-viya-analytics-pipelines-default Failed!
sas-viya-analytics-validation-default Failed!
sas-viya-annotations-default Failed!
sas-viya-backup-agent-default Failed!
sas-viya-backupmanager-default 00:01:02
sas-viya-businessrules-default Failed!

The timeout period expired with services remaining to be started:
sas-viya-authorization-default
sas-viya-identities-default
sas-viya-saslogon-default
sas-viya-files-default
sas-viya-folders-default
sas-viya-preferences-default
sas-viya-relationships-default
sas-viya-transformations-default
sas-viya-types-default
sas-viya-appregistry-default
sas-viya-audit-default
sas-viya-collections-default
sas-viya-comments-default
sas-viya-credentials-default
sas-viya-dataplans-default
sas-viya-datasources-default
sas-viya-deploymentBackup-defaul

also I see this message in log:
The timeout period expired with services remaining to be started:
sas-viya-authorization-default
sas-viya-identities-default
sas-viya-saslogon-default
sas-viya-files-default
sas-viya-folders-default
sas-viya-preferences-default
sas-viya-relationships-default
sas-viya-transformations-default
sas-viya-types-default
sas-viya-appregistry-default
sas-viya-audit-default
sas-viya-collections-default
sas-viya-comments-default
sas-viya-credentials-default
sas-viya-dataplans-default
sas-viya-datasources-default
sas-viya-deploymentBackup-default
sas-viya-analytics-components-default
sas-viya-analytics-flows-default
sas-viya-analytics-localization-default
sas-viya-analytics-pipelines-default
sas-viya-analytics-validation-default
sas-viya-annotations-default
sas-viya-backup-agent-default
sas-viya-businessrules-default

The same situation . with services on another the servers - prog.viya.sas, stateful.viya.sas, controller.viya.sas

p.s

systemctl show sas-viya-all-services.service | grep ^Timeout
TimeoutStartUSec=1h
TimeoutStopUSec=1h

Re: After restarting aws instances SAS Viya fails to start

PaulHomes — Sun, 26 Aug 2018 23:49:52 GMT

Increasing the sas-viya-all-services timeout, as you did, worked for me when I deployed SAS Viya 3.2 - as described at https://platformadmin.com/blogs/paul/2017/04/nudging-sas-viya-services/

However, when I recently deployed SAS Viya 3.4 I noticed that there is now a prerequisite step to increase the default timeout for all services (also documented for SAS Viya 3.3). For more info see SAS® Viya® 3.4 for Linux: Deployment Guide > Pre-installation Tasks > Perform Linux Tuning > Change the Default Time-outs.

After making those changes I would stop all SAS Viya services, confirm there are no more SAS processes running and then re-start them.

Before doing that, I would also recommend looking at the logs for the services that failed to start (sas-viya-authorization-default, sas-viya-identities-default, sas-viya-saslogon-default etc.) to see if there are clues as to why they failed to start.

Re: After restarting aws instances SAS Viya fails to start

BeNur — Tue, 04 Sep 2018 01:07:44 GMT

Hi Paul,

thank you for your response

Yes, I have already seen your article.

It this moment we don't have an issue anymore since we reinstalled viya

I guess the problem was somehow connected with memory utilization

also this sas note http://support.sas.com/kb/62/065.html was useful when after restarting some services' status was down

topic After restarting aws instances SAS Viya fails to start in Administration and Deployment

After restarting aws instances SAS Viya fails to start

Re: After restarting aws instances SAS Viya fails to start

Re: After restarting aws instances SAS Viya fails to start