After restarting aws instances SAS Viya fails to start

[BeNur]

 Hi,

I have a problem with starting SAS Viya 3.3. We installed it from quickstart ( https://aws-quickstart.s3.amazonaws.com/quickstart-sas-viya/templates/sas-viya-master.template.yaml)

Step to reproduce:

 1. Stop aws instances (visual.viya.sas, prog.viya.sas, stateful.viya.sas, controller.viya.sas and ansible)

 2. Start all aws instances

 3. Try to connect to SAS Viya through https://<DOMEN>/SASHome

getting 404 error: The requested URL /SASHome was not found on this server. 

4. login to visual.viya:  sudo /etc/init.d/sas-viya-all-services status

see that some of the services didn't start properly

5. trying to restart sas-viya-all-services
see log

Service start time summary:
SERVICE ELAPSED (+/-10)
sas-viya-consul-default 00:00:20
sas-viya-sasdatasvrc-postgres-node2-ct-pg_hba 00:00:20
sas-viya-sasdatasvrc-postgres-node2-ct-postgresql 00:00:20
sas-viya-vault-default 00:00:20
sas-viya-sasdatasvrc-postgres-node2 00:00:20
sas-viya-httpproxy-default 00:00:21
sas-viya-alert-track-default 00:00:21
sas-viya-authorization-default Failed!
sas-viya-cachelocator-default 00:00:44
sas-viya-cacheserver-default 00:01:07
sas-viya-identities-default Failed!
sas-viya-ops-agent-default 00:00:22
sas-viya-saslogon-default Failed!
sas-viya-watch-log-default 00:00:23
sas-viya-cas-management-default 00:01:14
sas-viya-files-default Failed!
sas-viya-folders-default Failed!
sas-viya-preferences-default Failed!
sas-viya-relationships-default Failed!
sas-viya-transformations-default Failed!
sas-viya-types-default Failed!
sas-viya-appregistry-default Failed!
sas-viya-audit-default Failed!
sas-viya-cas-access-management-default 00:01:41
sas-viya-cas-formats-default 00:01:14
sas-viya-cas-license-default 00:01:41
sas-viya-casproxy-default 00:01:41
sas-viya-casrowsets-default 00:01:15
sas-viya-collections-default Failed!
sas-viya-comments-default Failed!
sas-viya-credentials-default Failed!
sas-viya-crossdomainproxy-default 00:00:58
sas-viya-dataflowexecutionprovider-default 00:00:58
sas-viya-dataplans-default Failed!
sas-viya-datasources-default Failed!
sas-viya-datatables-default 00:00:57
sas-viya-deploymentBackup-default Failed!
sas-viya-analytics-components-default Failed!
sas-viya-analytics-flows-default Failed!
sas-viya-analytics-gateway-default 00:01:06
sas-viya-analytics-localization-default Failed!
sas-viya-analytics-pipelines-default Failed!
sas-viya-analytics-validation-default Failed!
sas-viya-annotations-default Failed!
sas-viya-backup-agent-default Failed!
sas-viya-backupmanager-default 00:01:02
sas-viya-businessrules-default Failed!

The timeout period expired with services remaining to be started:
sas-viya-authorization-default
sas-viya-identities-default
sas-viya-saslogon-default
sas-viya-files-default
sas-viya-folders-default
sas-viya-preferences-default
sas-viya-relationships-default
sas-viya-transformations-default
sas-viya-types-default
sas-viya-appregistry-default
sas-viya-audit-default
sas-viya-collections-default
sas-viya-comments-default
sas-viya-credentials-default
sas-viya-dataplans-default
sas-viya-datasources-default
sas-viya-deploymentBackup-defaul

also I see this message in log:
The timeout period expired with services remaining to be started:
sas-viya-authorization-default
sas-viya-identities-default
sas-viya-saslogon-default
sas-viya-files-default
sas-viya-folders-default
sas-viya-preferences-default
sas-viya-relationships-default
sas-viya-transformations-default
sas-viya-types-default
sas-viya-appregistry-default
sas-viya-audit-default
sas-viya-collections-default
sas-viya-comments-default
sas-viya-credentials-default
sas-viya-dataplans-default
sas-viya-datasources-default
sas-viya-deploymentBackup-default
sas-viya-analytics-components-default
sas-viya-analytics-flows-default
sas-viya-analytics-localization-default
sas-viya-analytics-pipelines-default
sas-viya-analytics-validation-default
sas-viya-annotations-default
sas-viya-backup-agent-default
sas-viya-businessrules-default

The same situation . with services  on another the servers - prog.viya.sas, stateful.viya.sas, controller.viya.sas

p.s 

systemctl show sas-viya-all-services.service | grep ^Timeout
TimeoutStartUSec=1h
TimeoutStopUSec=1h

[PaulHomes · Accepted solution]

Increasing the sas-viya-all-services timeout, as you did, worked for me when I deployed SAS Viya 3.2 - as described at https://platformadmin.com/blogs/paul/2017/04/nudging-sas-viya-services/

However, when I recently deployed SAS Viya 3.4 I noticed that there is now a prerequisite step to increase the default timeout for all services (also documented for SAS Viya 3.3). For more info see SAS® Viya® 3.4 for Linux: Deployment Guide > Pre-installation Tasks > Perform Linux Tuning > Change the Default Time-outs.

After making those changes I would stop all SAS Viya services, confirm there are no more SAS processes running and then re-start them.

Before doing that, I would also recommend looking at the logs for the services that failed to start (sas-viya-authorization-default, sas-viya-identities-default, sas-viya-saslogon-default etc.) to see if there are clues as to why they failed to start.

[BeNur]

Hi Paul,

thank you for your response

Yes, I have already seen your article.

It this moment we don't have an issue anymore since we reinstalled viya

I guess the problem was somehow connected with memory utilization

also this sas note http://support.sas.com/kb/62/065.html was useful when after restarting some services' status was down