A connection to the SAS Compute Server was unavailable

[elton_]

Hello Community!

I have no experience in SAS Viya administration, but I am trying to understand what is happening with my environment.

SAS: Viya 4

Product: SAS® Studio

Version: Stable 2022.1.1

When I'm trying to open the SAS Studio, I'm receiving this error: ✖️ SAS Studio compute context. A connection to the SAS Compute Server was unavailable.

I have no idea what is going on, because I don't know where I can get the logs.

Does anyone know where it is?

All my services is healthy and running, I've already tried to "kubectl -n viya4 delete --all pods", but without success

NAME                                                              READY   STATUS      RESTARTS   AGE
openldap-5fc5875cf8-szd2g                                         1/1     Running     0          18m
sas-activities-848ff8ccdf-jvfkk                                   1/1     Running     4          18m
sas-annotations-584c79f968-6vqmf                                  1/1     Running     5          18m
sas-app-registry-c694865d4-p5hw2                                  1/1     Running     4          18m
sas-arke-85cd8f89c9-vjmqm                                         1/1     Running     0          18m
sas-audit-56976bb9f5-8xdpg                                        1/1     Running     4          18m
sas-authorization-57d79cccc-8vkdc                                 1/1     Running     0          18m
sas-batch-7b8b7bdc49-glb6n                                        1/1     Running     5          18m
sas-cachelocator-0                                                1/1     Running     0          17m
sas-cachelocator-1                                                1/1     Running     0          12m
sas-cacheserver-0                                                 1/1     Running     0          17m
sas-cacheserver-1                                                 1/1     Running     0          18m
sas-cas-control-7599cd745b-6dbzc                                  1/1     Running     2          18m
sas-cas-operator-6959c5bc8-mrzwl                                  1/1     Running     0          18m
sas-cas-server-default-controller                                 3/3     Running     0          13m
sas-catalog-services-665f947bb6-f9hb8                             1/1     Running     5          18m
sas-code-debugger-68bd4db4b6-bqr6b                                1/1     Running     4          18m
sas-code-debugger-app-7fd944fd4d-bzxf8                            1/1     Running     4          18m
sas-comments-d4fdfd687-fh9sx                                      1/1     Running     4          18m
sas-compute-6fb7c446b-6jdng                                       1/1     Running     4          18m
sas-config-reconciler-5598446dc5-w2wbk                            1/1     Running     1          18m
sas-configuration-5b4f475695-r9rpw                                1/1     Running     4          18m
sas-connect-7485cff95-5xrrx                                       1/1     Running     4          18m
sas-connect-spawner-5dc5979857-44d9d                              1/1     Running     0          18m
sas-consul-server-0                                               1/1     Running     0          17m
sas-consul-server-1                                               1/1     Running     0          17m
sas-consul-server-2                                               1/1     Running     0          17m
sas-conversation-designer-app-5df6b6c499-nxwj2                    1/1     Running     4          18m
sas-credentials-794bbd6dd9-p2dpp                                  1/1     Running     0          18m
sas-crossdomainproxy-7cc8c59868-24xvs                             1/1     Running     1          18m
sas-crunchy-data-postgres-6986c9c54d-fdhdk                        3/3     Running     0          18m
sas-crunchy-data-postgres-backrest-shared-repo-6b9dc48b85-5r88r   1/1     Running     0          18m
sas-crunchy-data-postgres-jpyx-7868849b5-74lbn                    3/3     Running     0          18m
sas-crunchy-data-postgres-ocqa-6ccdbf6d78-sqtxg                   3/3     Running     0          18m
sas-crunchy-data-postgres-operator-7b8cdd46cc-dwk5f               4/4     Running     0          18m
sas-data-explorer-app-54df6db95d-qcldk                            1/1     Running     0          18m
sas-data-flows-7cfb488748-tk8s4                                   1/1     Running     5          18m
sas-data-plans-6bcc6fd5d7-4gxhd                                   1/1     Running     0          18m
sas-data-profiles-67db95774d-slfbz                                1/1     Running     4          18m
sas-data-quality-services-76fbfb6759-8smvb                        1/1     Running     5          18m
sas-data-server-operator-6b7cb9b555-r8fx8                         1/1     Running     0          18m
sas-data-sources-65c59786fc-c4lxm                                 1/1     Running     5          18m
sas-data-studio-app-75598c8768-z5rsl                              1/1     Running     4          18m
sas-deployment-data-7566755648-r72x5                              1/1     Running     4          18m
sas-device-management-9dfbfb4b4-jmtmz                             1/1     Running     4          18m
sas-drive-app-745f8cdcc7-5sv59                                    1/1     Running     4          18m
sas-drive-d6b8d6b94-8xwh6                                         1/1     Running     5          18m
sas-environment-manager-app-6b6664d9b8-lh6c7                      1/1     Running     4          18m
sas-feature-flags-69b899756d-wf4jd                                1/1     Running     4          18m
sas-files-6c5f858855-j6x92                                        1/1     Running     0          18m
sas-folders-cc76bfdf7-7fzpx                                       1/1     Running     5          18m
sas-fonts-8dd68c6f4-ng6m2                                         1/1     Running     4          18m
sas-formats-5d9b6fdd9d-v7l79                                      2/2     Running     4          18m
sas-geo-enrichment-9947986fb-fzrxk                                1/1     Running     4          18m
sas-graph-builder-app-69bbdbb68d-dmh7w                            1/1     Running     4          18m
sas-graph-templates-5858c6b48-lzhtj                               1/1     Running     5          18m
sas-identities-5597c5fb65-qmc2v                                   1/1     Running     0          18m
sas-import-9-5d47494c85-fhd4x                                     1/1     Running     0          18m
sas-information-catalog-app-6c774997c7-kxdcw                      1/1     Running     4          18m
sas-insights-79f8489bfb-4zz5d                                     2/2     Running     4          18m
sas-job-execution-56db44fd78-lghqh                                1/1     Running     0          18m
sas-job-execution-app-55cfb4f667-6q968                            1/1     Running     0          18m
sas-job-flow-scheduling-76c5677f98-dcsxp                          1/1     Running     0          18m
sas-launcher-c4d545688-5crfr                                      1/1     Running     0          18m
sas-lineage-app-6f5cb8bb8d-tw8j9                                  1/1     Running     5          18m
sas-links-6f6f88d958-d2d78                                        1/1     Running     4          18m
sas-localization-6b79787fc6-cvjkv                                 1/1     Running     0          18m
sas-logon-app-54bdc6fbcd-qmkld                                    1/1     Running     0          18m
sas-mail-5cd6db7746-9bv5c                                         1/1     Running     4          18m
sas-maps-d86c6d99c-b98vl                                          1/1     Running     4          18m
sas-model-repository-6bd44db6fd-wv54j                             1/1     Running     0          18m
sas-natural-language-conversations-86b4d49fd8-kfwcc               1/1     Running     0          18m
sas-natural-language-generation-98b4d5757-ggr7x                   1/1     Running     0          18m
sas-natural-language-understanding-6cd9c68d5c-vbl5c               1/1     Running     0          18m
sas-notifications-89d47f7cb-knzxq                                 1/1     Running     4          18m
sas-office-addin-app-7fc747fc8f-g969c                             1/1     Running     4          18m
sas-opendistro-default-0                                          1/1     Running     0          17m
sas-opendistro-exporter-7bd58f9977-pmcft                          1/1     Running     0          18m
sas-opendistro-operator-86f7576957-ws2ds                          1/1     Running     0          18m
sas-preferences-57ddd49f55-hk2ql                                  1/1     Running     4          18m
sas-prepull-7fcb7894d9-pwvv6                                      1/1     Running     0          18m
sas-rabbitmq-server-0                                             1/1     Running     0          17m
sas-rabbitmq-server-1                                             1/1     Running     0          17m
sas-rabbitmq-server-2                                             1/1     Running     0          18m
sas-readiness-656c664cf4-8jds2                                    1/1     Running     0          18m
sas-report-distribution-6685b85f66-522vb                          1/1     Running     5          18m
sas-report-execution-756684648b-vsbpv                             1/1     Running     0          18m
sas-report-renderer-7dbb58587b-ddl58                              1/1     Running     4          18m
sas-report-services-group-657c967d47-xwn5n                        1/1     Running     0          18m
sas-scheduler-64dc4b5f7d-2nsjn                                    1/1     Running     0          18m
sas-score-definitions-6c86bd86ff-nr7m8                            1/1     Running     5          18m
sas-score-execution-79f77f7977-xzc7v                              1/1     Running     5          18m
sas-search-6866f45696-xqxdp                                       1/1     Running     5          18m
sas-studio-5bb6bccdf8-2wzrr                                       1/1     Running     4          18m
sas-studio-app-89ff89546-chrwt                                    1/1     Running     0          18m
sas-studio-development-77d488f8df-8bcrr                           1/1     Running     4          18m
sas-templates-755d45c45f-svhxn                                    1/1     Running     0          18m
sas-theme-content-6c49c4bc49-9ccw5                                1/1     Running     4          18m
sas-theme-designer-app-5865867544-rzpqd                           1/1     Running     4          18m
sas-themes-7c8c4dbf9-7pmhp                                        1/1     Running     5          18m
sas-thumbnails-7656bffbb7-7bf52                                   1/1     Running     4          18m
sas-transfer-748d9d9ff-hg96t                                      1/1     Running     0          18m
sas-transformations-84fb5bcbbd-vfbrl                              1/1     Running     0          18m
sas-types-79d5b9ddcb-lcbcg                                        1/1     Running     4          18m
sas-visual-analytics-6df578f5d9-jc748                             1/1     Running     4          18m
sas-visual-analytics-administration-7c76ddf644-slp46              1/1     Running     4          18m
sas-visual-analytics-app-5cd959df4b-rstj5                         1/1     Running     4          18m
sas-web-data-access-868b577858-xc6vs                              1/1     Running     4          18m

[gwootton]

The Compute Server is requested through the compute service (sas-compute) and launched by the launcher service (sas-launcher). SAS Studio (sas-studio) performs the request.

The compute server itself appears as a pod with the name sas-launcher as well, though with a longer tail of characters. The pod is ephemeral though, so if it fails to start it will disappear.

So, the logs for sas-compute, sas-launcher and sas-studio, along with any kubernetes events from the time of the failure may be helpful for troubleshooting.

To get the logs for a pod you would use the command kubectl -n {{namespace}} logs {{pod name}} (this will output the log to stdout so you may want to redirect this to a file). You can view events with kubectl get events.

[doug_sas]

Adding onto what Greg said, the new computer server pod will try to instantiate a sas-programming-environment container. Due to the size of the container, it takes a long time to download the container to a node if it is not already present. The error you are seeing may be because the creation of the container is longer than the compute service is willing to wait (usually 60 seconds).

To get around this problem, the sas-prepull service will make sure sas-programming-environment containers are pre-downloaded to all nodes that have the "workload.sas.com/class=compute" label on the Kubernetes node.

So you can either keep retrying the context in SAS/Studio until the container is fully downloaded to a node or label the node so that the sas-prepull service will put the container on the node for you.