Tuning the authentication timeout for long-running jobs
Starting with SAS Viya 2021.2.4, you may encounter errors while executing programs that were previously fine. If you have long-running sessions, such as complex analytical models in SAS Model Studio or batch jobs submitted to SAS Workload Management, this may impact you. In this post, we’ll show how it may depend on changes implemented to enforce stricter security by default, and how to solve these issues.
SAS Logon manager is one of the key services for authentication in SAS Viya: it provides both an end-user interface for authentication and internal authentication to other services. This enables single sign-on within the SAS Viya environment between services.
SAS Viya authentication architecture is built around the OAuth 2.0 protocol, and its extension for authentication, OpenID Connect 1.0 (OIDC). According to these protocols, SAS Viya services exchange OAuth tokens to access protected resources. In layman's terms, SAS Viya services use these tokens to prove to each other that user authentication has occurred and is valid.
OAuth tokens include an expiration field; this is a security feature because tokens are usually non-revocable: once they are issued, they must be unconditionally accepted by services until expired.
With security being at the front and center of most SAS decisions, SAS is gradually reducing the default expiration for Access Tokens from the original value of 10 hours. Shorter-lived tokens reduce the window that a compromised token can be used by an attacker. To avoid possible disruption, this change is being implemented in steps. As of the stable release 2021.2.4 (February 2022), the Access Token default lifetime is decreased to 4 hours; in a later release, it will be further reduced to 1 hour.
The default behavior
Most SAS Viya applications are fine with short-lived Access Tokens. In general, it is the responsibility of client applications to interact with any called service using a valid token, i.e., a token that is non-expired for the lifetime of the request. This includes direct requests to called services, and subsequent requests where the service needs to call other services to carry out the request, and those services in turn call other services, and so on.
Most API calls complete quickly, allowing a short-lived access token to be used across many requests.
Client applications that call backend APIs on behalf of end-users (think SAS Studio, Model Studio, etc) usually create user sessions that may last longer than the original Access Token; these clients can use Refresh Tokens to re-authenticate on behalf of the user and get a new token. This ensures that there is always a valid, non-expired token when servicing a request.
Some applications inherently need longer-lived tokens, for example, applications that are involved in batch processing of long-running jobs, where the app may need to make calls to other services throughout the lifecycle of the jobs. These client applications may be registered with an expiration value specific to their needs, and this will have the effect of overriding the system-wide default value when any tokens are issued to the registered client ID. Such an example is SWAT, used to submit code – possibly long-running – from Python clients to CAS.
Why you may need to tune the default settings
There are situations where clients have long-running sessions that try to use expired Access Tokens – and obviously fail. In these cases, the default configuration may not be adequate. Here are a few examples.
- When a batch job is submitted to SAS Workload Management via the sas-viya CLI, it sends the OAuth token of the authenticated user as input along with the job. The sas-viya CLI framework does not refresh tokens for jobs. When jobs sit in a queue for too long, the OAuth token may expire before the job completes and this can result in job failures.
- When executing SAS jobs, either in batch or through interactive interfaces such as SAS Studio, the OAuth token of the authenticated user is provided at the startup of the backend compute session. Long-running jobs may try to use the token after it has already expired, resulting in failures. PROC HTTP or SAS/CONNECT SIGNON are examples of code that can use OAuth tokens to authenticate to SAS Viya services.
- When submitting very large models within SAS Model Studio, some nodes in the pipeline may require more time to execute than the default timeout and this can result in model execution failures.
In these cases, the cause of the problem is not always clear because the message for the authentication failure can be buried in the log of a service called by the front-end client. The client may simply present a generic error to the user, such as “A call to service X has failed” or “the job failed”.
A possible solution may be to increase the timeout for OAuth tokens to more than the longest job executing in that environment. Each customer should evaluate their business and security requirements and tune the environment to accomplish the right balance between them.
How to tune token expiration
The system-wide default token expiration can be changed on a per-deployment basis by SAS administrators in SAS Environment Manager by creating or modifying an existing configuration for the sas.logon.jwt configuration definition. There are two properties involved: policy.accessTokenValiditySeconds, and policy.global.accessTokenValiditySeconds. The latter is meant for multi-tenant deployments and is meant to apply to all tenants. The system-wide default will apply to all applications that are not registered with their own value.
Here is a simple step-by-step guide.
- Log on to SAS Environment Manager as an administrator and opt into Assumable Groups.
- Navigate to the Configuration area and filter for "sas logon". Select SAS Logon Manager, then click New Configuration
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page. - Select sas.logon.jwt
- Modify the following two parameter values, setting them to the desired value in seconds (for example, 108000 corresponds to 30 hours), and save:
- policy.accessTokenValiditySeconds
- policy.global.accessTokenValiditySeconds
Note: assess the max runtime required by the longest running job and set the timeout value accordingly. - Restart the SAS Logon service
# delete the pod(s), kubernetes will restart it kubectl delete pod -l app=sas-logon-app # now wait for the restarted service to be ready kubectl wait --for=condition=Ready pod -l app=sas-logon-app --timeout=180s
Conclusion
In this article we have seen how changes implemented to enhance SAS Viya default security may create failures in jobs that previously run fine. We have described how this may be due to the new default lifetime of OAuth tokens used by SAS Viya services to authenticate between themselves. Finally, we have seen how to tune the timeout to satisfy the requirements of long-running jobs.
Find more articles from SAS Global Enablement and Learning here.
Don't miss out on SAS Innovate - Register now for the FREE Livestream!
Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.
