SAS Open Model Manager – additional setup and configuration
I was going to call this advanced configuration, but these are really the steps that you should do to create a production or operational ready environment for SAS Open Model Manager.
Recap...
To recap before we get into the details, in the last blog we discussed that the default run_docker_container script is expecting all the input files to be in the same location with set names. The same rules apply with the configuration shown in this post, you will see the default names and my working directory was ‘~/working’. Also, SAS Open Model Manager 1.2 is only supported on Docker, it is not supported running on a Kubernetes platform or on a managed container service such as AWS Elastic Container Service (ECS). As such, the storage examples shown are for a single docker server configuration.
Storage overview and configuration
Generically in a containerized environment you have ephemeral and persistent storage. By default, all files created inside a container are stored on a writable container layer. This means:
- The data doesn’t persist when that container no longer exists, and it can be difficult to get the data out of the container if another process needs it.
- A container’s writable layer is tightly coupled to the host machine where the container is running. You can’t easily move the data somewhere else.
This is the ephemeral storage that I was referring to.
For persistent storage Docker has two options for containers to store files on the host machine, so that the files are persisted even after the container stops. The two methods are: volumes and bind mounts.
If you inspect the ‘run_docker_container’ script you will see that we make use of both storage types, see Figure 3 below. By studying Figure 3, you will see that the license file is mounted using a bind mount and volume mounts are used for the SAS Viya storage.
As discussed in the last post, as part of launching a container a set of volumes are created for your SAS Viya environment, using the naming format <viya_volume>-<container_name>. The following volumes are created:
- casdata
- caspermstore
- consul
- postgres
- sasmmastore
- sasmmsresources
This leads to the following configuration.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
However, for a production environment this isn’t enough, you really want persistent storage for the following:
- The user’s home directory
- Business data and code, and
- Log files.
It is recommended that the log files are on persistent storage. This is possibly an optional step but is recommended from a support and administration perspective.
Let’s look at how we can implement the following configuration.
In this example, we will configure persistent storage for the logs (the /var/log/sas/viya storage), user data and the user’s home directory. Note, to make use of the new location for the user’s home directory you will also need to update the SSSD configuration. More on that later.
For the SAS Viya log files, as these are specific to each container instance, I recommend using the same location as the other SAS Viya volumes discussed above. That is, under the /var/lib/docker/volumes/ path. Following the existing naming convention, let’s call the log volume ‘viyalogdata-<container_name>’. Therefore, you need to add the following mount command to the run script.
--mount type=volume,source=viyalogdata-${CONTAINER_NAME},target=/var/log/sas/viya \
For the user data and home directories we will use a bind mount to a directory location on the Docker server. In this example, we will mount /userData as /data and /userHome as /userhome. Therefore, you need to add the following commands to the run script.
--mount type=bind,source="/userData",target=/data \ --mount type=bind,source="/userHome",target=/userhome \
Let’s see how to configure this in the run_docker_container script.
You need to add the commands as shown above to the docker run script. Figure 3 illustrates the configuration of the run script. I created a script called 'run_logging_container' when I was testing.
Note, you do not have to add the mount statements in one continuous block, but it is a good practice to group the commands as shown.
Where did my logs go?
Once I started my new container (myTestOMM), the SAS Viya log files are under the following path on my docker server.
/var/lib/docker/volumes/viyalogdata-myTestOMM/_data
Note, the '_data' directory was created as part of the docker volume creation/mount process and all the log directories are located under this path. Now you do not have to exec into the container to find the log files. If I do a list (ls) of the directory above, you see the following output.
You can see that each of the SAS Viya services has its own directory for the log files.
While this example illustrates how to use local docker storage, it is possible to redirect the persistent volumes to an external NFS server. You might do this if you had more than one docker server and needed to share and persist data across the servers.
Updating the SSSD configuration
Host authentication is required for the SAS Open Model Manager application. To support this the System Security Services Daemon (SSSD) layer is included in the container. To enable the SSSD configuration you must provide a ‘sssd.conf’ file and any required certificate to connect to Active Directory or LDAP.
As previously stated, to make use of the persistent storage for the user’s home directory we also need to update the sssd.conf file to override the default location (this is the input file for the container, not the sssd.conf file on the docker host).
Using your favorite editor, you need to do the following:
- Find the "ldap_user_home_directory =” statement and change it to "ldap_user_home_directory = /userhome/%u"
- Add the "fallback_homedir = /home/%u" (This allows for the standard '/home' location to be used if there is a problem with the /userhome mount.)
- Find the "override_homedir =" statement and change it to "override_homedir = /userhome/%u"
Your sssd.conf file should now look like the example below.
#ldap_user_home_directory = unixHomeDirectory
ldap_user_home_directory = /userhome/%u
ldap_user_principal = sAMAccountName
ldap_user_name = sAMAccountName
:
:
ldap_use_tokengroups = true
fallback_homedir = /home/%u
# override the user home directory that is provided by the LDAP password entry
override_homedir = /userhome/%u
Figure. Example sssd.conf configuration
Along with the updates to the storage mounts, that completes the steps to move the user’s home directory to persistent storage.
Enabling TLS security
It is always good practice to enable strong security for any application. To avoid sending sensitive information, like your user credentials (username and password) in clear text over the network. Therefore, TLS encryption should be enabled for SAS Open Model Manger. This is called running in ‘TLS mode’.
To enable TLS mode, you must provide a CA Certificate and an associated Public Key file, and use the ‘--TLS’ flag when you launch the container.
The CA certificate must be named ‘casigned.crt’ and the public key file must be named ‘servertls.key’. These files also need to be your working directory. Using a certificate from a trusted authority is the best approach, you should copy the certificate and key for your site into the working directory and rename then as above.
However, if using a certificate from a trusted authority is not possible, then a site-signed certificate can be used as a temporary measure. You can use the following steps to create the certificate and public key. You will need to update step 1 with your company details.
Step 1. Create the input file
The first step is to create an input file (req.conf), this can be done using the following.
cat << 'EOF' > ~/working/req.conf
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = NC
L = Cary
O = Company
OU = Users
CN = server.company.com
[v3_req]
basicConstraints=CA:TRUE
keyUsage = keyEncipherment, dataEncipherment, keyCertSign
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = localhost
DNS.2 = server.company.com
EOF
Step 2. Create the self-signed CA certificate and public key
The second step is to create the self-signed CA certificate and public key using the input file you have just created. This can be done using the following command.
cd ~/working
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -keyout servertls.key -out casigned.crt -config req.conf -extensions 'v3_req'
You can check the result using the following command.
openssl x509 -text -noout -in casigned.crt
This completes the preparation steps to enable TLS mode. The two input files must be in the working directory and you need to start the container with the ‘--TLS’ flag and specify the ‘https-port’ to be used.
For example, to start a container instance called ‘myOMM’ using port 8443 for https (TLS) access to the SAS Open Model Manager visual interface and assigning the http-port to 8080, you would use the following command.
$ ./run_docker_container --container-name myOMM --image <image name> --order <SAS order number> --http-port 8080 --https-port 8443 --tls
Finally, when using the self-signed certificates, the user will be prompted to accept the certificate as it is not from a trusted source. As previously stated, using a certificate from a trusted authority is the best approach.
Updating the number of Threads for SAS Micro Analytic Service
To be able to publish Python models to the SAS Micro Analytic Service (maslocal) and to validate those models within the destination, you must first set the number of threads for the SAS Micro Analytic Service to '1'. This setting helps avoid a deadlock issue in the SAS Micro Analytic Service core.
Note, the best thread setting may vary with different workloads. You can use the steps in the SAS manuals and/or the approach outlined below to set the threads for their environment and processing.
The steps to update the threads are described in the SAS Open Model Manager 1.2: Administrator’s Guide.
However, it is possible to set the threads as part of the initial container configuration using the sitedefault.yml file. The advantage of this approach is the configuration is not a post deployment step requiring a restart.
If we look at the script that is supplied in GitHub , see here, the 'updateMASThreads.sh' script, you will see the following line.
/opt/sas/viya/home/bin/consul kv put config/microanalyticScore/sas.microanalyticservice.system/core/numthreads 1
This is using the 'consul kv put' command to set the key-value pair 'numthreads' to '1'. We can do this in the sitedefault.yml file using the following statements:
microanalyticScore:
sas.microanalyticservice.system:
core:
numthreads: '1'
To update the sitedefault.yml I added the statements above after the ‘identities’ configuration. For example, my sitedefault.yml looked like the following.
config:
application:
sas.identities.providers.ldap.connection:
host: 'ldap_server'
port: '389'
userDN: 'CN=ldap_user,OU=Users,DC=COMPANY,DC=COM'
password: 'connection_password'
url: 'ldap://${sas.identities.providers.ldap.connection.host}:${sas.identities.providers.ldap.connection.port}'
sas.identities.providers.ldap.group:
baseDN: 'OU=Users,DC=company,DC=com'
objectFilter: '(&(objectClass=group)(|(CN=Viya Users)(memberOf=CN=Viya Users,OU=Users,DC=COMPANY,DC=COM)))'
sas.identities.providers.ldap.user:
baseDN: 'OU=Users,DC=COMPANY,DC=COM'
identities:
sas.identities:
administrator: 'admin_user'
microanalyticScore:
sas.microanalyticservice.system:
core:
numthreads: '1'
Remember when updating the sitedefault.yml file the indentation is very important.
What did I learn from my testing?
Do NOT mount the logs target as ‘/var/log/sas’ (target=/var/log/sas). While this seems logical it does not work, as the spre and viya directories and sub-directories are not created. You could separate out the spre directory, but for SAS Open Model Manager there really isn’t any point. Hence, I recommend just creating the ‘viyalogdata’ volume as shown in the example above.
Do NOT try and replace the default location for ‘/home’, this will break your deployment. The ‘cas’ user needs the default location, that is ‘/home’. But you can move the end users to the persistent storage as shown here.
Conclusion and next steps
The key points to remember are:
- Don’t remap the /home location to persistent storage, but you can create a new (persistent) location for the user’s home directories.
- The default run script is expecting all the input files and the start script to be in the same directory.
- The run script can be renamed but the input files must be named as shown.
- The container name and ports being used must be unique.
- You need to specify the https port and the ‘--TLS’ flag to run in TLS mode.
- You need a SAS order with a valid license to be able to download and use the software.
I hope this is useful and thanks for reading.
Acknowledgements
I would like to thank Stuart Rogers and Erwan Granger for their input and assistance.
References
GitHub resources Manage data in Docker
Search for more content from our group: SAS Global Enablement & Learning.
