Updating the license file in a Single-Image Viya 3.5 Container – a howto

[#1]

As is often the case when I sit down to write about a topic, I thought I could write a short post about the ways in which you can update a License in a Viya Container. And as is often the case, a few thousand words later, I realize how wrong I was in the assumption that it would be short. 😞

Containers, like any other technology, require knowledge, experience, and skills. Although they can make life easier in some circumstances, this will be especially true for those who understand what makes containers tick. I hope that this post will help you acquire and cultivate some of that knowledge.

Help! My SAS container won't start anymore!

As container adoption is maturing amongst SAS customers, I suspect that there is a slow wave headed towards SAS Tech Support. This consists of those customers who have been using a Programming-Only Viya Container for the better part of a year, and are about to start seeing some License Expiry warnings in their logs. Should they ignore these warnings for too long, they might even be facing the even more dire prospect of said containers no longer starting up.

Starting and dying is just as bad as not starting up

I need a little parenthesis here to mention that yes, technically, the container will start, realize that the license has expired, the process inside will stop, and therefore the container will stop. Further to this, and because many might be the using --rm option on their docker run statement, the dead container will be cleaned up automatically. Which is why it really does look like it did not even start. Trust me on this: it started, it died within a second, and the body was removed immediately. If you are an avid fan of "CSI: Containers", you know it's hard to determine cause of death when there is no body.

 

What that looks like

In order to illustrate what that looks like, compare these 2 results, edited for brevity:

First, an interactive and expressive way of running the container with a BAD location for the license:

docker container run  \
    -it  \
    --volume ${PWD}/BAD/sasinside:/sasinside \
    prebuilt:v1

This results in something easy enough to parse for a human:

[INFO] : Logging files to stdout/stderr
[INFO]: License location is '/opt/sas/viya/config/license.sas'
Running host authentication script
ERROR: No license file found at /opt/sas/viya/config/license.sas.

However, if instead of the -it parameters, you have used -d --rm, there is much less help.

docker container run  \
    -d --rm \
    --volume ${PWD}/BAD/sasinside:/sasinside \
    prebuilt:v1

will only output the container id:

72dfd4ed889624ccb2894b662390b8b8cfb0cb5240221f7c1cbddfa5c2869e70

However, this container will be nowhere to be found, as it will have died and been removed already. So, if you are in this situation already, or if you are looking down the barrel of a soon-to-expire Viya license in a container, here is a piece of advice for you. Don't panic! This is the howto you've been looking for.

 

Harder than it first looked

I have to admit here that this has taken me a lot more time than I thought it would when my manager suggested it as a topic.

I had to do some digging, read a bit of the container recipe code, and even refer to the Viya 3.5 Documentation on applying licenses manually.

 

Getting your Viya 3.5 Programming-Only docker image

If you know your Viya containers, you will know that there are 2 ways of obtaining a Viya 3.5 Programming-Only Single Image. And it's important to know which one you are using.

I like to refer to those 2 ways as follows:

• Pre-Built
• BYO (Build-Your-Own)

In both cases, you are required to have a valid Viya 3.5 Order at your disposal. But the similarities stop there quickly.

 

Pre-Built

If you decide that the Pre-Built image is right for you, at a high level, you will:

1. Upload your SAS_Viya_deployment_data.zip file to a Linux Docker host
2. Extract files from the zip archive
3. Place certificates in some specific locations
4. Install the mirrormgr utility on that host
5. Use mirrormgr to determine the precise image tag to be used
6. Download the pre-built image from SAS-Hosted Docker Registries, using the docker pull command

 

Build Your Own

If instead you prefer to build your own image, then you will:

1. Upload your SAS_Viya_deployment_data.zip file to a Linux Docker host
2. Install the mirrormgr utility on that host
3. Use mirrormgr to download the RPMs in that Order
4. Serve the Yum Mirror over http/https
5. Download the SAS Container Recipes from Github
6. Execute the build.sh script to create the image

 

Naming conventions

To make things simpler to follow, we are going to assume the following:

• We have a linux Docker host
• We have both images available on that docker host
• They are called:
  • prebuilt:v1
  • byo:v1
• We'll assume that both images are currently working, with their respective current licenses, but we need to extend the licenses nonetheless.

 

Pre-Built Image - Test container

Let's start an interactive container from the Pre-Built image, on port 8888.

cd ~/working03/

docker container run \
    --interactive     --tty     --rm \
    --volume ${PWD}/sasinside:/sasinside \
    --volume ${PWD}/sasdemo:/data \
    --volume ${PWD}/cas/data:/cas/data \
    --volume ${PWD}/cas/cache:/cas/cache \
    --volume ${PWD}/cas/permstore:/cas/permstore \
    --env CASENV_CASDATADIR=/cas/data \
    --env CASENV_CASPERMSTORE=/cas/permstore \
    --publish 8888:80 \
    prebuilt:v1

 

BYO Image - Test container

And let's also start an interactive container from the BYO image on port 9999.

docker container run \
    --interactive     --tty     --rm \
    --publish 9999:80 \
    byo:v1

If you look at the above, you'll notice some differences in syntax between the byo:v1 image, and the prebuilt:v1 one. However, the important difference between the two is the lack of --volume ${PWD}/sasinside:/sasinside for the BYO image. This small difference is important. Keep it in mind for later.

 

Checking existing licenses

Now that both test containers are started, let's execute the following code:

proc setinit; 
run; 
cas mysess; 
cas mysess listabout;

For the Base SAS part of the license we can see something like:

And for the CAS part of the license:

 

Updating the License for the Pre-Built image

Now that we have established that both containers have a valid license that expires on 06JAN2021, we will go about updating them. For the purpose of this demonstration, I have another license file that will expire on 27JAN2021.

 

The quick way

In the case of the Pre-Built image, the license is NOT part of the image. It is placed into a folder that is then mounted inside the running instance of the container. You can see that by browsing the content of the sasinside folder on the docker host:

[cloud-user@dockerbox sasinside]$ pwd
/home/cloud-user/working03/sasinside
[cloud-user@dockerbox sasinside]$ ls -al
drwxrwxr-x  2 cloud-user cloud-user  142 Feb  3 06:25 .
drwxrwxr-x 10 cloud-user cloud-user  322 Feb 21 09:38 ..
-rw-rw-r--  1 cloud-user cloud-user    0 Feb  3 06:25 cas_usermods.settings
-rw-r--r--  1 cloud-user cloud-user 3607 Feb  3 06:25 license.sas
-rw-r--r--  1 cloud-user cloud-user 9118 Feb  3 06:25 SASViyaV0300_00AAA0_12345678_Linux_x86-64.jwt
-rw-rw-r--  1 cloud-user cloud-user    0 Feb  3 06:25 workspaceserver_usermods.sh
[cloud-user@dockerbox sasinside]$

So, to swap out the license, the operation could be as simple as the following (please don't do that quite yet, though!):

cd /home/cloud-user/working03/sasinside
mkdir old_license_06JAN2021
mv license.sas old_license_06JAN2021
mv SASViya*.jwt old_license_06JAN2021
cp /tmp/newlicense/SASViya*.txt ./license.sas
cp /tmp/newlicense/SASViya*.jwt .

Simple, right? Well, not so fast. First of all, you have made changes, but you have not yet confirmed that you got the intended result. Don't walk away yet, we still need to validate that this worked. However, and potentially more damaging, we have not really taken into account the fact that there may have been existing containers running already, and what this little manipulation might have done to them. The last thing you want is to kill someone's session just because you wanted to refresh the license.

 

The safer approach

If one was extremely worried about uptime, and not impacting the end-users (as one might need to be sometimes), one should then proceed a lot more carefully than what I have shown above. A better alternative could be as follows:

cp -rp ./sasinside/ ./sasinside_new/
rm ./sasinside_new/license.sas ./sasinside_new/SASViya*.jwt
cp /tmp/newlicense/SASViya*.txt ./sasinside_new/license.sas
cp /tmp/newlicense/SASViya*.jwt ./sasinside_new/

What this allows us to do, for newly launched containers, is to mount ./sasinside_new/ instead of ./sasinside/ into them.

You are now fully ready and able to test your changes without fear of impacting the end users. You can toggle back and forth between the 2 versions of the license on each container that you launch.

So now for the validation:

 

Validation

We stop the running container, and restart it, but pointing at sasinside_new rather than the original sasinside. Notice that we are using the same image:

docker container run \
    --interactive     --tty     --rm \
    --volume ${PWD}/sasinside_new:/sasinside \
    --volume ${PWD}/sasdemo:/data \
    --volume ${PWD}/cas/data:/cas/data \
    --volume ${PWD}/cas/cache:/cas/cache \
    --volume ${PWD}/cas/permstore:/cas/permstore \
    --env CASENV_CASDATADIR=/cas/data \
    --env CASENV_CASPERMSTORE=/cas/permstore \
    --publish 8888:80 \
    prebuilt:v1

Notice that the Base SAS License and the CAS License now report the newer date:

This confirms that we are indeed now using the newer license. (JAN 27 instead of JAN 06).

 

Updating the license for the BYO Image

In the previous section, we have seen how storing the license in a mounted folder seems very practical at first, but can also lead to issues if one is not careful. Eventually, the elegant solution is also more complex, and pretty much involves version control of the files.

While this seems fast and practical, I personally dislike this approach. (This is merely a personal preference, and some colleagues vehemently disagree with me.) I dislike it because it leaves me at the mercy of the presence of this file in that folder. If someone was to accidentally delete that file, it would start impacting the users very quickly and it might take some time before you figure out what's wrong, find a backup of the folder and license, restore it, and get back to normal.

The steps in this section will illustrate what I personally consider to be a "safer" way of doing things when dealing with containers. You are entitled to disagree, as long as you read my post until the end before commenting. Dissenting opinions are welcome.  😉🙂

Unlike the Pre-Built image, the BYO image contains the SAS License, straight inside the image. The downside is that if someone was to steal the image, they'd also steal its license. The other downside is that I will have to create a new image each year, with the new license in it. But the upside is that the image is much more self-contained, self-sufficient, and portable, which to me, more than makes up for the drawbacks.

I am reminded of this colleague who downloaded the Pre-Built image from a Linux Server down to his laptop, only to find it "not working". It had slipped my colleague's mind that he was also supposed to download the license file, and mount it identically on his laptop. The fact that the wound was self-inflicted did not lessen the pain of wasting a couple hours. The reaction was more along the lines of "So much for the promise of 'build once, run anywhere!'".

 

Re-running the Build.sh script

At this point, you might be tempted to re-run the initial build.sh script that you had used the create the first image. Although that could be an avenue, there are some serious considerations to take into account.

• Do you still have a copy of the RPMs that were used to build the initial image. If you did not follow my usual advice to create a mirror and build from it, let me save you some time: You do not (have a copy of the RPMs), and you won't be able to get one.
• If you don't have a copy of the original Mirror, you can build a new image, but chances are the new image will ALSO have a different version of the software. Are you sure you want to be changing 2 things (license, software version) at the same time? (Pro tip: You do not)
• You have to have same exact version of the Container Recipes code.
• Even if you can get the old version of the recipes, it might no longer work. (a change in Python2 packages has forced us to fix the last version of the recipes. Previous versions of the recipes do not have this fix in them, so you'd need to port that fix yourself)
• Even if you manage to get all of this sorted out, it's likely that the result will be quite inefficient: Your new image is likely to not have any layers in common with the old one, and therefore consume just as much space as the old one. (instead of leveraging the docker deduplication)

So, to sum up:

• Can you re-run the build script?
  • Probably.
• But should you?
  • Probably not.

 

Creating V2 from V1

The method highlighted here might seem overly complicated at first, but it is quite important to be able to grasp it if you want to get a good understanding of Containers. What we will do is:

1. Figure out what needs to happen to the V1 image to turn it into a working V2 that has a newer (updated) license file
2. Transcribe that into a Dockerfile file (not a typo, that's what it's called)
3. Build V2 from V1 so that the only difference is the license
4. Confirm it all works before you announce success and disappear on a well-deserved vacation

 

Figure out which files need replacing in the V1 image

At first, I thought I could easily do this without looking anything up. It turns that I was wrong. The following documentation pages ended up being quite useful: If you want to be able to snoop around in a fresh instance of the container, the following command can be quite useful:

## start a "blank" container with a bash prompt:
docker container run -it --entrypoint /bin/bash byo:v1

What this does is bypass the regular entrypoint, and "bashes" you into the container. Note that because of that, none of the services are started, and you might also be missing pieces that would have been performed by the regular entrypoint script.

But if you want to investigate the inside of the container, you can then run things such as:

find /opt/sas/ -'*.jwt'
find /opt/sas/ -'*license*'

 

 

How to create a new image

First, we need a working area and we should copy the files we need into it, while renaming them, for easier handling:

mkdir ~/image_v2/

cp /tmp/newlicense/SASViya*.txt ~/image_v2/license.txt
cp /tmp/newlicense/SASViya*.jwt ~/image_v2/license.jwt

In order to create a new image, we simply need to create a dockerfile. Read the code and the comments carefully:

tee ~/image_v2/Dockerfile > /dev/null <<'EOF'
## The source image we want to start from:
FROM byo:v1

## copy the 2 license files to the inside of the image
COPY ./license.txt /tmp/license.txt
COPY ./license.jwt /tmp/license.jwt

## Execute the license Update routine (as the user 'sas')
RUN su - sas -c "/opt/sas/spre/home/SASFoundation/utilities/bin/apply_license /tmp/license.txt /tmp/license.jwt"

## Delete the old license and replace with the new one:
RUN su - sas -c "rm -f /opt/sas/viya/config/etc/cas/default/SASViya*_Linux_x86-64.jwt ; \
    cp /tmp/license.jwt /opt/sas/viya/config/etc/cas/default/license.2020.01.21.jwt ; \
    rm -f /opt/sas/viya/config/etc/cas/default/sas_license.txt ; \
    ln -s /opt/sas/viya/config/etc/cas/default/license.2020.01.21.jwt /opt/sas/viya/config/etc/cas/default/sas_license.txt"

## Clean up those stray files we no longer need:
RUN rm -f /tmp/license.txt /tmp/license.jwt

EOF

Now that we have that file, we can run the following docker build command to create a second version of the image:

cd ~/image_v2/
time docker image build -f ./Dockerfile -t byo:v2 .

The output you get should look like this:

Sending build context to Docker daemon  16.38kB
Step 1/6 : FROM byo:v1
 ---> 1c1c66cdf6d5
Step 2/6 : COPY ./license.txt /tmp/license.txt
 ---> Using cache
 ---> edd3341b38eb
Step 3/6 : COPY ./license.jwt /tmp/license.jwt
 ---> Using cache
 ---> d12d395fba6a
Step 4/6 : RUN su - sas -c "/opt/sas/spre/home/SASFoundation/utilities/bin/apply_license /tmp/license.txt /tmp/license.jwt"
 ---> Using cache
 ---> db94abadb24c
Step 5/6 : RUN su - sas -c "rm -f /opt/sas/viya/config/etc/cas/default/SASViya*_Linux_x86-64.jwt ;     cp /tmp/license.jwt /opt/sas/viya/config/etc/cas/default/license.2020.01.21.jwt ;     rm -f /opt/sas/viya/config/etc/cas/default/sas_license.txt ;     ln -s /opt/sas/viya/config/etc/cas/default/license.2020.01.21.jwt /opt/sas/viya/config/etc/cas/default/sas_license.txt"
 ---> Running in c0a43094a5e9
Removing intermediate container c0a43094a5e9
 ---> 80845a19260e
Step 6/6 : RUN rm -f /tmp/license.txt /tmp/license.jwt
 ---> Running in 1d3267ff27fe
Removing intermediate container 1d3267ff27fe
 ---> d1b11d43547e
Successfully built d1b11d43547e
Successfully tagged byo:v2

real    0m1.770s
user    0m0.044s
sys     0m0.042s

This process will not have impacted any of the existing running instance based off of the V1 image. And now, you can start testing and validating V2 privately. Your end users need not be informed there is a V2 image until you are 100% sure it's good to go.

Validating

With the exact same syntax as earlier, save for the image name of byo:v2 instead of byo:v1, we restart our test container:

docker container run \
    --interactive     --tty     --rm \
    --publish 9999:80 \
    byo:v2

On running our proc setinit code, we can see that the licences for both Base SAS and CAS have been properly updated:

At this point, we can safely start shutting off the old container Instances that were launched from V1 and start replacing them with V2 instances instead.

 

Playing Tag

I usually don't venture too far out with tags. Mainly, I know I have not worked enough with containers to fully grasp image tagging best practices, and so I leave this to others. However, our license problem is presenting us with an interesting illustration of the power of tags.

(Obviously, for the Pre-Built image, this does not apply: we still only have one version of the image) Let's look at our BYO images:

docker image ls | grep byo

And we'll see:

REPOSITORY     TAG     IMAGE ID            CREATED             SIZE
byo             v2     7431acd8d684        57 seconds ago      10.5GB
byo             v1     f78122326319        8 hours ago         10.5GB

Anyone else than you will look at this and wonder what the difference is between V1 and V2. So let's add another round of tagging to make things clearer, for the good of future container-kind.

docker image tag byo:v1 byo:expires_20210106
docker image tag byo:v2 byo:expires_20210202

And now our images look like:

REPOSITORY      TAG                     IMAGE ID            CREATED             SIZE
byo             expires_20210202        7431acd8d684        3 minutes ago       10.5GB
byo             v2                      7431acd8d684        3 minutes ago       10.5GB
byo             expires_20210106        f78122326319        8 hours ago         10.5GB
byo             v1                      f78122326319        8 hours ago         10.5GB

Much more meaningful. Isn't it?

 

Conclusion

I have often stated in workshops the following: Bind mounts are for persistence, not convenience.

If you are externalizing a file as important as the license file simply because you think it will make updating it easier, you are opening yourself up for future human error.

If you own a car, you know that it is inconvenient to have to lock/unlock the car and start/stop then engine each time you need to use it.

And yet, you do it, because while leaving your car unlocked and the engine running might be convenient, there are risks associated with that, and most of us do not want to take those chances.

So do your future self a favor, and make it as difficult as possible for him/her to shoot himself/herself in the foot.

Search for more content from our group: SAS Global Enablement & Learning.