Best practice for OS accounts when deploying SAS Viya 3.2

[PaulHomes]

I'm initially posting this in the Admin community instead of the SAS Viya community (please see Where to post SAS Viya admin questions? and consider posting your thoughts/preferences on this).

I did my first deployment of SAS Viya 3.2 (SAS Visual Analytics 8.1) this week and was wondering if SAS staff or any other admins have any suggestions about recommended practices for the required operating system accounts when deploying SAS Viya. This is to expand on the documentation already provided in the User and Group Requirements and User Accounts (Reference) sections of the SAS Viya 3.2: Deployment Guide doc.

I did an Ansible based deployment with an Ansible controller, and mirror repository, separate from a single machine SAS Viya 3.2 target (a simple install for testing). After a few minor hiccups (first time doing this) I found the deployment went very smoothly indeed. When there were errors the reason was clear and it was easy to just rerun the process. Much of my time in the beginning was spent deliberating about which OS accounts to create (and which to have auto created) on which machines and which ones needed sudo access. Now that the deployment is up and running I have a better idea about which accounts are used where and how. I think this is an area where the documented could be slightly improved (it's still early days of course) and a few extra clarifications/use-cases/diagrams might be helpful.

What I ended up doing was:

• Manually creating an installer account on both machines to run the Ansible based installation. I named this account casinstall but wondered if I could have used cas or sas for this. Given that it needed sudo access on the target machine I decided against re-use and created this dedicated account. My initial thinking was affected by SAS 9 Linux-based installs where the installation account (e.g. sas) is also a service account for the running environment.
• Manually creating a local sas group on both machines. Afterwards I realised that this is not need on the Ansible controller and could be auto-created on the target machine during deployment (which I may try next time). Initially it was ensuring UID/GID alignment, and the fact my standard OS image for SAS deployments already had sas:sas, that pushed me in the direction of manually creating the accounts. Next time I will probably try with AD based cas/sas accounts rather than local accounts.
• Manually creating a local cas account on both machines (with primary group sas). I realised that this user is not need on the Ansible controller and could be auto-created on the target machine during deployment. After deployment I could see this account is used to run some Viya processes.
• Manually creating a local sas account on both machines (with primary group sas). I realised that this user is not need on the Ansible controller and could be auto-created on the target machine during deployment. After deployment I could see this account is used to run many Viya processes. At first I missed this as a required account but I happened to have it anyway as part of a standard OS image for a SAS 9 deployment.
• I manually setup password-less SSH access for the installer account (casinstall for me) to access casinstall on both machines and both the cas and sas accounts on the target machine. On reflection I don't think I needed casinstall>cas and casinstall>sas is it appeared to use casinstall>casinstall+sudo. I will try removing that superfluous config next time. I understand that cas>cas password-less SSH can be automated during the deployment.
• During the process I confirmed that cas and sas accounts did not need sudo access on the target machine. Only the installer account (casinstall for me) ended up requiring sudo access on the target machine. I felt this reinforced my earlier decision to use a dedicated casinstall account to run the deployment (and have it play no part in the running environment post-deployment).
• I was expecting a sasboot account to be auto-created on the target machine during deployment. It was not created but the deployment completed without error anyway.

So after this long intro, my question is, beyond the documentation, are there any SAS recommend practices, or a recommendations from other admins that may have deployed Viya this way, in terms of the operating system accounts. Do you just create an installer account and let everything else get auto-created (or use AD/LDAP accounts)? Do you use a dedicated account (like casinstall) to run the deployment process or (re)use the cas or sas account and potentially remove sudo access post-install.

Of course, now that I have written all this down, rewritten sections, and re-read relevant documentation, I feel like I have answered several of my own questions/doubts. However, I'm still posting it in case it is useful to others and also that I'm still be interested in the thoughts/practices of others in this area.

[LinusH]

Thanks for sharing your experience, much appreciated!

[ErwanGranger · Accepted solution]

Hello Paul, 

I see that you have given this a lot of thought already and have the start of some best practices. 

Kudos! 

I'll try to add a few points to your very interesting post.

These are based on my own experience and only from having been working with the software a bit longer than you. 

Installer account

I concurr with your approach to use a separate installer account. I usually do one of the following: 

• use the root account directly
• use a sudo-enabled account if there is one already available. (like ec2-user on AWS or cloud-user on OpenStack)
• create a new sudo-enabled account if there is none.
  (I usually call mine "viyadep". CAS is part of Viya, so I find "casinstall" to not be descriptive enough for my taste. Of course, it's completely fine and will work just as well). 

No matter what I chose, I make sure that this account exists on all the machines, and can SSH to all the machines and can SUDO on all the machines, both without being prompted for password. There are probably 12 alternatives to this, but they require a good understanding of the way Ansible works, and when I can, I prefer to keep things simple. 

Before even running my first Ansible test, I make sure that I can  ' ssh  server01 '  (without password) and once on server01 do a 'sudo -l'  to review what kind of sudo permissions I have. 

With such a setup, my 'viyadep' will ssh from the Ansible Controller to the target machines as 'viyadep'. Once there, it will use sudo to impersonnate other accounts. It may impersonnate 'root', 'sas' and 'cas'. So you have to make sure that your sudo permissions are flexible enough to allow that. 

Service Accounts

As you have read in the doc, Viya indeed requires 2 accounts: 'sas' and 'cas'. Note that while you could have an alternative name for 'cas', 'sas' cannot have any other name. 

In the documentation, we are being told that the 'sas' group and the 'cas' account have to be pre-created. Usually, since I have the hood open, I also pre-create the 'sas' account while I'm at it. ( I have built some home-grown Ansible Playbooks for all the pre-requisites, and I included the user creation ). 

Of great importance, is the fact that the UIDs and GIDs have to be consistent. Most dire, if the 'cas' account has inconsistent UIDs across machines, the CAS server will not work at all. 

Service Accounts - local or LDAP?

You could create 'sas' and 'cas' locally, or you could add them to LDAP and then configure PAM on your servers so they authenticate to that LDAP. The software will deploy either way. 

The advantage of the LDAP method is that you CANNOT have inconsisten UIDs/GIDs. 🙂 

However, I tend to prefer using local accounts myself, when I have a choice. 

• First, it's usually more involved and time-consuming to get those accounts created in LDAP than locally. I like being close to self-sufficient for this part of the work.
• Second, if I am to deploy more than one environment (dev/test/prod) then at least one of the accounts ('sas') will have to be "shared" by the environments, which I don't like too much. 

But mostly, unlike SAS 9.4, these are truly service accounts. Like the "mysql" account that gets created when you do a "yum install mysql".  And as such, even though you can assign them a password, it is better and more secure if they are created without a password. Their only role is to "own files" and "run processes". 

Once again, these are only my preferences, and there may be overriding factors that make this impractical. 

Passwordless SSH

The only service account that needs passwordless SSH is 'cas'. I do believe that the deployment process will create SSH keys for it if that's not already done. In an environment where CAS is spread accross multiple machines, only the CAS controller has a CAS service. (The workers do not). When that service starts, on the Controller, it will ssh to all the workers and start the required processes over there as well. 

Admin accounts

Since I never give 'sas' and 'cas' any password, that also means I cannot use them to log into any web application. Instead, once my LDAP has been properly configured in Environment Manager (after logging in as 'sasboot') I make sure to make myself an admin, based on my own LDAP credentials. 

sasboot

The sasboot account is pretty close to something like 'sasadm@saspw'. It is an internal account (the only one!), and it's a sort of 'safety admin account', which does not rely on either OS-level account or LDAP level account. And althrough you can use it to log into Environment Manager and VA, you won't be able to use it to log into SAS Studio or into CAS Monitor. That is why you do not see it being created at the OS level. 

Visual vs Programming

The authentication mechanism differs depending on whether you are using the Visual Interfaces (GUIs like VA, VDB, Env. Manager) or the Programming Interfaces (SAS Studio and its workspace). Everything I've described so far applies mostly to the Visual side. 

I have found that in order to have the most consistent experience, I highly recommand that PAM be configured so that the Linux servers authenticate to the same LDAP that is used by the Visual Web Apps. If this is done properly, then you can authenticate to all applications using the same LDAP credentials.

Before letting my customer try to log into SAS Studio, or even sometimes before the install, I usually present them with a linux prompt. I type '  su - <their-AD-username> ' . 

I ask them to enter "their windows password" .

If they can log into that Linux server in this way, then I have a high degree of confidence that they'll be able to use SAS Studio. If not, I have to have another discussion with their Linux admin to figure out why this does not work. (this is a simplified explanation. there are cases where an su-type login could be disallowed and yet a Studio login could be). 

Post-Install

This is an area where I've not had enough time to do too much work, but there is a lingering question for me as to what the best way would be to configure sudo access after the deployment is done. I have not field tested the following, so you may want to make sure it makes sense to you before recommending/implementing. 

Here is the dilemma: For the deployment, we do need a high level of sudo access. And that may be done by a Linux Admin. 

But after that, the SAS Admin should probably not have as high of a level of permission. 

In SAS 9.4, that was not an issue, because a SAS Admin would 

• know the password to the 'sas' account
• be able to stop and start most services from that 'sas' account. 

With Viya, neither of these is true: The sas account does not need to have a password, and the services are defined as Linux services, and therefore can only be managed by root.  

Here are some options to consider:

• Add an SSH key to the sas account, and share the private key with the SAS Admin. This would allow the admin to easily ssh in as 'sas', without needing a password on the account. This can be useful if a config file that belongs to 'sas' needs to be updated. 
• Give the SAS Admin's user ID (say 'erwan') the right to impersonnate the user 'sas'. This way, I can log in as myself ('erwan') and then type 'sudo -u sas vi /opt/sas/viya/config/....../somefiletoedit.cfg'.
  This is not very practical, but it adds some traceability. Also, you'd probably want to do that at the group level (sasadmins) rather than at a user level (erwan). 
• You can also allow certain users ('sasadmins') to impersonnate root but only for some operations. By configuring sudo in that way, you can allow a user (erwan) to administer the Viya Linux Services (start stop status, etc..) as root, but not to do anything else. 

I would say that even in cases where the Linux admin and the SAS admin are a single person, this is not a bad idea. Having too much root permissions can lead to very regrettable "incidents". 🙂 

Finally, having recently gotten a lot more familiar with Ansible, I believe that most common tasks or configuration changes could be written as Ansible playbooks, which would speed things up greatly, and avoid the risk of typos and other human-generated errors. Things like "edit the .cfg file, then restart the service then test that things still work, and if they don't, quickly roll back the change and restart the service again" can be automated fairly easily once you get past the initial learning curve of Ansible. 

Apologies for the novel, but like you, it helps me to put my thoughts down in writing. I hope it's useful nonetheless. 

Best Regards, 

Erwan Granger

[PaulHomes]

Hi Erwan,

Thanks for taking the time to provide such detailed comments and advice. That will be very helpful indeed.

It's good to hear that a dedicated service account for the deployment is the way to go. I went with it having sudo access with no password on all the machines too. I imagine that some customers will want to constrain this, but for my dev/test environment it simplifies things. In my initial sudo setup there was a subtle flaw in the way I had it configured that showed up with a no-sudo-to-sas error at one of the playbook steps. In the end what worked for me was to have the following in /etc/sudoers.d/casinstall

casinstall ALL=(ALL) NOPASSWD: ALL

Which would of course need be adjusted if a different deployment user id was used. Using viyadep does sound a bit better than casinstall so I might use that too in future (I like consistency).

I like the sound of the sas and cas accounts being service accounts with no ability to login with a password. I had created them with passwords (sas had one from the standard OS image I had), but I use SSH key based access to them (and casinstall) from my personal workstation account anyway. I will disable the password, for cas at least, in future deployments.

After I wrote my original question I did another deployment for a different product set and a different playbook. For that one I used all AD accounts rather than local accounts (for consistent UID, GID, passwords etc). I also tried to use the same Ansible controller. I was planning on also using it as a mirror repository, however, looking at the mirror repo instructions for this 2nd deployment they seemed different which got me wondering if an Ansible controller and mirror repo can be shared between different SAS Viya deployments or not. I ran into some difficulties with that deployment and, even after undeploying, am unable to re-deploy now, getting very odd errors about directories not existing when they would have been created by the playbook before. It sounds like there is some left over config that is being used to skip some initial playbook steps, but I'm not sure where it is yet. I'm going to start over with a new OS image and this time have a dedicated Ansible controller on the same machine as the SAS Viya deployment for simplicity.

That does leave me with some questions about Ansible controllers for SAS Viya deployments:

1) Can an Ansible controller be used for multiple differently configured Viya playbooks/deployments, or does the Ansible controller effectively become tied to that deployment, meaning a new controller for each different deployment?

2) If the Ansible controller can be shared is it intended that a mirror repo can service multiple different product mixes/playbooks/deployments? i.e. somewhat akin to having multiple orders in a SAS9 depot.

Any thoughts you have on that would be much appreciated.

With my AD-accounts only setup, I ran into an interesting issue: AD does not allow a user and group to have the same name, so whilst I had a sas user in AD, I had to name the AD group something other than sas - I use sassys. The Viya doc and playbook assumes the group is named sas. I saw that vars.yml had a casenv_group setting that I could change to sassys to set the group for the cas user. For the sas user I saw that I could edit the file group_vars/all and set INSTALL_GROUP to sassys. However that file has a comment saying "Do not alter the values in this file without the guidance of SAS Technical Support" so I was not sure how well it was going to work. In the end I decided to add a local group with the name sas but with the same GID as the sassys group in AD. The AD users cas, sas, and casinstall had their AD primaryGroupID set such that the sas/sassys group was their primary group. BTW I use realmd to setup my Linux images for AD auth - it makes it very easy - see https://platformadmin.com/blogs/paul/2015/07/active-directory-authentication-for-sas-on-linux-with-r... This method, by default, has UID and GID vaues derived from objectSid and so the numbers are largish but avoids the need to set unix/posix attributes for all users and groups in AD. Since that sas/sassys group only has members via the users primary group setting I also don't have to worry about keeping the local and AD group sas/sassys membership in sync (the local group has no real members just those primary group members).

So another question I have is:
3) Is it supported to name the cas/sas/casinstall primary group something other than sas and is it sufficient to just change just the 2 settings mentioned above, or are there other things that need to be done.

Thanks for confirming that sasboot is not an OS account. That explains why I couldn't find it post install 😉 I think it would be helpful if the SAS Viya 3.2 Deployment Guide clarified that this was an internal account and not an OS account. I also find this interesting because I had previously though that Viya only did AD/LDAP auth for users and I didn't know that it to ability to have internal users (only custom groups). I assume this is for SAS internal product use only and not for SAS customers to take advantage of?

I'm certainly looking forward to finding out more about Ansible and how I might be able to use it to automate some pre-req tasks and more!

Once again thanks for posting your thoughts and advice.

Cheers
Paul

[ErwanGranger]

Hello, 

I should be able to answer your first two questions, but I'll need more time for the third one. (Unless someone else wants to answer). 

---

PaulHomes wrote:

1) Can an Ansible controller be used for multiple differently configured Viya playbooks/deployments, or does the Ansible controller effectively become tied to that deployment, meaning a new controller for each different deployment? 

2) If the Ansible controller can be shared is it intended that a mirror repo can service multiple different product mixes/playbooks/deployments? i.e. somewhat akin to having multiple orders in a SAS9 depot. 

An Ansible controller can indeed be used for multiple deployments. If you were to deploy DEV, TEST and PROD, you could use the same single Ansible Controller to deploy all three. You would have essentially 2 options:

• Use a single Ansible Playbook folder. 
  • In that case, you would need to have 3 "inventory" files: hosts_dev, hosts_test and hosts_prod, each of them containing different servers. 
  • Every time you ran the playbook (ansible site.yml) you would pass the parameter ( -i ./hosts_dev ) to indicate against which set of machines you want this to run. 
• Duplicate the Playbook Folder.
  • You can also, instead, make copies of the folder (sas_viya_playbook_dev, sas_viya_playbook_test, sas_viya_playbook_prod). 
  • In each folder, your inventory (hosts) would have a different set of machines defined. 

Pros and cons. 

• I much prefer the second option for the following reasons:
  • if your vars.yml needs to be different, it's more easily achieved with option 2
  • all ansible activities are captured in "deployment.log". With option1, you'd have a mix of all 3 environments in the log. 
  • with option 1, you are at the mercy of a typo in " -i <inventory> ". It can also happen in option 2, but I'd think it would be less likely. 
  • With option 1, you would have no choice but to run dev, test and prod in sequence. With Option 2, you could run all 3 in parallel if you wanted to. 

Regarding your second question. I think it is much better to have a single, self-contained, mirror for each order. However, once you have a local mirror, you can definitely use it for multiple environments. Although it may be possible to store the RPMs from different orders into a single local repo, I have not tried to do that, as I expect it could lead to "too much" software being installed, and therefore, strange results. 

There are however many benefits to having a local mirror, and I'm glad to see that is the approach you have taken. 

Best regards, 

Erwan

[PaulHomes]

Hi Erwan,

That's good to hear re controller re-use. I think I prefer the safety of the duplicate-and-modified playbook directory for multiple deployments of dev, test, prod etc.

If I understand you correctly on the second question, it sounds like a shared repo mirror for the same order and multiple environments is good, but for different orders a different repo mirror is best. I might have a go at hosting a multiple repo mirror next time, but given my controller is virtual it is just as easy to spin up another one for a different order.

Thanks again,

Paul

[PaulHomes]

Just adding a quick footnote to this post that might help others install SAS Viya. I recently watched Erwan's series of 5 videos on installing SAS VA 8.1 on Viya 3.2 at https://www.youtube.com/watch?v=kbGmY-ryODY&feature=youtu.be Definitely worth watching.