SysV init, systemd and Viya – Service Basics
Many SAS software deployments on Red Hat over the past couple of years have relied on the System V init (aka SysV init) system to define and manage services. This system is used to control which services are started and stopped via runlevels. If you have automated the startup and shutdown of the sas.servers script on Red Hat 5 and 6, then you have used the SysV init system.
Beginning with Red Hat 7 a new service, systemd, replaced the SysV init service. The systemd init system introduces a modernized replacement for init processing and provides significant improvements in managing services. The enhanced capabilities of systemd make use of new commands and files to manage services and as a result will require familiarization before establishing new services on a host.
SysV init and systemd are not unique to Red Hat, but since most of our Linux deployments are on Red Hat, the content of this blog uses Red Hat machines. To understand the differences between the init systems, we will briefly review how SysV init is used to manage sas.servers with SAS 9.4 and then show how SAS Viya Data Mining and Machine Learning (DMML) 3.0 fits into the picture using systemd.
SysV init (and sas.servers)
At a very simplistic level, when a Red Hat machine is booted, the BIOS launches the boot-strap program from the primary hard disk. The boot-strap program then loads the kernel into memory. The kernel in turn loads drivers and required modules and mounts the root partition. The kernel then transfers control of the boot process to the init program. Once the init program is running it mounts file systems in /etc/fstab and then loads all services and user tools based on services defined in /etc/init.d and dependent upon the runlevel specified in /etc/inittab. If sas.servers was configured to start at boot time, then sas.servers is one of those services that init loads.
If you have ever captured information about the init process, you may have noticed that its PID is 1 (one). And if you look further, you would notice that the init process is the parent process for numerous processes. The following screen shot illustrates this as it shows part of the process tree on a VA 7.3 system where sas.servers was added to /etc/init.d. Notice that the parent process of Metadata server, Object Spawner, and DIPJobRunner is the init process.
Of course steps have to be taken to ensure sas.servers is enabled as a service and then started at boot time. Those steps can be found within the sas.servers script. Here is a snippet of instructions from the sas.servers that addresses starting services at boot time using /etc/init.d.
#
# # cd <sasconfigdir>/Lev1
# # cp sas.servers /etc/init.d
# # chmod 0755 /etc/init.d/sas.servers
#
# To install this script as bootable, as user "root", invoke:
#
# # /sbin/chkconfig --add sas.servers
# # /sbin/chkconfig --level 35 sas.servers on
# # /sbin/chkconfig --list sas.servers
Essentially the sas.servers script is copied to the /etc/init.d folder and permissions modified, then the chkconfig command is used to add and enable it as a service. I was going to step through what each of these commands accomplish, but shortly after starting this blog I discovered that Gerry Nelson has already nicely summarized these steps in his blog. Gerry's blog explains automating the startup of the SAS Deployment Agent, but the explanation of steps is essentially the same.
An Introduction to systemd
The same basic initialization process occurs in Red Hat 7 with systemd, except that the kernel transfers control to the systemd process instead of init when it completes. Once systemd has control it performs functions similar to SysV init, and potentially many more. It starts services that are enabled, and like the init process, systemd becomes the parent process of many services that are started. An example from Red Hat 7.2 is shown here. (If pstree is not available on your system, it can be installed using yum install psmisc.)
Once systemd is initialized the similarities diminish. Although part of systemd's role is similar to SysV init in that it manages services, the commands used and configuration of systemd are completely different. This would be an appropriate place to go into detail on systemd configuration, however the systemd topic is too wide ranging to cover in a blog, so we will just touch on some of the basics.
One advantage of systemd is that it starts services in parallel when possible. Of course the obvious advantage of parallel startup is that the system will require less time to be fully initialized. And since SAS Viya services do not have dependencies on each other, this is expected to speed startup. The improvement may not be so apparent with DMML, but it should greatly benefit startup once micro-services are introduced later this year. Additional advantages include using socket activation for starting services, on-demand starting of daemons, snapshotting and restoring of the system state, and others.
A key item to remember is that systemd is backwardly compatible with SysV init. As a result the commands used for SysV init, chkconfig and service, continue to be available along with systemd commands. It is possible to enable a service with chkconfig and then review the status of that service with systemd commands. A little more on this later.
Systemd introduces the concept of systemd units. These units are represented by unit configuration files that capture information about services, sockets, and other objects related to the init system. Units are stored in one of three locations:
- /usr/lib/systemd/system
- /run/systemd/system
- /etc/systemd/system
Here is an example of the cron daemon unit configuration file (i.e. crond.service). There are many different unit types. See the documentation for the complete list.
From the screenshot you can see there are multiple options used to define various aspects of the service. The options used for the crond.service is a small subset of the complete list available to the administrator. The complete list of options and their purpose can be reviewed by invoking the man page (i.e. man systemd.service). For example ExecStart options specifies the command to execute and related arguments when starting the service. These options may define how the when the service will start and stop, dependencies on other services, whether it try to restart itself if it fails, etc.
Once the unit configuration file is created, the service can be enabled using the systemctl command. The following command is issued to enable a service. The parameter name.service references the unit configuration file that was created and saved in one of the three locations noted above.
Now that the service is enabled we can query it. The command to query it is
Here is an example of the status command for the cron daemon, an existing service. The command shows that the service is loaded and enabled. It also shows that the daemon is running and the time it was started along with the PID.
This is all we'll cover on systemd basics in this blog. If you have heard of systemd and this is the first time you seen information on it, then the topics covered in this section hopefully have provided a glimpse into it, however that glimpse may be a little fuzzy (or maybe very fuzzy). To help clarify the picture I've included several links to official documentation at the end of the blog.
So what about Viya and Ansible?
You may be wondering how Viya's services and Ansible fit into the startup picture.
SAS Viya DMML 3.0 is supported on Red Hat 6.7+ and 7.1+. Therefore if SAS Viya DMML is deployed on Red Hat 6, then the SysV init commands would be used for automate startup. The process would be similar to automating sas.servers.
Alternatively if SAS Viya DMML is deployed on Red Hat 7, then it is possible to use systemd to automate startup. But at 16w20 the deployment does not define the service unit configuration files for each of the services. As a result it would be necessary to create them manually.
At the end of the DMML deployment Viya services are stored in /etc/init.d for all supported versions of Red Hat. These services are started as part of the deployment, but they are not enabled to start on boot.
The following shot shows the Viya services available in /etc/init.d on Red Hat 7.2 of a single machine deployment. SAS/Connect was in this order so the Connect spawner service is displayed. In a multi-machine deployment these services would be defined across machines, depending on their role.
Given that unit configuration files are not available as part of the deployment, the easiest path to enable them as services is to use the backwardly compatible components of systemd. This essentially means using the chkconfig commands shown earlier for sas.servers. Since Viya services are independent of each other, it is necessary to enable each of the services listed with the exception of the CAS worker (sas-viya-casworker-default). The CAS worker service is invoked by the CAS controller.
But wait, there is another way to enable the services. Since Ansible is the preferred tool used to deploy the software and it has the capability to enable services, then it stands to reason that it should be used.
It is far beyond the scope of this blog to delve into Ansible, but Ansible has hundreds of modules that run commands on desired hosts as a user of your choice. Of course it has be configured. (To obtain a basic understanding of the role of Ansible, please see Erwan Granger's blog on Desired State.)
One of those modules is the service module. This module provides the capability to manage services using an Ansible playbook. A playbook containing the service module to automate startup is not included at 16w20, but is expected to be incorporated in later ship events. As a result all services should be enabled as part of the deployment in future Viya ship events.
The following partial screen shot of an Ansible playbook shows an example of using the service module to enable the SAS Studio service so that it starts at boot time.
When this service module is executed the parameters indicate that the SAS Studio service should be enabled and started. In essence this module will execute the chkconfig command to enable the service to start at boot time.
If we now use the chkconfig command to list the service, we see that it is enabled for runlevels 2-5. Note that since we are running on a Red Hat 7.2 machine with systemd, messages explain that the chkconfig command is unaware of native systemd services.
However, it is also possible to use systemctl to display the status. Systemd is aware of the service but it is not enabled as a systemd service as we saw earlier with the crond service.
Okay, I think it's time to bring this to a close although we've just scratched the surface of these topics. So let's recap.
Summary
SysV init and its successor, systemd, are the tools used to automate startup and shutdown of services on Red Hat. SysV init uses the chkconfig and service commands to manage services, while systemd uses the systemctl command to do so. SysV init is the tool used on Red Hat 6.x, while systemd is available on Red Hat 7.x. SysV init provides basic functionality, while systemd introduces robust functionality. Systemd is backwardly compatible with SysV init. Enabling Viya or SAS 9.4 services on systemd will require creating unit configuration files. And finally, the Ansible service module can be used to enable services as SysV init services.
As more deployments transition to Red Hat 7, the use of native systemd functionality is expected to grow. And as we ramp up on Viya, you will be hearing a lot more about Ansible.
This blog was originally written for Viya 3.0. Although the number of services has increased with Viya 3.2, the concepts described here still apply.
Use the following links to drill further into each topic.
Red Hat 6 Boot Process
Managing Services with systemd
Ansible Documentation
SAS® Viya™ 3.2: Deployment Guide
Don't miss out on SAS Innovate - Register now for the FREE Livestream!
Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.
