Operating System Lifecycle for Red Hat Enterprise Linux – Enhancing SAS 9.4 on Cloud

[#1]

SAS’ continued commitment to improve customer experience in Cloud sees yet another enhancement for Cloud deployments of SAS 9.4. Public cloud providers offer various methods for managing and maintaining the Operating System (OS) in their corresponding virtualised compute (IaaS VM) offerings. For SAS customers on public cloud, the flexibility to manage and maintain the underlying OS of their SAS 9.4 deployments hasn’t always been all encompassing. Customers accustomed to on-premise OS patching and maintenance processes often find cloud vendor solutions for OS level patching to be at odds with SAS requirements for managing a SAS 9.4 deployment.

Background:

Based on the cloud provider subscription model, and at times their own internal policies, SAS customers find themselves constrained to bring their own satellite repository for RHEL to manage and maintain the OS using traditional 'yum' based approach. In public cloud environments, managing and licensing Red Hat Subscription and Satellite servers can be complex and may not be viable. This can hinder the ability to perform in-situ OS upgrades or patching using traditional methods. Many cloud vendors, for example Microsoft Azure, commonly recommend adopting a plug-and-play approach when it comes to managing the OS of an IaaS application in the cloud. In this model, organizations are encouraged to ‘destroy and rebuild’ cloud applications on a regular basis, using ‘Infrastructure as Code’ principles to effortlessly rebuild these using the most up-to-date OS or codebase.

Often, customer security teams also have a requirement for OS upgrades to new minor versions to be carried out on a regular cadence, as OS images are only maintained in the repository for a set period; until deprecated based on its support or vulnerability exposure.

The Challenge:

SAS recognise that the Disaster Recovery policies for SAS 9.4 [link] and SAS Viya 3.5 [link] do not currently cater for requirements such as maintaining the Operating System in cloud environments where organizations opt to adopt this methodology as the means to distribute OS rolling updates.

It has been particularly challenging for SAS 9.4 customers who need to meet these requirements, as the current DR policies state:

"Copying only the SAS deployment (the SASHome or SAS Configuration directories) from one system to another is not supported. In this scenario, SAS Deployment Manager tasks fail to properly manage and maintain the deployment and the SAS Deployment Agent (necessary to perform future backups and middle-tier clustering). Copying the SAS deployment as part of Option A is supported as long as the disaster-recovery system remains a clone of the production system."

The issue arises from the latter part of this statement, as with OS patch/upgrade, the target system cannot be a true clone of the source system. Until now…

Position:

SAS has now defined and tested a mechanism to decouple the SAS deployment from the underlying OS disks, enabling customers with SAS 9.4 deployments on the cloud to mount their existing deployment to a new VM host built with an upgraded minor release or patch for the OS. This process is detailed in the paper Mounting a SAS ® 9.4 Deployment from One Cloud Instance to Another after a Minor Operating System Re...

The scope of this process includes:

• Linux Distributions currently supported by the SAS 9.4 Mx release (excluding AIX and Windows). Note: Only Red Hat Enterprise Linux 8.x has been tested by SAS.
• Only minor OS release upgrades.
• Server virtualization technologies that allow decoupling of the OS disk from the data disks, enabling SAS applications and data to be unmounted and remounted to new infrastructure with an updated OS disk.

Design Principle:

The process will vary across cloud providers depending on their storage solutions and organizational standards, but typically involves:

• Storing SAS applications (SAS Binaries, SAS Config, SAS-provided third-party components) and content (SAS Data, Code, etc.) externally, in a storage solution separate from the OS disk, such as local/managed disks or shared storage solutions like Lustre or Azure NetApp Files.
• Implementing a strategy where VMs serve as “blank” canvases to which content is mounted, allowing for flexible infrastructure management.
• Using Infrastructure as Code (IaC) principles and automation tools to deploy and configure infrastructure components consistently and efficiently. For the initial build SAS deployment must be performed using SAS Deployment Wizard and SAS Deployment Manager tools.
• Employing strict guidelines for content retention and synchronization during the rebuilding process to maintain consistency and integrity.
• Minimizing business disruption and service downtime by provisioning target VMs with the desired OS version automatically, ensuring all customizations are applied, and then mounting existing application components to the new infrastructure.

Always review the System Requirement guide System Requirements for SAS® 9.4 Foundation for Linux® for x64 and Supported Operating System section SAS Supported Operating Systems. Additional steps may be required based on the software offering and solution deployed at your site.

Additional Considerations:

• This approach can only be performed during the initial SAS 9.4 deployment and may not be suitable for retrofit on an existing SAS system.
• Work along with your SAS Account team to define the architecture design and document any additional steps required for your site-specific requirements.
• You are responsible for ensuring that you have the appropriate processes, resources, holistic-system backups, skills, and competencies to guarantee the success of this approach at your site.

Risk Mitigation:

To ensure a smooth OS upgrade process, consider the following risk mitigation strategies:

• Regular Backups: Maintain regular backups of your SAS environment, including configuration files, data, and application binaries. This ensures that you can restore your environment quickly in case of any issues during the OS upgrade.
• Ad-hoc Backup Schedule: Trigger ad-hoc backup schedules before initiating any OS lifecycle activities. This provides an additional safety net in case of unforeseen problems during the upgrade process.
• Business Continuity Planning: Develop a robust business continuity plan that outlines the steps to take in case of a disaster or failure during the upgrade. This plan should include clear roles and responsibilities, communication protocols, and recovery procedures.
• Testing and Validation: Thoroughly test the upgrade process in a non-production environment before applying it to your live systems. Validate that all components function correctly post-upgrade and that performance meets expected standards. Validation should include ensuring the following:
  • deployment-tooling functions are successful
  • backups via the Deployment Backup and Recovery Tool are successful
  • SAS Deployment Manager updates passwords, rebuilds and redeploys web apps, updates certificates, and so on
• Documentation: Keep detailed documentation of the upgrade process, including any customizations or configurations applied and issues encountered. This documentation will be invaluable for troubleshooting and for future upgrades.
• Robust Rollback Plan: Have a clear rollback plan in case the upgrade does not go as expected. This should include steps to revert to the previous OS version and restore all data and applications from backups.

Conclusion:

For SAS 9.4 deployments in the public cloud that can decouple the SAS deployment itself from the underlying OS disks, SAS provides a viable solution for seamless upgrades to RHEL in cloud environments. This approach not only facilitates software currency by simplifying the OS upgrade process but also aligns with cloud-native practices. As SAS continues to enhance customer experiences in the cloud, this solution marks a significant step forward in addressing the complexities of OS management for SAS 9.4.

With that, if this is a requirement within your organisation? I would welcome your feedback and comments on this proposed method and encourage you to review the detailed documentation for further insights.