SAS Communities posts offer lots of amazing content on DevOps for SAS, but all of the technical detail can be a bit confusing. So let's step back and better define what we're talking about exactly.   What "DevOps" Typically Means   While DevOps incorporates many different functions from collaborative development to production monitoring, when most people say "DevOps" what they generally mean is automated code deployment. Instead of developers throwing their finished code "over the wall" to a dedicated team of release engineers who move it through the deployment process, DevOps aims to automate the release of code into production. Undoubtedly, this is an over-simplification. However, it is useful to help people understand what DevOps is all about. For example, while DevOps includes development, no one really uses the term "DevOps" to describe the collaborative development and versioning process. Most people really only say "DevOps" when the operations components come into play.   The "Dev" Part of DevOps, Development   To facilitate versioning and collaborative development, SAS developers use a code repository management and collaboration tool like Git or SVN. Developers create and modify SAS code and other files in their own dedicated repositories and branches dedicated to particular tasks, new features, bug fixes, etc. SAS developers can use SAS Studio's Git integration as well as SAS' integration with Visual Studio Code or their editor of choice to work with Git.   The "Ops" Part of DevOps, Operations   When development is complete, the finished SAS code and other files are pushed to the main branch on the central repository. This action triggers the Ops (Operations) part of DevOps to begin. These "Operations" are carried out by a CI/CD Pipeline. This is a fancy term but all it amounts to is an automated sequence of scripts which perform various administration functions like copying files to different environments. While these scripts may seem complicated, we can keep them in context by remembering that the point of them is to deploy the finished code, i.e. the desired functionality, into production.   The Most Basic SAS Viya DevOps Pipeline   To get a sense of what a DevOps pipeline does, let's start with the simplest possible example and build from there.   The simplest possible CI/CD pipeline is just some command to push the new/modified code to the production environment. In Gitlab, it would look something like this:     That's it, just a Linux SCP command to copy the contents of the shared repository to the production server. This is the essence of DevOps, coordinating development and getting the resulting work into use as quickly as possible.   Beyond Just Promoting Code, Libnames   Promoting code is the heart of the matter but there is generally more configuration and supporting assets that you'll need to make your SAS code work. Let's start with SAS libraries as our first consideration.   We want our code to be exactly the same in our development environment as it is in our production environment. Manually changing code to work in different environments (DEV, TEST, PROD, etc.) can introduce bugs as well as getting us away from our ultimate goal of fully automated code deployment. However, while our code will be the same, our data sources and targets will undoubtedly be different. We don't want to put unit test data in our prod tables, right? Additionally, SAS Viya has rich user interfaces we'll want to support as well as our code.   To support all of Viya, we'll want to configure libraries using the autoexec_code of our environment's server contexts. Again, this sounds complicated but it's just Viya's way of specifying autoexec.sas files in its application data.   In our scheme, we'll actually use an old fashioned autoexec.sas file that developers can put the library definitions straight in and promote up along with the code. We have structured all of our data stores to use environment-identifying strings (DEV, TEST, PROD, ...) somewhere in them so we can use Linux environment variables to specify which environment we are deploying as seen in the snippet below.     In our DevOps Pipeline, we'll set the ENV variable to "PROD," use the envsubst bash utility to resolve the variable, perform some additional formatting, and finally use the Viya command line interface to update the autoexec_code attribute. The resulting pipeline job would look something like this (again in Gitlab).     For a fully functional version, see this GitLab file.   Running SAS Code   Say we need to pre-populate a new table, run automated system tests, or create test data. There is no better tool than SAS. We can find a million reasons to use SAS for DevOps tasks. To run SAS code for DevOps purposes, simply include the code in your repository and execute it using the Viya command line interface.     For a fully functional version, see this GitLab file.   Importing SAS Packages   Git only works with file system files. So, promoting SAS content is not as straight forward as regular SAS code files. We need to export them from DEV and then import them into downstream environments like TEST and PROD. Below is an example of importing a SAS transfer package in a Gitlab CI/CD pipeline.     For More   For more information on DevOps for SAS with a host of facilitators like Gitlab, Jenkins, Azure, etc., check out SAS Communities posts as well as these two courses:   - Using Azure DevOps and Azure Pipelines in SAS® Viya®   - DevOps with SAS® Viya® and GitLab     Find more articles from SAS Global Enablement and Learning here. ... View more

Previously I wrote about Jobs and Flows' time-based scheduling capabilities and I even wrote about making your own DIY file event triggering scheme. With the 2025.02 release, Jobs and Flows includes its own file event triggering functionality. Let's take a look.   Defining File Event Triggers   Triggering jobs to run using the system clock is great. You can make jobs run at specific times on specific dates; repeatedly every few minutes, hours, days, .... But! What people usually need is event-based scheduling. When everything the job needs - input data, system resources, permission from the stake holders, etc. - is in place, then run the job. No clock required! So how can we know that everything is in place? We need a signal. The trigger file is that signal. When the trigger file appears, we know everything is in place and we can run. We set an agreement with the system owners and/or the source data providers, "When everything is ready, put a file named X in location Y."   Adding Event Triggers in Jobs and Flows   To add a file event trigger, right click the job flow and select Schedule Flow. On the following screen, click New and select File Event.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   From there, name your file event, point to a file in SAS content and add a condition. The conditions are described in detail here but, for our scheme, we'll choose the condition, "Exists."   So, as shown, by the graphic below, our job flow will not execute until the file nightlyFlow.trigger is detected in SAS content.     Combining Event Triggers with Time Triggers   In many scenarios, you'll want to combine a file trigger with a time trigger to achieve something like "Run the flow every midnight when the trigger file is detected." This combination gives you a regular execution cadence but also adds the trigger component to ensure that source data and other resources are ready.     Getting Files into SAS Content   As discussed earlier, the 2025.02 implementation only detects trigger files in SAS content. Writing to SAS content is fairly straightforward. It can be done with base SAS code as well as the Viya API. Below are examples.   Base SAS - Write a SAS Content File filename trgrFile filesrvc folderpath='/Public' filename='nightlyFlow.trigger'; data _null_; file trgFile; stop; run;   Python - upload2sas.py, code to Upload a File to SAS Content import argparse import os from sasctl import Session from sasctl.services import files, folders # Server and authentication details server_name = "server.demo.sas.com" user = "sasadm" password = "lnxsas" def upload_file(local_file, target_folder): """ Upload a single file to the SAS Content Server. Parameters ---------- local_file : str Path to the local file to be uploaded. target_folder : str Path to the target folder in SAS Content Server. Returns ------- None """ # Extract the file name from the local file path file_name = os.path.basename(local_file) # Ensure the target folder exists in SAS Content Server folders.create_path(target_folder) # Upload the file to the target folder files.create_file(local_file, target_folder, file_name) print(f"File '{file_name}' uploaded successfully to '{target_folder}'.") if __name__ == "__main__": # Parse command-line arguments parser = argparse.ArgumentParser(description="Upload a file to the SAS Content Server.") parser.add_argument("--local-file", required=True, help="Path to the local file to be uploaded.") parser.add_argument("--target-folder", required=True, help="Target folder in SAS Content Server.") args = parser.parse_args() # Ensure the local file exists if not os.path.exists(args.local_file): print(f"Error: The file '{args.local_file}' does not exist.") exit(1) # Exit with a non-zero status to indicate an error # Start a session and upload the file with Session(server_name, user, password): upload_file(args.local_file, args.target_folder)   Python - Commands to install the necessary Python libraries; Write a trigger file; Upload it to SAS content /opt/conda/bin/pip install sasctl touch /tmp/flow0.trigger python upload2sas.py --local-file /tmp/flow0.trigger --target-folder /Public   Base SAS - Delete a SAS Content File filename trgrMart filesrvc folderpath='/Public' filename='nightlyFlow.trigger'; data _null_; rc=fdelete('trgrMart'); run;     Find more articles from SAS Global Enablement and Learning here. ... View more

A lesser-known component of SAS Viya's user interface is the Jobs and Flows application within Viya Environment Manager. The application lets users schedule jobs and combine jobs into flows with complex dependencies.   Earlier I showed how to facilitate event-based triggers using a job running every 5 minutes and a few command line actions. In this post, I'll outline the current time-based scheduling capabilities of the application.   When users have assets like SAS programs, SAS Studio Flows, Visual Analytics reports, etc. ready for scheduling, they'll see them in the Viya Jobs and Flows application. To schedule, users simply need to right-click the job and select "Schedule."   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   This opens another window where users select a trigger.     Adding a new trigger, we see the time-based scheduling options available. Users can schedule their jobs daily, weekly, monthly, yearly, and on specific dates using a list. As shown the prompts change with each selection to offer a host of appropriate options.     Once scheduled, jobs will execute according to their schedule until cancelled. Execution can be monitored within the application as well as with the command line interface.       Find more articles from SAS Global Enablement and Learning here. ... View more

UPDATE: File-based event triggers are now available out of the box as of Viya 2025.02 stable. To learn more, see this post. SAS' scheduling capabilities have come a long way since the Warehouse Administrator AT days. SAS now supports multi-job flows with complex dependencies, complex execution schedules, sub-flows, command line integration, and more.   Sample Job Flow    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   However there is still one feature we have to implement ourselves, event based triggers. In this article, we'll explore creating event-driven processing with just SAS Environment Manager Jobs & Flows and some simple Linux commands.   The Need for Event Based Scheduling   In this scenario we want to trigger a SAS flow as soon as a third party process completes. Our SAS flow uses the output from the third party job as input. However we're told the process could finish anywhere from 9:00PM, to 1:00AM. We don't want to wait around until 1:00 every morning to start our flow though. We need to trigger when the job completes.   Our File-Based Event Trigger Scheme To build our Event-based trigger, we'll combine a few different elements.   1. The Trigger File First we'll ask the owners of the third party process to output a file when it completes. We'll call this a "process completion file." It is usually an empty text file but may contain some metadata-type information like the process completion time and/or how many records were written to the output files. Its sole purpose is to signal that the process is complete and its output is ready. Our file will be named "factTableComplete.tag." The creation of this file is the event we'll use to trigger our process.   2. A Linus Command to Detect the File Next we'll supplement our SAS Flow with a Command Line Action node. In it, we'll issue a Linux command to look for the trigger file. Our command will look like this:   'find "/gelcontent/data/factTableComplete.tag"'   With the command line action, our job flow will look like this:     3. A Return Code-Specific Dependency on the Find Command Using bash, the Find command will issue a return code of zero if it finds the file. If it does not, it will issue a one. So, we'll only let the downstream jobs execute if we get a zero. Thus, the downstream jobs will only execute if the 3rd party job is complete. Note that, we have three jobs directly dependent on the command line action. So we'll have to set this condition for all three dependencies.     4. A Short Duration Schedule that Runs the Job Flow Over and Over Again Now we'll schedule our job flow to run every 5 minutes for 60 occurrences. This gives us a 5 hour window for the flow to find the trigger file. Each execution before the trigger file arrives will end when the file is not found. When the file is found, the down stream jobs will execute.     5. Another Command Line Action to Remove the Trigger File Once the trigger file is located and the job flow runs beyond the initial Command Line Action, we only want the job flow to run once. So we'll have to remove the trigger file so the down-stream jobs don't run every 5 minutes. We'll also set the file deletion action to start when the dependent jobs start, not when they end. This will ensure that the file is removed in time for the flow's next execution.      What our Flow Execution Looks like   With this scheme in place and running, here is a sample of the flow execution history. First we see two executions where the flow was "Canceled." Digging deeper into the second canceled flow, we see that the 3rd party completion file wasn't found ending the flow. Then the completion file lands and the next flow execution succeeds because the file was detected and the down stream jobs executed successfully. Finally the flow goes back to the cancelled state since the completion file is not located again as the previous execution removed it.       Find more articles from SAS Global Enablement and Learning here.   ... View more

If you attended the SAS Innovate session on SAS Studio's Git integration capabilities, you may be interested in the links that were shown.  Here they are.   Instructions for you to replicate the demonstration shown with your own GitHub ID and SAS Studio Instance An article on SAS Studio Git Authentication Techniques A publicly available course that explains the SAS Studio Git integration The latest documentation on the SAS Studio Git integration Enjoy! ... View more

With its integrated SAS Compute server being an enhanced version of the traditional SAS engine, SAS Viya users have to change very little of their legacy SAS code when their implementation is configured to access the required data. For the most part, the SAS library definitions are all that need to change.   So how can we configure data access to on-premises data from SAS Viya in the cloud so we can migrate programs as easily as possible? We'll examine some strategies below.   1. SAS Cloud Data Exchange   SAS' Cloud Data Exchange (CDE) allows SAS Viya to access on-premises databases and odbc/jdbc sources as well as SAS Data Sets.   CDE utilizes a co-located data agent that sits within the SAS Viya cluster on the cloud and a remote data agent that sits on-premises. These services connect and pass the data back and forth between the cloud and the on-site network.   When utilizing CDE, SAS libraries need to be modified to access the CDE data source reference instead of the direct reference from when the code ran on-premises.   2. Cloud Provider Network Connections   Every modern cloud provider offers some mechanism(s) for connecting cloud networks securely with on-premises networks.   AWS offers Site-to-Site VPN connections as well as Direct Connect. Azure offers VPN and ExpressRoute Gateways. GCP offers VPN, Partner Interconnect, and Dedicated Interconnect.   Each of these technologies provides a network connection between your SAS Viya implementation and all of your on-premises resources. When using a technology like this, existing SAS programs will require almost no modification as the cloud resources should look like just other nodes on the network.   For more information on connecting SAS Viya on Azure to on-premises with VPN and ExpressRoute gateways see this post, Connecting Viya in Azure to On-Prem with Azure VPN, ExpressRoute (Intro).   3. Data Pipelines   Data Pipelines are sets of processes that move data from one platform to another. Pipelines can be used to synchronize key data sources from on-premises to SAS Viya in the cloud.   Like traditional ETL processes, data pipelines require development and regular maintenance. Also like ETL, initial loads, incremental loads, data transformation, required computing resources, execution frequency, and data staleness are all considerations.   When utilizing data pipelines, migrated programs may have to change more liberally as data storage formats may change. For example, database content may be migrated to parquet files in the cloud; Excel files may be migrated to CSVs; And so on.   For an example of a SAS Viya data pipeline using Singlestore see this post, SAS Viya with SingleStore: Near Real-Time Dashboarding Using SAS Visual Analytics.   4. Database Replication   Some on-premises data servers like Microsoft SQL Server offer replication capabilities to corresponding data servers in the cloud.   5. On Premises Cloud   Locating the cloud on premises using OpenShift is a completely different approach to those above. In this scenario, SAS Viya is deployed, not on a remote cloud provider like Azure, AWS, or GCP, but on local hardware using the OpenShift container platform. In this type of deployment, on-premises files can be accessed via mechanisms like the Kubernetes NFS storage class and databases can be accessed via mechanisms like traditional TCP/IP.   For information on deploying SAS Viya on Openshift, see this blog post, SAS Viya on Red Hat OpenShift – Part 1.   Further Considerations   Each of the strategies above comes with its own unique requirements including hardware, network, security, data latency, etc.. A Viya implementation may employ many different strategies for different data sources for reasons like data volumes, data volatility, and physical distance to the source.   Has your customer employed one of these strategies? Are there other strategies that they've employed? Thanks! ... View more

I am loading some data into a Viya 3.5 implementation that is being landed on the CAS controller node. The data is in the form of multiple CSV files within a zip file. Each file representing a different target table.    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Exploring Possible Load Strategies I was hoping I could treat the zip file as a directory and use CASL's elegant loading capabilities. However, I couldn't get CAS to treat the zip file as a directory. So I gave up on that plan.   Next, I next looked into using Viya 2021.1.4's enhancements around reading compressed files. However, these capabilities only to work for zip files that contain multiple files all of the same format intended to be loaded in the same target table. So again, I moved on.   The FILENAME ZIP Access Method Finally, I looked at modifying an approach I saw on this blog post that uses the FILENAME ZIP access method. This approach worked. The full code I used is below.   cas mysession sessopts=(metrics=true); caslib _all_ assign; filename inzip ZIP "/mnt/..../v1.3.0_csv_snapshot_2023_10_19_11_37_58.zip"; /* Read the files from the ZIP file and load their names into a work.contents */ data contents(keep=memname isFolder); length memname $200 isFolder 8; fid=dopen("inzip"); if fid=0 then stop; memcount=dnum(fid); do i=1 to memcount; memname=dread(fid,i); /* check for trailing / in folder name */ isFolder = (first(reverse(trim(memname)))='/'); output; end; rc=dclose(fid); run; /* Create a macro to copy a zip file member, write it to work, and load it to CAS */ %macro spin(csvfile); /* Write the file from the to the zip file to the work directory */ filename xl "%sysfunc(getoption(work))/&csvfile..csv" ; %put %sysfunc(getoption(work))/&csvfile..csv; data _null_; infile inzip(export/&csvfile.csv) lrecl=256 recfm=F length=length eof=eof unbuf; file xl lrecl=256 recfm=N; input; put _infile_ $varying256. length; return; eof: stop; run; /* Load the file from work into CAS */ proc casutil; load file="%sysfunc(getoption(work))/&csvfile..csv" outcaslib="casuser" casout="&csvfile" replace copies=0; quit; %mend; /* Spin over the zip file members */ data _null_; set contents; text=scan(memname,2); call execute('%spin('!!strip(text)!!');'); run;   As you can see, the code first examines the zip file and writes the names of all files within to a work data set named CONTENTS. Then we create a macro, SPIN, that extracts a CSV file from the zip file, writes it to the SAS WORK directory, and then loads it to CAS. Finally we use the CONTENTS data set to drive the macro with CALL EXECUTE so each file within the zip is processed one at a time by the SPIN macro.   Next Steps In our next post, we'll look at building a scheduling process for this code so that it can handle new files that appear.   Find more articles from SAS Global Enablement and Learning here. ... View more

I've had a number of requests for the hands-on session I led at this year's SAS Explore conference in Las Vegas. So, I thought I'd make the instructions publicly available. The hands-on exercises are now available on GitHub and should work for any properly configured SAS Studio environment.   The exercises step users through: Configuring their own GitHub repository complete with pre-built SAS programs and other content. Connecting SAS Studio with their new Github repository. Creating a local clone of the repository. Running SAS programs from the cloned repository. Staging, committing, and pushing modified SAS programs and SAS Studio Flows to the Git repository. The GitHub Repository with the hands-on instructions are located here: sas-studio-git-integration GitHub Repository.   The exercises are divided into three separate files, 1_SAS_Studio_GIT_Integration_Setup.md, 2_SAS_Studio_SAS_Program.md, and 3_SAS_Studio_Flows.md as shown in the image below.     ... View more

SAS' SingleStore partnership has led some of our customers to consider replacing their end-of-life SPD Server (SPDS) implementations with SingleStore.  After all, both are databases, more or less, and both are integrated with SAS in one way or another.   Here, we'll list the key functionality of SPDS and map it to the corresponding functionality in SingleStore (S2) to aid in migration.   1. SPDS Cluster Tables ≈ S2 Sharding SPDS Dynamic Cluster tables are, in SQL terms, essentially views of unioned tables. For example you might have 24 months worth of order data stored in 24 separate SPDS tables. A cluster table then could be created that combines all 24 into a single logical table.   Cluster tables offer enhanced query processing capabilities like member elimination and BY-group processing using information about the data in each member table as well as enhanced management capabilities that make rolling members in and out of the cluster easy.   Query Optimization SPDS cluster tables are very similar to Sharding in SingleStore. Sharding is essentially a form of range partitioning. Like cluster tables, sharding offers partition elimination as well as Group BY processing enhancement and other capabilities.   So, from a query optimization perspective, Singlestore should offer the same or better functionality. Therefore if a customer is currently using SPDS cluster tables for query optimization, they should choose a SingleStore Shard Key that achieves the same data distribution as the SPDS cluster table scheme.   Partition/Member Management Functionality was built in to SPDS to make rolling member tables in and out of cluster tables easy. That being a quick way to update the cluster table with the latest data while removing older, less relevant data.   SingleStore does not allow for manual manipulation of table partitions but claims that it doesn't need to since it can delete and load data so quickly.   Given the impressive numbers from the linked resources, it's clear that SingleStore could handle most customers' data volumes with simple SQL Delete and Insert operations instead of partition manipulation. However, if a customer is using cluster tables for member management, then its table maintenance processes would need to change when moving to SingleStore.   2. SQL Optimization SPDS has several advertised query optimization capabilities. However its star join capability is the true standout. It is truly world class for optimizing multi-table, star schema joins.   SingleStore also boasts world class query performance. This along with SingleStore's multi-machine processing capabilities ensure that S2 should perform as well or better than SPDS in almost all situations, assuming adequate hardware. However I would still check the S2 performance on any queries that triggered star joins in SPDS.    3. Indexes As a row-oriented data server, indexing is an important component of SPDS for optimizing queries. As a columnar data server, indexing in SingleStore fulfills a different role. As you can see in the linked material, the entire table structure in S2 is called the "Clustered columnstore Index." This is because, "in this layout, any one of the columns more closely matches the structure of an index in a row-oriented system." Thus we can think of the entire column store as an index for each field.   So, in general, there is no need to recreate all of the SPDS indexes in SingleStore. In fact, you couldn't even do it if you wanted to. You can of course create a primary key for each table as well as a shard key which determines how the data gets partitioned (see above). You can also create hash indexes but these are really to support transaction processing as opposed to the typical low cardinality uses of indexes in SPDS.   4. Sorting Similar to base SAS, SPDS relies heavily on pre-sorting for performance.   SingleStore also allows for pre-sorting by setting a sort key for each table.   Since sort keys can be any number of columns, make sure to create your SingleStore tables with a sort key that ensures the same sort order used for query optimization in SPDS.   5. SPDS Partition Files / Data Distribution Before multi-machine architectures were common, SPDS was designed to process big data efficiently by spreading table blocks over multiple I/O controllers and/or disks. This allowed SPDS to access multiple blocks in parallel with separate threads while avoiding I/O log-jams. This is a different concept than that of sharding / partitioning / cluster tables discussed above. It is at a lower level and occurs within each cluster table member.   Being a massively parallel processing engine, SingleStore utilizes multiple machines for both processing and storage. So, by its nature, it will also spread data across multiple disk arrays (assuming each machine is using local disk). Additionally disk striping could be used to ensure a similar affect to that of SPDS partition files.   As far as the topic of I/O optimization, SingleStore does not recommend one approach or another but simply requires certain I/O performance from its block storage. There are also I/O related parameters in SingleStore that can be configured to maximize performance. 6. SingleStore Table Structure All of the above information assumes that you will be converting to a SingleStore Columnstore table type. This is the optimal table structure for analytic and decision support style processing and is now the default table type in SingleStore's new Universal Storage offering which enhanced its columnstore format to handle transaction processing.   Given that columnstore is optimal for the same types of processing that SPDS is also designed for and now can handle transactional processing as well, it should be the table format chosen to replace SPDS. ... View more

Many prospective customers are curious about connecting SAS Viya on Azure with their on-premises data sources.  One option is SAS' CDE.  Another option is via Azure gateways.     As the above excerpt from the Microsoft Azure documentation shows, Microsoft recommends two different mechanisms for connecting SAS Viya in Azure to on-premises resources:   Azure VPN Azure ExpressRoute   Azure VPN allows for encrypted network traffic between an Azure virtual network and on-premises locations over the public Internet. ExpressRoute offers added security and performance by sending traffic over a dedicated private line. For an informative but non-technical video on VPN and ExpressRoute connections, see here.   Both methods require Azure Virtual Network (VNET) gateways to facilitate communication. (Site to Site VPN connections also require VPN devices configured on premises). Azure VPN gateways are discussed in the Azure VPN link above. Azure ExpressRoute gateways are discussed here. A video discussing Azure VNET gateways can be found here.   Microsoft offers documentation on configuring ExpressRoute and VPN gateways. There are also many youtube videos on the subject.   If you'd like more technical details on how to set up a VPN gateway and other considerations, check out Bogdan's series starting here, How to Connect SAS Viya in Azure to On-Prem with VPN Gateways – Part 1. ... View more