Git doesn’t need to be hard to understand and use, and in this post, I want to walk you through a practical way of using Git with SAS Studio in SAS Viya. SAS Studio provides a nice user interface to Git that makes working with Git much easier to use than using Git from the command line. No complex syntax to understand, no details to remember – you can simply point and click on what Git commands you want to use. And the information Git needs will be there for you - so no need to type in extra details.   See the screenshot below for a sample view of the Git user interface inside SAS Studio in SAS Viya:     Figure 1: Screenshot of SAS Studio Git user interface   My previous blog - Demystifying Git: A Survey of Data and AI Users on SAS - SAS Support Communities indicates that 59% from our local user community already use Git on at least a weekly basis for their data and AI work. The survey also tells us that 41% use Git less often – which tells us there is a potential to get more users to enjoy the benefits of using Git.   As you can find from this blog - Why Use Git for Data and AI Work in SAS Studio? - SAS Support Communities, there are good reasons to use a version control system like Git to save the day when you accidentally mess up your colleague’s work. Git will help you move back in time before the accident happen and get you back to a state when everything looked good.   In this post I will show you how to use SAS Studio with GitHub – one of the most popular hosted version control systems. SAS Studio also works well with Gitlab, Azure Repos and Bitbucket and quite a few others. And they will work quite similarly to how GitHub works with SAS Studio. Most of these version control systems can also be hosted by your own organization on-prem in case the hosted options are not available.   Prerequisites To be able to connect SAS Studio with GitHub, there are a few prerequisites that needs to be in place: A managed version control system – GitHub for instance, other systems work also, like GitLab, BitBucket, Azure Devops and quite a few others. Most of these also provides on-prem installations if there is a need to set these up inside your own network. SAS Studio enabled for Git integration – and integration through HTTPS if you want to connect through HTTPS, SSH Git integration if you want to connect through SSH An available authDomain where you can store a personal access token to authorize access to Git on your behalf – this was a change in SAS Viya from the 2024.12 release – see Prepare for Git Authentication Changes in SAS Studio on SAS Viya. If you don’t have any or don’t find any, ask your SAS Administrator to create one for you. Your personal Home folder on a file system mounted to SAS Studio. Although possible to use a commonly available file system, Git will be less able to properly track your changes when using a commonly shared folder.   Connecting through SSH or through HTTPS with Personal Access Token There are essentially 2 ways of connecting to GitHub from SAS Studio – through SSH or through HTTPS. What to use typically depends on policies in your organization – some may prefer SSH, and some may prefer HTTPS:   Using SSH involves a process of creating private and public keys and making those keys available to SAS Studio. Using HTTPS involves a process of creating a Personal Access Token in GitHub and making that token available to SAS Studio.   Although similar, I find HTTPS using a Personal Access Token more user friendly.   An overview of the process of how to connect SAS Studio with GitHub through HTTPS using a personal access can be illustrated as follows: Figure 2: Overview of connecting SAS Studio with GitHub   Generate Personal Access Token (PAT) in Github Enter SAS Environment Manager and store this PAT in My Credentials in an available authDomain Enter SAS Studio and Define a Git profile with HTTPS based on this authDomain (that holds the PAT) – use the userid and email that you have on your GitHub profile In SAS Studio, clone a git repository from GitHub based on this Git Profile Now you can work as usual in SAS Studio on the files you have access to in this cloned Git Repository. As this is cloned – you are now working on the local Git repository. Through the Git user interface in SAS Studio, you can stage the changed files if you are ready to commit these changes to your local repository. Commit with a meaningful commit message when are ready to do so. To get these committed changes back to GitHub, you need to Push the changes – and the Git user interface in SAS Studio helps you do that.   We will go through each of these in more detail, as there are some important details to be aware of. As the Personal Access Token is literally the key to your GitHub, care needs to be made both in GitHub when creating the token and when saving the token in SAS Viya.   For those interested in connecting to GitHub using SSH – there will be a separate blog about that approach.   So let us go through each of the steps above in more detail.   First let us log in to GitHub and create our Personal Access Token (PAT).   1. First time Git connection – creating the Personal Access Token in GitHub   Figure 3: Details on creating a Personal Access Token from GitHub   The following explains each point from the illustration:   Enter profile by clicking on your user image on the upper right corner Enter Settings Enter Developer settings in the menu that comes up Enter Personal access tokens (classic) Choose generate new token (classic) Specify name, expiration (duration) and repo scope – nothing else Hit generate new token at bottom of page Copy token Configure SSO if you need to authorize access to your organization’s repositories Hit the “Authorize” button to finalize authorizing access to the organization you want to authorize SSO into Now that we have our Personal Access Token, we can store it in My Credentials inside SAS Viya Environment Manager.   2. Saving the Personal Access Token in Environment Manager Figure 4: Details on storing the Personal Access Token from GitHub in My Credentials inside SAS Viya Environment Manager     The following explains each point from the illustration:   Enter My Credentials in Environment Manager Hit new credentials (asterisk) Pick an authDomain that is available – ask your SAS admin if you are unsure which to use or if you don’t find any, DO NOT use defaultAuth – it is used internally in SAS Viya Enter a userId for this credential. Use your full name - this is case sensitive, so type it in exactly as you want it – you might as well use the one you have in GitHub. When setting up the Git Profile in SAS Studio on the next page – SAS Studio will match against this UserId. Enter the personal access token (PAT) you created in GitHub as the password. We can now use our Personal Access Token by referencing this AuthDomain when defining the Git Profile in SAS Studio.   3. Define Git Profile in SAS Studio Figure 5: Details on defining a Git Profile in SAS Studio   The following explains each point from the illustration:   Under options menu, choose Manage Git Connections In the manage Git Connections window, choose Profiles Hit ‘+’ to add a git profile In the add profile window, choose the https option Give the profile a meaningful name – for instance My GitHub As the commit author name, enter the userId you stored in My Credentials (previous page)– exactly as written, this is case sensitive and will be matched against the entry with that userId As the commit author email, enter the email from your GitHub profile Select the authDomain where you stored the GitHub Personal Access Token (previous page) Hit ‘OK’ to save the new Git profile in SAS Studio Now that we have gotten ourselves a Git Profile in SAS Studio, we can use that Git Profile to clone Git repositories to our SAS Studio environment.   4.      Clone Git repository from GitHub Figure 6: Details on cloning a Git repository from GitHub to SAS Studio   The following explains each point from the illustration: In the ‘Manage Git Connections’ window, choose the Repositories page Hit the ‘+’ button to Clone a repository Enter GitHub in another browser tab, pick the repository you want to clone and hit the Code button (the green one) Select the HTTPS option Copy the link to the GitHub repository by hitting this button Paste the link to the GitHub repository you want to clone in this field In your home folder, create the folder you want to clone the repository to – you will not be able to clone into SAS Content as this is not a file system Name the folder for your local Git repository – the same name as the repository you are cloning from is a good practice - select the folder you created to get that back into the clone repository screen Select the Git profile you created in the previous page Hit ‘Clone’ and the GitHub repository will be cloned into the folder you specified   Now that we have cloned our repository from GitHub, we can work with the files and folders in that repository inside SAS Studio.   5.      Use Git from SAS Studio – develop and save work Figure 7: Details on how to develop and save work from a local Git repository in SAS Studio   The following explains each point from the illustration: The Git icon at the left of the folder indicates this is a Git managed folder in SAS Studio This button provides a list of Git repositories to work with inside SAS Studio You can now work in SAS Studio on files in this Git managed repository, save that work and Git will detect what files have changed This tab holds a user-friendly interface to work with Git – let us look at that in our next section.   5.4 Use Git in SAS Studio to stage and commit to our local repository Figure 8: Details on how to stage and commit changes to our local Git repository in SAS Studio   The following explains each point from the illustration: The user interface to Git in SAS Studio provides a list of files that Git has detected changes on locally inside SAS Studio This button performs a “stage” on all changed files – you can also stage files individually. Staging means we are telling Git that the changes to these files are ready to be committed to our local repository inside SAS Studio Once we are ready to commit these staged changes to our local repository – we can enter a meaningful commit message and hit the commit button.   Now that we have committed our changes to our local Git repository in SAS Studio, we can push those changes back to the remote repository in GitHub that we cloned it from.   6.      Push back to our remote repository in GitHub Figure 9: Details on how to push committed changes from our local Git Repository back to GitHub   The following explains each point from the illustration: As these changes are only committed to our local repository inside SAS Studio, we need to hit the push button to push these changes back to the repository we cloned it from. Notice the commit message now being visible in our remote GitHub repository which shows that these changes are now available in the remote GitHub repository as well   Summary – what we just did After all this detailed work, it makes sense to reflect on what we did – so let us review the illustration we started with: Figure 10: Overview of connecting SAS Studio with GitHub   First we generated our Personal Access Token (PAT) in GitHub. Then we stored this PAT in My Credentials in an available authDomain using SAS Viya Environment Manager. Then we defined a Git profile in SAS Studio with HTTPS based on this authDomain (that holds the PAT) – and we used the userid and email that we have on our GitHub profile Then we cloned a Git repository from GitHub based on this Git Profile in SAS Studio At this point, we are ready to work as usual in SAS Studio on the files we have access to in this cloned Git Repository. As this is cloned – we are now working on the local Git repository. Through the Git user interface in SAS Studio, we can stage the changed files when we are ready to commit these changes to our local repository. Commit with a meaningful commit message when ready to do so. To get these committed changes back to GitHub, we need to Push the changes – and the Git user interface in SAS Studio helps you do that.   So, with that – are you ready to use Git efficiently with SAS Studio in SAS Viya?   Questions and other feedback are welcome in the comments.   References used in this post: Demystifying Git: A Survey of Data and AI Users on SAS - SAS Support Communities Why Use Git for Data and AI Work in SAS Studio? - SAS Support Communities Prepare for Git Authentication Changes in SAS Studio on SAS Viya   ... View more

Recovering from a disaster – after deleting great work from my colleague Picture the scene – I accidentally saved a broken data pipeline after deleting some great work made by my colleague. With these days of rushing to get your work done – one could easily accidentally save something you shouldn’t – and in this case I happened to delete some nice work my colleague finished a few days earlier. So, his beautifully completed data pipeline now looks like this after I made a mess:     Luckily both me and my colleague used Git to save our previously committed changes, so his work was still available in the Git repository. Thus – by looking into our commit history we easily found my colleagues previously completed data pipeline:     By right clicking on the line with the highlighted commit message, we could reset our Git repository to that state, and we got his work back in again:     The ability to move back in time to a previous version is most likely the most important reason why you want a version control system like Git in your corner. In addition, Git provides you with a level of trust outside of our development environment as all committed changes are stored in the repository. Git also provides a safe space to collaborate in, even when working with colleagues who accidentally deletes your work.   SAS and Git Since its initial release in 2005, Git has grown in popularity and is today a ubiquitous part of the software development lifecycle. In a recent survey, over 93% of developers reported using Git.   SAS Studio provides Git integration to support a practical and efficient way of managing this code.   It is a standard practice to use Git with a managed version control system like GitHub, GitLab, Azure Repos, Bitbucket, and the like. They work quite similarly and can be hosted by your own organization or by the vendors who offer these systems.   As Git is a distributed version control system, each user will have a complete clone of the repository managed by Git.   A managed version control system typically holds the central and main clone of the repository being managed by Git. Thus, Git supports practices to synchronize the clones used by each user with a central version control system.   With Git and a centrally managed version control system, you get: Version control – Track changes and manage different versions of your code. Centralized backup – Ensure that your repositories are backed up on a central version control system. Documentation – Every committed version is documented, providing a clear history of changes. Collaboration – Git has built-in mechanisms for integrating work done across a team. Support for safe experimentation – Each clone can be worked on independently, and if anything happens that you don’t want back in – you just choose not to integrate this experiment. A practical way of reproducing a previous state (version) of your set of source code – Git provides ways of reverting back to any previous commits of a repository.   Over to you From a recent survey towards our local user community we discovered that 59% are using Git on at least a weekly basis – which shows that many data and AI users on SAS are already using Git – check out this blog: Demystifying Git: A Survey of Data and AI Users on SAS - SAS Support Communities.   I have a webinar coming up shortly – please sign up if you want to learn more about Git and how to use it effectively in SAS Studio for your data and AI work.   Register here:  Ask the Expert: Demystifying Git - Using git efficiently with SAS Studio | SAS UK ... View more

A recent survey of our local SAS user community in Norway found 59% of SAS data and AI users use Git at least weekly. The results tell us two things: Most SAS data and AI users actively use Git, but The SAS community’s use is behind that of the wider software development community, where over 93% of developers use Git   Key Benefits Among those who use Git actively, the survey also asked what they use Git for, which gives us an indication of its value.   The top three use cases all focus on collaboration, one of the five pillars of DataOps: Most SAS users use Git to support collaboration – which aligns with Git's intended purpose. As a distributed version control system, Git has great mechanisms for making repositories available across a team and integrating the work of all team members The Git platform allows users to share their work with others, and Allows users to benefit from work shared by others The popular practice of automating Continuous Integration and Continuous Delivery pipelines – known as CI/CD – is also gaining traction among this user group.   Conclusion There is tremendous value in all this usage of Git. Thus, I want to inspire the data and AI community to investigate the wonders of Git and give it a try. Modern user interfaces for Git, such as those available in SAS Studio, significantly lower the threshold for using it. I have a webinar coming up shortly – please sign up if you want to learn more about Git and how to use it effectively in SAS Studio for your data and AI work.   Register here:  Ask the Expert: Demystifying Git - Using git efficiently with SAS Studio | SAS UK ... View more

🎄 Is your organization working with data in Google Cloud and wondering how to integrate your SAS data? Look no further – this jolly holiday tip will gift you some wonderful, cost-efficient solutions! 🎁   SAS offers 3 practical ways of accessing data in Google Cloud: an optimized access engine for Google Cloud BigQuery read and write files stored in Google Cloud Storage, including parquet files a Spark access engine to work with data from Google DataProc See below for an illustration of these access methods: These access methods are supported on both SAS 9.4 and SAS Viya. The latest performance improvements for numerical data with BigQuery in SAS Viya have also been included in the SAS 9.4 M7 release. See below for a video showing you to do this, or read on in this juletip.     With CAS on SAS Viya, you have even more options, benefiting from our in-memory parallel analytical engine for enhanced performance.   Getting access 🎄 First things first – getting access! The easiest way is to use a JSON-based key file, which you can request from your Google Cloud administrator. Generating one based on a service account is convenient as we can reuse this across Google resources, like BigQuery and Google Cloud Storage. See below for a screenshot of the IAM & Admin page of Google Cloud Console how this may look like:     This service account should only have the necessary permissions – in this case, reading and writing to a Google Cloud Storage bucket and accessing BigQuery (read and write). Make sure to store this key file in a safe, protected location within your SAS system, such as your home folder. 🎁 The json-file typically looks like this:   {   "type": "service_account",   "project_id": "<your-google-project-id>",   "private_key_id": "<generated id>",   "private_key": "<-BEGIN PRIVATE KEY--a-quite-long-key--END PRIVATE KEY--\n>",   "client_email": "<an email address for the service account>",   "client_id": "<client-id>",   "auth_uri": "https://accounts.google.com/o/oauth2/auth",   "token_uri": "https://oauth2.googleapis.com/token",   "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",   "client_x509_cert_url":"<generated>",   "universe_domain": "googleapis.com" }   Google BigQuery   Let's start with BigQuery. SAS has simplified integration with various data sources, including Google BigQuery, through the libname statement. With the key-file that grants access to BigQuery, setting up a libname for BigQuery is straightforward:   libname bqlib bigquery cred_path="<path to key file>"     project="<project name>"     schema="google dataset name"     bulkload=YES     scanstringcolumns=yes ; The bulkload option is crucial for ensuring efficient write operations, while the scanstringcolumns option directs the access engine to scan string columns to determine their appropriate lengths.   Using this libname, SAS code can interact with data in BigQuery just as it would with path-based libnames on a local filesystem. This approach is highly practical for making data in BigQuery accessible for use outside of SAS, and not the least accessing data in BigQuery produced outside of SAS.   For example, the SAS Studio flows below will write to the three BigQuery tables on the right by changing the libname to point to BigQuery. SAS DI Studio on SAS 9.4 will look similar – and will offer a similar approach to making data available to Google BigQuery. You may notice that flows using BigQuery libnames might run somewhat slower compared to local path-based libnames. This is because BigQuery is accessed over the network from the SAS environment, and depending on the network configuration, some latency may be involved. The bulkload option helps mitigate this by storing data in a file close to BigQuery before bulk-inserting it into BigQuery.   With SAS Viya, you can leverage the in-memory parallel analytics CAS engine for enhanced performance. By using a CASLIB instead of a libname, you can load data into CAS and explore it directly from memory with SAS Visual Analytics. Since the data is already in memory, you won't need to send new queries to BigQuery, which helps reduce costs.   Data loading into CAS can be done in parallel across CAS worker nodes, and we've optimized the access engine for both loading data into CAS and writing from CAS to BigQuery. If you're curious about how this can be done, feel free to ask us in the comment field.   Google Cloud Storage Google Cloud Storage (GCS) offers a practical solution for saving files for use outside of SAS or for loading data into SAS from external sources. You can save various types of files, such as CSV, Excel, and even sas7bdat files, providing a secure place for later use.     Here's how to save a sas7bdat file to a bucket in GCS using the same key mentioned above from your SAS system:   /* Writing a sas7bdat to GCS */ filename sasin "<path of sas7bdat file in your sas system to export to gcs>" recfm=n lrecl=256 ; /* The file to be created will be accessible on GCS */ filename gcsout gcs "<path to file in your bucket>"                     key="<path to key file> "                     bucket="<name of bucket>" recfm=n lrecl=256 ; options msglevel=i ; /* Copy a SAS data set on GCS */ data _null_ ;    rc=fcopy('sasin','gcsout') ; run ; If you want to export a SAS dataset to a CSV file, use this code snippet:     /* Reference output file on GCS */ filename gcsout gcs "<path and file in the bucket>"    key="<your key file>"    bucket="<name of bucket>" ; /* Export a SAS data set to a csv file on GCS */ proc export data=sashelp.cars file=gcsout dbms=csv replace ; run ;   Importing data from a CSV file in GCS to a SAS dataset is similar:   filename gcsin gcs "<path and file in the bucket>"                     key="<your key file>"                     bucket="<name of bucket>" ; /* Export a SAS data set to a csv file on GCS */ proc import datafile=gcsin out= work.<data imported> dbms=csv  ;    getnames=yes; run ;   Google Cloud Storage is "bottomless," allowing you to easily add storage capacity as needed. This can be particularly useful if the storage mounted to SAS is reaching its limit.   Working with parquet files in GCS With the ability to load and save to Google Cloud Storage, you may also want to convert SAS data to an open file format like Parquet. You may even want to save data from Google BigQuery to the Parquet format in Google Cloud to save on queries to Google BigQuery. Compressed Parquet files save space and provide practical integration with the world outside of SAS. SAS has a Parquet engine that lets you read Parquet files directly from Google Cloud Storage quite efficiently. You can set up a libname towards the bucket as follows:   libname pqfiles parquet "<path in your GCS bucket>"    storage_platform="gcs"    storage_bucket_name="<bucket name>"    storage_gcs_key_file="<path to key file>"; Reading from such a libname works fine for most data types, including float, integer, and strings. Dates typically come across as strings, so some reformatting may be needed based on how the date has been formatted in the string. For instance, if the data is formatted as YYYY-MM-DD in the column Date, you can reformat it as follows:     data work.data;     set pqfiles.data;     Datefield = input(Date, yymmdd10.);     format Datefield yymmdd10.;     drop Date;     rename Datefield = Date; run;   Unfortunately, when writing to such a libname, not all data types get converted correctly; dates and floats may not be handled properly. With a little help from Python and a user-contributed sasdata2parquet function, we can sort this out. On SAS Viya with Python installed and enabled, you can use Python inside SAS Studio. Alternatively, you can use your own Python environment with saspy installed by running:   pip install saspy You need a configuration file to connect and authenticate to your SAS environment. Save this file with a .py extension in a well-protected area, such as your home folder, as sascfg_personal.py:   SAS_config_names=['httpsviya'] SAS_config_options = {'lock_down': False,                       'verbose'  : True,                       'prompt'   : True                      } SAS_output_options = {'output' : 'html5'}       # not required unless changing any of the default httpsviya = {'ip'       : '<server host name fully qualified>',              'context'  : 'SAS Studio compute context',              'user'   : '<your user id on sas>',              'pw'   : '<your password on sas>',              'options'  : ["fullstimer", "memsize=1G"]             } For more details and other authentication options, check out the documentation of saspy at Getting started — saspy 5.101.0 documentation.  With the configuration in place, you can convert your SAS dataset to a Parquet file with the following Python code: import saspy sas = saspy.SASsession(cfgname='httpsviya', cfgfile='<path to your configuration file>’) log=sas.saslib(libref='EXPORTLIB', path='<path to folder where you want to export sas data set from>’) sas.sd2pq(parquet_file_path ='<local path where you want to save your parquet file>', table='TOY_SUMMARY', libref='EXPORTLIB') Our newfound ability to save data to parquet files from SAS will also help us save data from BigQuery to Parquet using the same approach. Saving data from BigQuery to Parquet provide a cost efficient way of saving data from Bigquery into very cost efficient and convenient storage; ready for exploration and analytical consumption. A huge cost saving compared to loading data from BigQuery as each load of data will incur cost from Google Cloud. We just need to setup a libref to Bigquery and use the same approach as above - see the following:   import saspy sas = saspy.SASsession(cfgname='httpsviya', cfgfile='<path to your configuration file>’) log=sas.saslib(libref='BQLIBPY', engine='bigquery', options = 'cred_path="/export/home/users/norlsk/sa_key.json" project="sas-gelsandbox" schema="sas_innovate" bulkload=YES scanstringcolumns=yes') sas.sd2pq(parquet_file_path ='/export/home/users/norlsk/datafromBQ.parquet', table='TOY_SUMMARY', libref='BQLIBPY')   With the parquet files in place, you can then upload them to GCS – if the file is locally on your own laptop you can simply upload directly to your bucket using the user interface  for Google Cloud Storage as shown below: Within SAS, you can save the parquet file to GCS similarly to how you save your sas7bdat files: /* Writing our local parquet files out to GCS */ filename sasdata "/export/home/users/norlsk/data.parquet" recfm=n lrecl=256 ; filename sasbq "/export/home/users/norlsk/datafromBQ.parquet" recfm=n lrecl=256 ; /* The files to be created will be accessible on GCS */ filename gcssas gcs "data/datafromsas.parquet" key="/export/home/users/norlsk/sa_key.json" bucket="norlsk-fans-demo" recfm=n lrecl=256 ; filename gcsbq gcs "data/datafromBQ.parquet" key="/export/home/users/norlsk/sa_key.json" bucket="norlsk-fans-demo" recfm=n lrecl=256 ; options msglevel=i ; /* Copy SAS data and Bigquery data as Parquet files to GCS */ data _null_ ; rc=fcopy('sasdata','gcssas') ; rc=fcopy('sasbq','gcsbq') ; run ; If you are interested in data access through Dataproc and spark, we have covered spark-based access in this blog from earlier this year: Data to Databricks? No need to recode - get your existing SAS jobs to ... - SAS Support Communities.   Are you ready to work with SAS data in Google Cloud using BigQuery and Google Cloud Storage? Let us know in the comments what you plan to use it for.   Merry Christmas and a joyful New Year with SAS data on Google Cloud! 😊 ... View more

Freedom of choice is now more important than ever, especially when you want to get the most out of your data as an AI specialists. Boasting compatibility with 80+ data sources, including widely adopted open data standards and formats such as Parquet and ORC, the possibilities for data access are virtually limitless with SAS Viya. Check out my colleague Cecily’s blog on how easy it is to set up data access with SAS Viya. Databricks is one of these data sources that SAS Viya works well with, as explained in my colleague Jarno’s blog on harnessing the analytical power of Databricks with SAS. Most versions of Python are supported by Databricks, and as Jarno explains in his blog, you can write Python code natively in SAS Studio on SAS Viya as well. We will touch upon how to use Python inside SAS Viya as well as in your favorite Python tool of choice when using SAS Viya.     Figure 1: SAS Viya as the execution hub for SAS and open-source coders.   As the illustration above shows, SAS Viya not only excels in its native language but also: Embraces Python and other popular coding languages for analytics Offers a plethora of integration options, including intuitive visual interfaces Provides programming interfaces using Python, R, and other languages, with optimized libraries to access functionality within SAS Viya Delivers API interfaces for direct access to the majority of functionalities inside SAS Viya, accommodating various types of users See below for a deeper dive into these options and discover how to leverage SAS and Python for enhanced analytical prowess.     Use of Python in SAS Viya for Model development and Decision flows The tools in SAS Viya for model development and building decision flows have explicit support for Python in code nodes where you can write your own Python code directly. The illustration below shows how you can compare a Python gradient boosting model against an R random forest model and a couple of SAS models in Model Studio.   Figure 2: Analytical pipeline in SAS Model Studio showing a mix of Python, R, and SAS models.   Using Python inside SAS Studio In SAS Studio, the Data Engineering tool on SAS Viya, there is native support for Python in the editor. In addition, you can choose code nodes with Python code as well as SAS code when building visual data pipelines. You can even create custom transformation steps with python code and share with your colleagues to enhance their productivity. Figure 3: Writing Python code in a visual data pipeline in SAS Studio.   SAS Notebooks in Visual Studio Code with SAS Extension Visual Studio Code with SAS Extension supports SAS Notebooks where you can code with SAS blocks and Python blocks, similar to how notebooks work in Jupyter lab and Databricks. Figure 4: Using SAS Extension in Visual Studio Code with a SAS Notebook – SAS Code and Python.   Access to SAS Viya from Python environments Any Python environments that have network access to a SAS Viya environment can use specialized Python libraries curated and powered by SAS to provide efficient access to the power of SAS Viya: SWAT to access the in-memory based analytical engine. sasctl for convenient access to the platform for model management. sasdply to access SAS deep learning features sasoptpy  to quickly formulate and solve mathematical optimization problems CVPy  provides APIs to assist with the processing, analyzing, and visualization of images There is also a set of useful administrative utilities at pyviyatools that are useful for CICD automation and more. You can even access the REST APIs to SAS Viya directly to access most functions available.   Run Python inside the in-memory based parallel analytics engine of SAS Viya If you want to get even deeper in SAS Viya with your Python code, you can have your Python code executed directly inside the SAS in-memory parallel analytics engine using the in-memory gateway action set, see SAS Help Center: CAS Gateway Details. Figure 5: Python code running in parallel on the SAS in-memory server.   Summary SAS Viya offers a rich array of Python integration options, catering to both built-in SAS Viya applications and external Python environments seeking to tap into the formidable capabilities of SAS Viya. This versatility combined with access to most types of data sources available empowers data scientists, analysts, and AI teams with the freedom of choice to seamlessly collaborate across platforms.   Learn more about SAS and Databricks Expand your SAS and Databricks knowledge further by exploring these blogs by my colleagues, and anticipate our upcoming blogs posted every Wednesday in SAS Communities. Harness the analytical power of your Databricks platform with SAS Data everywhere and anyhow! Gain insights from across the clouds with SAS Elevated efficiency and reduced cost: SAS in the era of Cloud Adoption SAS and Databricks: Your Practical Guide to Data Access and Analysis Data to Databricks? No need to recode - get your existing SAS jobs to SAS Viya in the cloud     ... View more

What if you could make your valuable and business-critical SAS Data Integration jobs work with Databricks in the cloud in a fraction of the time it would take to recode the jobs in another language or system? The following illustration shows instant modernization of SAS 9 data pipelines moved to SAS Viya in the cloud.   Figure 1: SAS 9 data pipelines moved to SAS Viya and instantly modernized.   Transferring SAS Data Integration Jobs to SAS Studio in SAS Viya in the cloud will not only significantly reduce migration time, but also expedite the availability of your data and logic in the cloud thanks to the built-in automatic migration support in SAS Viya. Moreover, this will allocate more time and resources to further development work that adds value to your business, such as AI modeling, enhancing data quality, and other analytical processes that SAS Viya excels at and accomplishes faster than alternative solutions. For more details, refer to this independent study conducted by Futurum Group.   Take advantage of SAS Viya’s built-in automatic migration tools Leveraging SAS Viya's built-in automatic migration of your trusted SAS Data Integration jobs will notably mitigate risks in the process, allowing you to maintain the logic that has already proven effective. SAS migration tools capitalize on a thorough comprehension of the disparities between SAS 9 and cloud-native SAS Viya. This enables productive maintenance and management of logic through transformation steps resembling those in SAS 9 Data Integration Studio. Figure 2:  SAS Data Integration Studio Jobs automatically migrated to a SAS Studio flows in SAS Viya     Below are some useful resources that show and explain aspects of the migration process: SAS Help Center: SAS Viya Platform: Content Migration from SAS 9.4 SAS Viya 2021.2 – Migrating SAS Data Integration (DI) Studio Jobs to SAS Studio Flows Content Assessment delivers utility applications to support SAS Viya Content Migration SAS Demo | SAS Viya Migration of SAS Data Integration Studio Jobs to SAS Studio Flows   Switch from SAS tables to Databricks tables and continue with low code development. Using SAS Viya's access engine with Databricks ensures efficient and readily available data. This engine translates transformation steps into Databricks-compatible SQL, streamlining integration. Flows can be optimized to run entirely within Databricks, reducing or eliminating data movement to SAS Viya. As illustrated below, you have an option to store intermediate tables in Databricks so that the entire flow runs in Databricks.     Figure 3: Configuring SAS Viya to store intermediate tables in Databricks, enhancing data storage and processing within the platform.   With SAS Viya you have the flexibility to store and access data where you want – locally in SAS Viya, other data sources like Parquet in cloud storage, Databricks or another data platform. See my colleague Cecily's blog for how easy it is to set this up:  SAS and Databricks: Your Practical Guide to Data Access and Analysis. Use the option that suits your use case the best.   And now that you have your data pipelines available in SAS Viya you can evolve this further by tapping into new capabilities available on SAS Viya; adding python code, optimized machine learning, entity resolution or any other AI or data functions you can imagine.    Migration from SAS 9 to SAS Viya is so much faster and simpler than rewriting to another language and technology. And you get to run the pipeline on Databricks with access to data in Databricks. As an additional benefit you get the power of a highly optimized cloud-native data and AI engine in SAS Viya to further evolve your analytical pipelines.   In summary, the fastest, most cost-effective, and safest way of moving your SAS 9.4 data pipelines to Databricks in the cloud is through SAS Viya in the cloud. You get to leverage your existing intellectual property from the logic you already invested in from your SAS 9.4 data pipelines while making data available in Databricks in the process. You also have the choice of the logic running on Databricks or on SAS Viya.   Learn more about SAS and Databricks Expand your SAS and Databricks knowledge further by exploring these blogs by my colleagues, and anticipate our upcoming blogs posted every Wednesday in SAS Communities. Harness the analytical power of your Databricks platform with SAS - SAS Support Communities Data everywhere and anyhow! Gain insights from across the clouds with SAS Elevated efficiency and reduced cost: SAS in the era of Cloud Adoption SAS and Databricks: Your Practical Guide to Data Access and Analysis     ... View more

What if you could make your valuable and business-critical SAS Data Integration jobs work with Databricks in the cloud in a fraction of the time it would take to recode the jobs in another language or system? The following illustration shows instant modernization of SAS 9 data pipelines moved to SAS Viya in the cloud.   Figure 1: SAS 9 data pipelines moved to SAS Viya and instantly modernized.   Transferring SAS Data Integration Jobs to SAS Studio in SAS Viya in the cloud will not only significantly reduce migration time, but also expedite the availability of your data and logic in the cloud thanks to the built-in automatic migration support in SAS Viya. Moreover, this will allocate more time and resources to further development work that adds value to your business, such as AI modeling, enhancing data quality, and other analytical processes that SAS Viya excels at and accomplishes faster than alternative solutions. For more details, refer to this independent study conducted by Futurum Group.   Take advantage of SAS Viya’s built-in automatic migration tools Leveraging SAS Viya's built-in automatic migration of your trusted SAS Data Integration jobs will notably mitigate risks in the process, allowing you to maintain the logic that has already proven effective. SAS migration tools capitalize on a thorough comprehension of the disparities between SAS 9 and cloud-native SAS Viya. This enables productive maintenance and management of logic through transformation steps resembling those in SAS 9 Data Integration Studio. Figure 2:  SAS Data Integration Studio Jobs automatically migrated to a SAS Studio flows in SAS Viya     Below are some useful resources that show and explain aspects of the migration process: SAS Help Center: SAS Viya Platform: Content Migration from SAS 9.4 SAS Viya 2021.2 – Migrating SAS Data Integration (DI) Studio Jobs to SAS Studio Flows Content Assessment delivers utility applications to support SAS Viya Content Migration SAS Demo | SAS Viya Migration of SAS Data Integration Studio Jobs to SAS Studio Flows   Switch from SAS tables to Databricks tables and continue with low code development. Using SAS Viya's access engine with Databricks ensures efficient and readily available data. This engine translates transformation steps into Databricks-compatible SQL, streamlining integration. Flows can be optimized to run entirely within Databricks, reducing or eliminating data movement to SAS Viya. As illustrated below, you have an option to store intermediate tables in Databricks so that the entire flow runs in Databricks.     Figure 3: Configuring SAS Viya to store intermediate tables in Databricks, enhancing data storage and processing within the platform.   With SAS Viya you have the flexibility to store and access data where you want – locally in SAS Viya, other data sources like Parquet in cloud storage, Databricks or another data platform. See my colleague Cecily's blog for how easy it is to set this up:  SAS and Databricks: Your Practical Guide to Data Access and Analysis. Use the option that suits your use case the best.   And now that you have your data pipelines available in SAS Viya you can evolve this further by tapping into new capabilities available on SAS Viya; adding python code, optimized machine learning, entity resolution or any other AI or data functions you can imagine.    Migration from SAS 9 to SAS Viya is so much faster and simpler than rewriting to another language and technology. And you get to run the pipeline on Databricks with access to data in Databricks. As an additional benefit you get the power of a highly optimized cloud-native data and AI engine in SAS Viya to further evolve your analytical pipelines.   In summary, the fastest, most cost-effective, and safest way of moving your SAS 9.4 data pipelines to Databricks in the cloud is through SAS Viya in the cloud. You get to leverage your existing intellectual property from the logic you already invested in from your SAS 9.4 data pipelines while making data available in Databricks in the process. You also have the choice of the logic running on Databricks or on SAS Viya.   Learn more about SAS and Databricks Expand your SAS and Databricks knowledge further by exploring these blogs by my colleagues, and anticipate our upcoming blogs posted every Wednesday in SAS Communities. Harness the analytical power of your Databricks platform with SAS - SAS Support Communities Data everywhere and anyhow! Gain insights from across the clouds with SAS Elevated efficiency and reduced cost: SAS in the era of Cloud Adoption SAS and Databricks: Your Practical Guide to Data Access and Analysis     ... View more

When migrating your SAS 9.4 content to SAS Viya in the cloud – you want to work with data that is optimized for the cloud - how about using Snowflake? And make sure that the data falls right into and from Snowflake(winter pun for the season 🌨😀 ) Snowflake works well in the cloud – and only works in the cloud. And while migrating what you made in SAS 9.4 to SAS Viya in the cloud it is a simple matter to make data available into Snowflake. Through our close partnership with Snowflake, SAS has an optimized access engine with Snowflake that ensures smooth access taking advantage of pushdown SQL so that data movement can be minimized. We find that many customers of SAS use Snowflake in order to make data processed by SAS available for consumption outside of the SAS environment.   When doing that, you want to make sure that what you get in Snowflake is right – it may very well be that when migrating to a different platform we may get something different from what we expect.   In our case, we want to migrate this DI Studio job as shown below:   To this SAS Studio flow in SAS Viya: The migration as such was done fully automatic – but the destination tables were changed to a Snowflake based library so that we would make the aggregated data available in Snowflake for others to consume rather than SAS Datasets.   A very efficient way of making sure that we get what we expect in Snowflake is by comparing what we get in Snowflake with what was produced in the SAS DI Studio job as SAS datasets. It so happens that SAS comes with a PROC COMPARE function that does exactly that.   Still – there are a few things to be aware of when comparing SAS datasets to a relational datawarehouse like Snowflake: Data in relational data warehouses like Snowflake are unsorted – so for PROC COMPARE to work right you would need to sort the data And while you are at it, you may want to sort the SAS dataset you are comparing with even though SAS Datasets do have a default sort order. If you know the default sort order of the dataset – you can just sort the Snowflake data in the same sort order and compare directly Numbers in Snowflake are represented differently from SAS, so they may end up with some differences if they are a result of aggregation – so when comparing what kind of tolerance would you allow to accept the comparison? PROC COMPARE simply compare all data row by row – so you may want to control the number of rows you compare to ensure it complete in a reasonable amount of time. Use PROC COMPARE with a WHERE clause if you need to filter the data for instance.   Use of PROC COMPARE to compare the Snowflake table produced by the migrated SAS Studio flows to the expected SAS datasets produced by the original SAS DI Studio jobs will look like this: proc sort data=SNOWFL.Toy_Summary out=Toy_Summary; by Date ProductBrand ProductLine Product; run; proc sort data=EXPECTED.Toy_Summary out=exp_Toy_Summary; by Date ProductBrand ProductLine Product; run; proc compare base=exp_Toy_Summary compare=Toy_Summary METHOD = absolute criterion = 0.01 briefsummary; id Date ProductBrand ProductLine Product; run; proc sort data=SNOWFL.Novelty_Summary out=Novelty_Summary; by Date ProductBrand ProductLine Product; run; proc sort data=EXPECTED.Novelty_Summary out=exp_Novelty_Summary ; by Date ProductBrand ProductLine Product; run; proc compare base=exp_Novelty_Summary compare=Novelty_Summary METHOD = absolute criterion = 0.01 briefsummary; id Date ProductBrand ProductLine Product; run;   Here we are using the criterion of 0.01 to allow for a 2 decimal point difference when comparing numbers. This can make sense if we are considering these numbers used for accounting purposes.   Such a test can be used as part of automating the test of the SAS Studio flow we just migrated and by that be used in a CICD pipeline as well. If you found this juletip useful, you may enjoy these blogs as well on how to migrate your SAS 9.4 Enterprise Guide projects and SAS 9.4 DI jobs to SAS Viya Studio flows: Migration of Enterprise Guide to SAS Viya Migration of DI Jobs with SAS Datasets to SAS Viya while making data available in Snowflake Using CICD to automate testing and operationalization of your migrated SAS Studio flows with Snowflake   And by this approach we have validated that our Snowflake data falls 🌨 nicely and can be used for analytical consumption also outside of SAS. ... View more

This is the second part of the blog series on moving your data and analytics to the cloud with SAS Viya and Snowflake, based on the experience of supporting a global SAS customer. This blog describes the process of operationalizing data pipelines using a continuous improvement, continuous deployment (CICD) pipeline once you have completed testing in the development environment. At which point, you are ready to have the changes pushed to git so that a CICD pipeline can take it from there.   Moving into git Fortunately for us, SAS Studio has a user-friendly interface to git. It is therefore easy to commit the changes to git and push them into our remote repository managed by Azure DevOps. Azure DevOps will then start a CICD pipeline that is configured using the repository. In our case, the pipeline was triggered by the push of changes into the repository, but pipelines can also be triggered by time and other events. A CICD pipeline typically has three stages: Build, Test and Deliver. Each stage has a set of jobs to fulfil those stages. Some pipelines may also automate right through to deployment of artifacts to make them fully operational. In our case, that final step was left for those in charge of deployment. This is often the case in regulated industries or where there are other reasons to have those steps done manually. The three stages of deployment all have important roles: Build – We build what we need to be able to test and later deliver what is needed. This stage also includes preparations required to run a trustworthy test at the next stage. This includes cleaning out previous results so the validation will be based on the results created in the test run. Test – We run the same tests as we did when we developed our flow, but this time in our test environment. This is configured like the production environment, so that we can be confident that we will get similar results in production. Deliver – We do some preparations in the production environment before delivering the now tested artifacts all the way to production.   Requirements to support a CICD pipeline Three environments are required to enable a CICD pipeline (development, test and production), coupled with Azure DevOps and an Azure DevOps agent. In our case, the Azure Devops Agent was a self-hosted agent running on Linux that we registered to this Azure Devops project (there is a good primer on how to do this here). We followed this guideline and placed our self-hosted agent in the default agent pool. The self-hosted agent can run on any Linux machine (virtual or physical) that has network connectivity to Azure Devops and the environments it interacts with. In our case, this was the test and production environments. We equipped the agent with our SAS-Viya CLI (link will require that you identify yourself to sas.com - anyone can register), so it could communicate with the SAS Viya environment through scripting commands to be executed on the agent. We also added Python Viya Tools from GitHub to further facilitate communication with the SAS Viya test and production environments and enable the agent to run the required stages. The Azure Devops Pipelines was a business preference, but there are other pipeline options (there is a great video here that shows the key concepts of an Azure Devops Pipeline). Azure Devops provides several ways of structuring the pipeline, incorporating parameters, variables and predefined tasks that provide access to Azure-based resources. We used a few tasks that were proprietary to Azure, but mostly the pipelines were built around bash scripts that will run on most Linux-based distributions. Using bash scripts also provides a practical way of testing and developing these scripts in the pipeline. To create a pipeline in Azure Devops, enter the Pipelines section of your Azure Devops project and hit the “New pipeline” button. This will lead you through a wizard that connects the pipeline to the git repository and define the initial pipeline. The pipeline can be edited directly inside Azure Devops as well as in text-based editors. The pipeline is managed in git along with other artifacts in the repository. There is more about how to specify jobs in your pipeline here. However, just as an example, we had jobs in the Build stage to prepare the test environment, clean the results from the Snowflake data warehouse, load the flows from the local repository to the test environment, and generate the SAS code for the flows to be ready for execution.   Automating, accelerating, and migrating It is always important to consider the practical side of moving data and analytics to the cloud. This blog series has discussed the key lessons from one migration, but generalized them more widely. The key lesson for me throughout was to focus on the business expectations and requirements, and then look for the fastest, most efficient way to achieve the changes required. This underpinned most of the operational decisions made during the migration process, whether that was the order in which to move data, or minor issues about naming libraries. We also took full advantage of the tools within SAS Viya, such as the PROC COMPARE function. I think this provides a useful way to migrate data and analytics to the cloud, taking advantage of the efficient integration between SAS Viya and Snowflake. Automating the process through a CICD pipeline was perhaps the icing on the cake, but I can certainly recommend this addition.   ... View more

Many organizations are now moving analytics and data to the cloud. This has many benefits, including scalability, cost-efficiency, and accessibility. However, there are also some practical questions that admins need to consider. This blog series looks at some of those questions.  It is based on the experience of helping a global SAS customer to carry out this process. This first blog looks at what you need to consider before you start, and how you might make those decisions, up to the start of the operationalization process.   Before you start: decisions, decisions... The first aspect to consider is the need to provide access to both a modern analytics ecosystem, and a modern data fabric. This is needed to make data available for your analytics ecosystem in the cloud. We used Snowflake, which provides a cloud-based data warehouse, and is often used as a modern data platform. SAS Viya as a solid base for a modern analytics ecosystem works well with other cloud storage services as well, which is ideal, because data access needs to be provided through a set of storage services optimized for the customer.   The second aspect to consider is the data that need to be migrated. Essentially, you must understand the business needs, including compliance requirements, volume, variety, velocity, and sensitivity of the data. This affects how you will transfer the data in practical terms. You also need to consider any specific requirements. In our case, there was a requirement to have data available for consumption beyond the SAS analytics platform and making data available in Snowflake from SAS Viya supports that well.   A question of latency Data has gravity. It is therefore important to consider latency when transferring data between data centers. In our case, the customer’s current environment is in Europe, and the Snowflake data warehouse is served by Azure in the eastern US. The connection between the on-prem environment to Snowflake in the US is via a VPN connection, which causes high latency, and is therefore less suitable for transferring large volumes of data. The new SAS Viya environment is served by SAS Cloud on an Azure data center in Europe.   It therefore made sense to transfer the data to SAS Cloud first, then between Azure data centers to Snowflake. This order of transfer could leverage the Azure backbone network, and therefore provide much lower latency. This also means that data needed on the SAS Viya environment can be made available directly there, where they originate, reducing the volume of data transfer needed.   Consider alternatives It is also worth considering alternative approaches to use to evaluate them against each other. Generally, you want to evaluate each one on speed, cost, and general value as part of the process of moving to the cloud. In this case, we compared the approach of reusing existing assets in Data Integration Studio, rather than rewriting work to another platform in the cloud. We concluded that this approach would be: Faster - Shorter time to value because less work is required than rewriting. This approach only requires small changes to where data is written and read from to optimize it for the cloud. The changes are based on existing and proven data pipelines. Cheaper – reusing existing and proven data pipelines typically requires 5–10 times less work, which reduces the cost. We can also automate testing and delivery through a continuous improvement, continuous development (CICD) pipeline, which is also cheaper than manual testing and delivery. Better – writing analytical data to Snowflake improves accessibility of analytical data for consumption outside SAS. Risk and complexity are reduced when working from proven data pipelines in a productive environment. We would also be reducing the amount of work needed, and therefore freeing up resources.   Support business preferences You also need to know about any business preferences that you will need to accommodate. For example, in this job, the customer had some data integration jobs that needed to be migrated and amended to write into slightly different places. The business preference was to keep the jobs as they were until after migration, rather than amend first. This allowed them to decide later exactly how to use the jobs and maintained them in their existing form. It also made the migration easier, because we were able to simply export them as a package.   The key issue was to transfer workflows exactly as they were. You can then work on them in the new environment to ensure that they are fit for purpose there. We used the function in SAS Viya that allows you to import a SAS Package directly. You simply need to decide which folder to import the package into. It is helpful to reestablish the SAS Library with the same name as in the source environment because this saves mapping work. You can then amend the flows to write to Snowflake instead of SAS Datasets.   Test and validate Before any refactoring, it is good practice to write test code to validate whether your changes are implemented as expected. Within SAS, you can use the powerful PROC COMPARE functionality to compare what is being produced in Snowflake with the results from the source environment. You can also compare aggregated tables, including building in a level of tolerance for the different environments. We developed test code to run the tests in our CICD pipeline as well as to validate the changes to the flows. We reused test data and expected results from the source environment to have a baseline to work with for efficient validation.   Once you are confident that all the imported flows work as expected in the new environment, you can start the operationalization process. This is covered in the next part of the blog series, on Moving to Viya and Snowflake in the cloud using CICD and git.   ... View more

This session demonstrated using Azure DevOps and Git to automate the move from SAS® 9.4 DI Studio to SAS Viya Studio Flow while moving data to Snowflake using SAS code and the migrated SAS Viya Studio flows.   Presentation slides are attached to this post. ... View more

Hi, I just made a video to show how you can automate migration from SAS Enterprise Guide to SAS Viya.   It is based on a public repository here: https://github.com/larsatsas/auto-eg2viya so if you want to try this out yourself and have access to an environment with Enterprise Guide as well as a SAS Viya environment, you may want to try this out yourself.   As many now are considering migrating Enterprise Guide projects to SAS Viya - I wanted to see how we could automate this quite tedious process that requires tight attention to details.   As humans we are not always good at this – we get bored and sloppy, and mistakes may happen.   So why not automate this and have a computer do this for us? With CICD and devops being a thing, doing migration through CICD seems to be a sensible way of attacking this. Also – many of the tools for migration are well suited for automation – most of them have command-line options.   And as a CICD addict and true believer in all things agile, going cicd on migration makes a ton of sense. The repository have content and guidance on: Understanding and setting up the environments to prepare for CICD How to use git to transfer content between SAS94 and SAS Viya Using github actions to automate importing and validation of sas content to SAS Viya Watch the video and let me know what you think - and do take a peak at the repo mentioned above.   ... View more