As organisations are migrating to SAS/Viya, many SAS/DataFlux users are wondering how much functionality is available now in SAS Studio Flows that can replicate the functionality in DM Studio Data Jobs.  And can it be done without resorting to Code Steps in the Flow?   The answer is a definite YES!  A great many DMStudio Data Jobs can now be replicated as no-code SAS Studio Flows.   Here's a screenshot of one of a portion of my favourite demo DMStudio Data Job that does a variety of Data Quality operations, Address Verification, Matching, Clustering, and Survivorship:   With SAS/Viya I can now replicate the functionality of this flow in a SAS Studio Flow with a combination of standard steps and Custom Steps. (refer to the screenshots farther down in this article.  I also have the sample data and .JSON file exports available so you can import this demo into your own environment if you like).   As of SAS/Viya release 2024.09, there are now standard Steps for SAS Studio Flows that fill the gaps that were previously there in Viya: under Data Quality Clean Data, including tabs for: Standardization Casing Identification Analysis Gender Analysis Patten Analysis Match Codes Parse Data under Integrate: Load Table Merge Table under Enrichment: Geocoding Data Verify with Loqate: Address Verification Geocoding (using Loqate) Phone Verification (with enhanced info compared to DMStudio) Email Verification (with enhanced info compared to DMStudio) Under Transform Data: Branch Rows Calculate Columns Filter Rows Insert Rows Manage Columns Mask Data Query Rank Data Remove Duplicates Sort Split Columns Stack Columns Transpose Daa Union Rows Under Vizualize Data: sixteen types of charts, plots, and maps   There are also many SAS Studio Custom Steps published in the Github repository (https://github.com/sassoftware/sas-studio-custom-steps). Some Custom Steps that DMStudio users will find useful include: Anonymize and Mask Data Append Table CAS – Generate Unique ID Append Table DQ - Change Case DQ - Cluster Analysis DQ – Identify DQ – Create QKB Reference Table DQ – Match Code DQ - Parsing DQ – Surviving Record Export – Parquet Extract Text Features FTP Directory Listing FTP Download Files GeoDistance with Rounding Great Expectations – Execute Rule Great Expectations – Generate Expectation Suite Great Expectations – Run Expectation Suite HTTP Request Import – ADLS File Reader Import – CSV with long column names Import – Data Ingestion Auto Pilot Import – Extract Table from PDF Import – HTML Table Lookup Loop Deployed Object Loop Flow NLP – Extract Entities NLP – Identify Language OCR – Document Analysis Send SMTP Email Send Teams Message Surrogate Key Generator Translate Text Even the EEL code in DataFlux Expression nodes can be moved into SAS PROC EEL code, as described here:  https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-Using-PROC-EEL-in-SAS-Studio/ta-p/879625   This means that a great many DMStudio Data Jobs can now be replicated as no-code SAS Studio Flows.   Below are a couple of screenshots of my SAS Studio Flow, which shows a typical use case of Data Quality and Entity Resolution using Steps like: Branch Rows, Clean Data, Parse Data, DQ - Identify, Verify with Loqate, Calculate Columns, Manage Columns, Union Rows, Match Codes, DQ - Clustering, and DQ - Surviving Record.   It uses sample fictional data from the USA and Canada.   The first screenshot (below) shows the flow at the first node (incoming ACCOUNTS table), previewing the incoming data with obvious data quality issues and duplicate contact information in the 22 rows of sample data.     The next screenshot (below) shows the flow Previewing the resulting data at the end, after survivorship (using a simple record rule - best Address Verification Code), reducing the 22 rows down to 7 surviving rows.     If you want to take a closer look at all the steps in this flow you can import this demo flow, demo data, and the Custom Steps used, into your own environment.  I would just need to email you three .JSON export files and the  ACCOUNTS.CSV file of sample 22 rows of data.  Please message me here if you would like to get these assets.   If you need guidance on how to import these assets into your environment please refer to this SAS Communities article:  Migrating SAS Studio Flows containing Custom Steps - SAS Support Communities   For details about many of the steps in the flow you can also check out this excellent article here on SAS Support Communities written by my colleague Patric Hamilton:  Data Brilliance Unleashed: SAS Data Quality against Databricks - Preci... - SAS Support Communities           ... View more

Question: Why does my SAS Studio Flow not find the Custom Steps when I move it to a new environment?   Flow does not find Custom Steps if you just download it and move it to a new environment.   Answer: You need to Export and Import via SAS packages both the Flow and the Custom Steps.   Many users think all you need to do to move a SAS Studio Flow containing Custom Steps to a new environment is to just download it and copy it into the new environment (assuming all of the Custom Steps are there in the new environment).  Unfortunately this is not the case since SAS Studio has some workings going on under the covers that points the Custom Steps in the Flow to the location in the environment where those Custom Steps are stored.   There is no standard location for Custom Steps, and it's not enough to just use the same folder name for the steps across environments.   In order to make the connections work between the Flows and the Custom Steps in the new environment, you need to first Export to a SAS package the Flows and the Custom Steps.   You can either use the CLI or the interactive Export and Import utilities to do this.  This post will show how to use the interactive utilities.  For more info on how to use the CLI for Content Migration check out this YouTube video: SAS Demo | SAS Viya Content Migration Using the sas-viya CLI Transfer Plugin    Exporting The interactive, point and click, Export utility requires Administrator rights.  You get to it from the SAS Environment Manager app, Manage Environment, and then click the Content icon on the left.         Then you just need to navigate in the SAS Content  until you get to the flow, or the folder of flows, that you want to Export.   In the popup you need to give a name for the package file that will be created. naming the export package You should also check the "Include dependencies" check box if you are also moving reports as well as flows in a folder.  Unfortunately "Include dependencies" only includes dependencies for reports and doesn't include dependencies like Custom Steps for flows.    So you will need to also Export SAS package file(s) for the folder(s) where the Custom Steps are located.   Your exported SAS package files should appear in your download folder on your local machine as .JSON files.   Importing You can now import those .JSON files into your target environment. The interactive, point and click, Import utility requires Administrator rights.  You get to it from the SAS Environment Manager app, Manage Environment,  then Import icon on the left, and point to the source JSON file to Import.  You can Import the Custom Steps file(s) or the Flow file(s) in any order but they need to be imported individually, one at a time.   In the Mapping Tab you can map  the Table names and locations that are used by a Flow.   The Preview Tab show the items that will be imported,   Then you just have to hit the "Import" button in the top-right of the window.  The items will be imported and the Results pane will show the status of the Import action.   Once you've done the Import of all the Export files that you created for your flow(s) and Custom Step(s), then you should find that your Flow finds the Custom Steps and works as expected.   Of course this assumes that you've successfully migrated your data first. ... View more

Watch this Ask the Expert session to explore the ways in which Viya turns your raw data resources into valuable assets.    Watch the webinar You will learn: How to connect and use cloud data sources. Simple ways to continue to use on-premises legacy data. New methods to take advantage of the Viya massively parallel processing capabilities.     The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.   Q&A What can Viya do with Generative AI and my data? The first thing I would say is that we can make models better with generative AI for data, aka. synthetic data. That's a type of generative AI where we can create some synthetic data that’s hard to find in the real world, and then we can use that with in our predictive analytics and modeling to make those models better. The second thing is we can enable large language models, the API call. Within the SAS Studio flows that I was talking about earlier, there's a custom step. It's available on the public GitHub. If you go look for it, that custom step will allow you to download. You can install that, and then in your environment you can call the API from that custom step, call that LLM. We don't have a lot of customers doing that today, but we'll find some more use cases and examples of how that adds value as they do that. Also, I should mention that based on where you are, there's legislation in place that says you must practice trustworthy AI governance and business processes. In Canada, we have what's called the AIDA see The Artificial Intelligence and Data Act (AIDA) – Companion document (canada.ca). I'm sure wherever you are in the world, there's similar legislation. So, we've taken that to heart, and we've decided we want to become a leader in trustworthy AI. We ran a seminar on that recently with Reggie Townsend, who's one of our leaders in the AI space. He's also an advisor to the president the United States on that topic.  See these links for more info: https://www.youtube.com/watch?v=UfntNaxOlX0 https://www.youtube.com/watch?v=EWGDevmF3Yk https://www.youtube.com/watch?v=MmzsCFQmanU https://www.sas.com/en_ca/events/2023/q1/responsible-trustworthy-ai.html https://www.sas.com/en_us/news/press-releases/2022/april/sas-data-ethics-head-named-to-national-ai-advisory-committee.html   Should I be worried about my data on the cloud? Usually, cloud security is stronger than what you have in place on premises, in a general statement. Also, cloud security can rapidly evolve to new threats and stay on top of those new threats very quickly. Maybe you're in an organization where you have top notch security, so maybe that's not the case. In the end, I would say security is strong. But you should know your organization's policies, be smart on what you put in the cloud, and what the access control security policies you put in place there are. And finally, there are some organizations that have top secret implementations, like if you're RCMP or FBI. There are implementations that aren't connected to even the Internet. That's a different conversation though. But what my answer is that you shouldn't be that worried about data on the cloud.   What should I do with everything I have done on SAS 9? I think you first want to start with just lift and shift to Viya, get it running, analyze what's taking the most time, then optimize where it's the most expensive to run and where you can make the biggest gains. I should say in that example we have with the big bank there, we only changed seven jobs and we got big performance improvements. But if we were to change the next seven jobs, we wouldn't have got it to make a performance improvement. There's diminishing returns. Just focus on the stuff that's costing you the most, improve that. After that you might want to reevaluate how much more time you want to spend on improving performance.   What is CAS? CAS is the web-based version of SAS and it's the cloud version of SAS. It is architected to run the cloud and takes advantage of the cloud architecture. The objective is to do as much in memory as possible to improve performance.   How do I find local SAS representatives? If you don’t know your local SAS representatives, start with the “Contact Us” page on the SAS  website: https://www.sas.com/en_us/contact.html   How do you suggest clients use SAS Viya? Cost-wise, it’s very expensive. If they are already SAS customers, then I'd say bigger, faster, stronger, do more with costing less money.   Everyone I know are using PySpark instead of SAS Viya. There are some recent third-party benchmarks showing SAS Viya greatly outperforms competitors, including the open source options such as PySpark. See https://futurumgroup.com/sas-viya/   I need some hands-on experience with SAS Viya. Talk to your local SAS representative and find out what they can do for you in terms of getting you a demo Viya environment or a test environment. I know for certain customers we have here we've spun up Viya environments for quick training and learning experiences. That's something we can do.   Most of my friends are telling me Viya is not faster when compared to Base SAS. Well, we saw in those examples from that large banking customer, it was sometimes 34% faster, sometimes 29 times faster depending. It all depends on what? What the data is, how big the data is, what the code is, whether that code can gain some efficiencies by running all on memory. Sometimes it's 5% faster, sometimes it's 29 times faster. It all depends.   What is the suggested automatic way of scanning existing macros to convert SAS 9 macros/programs to SAS Viya in the most efficient way? We have a whole set of tools for migrating code, and that's not the topic of today's webinar, but we have toolkits such as the SAS Content Assessment tool that can help you do that. (for an overview see the article at https://communities.sas.com/t5/SAS-Communities-Library/A-first-look-at-Migration-from-SAS-9-4-to-SAS-Viya/ta-p/728498) I encourage you to talk to your local SAS representative and find out more.  We can scan your existing code and see which of those pieces of code are migratable as-is and which need some attention to get them to migrate. But things have been improving so almost everything is easily migratable now and you want to use the tools that we have available to do that.   Does SAS Viya make Enterprise Guide obsolete? We just announced at SAS Innovate in Las Vegas we are supporting SAS Enterprise Guide in Viya now. You can have Enterprise Guide connecting to SAS 9, Viya 3.5, and Viya 4. In fact, it's great for Enterprise Guide users who don't want to move. This is great if you have a very large investment in EG and the amount of time it would take you to change all those EG jobs in the SAS Studio flows is just prohibitive. In the end, I suggest somehow migrating into SAS studio flows to really take advantage and get the most benefit out of SAS Viya. But I understand that's a big lift, a lot of work for a lot of people. So that's why we just announced we're continuing to support Enterprise Guide and supporting it on Viya.   I am working with data I do not want anybody to see, is this granted in Viya cloud? Yes, you can control in Viya who's allowed to see your data. There's a whole range of security that you can put on and the administrator will be able to help you with that.   Can SAS Viya query Azure Serverless On-demand pool to query parquet files on Azure Data Lake in place?   Yes, a SAS Viya (CAS) user can ADLS2 parquet data in both serverless and dedicated SQL pools.  Users would use Synapse as an intermediary to read the parquet file and then SAS reads Synapse.  See the article:  SAS Viya and Parquet files – additional features    Recommended Resources Preparing data with CAS compared to SAS 9 using Data Step. LIBREFs and CASLIBs in SAS Viya Building no-code custom steps Moving from SAS®9 to SAS® Viya® Move to Viya Board Please see additional resources in the attached slide deck.   Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.         ... View more