In my last article, I introduced some ideas around persistent storage in SAS Viya 4 / Kubernetes. However, I only briefly touched on the actual mechanisms you'd use to find where your CASLibs and libnames actually point outside of the SAS Viya 4 containers. I'm currently tracking down some CASLibs on a test SAS Viya 4 env with Kubectl. Let's take a look at the commands and results I use to see where the physical files actually sit. . . . Get the Relevant Namespace and Pod Name I'm flying blind here. The first thing I need to do is understand what my Kubernetes environment looks like so I can figure out exactly where to look for the storage information. The first command I'll issue is below. This will tell me every single pod in the environment. . kubectl get pods --all-namespaces   The output contains numerous pods. I narrow down what I'm looking for to the CAS pods until finally I locate the CAS Controller pod. .. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   This not only gives me the CAS Controller pod name, sas-cas-server-default-controller, but also the namespace, gelenv-stable.   Describe the Pod Now that I know the relevant pod and namespace names, I can use them to look at how the pod is structured.   kubectl describe pod sas-cas-server-default-controller -n gelenv-stable . The pod contains numerous containers but I want the CAS container and, in particular, its mounts... .     Inspect any Ephemeral Volumes There are two mounts that draw my interest, /cas/data and /gelcontent. Respectively they are bound to the identifiers, cas-default-data-volume and gelcontent-volume. I search for these lower in the output. The identifiers appear in the Volumes section.   ..     Inspecting the volumes section, I can see that gelcontent-volume is an ephemeral volume and maps to /data/gel/content on the 10.96.13.223 machine.   Therefore, any CASLib or libname that uses the /gelcontent path within the SAS Viya 4 CAS environment maps to the /data/gel/content location on the 10.96.13.223 host. I can drop files there from external applications to use in SAS Viya. I can grab files from there that SAS Viya creates for use in other applications. .   Inspect any Persistent Volumes The cas-default-data-volume identifier points to a persistent volume claim (PVC), named cas-default-data. .     With the claim name, I execute the following command. .   kubectl get pvc cas-default-data -n gelenv-stable . The command tells me the Kubernetes Persistent Volume (PV) that the PVC claimed. ..   Now, I need to issue one more command to see where the PV maps.   kubectl get pv pvc-7c5f34af-1c64-4332-b8d9-bd03c113048e -o json . The mapping information appears under the "spec" element. ..   From this, I can see that the /cas/data path inside SAS Viya environment maps to the /srv/nfs/kubedata/gelenv-stable-cas-default-data-pvc-7c5f34af-1c64-4332-b8d9-bd03c113048e path on the intnode01 machine. So again, external applications can drop data here for SAS Viya to use in the various CASLibs and libnames that use the /cas/data path, and external applications can grab SAS Viya outputs from here.   . Applying This to Your Env Hopefully this exercise and these command examples will help you to analyze your SAS Viya 4 Kubernetes environment to see where your persistent data lives. .   Find more articles from SAS Global Enablement and Learning here. ... View more

You might have heard that the new SAS Viya (2020.1 and later) will deploy solely on containers orchestrated by Kubernetes. While Kubernetes introduces many benefits, it also introduces many changes, especially regarding storage. In particular, the containers that Kubernetes orchestrates are virtual with paths that map to virtual locations. So what happens when you have a source data admin wanting to pass data into containerized Viya? How do you answer the question, "Where should I drop the data?" All the paths you see inside the Viya env are virtual paths. They don't exist outside the cluster. How do you find out where they map on the physical hardware? . Kubernetes Pod to Physical Path Mapping Mechanisms While all container technologies offer storage mapping mechanisms (e.g. Docker bind mounts and Docker volumes), these should be avoided because they are tied to individual containers and live only as long as the container does. Kubernetes offers two main (there is always another way to do something right?) mechanisms for mapping internal paths to external storage resources -- Utilizing Persistent Volumes with Persistent Volume claims as well as defining volumes directly in a pod. We'll cover both mechanisms here. . Using Persistent Volumes with Persistent Volume Claims Kubernetes Persistent Volumes map the storage paths inside the containers to the storage resources available to the physical hosts. As the image below shows, this mapping is done via three levels of abstraction:   A persistent volume (PV) makes a physical storage location, NFS, AWS EBS, Azure File, etc., available to the cluster. . A persistent volume claim (PVC) requests access to the PV inside a cluster namespace . A volumeMount maps the PVC to a location within the Kubernetes pod. PV -> PVC -> volumeMount   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Defining Persistent Volumes Persistent volumes are defined in Kubernetes YAML files and are instantiated using the kubectl command line tool. An example of an NFS PV is shown below. Note that the file basically maps the name, nfs-pv to an nfs mount on the server, 192.168.1.40, at the path, /opt/k8s-pods/data. The claimRef reserves the PV for a "claim" that we'll make in the next step.   Sample PV Definition     Defining Persistent Volume Claims Persistent volume claims are also defined with YAML files. An example of a claim on the above PV is below. Note that the claimRef attribute from above reserves the PV for the PVC below. Also note that the volumeName attribute below explicitly claims the above PV. It these attributes are not included, a PVC will claim any PV that meets its specifications.    PVC Claiming a PV   Defining Volume Mounts The final step mounts the PVC to a local path within the pod. Again, an example YAML file is below. Note that the PVC we created above, nfs-pvc, is mounted to the /usr/data/ path within the pod.    volumeMount   There is, of course, much more to Kubernetes PVs and PVCs. For an in-depth understanding of Kubernetes persistent volumes, see this video. . Defining Volumes Resources within a Pod The other common alternative is to simply define an external storage resource as a volume directly within a pod and use a volumeMount to link the internal path to the external resource. While this may seem more convenient, the storage resource volume does not persist beyond the life of the pod and the resource is not available outside of the pod.   Direct Volume Mount   Making the Viya Connection Once you understand how kubernetes volumes work, you can explore your containerized Viya env with the kubectl command line tool (get pvc, get pv, describe pod, describe all and other commands) or the Kubernetes web ui to see where the internal paths of your caslibs and/or libnames point externally. Then you can tell the data source admin, "drop the data in there." . For More For more, on how to explore and customize your SAS Viya Kubernetes environment, see Gerry's article here. For more on Kubernetes volumes, persistent volumes, persistent volume claims, pods, and more, see the API documentation. ... View more

There are a few new features coming to Intelligent Decisioning in the new version of SAS Viya (2020.1 and beyond). An interesting one is the ability to compare different scenarios. Scenario analysis has wide applications in fields from finance to politics to energy and beyond. Let's take an early look at this new functionality and see what it has to offer. . . . . . How to Use it   Once your Decision or Rule Set is configured , go to the Scoring tab and select the new Scenarios sub-tab. Setup your scenarios just like tests  but specify only input values, no input tables. Once your scenarios are complete and run, compare them as shown below with the Compare -> Results pick.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Compare your Results   The Compare Results screen is shown below. It displays the scenario inputs and outputs side-by-side for easy comparison.     .. Going Further   For those looking to go beyond the OOTB comparison visualization, the output scenario data sets are easily located in the user's CASUSER caslib for use in Visual Analytics or elsewhere.       ... View more

Cloud object storage (S3, ADLS, GCS, etc.) keeps getting more and more relevant for our customers. They keep putting more and more data into it. Consequently, they want more and more SAS (native/direct please!) connectivity to it.   Let's dive into cloud object storage. First let's help understand object storage, look at its advantages, and then finally see how SAS can utilize and/or integrate with it. Understanding Cloud Object Storage . As a "Modern FTP Server" . Avoiding formal definitions and lots of technical details, the easiest way to understand cloud object storage is to think of it as a "modern FTP server." Object storage isn't actually an FTP server (although you can set that up) but functionally it plays the same role as FTP servers did in the past, and often still do today. You want to allow outside data to come into your network? You set up an FTP server and allow people to push or pull data into it. Object storage plays this same role for cloud platforms, although via different mechanisms. It allows outside data to come into your cloud network.   Say you wanted to bring data into your cloud database instance. You could configure a local database client to connect with your cloud db, open a bunch of ports (sounds risky!), and pass the data up that way. However, it's a whole lot easier just to push the data to a bucket in your cloud provider's object storage and access it from there.   Comparing Object Storage to a Traditional FTP Server Select the image to see a larger version. Mobile users: To view the image, select the "Full" version at the bottom of the page. . . REST-based put, get, mdir, mkdir, etc. (API) Also similarly to FTP, (at least for S3) we even use PUT and GET commands to load and unload object storage data. Regardless of exact command syntax, object storage platforms generally utilize simple, open REST APIs for data maintenance. This is what makes it "modern." Generally the entire object storage implementation is REST (HTTP) based, right down to each object (table, file, etc.) having its own URL. But it's Permanent Functionally, this is where the similarity to those traditional FTP servers ends. Because, unlike those old FTP servers which were generally meant for staging only, object storage is being used more and more to store data permanently. It's not as close to the processing engines as other storage options, like local disk or network attached storage. So, generally, performance against it isn't as good but object storage has several other advantages discussed later. Object Storage for Data Lakes So what are our customers storing out there and why are they choosing object storage? Well, remember the old Hadoop Data Lake value statement? It went something like   "Put your old data into Hadoop. It's cheap and Hadoop's big data capabilities will help you process it (if and) when you need it." This is the basic idea of a data lake and more organizations are using object storage for this purpose. Unlike a data mart or a data warehouse that is organized to answer questions, a data lake is simply a collection of raw tables, files, and whatever else you might want to keep for future analyses. You don't spend time modeling the data for ease of use but let data scientists (a.k.a. programmers) figure it out when the need arises.   Now, it looks like many organizations are replacing Hadoop with object storage for data lakes. Why? Well, as the linked article states, there are several reasons but four really stand out: Cost: Object Storage is seriously cheap, with different providers claiming 5+ times cost savings over local HDFS, and, with tiered pricing it can get even cheaper. Elasticity: Want to double your data? Just put it out there. There's no need to requisition and configure hardware. There's no need to add HDFS nodes. Integration: More and more applications are integrating directly with object storage. Want to use Spark to process your data? There's no need to put it in HDFS, just hook up your cloud Hadoop instance to the provider's object storage. Want to hook up SAS Viya to S3? No problem. Minimal Maintenance: All of the above comes at the push of a few buttons or some super simple scripting. There's no hardware or software to manage. There's no Hadoop expertise required. SAS and Object Storage If object storage is good for data lakes then SAS is great for object storage. After all SAS has been making sense of raw data for over 40 years.   Recently, SAS has been adding connectivity and integration with object storage to both its SAS and CAS data analytics platforms. Below is collateral that highlights SAS' current object storage capabilities along with some clever ways people have found to connect SAS to object storage. AWS Simple Storage Service (S3) Accessing S3 with the S3 CASlib PROC S3 to Access AWS S3 CAS and S3 Parquet Files Reading S3 (Avro, CSV, JSON, XML, Parquet, ORC) files to CAS and SAS via AWS Athena Writing CAS and SAS Data to S3 (Parquet, Avro, …) files via AWS EMR Reading S3 (Parquet, Avro, …) files to CAS and SAS via AWS EMR Accessing AWS S3 as NFS from CAS and SAS with Objective FS Accessing AWS S3 as NFS from CAS and SAS with AWS Storage Gateway Accessing AWS S3 as NFS from CAS and SAS with S3FS Azure Data Lake Storage (ADLS) Path CASlib and Libname to ADLS via BlobFuse SAS FILENAME statement to ADLS2 ORC Libname to ADLS2 ADLS2 CASLib Google Cloud Storage (GCS) Four Ways SAS can Access GCS SAS Accessing GCS with GCS FUSE Using REST (SAS PROC HTTP) to Access GCS ... View more

Some people need time to warm-up to new things. As a member of the Global Enablement and Learning team, I see this every day. While "late adopters" are generally quite willing to learn the latest twist or tweak on something they already know well, they balk at the truly novel and complete paradigm shifts. Such it is with CAS. Many SAS users are taking a "long" approach.   One group that might be reluctant to adopt CAS is surprisingly SAS' most ardent supporters. These users have invested lots of time into their SAS knowledge. With so much invested, they can be reluctant to work in another construct where they aren't so confident or skilled.   So, for you SAS true believers, let's look at how to get the best performance in CAS and compare it to SAS (standard disclaimers about these tests being done on non-optimal, virtual hardware...) and, hopefully, once you know how to get vastly better performance from CAS than single engine (but still awesome!) base SAS, you'll be more comfortable giving it a try. 1. Group BY Aggregation -- Low Cardinality When choosing data processing techniques in base SAS, you usually only have DATA Step or PROCs. You have these in CAS as well, but, when looking for optimum performance, the best place to start is with the CAS Actions, and aggregation is no exception.   Thanks to Nicolas Robert, we already know which CAS action aggregates the fastest (at least in his scenario), simple.summary. So, let's compare its performance on some big data against base SAS on the same. Test Parameters Test Parameter Value Input Rows 160 million Distinct BY Groups (Cardinality) 8 CAS Code -- Simple.Summary proc cas ; simple.summary result=r status=s / inputs={"revenue","expenses"}, subSet={"SUM"}, table={ name="mega_corp" caslib="visual" groupBy={"facilityType","productline"} }, casout={name="summaryMC", replace=True, replication=0} ; quit ; Base SAS Code -- PROC MEANS proc means data=mega_corp noprint; var revenue expenses; class facilityType productline; output out=summaryMC sum(revenue)=sumRevenue sum(expenses)=sumExpenses; run; Results Engine Method Real Time CAS Simple.Summary 7.39 SAS PROC MEANS 2:32.44 2. Group BY Aggregation -- High Cardinality There has been some talk that CAS does not perform well with high cardinality operations. Let's take a look by increasing the number of BY-Groups. We'll use the same code as above but replace the GroupBy and CLASS variables with productID, date, and unit. This gives us approximately 88,000 distinct groups. Test Parameters Test Parameter Value Input Rows 160 million Distinct BY Groups (Cardinality) 88,000 Results Engine Method Real Time CAS Simple.Summary 18.22 SAS PROC MEANS 2:31.65 3. De-Duplication As with aggregation, picking the right technique is key and, thankfully again, Nicolas Robert has already shown us which method to use for de-duplication, the simple.GroupBy CAS action.   So, let's compare simple.GroupBy with PROC SORT. Test Parameters Test Parameter Value Input Rows 160 million Unique Keys 88,000 CAS Code -- Simple.GroupBy proc cas; simple.groupBy result=r status=rc / inputs={"productID", "date", "unit"} table={caslib="casuser",name="mega_corp"} casOut={name="dedupMC",replace=true,replication=0} ; run ; quit ; Base SAS Code -- PROC SORT proc sort data=mega_corp nodupkey out=dedupMC; by productID date unit; run; Results Engine Method Real Time CAS Simple.GroupBy 12.57 SAS PROC SORT 3:09.29 Discussion So, there you have it. CAS is fast. It is plowing through some decent sized data here on a few (5), relatively small (4-way) virtual servers in seconds.   If you want performance like this however, you need to know which techniques to use. Luckily some of the hard work has already been done for you. In particular check out these posts: CAS answers to 4 common data manipulation tasks – Part 1 – APPEND CAS answers to 4 common data manipulation tasks – Part 2 – SORT CAS answers to 4 common data manipulation tasks – Part 3 – DE-DUPLICATE CAS answers to 4 common data manipulation tasks – Part 4 – AGGREGATE You'll also need to know more about CAS Actions. In particular, you'll need to know how to enhance them so they do exactly what you want. This post should help with that: CAS Action Computed Columns ... View more

In my last post, I compared CASL to Base SAS and SAS/MACRO, and provided examples of their similarities. The one big difference, however, was the mechanism used for referencing objects (tables, columns, librefs, directory paths, etc.) in generated code. With macro, we talked about using macro variable resolution. With CASL, we used CAS Action Result variables. CAS Action Result Option ≈ SAS PROC Output Statement CAS Action result variables are set using the CAS Action Result option.   CAS Action Result Option Example Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The function of the CAS Action result option is much the same as the SAS Procedure Output statement. It directs the results of the CAS Action to a code-consumable object. In the case of the procedure output statement, the results go to a SAS table. In the case of the result option, the results go to a CASL Variable. In the example, the results are sent to the tabInf CASL variable. CASL Variables What are CASL variables? If you're a SAS programmer, you can think of them like MACRO variables without all of the resolution timing concerns. They also can take on numeric and other data types.   CASL has the usual SAS character and numeric data types but also allows for arrays, boolean, dictionaries, and tables. We'll focus on these last two here as CAS Action result variables are generally either dictionaries, tables, or some combination of the two. CASL Describe Statement So how do we know what's in our result variable? Is it a dictionary? A table? A dictionary with a table inside? If there's a table inside of a dictionary, what's the name of the table? Use the Describe statement.   CASL Describe Statement   Looking at the TableInfo example above, we see that the result variable is a dictionary that contains one entry, a table named "TableInfo." CASL Dictionaries Before we consider the table, let's look at the dictionary. A CASL dictionary is an array of key-value pairs. If you've coded some CASL, you'll probably recognize them. The tbl dictionary below contains two key-value pairs that define a CAS table.   tbl={name="iris", caslib="casuser"}   Our CASL dictionary, tabInf, contains only one key-value pair and looks something like this:   tabInf={TableInfo="<The table containing the information about the CAS tables in the DM Caslib>"}     To reference the members of CASL dictionaries, you use simple two-level naming, e.g. "tabInf.TableInfo." CASL Table Variables So what can we do with CASL Table variables? We can do quite a bit actually. We can subset them, add computed columns to them, and select values from them. We can also return their row counts, column counts, table names, and attributes. Typically we'll be accessing their values by row and column.   Let's access some values from our TblInf.TableInfo result table.   Printing Values from a CAS Action Result Table   The first print statement accesses the name of the first table in the DM CASlib. The second prints the encoding (the 6th column of the table) of the second table. The Do loop prints the names of all the tables in the CASlib.   For an example of spinning through a CAS Action Result table to load files into CAS, see my previous post. All of my posts on CASL are listed below: CASL: It’s like Base SAS with the MACRO Built-In CAS is Fast CAS Action Computed Columns ... View more

One of the great features of SAS Viya is CASL ("CAS Language"). CASL is certainly not the only way to instruct CAS. CAS also supports DATA Step, SAS Procedures, and various open APIs. However, CASL is certainly a powerful option.   If I had to describe CASL to a SAS programmer, I'd say that it's like base SAS with the MACRO built-in. How CASL is like Base SAS Base SAS programs combine DATA Step with SAS Procedures. A simple one might look like this:   DATA public.aggregatedTable ; SET public.detailedTable ; by groupingVar; retain GroupSalesAmt; if first.groupingVar then GroupSalesAmt = 0; GroupSalesAmt = GroupSalesAmt + SaleAmt; if last.groupingVar then output; run; Proc Print data=public.aggregatedTable; run;   The equivalent code would look like this in CASL:   Proc CAS; datastep.runCode submit / code=" DATA public.aggregatedTable (replace=yes) ; SET public.detailedTable ; by groupingVar; retain GroupSalesAmt; if first.groupingVar then GroupSalesAmt = 0; GroupSalesAmt = GroupSalesAmt + SaleAmt; if last.groupingVar then output; run;"; table.fetch / to=5000 table={caslib="public" name="aggregatedTable"}; quit;   While the syntax is different, both approaches have two steps -- a DATA Step program followed by a printing mechanism. CASL is limited to running CAS Actions (no PROCS) but, since CAS PROCs are based on CAS actions, we should be able to do anything with CAS actions that we can do with CAS enabled PROCs. How CASL has the "MACRO Built-In" To see the "built-in MACRO," let's add some conditional logic to our simple example. Let's choose between printing the detailed data and the summarized data. Just change the assignment of aggreg8 between N and Y for each option respectively.   Now our Base SAS example looks like this. It uses SAS/MACRO for the conditional logic.   %macro conLogic; %let aggreg8=N; %if &aggreg8=Y %then %do; DATA public.aggregatedTable ; SET public.detailedTable ; by groupingVar; retain GroupSalesAmt; if first.groupingVar then GroupSalesAmt = 0; GroupSalesAmt = GroupSalesAmt + SaleAmt; if last.groupingVar then output; run; Proc Print data=public.aggregatedTable; run; %end; %else %do; Proc Print data=public.detailedTable; run; %end; %mend; %conLogic;   The equivalent code would look like this in CASL.   Proc CAS; aggreg8='N'; if aggreg8='Y' then do; datastep.runCode submit / code=" DATA public.aggregatedTable (replace=yes) ; SET public.detailedTable ; by groupingVar; retain GroupSalesAmt; if first.groupingVar then GroupSalesAmt = 0; GroupSalesAmt = GroupSalesAmt + SaleAmt; if last.groupingVar then output; run;"; table.fetch / to=5000 table={caslib="public" name="aggregatedTable"}; end; else do; table.fetch / to=5000 table={caslib="public" name="detailedTable"}; end; quit;   Again, both approaches are more or less equivalent but CASL requires no MACRO coding to accommodate the conditional processing. CASL variables (aggreg8 in the example) take the place of macro variables. CASL expressions (aggreg8='Y') take the place of MACRO expressions. CASL statements (IF, THEN, ELSE, DO) take the place of MACRO statements. Data-Driven Code Generation / Processing What about something more complex, like generating code from data? This can get fairly complicated with MACRO, requiring multiple levels of macro resolution (lots of ampersands strung together, e.g. "DATA p&&dsn&q..inputTab;" (see this paper for more info). A simple, single level (only one ampersand required) example is found in this article.   CASL can also do iterative data-driven processing using DO loops. However, you don't need MACRO resolution to reference the right table name, field name, library name, file name, directory name or whatever. CASL's dictionary and result table data types, which can be returned from previously-run CAS actions, can be used instead. (CASL is like Base SAS with the SCL Built-In?)   For really complex generated code, you can build the code in pieces and store/append it in/to CASL variables and execute it with the runCasl CAS action. For more info on the runcasl action, see this article by Deva Kumar.   Here is a simple example of iterating through and loading the SASHDAT files in a CASLib's data source location. The fileinfo action is used to store the data source file list in the filesToLoad CASL variable. It is a CASL dictionary field that contains a table in its sole FileInfo member. We then spin through the rows of the FileInfo table returning the file name to use in the loadTable action.   proc cas; table.fileinfo result=filesToLoad/ caslib="casuser"; do i = 1 to dim(filesToLoad.FileInfo); fname = filesToLoad.FileInfo[i,'Name']; table.loadtable / caslib="casuser" path=fname; end; run; quit;   Speak ... View more

CAS is massively parallel, spreading its processing across multiple threads on multiple machines but this affects the way you need to write your CAS DATA Step code. Let's look at how we might choose to create a unique row identifier in CAS versus SAS. . SAS DATA Step Creating a unique ID in SAS DATA Step is quite simple. We just use the _n_ automatic variable as shown below: . DATA tableWithUniqueID; SET tableWithOutUniqueID; uniqueID = _n_; run; . CAS DATA Step Creating a unique ID in CAS DATA Step is more complicated. Each thread maintains its own _n_. So if we just use _n_, we'll get duplicate IDs. Each thread will produce an uniqueID field value of 1. Each thread will produce an uniqueID field value of 2. And so on.... When the thread output is combined, we'll have a bunch of records with a uniqueID of 1 and a bunch with a uniqueID of 2.... This is not useful. To produce a truly unique ID, you need to augment _n_ with something else. CAS has its own set of automatic variables for such purposes. The _threadID_ automatic variable can help us get our unique ID as shown below: . DATA tableWithUniqueID; SET tableWithOutUniqueID; uniqueID = put(_threadid_,8.) || '_' || Put(_n_,8.); run; While there are surely other ways of doing it, concatenating _threadID_ with the _n_ ensures uniqueness because the _threadID_ uniquely identifies a single thread and _n_ uniquely identifies a single row output by that thread.   For more information on threads and CAS DATA Step, see this blog post. Speak Speak Speak Speak Speak ... View more

In my last post, I looked at extracting the generated DS2 code from SAS Intelligent Decisioning (SID). In this post, let's look at including that extracted code in SAS Event Stream Processing (ESP) projects. Our goals here are to: Better understand the differences between the "SID MAS" and the "ESP MAS". Better understand the SID to ESP DS2 conversion process for debugging and customization purposes. Better understand ESP MAS Maps. Sample Decision To aid us in understanding the whole process, we'll use a sample Decision as an example. The Decision supports auto auction bidding decisions and looks like this:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The decision contains 2 rule sets, a predictive model, a custom DS2 code node, and some branching logic. The Extracted Code We looked at extracting SID code in my last post. Using the methods described there and the following parameters, codeTarget=MICROANALYTICS, lookupMode=inline, we get the following code for our decision:   /* Rule Set autoAuction version 1.0 generated on Tue Apr 07 09:41:24 EDT 2020 */ package b_I2SJLZ3T2JBN7HYVMQV32RACEY / inline; dcl package logger _logger('App.SASDCM'); /* ---- RULE FIRED FLAG COUNT = 0 ---- */ method setruleFiredBasePosition(integer newBasePosition); end; method setDecisionNodeID(varchar newDecisonNodeID); end; method enableRuleFiredRecording(); end; method disableRuleFiredRecording(); end; method initRuleFireHashes(); end; method getRuleFiredFlagCount() returns integer; return 0; end; method recordRuleFired( in_out char ruleFiredFlags, in_out integer rulesFiredForRecordCount, char(36) this_rule_id); end; method execute( in_out double "Bid" ,in_out double "BlueBookPrice" ,in_out double "callOffice" ,in_out double "CurrentBid" ,in_out varchar "Make" ,in_out double "Miles" ,in_out varchar "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out varchar "state" ,in_out varchar "VIN" ,in_out double "Year"); "Bid" = 0 ; "callOffice" = 0 ; if (CurrentBid < BlueBookPrice) then do; if ("Miles" < 50000.0) then do; "Bid" = 1 ; end; end; "end_546d28a3-341e-462f-a06b-0d4083d9e2d5": if ("state" = 'PA') then do; "callOffice" = 1 ; end; "end_5b86f4d4-2055-4d96-995f-72f03bfba10c": rule_end: /* label for rule RETURN action */ end; endpackage; /* Custom Object californiaOverride retrieved on Tue Apr 07 09:41:29 EDT 2020 */ package c_EBWC2GAB6RB2BCVZ5DTMUWWRIU /inline; method execute(in_out double Bid, in_out varchar(100) bidAction, in_out char(2) state); if state = 'CA' then do; Bid = 1; bidAction = 'Buy anything from California!'; end; end; endpackage; /* Model Linear Regression (Pipeline 1) retrieved on Tue Apr 07 09:41:29 EDT 2020 */ ds2_options sas tkgmac scond=none; package m_3UWKJUHM5JBO5DSZSH34QQREHU/overwrite=yes; #_local _J_; #_local _I_; #_local _TEMP_; #_local _LINP_; #_local _BADVAL_; dcl double EM_PREDICTION; method score(char(17) "VIN", in_out double "EM_PREDICTION", in_out double "P_BlueBookPrice"); dcl char(17) _VIN_; dcl double _BETA_20892090_0_[15]; dcl double _XROW_20892090_0_[15]; _BETA_20892090_0_:=(8000, 1000, 0, 0, 4000, 0, 27000, 0, 0, 0, 20000, 4000, 37000, 0, 0); _BADVAL_ = 0.0; _LINP_ = 0.0; _TEMP_ = 0.0; _I_ = 0.0; _J_ = 0.0; _VIN_ = LEFT(TRIM(put(VIN, $17.))); do _I_ = 1.0 to 15.0; _XROW_20892090_0_[_I_] = 0.0; end; _XROW_20892090_0_[1.0] = 1.0; _TEMP_ = 1.0; select (_VIN_); when ('12345678901234560') _XROW_20892090_0_[2.0] = _TEMP_; when ('12345678901234561') _XROW_20892090_0_[3.0] = _TEMP_; when ('12345678901234562') _XROW_20892090_0_[4.0] = _TEMP_; when ('12345678901234563') _XROW_20892090_0_[5.0] = _TEMP_; when ('12345678901234564') _XROW_20892090_0_[6.0] = _TEMP_; when ('12345678901234565') _XROW_20892090_0_[7.0] = _TEMP_; when ('12345678901234566') _XROW_20892090_0_[8.0] = _TEMP_; when ('12345678901234567') _XROW_20892090_0_[9.0] = _TEMP_; when ('12345678901234568') _XROW_20892090_0_[10.0] = _TEMP_; when ('12345678901234569') _XROW_20892090_0_[11.0] = _TEMP_; when ('12345678901234571') _XROW_20892090_0_[12.0] = _TEMP_; when ('12345678901234572') _XROW_20892090_0_[13.0] = _TEMP_; when ('12345678901234573') _XROW_20892090_0_[14.0] = _TEMP_; when ('12345678901234574') _XROW_20892090_0_[15.0] = _TEMP_; otherwise do ; _BADVAL_ = 1.0; goto SKIP_20892090_0; end; end; do _I_ = 1.0 to 15.0; _LINP_ + _XROW_20892090_0_[_I_] * _BETA_20892090_0_[_I_]; end; SKIP_20892090_0: if (_BADVAL_ = 0.0) & ^MISSING(_LINP_) then do ; P_BLUEBOOKPRICE = _LINP_; end; else do ; _LINP_ = .; P_BLUEBOOKPRICE = .; end; if "P_BLUEBOOKPRICE" = . then "P_BLUEBOOKPRICE" = 22142.857143; EM_PREDICTION = "P_BLUEBOOKPRICE"; _return: ; end; endpackage; /* Rule Set autoAuctionNorthernStates version 1.0 generated on Tue Apr 07 09:41:24 EDT 2020 */ package b_4FNY7JHNGNDU7OT7FNJPAP2F6A / inline; dcl package logger _logger('App.SASDCM'); dcl varchar(512) lookup_value; dcl varchar(256) lookup_key; dcl package hash lookup_0; /* lookup for $LPUEPB73F7BGHBEGN4YJK6AMEBQ_K */ dcl package hash lookup_1; /* lookup for $LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K */ /* for inline lookups we initialize hashes in the package constructor */ method b_4FNY7JHNGNDU7OT7FNJPAP2F6A(); /* lookup: $LPUEPB73F7BGHBEGN4YJK6AMEBQ_K */ lookup_0 = _new_ [this] hash(); lookup_0.keys([lookup_key]); lookup_0.data([lookup_value]); lookup_0.defineDone(); lookup_0.clear(); lookup_key = 'Buick'; lookup_value = 'Buick'; lookup_0.ref(); lookup_key = 'Scion'; lookup_value = 'Scion'; lookup_0.ref(); /* lookup: $LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K */ lookup_1 = _new_ [this] hash(); lookup_1.keys([lookup_key]); lookup_1.data([lookup_value]); lookup_1.defineDone(); lookup_1.clear(); lookup_key = '0'; lookup_value = 'Do NOT bid on this car!'; lookup_1.ref(); lookup_key = '1'; lookup_value = 'Bid on the car!'; lookup_1.ref(); end; method getLookupValue() returns varchar(512); return lookup_value; end; /* $LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K is the format name for Lookup bidCommands. */ /* $LPUEPB73F7BGHBEGN4YJK6AMEBQ_K is the format name for Lookup unwantedMakes. */ method doLookupFind(varchar(256) lookupKey, varchar(100) lookupName) returns int; dcl integer rc; rc = 0; if ( missing(lookupKey) ) then do; lookup_value = ''; return 1; end; lookup_key = strip(lookupkey); select(lookupName); when('$LPUEPB73F7BGHBEGN4YJK6AMEBQ_K.') rc = lookup_0.find(); when('$LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K.') rc = lookup_1.find(); otherwise rc = 1; end; if rc then lookup_value = ''; return rc; end; /* ---- RULE FIRED FLAG COUNT = 0 ---- */ method setruleFiredBasePosition(integer newBasePosition); end; method setDecisionNodeID(varchar newDecisonNodeID); end; method enableRuleFiredRecording(); end; method disableRuleFiredRecording(); end; method initRuleFireHashes(); end; method getRuleFiredFlagCount() returns integer; return 0; end; method recordRuleFired( in_out char ruleFiredFlags, in_out integer rulesFiredForRecordCount, char(36) this_rule_id); end; method execute( in_out double "Bid" ,in_out varchar "bidAction" ,in_out double "BlueBookPrice" ,in_out double "callOffice" ,in_out double "CurrentBid" ,in_out varchar "Make" ,in_out double "Miles" ,in_out varchar "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out varchar "state" ,in_out varchar "VIN" ,in_out double "Year"); "Bid" = 0 ; if (0 = doLookupFind("Make",'$LPUEPB73F7BGHBEGN4YJK6AMEBQ_K.')) then do; if (CurrentBid < (0.9*BlueBookPrice)) then do; "Bid" = 1 ; end; end; "end_d71b5106-5b5d-4b44-8cee-a479673bd1c4": if(0 = doLookupFind(Bid,'$LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K.')) then;"bidAction" = getLookupValue();; rule_end: /* label for rule RETURN action */ end; method term(); if (null(lookup_0) = 0) then lookup_0.clear(); if (null(lookup_1) = 0) then lookup_1.clear(); end; endpackage; /* Decision autoAuction version 1.0 generated on Tue Apr 07 09:41:29 EDT 2020 */ package Decision_60e45dfc_ce58_4143_896 / inline; dcl package logger logr_perf('App.tk.SID.Perf'); dcl double timer_start; dcl double elapsed_time; dcl double total_timer_start; dcl double total_elapsed_time; dcl package b_I2SJLZ3T2JBN7HYVMQV32RACEY ruleset347(); dcl package c_EBWC2GAB6RB2BCVZ5DTMUWWRIU customobject348(); dcl package m_3UWKJUHM5JBO5DSZSH34QQREHU model345(); dcl package b_4FNY7JHNGNDU7OT7FNJPAP2F6A ruleset346(); method init(); end; method execute_internal( in_out double "BlueBookPrice" ,in_out double "CurrentBid" ,in_out varchar(20) "Make" ,in_out double "Miles" ,in_out varchar(20) "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out varchar(17) "VIN" ,in_out double "Year" ,in_out varchar(2) "state" ,in_out double "Bid" ,in_out double "P_BlueBookPrice" ,in_out varchar "bidAction" ,in_out double "callOffice" ,in_out integer "_filter_" ) ; dcl integer dg_filter_; dcl integer _masRC; total_timer_start = datetime(); timer_start = datetime(); model345.score("VIN", "BlueBookPrice", "P_BlueBookPrice"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: Linear Regression (Pipeline 1), Node ID: dd2ca4d0-ecea-42ee-8e59-91f7c842243d, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; /*model code*/; /*begin branch node*/ if ( "state" eq 'PA' ) then do; timer_start = datetime(); ruleset346.execute("Bid", "bidAction", "BlueBookPrice", "callOffice", "CurrentBid", "Make", "Miles", "Model", "OriginalInvoice", "OriginalMSRP", "state", "VIN", "Year"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: autoAuctionNorthernStates, Node ID: e15b8fa4-ed33-474f-ba7f-2b52f03f45f0, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; end; else do; timer_start = datetime(); ruleset347.execute("Bid", "BlueBookPrice", "callOffice", "CurrentBid", "Make", "Miles", "Model", "OriginalInvoice", "OriginalMSRP", "state", "VIN", "Year"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: autoAuction, Node ID: 46a495e7-73d2-42df-9f15-642bbd440226, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; end; /*end branch node*/ timer_start = datetime(); customobject348.execute("Bid", "bidAction", "state"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: californiaOverride, Node ID: 206c2d18-01f4-43a0-8ab9-e8e6ca5ad145, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; total_elapsed_time = datetime() - total_timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, TOTAL DURATION: ' .. total_elapsed_time); end; method execute( double "BlueBookPrice_" ,double "CurrentBid_" ,varchar(20) "Make_" ,double "Miles_" ,varchar(20) "Model_" ,double "OriginalInvoice_" ,double "OriginalMSRP_" ,varchar(17) "VIN_" ,double "Year_" ,varchar(2) "state_" ,in_out double "Bid" ,in_out double "BlueBookPrice" ,in_out double "CurrentBid" ,in_out varchar "Make" ,in_out double "Miles" ,in_out varchar "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out double "P_BlueBookPrice" ,in_out varchar "VIN" ,in_out double "Year" ,in_out varchar "bidAction" ,in_out double "callOffice" ,in_out varchar "state" ); dcl integer _filter_; "BlueBookPrice" = "BlueBookPrice_"; "CurrentBid" = "CurrentBid_"; "Make" = "Make_"; "Miles" = "Miles_"; "Model" = "Model_"; "OriginalInvoice" = "OriginalInvoice_"; "OriginalMSRP" = "OriginalMSRP_"; "VIN" = "VIN_"; "Year" = "Year_"; "state" = "state_"; execute_internal( "BlueBookPrice" ,"CurrentBid" ,"Make" ,"Miles" ,"Model" ,"OriginalInvoice" ,"OriginalMSRP" ,"VIN" ,"Year" ,"state" ,"Bid" ,"P_BlueBookPrice" ,"bidAction" ,"callOffice" ,"_filter_" ); end; endpackage;   The code is all DS2 and is organized into 5 packages -- 1 for each rule set (2), 1 for the custom DS2 code node, 1 for the predictive model, and 1 for the decision itself. The decision package calls the other packages and includes the branching logic. Also, one of the rule sets includes SID lookup tables.   Note that any generated SID lookup table code should utilize DS2 hash tables and not SAS formats. Setting the lookupMode=inline should take care of this but it's worth double checking. The other setting that applies here is the lookupStaticBinding parameter. For information on that, see this article on lookup tables. The generated lookup table code should include code that looks like this:       lookup_key = '0'; lookup_value = 'Do NOT bid on this car!'; lookup_1.ref(); lookup_key = '1'; lookup_value = 'Bid on the car!'; lookup_1.ref();     SID MAS vs ESP MAS But why do we need to export the code at all? Don't both SID and ESP use the MAS to run DS2 code? Yes, they do. However the MAS is implemented very differently in the two applications. In support of SID, the MAS is deployed as a web application with a REST API. In support of ESP, the MAS is integrated directly into the ESP server in the form of shared object libraries. So, while "SID MAS" needs DS2 packages pushed to it using HTTP POST commands, "ESP MAS" needs the DS2 packages presented to it using ESP's XML coding language. Getting our Extracted Code into ESP ESP executes DS2 code via ESP MAS modules and ESP Calculate windows. However, ESP only allows one DS2 package per MAS Module. So, how do we call our 5 DS2 package code? With 5 MAS modules of course. Put Each Package in its own File We break the code into 5 separate packages -- 1 for each rule set, 1 for the custom DS2 code node, 1 for the predictive model, and 1 for the overall decision. Each package is put into its own file. autoAuctionRuleSet.ds2: /* Rule Set autoAuction version 1.0 generated on Tue Apr 07 09:41:24 EDT 2020 */ package b_I2SJLZ3T2JBN7HYVMQV32RACEY / inline; dcl package logger _logger('App.SASDCM'); /* ---- RULE FIRED FLAG COUNT = 0 ---- */ method setruleFiredBasePosition(integer newBasePosition); end; method setDecisionNodeID(varchar newDecisonNodeID); end; method enableRuleFiredRecording(); end; method disableRuleFiredRecording(); end; method initRuleFireHashes(); end; method getRuleFiredFlagCount() returns integer; return 0; end; method recordRuleFired( in_out char ruleFiredFlags, in_out integer rulesFiredForRecordCount, char(36) this_rule_id); end; method execute( in_out double "Bid" ,in_out double "BlueBookPrice" ,in_out double "callOffice" ,in_out double "CurrentBid" ,in_out varchar "Make" ,in_out double "Miles" ,in_out varchar "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out varchar "state" ,in_out varchar "VIN" ,in_out double "Year"); "Bid" = 0 ; "callOffice" = 0 ; if (CurrentBid < BlueBookPrice) then do; if ("Miles" < 50000.0) then do; "Bid" = 1 ; end; end; "end_546d28a3-341e-462f-a06b-0d4083d9e2d5": if ("state" = 'PA') then do; "callOffice" = 1 ; end; "end_5b86f4d4-2055-4d96-995f-72f03bfba10c": rule_end: /* label for rule RETURN action */ end; endpackage;   autoAuctionNorthernStatesRuleSet.ds2: /* Rule Set autoAuctionNorthernStates version 1.0 generated on Tue Apr 07 09:41:24 EDT 2020 */ package b_4FNY7JHNGNDU7OT7FNJPAP2F6A / inline; dcl package logger _logger('App.SASDCM'); dcl varchar(512) lookup_value; dcl varchar(256) lookup_key; dcl package hash lookup_0; /* lookup for $LPUEPB73F7BGHBEGN4YJK6AMEBQ_K */ dcl package hash lookup_1; /* lookup for $LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K */ /* for inline lookups we initialize hashes in the package constructor */ method b_4FNY7JHNGNDU7OT7FNJPAP2F6A(); /* lookup: $LPUEPB73F7BGHBEGN4YJK6AMEBQ_K */ lookup_0 = _new_ [this] hash(); lookup_0.keys([lookup_key]); lookup_0.data([lookup_value]); lookup_0.defineDone(); lookup_0.clear(); lookup_key = 'Buick'; lookup_value = 'Buick'; lookup_0.ref(); lookup_key = 'Scion'; lookup_value = 'Scion'; lookup_0.ref(); /* lookup: $LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K */ lookup_1 = _new_ [this] hash(); lookup_1.keys([lookup_key]); lookup_1.data([lookup_value]); lookup_1.defineDone(); lookup_1.clear(); lookup_key = '0'; lookup_value = 'Do NOT bid on this car!'; lookup_1.ref(); lookup_key = '1'; lookup_value = 'Bid on the car!'; lookup_1.ref(); end; method getLookupValue() returns varchar(512); return lookup_value; end; /* $LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K is the format name for Lookup bidCommands. */ /* $LPUEPB73F7BGHBEGN4YJK6AMEBQ_K is the format name for Lookup unwantedMakes. */ method doLookupFind(varchar(256) lookupKey, varchar(100) lookupName) returns int; dcl integer rc; rc = 0; if ( missing(lookupKey) ) then do; lookup_value = ''; return 1; end; lookup_key = strip(lookupkey); select(lookupName); when('$LPUEPB73F7BGHBEGN4YJK6AMEBQ_K.') rc = lookup_0.find(); when('$LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K.') rc = lookup_1.find(); otherwise rc = 1; end; if rc then lookup_value = ''; return rc; end; /* ---- RULE FIRED FLAG COUNT = 0 ---- */ method setruleFiredBasePosition(integer newBasePosition); end; method setDecisionNodeID(varchar newDecisonNodeID); end; method enableRuleFiredRecording(); end; method disableRuleFiredRecording(); end; method initRuleFireHashes(); end; method getRuleFiredFlagCount() returns integer; return 0; end; method recordRuleFired( in_out char ruleFiredFlags, in_out integer rulesFiredForRecordCount, char(36) this_rule_id); end; method execute( in_out double "Bid" ,in_out varchar "bidAction" ,in_out double "BlueBookPrice" ,in_out double "callOffice" ,in_out double "CurrentBid" ,in_out varchar "Make" ,in_out double "Miles" ,in_out varchar "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out varchar "state" ,in_out varchar "VIN" ,in_out double "Year"); "Bid" = 0 ; if (0 = doLookupFind("Make",'$LPUEPB73F7BGHBEGN4YJK6AMEBQ_K.')) then do; if (CurrentBid < (0.9*BlueBookPrice)) then do; "Bid" = 1 ; end; end; "end_d71b5106-5b5d-4b44-8cee-a479673bd1c4": if(0 = doLookupFind(Bid,'$LYJBKPM4WRFA6ZP5JX2GVLYNDX4_K.')) then;"bidAction" = getLookupValue();; rule_end: /* label for rule RETURN action */ end; method term(); if (null(lookup_0) = 0) then lookup_0.clear(); if (null(lookup_1) = 0) then lookup_1.clear(); end; endpackage;   LinearRegression.ds2: /* Model Linear Regression (Pipeline 1) retrieved on Tue Apr 07 09:41:29 EDT 2020 */ ds2_options sas tkgmac scond=none; package m_3UWKJUHM5JBO5DSZSH34QQREHU/overwrite=yes; #_local _J_; #_local _I_; #_local _TEMP_; #_local _LINP_; #_local _BADVAL_; dcl double EM_PREDICTION; method score(char(17) "VIN", in_out double "EM_PREDICTION", in_out double "P_BlueBookPrice"); dcl char(17) _VIN_; dcl double _BETA_20892090_0_[15]; dcl double _XROW_20892090_0_[15]; _BETA_20892090_0_:=(8000, 1000, 0, 0, 4000, 0, 27000, 0, 0, 0, 20000, 4000, 37000, 0, 0); _BADVAL_ = 0.0; _LINP_ = 0.0; _TEMP_ = 0.0; _I_ = 0.0; _J_ = 0.0; _VIN_ = LEFT(TRIM(put(VIN, $17.))); do _I_ = 1.0 to 15.0; _XROW_20892090_0_[_I_] = 0.0; end; _XROW_20892090_0_[1.0] = 1.0; _TEMP_ = 1.0; select (_VIN_); when ('12345678901234560') _XROW_20892090_0_[2.0] = _TEMP_; when ('12345678901234561') _XROW_20892090_0_[3.0] = _TEMP_; when ('12345678901234562') _XROW_20892090_0_[4.0] = _TEMP_; when ('12345678901234563') _XROW_20892090_0_[5.0] = _TEMP_; when ('12345678901234564') _XROW_20892090_0_[6.0] = _TEMP_; when ('12345678901234565') _XROW_20892090_0_[7.0] = _TEMP_; when ('12345678901234566') _XROW_20892090_0_[8.0] = _TEMP_; when ('12345678901234567') _XROW_20892090_0_[9.0] = _TEMP_; when ('12345678901234568') _XROW_20892090_0_[10.0] = _TEMP_; when ('12345678901234569') _XROW_20892090_0_[11.0] = _TEMP_; when ('12345678901234571') _XROW_20892090_0_[12.0] = _TEMP_; when ('12345678901234572') _XROW_20892090_0_[13.0] = _TEMP_; when ('12345678901234573') _XROW_20892090_0_[14.0] = _TEMP_; when ('12345678901234574') _XROW_20892090_0_[15.0] = _TEMP_; otherwise do ; _BADVAL_ = 1.0; goto SKIP_20892090_0; end; end; do _I_ = 1.0 to 15.0; _LINP_ + _XROW_20892090_0_[_I_] * _BETA_20892090_0_[_I_]; end; SKIP_20892090_0: if (_BADVAL_ = 0.0) & ^MISSING(_LINP_) then do ; P_BLUEBOOKPRICE = _LINP_; end; else do ; _LINP_ = .; P_BLUEBOOKPRICE = .; end; if "P_BLUEBOOKPRICE" = . then "P_BLUEBOOKPRICE" = 22142.857143; EM_PREDICTION = "P_BLUEBOOKPRICE"; _return: ; end; endpackage;   californiaOverride.ds2: /* Custom Object californiaOverride retrieved on Tue Apr 07 09:41:29 EDT 2020 */ package c_EBWC2GAB6RB2BCVZ5DTMUWWRIU /inline; method execute(in_out double Bid, in_out varchar(100) bidAction, in_out char(2) state); if state = 'CA' then do; Bid = 1; bidAction = 'Buy anything from California!'; end; end; endpackage;   autoAuctionDecision.ds2: /* Decision autoAuction version 1.0 generated on Tue Apr 07 09:41:29 EDT 2020 */ package Decision_60e45dfc_ce58_4143_896 / inline; dcl package logger logr_perf('App.tk.SID.Perf'); dcl double timer_start; dcl double elapsed_time; dcl double total_timer_start; dcl double total_elapsed_time; dcl package b_I2SJLZ3T2JBN7HYVMQV32RACEY ruleset347(); dcl package c_EBWC2GAB6RB2BCVZ5DTMUWWRIU customobject348(); dcl package m_3UWKJUHM5JBO5DSZSH34QQREHU model345(); dcl package b_4FNY7JHNGNDU7OT7FNJPAP2F6A ruleset346(); method init(); end; method execute_internal( in_out double "BlueBookPrice" ,in_out double "CurrentBid" ,in_out varchar(20) "Make" ,in_out double "Miles" ,in_out varchar(20) "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out varchar(17) "VIN" ,in_out double "Year" ,in_out varchar(2) "state" ,in_out double "Bid" ,in_out double "P_BlueBookPrice" ,in_out varchar "bidAction" ,in_out double "callOffice" ,in_out integer "_filter_" ) ; dcl integer dg_filter_; dcl integer _masRC; total_timer_start = datetime(); timer_start = datetime(); model345.score("VIN", "BlueBookPrice", "P_BlueBookPrice"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: Linear Regression (Pipeline 1), Node ID: dd2ca4d0-ecea-42ee-8e59-91f7c842243d, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; /*model code*/; /*begin branch node*/ if ( "state" eq 'PA' ) then do; timer_start = datetime(); ruleset346.execute("Bid", "bidAction", "BlueBookPrice", "callOffice", "CurrentBid", "Make", "Miles", "Model", "OriginalInvoice", "OriginalMSRP", "state", "VIN", "Year"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: autoAuctionNorthernStates, Node ID: e15b8fa4-ed33-474f-ba7f-2b52f03f45f0, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; end; else do; timer_start = datetime(); ruleset347.execute("Bid", "BlueBookPrice", "callOffice", "CurrentBid", "Make", "Miles", "Model", "OriginalInvoice", "OriginalMSRP", "state", "VIN", "Year"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: autoAuction, Node ID: 46a495e7-73d2-42df-9f15-642bbd440226, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; end; /*end branch node*/ timer_start = datetime(); customobject348.execute("Bid", "bidAction", "state"); elapsed_time = datetime() - timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, Node Name: californiaOverride, Node ID: 206c2d18-01f4-43a0-8ab9-e8e6ca5ad145, NODE DURATION: ' .. elapsed_time); if ("_filter_" = 0) then return; total_elapsed_time = datetime() - total_timer_start; logr_perf.log( 'd', 'Package Name: Decision_60e45dfc_ce58_4143_896, TOTAL DURATION: ' .. total_elapsed_time); end; method execute( double "BlueBookPrice_" ,double "CurrentBid_" ,varchar(20) "Make_" ,double "Miles_" ,varchar(20) "Model_" ,double "OriginalInvoice_" ,double "OriginalMSRP_" ,varchar(17) "VIN_" ,double "Year_" ,varchar(2) "state_" ,in_out double "Bid" ,in_out double "BlueBookPrice" ,in_out double "CurrentBid" ,in_out varchar "Make" ,in_out double "Miles" ,in_out varchar "Model" ,in_out double "OriginalInvoice" ,in_out double "OriginalMSRP" ,in_out double "P_BlueBookPrice" ,in_out varchar "VIN" ,in_out double "Year" ,in_out varchar "bidAction" ,in_out double "callOffice" ,in_out varchar "state" ); dcl integer _filter_; "BlueBookPrice" = "BlueBookPrice_"; "CurrentBid" = "CurrentBid_"; "Make" = "Make_"; "Miles" = "Miles_"; "Model" = "Model_"; "OriginalInvoice" = "OriginalInvoice_"; "OriginalMSRP" = "OriginalMSRP_"; "VIN" = "VIN_"; "Year" = "Year_"; "state" = "state_"; execute_internal( "BlueBookPrice" ,"CurrentBid" ,"Make" ,"Miles" ,"Model" ,"OriginalInvoice" ,"OriginalMSRP" ,"VIN" ,"Year" ,"state" ,"Bid" ,"P_BlueBookPrice" ,"bidAction" ,"callOffice" ,"_filter_" ); end; endpackage; The files must be saved to a directory location accessible by the ESP server. For our example, we'll just use "/home". Create Your ESP MAS Module Code The MAS Module code is the ESP XML code required to register the 5 DS2 packages to ESP. It will ultimately become part of the ESP project code. Note that the code defines a "MAS Module" for each DS2 package, points the code tag to the package's file location, and sets the "func-name" (DS2 method) to execute.       <mas-modules> <mas-module module='autoAuctionRuleSet' language="ds2" func-names="execute"> <code-file>/home/autoAuctionRuleSet.ds2</code-file> </mas-module> <mas-module module='autoAuctionNorthernStatesRuleSet' language="ds2" func-names="execute"> <code-file>/home/autoAuctionNorthernStatesRuleSet.ds2</code-file> </mas-module> <mas-module module='californiaOverride' language="ds2" func-names="execute"> <code-file>/home/californiaOverride.ds2</code-file> </mas-module> <mas-module module='linearRegression' language="ds2" func-names="execute"> <code-file>/home/linearRegression.ds2</code-file> </mas-module> <mas-module module='autoAuctionDecision' language="ds2" func-names="execute"> <code-file>/home/autoAuctionDecision.ds2</code-file> </mas-module> </mas-modules>       Note: The decision is defined last, ensuring that it will be loaded last into ESP. This is necessary because it has dependencies on the other modules and they must exist before it is defined. Create Your ESP Calculate Window MAS Map Code The Calculate window maps the window input (SRCWin) to the decision MAS Module. The fields on the input are mapped to the Decision's EXECUTE method field definitions by name. So the Calculate window's input must provide every field required by the decision.       <window-calculate pubsub="true" name="Calculate1" algorithm="MAS"> ........ <mas-map> <window-map module="autoAuctionDecision" function="execute" revision="0" source="SRCWin"/> </mas-map> </window-calculate>     Put it All Together With your MAS modules defined and your calculate window MAS map created, integrate them with the rest of your ESP project and start testing. A basic project that simply applies the DS2 logic to streaming auto auction events would look like this:   <project name="SIDCodeTest" threads="1" pubsub="auto"> <mas-modules> <mas-module module='autoAuctionRuleSet' language="ds2" func-names="execute"> <code-file>/home/autoAuctionRuleSet.ds2</code-file> </mas-module> <mas-module module='autoAuctionNorthernStatesRuleSet' language="ds2" func-names="execute"> <code-file>/home/autoAuctionNorthernStatesRuleSet.ds2</code-file> </mas-module> <mas-module module='californiaOverride' language="ds2" func-names="execute"> <code-file>/home/californiaOverride.ds2</code-file> </mas-module> <mas-module module='linearRegression' language="ds2" func-names="execute"> <code-file>/home/linearRegression.ds2</code-file> </mas-module> <mas-module module='autoAuctionDecision' language="ds2" func-names="execute"> <code-file>/home/autoAuctionDecision.ds2</code-file> </mas-module> </mas-modules> <contqueries> <contquery name="cq1"> <windows> <window-source name="autoAuctionSource" pubsub="true"> <schema> <fields> <field name="Make_" type="string"/> <field name="Model_" type="string"/> <field name="state_" type="string"/> <field name="Year_" type="double"/> <field name="BlueBookPrice_" type="double"/> <field name="CurrentBid_" type="double"/> <field name="OriginalInvoice_" type="double"/> <field name="OriginalMSRP_" type="double"/> <field name="Miles_" type="double"/> <field name="VIN_" type="string" key="true"/> </fields> </schema> <connectors> <connector class='fs'> <properties> <property name='type'>pub</property> <property name='fstype'>csv</property> <property name='fsname'>auto.csv</property> <property name='transactional'>true</property> <property name='blocksize'>1</property> <property name='addcsvopcode'>true</property> <property name='addcsvflags'>normal</property> </properties> </connector> </connectors> </window-source> <window-calculate pubsub="true" name="Calculate1" algorithm="MAS"> <schema> <fields> <field name="Bid" type="double"/> <field name="bidAction" type="string"/> <field name="callOffice" type="double"/> <field name="VIN" type="string" key="true"/> <field name="OriginalInvoice" type="double"/> <field name="OriginalMSRP" type="double"/> <field name="BlueBookPrice" type="double"/> <field name="Make" type="string"/> <field name="CurrentBid" type="double"/> <field name="Miles" type="double"/> <field name="Model" type="string"/> <field name="P_BlueBookPrice" type="double"/> <field name="Year" type="double"/> <field name="state" type="string"/> </fields> </schema> <mas-map> <window-map module="autoAuctionDecision" function="execute" revision="0" source="autoAuctionSource"/> </mas-map> <connectors> <connector class='fs'> <properties> <property name='type'>sub</property> <property name='fstype'>csv</property> <property name='fsname'>calc.csv</property> <property name='snapshot'>true</property> </properties> </connector> </connectors> </window-calculate> </windows> <edges> <edge source="autoAuctionSource" target="Calculate1" role="data"/> </edges> </contquery> </contqueries> </project> Further Considerations Some minor code modifications may be required. In the example provided, the generated SID code changed the input variable names from how they were defined in the User Interface. Each variable included an underscore suffix, e.g. "originalInvoice_" and "BlueBookPrice_". This forced me to change the ESP input schema to match. Small modifications like this will likely be required to make SID code run in ESP. The main decision DS2 package must be modified to include the following statement, "ds2_options sas;." This should be included at the beginning. There are slight differences between what DS2 statements are supported in ESP versus SID. Please refer to the documentation for the latest details. In the above example, we referenced our MAS module code files by location. A more formal way to manage MAS modules in ESP is with MAS module stores. You can create the ESP MAS Modules, Calculate window, and ESP Project all in ESP Studio if desired. Speak ... View more

SAS Intelligent Decisioning is a cool little tool with lots of deployment functionality built in. However, as you field workers know, it usually only takes about 5 minutes before your customer is asking to deploy decision logic in some novel and unsupported way. So how can we get decision and rule set generated code to incorporate in other applications like ESP, remote SAS servers, etc.?   Making the Call For the purposes of this post, we're only concerned with extracting the generated decision code. So let's look at an example of the actual command used to extract the code:   https://intviya01.race.sas.com/decisions/flows/0b2a6c4f-afa3-4440-9705-16299cbc346c/code?codeTarget=MICROANALYTICSERVICE&lookupMode=inline&isGeneratingRuleFiredColumn=false As you can see, the command is a REST GET. It uses the Viya Decision Management REST API. Breaking the command into pieces, we can see how to use it:   Viya server: intviya01.race.sas.com (Not a real server name) API Command: /decisions/flows/{Decision ID}/code Decision ID: 0b2a6c4f-afa3-4440-9705-16299cbc346c Options: codeTarget: MICROANALYTICSERVICE lookupMode: inline isGeneratedRuleFiredColumn: false   For more information on the API and options, see the doc here and here. You can get the Decision ID from the Intelligent Decisioning User Interface (Decision --> Properties Tab) among other places.   The example above extracts the code from a Decision but you can also extract Rule Set code. A sample Rule Set code extraction call is below. Note the different URL path (/businessRules/ruleSets/😞   https://intviya01.race.sas.com/businessRules/ruleSets/a44445ab-3f65-4cc4-a206-bfe2e014e206/code?codeTarget=HADOOP&lookupMode=inline&isGeneratingRuleFiredColumn=false Below is an example of the API command being rendered through Chrome and producing the decision code.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Authentication Running the API command in Chrome as shown above will trigger the Viya UI login screen for authentication.   More information on authentication to Viya's REST API can be found here, https://blogs.sas.com/content/sgf/2019/01/25/authentication-to-sas-viya/.   Notes and Considerations The code can be rendered a few different ways. Change the codeTarget between MICROANALYTICS and CAS and the lookupMode among format, inline, and package to see which variation suites your needs. The doc is here. For most applications, you want to use lookupMode=inline as that puts all of the decision components in one file. The Decision code will contain lots of extra supporting code components including logging and administrative functions Intelligent Decisioning logic is rendered via SAS DS2 packages. The Decision itself as well as each individual component -- rule set, predictive model, custom code node -- will be defined as a package. The Decision package acts as a driver program calling the other packages. Branch logic is included in the main Decision package. The generated code is intended to run in the codeTarget specified. Using it in any other platform or circumstance will likely require modification. Search for more content from our group: SAS Global Enablement & Learning. . Speak Speak Speak ... View more

Deploying, implementing, and supporting software requires product knowledge "beyond the demo." So let's take a deeper look at SAS Intelligent Decisioning (SID) Lookup Tables.   In theory lookup tables are simple. They just hold reference data to be joined into transactions, summary tables, etc.. In SID, however, they have an unique implementation and unique considerations. For example they manifest differently in each of the different publishing destinations -- CAS, Teradata, Hadoop, and MAS. So, let's dive in and see what we need to know to utilize SID lookup tables in our decisioning environment optimally. .   SID Lookup Tables are Static When creating SID Lookup Tables, you do not simply point to a CAS table. You create it manually (or import it) in the SID User Interface. So, if you want your lookup data to change, you'll have to go into the SID interface and change it. (We'll hint at possible hacks to get around this later.)    The SID bidCommands lookup table Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SID Lookup Tables can be "In-Line" or "Shared" Depending on the lookupStaticBinding setting, SID lookup tables will either be coded into the published Rule Set/Decision (In-Line), or they will be treated as shared components (Shared).    The lookupStaticBinding Setting     Basically, when lookupStaticBinding is set to yes, the lookup table is hard-coded into the published rule set (or decision). Thus, any subsequent changes to the lookup won't be reflected in the rule set. When it is set to no, rule sets are published referencing the lookup table as a shared component. So the rule set will use the latest version of the lookup at execution time.   Bottom Line: When lookupStaticBinding=No ("Shared"), changes to the lookup table will be reflected in previously published Rule Sets. When lookupStaticBinding=Yes ("In-Line"), changes to the lookup table will not be reflected in previously published Rule Sets. NB: Changing the lookupStaticBinding value does not affect previously published rule sets. So changing the lookupStaticBinding to no will not change the behavior of rule sets published when it was set to yes and visa-versa. .   Publishing Destination Considerations -- Shared Lookups This is where the story ends for in-line lookups. For in-line lookups (lookupStaticBinding=Yes), the lookup is coded into the rule set that references it. Thus, in-line lookups work the same across all publishing destinations, CAS, MAS, Teradata, and Hadoop. They just become part of the published code.   Shared lookups (lookupStaticBinding=No), however, have special considerations for the different publishing destinations .   CAS shared Lookups -- CAS Formats CAS shared lookups are deployed as CAS Formats. If you're not familiar with SAS/CAS formats being used as lookups, see here. Basically PROC FORMAT is used to load a table's contents into a Format with one variable being assigned as a "lookup" value and another field being assigned as a "return" value. When the format is applied, it can translate the data from one "format" to another format. In the above example, a formatted "0" becomes "Nooooo!!!!!" and a formatted "1" becomes "Do it!!!!"   SID bidCommands Lookup as a CAS Format . Since CAS shared lookups manifest as formats, you'll need to make sure your Viya configuration is equipped to support them. Specifically check the sas.referencedata.casformats settings as well as the casstartup_usermods.lua file.   Also, since any CAS process with the proper authorization can alter the formats published by SID, you could theoretically make SID lookups more dynamic by updating the CAS formats regularly outside of SID or even possibly within published decisions using custom code nodes. If you've done this, please comment. Be aware that using any process outside of SID to modify the CAS formats derived from SID lookup tables will make the lookup table data within the SID user interface outdated. The CAS formats and the data shown on the SID lookup table screens would be out of sync. .   In-Database Shared Lookups -- Not Supported In SID 5.4, all in-database published rule sets must use in-line lookup tables. Shared lookups are not supported. In-database published rule sets/decisions with shared lookups simply fail during execution.  .   MAS Shared Lookups -- MAS Code Modules MAS shared lookups are deployed as MAS code modules. The image below shows the MAS Module for the sample bidCommands lookup table. Examining the source code, we see that the lookup rules are placed into a hash. While SID handles the publication of these modules for you, if you'd like to understand more about DS2 in the MAS, see here.    SID bidCommands Lookup as a MAS Module   As with the shared CAS formats, you could introduce a process outside of SID to modify the module using a PUT to replace the DS2. However, the same considerations apply here that apply to the shared formats, most notably that any "outside" process would cause synchronization issues between The SID interface and the lookup table shared modules. .   Further Considerations   SID Data Query Files as an Alternative to SID Lookups   If the static nature or some other aspect of SID lookup tables does not suit your requirements, v5.4 now supports custom SQL through Data Query Files. Note that this feature is only available to MAS-published decisions. You can find documentation on this new feature here as well as MAS usage and configuration information here. .   Speak ... View more