APIs in action: build an automated profiler with SAS Viya

5 Likes

Customer Profiler applicationRecently we were approached by a customer to tackle the following business challenge in their SAS Viya environment.

The challenge

The customer’s marketing operations team needed an easy-to-use tool allowing them to compare two different populations:

a targeted subset of their customer base
the entire population

This exercise would provide value for internal teams, exposing how particular audiences differed from the whole (e.g. responded to an email offer vs. no response). The solution required flexibility and allow the end-user to quickly assign and select different attributes to compare (such as acquisition channel and credit worthiness). The other requirement was for the solution to be automated and therefore shorten the time for the marketing team to take action.

We recognized by creating the application for this requirement we'd cover many other use cases for automation (simple to complex). Our SAS team designed a solution combining the ease and customization of a SAS Visual Analytics front-end with the automation and flexibility of SAS Job Execution Web Application. This was developed in SAS Viya version 3.5 on a cloud instance.

Solution overview

Customer Requirements

Easy-to-use
- The end-user uploads a sample list of customer IDs in the form of a CSV file that would filter a dashboard containing visualizations based on the entire list of customer ID’s, to reflect only the records in the uploaded sample
Automated
- The report refreshes automatically upon uploading the CSV
Intuitive
- Easily understand and compare the differences in populations
Scalable
- Support data volumes approaching 26 GB

Solution Components

SAS Viya:
- The primary component of this solution is SAS Viya (CAS), for its ability to support the large data volumes
CAS engine:
- CAS processes the filter requests very quickly because of its distributed, in-memory architecture, designed for fast performance on large volumes of data
Visual Analytics:
- Web-based interface will house the dashboard and visualizations
- Provides a breadth of dynamic, analytical graphics to present the data in a compelling way to uncover actionable insights
SAS Job Execution Web Application:
- Responsible for executing user-initiated or scheduled scripts comprised of SAS code
Additional features:
- This GUI allows users to manage and run custom SAS code behind the scenes to automate tasks with little manual effort
- Jobs are repeatable – build once, run many times
- An embedded HTML input form provides a user interface to the job that, combined with the job definition, executes in real time
- Build the reports once and update when the data does

End- User Experience

The user is presented an HTML form embedded within the Visual Analytics (VA) dashboard to upload a customer list (in the context of this project, it is the desired CSV file)
The HTML form prompts the user to upload a file to CAS
This triggers a backend script to automatically run the SAS Job that uploads the CSV and filters the report
The result is a 2-page VA report containing the HTML form to upload the customer list CSV file and another page comparing 2 sets of visualizations side-by-side– one set that reflects the uploaded customer list CSV and one that reflects the entire data set. Below is a video outlining a representation of the report and how it works.

(view in My Videos)

Application development

Build the report

We’ll start by building a simple report with static data. Afterward, we’ll embed a script to enable automatic updates when the user provides new data.

Let’s begin in SAS Visual Analytics. From a blank report, we can drag and drop visualizations into the workspace and assign data. Our simple report design includes:

A sample profile of the full customer population (right side)
A sample profile of the user-selected customer list (left side)

This video shows how to build this basic report and begin to populate data variables.

(view in My Videos)

Did you know…Visual Analytics allows you to join multiple tables directly in the report editor? It’s easy. Watch the video below for details.

(view in My Videos)

Build the front-end

Now that we’ve set up the profiler comparison dashboard let’s set up the user-input page. Create a new tab and drag in the Web Content object.

Web Content object

On the Options tab, the Web Content section is where we’ll insert our automated script. We’ll come back to this once our script is created.

Web Content URL

You can include more customized design features and align with your corporate style guides to increase user adoption. For tips on designing interactive and intuitive reports in SAS Visual Analytics refer to the following resources:

D3 and SAS Visual Analytics

SAS Visual Analytics Gallery

Data Can Be Beautiful site from the VA UX team

SAS Support Community posts by Xavier Bizoux

Automation and HTML creation

Next, let’s look at how to create the automated script and HTML.

Automated script

Why re-create the wheel? We’ll use the out-of-the-box example scripts provided in SAS Job Execution as a starting point to accelerate our development.

Access the sample scripts at http://<host>/SASJobExecution/samples

For this application, we used “Upload a CSV File” sample. You can select the check box next to it and then use the Copy To button in the top right to copy the sample to your location. Watch the video below for a demonstration.

(view in My Videos)

Once you’ve copied the sample, you can navigate back to Content and open your file.

We made a few enhancements to the sample source code in “Upload a CSV File.”

Connect to CAS in-memory engine.

/*  Set the options necessary for creating a connection to a CAS server.     */
/*  Once the options are set, the cas command connects the default session   */ 
/*  to the specified CAS server and CAS port, for example the default value  */
/*  is 5570.                                                                 */
/*****************************************************************************/

cas mySession host='&lt;server&gt;' port=5570;

NOTE: added this section to the file; replace <server> with your CAS host

Import the new user-selected file.

proc import datafile= upload
  out=work.Customer_List_in
  dbms=csv
  replace;
  getnames=yes;
run; quit;

NOTE: replaced the ‘out’ value

Verify the join variable (customer_id) is formatted correctly.
```
data work.Customer_List_stg (drop=customer_id_old);
	set work.Customer_List_in (rename=(&iuml;&raquo;&iquest;Customer_ID = customer_id_old));
	length Customer_ID $16.;
	Customer_ID = put(customer_id_old, 16.);
run;
```
NOTE: Added this section to the file. This is important for the data join. If your VA report does not load correctly, check that the variable name matches correctly.

Load the data to the Public CAS library.

*****************************************************************************/
/*  Drop Customer_List from Public and load/promote imported file            */
/*****************************************************************************/

 proc casutil; 
     droptable casdata="Customer_List_new" incaslib="Public" quiet; 
     *promote casdata="Customer_List_stg" casout="Customer_List" incaslib="Public" outcaslib="Public"; 
     load data=work.customer_list_stg casout='customer_list_new' outcaslib='Public' promote;
 quit;

NOTE: added this section to the file

Finally, the code below generates a simple HTML response after the file is loaded to specify that the file is available.

data _null_; 
 length filename $1024; 
 filename = htmlencode(strip("&amp;_WEBIN_FILENAME")); 
 call symputx('_WEBIN_FILENAME', filename); 
/*  if (upcase("&amp;_WEBIN_FILEEXT") ne 'CSV') then do;  */
/*    rc = dosubl('ods _all_ close;');  */
   file _webout; 
   put '&lt;!DOCTYPE html&gt;'; 
   put '&lt;html&gt;'; 
   put '&lt;head&gt;&lt;title&gt;File Loaded&lt;/title&gt;&lt;/head&gt;'; 
   put '&lt;body&gt;'; 
   put '&lt;h2&gt;Uploaded file "' filename +(-1) '" is now available as Customer_List.&lt;/h2&gt;'; 
   put '&lt;h2&gt;You may now refresh your report and close this window.&lt;/h2&gt;';
   put '&lt;/body&gt;'; 
   put '&lt;/html&gt;'; 
   abort cancel; 
/*  end;  */
 run;

NOTE: added this section to the file

Customize the HTML file

Modify the HTML form as needed. Access the HTML by right clicking on your Job Definition and going to Edit > HTML form as shown here:

Access HTML generated file

NOTE: In this example we are using SAS Viya version 3.5. If you are using Viya 3.4, you need to copy both the Job Definition and the Job Form from the Sample page and you’ll see them as two separate elements.

The following highlights key updates in the example script.

You’ll notice several pieces of display text; change these as needed for your job to instruct the end user.

<body role="main">

<div>SAS<sup>&#174;</sup> Job Execution</div>

<h1>Upload a CSV File</h1>

<p>
Use this page to upload a CSV file from your local
machine to the SAS server machine.
</p>

In the following code, the hidden input types serve as job parameters
1. If your program has a different name or location than your HTML form, change the _program parameter. If your HTML and definition have the same name keep this as $PROGRAM$.
2. If you are using the form within a VA report as we are in this example, remove the target= “_tab” parameter. In the following code, the hidden input types serve as job parameters.
```
<form action="/SASJobExecution/" method="post"
  enctype="multipart/form-data">
<input type="hidden" name="_program"     value="$PROGRAM$">
<input type="hidden" name="_action"      value="execute"/>
<input type="hidden" name="_output_type" value="ods_html5"/>
<input type="hidden" name="_csrf"        value="$CSRF$">
<input type="hidden" name="_odsstyle"    value="HTMLBlue">
```

Below is the code for the “Submit” button. Change the command name on the button if desired. You can also remove the “Show SAS Log” checkbox.

<input type="submit" value="Upload Data" class="pointer"/>
<input type="checkbox" name="_debug" id="_debug" value="log" class="pointer"/><label for="_debug">Show SAS Log</label>

Add endpoint to the report

Once we have the Job Execution SAS code and the HTML form, locate the endpoint to embed in the front-end report.

Right-click on your Job Execution project, then select Properties, and open the Details tab.

Job Execution property details

Copy the Job submit action and paste this back to your Visual Analytic web form.

Finish building the application

Let’s return to Visual Analytics to complete the application. In the web content form, go to Options and then enter the job submit URL under Web Content.

Web Content URL entry field

Save the report and select View Report to test functionality.

View Report option

Check out the application

Congratulations! If you’ve followed along through this article, you now have access to the final application.

Final Customer Profile report

Want a quick-start?

Clone the assets used in this article from the sas-viya-job-execution-service-automated-profiler GitHub repository.

You’ll need the Upload_a_CSV_File_definition.json file to configure the automated script in SAS Job Execution. As discussed previously, in Viya 3.5, the job definition and the job HTML are contained in the one file.

Import the json file and the VA report via the SAS Environment Manager > Content > Import button.

Import the JSON file & VA report

Make the following required modifications.

In SAS Job Execution, you need to enter your server name in the CAS connection
In the Visual Analytics Report, edit the Web Content link to your server

That's a wrap

We've demonstrated how to build an application using a prepackaged sample in the SAS Viya Job Execution Web Application. The customer was quite pleased with the POC we created. In fact, they were impressed at how easy the process was to complete. They immediately realized the benefits and took this approach to create several other applications.