In SAS Intelligent Decisioning (as of stable 2024.01 or LTS 2024.03), you can create advanced lists and reference them in rule sets.  Advanced lists can also be shared among SAS applications and workflows.   An advanced list is a tabular data set that can have multiple keys and values.  It enables you to manage tabular data in your environment allowing you to publish and query that data in a low-latency, key-value data store.  Advanced lists also provide the ability to protect sensitive data in your lists using data masks. When you mask data in a list, all or a portion of the data values for an entire list column are redacted in the output.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Advanced lists are stored in Redis. You must provide, configure, and maintain your own Redis instance. For information about configuring Redis for your environment, refer to the List Data Service topic in the SAS Viya: Administration guide.  In addition, an administrator must add any users who need to work with advanced lists to the Advanced List Users custom group in SAS Environment Manager.     Creating an Advanced List   To create an advanced list, select New list on the Advanced Lists page within SAS Intelligent Decisioning.​     The first step in creating in advanced list is to define its properties.  You must enter a name for the advanced list.  Optionally, you can also specify a label and description.     Check the box for deploy list on creation if you want the advanced list to be available for use by other SAS applications and workflows upon its creation.  If you don’t check this box now, you can deploy the list after creation.  Check the box for is immutable to prevent the contents of the list being changed after you add data.     The Products\List Data folder in SAS Content is the default location for storing new advanced lists.  If you want to store it in a different location, select the folder icon and navigate to that location.     For the list content, you have the option to define an empty list or import its contents from a CSV file.  You also have the option to include an expiry date column for the list items.     Next, review the column names from the imported CSV or if creating the list from scratch, then add the column names for your advanced list. If you are importing the items from a CSV file, the column names are taken from the first row of the file.  Confirm you have selected the correct data type for each column.  The available data types are string, numeric, datetime, currency, and currency code.   You can also select whether you want to mask the column.  When masking a column, you can mask all characters or all but the last four characters.  Currently, if you mask the data in a column, then that column cannot be returned in a rule set.  Also, do not select to mask any of your key columns.     Finally, select one or more columns to be the key for your advanced list.  In other words, what column or columns are used to uniquely identify a row on your advanced list.  Select Finish to create your advanced list.     Here is the created advanced list with the data imported from the specified CSV file.     In this case, CLIENT_ID is the key column and EMAIL, PHONE1, and PHONE2 are the value columns.  Depending on your privileges, you may be able to view and/or edit the list properties and its items.   The status of a list item can be deployed, developing, or archived.  Archived is only a status option if you have set an Expiry column for your advanced list. I select to deploy all rows for my advanced list.     This changes all the list items to the status of deployed.  Only deployed items are returned when referencing an advanced list.       Referencing and Advanced List in a Rule Set   Once a list is deployed, you can use an advanced list reference in an assignment or common rule set to retrieve values from the list.     The first step is to select the advanced list you want to reference.  Only deployed advanced lists are available for selection here.     Next, select the value columns you want to retrieve.  Only the value columns that are not masked are listed here.     Finally, map the input variable or variables to the key column or columns on the advanced list.  Select Finish to complete the reference logic for the rule set.     This is what the reference to the advanced list looks like in the rule set.     Below are the results of testing that advanced list reference.       Summary   The advanced lists features enable you to manage tabular data in your environment and query that data in a low-latency, key-value data store. For more information, refer to the SAS Intelligent Decisioning documentation on Managing Advanced Lists.     Find more articles from SAS Global Enablement and Learning here. ... View more

In SAS Intelligent Decisioning (as of stable 2023.12 or LTS 2024.03), you can add a Parallel Process node to a decision.  This node processes multiple decisions in parallel within a decision.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   There are several rules that must be followed when using a Parallel Process node in a decision.​   A parallel process node can contain only decisions. The same decision cannot be added to the same parallel process node more than once.​ None of the decisions or their sub-decisions that are added to a parallel process node can contain parallel process nodes.​ All the decisions that are added to a parallel process node must return exactly one output variable of type data grid. For all decisions that are included in the same parallel process node, input variables that have the same name must have the same data type, and the columns in the output data grids that have the same name must have the same data type.   You can publish decisions that contain parallel process nodes to SAS Micro Analytic Service (MAS), Container destinations, and SAS Cloud Analytic Services (CAS). By default, decisions in a parallel process node are processed in parallel only for MAS and Container destinations. In CAS, the decisions in the parallel process node are processed sequentially.​   ​To successfully execute the decisions in parallel process nodes, you must configure the time-out limit and the maximum number of threads for the destination to which you publish decisions that contain the parallel process nodes. For more information, refer to the Configuring Support for Parallel Process Nodes topic in SAS Intelligent Decisioning: Administrator’s Guide.​     Example Decision with Parallel Process Node   Here is an example decision using the Parallel Process node.     The parallel process node executes two sub-decisions at the same time.  These two sub-decisions use two slightly different methods to determine if a customer’s mortgage is underwater (i.e., the customer owes more than the property is worth).  Note:  I could add other nodes before or after the parallel process node in the decision flow, but for the purposes of this post solely focusing on the parallel process node.   Let’s look at both of the sub-decisions from the Parallel Process node.     Both of the sub-decisions only have one Output variable, and it is of type data grid.  Also, they both have an Input variable called CLIENTID and in both sub-decisions it has the same data type of Character.     For both sub-decisions where their Output data grids have the same variable names they are also of the same type.     The main decision has the needed Input and Output variables.     The PPN_OUT_1 data grid is the merged data grid of all the sub-decisions in the Parallel Process node.  It also includes columns related to the Parallel Process node (PPN_ columns).  For example, the PPN_DECISION_NAME column lists which sub-decision produced the row in the data grid.     Now, let’s test the Parallel Process decision.  Here you can view in the output data grid which sub-decision produced the row in the output.       Summary   Using a Parallel Process node in a decision is especially useful when you have many sub-decisions that require a significant amount of input and output operations or that have a lot of complex logic, and all these decisions need to be run before a final decision is reached. For more information, refer to the SAS Intelligent Decisioning documentation on Adding Parallel Processing Nodes.   You can also build the example decision using the Parallel Process node from this post by taking the SAS Intelligent Decisioning: Beyond the Essentials course.     Find more articles from SAS Global Enablement and Learning here. ... View more

In SAS Intelligent Decisioning (as of LTS 2023.10), you can create segmentation trees and value lists.   A segmentation tree is a hierarchical structure that consists of a root node, branches, and final leaf nodes called outcomes. Any branch can lead to additional branches or to an outcome. Each branch in the tree specifies criteria that determine whether that particular branch is executed. The criteria for a branch might be Boolean expressions, individual variable values, or combinations of variable values called a matrix.  Outcomes define actions to be taken when all of the branch criteria leading to the outcome are satisfied.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When defining a variable for the segmentation tree, you can associate the values of the variable to a value list. A value list is the list of acceptable values for a variable.  Value lists are used for Segmentation tree branches of type Variable or Matrix.   For a large, complicated decision that uses simple conditions, a segmentation tree can be an effective way to visually represent the decision's model and to manage the associated outcomes.   Value Lists   Value lists can be of type Character or Numeric.     The entries in a numeric value list can be discrete values or ranges.   To specify a range of numeric values, use v to indicate the value of the variable that is associated with the list. Numeric ranges cannot overlap.  To enter multiple value specifications in one entry, separate the specifications with a semicolon.  Multiple entries within the same list cannot specify the same value.   Here is an example of a value list for credit score ranges.     For more information, refer to the Specifying Value List Entries topic in the SAS Intelligent Decisioning documentation.   Segmentation Trees   SAS Intelligent Decisioning provides a low-code interface for designing segmentation trees, and it displays the segmentation tree in a simplified, compact format.  The three main elements used to build a segmentation tree in SAS Intelligent Decisioning are: Variables, Branches, and Outcomes.   Just like other objects in SAS Intelligent Decisioning, you list the variables needed for segmentation trees on the Variables tab.  For each variable, you specify a name, data type, whether it is an input, an output, both, or neither.  You can also specify an Initial value for an output variable.   What is different about variables for a segmentation tree is that you can associate a value list with a variable; this means that values for this variable come from the associated value list.     There are three types of elements used to create branches in a segmentation tree – Boolean expression, Matrix, and Variable. When you add one of these elements to a segmentation tree, SAS Intelligent Decisioning creates branches for each of the possible values or result strings that can be produced by the element. For each value or result string, you can either add an outcome to be executed, or you can add additional branch elements to the tree.   A value list must be associated with the variable that is used to create the branch type of variable or matrix. A matrix requires two variables associated with different value lists thus producing a matrix whereas a variable branch is a single variable associated with a value list.     An outcome is the last leaf in a segmentation tree branch. An outcome is a named set of actions to be taken when all the branch conditions leading to the outcome are satisfied. An outcome can be one or more assignment statements, one or more DS2 code files, or a combination of both. You can reuse the same outcome in multiple places in a tree. You can also customize each instance of the same outcome as needed.       Using a Segmentation Tree in a Decision   I can now finally use the segmentation tree in a decision flow.  Adding a segmentation tree to a decision flow is similar to adding any other object to the flow.  Once the segmentation tree is added, you can select to open it in its editor.  You can publish a decision with a segmentation tree to any of the publishing destinations.       Summary   Using Segmentation Trees with value lists in your decisions can be a useful combination for representing your decision logic.  For more information, refer to the SAS Intelligent Decisioning documentation on Using Value Lists and Working with Segmentation Trees.     Find more articles from SAS Global Enablement and Learning here. ... View more

In SAS Intelligent Decisioning when creating rule sets, you can create a Common rule set that can be referenced by many Assignment rule sets.  When an Assignment rule set containing a reference to a Common rule set executes, the Common rule set is executed as if it was defined directly in the Assignment rule set.   Let’s look at an example of creating and referencing a Common rule set in SAS Intelligent Decisioning.   When creating the Common rule set, you must select the type of Common.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Building a Common rule set is the same as building an Assignment rule set.  You can use IF-THEN-ELSE logic and add assignments.  Here is the Common rule set, I have built for this example.     Next, you need to create a rule set of type Assignment.  Common rule sets can only be referenced in Assignment rule sets; they cannot be referenced in a Filtering rule set.     In this rule set, select Add rule and then select Common rule set reference to open the Add Common Rule Set References dialog.     Note: If rules have already been added to this assignment rule set, then you select Add > Add reference.   First step, select the Common rule sets, you want to add as a reference, then select Next.     Second step, select the version to reference for each of the selected Common rule sets, then select Next.     Note: If this referenced version of the Common rule set changes, then the new logic is used by this Assignment rule set.   Third step, map the input variables.  You can select Add Missing Variables to automatically add the input variables from the Common rule set to the Assignment one you are creating.     Fourth step, map the output variables.  You can select Add Missing Variables to automatically add the output variables from the Common rule set to the Assignment one you are creating.     Fifth step, order the selected Common rule set references using the arrow buttons and select Finish.     The Common rule set(s) are referenced in the Assignment rule set.  You can add other rules within this Assignment rule set and test and publish this Assignment rule set that contains a reference to the Common rule set as needed.     You can also expand any referenced Common rule set to view its current logic.     Summary   The use of Common rule sets allows for reusability of rule logic and if a change is needed, then you only need to make it in one place.  For more information, refer to the SAS Intelligent Decisioning documentation on Rule Set Types.     Find more articles from SAS Global Enablement and Learning here. ... View more

As of the stable 2024.04 SAS Viya release, you can specify whether the values for the column selector control in a custom step come from an input table or from another column selector control.   Here is the new option:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In this example custom step, the first column selector has its selections tied to an input table.     The second column selector is tied to the first column selector which means its available values for selection will come from the values selected in the first column selector.     Note:  The Link to column selector option lists the column selector controls in the step that are linked to input tables. This list does not include column selectors that are linked to another column selector control. In other words, the second column selector will not show as possible link to another column selector.   I also used a new style feature available for custom step controls to indent the second column selector to make it more visually obvious that is related to the first column selector.     When I preview the custom step, I can select certain columns for the first column selector control.     Once I have selected those four columns for the first column selector control, I can only select from those for the second column selector control.     Summary   This is an enhancement to the column selector control that you can take advantage of in your custom step design.  For more information on the Column Selector control in a custom step, refer to the following: Column Selector Control documentation SAS Studio Custom Steps - Coding Made Easier Using the Column Selector Control blog SAS Viya: Using Cascading Prompts in a Custom Step blog. ... View more

As of the Viya stable 2024.01 release, there is a Link control for Custom Steps. Use the Link control to add hyperlinks to the user interface of your custom step design. For security reasons, this feature must be enabled by a SAS administrator in SAS Environment Manager on the Configuration page before it can be added to a custom step.   Enable Link Control for Custom Step   A SAS administrator must log on to SAS Environment Manager to configure the use of the Link control in a Custom Step.  On the Configuration Page, search for flow.  Select the Data Flow service.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   For sas.data.flows section, select  to edit the settings for the allowedStepLinks option.     Add supported domains (e.g. https://, file://, vscode://, etc.) in the format of a regular expression (regex) and use the OR character (|) in a regular expression to separate multiple domains.  Note: JavaScript is not supported.  I enter the following regular expression and select Save.   ^https?:\/\/(?:.+\.)?sas\.com(?::\d+)?(?:\/.*)?|^vscode:\/\/file\/C:\\.+     This allows links to be added in Custom Steps for https:// for various sas.com sites OR files to open in VSCode that are only located on the C:\ drive.   For help with regular expressions, you can visit the site https://regexlearn.com/.   Use Link Control in a Custom Step   In the Custom Step Designer, there is now a Link control.     I add the Link control to my custom step canvas and enter the link: https://learn.sas.com.     I select Preview and the link is displayed since it matches one of the regex patterns entered in the configuration.     I select the link and website for the entered link is opened in the browser.     I now change the link to: https://google.com.     I select Preview and a message is displayed that link does not match one of the regex patterns entered in the configuration.     I now change the link to: vscode://file/C:\PSGEL315-sas-viya-sas-studio-flow-and-custom-steps\README.md     I select Preview and the link is displayed since it matches one of the regex patterns entered in the configuration.     I select the link and specified file is opened in VSCode.     Note:  You may have issues with using the link control especially the file:// protocol.  Exposing local system files is a major security concern that browsers handle in their own ways.  You may need to adjust some settings and/or add browser extensions.  Some variables that are out of SAS Studio’s control are:   The browser being used. The version of the browser being used; policies change with every release. The company IT policies (whether company guidelines or via managed software).   Summary   There is now a Link control in Custom Steps that you can use to add hyperlinks to your custom step user interface.  This allows you to link to sites and files that provide more information to aid the user in completing the user interface of your custom step or provide some documentation/background about the custom step.  For more information, refer to the documentation.     Find more articles from SAS Global Enablement and Learning here. ... View more

As of the Viya stable 2023.11 release, you have the option to change the default library for the writing of temporary output tables when executing a flow in SAS Studio.  This new option only applies to SAS Studio Flows.   By default, the WORK library is used but you now have the option to set a different library for these temporary tables.  You can set this option for all SAS Studio flows you create or set it for a single SAS Studio flow.   Set for All SAS Studio Flows   In SAS Studio, select Options → Preferences from the menu.  Select Tables to view the table preference options.  The newly added section is Default Output Libraries.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The application default is the WORK SAS library.  Select the radio button for Output library location and then select   and navigate to the library you want to set as the new default for SAS Studio Flows.     Click Save to save the default output library selection.  This sets the default output library for all your SAS Studio Flows.   Set for Single SAS Studio Flow   Open the SAS Studio Flow where you want to set a different default output table library option.  On the right-hand side of the flow, select    to open the Properties pane for the flow.     In the Default output library for intermediate tables section, the application default is the WORK SAS library.  Select the radio button for Output library location and then select    and navigate to the library you want to set as the new default for SAS Studio Flows.     Save the flow to save the change to the properties.  This sets the default output library just for this SAS Studio Flow.   Summary You can now change the location where temporary output tables are written when executing a SAS Studio Flow.  For more information, refer to the documentation.   Find more articles from SAS Global Enablement and Learning here. ... View more

As of the Viya stable 2023.10 release there is another option for dependencies between controls – the Enablement dependency option.  Therefore, there are now two dependency options – Visibility and Enablement.  I will review both dependency options in this post.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Visibility versus Enablement   The Visibility dependency option is used to determine whether or not the control is displayed to the end user.  A control that uses the Enablement dependency option is always displayed and the dependency is used to determine whether or not the control is available for editing.   Example   Let’s look at a working example.  I downloaded the GeoDistance with Rounding custom step from the GitHub - sassoftware/sas-studio-custom-steps: Repository to share SAS Studio Custom Steps to illustrate the dependencies options.  For more information on how to do this, view this YouTube video.   I open the custom step in Designer mode and look at its Options tab to review the dependency options.     Visibility   Currently, the Decimal places text or numeric control has the dependency for Visibility set.  If the Round checkbox control is checked, then the Decimal places text or numeric control will be displayed; otherwise, the Decimal places text or numeric control will not be displayed.  For more information on the expression syntax for dependencies, view its documentation.     Let’s preview this.  With the Round checkbox control unchecked, the Decimal places text or numeric control is not displayed.     When the Round checkbox control is checked, the Decimal places text or numeric control is displayed.     Enablement   Now, let’s change so the dependency for the Decimal places text or numeric control uses the Enablement option instead.     When the Round checkbox control is unchecked, the Decimal places text or numeric  control is displayed, but it is read-only (i.e., not enabled).     When the Round checkbox control is checked, the Decimal places text or numeric is displayed and able to be edited (i.e., enabled).     Summary   The new dependency feature, Enablement, gives you more design possibilities for your custom steps.   For more information on the new Enablement dependency review its documentation.     Find more articles from SAS Global Enablement and Learning here. ... View more

In a previous article, I demonstrated how you can use a SAS Quality Knowledge Base (QKB) reference table in a custom step. Those reference tables are based on an out-of-the-box QKB. What if you have a customized QKB? In this blog, I show how you can create the QKB definition reference tables for your customized QKB. Then, you could use those custom QKB reference tables in custom steps or in the SAS-delivered QKB steps (e.g., Match Codes).     The high-level steps to create the data quality reference library for a customized QKB are: Add the Customized QKB to SAS Compute as Default QKB Create a Folder on the SAS Server for the Reference Tables Run the DQ - Create QKB Reference Tables Custom Step Add the Created QKB Reference Tables Location to SAS Studio AUTOEXEC Add the Customized QKB to SAS Compute as Default QKB The DQ - Create QKB Reference Tables custom step analyzes the default QKB in SAS Compute to create the QKB reference tables. Therefore, follow the steps in this article to set your customized QKB as the default for SAS Compute.     Create a Folder on the SAS Server for the Reference Tables Next, you need to create a folder on the SAS Server to save the output data sets for the QKB reference tables. In SAS Studio, on the Explorer tab, navigate to the SAS Server location where you want to create the folder. Right-click and select New folder. Note: You should be logged in as a SAS Administrator to perform this function.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Enter the name of the folder you want to create and select OK.     Make note of the location of the SAS Server folder; you will need it when you run the custom step to create the QKB reference tables.     Run the Create QKB Reference Tables Custom Step The DQ - Create QKB Reference Tables custom step is located here. Select    to download the .step file.     In SAS Studio log on as a SAS Administrator. Navigate to the Explorer tab, upload the DQ - Create QKB Reference Tables.step to a SAS Content folder.   Open the custom step in its own tab.     Enter the desired name for the SAS Library and select the SAS Server path you created for the output tables.     Run the custom step and view its results. The specified SAS library is created with the QKB reference tables for each of the definition types. In my customized QKB there is a parse definition at the English language level called Clothing and it has the tokens of: Department, Item, Color, and Size.     Add the Created QKB Reference Tables Location to SAS Studio AUTOEXEC After running this custom step, you may want to consider adding libname SASDQREF "path-containing-your-QKB-Reference-Tables"; to your autoexec in SAS Studio or SAS Environment Manager. This results in the Data Quality steps that use the QKB to utilize your newly created reference tables. I am going to do this in SAS Environment Manager, so it is available for all SAS Studio users.   In SAS Environment Manager on the Configuration page, select Compute Service. Edit the sas.compute.server: autoexec_code section. Note: You should be logged in as a SAS Administrator to perform this function.     In the Contents area enter the following:   /* Custom QKB Reference Tables */ libname SASDQREF "/qkb/reference/path" ;   Log out of all SAS Compute sessions. Must re-open compute session for change to take effect. I sign-out and close my browser with Viya. I re-open and sign back into my Viya installation. The SASDQREF Library with my custom QKB reference files should now be listed anytime a new SAS Compute session is started.     Now, any steps that use the QKB (e.g., Match Codes) now reference my customized QKB reference tables. As shown in the screen shot below, the Match Codes step allows me to select my custom match code definition of Clothing.     Summary The DQ - Create QKB Reference Tables custom step analyzes the default QKB in SAS Compute to create the QKB reference tables. You can then use these reference tables based on your customized QKB in the supplied QKB steps or in your own custom steps. For more information, please refer to this blog.   Acknowledgements Thanks to Charles Norman, Brant Kay, and Nicolas Robert for their assistance with the Program code for this custom step.   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Studio Custom Steps enable you to create a user interface for users at your site to complete a specific custom task. There is a public GitHub repository of these custom steps designed by both SAS employees and other customers that you can import and use at your site. You can use them as designed or tweak them to fit your specific use case. In this presentation, you will learn how to access this public GitHub repository of custom steps and import them into your environment. This demonstration will use two custom steps from the public GitHub repository in a SAS Studio Flow that anonymizes a column in the input data, creates a new output table, and then sends an email that the output is ready for viewing. ... View more

In SAS Viya, you can use the SAS Quality Knowledge Base (QKB) to perform data quality operations (e.g., parse and standardize) on your data either in SAS Data Studio (CAS) or in SAS Studio (CAS and/or SAS Compute).  As an Administrator, you can add a customized QKB to your SAS Viya installation.   Note:  To learn how to customize a QKB, you can take the Understanding the SAS Quality Knowledge Base VLE course. The QKB Customization Tool can be downloaded from here.   The high-level steps to add a customized QKB in SAS Viya for CAS and SAS Compute are:   Create a QARC (QKB Archive) File for the Customized QKB Add Customized QKB QARC File to CAS Add Customized QKB QARC File to SAS Compute   Create a QARC File (QKB Archive) for the Customized QKB   A QKB must be in the QARC (QKB Archive) format to be imported to a CAS Server and/or SAS Compute in SAS Viya.  A command-line QARC tool is provided to convert your QKB to this format.  The tool to create the QARC file is available for download here.   I downloaded the Windows version of QARC program since the tool used to Customize a QKB is Windows-based, so it is easier to run this process there.   Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   The tool needs to be extracted.  I extract the program to my C:\Customized QKBs directory.     To create a QARC file, first locate the directory of your customized QKB.  In my case it is located at: C:\Customized QKBs\CI33_customized.     Once you have located the directory that contains your customized QKB, then from a Windows command prompt run the following code:    Cd \path\to\qarc.exe qarc create \path\to\qkb myqkb.qarc   In my case I submit the following commands.    cd "C:\Customized QKBs" qarc create "C:\Customized QKBs\CI33_customized" CI33_customized.qarc   Note:  It may take about 10 – 20 minutes to create the QARC file depending on the size of your customized QKB.     The QARC file has been created and is ready to be imported to CAS.   Add Customized QKB QARC File to CAS   Note:  In order to import the QKB file using the user interface, the QARC file must be in a location accessible from your local machine.     Next, log on to SAS Environment Manager as an Administrator to the Viya installation where you want to add the customized QKB.  You must have SAS Administrator privileges to import a QKB QARC file into SAS Viya.  When signing in, select Yes to assume the group SASAdministrators, so you have the appropriate privileges.     Select Manage Environment to access SAS Environment Manager, then select the Quality Knowledge Bases tab.     To import a QARC file, select  and navigate to where the QARC file for the customized QKB is located by using the Browse button. Enter the target name and server for the QKB.  Select Import to import the file.     Note:  It may take about 5 – 10 minutes to import the QARC file.   Once the import is complete, confirm that the imported QKB is now listed.       Set Customized QKB as Default QKB in CAS   After importing the QKB, to designate the newly imported QKB as the default QKB for your CAS server, add (or change) the configuration settings for the CAS server.   In SAS Environment Manager on the Configuration page, select the name of your CAS server. By default, this is cas-shared-default. Edit the sas.cas.instance.config: config section.      In the Contents area enter the following:     cas.DQSETUPLOC="DefaultQKBName" cas.DQLOCALE="DefaultLocale"   Note:  The DefaultQKBName must match the name found on the Quality Knowledge Bases page of SAS Environment Manager.  The DefaultLocale is the 5-character QKB locale abbreviation.  For example, ENUSA = English-United States and ESESP = Spanish-Spain. SAS Quality Knowledge Base for Contact Information 33 - QKB Locale ISO Codes   Below are my modified config settings to set my imported Customized QKB as the new default QKB.     After editing this configuration setting, you need to restart the sas-data-quality pod for the change to take effect by issuing the following commands:     kubectl -n name-of-namespace delete pods -l app.kubernetes.io/managed-by=sas-cas-operator kubectl -n name-of-namespace get pods | grep sas-data-quality kubectl -n name-of-namespace delete pod sas-data-quality-pod-name   Note:  In the last command, replace sas-data-quality-pod-name with the name of the pod that is returned by the second command. This enables the automatic import of the QKB that is specified by the DQSETUPLOC server option.   Here is what I submitted to restart the sas-data-quality pod for my SAS Viya installation.  Note: My namespace is gelenv.     Finally, after waiting a few minutes for the sas-data-quality pod to restart, return to the Quality Knowledge Bases tab in SAS Environment Manager and refresh the display to confirm that the Customized QKB is now set as the default QKB.     Note: It will take a few minutes for the pod to restart and reflect this change.  It will display a message “No QKBs were found” until the pod is fully restarted.   Add Customized QKB QARC File in SAS Compute   To add the customized QKB to SAS Compute, upload its .qarc file to a location on your SAS Server. I uploaded mine here: /shared/gelcontent/Customized_QKBs/.     Next, you need to find the location of the QKB bash scripts for your SAS Viya installation by issuing the following command.    find . -name containerize-qkb.sh     In my case the QKB bash scripts are located here: ./project/deploy/gelenv/sas-bases/examples/data-quality/scripts/   Next, change directory to that location:     Then, run the following command to containerize your customized QKB:   bash containerize-qkb.sh "My Customized QKB" /path/to/myqkb.qarc registry.mycompany.com/mycustomizedqkb:v1   Note: registry name must be in lowercase with no spaces.     Note: It may take a few minutes for this script to complete.   Next, run the following command to deploy the customized QKB.   bash deploy-qkb.sh registry.mycompany.com/mycustomizedqkb:v1 | kubectl apply -n my-namespace -f –       Then run this command to create the sas-quality-knowledge-base-list-job.   bash list-qkbs.sh | kubectl apply -n my-namespace -f –     Then run the following command to get a list of registered QKBs for SAS Compute:   kubectl -n my-namespace logs job.batch/sas-quality-knowledge-base-list-job-nameReturnedFromPreviousCommand     All registered QKBs in SAS Compute are displayed including the customized one you just registered. Finally, run the following command to delete the sas-quality-knowledge-base-list-job you just created.   kubectl -n my-namespace delete job.batch/sas-quality-knowledge-base-list-job-nameReturnedFromPreviousCommand       Set Default QKB in SAS Compute   In SAS Environment Manager on the Configuration page, select Compute service. Edit the sas.compute.server: configuration_options section.        In the Contents area enter the following:   -DQSETUPLOC "DefaultQKBName" -DQLOCALE (DefaultLocale)   Note:  The DefaultQKBName must match the name of the customized QKB returned from the listing of registered QKBs for SAS Compute.  The DefaultLocale is the 5-character QKB locale abbreviation.  For example, ENUSA = English-United States and ESESP = Spanish-Spain. SAS Quality Knowledge Base for Contact Information 33 - QKB Locale ISO Codes     Log out of all SAS Compute sessions. Must re-open compute session for change to take effect.  I sign-out and close my browser with Viya.  I re-open and sign back into my Viya installation.   In SAS Studio, test with a definition in the default locale specified for your default customized QKB to confirm customized QKB is being used.   A custom parse definition in my customized QKB is Clothing.      data _null_; version=DQVERQKB(); token_names=DQPARSEINFOGET('Clothing'); put version; put token_names; run;     I have now confirmed that SAS Compute is using my customized QKB and default locale setting. Refer to the SAS Data Quality Language Reference section in the SAS Data Quality guide other code examples you can test with.   Summary You can use your customized SAS Quality Knowledge Base (QKB) in both CAS and SAS Compute in SAS Viya to perform data quality operations.  Refer to the QKB Management section in the SAS Data Quality guide for more information.     Find more articles from SAS Global Enablement and Learning here. ... View more

In my previous post, I reviewed how to use the Make control read-only and hide the control at runtime options for referencing table information to populate a drop-down list selector in a Custom Step. In this post, I will cover how to use those new options with the new SAS Quality Knowledge Base (QKB) reference tables in a Custom Step, so you can include the QKB locale and definition selections in your Custom Step.     As of the 2023.06 release, there is a new SAS library called SASDQREF.  It contains data quality reference tables; the majority of which contain the metadata from the SAS Quality Knowledge Base (QKB).  The tables are located on the SAS Server at: /opt/sas/viya/home/commonfiles/home/share/DMData.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   CONTACTS – listing of contact records that can be used to demonstrate data quality functions.  This is the same data as the CONTACTS table delivered with Data Management Studio. DM_CADEF - listing of available casing definitions in the QKB DM_DEF - listing of all definitions in the QKB DM_EXDEF - listing of extraction definitions in the QKB DM_GNDEF - listing of gender analysis definitions in the QKB DM_IDDEF - listing of identification analysis definitions in the QKB DM_LGDEF - listing of locale guessing definitions in the QKB DM_LOC - listing of all locales in the QKB DM_MADEF - listing of all match code definitions in the QKB DM_MKDEF - listing of all data masking definitions in the QKB.  This is a subset of the standardization definitions in the QKB and are used to anonymize and mask data. DM_PADEF - listing of all parsing definitions in the QKB DM_PTDEF - listing of all pattern definitions in the QKB DM_STDEF - listing of all standardization definitions in the QKB   To demonstrate how these tables can be used I am going to review how I can now update my Anonymize and Mask Data Custom Step from a prior post.  (Note:  The user interface for creating custom steps has had improvements since I wrote that post.)    In that post when I built that version of the Anonymize and Mask Data Custom Step, I manually entered the data masking standardization definitions in the drop-down list control and hard-coded QKB locale to ENUSA (English – United States) in the Program (Template) code.  In this post, I will show how I can use the DM_MKDEF table that lists all the data masking definitions in the QKB to populate the drop-down lists for the data masking definitions as well as one for the QKB locale.    Below is an excerpt from the table.  As part of my custom step, I will be referencing the columns highlighted in yellow.       To start my revised Anonymize and Mask Data Custom Step, I add the controls for an input and output table.     Next, I add another input table control.  This one I use to hard-code its table to the DM_MKDEF table in the SASDQREF library.  I set its properties to read-only and hide control at runtime so the end user of the custom step will not see this control.     I add a column selector control and hard-code it to the Locale description column (DM_DESC) on the DM_MKDEF table. I set its properties to read-only and hide control at runtime so the end user of the custom step will not see this control.     I add a column selector control and hard-code it to the data masking definition column (DM_NAME) on the DM_MKDEF table. I set its properties to read-only and hide control at runtime so the end user of the custom step will not see this control.     Next, I add a column selector control to allow the user to select the column to mask from the input table they select.     I add a drop-down control for the QKB locale selection and make it a dynamic list based on the Locale column selector I added previously.     I add a drop-down control for the data masking definition selection and make it a dynamic list based on the Masking definition column selector I added previously.     Next, I create a prompt hierarchy between the two drop-down fields.  The available masking definitions to select are filtered by the QKB locale selection.  For more information on prompt hierarchies, refer to this post.     I had a new column control for the new masked column and give it a default name of maskedColumn; however, the end user can change that name if desired.     For the last part of my custom step design, I drop the column to be masked from the output port and add the new masked column.     To complete my custom step, I add the code to be executed on the Program tab.  As part of the code, I return the five-character QKB Locale that is associated the description of the locale that the end user selected.  Then, that locale is loaded and the dqStandardize function is used to apply the selected masking definition for that locale to return the masked column.   /* Create SASDQREF library in case it doesn't exist in the context where this custom step is being executed */ %macro checkSASDQREF ; %if %sysfunc(libref(SASDQREF)) %then %do ; %if %sysfunc(libname(SASDQREF,!sasroot/../commonfiles/home/share/DMData)) %then %put %sysfunc(sysmsg()) ; %else %put NOTE: SASDQREF library assigned to %sysfunc(pathname(SASDQREF)). ; %end ; %else %put NOTE: SASDQREF library exists. ; %mend checkSASDQREF ; %checkSASDQREF ; /* Resolve selectedLocale to its 5-character QKBLocale */ proc sql ; select distinct DM_LOCALE into :QKBLocale from SASDQREF.DM_MKDEF where DM_DESC="&selectedLocale"; run ; %put &=QKBLocale.; /* Load Selected QKB Locale */ %DQLOAD(DQLOCALE=(&QKBlocale)); /* Apply Data Masking Standardization Definition */ data &outTable (drop=&incolumn_1_name); /* also specify the drop column in the output table metadata */ set &inTable; /* also specify the new column name in the output table metadata */ &maskedColumn_name=dqStandardize(&inColumn_1_name,"&selectedDefinition", "&QKBlocale"); run;   Now I can test my revised Anonymize and Mask Data custom step.  I select an input table and select to mask the CONTACT column.  I also select to use the English (United States) QKB locale and its Mask All Characters Except First and Last masking definition.  The maskedColumn  I rename to CONTACTmasked on the output table.     Note:  The controls that I hard-coded are hidden from the end user since I selected to hide those controls at runtime.     Summary   As of the 2023.06 release, there is a new SAS library called SASDQREF.  It contains data quality reference tables; the majority of which contain the metadata from the SAS Quality Knowledge Base (QKB).  These can help you build custom steps where you want to use definitions from the QKB.   The revised Anonymize and Mask Data custom step that I built as part of this post can be downloaded from here.        Find more articles from SAS Global Enablement and Learning here. ... View more

As of the March stable release (2023.03) the custom step author can now specify the table and column to populate a selection list and they also have the option to hide or show that information to the end user.  Let’s look at these new control options of Make control read-only and Hide control at runtime in action by reviewing a custom step I created to perform currency conversion.   In my custom step, I get the list of currencies from the CURRENCY_CD column on the SASHELP.SAS_CURRENCY table.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I added an input table control and selected the option to Make control read-only and set the default library to SASHELP and the default table to SAS_CURRENCY to hardcode it as the reference table for the list currencies.  I have labeled this input reference table as the Currency reference table.  For now, I leave the Hide control at runtime option unchecked.     I then added a column selector control and linked it to my Currency reference table.  I selected the option to Make control read-only and set the default column to CURRENCY_CD to hardcode it as the reference column for the list currencies.  I have labeled this reference column as the Currency code column.  For now, I leave the Hide control at runtime option unchecked.     I then add a drop-down list controls to select the currency to convert from and to. They both use a dynamic list and the reference column is the Currency code column.     For the rest of the custom step, I have the input table, column that contains the currency value to be converted, the new column to contain the converted currency value and the output table.     When I select Preview the Currency reference table and Currency code column are displayed, but the user is not able to change the values since those controls are set to read-only. Also, the Convert currency from and Convert currency to drop-down columns contain the list of CURRENCY_CD as their selections.     Note:  When using this Custom Step in a SAS Studio Flow the Currency reference table and Currency code column controls wll be displayed inside the properties pane of the custom step since the option to hide at control at runtime.   There is also a page to enter the API key for performing the currency conversion lookup.     Here is the full Prompt UI for the currency conversion step:   { "showPageContentOnly": true, "pages": [ { "id": "currConv", "type": "page", "label": "Currency Conversion", "children": [ { "id": "currencyTable", "type": "inputtable", "label": "Currency reference table", "required": true, "placeholder": "", "visible": "", "readonly": true }, { "id": "currencyCode", "type": "columnselector", "label": "Currency code column", "order": false, "columntype": "a", "max": 1, "min": 1, "visible": "", "readonly": true, "table": "currencyTable" }, { "id": "convertFrom", "type": "dropdown", "label": "Convert currency from:", "items": { "ref": "currencyCode" }, "required": true, "placeholder": "", "visible": "" }, { "id": "convertTo", "type": "dropdown", "label": "Convert currency to:", "items": { "ref": "currencyCode" }, "required": true, "placeholder": "", "visible": "" }, { "id": "inTable", "type": "inputtable", "label": "Select the input table:", "required": true, "placeholder": "", "visible": "" }, { "id": "currencyCol", "type": "columnselector", "label": "Select the column to convert its curency", "order": false, "columntype": "n", "max": 1, "min": 1, "visible": "", "table": "inTable" }, { "id": "convertedCol", "type": "newcolumn", "label": "Specify the column for the converted currency value:", "required": true, "placeholder": "", "hideproperties": false, "readonly": false }, { "id": "outTable", "type": "outputtable", "label": "Select the output table:", "required": true, "placeholder": "", "visible": "" } ] }, { "id": "apiKey", "type": "page", "label": "API Key", "children": [ { "id": "_apiKey", "type": "textfield", "label": "Specify your API key:", "placeholder": "", "required": true, "visible": "" } ] }, { "id": "about", "type": "page", "label": "About", "children": [ { "id": "aboutText", "type": "text", "text": "The \"Currency Conversion\" step uses a proc http call to a currency rate API to get today's exchange rate between two currencies. It requires an API key. Refer to the notes on the Program tab.\n\nVersion: 1.0 (29JUN2023)", "visible": "" } ] } ], "values": { "currencyTable": { "library": "SASHELP", "table": "SAS_CURRENCY" }, "currencyCode": [ { "value": "CURRENCY_CD" } ], "convertFrom": null, "convertTo": null, "inTable": { "library": "", "table": "" }, "currencyCol": [], "convertedCol": { "value": "converted_CurrencyValue", "label": "converted_CurrencyValue", "type": "n", "format": "NLMNYIw.d" }, "outTable": { "library": "", "table": "" }, "_apiKey": "" } }     Note: the "readonly": true and "visible": "" options in the Prompt UI for the currencyTable and currencyCode controls.  If the Hide control at runtime option was selected, then the option would be "visible": false.   Here is the Program code for the currency conversion custom step:   /* Currency Exchange from RapidAPI */ /* For more information and to sign up for an API Key: https://rapidapi.com/fyhao/api/currency-exchange/details */ /* Note: This free API may not support all currencies on the SAS_CURRENCY list */ /* Currency Conversion API endpoint */ %let _APIendpoint=https://currency-exchange.p.rapidapi.com ; /* Make call to API to get current Exchange Rate */ filename _curRate temp ; proc http method=get url="&_APIendpoint/exchange" query= ("from"="&convertFrom" "to"="&convertTo" "q"="1") out=_curRate ; headers "X-Rapidapi-Key"="&_apiKey" "X-Rapidapi-Host"="currency-exchange.p.rapidapi.com" ; run ; /* Write the exchangeRate to a Macro */ data _null_ ; infile _curRate ; input rate ; /* Avoid notes about data type conversions */ call symput('exchangeRate',strip(put(rate,32.12))) ; run ; %put ***Exchange Rate of 1 &convertFrom to &convertTo is &exchangeRate ; /* Calculate the converted rate for the selected column */ data &outTable ; set &inTable ; &convertedCol_name=round((&currencyCol_1_name * &exchangeRate),0.01) ; run ;   I have now decided that I want to edit the custom step and hide the Currency reference table and Currency code column selections from the user.  To do this I edit their respective properties and check the Hide control at runtime option.       When I select Preview the Currency reference table and Currency code column are not displayed, but the Convert currency from and Convert currency to drop-down columns contain the list of CURRENCY_CD as their selections.      I designed my custom step, so now it is time to test it in Stand-alone mode.  Here I am converting the ACTUAL column on the SASHELP.PRDSALE table from USD to EUR.         Next, I test the custom step in a SAS Studio Flow. Here I am converting the ACTUAL column on the SASHELP.PRDSALE table from CAD to AUD.         Summary   You now have the option to hardcode the values for an input table and/or column in your custom step giving you the ability to use reference table information for drop-down list selections in your custom steps. You also have the option to show or hide these hardcoded values to users.  For more information, please refer the documentation.   Stay tuned for my next blog on using this new hardcoded value feature with the new SASDQREF library tables for the Quality Knowledge Base (QKB) metadata.   Find more articles from SAS Global Enablement and Learning here. ... View more

In a previous blog, I reviewed using PROC EEL in SAS Studio powered by SAS Viya.  In this four-part blog series, I cover examples of the three data monitoring rule types (row, group, and set) and an example with multiple rules by using PROC EEL in both SAS code and Custom Steps in SAS Studio. Row-based Data Monitoring Rule using PROC EEL in SAS Code and a Custom Step Group-based Data Monitoring Rule using PROC EEL in SAS Code and a Custom Step Set-based Data Monitoring Rule using PROC EEL in SAS Code and a Custom Step Multiple Row-based Data Monitoring Rules using PROC EEL in SAS Code and a Custom Step (this blog)   Multi-Rule Monitoring using PROC EEL For this demonstration on using PROC EEL for multi-rule I am going to check some columns on a table for missing values.   Below is the Contact List with Email input table, I use for my rule check.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Here is the PROC EEL code that builds the rule table and performs the check for the Address, Zip, Phone, and Email columns for missing values.  Note:  I specified the primary key column for the input table using the inputkey option.     Here are the results when I run this code against the input table.  The output is written to the table called MISSING as specified in the code.     The table lists the primary key (ID) and the rule_name that was violated for that record.  A record could fail more than rule check (e.g., IDs 3 and 6).   Multi-Rule Monitoring in a Custom Step Instead of hard-coding the input table, the primary key column, the columns to check for blanks, and the output table, I decided to create a custom step where the user can select that information.  For more details on creating custom steps refer to the following: SAS Demo - Create a Custom Step in SAS Studio on SAS Viya (YouTube) My First SAS Studio Flow Custom Step | Global Enablement & Learning Blog SAS Documentation on Custom Steps in SAS Studio. Creating the Custom Step On the main page of my custom step, I add controls for the selection of the following: Input table Primary key column Output table.   Note:  The columns for the rule table are determined in the Program code for the custom step.  It will create a rule to check all columns except the primary key column for missing values (blanks).   Below is the JSON for the Prompt UI design for this custom step.   { "showPageContentOnly": true, "pages": [ { "id": "page1", "type": "page", "label": "Multiple Rules - Check all fields for missing values", "children": [ { "id": "inTable", "type": "inputtable", "label": "Select the input table", "required": true, "placeholder": "", "visible": "" }, { "id": "primaryKey", "type": "columnselector", "label": "Select the primay key field", "order": false, "columntype": "a", "max": 1, "min": 1, "visible": "", "table": "inTable" }, { "id": "outTable", "type": "outputtable", "label": "Select the output table", "required": true, "placeholder": "", "visible": "" } ] }, { "id": "page2", "type": "page", "label": "About", "children": [ { "id": "aboutText", "type": "text", "text": "This custom step performs multiple rule checks. It checks every column of the selected input table (except for the primary column) for blank values.\n\n\nVersion 1.0 (12APR2023)", "visible": "" } ] } ], "values": { "inTable": { "library": "", "table": "" }, "primaryKey": [], "outTable": { "library": "", "table": "" } } }   Here is the Program code to perform the row rule check using PROC EEL.  Note: It is similar to the code in my above examples but uses the functions FieldCount and FieldName from the expression language to retrieve the field count and field name values for the columns on the input table.   /* Multiple Row-rules for missing information */ /* Build Rules File to check for missing data from all fields from the input table except the one indicated as the primary key */ proc eel data=&inTable out=ruledata (keep=rule_name rule); submit ; pre: string(255) rule_name string(32767) rule /* Loop thru to get all the fields from the input table */ hidden integer i for i = 1 to FieldCount() begin if FieldName(i)!=getvar("primaryKey_1_name") begin rule_name=FieldName(i) & " MISSING" rule=FieldName(i) &"==''" pushrow() end end main: /* do not return any row info from the input table */ return false endsubmit ; run ; proc eel data=&inTable rules=ruledata out=&outTable inputkey=&primaryKey ; quit ;   Running the Custom Step   When I run the custom step, I use the same input table as my example above and select the primary key column as ID and set the output table.     The RULEDATA table is created with a rule check for MISSING for each column of the input table except the specified primary key column.     My results are the same with the exception that I have now run the check for all columns on the table except the primary key column.     Since I have set this as a custom step where the inputs for the set rule check are generic, I can easily change the table to check for missing values.   I have changed the input table to ORDERS and its primary key field is ORDER_ID.     In this case, the RULEDATA table now has the rule checks based on this input table’s columns.     And the MISSING_CHECK table has its results.     No rows were written to the output table indicating that none of the columns on the specified input table (ORDERS) are missing any values. Summary In conclusion, you can use PROC EEL combined with Custom Steps to perform multi-rule monitoring of your data where the inputs for the rule check are not hard-coded into the program.  For more information on Expression Engine Language coding, please refer to the following: PROC EEL Statement documentation Expression Language: Reference Guide Using PROC EEL video by Arnold Toporowski   Acknowledgements Special thanks to: Mickey Schauf for providing the inspiration to create this blog series Andrew Shakinovsky for taking the time to let me talk through my ideas for this blog series enabling me to confirm that the concept was possible!   Find more articles from SAS Global Enablement and Learning here. ... View more