Introduction   Hi everyone! 👋   In this article, I’ll guide you on how to streamline SAS Visual Analytics (VA) reports by reducing the number of graphs through the use of a Control object. Often, reports include several graphs that are nearly identical, differing only in the variables they display. Instead of cluttering your report with multiple similar graphs, you can use a single graph with a drop-down list for variable selection, creating a cleaner and more interactive experience. For example, let’s consider a dataset with 2015 monthly data on energy price components, including both fixed and variable costs for different periods (peak, mid-peak, and off-peak). This dataset, sourced from Kaggle, contains the following six variables: price_mid_peak_fix price_mid_peak_var price_off_peak_fix price_off_peak_var price_peak_fix price_peak_var Instead of creating six separate time series plots to display the average monthly trends of these variables, a single time series plot with a drop-down list allows users to dynamically choose the variable they want to visualize. This approach simplifies your report while enhancing its interactivity. The video below demonstrates the outcome we’re aiming to achieve:     In the steps below, I’ll guide you through the process of setting this up using a dummy dataset. Let’s dive in!   Note: This procedure is based on this SAS paper. In this article, I provide a detailed explanation of the steps required to implement the same functionality using the latest versions of SAS Viya (the screenshots and procedure are based on release 2024.12).     Step 1: Prepare the Data   First, we’ll prepare the data needed to reproduce the result shown in the video. The price_data_raw table has been loaded into memory using the Import Local Files section of the Import tab in SAS Data Explorer:     The va_dummy_data table has been created, promoted to global scope, and saved as a .sashdat file using the code in the attached Data preparation.sas file. This dataset consists of a single variable, dummy_data, with a number of rows equal to num_rows, and contains values ranging from 1 to num_vars (which is set to 6 in this example). To adapt this procedure to your use case, adjust the value of the num_vars macro variable in the code to generate distinct values equal to the number of variables you want to use in your dynamic plot. Additionally, the code converts the price_date variable in the price_data_raw table from character to date format, enabling its use in the Time Series plot.     Step 2: Set up the report in VA and create a custom category   Now, we’ll create a new report in SAS Visual Analytics and set up the necessary components. Add both the price_data and va_dummy_data tables to the report via the Data pane (the topmost icon on the left pane).     With the va_dummy_data data set selected, create a new custom category by clicking New data item --> Custom category.     Name the custom category Variable Select and set Group by to Distinct values. Next, add new groups until you have a number of groups equal to num_vars (6 in my case): Group 1 to num_vars-1 and the Remaining Values group. Rename the first num_vars-1 groups to distinct variable names and assign a value to each. For the Remaining Values group, change the default name Other to the last price variable name you didn't use yet.     Create a new character parameter called Variable Select Parameter by selecting New data item --> Parameter and set its type to Character. Drag a Drop-down list object to the canvas, flag the Required option and change the title to Select a price variable to display the corresponding Time Series plot. Link the Variable Select Parameter parameter to the drop-down list. This can be done in the Actions pane by selecting the parameter through the drop-down list under Parameter Links.     The parameter allows us to create a connection between the Control object and the dynamic variable, which we'll create in the next step.     Step 3: Create a Calculated Item   Select the price_data data set in the Data pane and click New data item --> Calculated Item. Name it Price Variable and define it using the following code: if(LowerCase('Variable Select Parameter'p) = 'price_mid_peak_fix') return price_mid_peak_fix else if(LowerCase('Variable Select Parameter'p) = 'price_mid_peak_var') return price_mid_peak_var else if(LowerCase('Variable Select Parameter'p) = 'price_off_peak_fix') return price_off_peak_fix else if(LowerCase('Variable Select Parameter'p) = 'price_off_peak_var') return price_off_peak_var else if(LowerCase('Variable Select Parameter'p) = 'price_peak_fix') return price_peak_fix else price_peak_var     The calculated item dynamically takes the values of a variable based on the value of the Variable Select Parameter parameter. Now, let’s create the Time Series plot with a dynamically changing price variable.     Step 4: Add the Time Series Plot to the report   Add a Time series plot object to the canvas below the drop-down list and assign the roles as follows: Time axis: price_date Measure: Price Variable  (Optional) The dynamic variable name is static, i.e. it doesn't change based on the parameter value. Therefore, at the moment, the user only knows which variable is being displayed from the drop-down list selection. To make the displayed variable more explicit, a dynamic title can be defined: Remove the plot title by selecting No title in the Options pane. Add a Text object above the plot, insert the Variable Select Parameter by selecting it from the Data dropdown list on the toolbar, i.e. the rightmost button, and complete the title as desired.       Conclusions   This ends the detailed explanation on how to create a dynamic variable in VA and use it to reduce the number of graphs in a report. I used this approach in a churn prediction project, which you can read about here. Feel free to adapt this method to your specific use case, and comment on this article or reach out to me via email (Mattia.Guglielmelli@sas.com) for any feedback or questions. I hope you found this article helpful, see you next time! 😊 ... View more

Introduction   Hi everyone! 👋   In this article, we’ll explore how to predict customer churn in the energy sector using SAS Viya. Specifically, I worked with data from Kaggle that provides insights into Small and Medium Enterprises (SMEs) served by the PowerCO energy supplier. The goal is to identify the key factors driving customer churn and build predictive models to target retention campaigns at customers with a high likelihood of churning. The project uses two datasets: The first dataset, client_data, includes variables such as client ID, sales channel codes, electricity and gas consumption over the past 12 months, customer tenure, and details on potential product modifications. It also contains variables that help evaluate financial performance, such as forecasted energy consumption, client margins, and more. Of course, it includes the churn variable, indicating whether a client churned within the next three months. This serves as the target variable for training predictive models. The second dataset, price_data, contains details about variable (usage-based) charges and fixed charges. Variable charges depend on energy consumption, while fixed charges reflect infrastructure costs to meet energy demand and handle peak loads. These charges are further categorized into off-peak, peak, and mid-peak periods, reflecting typical demand patterns throughout the day. The project follows these key steps: Data Preparation and Exploratory Data Analysis (EDA): Performed using SAS Studio flows and SAS Visual Analytics to clean, explore and transform the data, uncover insights, and ensure quality for modeling. Predictive Modeling: Training several models in SAS Model Studio, selecting a champion model, and scoring test data. Model Interpretation: Interpreting the champion model using SAS Model Studio interpretability tools, saving the interpretability data, and visualizing it in a Model Interpretability dashboard created in SAS Visual Analytics. Retention Campaign Simulations: Simulating various retention campaign scenarios in SAS Studio and analyzing the overall profits resulting from different discount rates and campaign efficiencies in SAS Visual Analytics. By the end of this journey, you’ll have a comprehensive understanding of how SAS Viya can be used to tackle churn prediction challenges in the energy sector. I hope this article inspires you to experiment with SAS Viya and apply its powerful capabilities to solve similar challenges in your industry!     Step 1: Data Preparation and Exploratory Data Analysis (EDA) in SAS Studio and VA   To start, we’ll import the data into the CAS server using SAS Data Explorer, accessible from the Applications menu by selecting Manage Data. Once Data Explorer is open, navigate to the Import tab and use the Local File Import section to import the two CSV files containing the client_data and price_data tables. These will be imported as CAS tables named client_data_raw and price_data_raw, respectively, and stored in the casuser caslib. This process not only loads the tables into memory but also saves them as sashdat files if the 'Save as in-memory data only' option is deselected, which is the default setting. You can preview the imported tables directly in Data Explorer by selecting them from the In-memory Data (available) or Sources tab.   Figure 1: Example of the Sample Data tab for the client_data_raw in-memory table. From this initial look at the two tables, the following observations can be made: The date variables are incorrectly interpreted as character variables. During the data manipulation phase prior to the exploratory data analysis (EDA), these variables will be converted into SAS date formats to enable proper visualization. Some categorical variables have been anonymized into hashed text strings to protect the privacy of the original data. To create more understandable visualizations, these hashed values will be mapped to human-readable labels, such as 'Channel 1', 'Channel 2,' etc., for the channel_sales variable. To address these issues, I transitioned to SAS Studio (accessible via Develop Code and Flows in the Applications menu) to perform the required data manipulations and create useful tables for the EDA report. These steps were organized into a flow named pre_eda.flw. Flows are one of the primary point-and-click tools in SAS Studio, allowing you to combine prebuilt and custom steps to import various file types, query, transform, and export data efficiently.   The pre_eda flow consists of multiple sub-flows, with each one placed in a different swimlane to control the submission order. It creates and loads into memory the following CAS tables: client_data and price_data: These are the processed versions of the original client_data_raw and price_data_raw tables. The processing involves converting date columns to the SAS date type, removing potential duplicates, mapping categorical variables to more human-readable strings, and performing data quality checks, such as verifying that some numeric variables contain only positive values, ensuring consistency between related columns (e.g., gas consumption and gas availability), checking for logical date order, and validating that gross margins are not smaller than net margins. Figure 2: Swimlanes for the creation of the client_data and price_data tables. client_numeric_summary and price_numeric_summary: These tables contain summary statistics for the numeric variables in the client_data and price_data tables (by default, Cardinality, Minimum, Maximum, Mean, Number of Observations, Number of missing values, and Standard Deviation). client_date_summary and price_date_summary: These tables provide summary statistics for the date variables in the client_data and price_data tables (Minimum and Maximum). Figure 3: Swimlanes for the creation of the client_numeric_summary, price_numeric_summary, client_date_summary and price_date_summary tables. id_obs_summary: A table containing the number of observations and unique IDs for the client_data and price_data tables. Figure 4: Swimlane for the creation of the id_obs_summary table. joined_data: A table resulting from an inner join between client_data and price_data, retaining only churn, channel_sales, and date_activ for the former. Figure 5: Swimlane for the creation of the joined_data table. dummy_table: A table with six rows and a single variable, dummy_variable, containing values ranging from 0 to 5. The purpose of this table will be explained shortly. Figure 6: Swimlanes for the creation of the dummy_table table and termination of the CAS session. These tables were then used in a report in SAS Visual Analytics, accessible through the Explore and Visualize option in the Applications menu, to visualize the data and determine which types of transformations to perform on them before building predictive models for churn prediction. The report consists of four distinct pages:   1. Client EDA. This page analyzes the variables in the client_data table and includes, at the top, the number of unique IDs and two tables showing the summary statistics for numerical and date-type variables. At the bottom left, there is a bar chart for each categorical variable, with the option to filter them based on the value of the churn variable. On the bottom right, there is a correlation matrix to display the degree of correlation between pairs of numeric variables.   Figure 7: Client EDA dashboard. 2. Price EDA. This page analyzes the variables in the price_data table and has a very similar structure to the previous one, except for the Time Series Plot at the bottom left instead of the bar charts. This plot allows you to visualize the monthly trend (obtained by averaging across all IDs) of each price variable, which can be changed through the drop-down list above it. To create this Time Series Plot, it was necessary to use the previously created dummy_table. For more details, refer to this article I wrote, which explains in more depth how to create this plot.   Figure 8: Price EDA dashboard. 3. Joined data EDA. This page leverages the joined_data table to provide insights into customer churn and price trends. At the top, a pie chart illustrates the proportions of churners and non-churners among clients present in both tables. To the top right, a table lists the IDs with fewer than 12 observations in the price_data table, indicating missing price values for some months for these customers. At the bottom, a scrolling container showcases time series plots for all price variables, with average trends displayed separately for churners and non-churners. Additionally, the time series plots can be filtered based on the values of the channel_sales variable, enabling more granular analysis.   Figure 9: Joined data EDA dashboard. 4. Deep-dive on low-cardinality variables. This page focuses on three numeric variables from the client_data table that exhibit a low number of distinct values: nb_prod_act, num_years_antig, and forecast_discount_energy. The scrolling container displays, on the left, the frequency distributions of the original variables converted into categorical form, and on the right, the same distributions with rare categories grouped into a single category. The idea is that this grouping mitigates noise caused by infrequent values, allowing the model to focus on meaningful patterns and improve predictive performance.   Figure 10: Deep-dive on low-cardinality variables dashboard. Based on the findings from the VA report, a dedicated flow (data_preparation.flw) was built in SAS Studio. This flow processes and transforms the data from two source tables (client_data and price_data) and combines them into a single resulting table, which serves as the input for SAS Model Studio to train machine learning models. Apart from the 'Create CAS Session' and 'Terminate CAS Session' swimlanes, the flow consists of three main swimlanes: 1. Client data preparation. This sub-flow starts with the client_data table and produces the client_data_final table through the following transformations: - Removal of infrequent categories in the channel_sales and origin_up variables. - Creation of contract_age_at_last_modif, has_modified, and days_to_end variables based on the available date variables. - Conversion of num_years_antig, forecast_discount_energy, and nb_prod_act into categorical variables, by grouping rare classes.   Figure 11: Swimlanes for the creation of the CAS session and the client_data_final table. 2. Price data preparation. This sub-flow processes the price_data table to create the price_data_final table, involving the following steps: - Imputation of missing price values using the median for the corresponding months. - Computation of summary statistics (minimum, maximum, mean, and maximum difference between consecutive months) to represent price trends.   Figure 12: Swimlane for the creation of the price_data_final table. 3. Joining the 2 prepared tables for Model Building. This sub-flow combines the prepared tables into the prepared_data table through the following steps: - Merging the client_data_final and price_data_final tables. - Removal of numeric variables to eliminate all correlation values greater than 0.9, reducing redundancy and enhancing model interpretability.   Figure 13: Swimlane for the creation of the prepared_data table. Next, we will use the prepared_data table in Model Studio, accessible through the Build Models option in the Applications menu.     Step 2: Predictive Model Building in Model Studio   To begin developing models in Model Studio, a new project was created by clicking on New Project in the upper-right corner of the Projects page and selecting the prepared_data table as the data source. During project creation, clicking the Advanced button allows you to adjust settings such as performing random undersampling or modifying the data partitioning proportions. For example, I opted for an 80/10/10 train/validation/test split to allow the model to train on more data compared to the default settings. Once configured, clicking the Save button finalizes the project creation. After opening the project, the roles of the id and churn variables were updated in the Data tab to Key and Target, respectively. Assigning the Key role to the id variable enables its use in the Model Interpretability dashboard to specify the instances to explain, as detailed later. Three pipelines were then built and executed: Custom pipeline This pipeline was manually constructed from a blank template to compare Logistic Regression, Random Forest, and Gradient Boosting models across two scenarios: Using the prepared_data table directly. Using an enriched version of the table by adding the Feature Machine node between the Data node and the Supervised Learning nodes. This node generates new features to fix common data quality issues, such as high skewness, missing values, outliers, high cardinality, and more. Advanced template for class target with autotuning This pipeline is one of the out-of-the-box templates you can choose from during project creation or when adding a new pipeline to the project. SAS Automatically Generated Pipeline This pipeline was created by selecting the Automatically generate the pipeline option when adding a new pipeline. SAS Viya uses automated machine learning to build the pipeline dynamically, adhering to a user-defined time limit. The resulting pipeline contains the top 5 models selected from a pool of candidates, which can be customized in Advanced Settings. For further details, refer to the documentation on Automated Pipeline Creation. Figure 14: Custom pipeline. Figure 15: Advanced template for class target with autotuning. Figure 16: SAS Automatically Generated Pipeline. After running the pipelines, model performance within each pipeline was compared using the Model Comparison node, while models from all pipelines were compared through the Pipeline Comparison tab. The chosen evaluation metric was the Area Under the Curve (AUC), selected by navigating to Project Settings via the cogwheel in the upper-right corner, then under Rules, setting the Class selection statistic to Area under curve (C statistic). AUC was chosen because it evaluates the ranking quality of probability estimates, which should give high probability to the positive examples (true churners) and small probability to the negatives (non-churners). As AUC is a well-accepted ranking metric for unbalanced datasets, and the goal is to create a reliable scoring list for selecting customers to target in a retention campaign, ranking quality is more critical than predictive accuracy for this use case.   The Pipeline Comparison tab provides a summary of results after running the three pipelines:   Figure 17: Pipeline Comparison tab. As shown, the champion model for each pipeline is displayed. Additionally, you can add challenger models to this tab for comparison. Simply right-click on the desired Supervised Learning node and select Add Challenger Model. For instance, by examining the Model Comparison nodes of each pipeline, I identified that the Random Forest model from the custom pipeline trained on the original data achieved the highest AUC score, excluding the Ensemble models. I chose to add this model as a challenger in the Pipeline Comparison tab:   Figure 18: Pipeline Comparison tab after adding a challenger model. For this use case, I selected the Random Forest model as the final one to score customers based on their churn likelihood. While its performance is comparable to that of the Ensemble models, the Random Forest model also provides native insights into the relative importance of input variables by considering them collectively. This differs from the approach used in Ensemble nodes, where importance is estimated separately using one-level decision trees for each variable. The scored data from the final model was saved to the scored_data.sashdat file and loaded into memory as the scored_data table. This was accomplished by adding a Save Data node as a child of the model node.   Figure 19: Save Data node.   Step 3: Model interpretability dashboard in VA   The Post-training Properties of the selected model node were configured to leverage SAS Viya's interpretability tools, including: Variable Importance Partial Dependence (PD) Plots Individual Conditional Expectation (ICE) Plots Local Interpretable Model-Agnostic Explanations (LIME) Kernel Shapley values (HyperSHAP) These model interpretability techniques are model-agnostic, meaning they can be applied to any model generated by a supervised learning node! I decided to execute the node twice to provide insights into both the top five customers with the highest churn likelihood and the top five with the lowest churn likelihood, utilizing the Specify instances to explain section. Assigning the id variable the Key role during project creation allowed us to specify the instances for this analysis more easily. In the first run, data for Variable Importance, PD and ICE Overlay Plots, LIME Explanations and HyperSHAP values were downloaded from the Results pane. In the second run, the same results were downloaded, except for Variable Importance, as it provides global interpretability rather than local insights and remains unchanged when only the instances for computing local interpretability tools are modified. Finally, this data was used in SAS Visual Analytics (VA) to build a dashboard, offering a comprehensive understanding of the selected model and its predictions.   Figure 20: Model Interpretability dashboard. The dashboard features a bar chart on the left side that displays the relative variable importance. On the right, a scrolling container contains two distinct boxes: one showcasing the top 5 clients with the highest churn likelihood, and the other highlighting the top 5 clients with the lowest churn likelihood. Each box includes the following: Left side: A bar chart visualizing the HyperSHAP values for the ID selected in the drop-down list above. This same drop-down list determines which ID is displayed in the plots on the right. Right side: The PD and ICE Overlay Plot for the top 5 most important variables, along with the LIME Explanations graph. In the overlay plot, the variable to display can be chosen via a drop-down list located above the corresponding plot. For a detailed explanation of how the HyperSHAP values and LIME Explanations bar charts have been configured to sort the bars by their absolute values, check out this article that walks you through the entire process.     Step 4: Retention Campaign Scenarios in SAS Studio and VA   Starting from the scored_data table, I simulated various retention campaign scenarios aimed at incentivizing high-risk customers to remain with the energy supplier by offering targeted discounts. This simulation was conducted on the test set of the scored_data table to evaluate the model's behavior on unseen data. It was executed using the SAS macro %simulate_profits, which calculates profits across various combinations of discount rates, discount efficiencies, and probability thresholds. The profit represents the average annual profit and has been calculated using the margin_net_pow_ele, which denotes the net margin on power subscriptions. This margin is assumed to refer to the average monthly margin and was therefore multiplied by 12 to derive the annual value.    How the Profit is Computed   Profit Without Campaign: The baseline profit, if no retention campaign is conducted, is computed as: Profit Without Campaign = Gains − Loss where: Gains represents the total potential profit if no customers churn, therefore this value assumes a perfect scenario where all customers are retained. Loss represents the total profit lost due to customers who churned. This value accounts for the revenue lost because these customers are no longer contributing profit. This value represents the baseline scenario in which no effort is made to prevent churn, and the business simply absorbs the loss from churned customers. It serves as a reference point for evaluating the effectiveness of any retention campaign. Prevented Loss and Discount Loss: For each scenario, the retention campaign targets all customers with a predicted churn likelihood exceeding the probability threshold. Prevented Loss: The profit gained from customers who were predicted to churn and actually churned, but would've been retained due to the campaign. This is scaled by the discount efficiency. Prevented Loss = Sum of profits from targeted churners x Discount Efficiency Discount Loss: The total cost of offering discounts to targeted high-risk customers, broken down as follows: For non-churners: The profit from customers who were predicted to churn but did not actually churn, multiplied by the discount rate, as they would've been targeted by the campaign. Discount Loss (Non-Churners) = Sum of profits from targeted non-churners x Discount Rate For Churners: The prevented loss scaled by the discount rate. Discount Loss (Churners) = Prevented Loss x Discount Rate The total discount loss is the sum of these two components: Discount Loss = Discount Loss (Non-Churners) + Discount Loss (Churners) Final Profit for a Scenario: For each combination of parameters (discount rate, discount efficiency, and probability threshold), the profit is calculated as: Profit = Gains − Loss + Prevented Loss − Discount Loss A lower probability threshold expands the retention campaign to more customers, leading to a higher Discount Loss. Conversely, a higher probability threshold excludes some potential churners from the discount offer, increasing the risk of losing them and, therefore, the Loss term.     Simulation Analysis and Visualization   The SAS macro evaluates all possible combinations of parameters to generate a comprehensive profit simulation. The results include: Worst Profit Table: Contains the baseline profit assuming no retention campaign is implemented. Profit Scenarios Table: Stores the profits for all combinations of discount rates, discount efficiencies, and probability thresholds. Max Profit Table: Identifies the optimal probability threshold for each scenario, maximizing the profit. These tables are integrated into a VA dashboard, which allows for interactive exploration and comparison of various retention campaign scenarios. The parameters for these scenarios can be configured using drop-down lists. The dashboard provides insights into: How profit varies for specific discount rates and efficiencies as the probability threshold changes. The impact of retention campaigns on profit compared to the baseline, both in absolute and percentage terms. The customers who should be targeted in the retention campaign for a given scenario.   Figure 21: Retention campaign scenarios dashboard. If we had an ideal model with a probability threshold p* that perfectly discriminates between churners and non-churners, the profit would increase until p* and then decrease until reaching an asymptotic value corresponding to the no-retention-campaign scenario. In a real-world scenario, such an ideal p* does not exist, as there is some overlap between the two client classes. Consequently, increasing the probability threshold p by 0.05 (i.e., the step size I used) could lead to two extreme cases: All clients with churn likelihood between p and p + 0.05 are churners: In this case, these clients are no longer classified as churners, resulting in a decrease in overall profit. All clients with churn likelihood between p and p + 0.05 are non-churners: Here, these clients are already non-churners, so avoiding targeting them with a discount increases overall profit. These dynamics explain the fluctuations observed in the profit curve shown in Figure 21. By leveraging this simulation and visualization, the analysis highlights the most profitable strategies for retaining high-risk customers, helping the energy supplier optimize its retention campaigns effectively. Below are some final comments on the retention campaign scenarios: The efficiency of the retention campaign may, of course, depend on competitors' tariffs, which in turn can vary based on the client's location. Therefore, it could be beneficial to incorporate this information into the model-building phase, when possible. The retention campaign has its own costs, which should be considered when computing the overall profit and determining whether it's profitable to run.     Conclusion   Thank you everyone for reading this article on churn prediction and prevention in the energy sector using the SAS Viya platform, I hope you found it useful! 😊 Feel free to comment on this article or reach out to me via email (Mattia.Guglielmelli@sas.com) for any feedback or questions, and I encourage you to experiment with SAS Viya to explore how its powerful tools can help solve similar challenges in your own industry! ... View more

Introduction   Hi everyone! 👋   In one of the recent projects I worked on, I developed a Model Interpretability dashboard to better understand how a machine learning model makes predictions and scores data. Among the various visualizations in the dashboard, one graph displayed the HyperSHAP values for a selected client, which could be chosen via a drop-down list. HyperSHAP values indicate how much each feature contributes to a prediction, with contributions being either positive or negative. For more information on Shapley values, refer to the references below. One challenge I faced was sorting the bars in this graph by their absolute values rather than their raw values. In this article, @pathew and I will guide you through the steps to create a custom bar chart in SAS Visual Analytics. This chart will display HyperSHAP values (whether positive or negative) and order the bars based on their absolute values, using the data provided in the attached HyperSHAP_Values.csv file. The video below demonstrates the outcome we will be achieving by the end of the article:     To achieve this goal, we will leverage the power of SAS Graph Builder to create a custom graph object, which provides flexibility for visualizing your data. Let's get started!     Step 1: Create the Custom Bar Chart   To create a custom bar chart in SAS Graph Builder capable of sorting by a value different from what is displayed, we will employ a technique using two overlaid bar charts. Begin by accessing SAS Graph Builder through the Viya application menu under the Administration group, selecting Build Custom Graphs, and initiating a "New Custom Graph." Drag and drop two Bar Chart elements from the Graph Elements pane onto the canvas.     To display only the desired values (positive or negative) and not absolute values, we will effectively "hide" the second bar chart. Achieve this by selecting the canvas to open the custom graph Options pane. Select "Bar Chart 2" from the drop-down list and set the "Transparency" to 100%.       Note: If you want to change the direction of the bars from vertical to horizontal, change the Direction for one of the two bar charts, and this will automatically change the direction of the remaining bar chart as well. Then, switch to "Discrete Legend" in the drop-down list options, deselect the checkbox for Bar Chart 2, effectively concealing it and its legend.     Examine the Role Definitions for your custom graph in the Roles pane. It will feature one "Shared Role" utilized by both bar charts, along with a separate "Response" role for each. You can refine these role names to better align with SAS Visual Analytics terminology. For instance, change "Shared Role 1" to "Category", "Bar Chart 1 Response" to "Measure", and  "Bar Chart 2 Response" to "Hidden Measure." This hidden measure now allows us to assign actual values and absolute values to an individual chart object without displaying both, providing the desired sorting flexibility. Finally, save the completed custom graph to a readily accessible location within SAS Content, making it available for import into SAS Visual Analytics.   Step 2: Use the custom graph in Visual Analytics   To use the custom graph in VA, we need to import it first. The steps to do that are the following: 1. In the Objects pane, accessible via the left pane, click on the Objects menu on the top-right corner and select Import custom graph.      2. Select the custom graph from the folder where you saved it and click Open. The graph is now ready to be used and is displayed together with the other objects.     Note: The same steps are also explained in the documentation. Therefore, we can now drag the custom bar chart onto the canvas and add data to it. If you use the data attached to the article, the role assignment reads: Measure: Estimate Hidden Measure: absEstimate Category: Variable Since this dataset contains the HyperSHAP values for five different customers, let's now add a Drop-down list to allow the user filter the bar chart based on the desired customer. We assign the Category role to the id variable, flag the Required option in the Options pane, and mark the custom bar chart in the Object Links section of the Actions pane.       Next, right-click on the graph and select absEstimate: Ascending (or Descending) for the Sort option.      If you followed all the steps above, you should now have obtained a report very similar to the one visible in the video attached at the beginning of the article!   Conclusions   This ends the detailed explanation on how to create a custom bar chart in SAS Graph Builder to sort the data by a value different from what is being displayed. The one above is just an example, and you can customize the same procedure based on your data sets and needs! Of course, feel free to comment on this article or reach out to us via email (Mattia.Guglielmelli@sas.com and Paul.Thewalt@sas.com) for any feedback or questions. I hope you found this article helpful, see you next time! 😊   References   For more information on Shapley values, check out this article that also gives an overview of all the approximation methods that are available in SAS Viya to estimate them. To know which are the model interpretability tools that can be applied to any model that is generated by a Supervised Learning node in SAS Model Studio, see here for an overview.    If you want to know more on SAS Graph Builder, start by reading its documentation here. ... View more

The field of time series forecasting has traditionally been dominated by statistical and machine learning models, but recent advancements in deep learning, particularly the development of foundation models (FMs), are beginning to revolutionize this domain. Foundation models, also known as large pre-trained models, are deep learning models trained on vast and diverse datasets, enabling them to acquire broad general knowledge and patterns across multiple domains. These models have shown remarkable success in fields like computer vision and natural language processing, where their ability to be fine-tuned for specific tasks with relatively small amounts of data has proven highly effective. Inspired by these successes, the concept of Time Series Foundation Models (TSFMs) has emerged as a promising approach for time series analysis. TSFMs leverage the foundation model paradigm to create generalized models capable of understanding and forecasting time series data across various applications. Examples of TSFMs include Amazon Science's Chronos models.   Chronos is a language modeling framework adapted for time series forecasting. It employs a minimalist approach by tokenizing time series values into a fixed vocabulary (through scaling and quantization of real values) and training existing language model architectures on these tokens without any time-series-specific design or features. Despite its simplicity, Chronos has shown to be effective, having comparable and occasionally superior zero-shot performance on unseen datasets, relative to methods that were trained specifically on them. For more details on Chronos's methodology and performance, refer to the paper linked here.   This article explores the application of foundation models for time series forecasting and demonstrates how users can leverage SAS® Visual Analytics (VA) to train and compare these advanced models with traditional forecasting techniques, such as Moving Average, Seasonal Naive, and Exponential Smoothing (ESM).   To enable users to train and compare models directly within the VA dashboard, this solution combines Data-Driven Content and SAS® Jobs. Specifically: The SAS® Job will be used to execute SAS® code and pass parameters to it, via HTTP calls to the SAS® Job Execution Web application. The latter allows to create, manage and execute SAS® Jobs, and can be accessed at: http://<your.host.name>/SASJobExecution/. The Data-Driven Content (DDC) object will be used to embed the job into the VA report and transfer information from SAS® Visual Analytics to SAS® Jobs. The executed SAS® code updates the CAS table used in the dashboard, allowing users to visualize different forecasted values in a single time series plot and select the best model. This solution supports datasets with: 1 date variable 1 BY variable 1 dependent variable This means your original dataset can include additional variables, but only these three—date, BY, and dependent variable—will be utilized by the SAS® code and VA dashboard. However, the solution can be expanded in future versions to include multiple BY variables or independent variables.   This article is divided into two sections: The first section explains the HTML form and SAS® code behind the job to clarify the logic. The second section guides you through deploying this project for your use.   Note: The SAS® Job's code accesses the Chronos model by calling an API endpoint where it is deployed. If you want to learn how to achieve this within your SAS® Viya installation, check out the first of a two-part series of articles on incorporating pre-trained forecasting models into SAS® Visual Forecasting. Project Description For clarity, this section utilizes a specific dataset, such as the PRICEDATA table from the SASHELP library, though you can easily adapt the steps to any dataset of your choice.   Before diving into the code, let's first examine the final outcome, i.e. the VA dashboard:   Figure 1 - ModelComparison report   The dashboard comprises four different objects: Drop-down List: Filters the other dashboard objects, allowing focus on a specific subset of data. Time Series Plot: Displays both historical data (in blue) and forecasted values from various models. List: Enables users to toggle the visibility of forecasted values for each model. Data-Driven Content (DDC): Allows users to select which model to train based on the subset of data defined by the drop-down list. If no selection is made, the model is trained on the entire dataset. This implies that different models can be trained for different values of the BY variable. Behind the scenes, the CAS table is updated with the forecasted values after each model is trained. To ensure the time series plot and list objects reflect these updates, automatic refresh is enabled; otherwise, these objects won't refresh as the data changes with the job execution.   Note: Automatic refresh only works when viewing the report, as explained in the documentation.   Data Pre-Processing As mentioned, the forecasted values are plotted alongside historical data in the time series plot. To achieve this without altering the table structure (such as adding a new column for the forecasted values of the selected model), the PRICEDATA table (or your chosen dataset) undergoes pre-processing. This pre-processing is handled by the code in the following two screenshots (from the preprocessing.sas file). The red boxes highlight all the macro variables you can modify to adapt the code to your dataset, which are: originaltab, which stores the name of the original table in the format libname.tablename. castab, which stores the name of the final table in the format libname.tablename. target, which stores the name of the dependent variable to forecast. date, which stores the name of the date variable. byvar, which stores the name of the BY variable. Figure 2 - SAS pre-processing code: part 1 Figure 3 - SAS pre-processing code: part 2   The pre-processing is performed in the PROC CAS procedure. Here’s a step-by-step explanation of what the code does: A dictionary casTbl is created, storing CAS library and CAS table names. A SOURCE block is used to embed an SQL query in a variable named long_table. This query creates a new table (&castab) by selecting and summing the variable &target grouped by &byvar and &date from a temporary table &castab._tmp. The SQL query is executed in the CAS server using the execDirect CAS action. The just created table is reshaped from wide format to long format using the wideToLong action from the dataShaping CAS action set. The final table contains the columns specified in the id parameter and three other columns: _Variable_, whose values are variable names in the input table. _Value_, whose values are values in the input table. _C0_, which stores the character component of the variable names. In this case, it coincides with _Variable_ column, since all the original column names don’t contain numeric values. See here for more information. The _C0_ variable is dropped from the table, while the _Variable_ and _Value_ columns  are renamed to variableName and variableValue respectively, by means of the alterTable CAS action. Finally, the &castab table is promoted to a globally accessible table in the CAS environment, so that it can be used in Visual Analytics. Keeping the variables in the specified order (retain statement) and setting the length of the variableName to 23 characters (length statement) is important not to have problems in the following, when we will append the forecasted values to this table. As an example, the following image illustrates the PRICEDATA table before and after pre-processing:   Figure 4 - PRICEDATA table before and after pre-processing   With the dataset now prepared, we can begin describing the SAS® Job.   SAS® Job Code The SAS® code is composed of several sections: Initialization. This section sets up the environment and defines necessary macros and options: Figure 5 - SAS Job code: initialization In particular: The sendApplicationMsg macro is defined, which sends a message back to the client in JSON format, indicating the success or failure of the operation. If the process fails (success=false), it closes the CAS session and aborts the job. A CAS connection is established and the &caslib library is assigned. The checkParams macro verifies the presence of all required SAS® global macro variables (model, castab, byvariable, target, date, date_interval, byvar). These variables are automatically generated from the job input parameters included in the HTTP request sent by the HTML form (as explained in the next section). If any variables are missing, the macro triggers an error message and terminates the process. Main Processing. This section handles the core processing based on the value of the byvariable and model parameters. Data Preparation. Depending on whether byvariable is provided, the code either groups by date alone or by both byvar and date. It creates a new CAS table (&castab._aggregate) with the summed variableValue. Figure 6 - SAS Job code: data preparation   Model-Specific Forecasting. The code then applies different forecasting models based on the model parameter. Each block uses the PROC TSMODEL procedure to run a time series model on the data and generate forecasts. As an example, the following screenshots illustrate the code used to retrieve forecasted values from the Chronos model: Figure 7 - SAS Job code: model-specific forecasting (Chronos example), part 1 Figure 8 - SAS Job code: model-specific forecasting (Chronos example), part 2 Figure 9 - SAS Job code: model-specific forecasting (Chronos example), part 3   For a detailed explanation of the code and step-by-step instructions on building a node for the Chronos model in SAS® Visual Forecasting, refer to the second article in the two-part series mentioned in the introduction. Note: Despite its name, the Seasonal Naive model simplifies to a standard Naive model in the absence of seasonality, which occurs when the default length of the seasonal cycle associated with the &date_interval is 1 (for instance, when &date_interval=year) or when SEASONALITY=1 is explicitly set in the PROC TSMODEL statement. Post-Processing. After generating the forecasts, the code reshapes and formats the output table by dropping unnecessary columns and converting it from wide to long format. The columns are then renamed and formatted as needed before appending the table’s rows to the final dataset. Figure 10 - SAS Job code: post-processing Termination. Finally, the CAS session is terminated, and a final success message is sent back in JSON format using the sendApplicationMsg macro.  Figure 11 - SAS Job code: termination   HTML Form Let's examine the HTML form, beginning with the head section.   Figure 12 - HTML form: head section   The JavaScript code embedded in the HTML (described below) leverages some functions from two modules defined in the GitHub project called sas-visualanalytics-thirdpartyvisualizations. For this reason, they have been uploaded to the Files service and included in the HTML form using the SAS® Job Execution Web Application with a SCRIPT tag (as explained in the documentation). In the final section of the article, the complete procedure to do that will be provided.   The body section defines: A heading titled “Select a Model to Train”; Four buttons, each representing a different forecasting model: Moving Average Seasonal Naive Exponential Smoothing Model (ESM) Chronos Initially, all buttons are disabled and are only enabled once the data is received from the VA dashboard. A div element with ID “JobResults” to display the outcome of the job execution. It will show messages like "Waiting for job to finish...", "Done!", or error messages depending on the job's success or failure. Figure 13 - HTML form: body section   Next, let's examine the script section. The basic structure is shown in the screenshot below: Figure 14 - HTML form: script section   The script begins by defining global variables. Variables with an underscore prefix store parameters received from the VA dashboard, while the model parameter holds the name of the selected model when a form button is clicked. The script then listens for the DOMContentLoaded event to ensure the DOM is fully loaded before executing the code. This is a best practice when working with JavaScript. When the event occurs, the onDataReceived callback function is triggered, which processes the data received from the VA dashboard. Specifically, it verifies if: At least one column and one row are passed to the DDC object. All necessary parameters are defined in the VA dashboard. If both conditions are met, the model selection buttons are enabled. When a model button is clicked, the trainModel function is called with the model name as an argument. This function initiates a job via the callJob function and displays the job’s execution status in the jobResults section. The callJob function prepares a form data object with necessary parameters, including the selected model and other VA-specific parameters, sending it via an HTTP call. The job is called with the _action parameter set to json. This results in the job response being a list in JSON format of output files created by the job. From the list, the files’ content can be retrieved using the URIs specified in the HREF keys (for more details, see here). The response is then processed to display a success or error message in the jobResults section: If unexpected execution errors occur, the error information is returned in a JSON structure with the following fields (according to the documentation)​: Figure 15 - Example of JSON response when there is an error in the SAS code In this case, an error message is displayed in the DDC object, prompting the user to check the browser’s developer console, where the request response and the SAS® log are displayed to identify the lines in the SAS® code causing the error. The SAS® log is retrieved and processed by the obtainSASLogLines function to extract only log lines. If no execution errors are found, the _appout.json file is inspected using the obtainAppExecutionInfo function to ensure no application-level errors exist (e.g., all input parameters are present, and the CAS table exists). If both checks pass, a “Done!” message is displayed in the DDC object for 2 seconds before disappearing.   How to deploy this project Prerequisites: As previously mentioned, the Chronos model should be deployed on the same Kubernetes Cluster as your SAS® Viya installation. For a detailed step-by-step procedure, please refer to this article.  For running the Chronos model with PROC TSMODEL, the EXTLANG package should be installed and configured, ensuring that numpy and requests are included in the setup. Deployment steps: Download the zip file attached to the article. Additionally, the same files are available for download from this GitHub repository, under the folder named ModelComparison. Open the SAS® Job Execution web application by navigating to http://<your.host.name>/SASJobExecution/. Create two new JavaScript files in a folder of your choice by using the New File option and selecting the appropriate File type. The steps are shown in the screenshot below. Figure 16 - Creating a JavaScript file in SAS Job Execution Web application   Name the files as follows and paste the corresponding content from the GitHub repository’s util folder: contentUtil.js messagingUtil.js This ensures the files are correctly loaded as JavaScript files. They will be included in the HTML form, as already seen in the corresponding section. Create a new job in your preferred folder and give it a name. Follow a similar procedure as in the previous step. Figure 17 - Creating a SAS Job in SAS Job Execution Web application Right-click on the job, select Properties, then navigate to Parameters and add a new parameter with the following details: Name: _action Default value: FORM Leave all other settings as default.   Setting _action to FORM (case-insensitive) causes the SAS® Job Execution Web application to transfer execution to the HTML form, which then calls the SAS® code stored in the job (with _action=JSON). Save the changes made to the job. Right-click on the job and choose Edit > Source code. Copy and paste the contents of the ModelComparison.sas file. Replace the CHRONOS_ENDPOINT value with your specific Chronos endpoint: Figure 18 - Code line for Chronos endpoint to update   Save the job after editing the source code. Right-click on the job, select Edit > New HTML form, and copy-paste the content from the ModelComparison.html file. Update the path of the JavaScript files. To get them, right-click on the files and select Properties. Here is an example for the contentUtil.js file: Figure 19 - Obtaining JavaScript files' full path   Replace the path after the equal sign in each of the following lines with the paths to your own files: Figure 20 - Modifying src with your JavaScript files' full path Pre-process your input dataset using the preprocessing.sas file. If needed, customize the values of the five macro variables (originaltab, castab, target, date, byvar) in the first few lines of the code before running it. Note: Be sure that all the variables types and formats are correct in your input dataset. For example, verify that your date variable has a date format, otherwise it will be considered by Visual Analytics as a numeric variable. Go to http://<your.host.name>/SASVisualAnalytics/ and log in. Create a new empty report in SAS® Visual Analytics. With the report open, press Ctrl+Alt+B to open the SAS® Visual Analytics Diagnostics window. Replace the existing XML content with the content from the ModelComparison.xml file. Click Load to apply the XML content. First off, add your pre-processed CAS table in the Data pane on the left-hand side: Figure 21 - Adding data to the VA dashboard In the Parameter section of the Data pane on the left-hand side, modify the following parameters: _job_cas_table: Set this to match the value of the castab macro variable in preprocessing.sas. _target: Set this to match the value of the target macro variable in preprocessing.sas. _date: Set this to match the value of the date macro variable in preprocessing.sas. _byvar: Set this to match the value of the byvar macro variable in preprocessing.sas. _date_interval: Specify the frequency of the accumulated time series (e.g., year, month, quarter, week). Refer to the documentation for a summary of interval types for SAS® date values. _end_historical_data: Use this to display only the forecasted values for the forecast horizon. Set it to the last date value for which you have data. Note: The parameters _by_variable and _variable_name_parameter do not need modification, as they are automatically set by the Drop-down list control and the List control objects, respectively. Assign the following Data Roles, Display Rules and Filters to the report objects: List.  Category: Variable name. Filters. Click New filter > Advanced filter and copy-paste the following expression: 'Variable name'n = UpCase(_target) OR (IF _by_variable = '.' RETURN StartsWith('Variable name'n, 'forecasted') AND 'Sales Region'n = '.' ELSE StartsWith('Variable name'n, 'forecasted') AND 'Sales Region'n = _by_variable) Substitute 'Sales Region'n with your BY variable both at the end of line 3 and 4. Options: Replace the Initial value for the control with the name of your target variable in uppercase: Figure 22 - Changing required initial value for List control object   Time Series Plot. Time axis: your date variable. Measure: remove Frequency and add Variable value. Group: Variable Name.  Display Rules: Figure 23 - Display rules for the Time Series plot Compared to the screenshot, you only need to substitute SALE with your target variable's name in upper case, which should be actually prompted to you after writing the first letter. Of course, the color choice can be modified based on your taste. Filters. Click New filter > Advanced filter and copy-paste the following expression: 'Variable name'n = UpCase(_target) OR (IF _by_variable = '.' RETURN StartsWith('Variable name'n, 'forecasted') AND In('Variable name'n, _variable_name_parameter) AND 'Sales Region'n = '.' AND 'Order Date'n > _end_historical_data ELSE StartsWith('Variable name'n, 'forecasted') AND In('Variable name'n, _variable_name_parameter) AND 'Sales Region'n = _by_variable AND 'Order Date'n > _end_historical_data) Also in this case, replace 'Sales Region'n with your BY variable at the beginning of lines 5 and 9, and 'Order Date'n with your date variable at the beginning of lines 6 and 10. Data-driven Content. Variables: Variable name. Filters. Click New filter > Advanced filter and copy-paste the following expression: (_job_cas_table = _job_cas_table) AND (_by_variable = _by_variable) AND (_target = _target) AND (_date = _date) AND (_byvar = _byvar) AND (_date_interval = _date_interval)​ The reason of this dummy advanced filter (that always returns true) is that VA parameters are only passed in the JSON message to the DDC if the parameters affect the data that is being passed to the DDC (as explained in this article).  Drop-down List. Category: your BY variable. Filters. Click New filter > Advanced filter and copy-paste the following expression: ('Sales Region'n) <> ('.')​ Substitute 'Sales Region'n with your BY variable. Select the DDC object in the report, then go to the Options pane on the right. Replace the URL with the one found in the job’s Properties > Details pane: Figure 24 - Obtaining SAS Job URL Save the report after making all changes.   Conclusion In conclusion, the integration of foundation models into the realm of time series forecasting marks a significant advancement in the field. As foundation models continue to evolve, their application in time series forecasting will likely expand, offering even more effective solutions in the future. With the DDC object described in this article, you can easily train and compare Time Series Foundation Models (TSFMs) like Chronos with traditional forecasting techniques using a VA dashboard. This allows you to evaluate their performance on your specific datasets. The models shown here are just examples, and you can customize the solution to include your preferred models and additional variables. Feel free to use this article as a starting point to create your own tailored version! 😊   Acknowledgements I would like to express my sincere gratitude to Arpit Jain and David Weik for the opportunity to work on this project and for our discussions and idea exchanges, which led to the final outcome. I also want to thank Iman Vasheghani Farahani for his assistance with questions regarding the TSMODEL procedure, as well as Michael Carman and Renato Luppi for their guidance in incorporating JavaScript files into the HTML form.   References To start understanding how to use the DDC Object in Visual Analytics, the four-part series of articles beginning with this one is really enlightening! The series of articles starting with this one, along with this one, inspired me to create the basic structure for my SAS® Job. For more information on how to deploy image, CSS and JavaScript files in the SAS® Content Server and add them to the HTML form, check out this article. I found this article really helpful for understanding how to build custom models in PROC TSMODEL. Another valuable resource is this one, which outlines all the steps to fit a model using the Time Series Model (TSM) Package in the TSMODEL Procedure. For more information on CAS Actions, check out CAS Action! - a series on fundamentals - SAS Viya Programming. ... View more