Variable selection is an important data preprocessing task that improves model performance by removing irrelevant and/or redundant inputs, enhancing accuracy, and minimizing computational complexity. Variable selection can be supervised (using the target variable) or unsupervised (ignoring the target). SAS Viya supports both, with supervised methods reducing irrelevant inputs and unsupervised methods removing redundant ones. Here, we highlight a specific unsupervised method known as variance-based unsupervised variable selection. Another approach is variable clustering. To learn more about clustering, check out the SAS tutorial titled "Feature Selection Using Graphical Lasso" on YouTube.   Unsupervised Variable Selection   In SAS Viya, the unsupervised variable selection method can be implemented programmatically using the VARREDUCE procedure or the UNSUPER action, and interactively through the Variable Selection node in Model Studio pipelines.   The unsupervised variable selection method selects a set of variables that can jointly explain the maximum amount of variance in the input space. Variance is represented using one of three matrices: the Pearson correlation matrix, the covariance matrix, or the sums of squares and cross-products (SSCP) matrix.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   This method reduces dimensionality by forward selection of the variables and the output lists the variables in order of their contribution to data variance. In forward selection, the process starts with an empty set of variables and iteratively adds variables that contribute the most to the overall variance. You can specify the fraction of the total variance to be explained or the minimum increment of explained variance or both.   Forward selection adds inputs one at a time by computing the variance explained by each variable individually and selecting the one that maximizes the increase in variance explained at each step, if it is under a predetermined threshold value, called the entry cutoff. The process continues until a predetermined stopping criterion is met, such as reaching a specified number of selected variables or when adding additional variables no longer significantly increases the overall variance explained. Adding terms in this nested fashion always selects the most important variables and reduces redundancy. This method is particularly beneficial in applications where retaining the original variables is important for model exploration and interpretation.   This all sounds interesting, but how is it actually done in sequential forward selection?   An Optimization Algorithm   Unsupervised variable selection can be successfully approached as an optimization problem by means of global optimization heuristics if an appropriate objective function is considered.   We will use the getStarted dataset from the SAS Viya documentation containing 100 observations and 12 variables. The variables include interval inputs generically named X1 through X10, a binary target variable Y, and a categorical input variable C. The binary target variable Y is only needed for supervised variable selection. Thus, our input space X consists of 100 observations and 11 variables, one of which is categorical. Categorical variables are dummy coded using the GLM method of parameterization. Our objective is to maximize the explained variance in this input space, essentially making it an optimization algorithm.             This input space can be thought of being divided into two parts: the variables selected by the unsupervised method and those that were not. Suppose S contains the five selected variables (X3, X5, X7, X10 and C), while U contains the remaining six variables (X1, X2, X4, X6, X8 and X9). The problem is then approached as a minimization rather than a maximization problem by utilizing the concept of null space.     The null space of a matrix signifies the set of all vectors that, when multiplied by the matrix, result in the zero-vector. The idea of null space helps identify relationships or dependencies between variables. If there are non-zero vectors in the null space, it means some variables can be expressed in terms of others. And so, the unselected variables matrix U is projected to the null space of the selected variables matrix S to determine if the variables in U have any impact on the variations in the data.   The data variance that resides in the null space of the selected variables is measured. Since U is projected in the null space of S, this data variance is equivalent of what cannot be explained by the variables in the selected set and thus need to be minimized. This process helps select variables that jointly explain the maximum amount of the variance in the original data. Minimizing unexplained variance is equivalent to maximizing explained variance.   Implementation in Model Studio   The Variable Selection node in Model Studio includes variance-based unsupervised variable selection as one of its many available methods. A simple pipeline has been created with the Data node connected to a Variable Selection node.   In the Variable Selection node, only the Unsupervised Selection slider is enabled. The maximum number of variables to be selected is limited to 5. The Correlation Statistics field is set to its default option, Correlation, while alternative options include Covariance and Sum of Squares and Crossproducts. The cumulative variance cutoff is set to 0.9, and the incremental variance cutoff remains at its default value.   After running the Variable Selection node, Cumulative Variance Explained window displays a bar chart giving the proportion of variance explained as each variable is considered. The parameters are ordered by decreasing incremental variance.                             The Output window displays the SAS output of the variable selection run including the selection summary. The Selection Summary table displays for each iteration the name of the selected effect, the name of the selected level, and the total variance explained after the iteration.     The Selected Variables table (not shown here) summarizes which variables were selected in the selection process. It also provides information about the variable type of each selected variable. Five variables were selected: X3, X7, X10, C and X5.   You can also implement this programmatically. To learn how to perform unsupervised variable reduction using the VARREDUCE procedure in SAS Viya, watch this video: "Unsupervised Variable Reduction Using the VARREDUCE Procedure in SAS Viya" on sas.com.   Concluding Remarks   Variable selection is the obvious way to thwart the curse of dimensionality. Unfortunately, reducing the dimension is also an easy way to disregard important information. In predictive modeling, relying solely on unsupervised variable selection can result in the inclusion of less relevant predictors. This is because unsupervised variable selection focuses solely on eliminating redundant inputs rather than removing irrelevant ones. To overcome this limitation, it is recommended to integrate both supervised and unsupervised methods for variable selection. That said, if the goal is purely to remove redundant inputs, such as in cluster analysis, using only unsupervised variable selection is particularly important.     Find more articles from SAS Global Enablement and Learning here. ... View more

I am excited to talk about one of the grandest events underway on our planet Earth, the Maha Kumbh Mela (festival of the sacred pitcher), where spirituality is converging with science and technology. Look at some staggering numbers displayed in the exhibit related to this festival!   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The Kumbh Mela series draws spiritual seekers from all over the world, including Hollywood stars, global thought leaders and influencers. Yoga enthusiasts, researchers of Indian spirituality and pilgrims not only from Indian subcontinent but from the Americas, Europe, the Indo-Pacific and West Asia are also drawn to it. Clearly, the Kumbh Mela has evolved into a global pilgrimage, and this year it is even grander due to unique planetary alignments, earning the name Maha Kumbh Mela. In Sanskrit, "Maha" means great or big, "Kumbh" refers to a pitcher or pot, and "Mela" signifies a fair or fest.     The Kumbh Mela is so massive that it is visible from space. The first high resolution shot with stunning details was captured by Space Imaging’s IKONOS satellite in 2001. Indian space agency ISRO released photographs of the Kumbh Mela in 2019.     UNESCO named Kumbh Mela as an Intangible Cultural Heritage of Humanity in 2017. The U.N. agency says that the event encapsulates the science of astronomy, astrology, spirituality, ritualistic traditions, and social and cultural customs and practices, making it extremely rich in knowledge.     AI-powered technology takes center stage at Maha Kumbh 2025 in Prayagraj, Uttar Pradesh, India redefining how devotees engage with their faith and cultural heritage during the 45-day event (January 13–February 26, 2025). The mega event is poised to leverage cutting-edge AI technologies to manage millions of pilgrims while ensuring safety, efficiency, and sustainability.  This blend of tradition and technology sets a new benchmark for managing gigantic public gatherings.   If you're new to AI, consider reading SAS' article, "Artificial Intelligence: What it is and Why it Matters," for an introduction. To learn about SAS Viya—a modern, scalable platform for AI, analytics, and data management—check out the "AI and the Analytics Life Cycle" video.     Here’s a list of key application areas and how AI is transforming this mega event:   Advanced Crowd Management and Monitoring Real-Time Surveillance and Crowd Control   Managing millions requires precision. Over 2,700 AI-powered cameras, integrated with IoT devices, monitor crowd density and movement in real-time. Computer vision identifies overcrowding and unusual behavior, enabling authorities to take proactive measures. At the heart of operations is the Integrated Command & Control Center (ICCC), a nerve center overseeing real-time activities. The ICCC consolidates data from drones, CCTV feeds, and IoT devices to detect congestion and optimize decongestion measures efficiently.   Explore the SAS Demo: Example Use Case for SAS Computer Vision Capabilities, which highlights a practical example of how SAS Computer Vision can be applied.   Predictive and Optimization Models for Safety   AI models analyze historical data and real-time inputs to forecast crowd flow, peak attendance periods. AI algorithms identify high-risk zones and provide early warnings for potential bottlenecks and stampedes. GPS-enabled systems track real-time crowd movement, allowing authorities to deploy resources efficiently using mathematical optimization models. Over 2.5 terabytes of data generated daily provide insights into crowd behavior and preferences for future planning.   To learn how to train and deploy predictive models using Model Studio pipelines, watch the "Build Models with SAS Model Studio" video. Additionally, explore the "Monitor Assignment for Students with Disabilities Using SAS Optimization: Boston Public Schools" project, a part of SAS' Data4Good initiative.     Smart Transportation and Parking Solutions AI-Powered Traffic Optimization   Machine Learning models predict traffic flow and AI-powered GPS systems suggest optimal routes based on real-time traffic conditions. Video analytics systems enhance traffic management by detecting wrong way driving or barricade jumping, monitoring entry/exit points, and ensuring smooth vehicular movement.   AI-Integrated Parking Solutions   Automatic Number Plate Recognition (ANPR) systems streamline entry and exit for over 550,000 vehicles across 30+ parking zones. Integration with FASTag, an electronic toll collection system, enables cashless transactions, reducing congestion and wait times. AI predicts parking demand and monitors fill rates for optimized operations.   To learn more about how analytics combined with new IoT data is transforming cities smarter, watch "Analytics Powers the Internet of Things (IoT) for Smart Cities" on the SAS video portal.     Enhanced Public Safety and Security Facial Recognition and Biometric Systems   AI-enabled facial recognition identifies missing persons and reunite lost individuals, particularly children and the elderly, with their families quickly. For instance, over 250 individuals were separated from their families in the dense crowd during the early hours of the first day of Maha Kumbh. Remarkably, civil defense teams successfully reunited all of them with their families within just 90 minutes.   Biometric identification, integrated with Aadhaar, a twelve-digit unique identity number for Indian residents just like SSN in the US, ensures robust identity verification and enhances security. The digital 'Khoya Paaya Kendra', that is, lost and found centers enable quick registration and identification using facial recognition and biometric technology.   Discover how a face patch is converted into a feature vector in "Face Recognition Using SAS Viya." Also, watch SAS' blog, "Facial Recognition: Monetizing Faces in the Crowd," to learn how facial recognition systems go beyond identifying individuals, offering applications in marketing and demographic analysis.   Aerial and Underwater Surveillance   Drones equipped with high-resolution cameras monitor congestion and provide night-vision support. Underwater drones track activities below the surface in real-time, ensuring safety during ceremonial bathing.   Cybersecurity Measures   A dedicated team of 56 cyber experts monitors online threats and educates pilgrims about safe digital practices via Variable Messaging Displays (VMDs). AI tools identify and neutralize suspicious websites and online scams across platforms like Facebook and X.   To begin your journey in cybersecurity, read "Cybersecurity Analytics and the Vital Arterial Pulse Wave That Flows Through Your Network" and "The Latest Approach to Cyber Defense: Cyber Analytics" on the SAS website.     Pilgrim-Centric Digital Solutions Android and IOS Mobile App   The official mobile app 'Maha Kumbh Mela 2025' provides navigation, emergency contacts, live updates, and a Lost & Found section. Multilingual support caters to international and regional visitors.   RFID Wristbands and AI Chatbots   Pilgrims are provided with RFID wristbands that, with their consent, track their movement including entry and exit times, ensuring safety and assisting in crowd regulation. The AI chatbot 'Kumbh Sah’AI’yak' offers real-time assistance in 11 languages, providing event information and navigation help.   Learn more about chatbots and intelligent assistants by reading the article "What are Chatbots and How Can You Combine Them with Analytics?" on the SAS website. Additionally, explore potential future applications of intelligent assistants in SAS Viya in the second half of the video "Sharing Reports, Generative AI Intelligent Assistants | SAS Viya December 2023 Release."     Healthcare and Emergency Response AI-Enhanced Medical Services   Hospitals and mobile health units equipped with AI cameras and microphones monitor patients in real time and alert staff during emergencies. Wearables track health metrics of volunteers and paramedical staff and predict hotspots for potential health issues like drowning, suffocation, cold stroke etc.   For related information, visit the "Health Care Analytics" page on the SAS website. Also, watch the customer success story, "Duke Health: Advancing Health Care with Data and AI," to see how innovative AI-powered tools are being developed to enhance health care delivery processes.   Real-Time Translation Services   AI translation systems bridge language gaps, enabling seamless communication between doctors and patients in regional and international languages for better accessibility.   Digital Twin for Disaster Preparedness   A virtual replica of the mela grounds simulates crowd overflow scenarios and helps plan evacuation strategies for emergencies, ensuring safety for all attendees. AI-driven scenario analysis enhances disaster preparedness.   Learn more about digital twins by reading the SAS post, "Unleashing the Power of Generative AI: Transforming Digital Twins into Intelligent Twins."     Environmental Sustainability Smart Waste Management   IoT-enabled bins and AI analytics optimize waste collection schedules, ensuring cleaner mela grounds. Real-time monitoring systems track waste disposal rates and promote eco-friendly practices.   Water Quality and Resource Monitoring   AI sensors continuously monitor the Ganges' water quality, tracking pollutants and suggesting remediation measures. This ensures water remains safe for holy dips and other rituals. Solar-powered energy stations and AI-driven insights into energy and water usage reduce the event’s environmental footprint. Analytics optimize resource consumption, reducing the mela’s carbon footprint.   As part of its corporate social responsibility, SAS is dedicated to achieving net-zero emissions by 2050. Learn more in the article "SAS Propels Corporate Responsibility Through Innovative AI and Sustainability Initiatives."   Concluding Remarks   Maha Kumbh Mela 2025 exemplifies the transformative power of AI in managing one of the world’s largest gatherings. From enhancing safety and efficiency to promoting sustainability, AI-driven solutions are paving the way for a seamless and spiritually enriching experience for millions of devotees.     Find more articles from SAS Global Enablement and Learning here. ... View more

Unsupervised learning methods are a type of machine learning algorithms that work with data that have no historical labels or target variable. The system is not told the "right answer" and the algorithm must figure out what is being shown. The goal of unsupervised learning is to explore the data, uncovering hidden patterns within it and find some structures and relationships in the data.   Unsupervised learning methods are crucial in the field of machine learning, particularly for handling high-dimensional data, and are applied in various areas such as visualization, feature extraction, feature selection, data reduction, anomaly detection, association rules mining and more.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     SAS Viya provides a rich suite of unsupervised machine learning techniques, available in both point-and-click and programming modes. These unsupervised learning methods provide powerful tools for understanding and utilizing high-dimensional data, enabling insights and applications across various domains.   Here's a tabular summary of unsupervised learning methods in SAS Viya:   Application / Task Method / Technique SAS Procedure Model Studio Node CAS Action Product Visualizing High Dimensional Data t-distributed Stochastic Neighbor Embedding PROC TSNE Data Exploration node tSne Action SAS Machine Learning Feature Extraction Principal Component Analysis PROC PCA Feature Extraction node eig Action, itergs Action, nipals Action, randompca Action SAS Visual Statistics Singular Value Decomposition PROC RPCA Feature Extraction node robustpca Action SAS Machine Learning Robust Principal Component Analysis PROC RPCA Feature Extraction node robustpca Action SAS Machine Learning Moving Window Principal Component Analysis PROC MWPCA - - - mwpca Action SAS Machine Learning Autoencoding Neural Networks PROC NNET Feature Extraction node annTrain Action SAS Machine Learning Non-Negative Matrix Factorization PROC NMF - - - nmf Action SAS Visual Statistics Feature Selection Unsupervised Selection Method PROC VARREDUCE Variable Selection node unsuper Action SAS Visual Statistics Variable Clustering PROC GVARCLUS Variable Clustering node gvarcluster Action SAS Machine Learning Data Reduction using Cluster Analysis Partitive Clustering Using k-Means PROC KCLUS Clustering node kClus Action SAS Visual Statistics Model-Based Clustering PROC MBC - - - mbcFit Action SAS Visual Statistics Nonparametric Bayesian Gaussian Mixture Model PROC GMM - - - gmm Action SAS Machine Learning Anomaly Detection Support Vector Data Description PROC SVDD Anomaly Detection node svddTrain Action SAS Machine Learning Isolation Forest PROC FOREST - - - forestTrain Action SAS Visual Analytics Association Rules Mining Market Basket Analysis PROC MBANALYSIS - - - mbanalysis Action SAS Machine Learning   Let's briefly discuss about each one.   Visualizing High Dimensional Data   Unsupervised learning methods are instrumental in reducing the dimensionality of high-dimensional data for visualization, allowing us to interpret complex datasets.   t-distributed Stochastic Neighbour Embedding (t-SNE) is a machine learning algorithm for dimensionality reduction that is particularly well-suited for visualizing high-dimensional data sets, transforming them into two- or three-dimensional points while preserving the pairwise distances between closely neighboring observations and relaxing distances for non-neighboring observations. For further details, check out this post: "Visualizing High-Dimensional Data with t-SNE" on sas.com.   Feature Extraction   Feature extraction involves creating new features from raw data to improve model performance, with unsupervised methods helping to identify and extract important features.   Principal Component Analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components, with the goal of reducing the dimensionality of the data while retaining as much as possible of the variation present in the dataset. To learn how to perform principal component analysis using PROC PCA in SAS Viya, watch this video: "Principal Component Analysis Using the PCA Procedure in SAS Viya" on sas.com. Singular Value Decomposition (SVD) is a matrix decomposition method that breaks down a matrix into three separate matrices, projecting high-dimensional document and term spaces into a lower-dimension space, and is often used in text mining to transform a term-by-document frequency matrix into a data set suitable for data mining purposes. For further details, check out this blog post: "The Singular Value Decomposition: A Fundamental Technique in Multivariate Data Analysis" on sas.com. Robust Principal Component Analysis (RPCA) is a matrix decomposition algorithm that decomposes an input matrix into a low-rank matrix and a sparse matrix, which can be used for feature extraction and anomaly detection, respectively, and is particularly useful in handling data that might be noisy or contain outliers. Moving Window Principal Component Analysis (MWPCA) implements principal component analysis over sliding windows of observations, allowing for the capture of changes in principal components over time and the detection of relative changes in parts of a system compared to the overall system. Autoencoders are types of neural networks used for efficient codings, feature extraction, and nonlinear principal component analysis, which seeks to model its inputs through an architecture consisting of an input layer, hidden layers (encoding layers), and an output layer (decoding layer) that is a duplicate of the input layer. For more information check out this SAS tutorial "Unsupervised Learning Example: Autoencoders" on youtube.com. Nonnegative Matrix Factorization (NMF) is a dimension reduction technique that approximately decomposes a nonnegative data matrix into two low-rank nonnegative factor matrices, often used for feature identification and extraction in various fields such as image processing, text mining, bioinformatics, and spectral data analysis.   Feature Selection   Feature selection focuses on choosing a subset of relevant features for model construction, enhancing interpretability and performance.   Unsupervised Selection Method identifies a set of input variables that jointly explain the maximum amount of data variance, without considering the target variable. Unlike PCA, this method reduces dimensionality by selecting a subset of the original variables, thus preserving model interpretation. To learn how to perform unsupervised variable reduction using the VARREDUCE procedure in SAS Viya, watch this video: "Unsupervised Variable Reduction Using the VARREDUCE Procedure in SAS Viya" on sas.com. Variable Clustering groups input variables that are highly correlated, helping to reduce redundancy and collinearity in the data. This method involves performing graphical LASSO modeling, creating tables that include edge and vertex information, defining an undirected graph. In this way, the relationships among all the variables are expressed through an undirected graphical model. For more information check out this SAS tutorial "Feature Selection Using Graphical Lasso" on youtube.com.   Data Reduction Using Cluster Analysis   Data reduction aims to decrease the volume of data while preserving its informational content, facilitating efficient storage and processing.   Partitive Clustering partitions observations into clusters so that observations in the same cluster are similar and observations in different clusters are dissimilar, using methods like k-means, k-modes, and k-prototypes for interval, nominal, and mixed input variables respectively, and provides a technique called the aligned box criterion for estimating the number of clusters in the data table. To learn how to perform k-means clustering and segmentation in SAS Viya using PROC KCLUS, watch this video: "Unsupervised Segmentation Using the KCLUS Procedure in SAS Viya" on sas.com. Model-Based Clustering (MBC), such as the Gaussian mixture model, is a soft clustering method that assumes each data point is generated from a mixture of normal distributions, requiring you to specify the number of clusters in advance and fit a fixed number of Gaussian components as many as the number of clusters. For further details, check out these posts: "Model-Based Clustering (Part 1): Exploring Its Significance" and "Model-Based Clustering (Part 2): A Detailed Look at the MBC Procedure". Nonparametric Bayesian Gaussian Mixture Model, is a probabilistic model that assumes all data points are generated from a mixture of Gaussian distributions, generalizing k-means clustering to include information about the data's covariance and the centers of the latent Gaussians, thus providing soft clustering and allowing for a more flexible and data-driven determination of the number of mixture components, accommodating varying complexities in the underlying data distribution. For further details, check out this post: "Model-Based Clustering (Part 3): The GMM Procedure Demystified".   Anomaly Detection   Anomaly detection identifies unusual patterns that deviate from expected behavior.   Support Vector Data Description (SVDD) is a one-class classification technique that identifies a minimum-radius hypersphere around the training data, providing a geometric description of the data, and is particularly useful in applications where data for one class is abundant but scarce or missing for other classes, enabling outlier detection. Isolation Forest is used for anomaly detection that identifies outliers by constructing a forest of decision trees, where anomalous observations are likely to have a shorter path from the root node to the leaf node than non-anomalous observations. For more information see this SAS paper: "Detecting Fraud and Other Anomalies Using Isolation Forests".   Association Rules Mining   Association rules mining uncovers interesting relationships or associations between variables in large datasets.   Market Basket Analysis (MBC) is a technique used in data mining that uses association rule mining to discover the co-occurrence relationships among a set of items, typically used to analyze customer purchasing patterns by identifying the items they buy together. For further details, check out this blog post: "Visualizing the Results of a Market Basket Analysis in SAS Viya" on sas.com.   Unsupervised learning plays a pivotal role in the machine learning field by enabling the extraction of valuable insights and patterns from unlabeled data. It facilitates data exploration, dimensionality reduction, clustering, anomaly detection, and feature learning, all of which are critical tasks in various domains such as finance, healthcare, and image recognition.   SAS boasts a rich suite of unsupervised learning methods, offering a comprehensive range of techniques and tools to tackle complex data analysis tasks effectively. From clustering algorithms like K-means to dimensionality reduction techniques such as PCA, SAS provides a robust framework for researchers and practitioners to explore, analyze, and derive meaningful insights from their data.     Find more articles from SAS Global Enablement and Learning here. ... View more

In earlier installments of this series, Model-Based Clustering (Part 1): Exploring Its Significance, we explored the concept of model-based clustering, highlighting its benefits and examining the various models offered by SAS Viya. Then, in Model-Based Clustering (Part 2): A Detailed Look at the MBC Procedure, we delved into the practical steps of implementing model-based clustering using the MBC procedure.   We will now apply PROC GMM to analyze the Census2000 dataset, which provides a summary of the 2000 United States Census at the postal code level. Recall that the goal of the analysis is to group geographical regions in the United States into distinct subsets based on three demographic factors: Average household size in the region (MeanHHSz), Median household income in the region (MedHHInc), and Region population density percentile (1=lowest density, 100=highest density) (RegDens). These factors are common to commercial lifestyle and life-stage segmentation products.   Although model-based clustering can be executed in SAS Viya solely through code, this doesn't prevent us from leveraging it within the Model Studio pipeline interface. Despite the lack of a dedicated node for GMM, similar to the MBC procedure, we seamlessly integrated a SAS Code node into a pipeline, as illustrated below. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The importance of incorporating a Data Exploration node, Filtering node, and Clustering node was previously elucidated and discussed in the preceding installment of this series.   Fitting A Nonparametric Bayesian Gaussian Mixture Model   Now, let's fit a nonparametric Bayesian Gaussian mixture model. Following code is used in the GMM SAS Code node:   /* Executing Nonparametric Bayesian Gaussian mixture model using GMM procedure */ proc gmm data=&dm_data seed=12345 maxClusters=50 alpha=1 inference=VB (maxVbIter=1000 covariance=full threshold=0.001)clusterSumOut=casuser.clustersum clusterCovOut=casuser.clustercov; input %dm_interval_input; score out=casuser.Score_GMM copyvars=(_ALL_); ods select nObs descStats modelInfo clusterInfo; run; /* Printing scored and output datasets */ proc print data=casuser.Score_GMM (obs=10);run; proc print noobs data=casuser.clustersum;run; proc print noobs data=casuser.clustercov;run;   The above statements run the GMM procedure and output the results to CAS tables and ODS tables. In the GMM procedure, the Gaussian mixture model is Bayesian. It generates the model parameters (means, covariances, and mixture proportions) along with their prior distributions. Observations are then generated based on these parameters and their likelihood distributions. The parameters are estimated by their posterior distributions.   In PROC GMM, the number of clusters is controlled by the Dirichlet process. You can specify the mass parameter of the Dirichlet process using the ALPHA option. A larger value for ALPHA tends to discover more clusters in the input data. By default, ALPHA is set to 1. Additionally, you can set the maximum number of possible clusters using the MAXCLUSTERS option. This option specifies the upper limit for the number of clusters that the algorithm can generate.   In this example, COVARIANCE=FULL is specified to capture the possible variations between the clusters. Also, the MAXVBITER= option is specified using a large value and the THRESHOLD= option is specified using a small value to allow a sufficient Variational Bayesian convergence. The VB inference that PROC GMM uses provides a choice of two covariance matrix types for the Gaussian distributions in the Gaussian mixture model: diagonal and full.   The output is stored in the casuser.Score_GMM data set, where casuser refers to a CAS library reference.  The first ten observations of the scored data are printed for review. The GMM procedure also generates the cluster covariance table casuser.clusterCov. and the cluster summary table casuser.clusterSum.   Finally, cluster visualizations are generated using two-dimensional scatter plots of the MeanHHSz, MedHHInc and RegDens variables, labeling each point with a cluster number.   /* Creating cluster visualizations */ ods graphics / reset width=4in height=4in imagename='Clusters'; proc sgplot data=casuser.Score_GMM; scatter x=MeanHHSz y=MedHHInc / group=_predicted_cluster_ markerattrs=(symbol=CircleFilled size=9); run; proc sgplot data=casuser.Score_GMM; scatter x=MedHHInc  y=RegDens / group=_predicted_cluster_  markerattrs=(symbol=CircleFilled size=9); run; proc sgplot data=casuser.Score_GMM; scatter x=RegDens y=MeanHHSz / group=_predicted_cluster_  markerattrs=(symbol=CircleFilled size=9); run;     Now that the model has been fitted, let's review the results of the GMM SAS Code node.     The Cluster Information table provides basic information about the clustering results. This information includes the cluster weights, cluster means, and cluster variances (the diagonal elements of the cluster covariances).     The Model Information table displays basic information about the parameters that are used in the Gaussian mixture model. The GMM procedure used the Dirichlet process in the Gaussian mixture model and identified 19 as the optimal number of clusters for the input data. It used a parallel variational Bayesian (VB) method of model inference. Variational Bayesian methods are a family of techniques for approximating the intractable integrals that arise in the posterior distributions of Bayesian models with computable alternative distributions.         In the scored data set, the _PREDICTED_CLUSTER_ variable contains the cluster assignments, and the _CLUSTER_k_ variables hold the posterior weights, which are the soft clustering values.     In the Cluster Summary table, you can see that PROC GMM performs well in assigning the observations to 19 clusters and also finds the cluster centers.   In contrast to the MBC solution where outliers were not assigned to any clusters, the GMM solution incorporated these outliers into a more flexible cluster framework, allowing for a higher number of clusters. Some of the smallest clusters contain as few as 2 or 6 observations, while the largest clusters encompass 7849 and 6908 observations. The GMM procedure assumes that each data point is generated from a mixture of Normal distributions. This method inherently models the noise component as part of these distributions. In the context of GMM procedure, the noise can be thought of as the variability within each of the Gaussian distributions that make up the mixture model.     The marker symbol and color indicate which cluster each point was assigned to. These results depict nineteen clusters, exhibiting varying degrees of correlation among them. The marker is determined by the PREDICTED_CLUSTER_ variable in the output data set.   Concluding Remarks   PROC GMM is a soft clustering method that assumes each datum is generated from a mixture of Normal distributions, generalizing k-means clustering to include information about the data's covariance and the centers of the latent Gaussian models. It uses a Dirichlet process to determine the best number of clusters and is particularly useful for data exploration.   If you have a general idea about the number of clusters and prefer to proceed without imputing missing values and/or identifying outliers, then PROC MBC is suitable. On the other hand, if you are uncertain about the number of clusters and seek a more flexible clustering solution where each observation is assigned to mutually exclusive clusters, then PROC GMM is the preferred choice.   Unlike conventional clustering methods, both PROC GMM and PROC MBC are capable of generating "soft" clusterings through posterior probabilities. This eliminates the need to rely solely on cluster memberships, allowing for more subjective exploration using the soft clustering values.     Find more articles from SAS Global Enablement and Learning here. ... View more

In the previous installment of this blog series, Model-Based Clustering (Part 1): Exploring Its Significance, you were introduced to model-based clustering, its advantages, and the available models within SAS Viya.   We will now apply PROC MBC to analyze the Census2000 dataset, which provides a summary of the 2000 United States Census at the postal code level. The goal of the analysis is to group geographical regions in the United States into distinct subsets based on three demographic factors: Average household size in the region (MeanHHSz), Median household income in the region (MedHHInc), and Region population density percentile (1=lowest density, 100=highest density) (RegDens). These factors are common to commercial lifestyle and life-stage segmentation products. The data is suitable for the creation of life-stage, lifestyle segments using clustering technique.   While model-based clustering can be implemented in SAS Viya using only code, this does not limit us from utilizing it within the Model Studio pipelines interface. Despite the absence of dedicated node for MBC, we integrated SAS Code node into a pipeline as demonstrated below. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The findings from the Data Exploration node showed a notable spike in the histogram of MeanHHSz around or near zero.     Having a household size of zero is not plausible in census data. Consequently, a lower limit of 0.1 is specified in the metadata. All instances with an average household size below 0.1 are filtered out for further analysis. The Filtering node excluded 1081 cases with a household size of zero. Subsequently, a Clustering node is introduced downstream to conduct traditional k-means clustering for comparative purposes. k-means clustering suggests a five-cluster solution. If you're interested in learning about nitty gritty of k-means clustering, I recommend the Course: Advanced Machine Learning Using SAS® Viya®   A SAS Code node was inserted below the Filtering node to execute the MBC procedure for fitting a Gaussian mixture model.   Fitting A Gaussian Mixture Model   First, let's fit a Gaussian mixture model. Following code is used in the MBC SAS Code node:   /* Executing Gaussian mixture model using MBC procedure */ proc mbc data=&dm_datanclusters=(2 3 4) noise=(n y) covstruct=(vii vvi vvv eev); var %dm_interval_input; output out=casuser.Score_MBC copyvars=(ID %dm_interval_input) maxpost nextclus=wt; run; /* Printing scored data */ proc print data=casuser.Score_MBC (obs=10);run;   &dm_data is a macro variable identifies the CAS training table from the preceding node.   There are three options that control the models that will be evaluated:   The NCLUSTERS option specifies a list of values for the number of multivariate Gaussian clusters - two, three or four. PROC MBC fits a separate model for each value in the list. The NOISE option specifies whether the model should include a uniform noise component. The existence of outliers is a good motivation to include a noise component in the model, because the outliers can otherwise distort the cluster structure in the model. Specifying yes (or y) and no (or n) will generate models both with and without a noise component. The COVSTRUCT option specifies the covariance structure to use in the model. Nine covariance structures are available for fitting Gaussian mixture models: COVSTRUCT=EEE | EEI | EEV | EII | EVI | EVV | VII | VVI | VVV. See the Details: Covariance Structure section of SAS Documentation.   Each element in these lists will be used in combination with the elements from the other lists, giving 24 models (3 clusters × 2 noise options × 4 covariance structures) in all, 12 with noise component and 12 without noise component.   The output is stored in the casuser.Score_MBC data set, where casuser refers to a CAS library reference.  The first ten observations of the scored data are printed for review.   Finally, cluster visualizations are generated using two-dimensional and three-dimensional scatter plots of the MeanHHSz, MedHHInc and RegDens variables, labeling each point with a cluster number.   /* Creating cluster visualizations */ ods graphics / reset width=4in height=4in imagename='Clusters'; proc sgplot data=casuser.Score_MBC; scatter x=MeanHHSz y=MedHHInc / group=maxpost markerattrs=(symbol=CircleFilled size=9); run; proc sgplot data=casuser.Score_MBC; scatter x=MedHHInc  y=RegDens / group=maxpost markerattrs=(symbol=CircleFilled size=9); run; proc sgplot data=casuser.Score_MBC; scatter x=RegDens y=MeanHHSz / group=maxpost markerattrs=(symbol=CircleFilled size=9); run; /* Create Shape and Color columns based on cluster membership in MaxPost column*/ proc fedsql sessref =&dm_cassessref;  create table casuser.Score_MBC_new {options replace=true} as  select *, case when maxpost=0 then 'balloon' when maxpost=1 then 'club' when maxpost=2 then 'diamond' when maxpost=3 then 'spade' when maxpost=4 then 'heart' when maxpost=. then 'point' end as Shape, case when maxpost=0 then 'Burlywood' when maxpost=1 then 'DodgerBlue' when maxpost=2 then 'Crimson' when maxpost=3 then 'LimeGreen' when maxpost=4 then 'MediumOrchid' when maxpost=. then 'Grey' end as Color from casuser.Score_MBC;  quit; proc g3d data=casuser.Score_MBC_new; note j=r "Clusters:"c=Burlywood "Outliers" j=r c=DodgerBlue "Cluster 1" j=r    c=Crimson "Cluster 2" j=r c=LimeGreen "Cluster 3" j=r c=MediumOrchid "Cluster 4" j=r c=Grey "Missing"; scatter RegDens*MedHHInc = MeanHHSz / color=Color shape=Shape size=1 noneedle; run;   Now that we have fit the model, let's examine the results of the MBC SAS Code node.     The Model Information table displays basic information about the model. The selected model has four Gaussian clusters. The selected model includes a noise cluster also. Estimation is carried out using the Expectation-Maximization (EM) algorithm. The best model is selected based on the Bayesian Information Criterion (BIC).     The parameter estimates and mixing probability estimates describe the mixture model that was selected. The parameter estimates describe the mean and covariance of each Gaussian cluster. In the Mixing Probability Estimates table, the numbering starts at zero because this model did include a noise component; component zero is the noise component. These values correspond to the prior weights in the mixture model likelihood that is presented in the section Log-Likelihood Definitions.     The Model Selection Summary table displays several likelihood-based measures of fit for the specified models. The table also includes the number of parameters that are used in the computation of the fit statistics.  The top model is the selected one because it has the lowest BIC value.     In the scored data set, the MAXPOST variable contains the cluster assignments, and the WTk variables hold the posterior weights. You can examine these soft clustering values, which are stored in the wt0, wt1, wt2, wt3, and wt4 variables in the output data set. The wt0 values indicate strength of association with the noise cluster.       The marker symbol and color indicate which cluster each point was assigned to. These results represent four clusters, each with strong correlation, and some background noise represented by blue dots. The marker is determined by the MAXPOST variable in the output data set. Clusters 1 and 2 are not clearly visible in these 2-dimensional scatter plots. A 3-dimensional scatter plot might provide better clarity.     All clusters are clearly visible in the three-dimensional scatter plot. Outliers are indicated by beige balloon markers. These outliers typically have extreme values in one or more inputs, placing them far from or outside the dense cloud of each cluster.   Concluding Remarks   PROC MBC fits mixtures of multivariate Gaussian and uniform distributions for unsupervised and semisupervised clustering of data. It treats cluster memberships as missing data and uses the expectation-maximization (EM) algorithm to maximize the likelihood. Unlike other clustering procedures, PROC MBC can produce "soft" clusterings based on posterior probabilities, indicating relative strengths of affinity that each observation has for each cluster in the model.   The subsequent section of this series Model-Based Clustering (Part 3): The GMM Procedure Demystified will cover the implementation of PROC GMM.     Find more articles from SAS Global Enablement and Learning here. ... View more

The traditional goal of clustering has been to identify groups of observations that are similar by some measurement. Partitive clustering, also known as optimization clustering, achieve this primarily through heuristic methods, by using a measurement such as the distance to a center or a boundary. They essentially divide a data set into clusters by trying to optimize some specified criterion. For example, k-means clustering minimizes within-cluster sum of squares, so that the members of the cluster are homogeneous and maximizes between-cluster sum of squares to get the clusters well separated. Partitive clustering methods, such as k-means, are typically considered "hard" clustering algorithms, as each data point is assigned exclusively to one cluster, including outliers. These methods scale linearly with the number of observations, making them the only practical choice when dealing with a large dataset.   Partitive clustering methods suffer from significant limitations due to their heuristic nature rather than being grounded in formal or strict statistical models. They have drawbacks such as the need to guess the number of clusters, sensitivity to the initial location of the reference vectors (seeds), outliers, and even the order in which the observations are read, and assumptions about cluster shapes (often spherical). Additionally, determining the best partition of observations is computationally infeasible due to the exponentially growing number of possible partitions.   Model-Based Clustering and Why to Use it?   Instead of taking a heuristic approach to build a cluster, model-based clustering uses a probability-based approach. Model-based clustering assumes that the data is generated by an underlying probability distribution and tries to recover the distribution from the data. In other words, similarity is determined by the likelihood, not the distance. This likelihood is derived from a finite mixture model. A finite mixture model has mixture components, and these mixture components represent the clusters.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Each cluster is modeled by a distinct probability distribution (such as multivariate Gaussian or uniform distributions), and the parameters of these distributions (like means and covariances) are estimated from the data using techniques such as the Expectation-Maximization (EM) algorithm to maximize the likelihood.   This all sounds interesting, but why use model-based clustering?     Model-based clustering is a formal model based on a likelihood, so you have well-defined statistical properties for formal inference, and you can use well-established statistical methods for model selection.   Also, the clustering can be "soft" - this means that each clustering has a measure of strength, rather than being a rigid classification. This typically involves estimating fit probabilities, reflecting the degree of belongingness to each cluster, for each of the observations. This will permit observations to belong to multiple clusters simultaneously. This soft assignment provides more nuanced insights into the structure of the data, especially in scenarios where observations exhibit characteristics that align with more than one cluster.   Observations which fit the distribution well will have high fit probabilities, whereas anomalies will have very low fit probabilities. This will allow to identify outliers that might not be members of any of the clusters found.   Another key advantage is that the model-based approach employs statistical criteria to suggest the optimal the number of clusters and the most likely model.   Finally, this approach allows for flexible clustering that can accommodate different shapes and sizes of clusters. The form of each mixture component can vary in shape from the others, meaning that you do not need to be restricted to one cluster shape.   A Simple Example   What does this all look like? Here's a simple example sourced from Dave Kessler's video on Model-Based Clustering with PROC MBC - SAS Video Portal.     In the analysis data, you see two clusters where there is strong correlation within each cluster; that is the "X" shape. You also see some points outside the clusters, which represent some amount of background noise.   If you use a traditional clustering method like k-means, what will happen?  k-means quickly does what it was built to do. It has resulted in a 5-cluster solution. The shape and color of the marker indicates the cluster to which k-means assigned that point, and the dark markers indicate the center of different clusters. The challenge here is that k-means is designed to find the best circular clusters in the data, and the clusters you see are clearly not circular. Now see how model-based clustering approaches this problem!   As in the k-means clustering plot, the marker symbol and color indicate which cluster each point was assigned to. There are clearly two multivariate Gaussian clusters. These results conform to your impression of two clusters, each with strong correlation, and some background noise.     The output of the model-based clustering can include soft clustering information in addition to hard assignments. You can store your fitted models to apply to new data sets. The output contains weights of association between each observation and each cluster. In the above plot, the color of the point is darker for stronger associations. When you look at the noise weight values, which indicates strength of association with the noise cluster, they are highest outside of the "X" shape. When you look at the cluster1 and cluster2 weights, in areas where two clusters cross, the strength of association for any cluster is lower, producing gray areas.   Model-Based Clustering in SAS Viya   There are various algorithms available for conducting model-based clustering, with Gaussian mixture models being among the most widely used probabilistic models for this purpose. In Gaussian mixture models, it is assumed that the data is generated from a mixture of several Gaussian distributions.   In SAS Viya, Gaussian mixture models can be fitted using either the MBC or GMM procedures. PROC MBC and PROC GMM are both model-based clustering methods, but they differ in their approach and capabilities.     PROC MBC supports two types of Gaussian mixtures distinguished by their use of latent factors in the model. The models that do not use latent factors are referred to as Gaussian mixture models (GMMs), and the models that use latent factors are referred to as Parsimonious Gaussian mixture models (PGMMs).   On the other hand, PROC GMM fits a nonparametric Bayesian Gaussian mixture model, which can be regarded as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians.   The MBC procedure is part of SAS ® Visual Statistics and the GMM procedure is part of SAS ® Machine Learning.   GMMs with the EM algorithm require specifying the number of clusters in advance and fit a fixed number of Gaussian components, whereas nonparametric Bayesian Gaussian mixture models allow for a more flexible and data-driven determination of the number of mixture components, accommodating varying complexities in the underlying data distribution.   Concluding Remarks   Model-based clustering, such as the Gaussian Mixture Model and the nonparametric Bayesian Gaussian mixture model, is a type of soft clustering method that assumes data is generated from a mixture of different distributions, often normal distributions. This method generalizes k-means clustering to include information about the covariance of the data and the centers of the latent Gaussian models. The number of clusters is determined using processes like the Dirichlet process. This method is particularly useful in data exploration scenarios. It's important to note that the effectiveness of model-based clustering can be influenced by the scale of the variables, with variables having larger variances tending to have more influence on cluster formation.   The subsequent sections of this series Model-Based Clustering (Part 2): A Detailed Look at the MBC Procedure and Model-Based Clustering (Part 3): The GMM Procedure Demystified will cover the implementation of PROC MBC and PROC GMM.     Find more articles from SAS Global Enablement and Learning here. ... View more

Digital twins are not inherently generative in nature, but the evolution of generative AI is poised to revolutionize them with unprecedented intelligence and predictive capabilities, thus transforming them into "intelligent twins”.   Digital twin is a virtual representation of a physical entity – an object, behavior, process, system, or product. It is essentially an animate ecosystem that is trained to mirror and sync with the physical entity. A digital twin is created by collecting real-time data from sensors embedded in physical entities and using that data to create a digital replica. This digital replica can then be used for various purposes such as monitoring, analysis, simulation, and optimization.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   A digital twin ecosystem is an inherently dynamic, interconnected network comprised of software, generative and non-generative algorithms, batch and streaming data, and of course, business logic. Digital twins often converge AI/ML, simulation, forecasting, optimization, and visual and streaming analytics to stress test a physical entity in the digital world to prescribe actions that optimize the entity in the physical world – to improve the lives of individuals, the lives of populations, the environment, cities, organizations, systems, products, and more.   A simple example of a Factory Digital Twin is illustrated in a SAS post authored by Biller, B. (2022) titled Digital twin development: Why simulations are critical.   Generative AI as a Digital Twin   Generative AI is set to revolutionize digital twins by transforming them into “intelligent twins”.   Envision a friendly looking doctor wearing a white lab coat with a stethoscope around her neck. She has a warm and approachable smile, exuding confidence and professionalism. Her eyes convey empathy and understanding, and she appears focused yet compassionate. This is an AI-generated version of your doctor.     AI-generated version of doctor is created using https://app.simplified.com/graphic-design/tools/text-to-image/results    Now, imagine a scenario where you can post a question to this virtual persona who embodies expertise, care, and a dedication to helping others and could serve as a virtual advisor. This AI-powered digital twin would respond to your queries based on the knowledge it has acquired from the imperfections and behavioral traits of your doctor. It goes beyond simply analyzing real data. Instead, it taps into the essence of your doctor's persona and assisting in decision-making, providing expert insights, and even offering personalized treatment recommendations.   This concept of generative AI digital twins is still in its early stages, but it has already captured the attention of researchers and industry experts.   Generative Ecosystem of Digital Twins   Digital twins can be generative in nature. The simulation aspect of a digital twin is often emphasized, but the role of generative AI is severely undervalued.   The interconnected networks in a digital twin are comprised of numerous deep learning algorithms that are inherently generative. For example, generative adversarial networks (GANs) are increasingly being adopted in the field.   A digital twin uses real-time data to generate or create simulations that mirror the behavior and characteristics of real-world entities. Besides, a digital twin uses generative outputs as automated inputs to the ecosystem. For example, it generates synthetic data as an output from the ecosystem that is, in turn, used as a feedback loop and contributes back as an input into the ecosystem to optimize the ecosystem.   A digital twin generates new data and insights about an entity’s behavior in different scenarios and conditions to proactively guide real-world strategy and to improve real-world outcomes – not as a static replica, but as an animate virtual representation of an entity.   Similarly, it generates new observations to continuously learn and adapt to changes in the real-world entity, leading to more accurate predictions and better decision making about the real-world entity.   What is more, a digital twin generates large amounts of synthetic data in a controlled ecosystem when real-world data is limited or too expensive to collect. This real-time synthetic data creates simulations to mirror and emulate real-world objects, systems, or processes.   Methodological Perspective of Digital Twin Ecosystem   Frank Emmert-Streib presented a methodological perspective of a digital twin system (shown below in a light blue rectangle) with an interface to a physical entity and external data. Its structural architecture highlights the various contributions that AI can offer for optimizing the outcomes of a digital twin. These contributions will be represented from 1 to 6.   A digital twin is created corresponding to a dynamic system that is implemented via a computer simulation capturing essential features of the physical entity. The physical entity to be modeled can come from a wide range of applications including engineering, manufacturing, healthcare, urban development, and more. Usually, this involves external data.     Source: Emmert-Streib, F. What Is the Role of AI for Digital Twins? AI 2023, 4, 721-728.   The additional external data is used to estimate the parameters of the digital twin. This is an optimization process of model creation, the first AI contribution. Importantly, there is another optimization step involving AI that is different from the first one.   The second optimization step ensures the synchronization of the digital twin and its physical counterpart during its operation. This leads to the optimization of the updating mechanism of the digital twin called model updating.   The third AI involvement is generative modeling. For example, a GAN can be used to generate data with characteristics learned from large-scale data to replace or complement a traditional simulation model. That means AI does not only help optimize a simulation model, but it can constitute the simulation model itself.   The fourth and fifth AI associations are principally the data analysis steps. However, the sources of the data for such an analysis are entirely different.   External data is thoroughly analyzed to identify patterns and relationships. This data analysis step is called descriptive analytics. Descriptive analytics seeks to describe an event, phenomenon, or outcome. It helps understand what has happened in the past and provides businesses the perfect base to track trends and can also be used to compare the performance with others within the same industry.   A lot of data is generated by the digital twin. The data generated by the digital twin and the data that is coming from descriptive analytics using the external data are combinedly used to predict future outcomes like component failure of a particular component in a manufacturing unit. This process is called predictive analytics. Evidently, the source of data has a crucial influence on the interpretation of an analysis. Hence, descriptive analytics and predictive analytics provide complementary means, and the technique we discuss next aims at their integration.   The sixth AI application, called decision making, is used for summarizing all individual results achieved up to this point to enable decision makers to base their choices on factual data and insights rather than intuition or assumptions without testing the physical entity. This step integrates everything together and produces a quantitative or qualitative summary that can be seen as the ultimate output of a digital twin system.   How Digital Twin Is Different from Simulation   Data simulation is not a new idea, and it has been a fundamental tool for statistical modeling. Digital twin technology is similar to simulations as both utilize digital models to replicate various processes of a system. However, there are some key differences between the two.     The most notable is that a simulation model is a brute-force computational technique that relies on repeating a computation on many different static random samples to estimate a statistical quantity where modelers must set all parameters manually, whereas digital twin technology uses real-time data to mimic processes by integrating all relevant data into a digital system, effectively mirroring the entirety of the product, service, or process.   It is important to understand that a simulation model is not equipped with an updating mechanism, whereas a digital twin is a mathematical model with an updating mechanism. Digital twins are designed around a two-way flow of information that first occurs when object sensors provide relevant data to the system processor and then happens again when insights created by the processor are shared back with the original source object.   A simulation replicates what could happen to a product, service, or process. Any changes to a simulation are limited to the imagination of the modeler. But a digital twin replicates what is happening to an actual outcome in the real world as it offers active interaction. Thus, the modeler can see if it is working as intended and then determine any improvements based on actual use.   The difference between digital twin and simulation is largely a matter of scale. A simulation studies one process, but a digital twin can itself run any number of useful simulations to study multiple processes. A digital twin is far more flexible, allowing modelers to test the entity through all stages in many different settings. As long as you have the data, the digital twin will replicate the entire process. Digital twin simulations are more versatile and offer a deeper understanding of processes when compared to simulations.   Concluding Remarks   Generative AI represents a paradigm shift in the realm of digital twins, unlocking new possibilities for intelligent decision-making, predictive analysis, and scenario forecasting. By harnessing the power of generative models, digital twins are evolving into intelligent twins that not only replicate physical entities but also simulate, predict, and optimize their behavior in real-time. As this transformative technology continues to mature, it holds the promise of revolutionizing diverse industries and shaping the future of smart, interconnected systems.     Find more articles from SAS Global Enablement and Learning here. ... View more

The Decision Tree node in Model Studio pipeline can build decision tree models autonomously or interactively. To build a model autonomously, simply run the Decision Tree node. Building models interactively, however, gives you more control and can be more informative and is helpful when you first learn about creating decision trees, and even when you are more expert at predictive modeling.   After you run a Decision Tree node in Model Studio pipeline, you can modify the splitting and pruning logic of nodes when you open the decision tree. The splitting logic that you can modify varies, depending on whether you want to modify a category predictor (class input) or a measure predictor (continuous input).   Interactively Editing a Decision Tree   To edit split for a node, right-click the Decision Tree node and select Open. Switch to the Stack view from the top right corner of the window. Stack view displays the objects as if they are in a slide deck. Only one object is displayed at a time, however, a control bar lets you move between objects. By default, Decision tree object is open.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Right-click the node in the decision tree that you want to split and select Edit split for node. The Split Node window appears.   Decision tree models involve recursive partitioning of the training data in an attempt to isolate concentrations of cases with identical target values. Regardless of the type of predictor that you select, you can edit rules when splitting a node.   Under Variable, select the variable that you want to use for splitting. Variables are sorted by -Log(p) or logworth. The logworth is used to determine how well a variable divides the data. As the logworth increases, the partition better isolates cases with identical target values. You can choose to split the data on any of the listed inputs and also edit the splitting point given additional information.     Under Branches, specify the splitting criteria for each branch that stems from a node. You can directly replace the values that you want to use for your splitting criteria in the Values field.     You can also add branches and specify the splitting criteria for each of those branches. If you want to add branches to your splitting criteria, click Add new branch. You can clear all of the branches or reset the branches to reflect the best split by clicking the Actions (the three vertical dots icon) and selecting either Set to best split or Clear all branches. Any changes that are made at this step override the value that is defined for Maximum number of branches in the options pane for the Decision Tree node.   Under Missing values, specify how to treat missing values when splitting the node.     Once done with editing the splitting rules, click OK to apply your changes to the decision tree. You can undo or redo edits that were made to your decision tree by clicking Undo or Redo icons in the upper right corner of the window.   Interactively Pruning and Training a Decision Tree   In order to prune your decision tree to a node, right-click the node that you want to prune to and select Prune to node. The Prune to node option removes all nodes beneath the selected node and turns that node into a leaf node, resulting in a little simpler tree.     If you want to undo pruning, you have two options – you can either split the node one level or you can train more than one level beyond the leaf node.   For the former one, select Split from node and the Split Node window appears to specify the splitting rules. You can select the variable that will be used to split the node one level and edit the splitting point.   For the latter one, select the Train from node option and the Train Node window appears to select the variables to use in splitting. Variables are still sorted in descending order by their logworths at that node.     The Train node window appears with all the variables selected by default. However, you can deselect variables in the list based on business requirements or based on any other reasons. You can also change the Maximum depth of subtree.   Interactive Editor   When you close the interactive editor, you are prompted to save your changes. To save your changes, click Save and Run. If you do not want to save your changes, click Don’t Save. To continue editing your decision tree, click Cancel.   You cannot copy a Decision Tree node when it is in interactive mode. To copy a Decision Tree node, right-click the node and select Enable Properties. When the Enable Properties window appears, select Yes. You can now copy the Decision Tree node. When you enable properties after interactively editing a decision tree, the changes that you made to the decision tree are lost. During an interactive editing session, the Decision Tree node is locked so that other users cannot simultaneously make edits in interactive mode. The node is also locked when a second user who has access to the project attempts to open the editor while it is in use by another user or the same user attempts to open the editor in a different tab or browser. If you encounter an error when you try to save your work, you are returned to the interactive editor. The interactive editing session lasts 30 minutes by default.   Concluding Remarks   In a Model Studio pipeline, you can interactively edit splitting variable or split point or both in one or more than one decision tree nodes, of course, one at a time. You can interactively control the missing values allocation when splitting the node. Additionally, you can create entire tree interactively by first pruning it up to the root node and then use the Split from node or the Train from node options. The tree will then be created by splitting the source data, constituting the root node of the tree, into subsets, which constitute the child nodes. The splitting is based on a set of splitting rules based on classification features (Shai and Shai 2014). This process is repeated on each derived subset in a recursive manner. The recursion is completed when the subset at a node has all the same values of the target variable, or when splitting no longer adds value to the predictions. This process of top-down induction of decision trees (Quinlan 1986) is an example of a greedy algorithm, which is based on the concept of heuristic problem solving by making an optimal local choice at each node. By making these local optimal choices, you reached the approximate optimal solution globally.   If you're just getting started with Model Studio, check out Build Models with SAS Model Studio | SAS Viya Quick Start Tutorial. It guides you through the process of training and deploying machine learning models using Model Studio pipelines.     Find more articles from SAS Global Enablement and Learning here. ... View more

Generative Artificial Intelligence, abbreviated as GenAI or Generative AI, refers to a type of artificial intelligence that is designed to generate new content, data, or information that is similar to, or indistinguishable from, what a human might produce. One of the prominent models in generative AI is the Generative Pre-trained Transformer (GPT), which is capable of generating coherent and contextually relevant text based on the input it receives.   Generative AI systems learn real-world data to generate seemingly new content, such as text, images, audio, video, tabular data, synthetic data, and code, with similar probabilistic distributions and characteristics of the real-world data. Generative AI can produce highly realistic and complex content that mimics human creativity, making it a valuable tool for many industries such as art, writing, software development, product design, health care, finance, marketing, fashion, and more. Generative AI is an exploding area within the field of artificial intelligence. Therefore, it is important to explore the AI backdrop and see where generative AI fits in.   Artificial Intelligence (AI)   SAS describes Artificial Intelligence (or AI) as “the science of designing ethical and transparent systems to support and accelerate human decisions and actions.”   AI is a branch of computer science that deals with the creation of intelligence agents, which are systems that can reason, learn, and act autonomously. Essentially, AI has to do with the theory and methods to build machines that think and act like humans. Advancements in computer processing and data storage made it possible to ingest and analyze more data than ever before. Around the same time, we started producing more and more data by connecting more devices and machines to the internet and streaming large amounts of data from those devices. All these advancements brought artificial intelligence closer to its original goal of creating intelligent machines, which we're starting to see more and more in our everyday lives. From recommendations on our favorite retail sites to auto-generated photo tags on social media, many common online conveniences are powered by artificial intelligence. Most artificial intelligence examples that you hear about today, from chess-playing computers to self-driving cars, rely heavily on machine learning that gives the computer the ability to learn without explicit programming.   Machine Learning (ML), Deep Learning (DL) and Beyond   Artificial intelligence is the broad science of mimicking human abilities, and machine learning is a specific subset of artificial intelligence that trains a machine how to learn. Machine learning automates analytical model building. Early work with neural networks stirred excitement for "thinking machines." A neural network is a kind of machine learning inspired by the workings of the human brain. It's a computing system made up of interconnected units (like neurons) that processes information by responding to external inputs, relaying information between each unit. The process requires multiple passes at the data to find connections and derive meaning from undefined data.   Machine learning and deep learning are subfields of artificial intelligence, and deep learning is a specific field of machine learning. Whereas machine learning mostly dealt with huge tabular data, deep learning generally dealt with images, audio, and text as input data that results in even larger tables. Recent deep learning breakthroughs are driving an AI boom. Deep learning uses huge neural networks with many layers of processing units, enabling them to process more complex patterns than machine learning in large amounts of data. Deep neural networks leverage artificial neurons, backpropagation, weights, and biases to identify patterns based on the inputs.   Artificial Intelligence applies machine learning, deep learning, and other techniques to solve problems in areas such as computer vision (CV), natural language processing (NLP) and internet of things (IoT). In addition, several technologies enable and support AI including GPUs, advanced algorithms and APIs.   Graphical processing units, or GPUs, are key to AI because they provide the heavy compute power that's required for iterative processing. Training neural networks requires big data plus compute power. Advanced algorithms are being developed and combined in new ways to analyze more data faster and at multiple levels. This intelligent processing is key to identifying and predicting rare events, understanding complex systems, and optimizing unique scenarios. Application programming interfaces, or APIs, are portable packages of code that make it possible to add AI functionality to existing products and software packages. They can add image-recognition capabilities to home security systems and Q&A capabilities that describe data, create captions and headlines, or call out interesting patterns and insights in data.   In summary, the goal of AI is to provide software that can reason on input and explain on output. AI will provide human-like interactions with software and offer decision support for specific tasks, but it's not a replacement for humans. And it won't be any time soon.   Generative AI: Subset of All Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Now we finally get to where generative AI fits into this AI landscape. Generative AI is a specific field of artificial intelligence that uses several techniques that continue to evolve. Foremost are deep learning algorithms such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which consist of multiple layers of interconnected nodes that accept large quantities of data as inputs and identify patterns or structures within the data set. CNN is mostly used for image recognition and computer vision tasks. It works by pooling important features from an input image. RNN is used in advanced fields like natural language processing, handwriting recognition, time series analysis, and so on.   Generative AI can process both labeled and unlabeled data using many machine learning algorithms like neural network, k-nearest neighbor, and more. Generative AI also uses some of the reinforcement learning algorithms that focuses on sequential decision-making problems. The goal for such algorithms is to train an agent to perform actions in order to optimize an objective based on a reward signal.     A recent breakthrough of transformer models has revolutionized how we use machine learning to analyze unstructured data. The transformer in NLP is an innovative architecture that aims to solve sequence-to-sequence tasks while handling long-range dependencies with ease. One of the key innovations in transformer models is the use of the self-attention mechanism, which enables the model to weigh the importance of different parts of the input and output without using sequence-aligned RNNs or CNNs. Large language models (LLMs) are also a subset of deep learning. The use of deep learning models which will employ a Variational autoencoder or Generative Adversarial Network model uses methods for generating synthetic data.     AI systems can be used to do different things for us, such as forecast the demand for a product or the weather, recommend customized product suggestions for e-commerce sites, identify objects from images and video, detect fraudulent bank transactions, or optimize large scale production systems. On the other hand, generative AI is a type of AI whose job is simply to generate content.   Arguably, because machine learning and deep learning are inherently focused on generative processes, they can be considered types of generative AI, too. Broadly speaking, whereas traditional AI/ML systems recognize patterns and make predictions, GAI systems learn real-world data to generate data – like text, images, audio, video, synthetic data, code – with similar probabilistic distributions and characteristics of the real-world data.     Find more articles from SAS Global Enablement and Learning here. ... View more

Supervised learning techniques focus on using the data that have labels to train classifiers and then use the trained classifiers to do predictive modeling. However, in real-world applications such as fraudulent insurance claims, the labeled data are usually limited and very expensive to obtain.   Moreover, the key to supervised learning algorithms, such as k-NN and SVM, is that, in general, they depend on the assumption that nearby points are likely to have the same label, regardless of the underlying data structure. This is more an assumption of local consistency and thus might result in inferior classifiers. However, points on the same structure (typically referred to as a cluster) are likely to have the same label. This assumption is often called the cluster assumption and ensures global consistency. Semi-supervised learning ponders both of these assumptions.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Semi-supervised learning techniques combine labeled data and unlabeled data to address challenges posed by supervised learning and to improve classification performance. Semi-supervised learning is used for the same applications as supervised learning. But it uses both labeled and unlabeled data for training—typically, a small amount of labeled data with a large amount of unlabeled data (because unlabeled data are less expensive and take less effort to acquire). The target variable specifies the variable that contains the label. Input variables are used for distance calculation. Only interval variables are supported as input variables.   This type of learning can be used with methods such as classification, regression, and prediction. The goal is to predict the labels of the unlabeled points. The performance of an algorithm is measured by the error rate on these unlabeled points only. Semi-supervised learning is useful when the cost associated with labeling is too high to allow for a fully labeled training process. Early examples of this include identifying a person's face on a webcam. The semi-supervised learning algorithm has numerous applications, including fraud detection, web page classification, image recognition, medical imaging, natural language processing, and action recognition.   Graph-Based Semi-supervised Learning   A principled approach to semi-supervised learning is to design a classifying function that is sufficiently smooth with respect to the intrinsic structure collectively revealed by labeled and unlabeled observations.   The SEMISUPLEARN procedure in SAS Viya implements the graph-based semi-supervised learning algorithm, which is well known for its good performance and scalability for big data. Here the assumption is that the data (both labeled and unlabeled) are embedded within a low-dimensional cluster or manifold that can be reasonably expressed by a graph.   In the graph-based methods, label information of each sample is propagated to its neighboring sample until a global stable state is reached on the complete data set. For each observation in the query data table, PROC SEMISUPLEARN returns the predicted labels for the observations in both the unlabeled data table and the labeled data table.   Graph-based semi-supervised learning includes a 3-step process:   Similarity Measure: Similarities (distances) between the pairs of the observations in the unlabeled and labeled data tables are computed to determine similar data points. The SEMISUPLEARN procedure offers both a dense kernel (the radial basis function, or RBF, kernel) and a sparse kernel (the k-nearest-neighbors, or KNN, kernel) for computing the similarity measure or distance. Graphical Network: A network graph is constructed to represent the intrinsic structure of the data. This graphical network consists of vertices and edges, where the vertices are specified by unlabeled and labeled observations, whereas the edges specify the similarities among the labeled as well as unlabeled observations. Thus, each data sample is represented by a vertex in a weighted graph with the weights providing a measure of similarity between vertices. Label Propagation: Label of each data sample is progressed to its neighboring points to infer labels on unlabeled nodes in the graph. Here the assumption is that the points within the same structure (such as a cluster or a manifold) are likely to have the same label. Given this assumption, the aim is to use unlabeled data to uncover this structure.   Model Studio Demo: Detecting Fraud   The semi-supervised learning algorithm to the PaySim data from Dr. Lopez-Rojas et. al. is applied. A graph-based semi-supervised learning model is created using a SAS Code node in Model Studio pipeline. The PaySimLabeled data set consists of 430 observations, 130 of which are labeled as fraud. The PaySimUnlabeled data set has 5000 observations, and none are labeled. The objective is to detect fraud in the unlabeled data.   In the Code Editor window, first, a CAS session is started, and all the default libraries are assigned.   options cashost='&dm_cashost' casport=&dm_casport; cas; caslib _all_ assign;   The next part of the program loads labeled and unlabeled SAS data files from a client location into the specified caslib and saves it with specified table name.       proc casutil; load file="/home/PaySimLabeled.sas7bdat" outcaslib="casuser" casout="PaySimLabeled"; load file="/home/PaySimUnlabeled.sas7bdat" outcaslib="casuser" casout="PaySimUnlabeled"; run;   The following piece of code invokes the SEMISUPLEARN procedure to perform graph-based semi-supervised learning on the unlabeled data table casuser.PaySimUnlabeled and the labeled data table casuser.PaySimLabeled and output the results to ODS tables. The INPUT statement specifies that the interval variables are to be used as inputs. The OUTPUT statement requests that the predicted labels for the unlabeled and labeled tables be written to the data table casuser.PaySimOut.   proc SEMISUPLEARN data= casuser.PaySimUnlabeled label = casuser.PaySimLabeled gamma = 1000; input %dm_interval_input; output out = casuser.PaySimOut copyvar=(_all_); target %dm_dec_target; run;   By default, the radial basis function is used as a kernel to calculate the similarity in distance computation using the Gaussian kernel. The GAMMA parameter specifies 1000 as the inverse of the variance for Gaussian kernels when you compute the pairwise distance between data observations. This is essentially the RBF kernel width.   The following statements sort the output of PROC SEMISUPLEARN by id and show the observations from 100 to 200:   data PaySimOut; set casuser.PaySimOut; run; proc sort data=PaySimOut; by %dm_id; run; proc print data=PaySimOut(firstobs=100 obs=200); run;   The successful run of the SAS Code node results in the following:     The Model Information table shows the number of unlabeled observations, number of labeled observations, number of levels for the target variable, gamma value, maximum number of iterations, the kernel used in the computation, number of nearest neighbors, and the loss.   You can try experimenting with GAMMA values to minimize the loss.   The PAYSIMOUT table is printed.     The table shows the values of the input and ID variables, the predicted labels for the 100 observations, and the indicators for the labeled or unlabeled data. For each row in the output data table, the _WARN_ column indicates whether the row of data is coming from labeled data or unlabeled data. The _WARN_ value is 1 for labeled data and 0 for unlabeled data. I_isFraud is the predicted target variable, which is generated for labeled as well as unlabeled samples.   PaySim first paper of the simulator: Lopez-Rojas, E. A., A. Elmir, and S. Axelsson. 2016. PaySim: A financial mobile money simulator for fraud detection." In the 28th European Modeling and Simulation Symposium-EMSS, Larnaca, Cyprus.     Find more articles from SAS Global Enablement and Learning here. ... View more

  The purpose of supervised machine learning is to generate accurate predictions for the future. Good models strive to achieve low training error, but it is just as important to achieve low generalization error. The training process needs to account for this compromise and make an honest assessment of the accuracy of the model. Assessing a candidate model on the data that is used to train the model would direct the algorithm to overfit to that training data. Instead of doing this, we normally take the historical data with known target, and split it into three partitions: training, validation, and test.   A major portion is used for fitting the model - the training data set.   Validation data is used to assess the model during training for the purpose of selecting variables and adjusting parameters. Validation data sets are instrumental in preventing overfitting. In lieu of a separate validation set (which might not be feasible for smaller data sets), SAS Viya Machine Learning offers k-fold cross validation through the Autotuning capability. Whether through a validation set or through k-fold cross validation, ensure that the training process assesses the error on data that is not used to train the model.   Test data is used at the end of model fitting to obtain a final assessment of how the model generalizes to new data. The reason for using test data (instead of validation data) is that validation data plays a role in the model training process. Hence, using validation data might lead to the same biased assessments as using training data. For this reason, a test data set should be used only at the end of the analysis and should not play a role in the model training process. For example, you make predictions for new data (say 2024) using a model trained, validated and tested on 2019–2023 data. This classic setup is illustrated below.   Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post   This is essentially an in-time testing approach that could be easily implemented in Model Studio while creating a Data Mining and Machine Learning project. You can either, choose to Create partition variable checkbox under the Advanced project settings and then specify the training, validation and/or test percentages, or, set a pre-existed partition variable within the data by specifying its Role as Partition.   By default, Model Studio uses the test data to select the champion model, if a test data set exists. Otherwise, Model Studio uses the validation data to select the champion model. If a validation data set does not exist, Model Studio uses the training data to select a champion.   Simple! Nonetheless.   “My team build a churn model for a telecommunication company where the churn rate was 10%. The model’s sensitivity (% of actual churners correctly predicted by the model) on test data was 72% but when they rolled out the model and tracked the result after 5 months, the sensitivity was dropped to 45%.”   Have you encountered such a problem? There could be many reasons for such a fallout. Perhaps doing an out-of-time testing would have helped identify the issue with the model before it was rolled out.   Out-of-time testing is the process of validating the model on the latest unseen data and capturing its performance to see whether there is any performance dip happening on the model prediction output. For example, you hold-out 2023, train the model using 2019–2022 data, and out-of-time test it on 2023 hold-out data. This out-of-time testing is illustrated below.     Holdout data is a capability that is new to Model Studio. Since test data was used in model comparison, using it again in comparing different pipelines might introduce bias. By setting aside data for the holdout partition, you allow for a further safeguard against generalization error. However, one of the biggest drawbacks of this approach is that you are not using the most recent data to make predictions.   Note that the holdout data set generally comes from a different time period than the training, validation and/or test sets and thus cannot be partitioned while creating project in Model Studio. The proof of a model's stability is in its ability to perform well year after year. A data set from a different time period, often called an out-of-time holdout data, is a good way to verify model stability, although such a holdout data set is not always available.   Once you have a holdout data you can score one or more models on it in Model Studio. To score holdout data, complete the following steps:   Select the three vertical dots icon in the upper right corner of the Pipeline Comparison tab and then select Score holdout data.       The Browse Data window appears. Select the data set that contains the holdout data that you want scored. Click OK. Model Studio will score the holdout data.     Once all the models are scored, click Close. To see the results of this process, use the Data menu below Pipeline Comparison to select Holdout.     This enables you to see the performance of the champion model (which was selected using the test data) on the holdout data to get an idea of expected performance (e.g., KS Youden) when this model is implemented in the field.     Close the Results. Switch your attention to the Insight tab by clicking on it.     The Insight tab provides model assessment results on holdout data along with other partitions.   Some analysis complications might introduce while interpreting these model assessment results. For example, if you have selected Enable event-based sampling, you must use model performance measures (like KS Youden and ROC curve) that are not drawn from the outcome proportions. Note that Model Studio uses the score code while scoring holdout data. The score code contains a section titled Adjust Posterior Probabilities. This code block modifies the posterior probability by multiplying it by the ratio of the actual probability to the event-based sampling values specified previously. Most model fit statistics, especially those related to prediction decisions (like misclassification rate), and some of the assessment plots (like cumulative lift) are closely tied to the outcome proportions in the training data. If the outcome proportions in the training, validation and test samples do not match the outcome proportions in the scoring holdout sample, model performance can be greatly misestimated.   So, what is an effective out-of-time testing approach in Model Studio? You might want to follow the following process for implementing out-of-time testing in Model Studio.   Find the best model based on Test data using Model Comparison node and/or Pipeline Comparison tab functionalities in Model Studio. You can also use only two partitions: train and validate instead of using three: train, validate and test. Use holdout data to find the best model based on the out-of-time sample. If the model based on holdout is different and if you feel it to be compelling enough to be chosen as the best model you can choose the best model from the comparison screen manually and pass it over to SAS Model Manager for scoring.   Find more articles from SAS Global Enablement and Learning here. ... View more

  It’s here!   Statistics is a core component of data analytics and machine learning. Despite the "bigness" of the data, statistics still has a lot of application. The role of statistics remains what it has always been and is even more important now. Perhaps the core statistical task in (traditional) statistics is inductive inference from data to models and scientific conclusions. This core task is still very relevant in the advent of massive data sets.   Replicability, stability, heterogeneity, causality, and uncertainty are the five basic principles of statistics, and they all hold equally well with big data.     Mobile users: To view the images, select the "Full" version at the bottom of the page.   Ideally, in big data scenario too, the conclusions and findings are replicable and generalizable. If you imagine running the analysis again, now on a new data set, would the outcome be similar, meaning that the model is stable? How would you find out what similarity in outcomes means and how to evaluate accuracy, to quantify uncertainty. Understanding heterogeneity in large-scale data sets is more crucial and comprehending causality and its connection to robust prediction is still interesting.   Are you interested in machine learning and want to grow your career in it? The key to machine learning is using the right data preprocessing techniques, understanding the algorithm, cutting through the equations and Greek letters, and making sense out of complex results.   Developing an accurate understanding of statistics will help you build robust machine learning models that are optimized for a given business problem. SAS launched a new course that provides a comprehensive overview of the fundamentals of statistics that you'll need to start your data science journey. This course is also a prerequisite to many courses in the SAS data science curriculum.   In this course, you learn how to: explain the relevance of statistics in the big data and machine learning world relate statistical and data science terminology generate descriptive statistics, explore data with graphs and plots, and perform testing of hypotheses detect associations among variables and perform linear regression compare explanatory modeling with predictive modeling describe trade-offs between bias and variance fit a logistic regression model and score new data explain the statistical foundations of machine learning prepare your data for machine learning modeling using transformations, imputation, standardization, and variable reduction discuss data difficulties and modeling issues, and their statistical solutions.   It also gives you opportunity of hands-on using SAS Studio tasks to perform your data analysis. This course is available in three modes: face-to-face classroom training, remotely connected live web, and self-paced e-learning mode.   More details on this brand-new SAS course can be checked out at: SAS Training in the United States -- Statistics You Need to Know for Machine Learning   See this course on the Data Science Learning Path: SAS Training - Data Scientist learning path   Also check out On-Demand Webinar: The Data Scientist Learning Journey: Statistics You Need to Know for Data Science | SAS     Find more articles from SAS Global Enablement and Learning here. ... View more

Today it is imperative to deal with the high dimensional data. The dimension of your data refers to the number of input variables. The analysis of high-dimensional data offers a great challenge because the human intuition about geometry of high dimensions fails. One of the biggest challenges in data visualization is to find general representations of data that can display the multivariate structure of more than two variables. Appropriate visuals are especially helpful when you're trying to find relationships among hundreds or thousands of variables to determine their relative importance – or if they are important at all.   The t-distributed stochastic neighbor embedding (t-SNE) is a method in SAS Viya for visualizing high-dimensional data. The t-SNE method computes a low-dimensional representation, also called an embedding, of high-dimensional data into two or three dimensions.   Unlike other dimension reduction methods, such as principal component analysis (PCA), t-SNE is appropriate for nonlinear data and emphasizes existing groupings in the data. The method is named t-SNE because it models the pairwise distances in low dimensions according to Student’s t-distribution. The t-distribution with one degree of freedom has heavier tails than the Gaussian distribution, which means that it assigns higher probability values to large distances. This enables t-SNE to relax pairwise distances for non-neighboring observations, whereas distances between closely neighboring observations are more exactly preserved. This behavior is desirable because it mitigates the crowding problem in high-dimensional data representation and makes existing groups in the data visually evident.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   For example, in the above illustration, the point-of-interest observation is represented in green, and the other two observations, 1 and 2, are represented in orange. Observation 1 has a higher similarity score than observation 2. Thus, it is closer to the point-of-interest observation.   The t-SNE plot is available using the Data Exploration node in Model Studio, which might help identify naturally occurring clusters in the data. You can also use the TSNE procedure in a SAS Code node.   The t-SNE method operates on an input data table. For each observation in the input data table, the procedure returns either two or three computed columns that contain the embedding coordinates. It computes the embedding by minimizing the Kullback-Leibler divergence between the joint probabilities in high dimensions and the joint probabilities in low dimensions, and it stores the embedding in the output data table. To speed up computations, the t-SNE method uses the Barnes-Hut approximation to the perplexity loss.   In the Data Exploration node, the t-SNE projection property specifies whether to perform t-SNE projections of input variables. This option is deselected by default. The t-SNE perplexity property specifies the t-SNE perplexity that controls the separation of points in the projected space. Possible values range from 1 to 100. The default value is 30.   Following example shows how to use the TSNE procedure to obtain an embedding from observations in a data table. The example uses the Iris data from Fisher (1936), which contain morphological measurements of 50 specimens from each of three different species of iris flowers: Iris setosa, I. versicolor, and I. virginica.         proc tsne data = mycas.iris nDimensions = 2 perplexity = 5 learningRate = 100 maxIters = 500; input SepalLength SepalWidth PetalLength PetalWidth; output out = mycas.tsne_out copyvars=(id species); run;         The PROC TSNE statement and at least one INPUT statement are required. You can specify multiple INPUT statements. The NDIMENSIONS=2 option requests that the model return two embedding dimensions; the PERPLEXITY=5 option specifies the perplexity value; the LEARNINGRATE=100 option specifies the learning rate for the optimization; and the INPUT statement specifies that the SepalLength, SepalWidth, PetalLength, and PetalWidth variables be used as inputs.   PROC TSNE returns a two-dimensional representation of each observation. The OUTPUT statement requests that the embedding be written to the data table mycas.tsne_out, and the COPYVARS= option requests that the ID and Species variables be copied to the output.   The following PROC SGPLOT statements plot embedding dimension _DIM_2_ against embedding dimension _DIM_1_:         proc sgplot data=tsne_out; title "Iris embedding"; title1 "Scatter plot of iris embedding"; scatter x=_DIM_1_ y=_DIM_2_ / group=species markerattrs=(symbol=CircleFilled); run;         The visualization below shows the resulting scatter plot.     The colors in the scatter plot indicate three distinct clusters, which correspond to the three iris species. The two axes here in a low dimensional space do not have a particular meaning.   There are two methods available for the algorithm to use for gradient calculation: METHOD=BARNES_HUT | EXACT. By default, METHOD=EXACT. For larger data sets, Barnes-Hut method is recommended over Exact method as it gives good visualisation accuracy when plotted.   The t-SNE method doesn't construct explicit mappings relating the high dimensional and low dimensional spaces. In fact, t-SNE captures structure in the sense that neighboring points in the high-dimensional input space will tend to be neighbors in the resulting low-dimensional space. Care should be exercised while interpreting the t-SNE plot: Unlike PCA or MDS, larger distances can't necessarily be interpreted. If points are separated in input space, t-SNE separates them in the low dimensional space, however, it doesn't care how far they are. At times, t-SNE breaks overlapping or continuous segments of data into contiguous pieces and artificially separates them, particularly at low perplexity settings. Note that the t-SNE method is well suited for visualization of high-dimensional data, and less so for dimension reduction, feature engineering and preprocessing for subsequent clustering and modeling.   Related Links: For more information on The TSNE Procedure see the Data Mining and Machine Learning Procedures documentation: (https://go.documentation.sas.com/doc/en/pgmsascdc/default/casml/casml_tsne_overview01.htm) StatQuest: t-SNE, Clearly Explained (https://www.youtube.com/watch?v=NEaUSP4YerM) is a good resource to understand the concept behind the t-SNE method. Find more articles from SAS Global Enablement and Learning here. ... View more