The purpose of this post is to illustrate how the REGEX LITI rule can be used to extract recurring patterns of information from textual data using SAS Visual Text Analytics (VTA) through the Model Studio interface. This post assumes readers have a basic familiarity with VTA software running in Model Studio.   The Concepts node is a powerful and useful node in a VTA pipeline that has the purpose of extracting information from unstructured, textual data. The node has 9 predefined concepts: nlpDate, nlpMeasure, nlpMoney, nlpNounGroup, nlpOrganization, nlpPercent, nlpPerson, nlpPlace, and nlpTime. These predefined concepts use natural language processing to extract different types of common information from a document collection. But what if these predefined concepts do not extract the exact information an analyst needs? Well, in addition to the predefined concepts, an analyst can also write custom concepts using the LITI (Language Interpretation for Textual Information) language. The LITI language for extracting information from a document collection is powerful, comprehensive, and contains far too many rules for a single post. In fact, entire books have been written on the LITI language 1 . This post will feature the REGEX rule. REGEX, short for Regular Expressions, is a type of coding rule used to extract information in the form of recurring patterns. The term “Regular Expressions” first appeared in a 1951 publication titled Representation of Events in Nerve Nets and Finite Automata, written by mathematician Stephen Kleene. Perl Regular Expressions are a well know variation of the coding rules which have been used for decades 2,3 and were even incorporated into SAS 9 software circa 2000. REGEX rules are used to extract recurring patterns in textual data such as telephone numbers, email addresses, birthdates, social security numbers, serial numbers, and part numbers, just to name a few. This post will not teach you how to write REGEX rules from the ground up, rather it will provide examples and insights that will help you get started in using them within SAS VTA software. All examples provided in this post are performed within the Concepts node of a VTA pipeline.   Social Security Numbers:   To get started with some basic functionality and operators of REGEX rules, let’s do so in the context of a simple example. Through the example I’ll provide some useful and common operators (also called markers, special characters or metacharacters) and explain what they do. As stated above, it may be necessary to read several textbooks 4 and take specific training classes 5 to become proficient with writing Regular Expressions and other LITI rules. In this example, we want to extract American social security numbers from a document collection. We’ll define social security numbers such that they have 9 total digits in a specific pattern; 3 digits followed by a dash (-), followed by two digits and a dash (-), and ending in four digits. We want the REGEX rule to return only social security numbers, and not other numbers such as 10-digit phone numbers (which also may contain dashes) or dates.   Here’s the REGEX code written for a custom concept within the Concepts node in Model Studio:     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In this code each \d marker returns a single digit, 0-9, and each \- marker returns a single dash (-). Writing three \d markers in a row returns exactly three digits, each 0-9, writing two in a row returns exactly two digits, and writing four in a row returns exactly four digits. We can test this rule on the following sentence, which contains a fake social security number as well as a fake phone number: “This rule will match a social security like 123-45-6789 but not a phone number like 123-456-7890”. We want the rule to return a match on the social security number only and not the telephone number. Model Studio also has a Test Sample Text feature which allows the analyst to test rules on sample documents, which may be as simple as a sentence or just a few words. Using the Test Sample Text feature rather than running the rule against the entire document collection can help with debugging and fine-tuning rules and can also save time, for example, when document collections are very large. Here’s the sample sentence in the Test Sample Text window of the Concepts node after the rule has been applied to it:     Returned matches in Model Studio are colored in blue. The rule is verified to work in that it returns a match only on the (fake) social security number and not the (fake) phone number. REGEX rules often have multiple markers that can be used to extract the same information. For extracting single numerical digits, the marker [0-9] could also be used in place of \d. So, an alternate way to write the rule for social security numbers is:     Parts Numbers:   In this example, let’s suppose the information to extract are parts numbers from a document collection consisting of written descriptions of parts and their uses in a manufacturing scenario. The multi-digit part numbers are always decimal values, separated by either a decimal (.) or a comma (,), and consist of varied numbers of digits before and after the decimal or comma. This example will require a rule that’s more flexible than the rule used for social security numbers. One such example of a part number is: 234242.12342143. Here is a hypothetical example of a document, in this case a single sentence, in the data to be analyzed: “The required ball bearing type to be used within the axle has part number 234242.12342143.”   Here’s the REGEX code written within the CONCEPTS node in Model Studio:    This is what each operator does:   \d matches any single digit (0-9) + matches the previous character one or more times (the previous character is a digit) [\,\.] matches either a comma (,) or a decimal (.) \d matches any single digit (0-9) + matches the previous character one or more times (the previous character is a digit)   So, when the sample text is tested, the match is on the part number including all digits before and after the decimal:     Even if the part number is written with a comma, the part number is still matched:     And of course, the rule would still match part numbers that have a single digit before and after the decimal.     To add a wrinkle to the example, suppose that the part number may or may not have a decimal or comma. Thus, we may want to extract a part number that is simply “37”. This can easily be handled by adding a question mark (?) to the rule which is a marker that makes the previous character optional. In this case, the adjusted REGEX rule would be:     The question mark makes the previous character, here a decimal or comma, optional. This adjusted rule returns two matches in the following sample text:     Email Addresses:   The final example will require the most flexible rule yet. Suppose we want to extract email addresses from a document collection. Email addresses could contain a variety of punctuations, for example periods or underscores, could be from different domains, such as .com, .gov, or .edu, and they all contain the @ symbol. There is no standard when it comes to the length of email addresses. Here’s a rule that would match most common email addresses:    Here is how the rule works:   [_a-zA-Z\d\_\-.] matches any lower-case letter (a-z), upper case letter (A-Z), digit (0-9), underscore (_), dash (-), or period (.) + matches the previous character one or more times (the previous character is defined above) @ matches an @ symbol [_a-zA-Z\d\-] matches any lower-case letter (a-z), upper case letter (A-Z), digit (0-9), or dash (-) + matches the previous character one or more times (the previous character is defined above) \. matches a period (.) [_a-zA-Z\d\-] matches any lower-case letter (a-z), upper case letter (A-Z), digit (0-9), or dash (-) + matches the previous character one or more times (the previous character is defined above)   In the following sample document, three email addresses are returned:     The LITI language is used to write custom concepts in the Concepts node using SAS VTA in Model Studio. It is a powerful natural language processing coding language for extracting information from unstructured, textual data. The REGEX rule is only one small part of the LITI language and it is specifically used to extract recurring patterns. This post is far from a complete guide on learning the REGEX LITI rule but it provides and explains some basic examples as a starting point to understanding and using the rule. For more examples using the REGEX rule and a more complete list of markers (a.k.a. special characters or metacharacters), see VTA product documentation 6 .   References   (1.) Jade, Teresa, Biljana Belamaric Wilsey, and Michael Wallis. 2019. SAS ® Text Analytics for Business Applications: Concept Rules for Information Extraction Models. Cary, NC: SAS Institute Inc. (2.) Schwartz, Randal L. 1993. Learning Perl. Sebastopol, CA: O’Reilly and Associates, Inc. (3.) Quigley, Ellie. 1998. PERL by Example. Upper Saddle River, New Jersey: Prentice Hall PTR. (4.) Windham, K. Mathew, 2014, Introduction to Regular Expressions in SAS ® , Cary, NC: SAS Institute Inc. (5.) SAS Course: SAS Visual Text Analytics in SAS Viya. (6.) SAS Documentation: Using Regular Expressions (Regex).       Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this blog is to shed light on the new Document Text window that accompanies Term Maps in the Text Parsing node in Visual Text Analytics. This blog assume readers are already familiar with Visual Text Analytics (VTA) software running in Model Studio.   With the latest release of SAS Viya (at the time of the writing of this blog), a new and useful window is provided to aid in interpretation and understanding of the results of Term Maps constructed in the Text Parsing node of VTA. Term Maps are used to identify relationships between terms in a document collection. Prior to the latest Viya 2024 release, term maps were stand alone and illustrated relationships between terms, without providing a means to the user of a way to see where the terms exist within documents.   Matched documents for terms have always been visible on the main screen when the user is interacting with the Text Parsing node, but now matched documents can be seen while the Term Map is open. This provides the opportunity for the analyst to view matched documents while investigating the relationships between terms on the Term Map. A Term Map uses information such as Information Gain and document match counts to provide insight to an analyst on the relationship between a main, or centered, term and other terms within a document collection.   The Term Map below is based on a document collection consisting of movie descriptions. The central term, police, is of main interest and terms that may be related to police are also being investigated.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Several terms are related to police. Anyone who has seen movies or television shows centered around law enforcement and criminal activity can easily understand the other related terms. Suppose you wanted to investigate the relationship with undercover and specifically see the movie descriptions that contain these terms without closing the Term Map. Until Viya 2024, to see where these terms appear in documents, the Term Map would need to be closed so the view could return to the main Text Parsing page where the Document window could be observed. But in this case, the information in the Term Map would no longer be visible. Now, however, a new Document Text window immediately adjacent to the Term Map is available.     When a term in the term map is selected, the Document Text window shows where that term exists within documents in the text collection. Thus, the Document Text window shows the matched documents with the selected term highlighted. Of the 2137 documents (movie descriptions), 68 of them contain the term police. Those 68 matched documents with the term police highlighted are shown in the Document Text window. (In the image below, the term police has been selected by the user and the mouse arrow is hovering over the term.)     We want to investigate how the terms police and undercover are related. Of the 22 documents that contain the term undercover, 6 of them also contain the term police. The 22 documents containing undercover are shown in the Document Text window with undercover being highlighted. Because the documents are visible, the analyst can see that the first movie description displayed contains both terms: undercover and police. (In the image below, the term undercover has been selected by the user and the mouse arrow is hovering over the term.)     The analyst can further investigate the relationship between these terms by looking at the Information Gain while the matched documents are still visible. (In the image below, the user is hovering the mouse arrow over the line connecting police and undercover.)     The new Document Text window is a useful and handy addition to the Term Map when using Visual Text Analytics within Model Studio. It provides a means for the analyst to see terms within matched documents while gaining insights into the relationship between terms when using a Term Map.   For more on: Term Maps within Model Studio: https://go.documentation.sas.com/doc/en/ctxtcdc/8.5/ctxtug/p0j3ur41igx5gkn1kvlutjfkg3ic.htm#p02ee3p54qyuz7n1cp2rna6d82v6   Term Maps by writing code: https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/casvtapg/p1ns1ic0qmjd8dn1incqd7mn4x1k.htm   Training in Text Analytics: https://learn.sas.com/course/view.php?id=127   SAS Visual Text Analytics: https://support.sas.com/en/software/visual-text-analytics-support.html   VTA pipeline overview: https://communities.sas.com/t5/SAS-Communities-Library/Anatomy-of-a-Visual-Text-Analytics-Pipeline/ta-p/881148 ... View more

This post is part 2 in a multi-part series that has the purpose of introducing Model Studio to Enterprise Miner users. If you did not get a chance to check out Part 1 in the series, do so by clicking here. In this post, I’m going to cover all things data related for the two SAS tools. I’ll discuss how data sources are created in Enterprise Miner, some of the project Advisor Options in Model Studio, and the benefits of the Data tab in a Model Studio project. For brevity, I’ll continue using the same abbreviations for each tool that I introduced in my first post: E-Miner for Enterprise Miner and M-Studio for Model Studio.   A Data Source in E-Miner:   A data source must be brought into an existing project in E-Miner, so a project must be created first. I covered E-Miner projects in my first post in this series. When a data table is imported into an E-miner project, using E-Miner lingo, it becomes a data source. A data source includes the original data table, but it also has additional metadata attached. For E-Miner, the additional metadata, or information about the data, includes specific information in three areas: the variable roles, variable measurement levels, and the role of the table itself. This metadata allows E-Miner to use the data efficiently and make certain decisions for the data scientist during the analysis. These decisions remove mundane tasks from the analyst and allow him or her to focus on more important details during their work. (If this makes you nervous, don’t be. In E-Miner the user can always change or undue the decisions the software makes.) An example of E-Miner making such a decision is the case where the data has a binary target, and a regression model is to be built. E-Miner will automatically invoke logistic regression over linear regression thus removing the mundane decision of which type of regression model to create from the user. Variable roles include, but are not limited to input, rejected, target, and ID. Variable measurement levels include, but are not limited to interval, nominal, ordinal, and binary. These variable measurement levels go beyond the two measurement types that are usually used when writing a SAS data step: numeric and character. And the possible table roles include, but are not limited to input, score, training, validation, and transaction.   From within a project, a new data source is created by using the “File” pull down menu, the “New” short cut button, or by right clicking on “Data Sources” in the project panel. A new data source is created through the Data Source Wizard that takes the user through a series of steps such as naming the data source, selecting the library where the data table exists, establishing metadata for the data source and deciding if a sample of the data should be taken. Here’s a screen shot of step 1 in the Data Source Wizard:     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The number of steps in the wizard updates depending on the metadata as it is established. For example, when the column (i.e., variable) metadata is defined, if the target is given a measurement level of binary, there is an additional step in the wizard for “decision processing”. Decision Processing can be used to update predicted probabilities for a binary target if the event rate in the observed data is different from the true population event rate, as would be the case if the data are over-sampled. I won’t discuss each step in the Data Source Wizard here, but I will cover the Metadata Advisor Options for establishing column metadata which takes place in step 4. Step 4 in the wizard has two options for the Metadata Advisor: Basic, which is the default, and Advanced. When the Basic option is used, every variable is assigned the role of Input, unless a variable name contains the word “target” or “ID”. In these later cases, a variable is then assigned a role of Target or ID, respectively. No variables are rejected when the Basic advisor is used. Further, each variable’s measurement level is assigned as Nominal if it is a categorical variable and Interval if it is numeric. No variable is assigned a measurement level of ordinal or binary using the Basic advisor. E-Miner takes further steps at establishing metadata when the Advanced Advisor is used. Take a look at the Advanced Advisor options below:     There are three main options: Reject Vars with Excessive Missing Values, Detect Class Levels, and Reject Vars with Excessive Class Values. Each option can be on or off and each has its own default threshold which also can be changed by the user. Two of the options guard against having problematic variables in the data source. “Reject Vars with Excessive Missing Values” protects against having an input variable with a high amount of missing information. The default cut-off is 50%. Categorical variables with too many levels can also create problems when modeling so the option “Reject Vars with Excessive Class Values” guards against this. E-Miner will reject any categorical variable with more than 20 levels when this advanced advisor option is used. Finally, “Detect Class Levels” is for converting numeric variables with a small number of distinct values to categorical, or “Class” variables. The default threshold is 20. So, a numeric variable having only 10 distinct values for example, would be assigned a role of nominal. This option changes numeric variables with 2 distinct values to binary and any numeric variable with a single value to unary.   Of course, whether the basic or advanced advisor is used, the analyst can always make individual changes to variables in the Column Metadata step, which is step 5 in the wizard. The second to last step in the wizard is where the role of the table is defined, and the final step is always a summary of the data source including its assigned metadata. When a data source is added to a project, the name appears under the Data Sources folder in the project panel. Something different for E-Miner compared to M-studio, is that an E-Miner project can contain more than one data source.   Once a data source is defined, the role of the table can be changed by selecting the data source name in the project panel and then changing the Role property in the properties panel which is located under the project panel. Variable metadata for a data source can be changed within the project panel itself, by right clicking on the name of the data source and selecting Edit Variables…     Variable metadata can also be changed within a pipeline by using the Metadata node. More on E-Miner nodes in my next post.   Data Access and Advisor Options in M-Studio:   As stated in my first post, a project in M-Studio cannot be created without data being assigned to it during the creation process. This behavior is a bit different from E-Miner when data sources are added after the project is created. In M-Studio, creating a project and defining data are inseparable. For M-Studio, contained within the “New Project” window is a required field for Data. The New Project window was covered in my first post in this series. When the user clicks “Browse” to select a data table, M-Studio opens a “Choose Data” window which provides three options: Available, Data Sources, and Import.     M-Studio works only with data in CAS memory. Data tables already distributed and available in CAS memory are found under the “Available” tab. Data tables already on the CAS server but not yet loaded into CAS memory are available under the “Data Sources” tab. And data that needs to be imported into CAS, distributed, and loaded into memory are done so by using the “Import” tab.   Once an in-memory data table is selected, M-Studio helps establish variable metadata just as the advanced advisor options do for E-Miner. These metadata advisor options are available by clicking the “Advanced” button on the New Project window. One of the advanced options when creating an M-Studio project is for “Advisor Options”. Here’s a screen shot of them:     The three advisor options are the same as the advanced advisor options discussed above for E-Miner, but M-Studio describes them differently. “Maximum Class Levels” here is the same as “Reject Vars with Excessive Class Values” for E-Miner. “Interval cutoff” and “Maximum percent missing” for M-Studio are referred to as “Detect Class Levels” and “Reject Vars with Excessive Missing Values”, respectively, for E-Miner. These options have the same cut-offs as those for E-Miner. Further, in E-Miner when the advanced advisor options are used, the user can decide whether each option is enabled or turned off. For M-Studio, they are automatically used, except for Maximum Percent Missing, which can be disabled by deselecting the “Apply” checkbox. During the creation of an M-Studio project is the only time these metadata options can be seen.   The Data Tab in M-Studio:   In M-Studio, once the project is created, the project automatically opens on the Data tab. This is a huge difference from E-Miner. The Data tab is possible because in M-Studio only one data set is applied to each project. The Data tab is where M-Studio’s assigned metadata can be viewed and changed by the user. So just like in E-Miner, an analyst is never locked into one of the metadata settings decided upon by the software. Here’s a look at the Data tab for a Data Mining and Machine Learning project:     Aside from the Data tab itself, M-Studio takes metadata to a whole new level compared to E-Miner. In addition to the variable roles and measurement levels, M-Studio also allows the analyst to apply certain data preprocessing rules to each variable. Applying these preprocessing rules is optional. On the data tab, when a variable is selected, the metadata fields open in a pane on the right of the Data tab screen. But in addition to the variable role and measurement level, other fields are shown and the fields available depend on the measurement level of the variable. These additional fields allow for data preprocessing rules to be defined for variables. For interval inputs, the additional fields are for Transform, Impute, Lower limit and Upper limit. Rules can be established here for types of transformations (such as inverse, log, square, etc.), imputations (such as custom constant value, maximum, mean, etc.) and lower and upper limits for value replacement. The Data tab only establishes the rule but the data preprocessing action and rule itself are invoked by using the appropriate node within a pipeline. For example, if you set a variable to have an Impute rule of median, and then you place an imputation node in a pipeline, when the pipeline is run, the median is imputed for that variable. For nominal variables rules can be set for Order (such as ascending or descending), Transform, and Impute. Nominal inputs can be assessed for bias, and this is toggled on and off also in the metadata pane on the right. This is another leap beyond E-Miner, as it does not have the ability to assess categorical variables for bias. Below is a view of the metadata pane for an interval input.     An additional role for a variable in M-Studio that is not available for E-Miner is Partition. The user can create their own partition variable and partition the data according to this variable. The role of partition is only available for variables with a measurement level of nominal or ordinal. Another difference from E-Miner is that to create an M-Studio project, some variable roles are required. For example, to create a Data Mining and Machine Learning project, a single variable must have the role of Target, or a pipeline cannot be run. A Forecasting project requires two variable roles: Time and Dependent. For Text Analytics projects, a variable must have a role of Text. Further, when it comes to the role of Target for a Data Mining and Machine Learning project, M-Studio can have just one. E-Miner allows for two variables with a role of target because it has two-stage modeling capabilities. M-Studio does not yet have that capability, at least within a single project. In M-Studio, two-stage modeling could be done using two projects, one for each target variable.   So, it’s pretty clear that although there are some similarities between “data” in E-Miner and M-Studio, how the tools handle and display them as well as the full capabilities of each are quite different. M-Studio only works with data that resides in CAS memory, and this is simply not the case for E-Miner. This fact alone leads to a lot of the differences. For me, as a user of both tools, it took just a bit of time getting used to the differences. I quickly fell in love with the Data tab and its capabilities in M-Studio once I got used to it. One final warning about the Data tab before wrapping up this installment in my series. After pipelines are run in M-Studio, if changes are made on the Data tab, then all pipelines would require to be rerun. Yes, this can sometimes be frustrating and a bit time consuming, but the tool must operate this way. If I go back and reject a variable that was an input in my prior runs, the pipelines must be rerun because that same variable may have ended up in my final models.   For the next post in the series, I’ll be covering differences in how models are built between the two tools. I’ll be covering the SEMMA tools pallet in E-Miner, the modeling nodes in each of the three types of M-Studio projects, and pipeline templates in M-Studio. I hope you are enjoying your journey from the E-Miner world into the new and modern world of M-Studio on SAS Viya. And if you are, stay tuned, there’s a lot more to come!     Find more articles from SAS Global Enablement and Learning here. ... View more

  The purpose of this blog is to compare SAS Enterprise Miner with SAS Model Studio. This blog is part 1 in what will be a multi-part blog series to help current Enterprise Miner users transition to Model Studio. SAS Model Studio, a point and click interface for building machine learning models in SAS Viya, has been out for several years, so why blog about transitioning to it now? Well, the simple fact is that there are still many SAS customers that are currently making the move from SAS 9 (the foundation for Enterprise Miner) to SAS Viya. So, comparing the two products is as relevant today as it was when Model Studio was first released.   Since I plan on releasing a few blogs on this topic, I’d like to start at a high level here and then drill into things with more specifics in following posts. SAS Enterprise Miner (henceforth for brevity, E-Miner) is SAS’s flagship data mining tool, that was designed around SAS 9 technologies. What’s the “engine under the hood” for E-miner? For the most part, it essentially uses SAS 9 procedures but it does come with special procedures that were developed specifically for E-Miner to work with large data sets. It is a point-and-click interface, that can be installed and run on a desktop but its ideal use is at the enterprise level where it is a server-based application. The main idea behind E-Miner as a data mining tool is efficiency. E-Miner saves the analyst time and frustration by automatically taking care of tasks that would typically be done manually when writing code. Primarily E-Miner is centered around its supervised modeling capabilities, but it also has tools for data preparation, feature engineering, unsupervised modeling, model deployment and more.   For SAS Viya, there are three main areas for analysts to work: Data Mining and Machine Learning, Forecasting, and Text Analytics. Model Studio (henceforth for brevity, M-Studio) is a point and click interface for working in these three areas. The “engine under the hood” for Model Studio is SAS Viya procedures and the CAS server. CAS stands for Cloud Analytic Services and is the in-memory engine for SAS Viya. It works with distributed data using a massively parallel processing architecture. Like all Viya products, M-Studio is a web-based application. So, it is accessed using a web browser with a specific URL. M-Studio running on Viya 4.0, the latest release of Viya, can be installed on-site at an enterprise using their hardware, it can run in the cloud, or it can be third-party hosted such as by AWS. Thus, M-Studio is more modern and flexible compared to E-Miner in terms of its hosting options. M-Studio has the same goal as E-Miner in providing efficiency to the user through a point-and-click interface. M-Studio’s focus is on building models in the three areas stated above, but as with E-Miner, it also has capabilities in data preparation, feature engineering, unsupervised modeling, and model deployment.   Getting Started   Let’s start with comparing the first thing a user sees when they access each tool. For E-Miner, after logging in, the user sees a Welcome screen. (Side note: Logging in is only required for server-based instillations of E-Miner, not the desktop version.) Here’s what the Welcome screen looks like:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   From the welcome screen users can perform actions such as creating a new project, accessing all existing or recently accessed projects, and even accessing E-Miner’s help documentation. If the user chooses the option “Open Projects…” from the initial Welcome screen, a list of all existing projects is then shown:     To open a project, select it and click OK. I’ll address creating a new project later.   When a user accesses M-Studio, what they initially see is quite different compared to E-Miner. For M-Studio the user initially accesses a Projects page. The projects page automatically shows all current projects that exist in M-Studio and provides the option to create a new project. (M-Studio requires a log in, but the log in is done immediately upon accessing SAS Viya which takes users to SAS Drive. From SAS Drive, M-Studio is accessed.) Here’s what the Projects page looks like, for an installation where several projects already exist:     So unlike E-Miner, projects themselves are shown immediately and are directly accessible. The project type- Data Mining and Machine Learning, Forecasting, or Text Analytics- can also be seen for each existing project. Users can also create new projects from the projects page. If no projects have yet been created, then none exist to be shown, and the main option is simply to create a new project.   Organizational Hierarchy   Now let’s get into what I like to think of as the organizational hierarchy of each tool. What I mean by this is the top-down structure of the main components used in an analysis and how they assist an analyst in organizing their work. What’s the highest level of organization down to the smallest? E-Miner and M-Studio are somewhat similar in this regard. As what may be obvious from what is stated above, the highest level of organization for E-Miner is a project. Contained within an E-Miner project is one or more data sources. More details on data sources will come in a future blog. Also, contained within a project is one or more diagrams. Diagrams are the level of organization for E-Miner where the work is actually done. A Project Panel exists within a project that allows the user to navigate through what the project contains, in other words, navigate through the organizational hierarchy. Here’s a look at the Project Panel for an E-Miner project. It shows a project named Advanced Predictive Models and that the project contains several data sources (Claim, Fraud, Organics, etc.) and diagrams (Advanced Models, Claim, Donation Analysis, etc.).     Projects can also be as simple as containing a single data source and a single diagram. The user creates what’s known as a process flow in a diagram, where analytical tools are connected based on what is needed to be performed within an analysis. The analytical tools are known as nodes, and they’ll also be covered in detail in a future blog. So, the node is really the smallest level of the analysis. It represents a step performed in the analysis which is typically some action taken on the data. A node can be used to explore data, partition data, create new inputs, build supervised or unsupervised models, compare models, score new data with a model, and much, much more. Here’s a typical process flow contained in an E-Miner diagram. The process flow starts with a data node, includes some basic data preparation, builds four predictive models, and performs model comparison:     For M-Studio, the highest level of organization is also the project. When we drill into an M-Studio project, we do begin to see some differences from E-Miner. For M-Studio it is better to think of the project being linked to a data set, rather than the data being contained within the project. M-Studio works with data that is in-memory. Whereas in E-Miner you add data sources after the project is created, for M-Studio you have to link the project to an in-memory table during the process of creating the project. In fact, you cannot even create a project without linking it to a data source. There is no project panel in the same way there is in E-Miner. You navigate around the organizational hierarchy in an M-Studio project by using a series of tabs across the top of the project. Each tab takes you to a different place, or view, within the project. The tabs that you see are different depending on the type of M-Studio project. For Data Mining and Machine Learning projects there are four tabs: Data, Pipelines, Pipeline Comparison, and Insights. For Text Analytics projects there are two tabs: Data and Pipelines. And for Forecasting projects there are five tabs: Data, Pipelines, Pipeline Comparison, Overrides, and Insights. Here are the tabs for a Data Mining and Machine Learning project:     No matter the type of project, they must each be linked to a data source and consist of at least one pipeline where analytic activities take place. The Data tab is definitely something very different from E-Miner. Keep in mind that for M-Studio only a single data source applies to a project. Thus, more information about the data set and the variables it contains can be directly displayed and available to make changes, from the M-Studio project. On the Data tab, all columns included in the in-memory table are displayed with their associated metadata. In M-studio, part of the variable metadata includes rules that may be assigned to each variable for various data preparation activities such as methods of imputation and types of transformations. I’ll cover this in more detail in another blog. The next tab on the interface is the Pipelines tab. Pipelines are structured flows of analytic actions. They are essentially like the diagrams and process flows in E-Miner. Pipelines are where the work is done in M-Studio. Just as E-miner can contain a single or several diagrams, the same is true for pipelines within M-Studio. Here is a typical pipeline for a Data Mining and Machine Learning project. It performs similar actions as the E-Miner diagram above.     In M-Studio, pipelines are built vertically. In E-Miner, the user can decide whether to build the process flow horizontally (the default) or vertically.   For Data Mining and Machine Learning projects, the Pipeline Comparison tab is another feature in M-Studio that E-Miner does not have. When multiple models are built within a single pipeline, they are compared in a model comparison node (see the image above); just as multiple models can be compared with the same diagram in E-Miner. However, M-Studio has the ability to compare models across pipelines. The pipeline comparison allows for a sort of “champion of champions” comparison. Each pipeline may have its own champion model, but the Pipeline Comparison tab allows for an over-all project champion. The Pipeline Comparison tab automatically compares each pipeline’s champion model, but the user can add any other challenger models from pipelines they choose. Here’s a partial view of the pipeline comparison window for an M-Studio project containing two pipelines, where a logistic regression model has been declared the project champion based on the KS statistic:     For Forecasting projects, the Overrides tab provides a way for the analyst to apply their business knowledge to change the forecasted value for a certain time point that has been predicted by a model. The Overrides tab is only available for Forecasting projects.   Finally, the Insights tab provides a summary of the entire project. The Insights tab is only available for Data Mining and Machine Learning and Forecasting projects. It summarizes the data, states and summarizes the over-all champion model (selected from the Pipeline Comparison tab), and provides other information such as the most important variables across all models built in the project. The results of the Insights tab (and any node from within a pipeline, for that matter) can also be saved to a PDF for reporting purposes.     E-Miner has something close to insights, but it is not automatic, and it only summarizes a single process flow within a diagram rather than the entire project. The tool for providing this process flow summary is the Reporter node. The Reporter node states and summaries each node in the process flow. It creates a summary document as either a PDF or RTF.   Creating a New Project   Before we wrap up this initial blog in this series, let me compare how projects are created in each tool. E-Miner takes a wizard approach to creating a project. By “wizard” I’m referring to a process where one window is followed by the next as steps are performed to complete the task. A new project can be created from the initial Welcome screen or from the interface in general using the File pull-down menu at the top (see below) or a short-cut button (    ).     Once the initial request to create a new project is performed, E-Miner opens a Create New Project window.     The wizard for creating a new project goes through a 4-step process, assuming a full server-based installation. (It is a two-step process for the desk-top version.) The multi-step wizard asks the user to provide information such as the server where E-Miner processing will take place, the name of the project, and where the project files will be saved. The wizard also provides a summary of the new project information in the final step.   For M-Studio, a project can only be created from the initial Projects page. Rather than a step-by-step wizard, M-Studio opens a single New Project window where the user enters required information into several fields.     Required fields are indicated with a red asterisk. These fields are for the project Name, project Type (Data Mining and Machine Learning, Text Analytics, or Forecasting, depending on licensing), and Data. Optional fields are for pipeline Template and project Description. Advanced project options are also available with a single click, and I’ll cover these in an upcoming blog.   The next blog in this series will focus on how each tool looks at and uses data. I’ll discuss how data sources are created in E-Miner, the benefits of the Data tab in an M-Studio project, and some of the advanced project settings in M-Studio. Plans for other future blogs in this series include but are not limited to: creating a process flow in E-Miner versus creating a pipeline in M-Studio, importing E-Miner diagrams into M-Studio pipelines via batch code or score code, automation methods in each application, and a comparison of all supervised modeling methods available between the two tools.   Additional Resources concerning Model Studio   Training: Machine Learning using SAS Viya: https://learn.sas.com/course/view.php?id=343   YouTube videos: Build Models with SAS Model Studio/SAS Viya Quick Start Tutorial: https://www.youtube.com/watch?v=CtuBWJW46Zk SAS Tutorial/End to End Model Building and Machine Learning in SAS Viya: https://www.youtube.com/watch?v=8Ey06q9FHyM   Blogs on Model Studio and Data Mining and Machine Learning: An Introduction to Machine Learning in SAs Model Studio: https://communities.sas.com/t5/SAS-Communities-Library/An-Introduction-to-Machine-Learning-in-SAS-Model-Studio/ta-p/893384 SAS Visual Data Mining and Machine Learning: Getting Started: https://communities.sas.com/t5/Ask-the-Expert/SAS-Visual-Data-Mining-and-Machine-Learning-VDMML-Getting/ta-p/385455 ... View more

  The purpose of this blog is to provide a method in SAS Model Studio to make use of model interpretation tools for a model based on generated, non-interpretable features.   SAS Viya users, specifically those who build machine learning models using Data Mining and Machine Learning tools, likely have used or at least are aware of model interpretability features to gain insight into black-box models. Although model interpretability can be deployed through code, for example using SAS Studio, this blog will focus on the point-and-click interface for SAS Viya, Model Studio. The model interpretability features in Model Studio use tables and plots to provide the user with insights and interpretable information about how the model makes predictions which are otherwise typically not available for black-box, machine learning models.   Machine Learning models often trade interpretability for accuracy. Neural networks, forests, gradient boosting models, and support vector machines are perfect examples of machine learning models that can provide accurate predictions but are considered black-box models. Model Interpretability tools do not magically make the black-box model directly interpretable (in the same way a regression model or decision tree are interpretable), but they do provide insights such as the relative importance of inputs in the model and how the model is making predictions based on the inputs. The following tools are all available for model interpretability: a Variable Importance table, Partial Dependance (PD) plots, Individual Conditional Expectation (ICE) plots, a Local Interpretable Model-Agnostic Explanation (LIME) method, and a Shapley Explainer (HyperSHAP) method. All these tools, aside from the variable importance table, are visual plots to help explain model predictions. The tools can provide insight into the model globally (variable importance and PD plots) or locally for individual observations in the data (ICE, LIME and HyperSHAP plots). Resources on model interpretation are provided below.   Another common practice in predictive modeling is to generate new features from the available data and use these new features as inputs into the model.  New features can be created by doing straight forward mathematical transformations on inputs, such as taking the log or squaring an input, but can also be based on more advanced mathematical transformations which may be constructed using all inputs. One node in Model Studio that generates features using advanced mathematical transformations is the Feature Extraction node. This node generates new features using methods such as principal components, singular value decomposition, and autoencoders. All these methods, however, generate new features based on combinations of original variables in the data set, and these newly generated feature are typically not interpretable on their own. The new features are often linear or nonlinear combinations of all original inputs.  Resources on feature creation are provided below.   So, what happens when an analyst wants to gain insight into a model based on such uninterpretable features? Let’s walk through an analysis which is based on telecom data used to predict customer churn. Below is the initial Model Studio pipeline.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     The imputation node solves the missing data issue. The feature extraction node generates new features. In this case, the feature extraction node is used to create principal components; reducing the dimensionality of the input space, by creating new features based on linear combinations of the original inputs. (By default, principal components are constructed only using interval inputs, where nominal and ordinal inputs would be ignored by the node. This will be relevant later.) The principal components and the original categorical inputs are used to construct a gradient boosting model. Gradient boosting models are powerful machine learning models, that are based on an ensemble of individual decision trees. They are not directly interpretable, and thus are considered by most data scientists to be black-box models. To gain insight into how the gradient boosting model makes predictions, an analyst may choose to use model interpretability tools. However, as can be seen below, these model interpretability results would be based on the principal components. Below are the PD, ICE, and LIME plots for the gradient boosting model used in the analysis:         Notice that the plots are based on the uninterpretable principal components (for example, PC10 and PC14) as well as categorical variables ignored by the feature extraction node. The other model interpretability tools, namely the variable importance table and the HyperSHAP plots, would similarly be based on a combination of the original categorical inputs and principal components. The trick to have these same model interpretability tools based only on original inputs, is very simple! Build an ensemble model using only the gradient boosting model and have the ensemble produce the desired model interpretation tools. This is done by connecting the gradient boosting model to an Ensemble node. This trick, of course, works for any machine learning model. Not just gradient boosting models. The updated pipeline will appear as follows:     When model interpretation tools are turned on for the ensemble model, they will only be based on original inputs. Keep in mind that in this case, the ensemble model is just the original gradient boosting model. Nothing in the model itself changes. It is still based on the same exact combination of categorical inputs and principal components. It is the same gradient boosting model, making the same exact predictions. The only difference is that the model interpretability tools will all be based on original inputs only. Below are the same plots as above but being generated from the Ensemble node.           Now the analyst can gain insight into the gradient boosting model by having model interpretation based only on the original, interpretable inputs.   For more on model interpretability: https://www.youtube.com/watch?v=6LcyVSLwVck https://www.youtube.com/watch?v=ZjhvxsE9n4I&t=19s https://communities.sas.com/t5/SAS-Communities-Library/Complex-models-and-easy-interpretability-is-that-possible/ta-p/831223   For more on generating new features: https://www.youtube.com/watch?v=D9_jXt_sdX8 https://communities.sas.com/t5/SAS-Communities-Library/Automated-Feature-Engineering-for-AI-Marketing-with-SAS-Customer/ta-p/690480 https://learn.sas.com/course/view.php?id=328   ... View more

  The purpose of this blog is to introduce the Cortex Analytics Simulation game. The game is available as a learning tool to SAS’s commercial and academic customers and partners.     The Purpose of Cortex:   The purpose of the Cortex game is to teach predictive modeling and machine learning concepts in a hands-on, fun, yet competitive environment. Cortex breaks from the traditional instructor led training class which typically consists of lecture followed by computer demonstration. That’s why it is referred to as a game and not training.  In the Cortex game, players are thrown into an analytics sandbox and given a bunch of predictive modeling toys to play with, at their own pace. Through a Cortex game page, players have access to a real-world business case study as well as online instructions and tutorials, in the forms of documents and videos. They can also create their own virtual lab reservations where they have access to SAS software and all data needed to play the game. Further, during the duration of the game (typically about 2 weeks, but more on that later), players have access to live instructors/mentors through pre-scheduled office hours. Finally, players have access to an online leaderboard where they submit results and get immediate feedback to see where they stand against other players in the game.   One huge benefit of Cortex is that no modeling experience is necessary. Read more about this in the “Playing Cortex” section below. Further, each organization plays their own private game with their own unique leaderboard so there is as much team building happening as there is learning! Below is a screen capture of the game page for one of the Cortex scenarios.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The History of Cortex:    Cortex was first developed at the university level by the Canadian business university HEC Montreal, in partnership with SAS. A small group of business professors came up with an idea for an out-of-the-box way for analytics and data science students to learn about building predictive models without using the framework of a traditional classroom. They wanted their students to jump into real-world data with minimal guidance and build models using trial-and-error. Simply put, they wanted the students to learn by doing. To make it fun for college students the idea was to treat it like a competition with a leaderboard to display winning entries. The students/players submit lists of customers the hypothetical company should contact and calculate an over-all profit or operating surplus to rank players. This learning method worked so well in the academic space, that SAS decided to make the game available for its commercial customers. It has been a huge success.   The Scenarios of Cortex:   The first application developed for the game used SAS Enterprise Miner. The scenario in this first application is a non-profit organization that is seeking donation contributions. Players build models to predict which potential donors to contact during a targeted marketing campaign. This fundraising scenario consists of two rounds. In the first round, players predict donation amount and market to those with the largest predicted value for amount. Operating surplus from contacting these potential high value donors is calculated on the leaderboard as players submit lists of possible donors to contact. More advanced topics are used in round two of the game where the players consider two targets: a binary target of whether customers donate or not, and a numeric target of how much they donate if they do. Such an application is known as two-stage modeling. In addition, the concept of Net Lift (also called Uplift or Incremental Response) modeling is introduced. The idea in Net Lift modeling is to investigate which potential donors can be persuaded to donate by being contacted, and then market to these persuadable donors. Does being contacted provide a lift, or increase, in the probability they donate or how much they donate?   Over time, HEC Montreal found such success with Cortex, that they decided to expand the game to other scenarios and analytical applications. With help and additional development from SAS, the games were extended into SAS Viya. Below is a summary of all current scenarios:   • Fundraising: The Fundraising scenario is described above. As it was the first scenario developed, it is the most mature of all the applications. Versions of this scenario exist for SAS Enterprise Miner, Model Studio on SAS Viya, and also a coding and open-source version which uses applications like SAS Studio and Jupyter notebook.   • Customer Retention (Available for SAS Model Studio and SAS Enterprise Miner): Players are working for a fictitious tele-com company that is trying to prevent customers from leaving, also known as customer churn. Predict which customers have a high chance of churn and try to get them to stay! Part of the scenario is to investigate whether inviting high risk, current customers to a high-end dinner party will prevent them from churning.   • Credit Risk (Available for SAS Model Studio and SAS Enterprise Miner): Players work for a financial services company that is deciding to which customers credit will be granted. Players of the game predict customers’ creditworthiness based on demographics, financial indicators, historical loan information and credit related metrics. Players will use the Credit Scoring add-ons for Model Studio or Enterprise Miner to predict which customers should be approved or rejected for the loan. Part of the scenario allows the players to apply the concept of Reject Inference to the game. Two powerful tools available in the Credit Scoring add-ons are used in this scenario: the Interactive Grouping node and the Scorecard node. These nodes in the prebuilt pipeline for the Model Studio version of the game are shown below.     A typical timeline for a Cortex game:   Academic applications of the game can be very customized in terms of how long they run. An instructor could choose to run the game over several weeks or months, where students play the game periodically while receiving traditional classroom-based instruction. Or they may decide to have students fully immersed in the game for full-days of gaming activity, where the entire game would be run in just one or two days. The timeline for commercial games is usually a bit more consistent.   Commercial games generally take just over two weeks to run; about 2 weeks of actual game time, plus an additional kick-off session at the start and a wrap-up session at the end. There is an initial 1-hour kick-off session typically in the week prior to the game actually being played. In the kick-off, students are introduced to the game scenario, given instructions on accessing all game needs (the game page, how to create virtual lab reservations, the leaderboard) and a few pointers on how to get started. During the following two weeks, players play the game when and for how long they choose. The more time players put into the game, the more they learn from it. This is one of the big benefits of being outside the traditional classroom. Players make their own lab reservations, and they play the game when convenient for them.   During those two weeks of playing the game, there are four, 1-hour, instructor led office hours. No two office hours are ever alike. Instructors typically start by discussing the leaderboard and the standings for the players. Then the instructors address any and all questions the players have. Finally, time permitting, the instructors can give pointers on predictive modeling concepts or specific details about using the software, all with the intent of helping players be more competitive in the game. The leaderboard shuts down at the end of that two-week period. The game then concludes with a 1-hour wrap-up session where the winners are congratulated and, in some cases, awarded prizes. All players are given the opportunity to discuss their experiences playing the game, what they learned, and what worked and what didn’t work for them to be successful.   Playing Cortex:   Getting started playing the game is easy. For each game, players are given a pre-built project or pre-written code as a starting point. So even players brand new to predictive modeling or that have never used advanced software before can sit down, basically “click run”, and get started. They’ll have two predictive models trained, assessed, and compared immediately upon starting the game. Below is an example of the out-of-the-box pipeline available in the pre-built project for the Fundraising scenario.     Cortex takes a crawl, walk, run approach as the game progresses. All scenarios for the game have two distinct rounds. Round one for each scenario typically covers the basics: predictive modeling and machine learning fundamentals and getting players comfortable with new software. Round two typically covers more advanced analytical methods, beyond those which are covered in round one. Maximizing earned profit, in other words, being competitive in the game, typically is done by applying the round two concepts.   The Leaderboard:   We have a winner! A major component to the game is the online leaderboard, which is unique for each game. Players submit a list of customers to the leaderboard. This mimics a real-world situation in which a data scientist submits a list of customers to contact in a marketing campaign or to grant credit. A data set with known answers (for example in the fundraising scenario, who donated and how much money) is held back from players. This hidden data set is used to calculate operating surplus or over-all profit based on the specific list of customers that the players submit. Players can submit up to 100 different times to try to improve their score and get the coveted #1 spot on the leaderboard. In the screen shot below, from a hypothetic game, 9 players have made submissions, where expenses and operating surplus are calculated for each submission. The player with highest over-all operating surplus is in first place.     To sign up for the game: If you are a SAS customer and would like to play Cortex with your team, fill out this form and we’ll be in touch soon: https://www.sas.com/en_us/training/forms/cortex-for-business-pricing-form.html   If you are at an academic institution and you are interested in Cortex for your class, send an email here: Cortex@hec.ca   More on Cortex: Cortex information page at SAS.com: https://www.sas.com/en_us/training/for-businesses/cortex-analytics-simulation-game-for-business.html#the-scenarios Cortex for business: https://learn.sas.com/course/view.php?id=850 Cortex for academics: https://learn.sas.com/course/view.php?id=853 Video content on the SAS Users YouTube channel: https://www.youtube.com/watch?v=FPZCZfnnhQE From HEC Montreal: https://erpsim.hec.ca/en/cortex         ... View more

  The purpose of this blog is to provide a tour of SAS Model Manager and discuss at a high-level the role it plays in ModelOps. Are you a data scientist who not only builds machine learning models, but is also responsible for putting those models into a production scoring environment? Are you the model steward at your organization who oversees hundreds or even thousands of models and must monitor the performance of these models over time? Or are you just curious to learn more about this so called ModelOps stuff (by the way, ModelOps is short for Model Operations)? No matter what your answer is to any of the above questions, there’s likely to be something new for you to learn from this tour of SAS Model Manager.   SAS Model Manager is the flagship SAS product to perform ModelOps activities. ModelOps means making the deployment and monitoring of machine learning models scalable, to be able to handle any number of models used by an organization. With SAS Model Manager you can:   • track data, models, and score code • work with models developed in SAS 9 and SAS Viya as well as Open Source languages • run champion-challenger tournaments • assess the performance of models over time • use workflow to assign tasks to users when ModelOps activities are required • and much more.   Here’s a quick tour and some highlights of the SAS Model Manager interface and its capabilities. There are 5 main category views for Model Manager, and each is represented by an icon in a menu found to the left of the interface. You navigate around Model Manager for different ModelOps activities using these category views. The five categories along with their representative icons are: Home , Models  , Projects  , Deployments  , and Tasks  .  (Just as a side note, earlier versions of Model Manager only contained categories for Models, Projects, and Tasks.)   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When opened, Model Manager starts at the Home category view. The Home view shows existing projects and models and provides other sources of information that Model Manager users will find helpful. The Home view provides links to SAS communities as well as to documentation and videos for information on topics like “What’s New” and “How To”.   Let’s continue the tour by skipping to the third category on the list, Projects, since this is the category where most of the activities in Model Manager are performed and where users spend much of their time while doing ModelOps. The initial Projects view shows all projects that currently exist within Model Manager. You can think of this initial page as the “view all” page; all projects are shown. New projects are created in this “view all” Projects view, but can also be created from the Home view. In the screen shot below, the Projects view shows there are currently 3 projects that exist in this instance of Model Manager. Some basic metadata about the projects is shown.     Individual projects can be opened to reveal what they contain. Projects may contain one or several models, information and results from scoring tests and performance reports, and a history record of all activities performed within the project. When a project is opened, there are several tabs across the top that are used to navigate through and work with the project.  Some of these tabs include Models, Variables, Properties, Scoring, Performance, and History.  In the screen shot below, a project named QS_HMEQ is opened.  The project is open on the Models tab, where it can be seen that 3 models are listed, thus the project contains these three models. A model called QS_Tree1 has been declared the project champion (note the special symbol in the Role column next to the name of that model).     In the categories view list, the second category from the top is for Models. As with the Projects view, a summary of all models contained within Model Manager can be displayed, but individual models can also be opened to reveal their properties. The screen shot below is showing the Models page when no specific model is opened; this is the “view all” page for models. It shows that there are 9 models contained within Model Manager, the projects the models are contained in, and even the type of score code for each model.     When an individual model is selected, the Models view changes to show information and properties on that specific model. Machine learning models are a lot more than just score code. Additional files for models contain information on input and output variables, model properties such as code language and score code type, and summary information on model performance. Just as within projects, there are several tabs that are used to navigate the information about models. For example, in the screen shot below, a model named QS_Reg1 is open on the Files tab. Files for the score code, input and output variables and other properties about the model can be seen.     Now continuing to move back down the list of categories (the icons on the far left), second to last is for Deployments.  How many models are deployed?  Where are they deployed?  Where are models published and how many have been validated?  This type of information about ModelOps activities is summarized on the Deployment page.  The screen shot below shows that 3 models are deployed and that 3 projects have published models.     The final category to navigate through Model Manager is for Tasks. Tasks use an application called Workflow, that comes with Model Manager, but is not required to be used with Model Manager. A workflow is simply a series of tasks. The tasks can be connected into a node-link diagram to create a model of the workflow called a workflow definition. The tasks in a workflow definition may be assigned to specific users or roles and may need to be performed in a certain order. These tasks consist of ModelOps activities such as importing models, setting and approving a champion model, and publishing of score code to a scoring environment.     Well, sorry to say folks, but our tour ends here.  We’ve discussed the reasons for SAS Model Manager and what its capabilities are.  We’ve also explored and seen highlights of all 5 category views.  Now that you have the basic idea of Model Manager and the role it plays in streamlining ModelOps, the next step is up to you!   Are you interested in learning more about SAS Model Manager or ModelOps in general? There are lots of ways to do so. -Consider taking the instructor-led or self-paced e-learning course, Managing Models in SAS Viya. -Check out some of the free videos and tutorials that are available, such as these from the SAS Users YouTube channel: Deploying Model in SAS or SAS Viya- Model Management/Execution for Python, R and SAS. -I’m also covering the topic at the SAS Explore conference in Las Vegas, in a hands-on-workshop. My workshop will be held on Wednesday, September 13 at 4:00 PM. Here’s where to register for the conference, if you haven’t done so yet: explore.sas.com     ... View more

Unstructured, text data is everywhere.  And organizations are collecting more and more text-based data with every passing day.  But how do organizations gain insight from such data?  Using SAS Visual Text Analytics in Model Studio, it's easy!  Let's discuss some of the essentials on how to get started with SAS Visual Text Analytics and cover the components of the default text analytics pipeline in Model Studio. Here are some of the main steps that may be involved in a text analysis:   create a project access a corpus of text documents extract concepts, terms, and topics interpret your findings   When you first create your Visual Text Analytics project, Model Studio provides a default pipeline.  Here’s what it looks like with a brief discussion of its capabilities:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The pipeline starts with data, which at a minimum for a text analysis, is a collection of documents.   Next comes the Concepts node which is all about extracting information.  The software has 9 predefined concepts, including but not limited to, dates, measures, and organizations.  Custom concepts, specific to the analysis at hand, can be created by the user as well.   After the Concepts node comes the Text Parsing node. The Text Parsing node starts by using natural language processing to determine parts of speech for terms found in the document collection. The node then generates the start and stop lists for the analysis.  The start list, also called the kept terms, is the list of terms in the document collection that are used in the analysis.  All following nodes in the pipeline make use of these kept terms.  The Test Parsing node places any term in the document collection that will be ignored during the analysis into the stop list, also called the dropped terms. Terms may be placed into the stop list for not providing useful information for the analysis or for appearing in too few documents. Because the Concepts node comes before the Text Parsing node, concepts will be allowed to appear in the start list.   Next the Sentiment node is used to determine positive, neutral, or negative sentiment about each document.   The Topics node uses latent semantic analysis to discover natural groupings of important terms.  That is, terms in the start list that often appear together in single documents within the document collection.  The topics are automatically generated and are assigned to documents.  More than one topic can be assigned to a single document. Topics can help reveal prevailing “themes” that occur within the document collection.   Finally, the Categories node can be used to extract Boolean rules to identify documents that belong to categories.  The categories can be based on variables in the input data which have defined roles of category, user created custom-categories, or even topics that have been promoted to categories.   This default pipeline can be edited, to allow for a custom Visual Text Analytics project on subsets of the nodes.   What if you need to process a document collection not written in English? No problem! Visual Text Analytics currently supports over 30 languages.   Are you interested in learning more?  One way is to take an instructor lead course.  The course SAS Visual Text Analytics in SAS Viya will definitely get you well on your way into the wonderful world of text analytics.  There's an alternative way to start your text analytics journey, but allow me to start by asking another question.  Are you attending the SAS Explore Conference, September 11-14, in Las Vegas, NV?  If not, be sure to register soon (explore.sas.com)!  Once you are there be sure to check out my Hands-on-Workshop, SAS Visual Text Analytics in SAS Viya, which will take place 11:00 AM on Tuesday, September 12. Whether you are new to text analytics, a seasoned expert, or just looking for something new and interesting to try out, there will be something for you in the workshop!   For the workshop, attendees will be provided with access to SAS Visual Text Analytics software, all required data, and step-by-step instructions to get started.   I hope to see you in class soon or at my workshop in September at SAS Explore!   Find more articles from SAS Global Enablement and Learning here. ... View more