In this post, I’ll outline the steps for adding a Grafana plugin to SAS Event Stream Processing (release 2022.07 or later). We’ll use the plugin to create a dashboard that shows the current location of the International Space Station (ISS) in real time with a Geomap.   Internet of Things (IoT) applications provide new capabilities and benefits that are only limited by our imaginations. For years, SAS and our customers have been extending the limits of what is possible with IoT. SAS Event Stream Processing is a part of the IoT landscape that many use to build new applications that exploit streaming data.   Data dependencies, innovation, experimentation, trying new things are part of bringing businesses into the age of the streaming and connected world of today. Building and testing prototype IoT projects now occurs with increasing frequency. One piece of the analytical toolkit has always been the ability to visualize the stream of data as it flows through the phases of a project. This is where SAS Event Stream Processing Streamviewer fits the bill in my experience.   As of Release 2024.07, SAS Event Stream Processing Streamviewer was no longer supported. Users were instead encouraged to explore the SAS Event Stream Processing Data Source Plug-in for Grafana as a replacement. I don’t know about you, but with changes like this, sometimes it helps to have someone share a few pointers to get started with the transition.     Grafana may be familiar to you as a tool for visualizing logs and traces such as those produced by Kubernetes environments providing alerts to system administrators. Well, we have integrated it into SAS Event Stream Processing as a visualization tool you can use for testing new projects developed in SAS Event Stream Processing Studio.   As you can guess, I happen to be a fan of the now-discontinued Streamviewer when creating new IoT projects with SAS Event Stream Processing. Streamviewer provided a graphic interface you could use to make sure the data flowing through a project matched your expectations as any data transformations, computer vision detection and other advanced analytics were applied. The documentation says that the use of the Grafana plugin is not intended for monitoring IoT projects, but I still find it useful for validating data in new projects. There’s nothing wrong with having a little fun when working on projects and gaining new insights.   There is a nice communities post that mentions Grafana as part of a computer vision project. You can check out this example that mentions the plugin. The post also describes the recent SAS Event Stream Processing project package organization structure that makes it so easy to work with SAS Event Stream Processing projects in a Kubernetes environment without having to separately worry about keeping track of your data in persistent volumes PV and persistent volume claims PVC.   In this post, I’ll show you a few steps of how to add a Grafana dashboard to easily show the current location of the International Space Station (ISS) in real time with a geomap. To recreate the results shown in this post, you must have SAS Event Stream Processing release 2022.07 or later and the Grafana plugin correctly installed on your environment, which is a straightforward task for an experienced systems administrator these days.   For this example, I presume that you have some familiarity with SAS Event Stream Processing Studio, but if you don’t, that’s ok since by continuing to read, you will gain some intuition on SAS Event Stream Processing.   I have a completed SAS Event Stream Processing Studio project saved on my ESP Server. This project has three source windows, one of which makes the connection to the streaming data source having the latitude and longitude coordinates of where the International Space Station is in real-time. The source window uses a SAS Event Stream Processing URL Connector to access an API that returns the location information in JSON format. The name of the source window is “iss”, and this name will be used later in the dashboard.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I run the SAS Event Stream Processing project in test mode to confirm that I am receiving data from the API. This screen shows that the iss project window is receiving streaming data from the space station. (Note: If you want to run this project in Grafana in your environment, be sure to change the output schema for the ISS source window so that the datetime is set to Type: stamp and not date.)     You can get to Grafana from SAS Viya in the Streaming Analytics application section: Welcome Experience page. (If you are not running release Stable 2024.11 or later, the choice is visible after you first enter one of the Streaming Analytics applications.)       In Grafana, the SAS Event Stream Processing plugin in my environment has been installed and configured with a data source that connects to the SAS Event Stream Processing application. There is a “Build a dashboard” button on this window.     I add a visualization to the new dashboard.     The sasesp-plugin connects to the ESP Server and can access any of the running public projects and any other projects that I have been granted authorization to. To start the dashboard, I use the plugin to find and connect to the running project and then make some selections deciding whether I want to see a table of the data fields, or some other visualization. Multiple visualizations can be added to a single dashboard.     I just had to select the right data source fields at the bottom of the screen to reference the “iss” project and SAS Event Stream Processing window I am interested in viewing. I changed the default Time series view to Geomap and set a few of the listed properties to reference the latitude and longitude fields in the running SAS Event Stream Processing project. I selected location mode > Coords and then added the lat and long fields to the properties panel. I then clicked Apply and Save to add this visualization to the new dashboard.     Here is the Geomap. At the time of writing this post, the space station was heading into the Gulf of Mexico and over the eastern seaboard of the United States.     The SAS Event Stream Processing plugin is easy to use when testing SAS Event Stream Processing projects, and now that I have worked with it for a while, I’m pleased to say that for those of you that liked working with Streamviewer, I can recommend the plugin as a worthwhile replacement.   For anyone interested in trying out this ISS project in your own SAS Event Stream Processing environment, you can get all the details and download a copy of the project from GitHub. This project has more illustrations of SAS Event Stream Processing capabilities and is a good project to run to get some practice with the application. Also, check out this Communities article on Tracking the International Space Station.   Thanks for reading!   Don’t forget that you can always reach out to your local SAS Representative to ask about using your Education Training points to arrange for private online mentoring sessions on SAS Event Stream Processing or any other SAS initiative you are working on.     And remember, with SAS Event Stream Processing the sky’s the limit! 😊     Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this post is to consolidate information available on SAS Viya Workbench in one handy location. First off, what is it?   SAS Viya Workbench is a cloud-based application for Python and SAS developers and data scientists that is accessed through a browser. You can run SAS 9 code and SAS Viya analytics within the same environment without having to install either SAS 9 or SAS Viya.   SAS Viya Workbench is for SAS programmers who want to access SAS in VS (Visual Studio) Code using their choice of running traditional SAS program files (.sas) or using the SAS Notebook (.sasnb) format with features familiar to users of Python notebooks such as markdown cells with inline output and log displays. It is also for Python programmers who use VS Code or Jupyter Notebooks.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In SAS Viya Workbench, users can easily manage their own environment without additional IT support. Users can also choose the compute power to match the needs of the project.   The compute power can be scaled up or down by the user with a simple restart of the workbench to make sure you only use the resources needed for your specific analysis task. This can reduce the cost of running workbench since cloud providers charge by the resources used. No need to be concerned about resizing your environment since all work is saved between restarts of the environment so that nothing is lost. After a restart, you pick up where you left off in the previous session.   Here are some capabilities of SAS Viya Workbench.     You might want to think of SAS Viya Workbench as a convenient standalone cloud-based application that can also be a stepping stone towards a full SAS Viya environment if-and-when enterprise capabilities are needed. Additional Viya enterprise capabilities that you may want are shown in the chart below.     A really good way to get a taste of using Workbench is to view the quick start tutorial youtube videos. I highly recommend you watch the 12-minute Getting Started video from this series of Quick Start Tutorial videos.   The sasviya.ml package   We also provide you with a repository of Python notebooks, SAS programs and sample data for use in SAS Viya Workbench. SAS9 users can use their code and experiment with new Viya procedures that can coexist and run in workbench without requiring a full Viya environment. The SAS procedures run multi-threaded where it makes sense, providing performance benefits that reduce the amount of time and cost required to run your analysis.   Python users can also benefit from using SAS Libraries that offer performance improvements over traditional libraries. SAS wrote the sasviya.ml package for Python users to run in SAS Viya Workbench. This is from a recent post:   The sasviya.ml package is designed to integrate with scikit-learn, so the sasviya.ml model objects have the same syntax and most of the same functionality as the equivalent scikit-learn model objects. The major difference is in the execution, the models in the sasviya.ml package execute using optimized SAS libraries, which takes advantage of multithreading and can yield much faster runtimes than the equivalent models in scikit-learn.   Examples   These examples provide an excellent framework you can modify to quickly run your own data through machine learning algorithms.   In addition to data mining and machine learning procedures, SAS Viya Workbench provides procedures for sampling, data exploration, clustering, dimension reduction, model assessment, computer vision image processing and supervised learning.   Additional Resources are available in the SAS Viya Community where interesting and relevant posts are frequently updated by community members. Examples of community topics for Python and SAS users include tips such as data access, machine learning, and using Git for source control.   E-Learning   In collaboration with my colleagues, we developed a free E-Learning training course on SAS Viya Workbench that is now publicly available on https://learn.sas.com. Simply log into your profile, search for the course Modern Data Science with SAS® Viya® Workbench and Python and activate it to get started.   Product Documentation   The official product documentation has all kinds of useful information including example use cases, data access techniques, programming examples and more.   Why use SAS Viya Workbench?   Cost-efficiency: Combined with cloud providers, you can manage computing costs by user, project and department, scaling resources as needed to only pay for what you use.   Ready-to-use environments using SAS code or Python through Visual Studio Code or Jupyter Notebook interfaces.   Projects can be shared using Git for version control and collaboration.   Improve performance by using multi-thread supported SAS libraries.   There is lots of excitement around SAS Viya Workbench, and I think these links will help guide you to the resources you need to keep you in the know.   Thanks for reading and I hope you enjoy exploring SAS Viya Workbench!     Find more articles from SAS Global Enablement and Learning here. ... View more

Imagine that you have an enormous collection of transcribed phone calls from bank customers as your data source. If you were to ask someone to identify calls related to ‘accounts’ in a bank, what do you think they would say?   Sure, some may simply tell you they’re not interested in whatever you’re selling. Others may indulge your question and provide a thoughtful response possibly suggesting that you:   Look for savings, checking, account balance, and interest rates. Search for overdraft, insufficient funds, closed account. Don’t forget the terms: fraud, check, or any word containing ‘cash’.   With a bit of work, we can come up with convincing search arguments that seem to do a pretty good job of identifying the concept we are looking for. Even with well-crafted arguments, it is possible that some relevant documents remain unidentified.   Is there anything else that can improve our information retrieval searches when we have run out of ideas based on typical human thinking patterns?   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Refer to my previous post where I show how to use SAS Visual Text Analytics to automatically generate concept rules that can build on our research related to ‘accounts’.   In this post, I want to highlight some less-obvious available syntax for constructing various types of concept rules. This screen capture from my previous post shows a handful of generated rules. Think of these as a source of concept rule ideas you can experiment with and use in rules that you have already written.     Let’s consider how we can use these for inspiration to create specialized rules that can be useful in many different scenarios. What do you think of this rule for starters? We can use it to determine what actions have been taken regarding user accounts.   C_CONCEPT:_c{:V} my account   The C_CONCEPT rule returns the _c portion of the rule when a complete rule match is found in a document. This rule looks for a part of speech :V (verb) immediately followed by the term ‘my account’. This rule returns any actions applied to a user’s account. The action will likely be a verb such as opened or closed, but we might discover some unusual or unexpected actions that provide new insight into our customer data.   This information can then be used to identify trends in the number of specific actions that have been taken against user accounts over time. (A SAS Visual Analytics dashboard using the scored concepts is a good way to report on this information).   Different parts of speech can be substituted in this type of rule depending on the information you are looking for. Nouns, verbs, adjectives, prepositions and determiners are examples of parts of speech that can be identified by the Natural Language processing capabilities in SAS Visual Text Analytics. This table lists the available parts of speech for all supported languages. Here are a few examples of additional automatically generated rules to ponder.   C_CONCEPT:_c{:N} from :PN   This rule highlights a noun :N followed by ‘from’ followed by a proper noun :PN (upper case noun). This rule can identify interactions between parties that may be of interest in a business context. Here are some matches for this rule from my data collection of customer complaints.   Matches:   retroactive spousal payments from Department of Veterans Affairs   contacted so many attorneys from New Jersey to Georgia to pursue a case   After hearing from IRS The next example looks for the term fraud in the same sentence as any monetary amount found by the predefined nlpMoney concept. The elements in this rule can appear in any order as long as they are in the same sentence. Combining predefined concepts with specific information you are looking for can be very powerful.   CONCEPT_RULE:(SENT, "_c{Fraud}", "nlpMoney")   Here are three returned matches from this rule.   Bank 's Fraud Department regarding an online purchase of over {$1500.00} in which I stated…   …was victim on fraud in the amount of {$18000.00} on XX/XX/2022…   Overall, I lost over {$13000.00} to this fraud and I really want my money back. Next is a rule that requires items to appear in a specific order for a match to occur. It looks for any word form of either fraud or scam within 5 terms of a proper noun.   CONCEPT_RULE:(ORDDIST_5 , (OR, "fraud@", "scam@"),"_c{:PN}")   Matches:   I was scammed. The FBI is now involved.. I filed a fraud report with local XXXX Sherrif 's police department… Here is a generated rule looking for items that occur in any order in a sentence. This rule combines a predefined concept with a morphological expansion (the @ symbol syntax) of the term advance. (any form of the term 'advance')   CONCEPT_RULE:(SENT, "_c{advance@}", "nlpMeasure")   Matches:   It was not until two days later ( XX/XX/18 ) that I found out that they transferred money from my own credit card as a cash advance to make me believe it was money from somewhere else.   mailing in my payments 1 month in advance   We will send you an advance notice approximately seven days before we reapply the negative balance.   There were 75 cash advances If you are looking for any mention of an action, this rule identifies what’s going on in your documents by looking for a verb before ‘that’ followed by a noun. Here are some examples.   C_CONCEPT::V that _c{:N}   Matches:   lose that amount of money.   stop that retaliation   I sent that money   protect that data or delete that data     I would not have thought of these rules were it not for the automatic generation capability. Consider incorporating some of these ideas in your concept rules to see if it improves your information retrieval success. Who knows, if your current rule ain't broke, you may make it better!   Thanks for reading and I wish you much success with your text analysis adventures.     Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this post is to describe how to enhance concept rules to retrieve better focused information from a collection of documents.   We'll explore concept rules and ideas for refining them in the context of an analyst processing a large number of customer complaints at a large bank.   Better concept matches start with interactive exploration in SAS Visual Text Analytics.   For an introduction to the capabilities in SAS Visual Text Analytics see my post.   Concepts are used to extract relevant information from documents. This analytical journey starts with exploratory analysis. When processing customer complaint documents, concepts help us: identify conditions where someone needs help, identify opportunities to improve customer service, discover social media comments to address before a complaint goes viral, and more.   First some freebie tips.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Predefined concepts are automatically identified by checking a box in the properties panel. These use Natural Language Processing (NLP) techniques to identify concepts such as measures, money, organizations, people, places and more. The nlpMeasures concept for example, returns matches for terms such as (2 years, two days, 5 mg, $250., 8 feet). The nlpMoney concept matches anything related to currency such as ($2, 2.50€, 40 yen, 20 pounds).   You can combine predefined concepts with new custom concepts to easily identify documents having specific information. Keep these freebies in mind to assist you with building your own concepts.   For example, the rule argument (SENT,”_c{cashed}”, ”nlpMoney”}) returns matches for any document that has the term “cashed” and any currency amount in the same sentence. SAS will recognize currency whether it is U.S. dollars, British pounds, etc.   Your mission…   Let’s consider an example scenario.   You work for a bank and one of your responsibilities is to evaluate 10,000 recent customer complaints that came in through various channels and are available in computer readable text. Your mission is to investigate complaints related to the handling of cash and recommend actions your company might take to improve the customer’s experience.   What is the first thing you would try?   Exactly, Visual Text Analytics!   Open the documents in SAS Visual Text Analytics, run the text parsing node and look at the kept terms and list of documents. You might then try searching the documents for the word cash and see if this gives you any ideas. Your search returns 700 documents and you see phrases in the documents like:   Cash a check Cash advance Cash back Cash price Cash deposit / withdrawal     Remember these pointers to structure an effective search:   Placing the "+" in front of a word requires the word to be in a document. A term without a preceding + returns the term if it exists in the document but it does not have to be in the document. A "–" in front of a word finds documents that do not have that term. Place a * at the beginning, middle or end of a search query to get wildcard matches. The query term +*ion returns terms such as exception or action. Quotation marks return the specific content in between such as “fraudulent check”.   You might then shift your attention to the kept terms list at the top and search for cash again and notice the following.      In this display, cash is recognized as a noun, verb, proper noun, and in a noun group by the NLP process. You decide to look at the verb cash in the term map next.     This map shows the terms related to cash based on information gain. One path from this chart suggests that some documents contain (cash, check, issue) and do not contain (offer or deposit) where the ~ sign indicates “not”. These ideas represent new insights to consider as you construct new custom concept rules.   In the parsing node you can click the similarity scores icon, remove “cash” from the kept terms filter, and notice the additional new terms that appear. “Check fraud” and “forged check” didn’t initially come to mind when exploring cash, but you may want to expand your investigation for occurrences of fraud related to cash!     Concepts Node   Next on your journey, you add and run a concepts node to identify predefined concepts in the documents and then create a custom concept looking for various morphological expansions of the term “cash”, with fraud and check. The initial LITI (language interpretation for textual information) CONCEPT and CLASSIFIER rules follow.     These results look promising, but a lot of documents matched. You wonder if there might be some additional ideas for rules. Click the Autogenerate concept rules icon in the concept that has just 3 simple rules.     The fresh new rules were written to the sandbox. The generated rules will need more explanation especially if the LITI syntax is new to you. We plan to cover some of these in the next post, but here is a sample of the autogenerated rules in the meantime.   After building concept rules, the resulting concepts model can score the current or new documents for concept matches. SAS Visual Analytics can also be used to report on the results.   If you can’t wait until the next post to make sense of these rules, reference this documentation, otherwise, I look forward to sharing some more insights with you in the next post! It's your move!       Find more articles from SAS Global Enablement and Learning here. ... View more

  A customer recently posed this question: “I know how to match a concept at the end of a sentence, but how can I match it at the conclusion of a document?” (For a review of concepts and other basics, see my previous post.) The purpose of this post is to match concepts near the end of a document using a technique that leverages multiple nodes. LITI is Language Interpretation for Textual Information, and it is the programming language used to code concept and category rules in SAS Visual Text Analytics.   Customer complaint call transcripts can end up being quite verbose. Sure, for starters, you could run a summary of the documents to get the main ideas being raised, and this would be a good thing to do. (see my post on summarizing documents). But isn’t there something else I could also try?   The complete customer interaction likely makes some progress over its duration. The follow-up actions or conclusions of the dialog can be expected to occur towards the end of the event, after the back-and-forth discussion has become more focused on the real issues. It may not be necessary to read through the entire interaction to gain some actionable insights.   You may be most interested in returning the matches for certain conditions that occur in the last 50 terms or so of an exchange between the customer and the call agent. The text near the close of the conversation likely holds information to help you determine any appropriate followup actions. Designing the structure of your rules thoughtfully can provide meaningful insights. You could also create a dashboard of these conclusions over time for extra credit.   Consider this example where a Concept rule finds matching terms occurring in a document based on a list of conditions and activities that are stored in supporting rules. In the consumer financial services area, you might have a concept rule containing LITI syntax like this:   CallCenterMatches   CONCEPT_RULE:(AND,(DIST_20,"_CONDITION_", "_c{_ACTIVITY_}"))   This rule matches documents that have an item from the Condition rule within 20 tokens of an item from the Activity rule. A match requires that at least one item from each table is in the document due to the use of the AND operator.   Two example tables below use the LITI language syntax for a table of conditions and a table of activities. The items from the columns can appear in any order. The @ symbol indicates morphological expansions of the term (all forms of a term like report, reports, reported, reporting…)   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Here are some of the 635 matches from SAS Visual Text Analytics of the CallCenterMatches rule with the activity name highlighted as the rule requested.     OK, this is great, but we already know how to use the CONCEPT_RULE and all its bells and whistles. What else can I do?   What if you have a long call transcript and only want to know what happens towards the end of the call? This kind of focus may do a better job than our CallCenterMatches rule on its own to identify followup actions with the customer. To do this, we’ll add a category rule to find a concept if it exists at the end of the interaction (call or chat or transcribed audio)   Next, we will create a category rule based on the concept and use an easily overlooked statement available in the Category node.   I can connect a categories node directly to a concepts node as shown in the right-hand branch of this pipeline, since both nodes can do their own parsing.     I added the following Categories rule so I could see only the matches at the end of the document. The syntax of the Categories node is different than the Concepts node, so this syntax is not valid for use in the Concepts node.   I created my new Category rule using this syntax:   (END_40, "[CallCenterMatches]")   The square brackets in double quotation marks tells the Categories node to use a concept rule rather than a term in the rule.   After applying the concept matches to the categories node in the last 40 tokens of these call reports, there are only 190 matches whereas previously there were 635 document matches in the concepts node.   This is the end of one of the matched documents with the highlighted conditions and actions terms within the final 40 tokens!     This is the description of the END_n Categories rule.   END_n (From the end of the document)   Takes a value for n and one or more arguments. Matches if each argument occurs within n tokens from the end of the document. For example, the rule (END_35, "conclusion") produces a match if conclusion is found within 35 words from the last token in the document. Note: Punctuation counts as a token. In some languages, words that include hyphens are counted as one word (for example, merry-go-round is one token).   So, there you have it! You can combine concept rules within categories to get specific matches based on your needs. There are many other hidden treasures in the land of LITI rules and who knows, I may reveal some more in posts yet to come!!!   Thanks for reading and let me know if you use a form of this rule.   BTW,   If you really want to learn this stuff, here is an exercise for you to try. See if you can identify what each of these rules will return. Do some rules not give you what you are really looking for? Do any of these rules return more than the information we are looking for? Read about the DIST_n, AND, OR, and _c operators for Concept rules for hints.   CONCEPT_RULE:(DIST_20, "_c{_CONDITION_}", "_ACTIVITY_") CONCEPT_RULE:(AND, "_c{_CONDITION_}", (OR, "_ACTIVITY_")) CONCEPT_RULE:(OR,(DIST_20, "_c{_CONDITION_}", "_c{_ACTIVITY_}")) CONCEPT_RULE:(AND,(DIST_20,(OR,"_c{_CONDITION_}", "_c{_ACTIVITY_}")))     Find more articles from SAS Global Enablement and Learning here. ... View more

  Hello, and welcome to my blog! The purpose of this blog is to encourage you to experiment with some of the options that are available for taking your Text Topic Modeling beyond the basics. If you are new to Visual Text Analytics, see my article on Getting Started with Text Analytics to get an idea of the capabilities.   Starting with a collection of documents, these can include customer call reports, patients’ comments, product reviews or movie recommendations, SAS provides the Natural Language Processing (NLP) capabilities that can turn the documents into useful insights.   The Topics node is usually inserted near the end of a text processing pipeline. This node provides you with controls that regulate the topic discovery process. Controls include setting cutoff thresholds for terms and documents. A lower threshold captures more items while a higher threshold captures fewer items. The threshold is based on the number of standard deviations above the mean term or document weight.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Note that when running a Topics node, some documents will likely not fit into any of the generated topics, and they will be accounted for in a group designated as “No Matching Topic”. Re-running the node with lower thresholds specified will result in fewer documents falling into the “No Matching Topic” group. These settings provide users with adequate controls for extracting reasonable topics when getting started with a text analysis project. Machine generated topics can be combined or split, and additional user-created topics can also be added.   To illustrate this point, I ran the topics node with the default settings, and it returned 11 topics. The right-most bar represents documents that did not fit into any discovered topic.     I then reduced the document cutoff control setting from 1 to 0.7 standard deviations, and now more documents are accounted for in the generated topics as shown below. The 5th bar from the right in the chart below represents the reduced number of documents with no matching topic. A good approach is to examine the results you get by spot checking the documents in your topics and then make adjustments to the thresholds until you are happy with the settings.     The Latent Dirichlet Allocation (LDA) Topic Model   The results above were generated under the default Singular Value Decomposition (SVD) algorithm also known as latent semantic indexing (LSI). If you wish to experiment with generating topics using a different algorithm, the Latent Dirichlet Allocation (LDA) Topic Model technique is also available. Running LDA for topic generation may produce different topics and document assignments. It utilizes Bayesian and Expectation Maximization techniques. Now, let's discuss how we'd implement it by calling a CAS Action in syntax.   Implementing the Latent Dirichlet Allocation algorithm: ldaTopic.ldaTrain action creates topics from a document collection using just a document ID, input table, and the variable name containing the documents at a minimum. It defaults to creating only 2 topics, but this can be changed using the k= parameter. In my movie description example, I asked for k=5 topics to be created.   From the documentation, the minimum syntax needed to run with default settings is shown here. This code would run in SAS Studio.   /* Connect to CAS and load the data table. */ options casport=5570 cashost="cloud.example.com"; /**/ cas casauto; caslib _all_ assign; proc cas; /**/ session casauto; table.loadTable result=r / /**/ caslib="Library_name_to_use" path="Path_to_my_file" casOut={name="CAS_file_name", replace=true}; run; /* Run the ldaTrain action – this section defines the output table names */ ldaTopic.ldaTrain / /**/ casOut={name="Topics", replace=TRUE } docDistOut={name="DocDist", replace=TRUE } /* This section defines the input table name, id variable and text variable */ docId="D_ID_variable_name" table={name="Table_name"} text={{name="T_variable_name"}}; run; quit; After running code based on this outline, the following output table shows the proportion of terms for each topic that was found in a document. There were 5 topics generated, numbered 0-4, and each document id has a proportion value for each topic.     The following output table shows what topic (0-4) each term was placed into as well as the assigned probability the term will represent the specific topic.     As I mentioned before, I added a parameter k=5 to return 5 topics rather than the default 2 topics when I ran against a movie description document collection. The hyperparameter Alpha with a minimum value of 0 and a default value of .1 specifies the Dirichlet hyperparameter for the document's topic proportion. Beta specifies the Dirichlet hyperparameter for the topic distribution.     Additional optional parameters described in the product documentation let you do additional configuration including adding stop words, modifying the number of iterations, entity identification and stemming.   You can run LDA action sets from CASL, PYTHON, R and Lua, as well as from a SAS STUDIO task for those of you who might prefer to work with a user interface. Let’s take a quick look at SAS Studio for inspiration.     On the left you see the text parsing and topic discovery selection. In the center, the Options tab is selected as is the LDA radio button. On the right you see the generated code calling the ldaTopic.ldaTrain action. Even if you are a ‘coder’, the SAS Studio task is a great way to quickly get a working copy of code showing correct syntax and the placement of additional options.   The SAS Studio tasks are straight forward easy to use and can be an inspiration on your way to becoming a text analytic practitioner!   In a subsequent blog we will explore scoring these models and ways to deploy them in a production environment. Until then, I wish you much success with your analysis and may all your topics be true!     ... View more

Hello, and welcome to my post! The purpose of this post is to showcase the Text Summarization task in SAS Studio (in less than 5000 pages 😒).   The volume of text data seems to increase each day, and we often encounter information overload in our daily activities. It seems that everyone has something to say, whether it is in a video on how to clean a yucky coffee pot, another breaking news report, customer product complaints, or transcribed audio. How do you find meaning in the vast amount of data that is growing every day?    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS Visual Text Analytics (VTA) users certainly have the tools needed to find answers buried in document collections. (See my post on Getting Started with Text Analytics).   The Text Summarization task in SAS Studio is a VTA tool that provides functionality to summarize a document. Ever think “TL;DR” when looking at social media posts? I know that you know this but… TL;DR is an abbreviation for "too long; didn't read."   You probably have found yourself in a situation where you could use some help getting a quick summary of a document collection into a reasonable number of sentences. Especially if the alternative is going through “billions” of pages of text.   The Text Summarization task in SAS Studio is one possible way to summarize documents quickly and easily. It uses natural language processing (NLP) techniques to summarize a document by identifying representative sentences. It can also be run via the TextSummarization action set in code. It computes the summary for each document in the data set, and it can also generate a single summary for the entire data set.   There are various techniques that are used for summarizing text. The purpose of the Text Summarization task is to perform an extractive summary. The software tries to identify meaningful sentences by using NLP (Natural Language Processing) techniques involving terms, entities, and noun groups. Another summarization technique you may have encountered is referred to as an abstractive summary that can use deep learning methods to analyze the text.   A fun fact is that the summarization result will be identical to the original document if it has only 2 sentences and you ask for a summary of 2 sentences. If you try this task on really short documents, you might be disappointed with the findings, but now you know why this would happen. Creating a summary of the entire short-document collection may be more valuable, and that’s what we’ll describe next.   The table in this example has hundreds of historical customer reviews of an electronics product, and I want to condense each review down to 2 sentences. Some of the reviews are several paragraphs in length and others are only a single sentence.   You can find the Text Summarization task in SAS Studio on Viya to get started.     The Text Summarization task in SAS Studio reduces content down to a maximum number of sentences that you can specify. The maximum number is capped at 32. A maximum number this large may be useful if you are summarizing chapters of a novel or scholarly abstracts. In this case, I’m processing consumers’ product reviews, so I chose to represent each review by only 2 sentences. Since I am also interested in the overall ‘sense’ of the reviews, I am also going to ask for a summary of the entire corpus.   Summarizing documents will produce different results depending on what features you generate for the task. You have a choice of parsing terms for the analysis, using entities and noun groups, or selecting both options as I have in this example. Try different combinations of these options with your data and see what options give you the best results for the kind of data you are processing.     This type of text summarization is different than a summary that would be created by a Large Language Model (LLM). Some analysts are wary of using LLMs to summarize documents since they may ‘hallucinate’ or just make stuff up, and that is not what you want. There are situations where LLMs may be useful when they are applied after a summarization is done.   Here is one document from my dataset. At the risk of getting a TL;DR from you, go ahead and read the original review and the subsequent summarized review. Do you think the summary did a good job capturing the important sentences?   Original Review:     Summary of Review:     When you experiment with your own text data, ask for a corpus summary and see what combination of parsing options provides you with the best result. This task is pretty nifty to have in your bag of tricks, but it's not designed for all situations. Remember this is only one of the many resources available to process text. I hope that you enjoyed reading about this task and encourage you to watch for additional articles.   Thanks for reading!     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS® Visual Text Analytics: How you can Create concept rules programmatically.   Hello, and welcome to my post! The purpose of this post is to provide helpful insights for working with text document collections. This post addresses questions students have posed recently that you may also find helpful when working with SAS® Visual Text Analytics.   In this installment we will consider "case sensitivity" in custom concept rules as well as how to create concept rules programmatically.   There are excellent sources of information already available on writing custom concept rules. Some sources are the product documentation or the SAS Support Community articles. These are great starting points if you are interested in learning the basics of writing concept and category rules.   Let’s start with a quick tip that you may not be aware of. Did you know that when you create your own concept rule it is not case sensitive? The following example concept rule identifies several types of desserts.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   It includes some capitalized words, but it will generate matches regardless of the actual case in the documents. There are four matches returned with the default rule behavior.     Suppose now you want to identify terms based on the specific case that is used in a rule.  With the Concepts Node open, right click the rule name that you want to be case sensitive and click “Match Case”. This will place an icon Aa next to the rule name as shown below. This easily distinguishes case-sensitive rules. Notice that now there are fewer matches in the text document with case sensitivity enabled for the rule (only two matches this time).     Using ActionSets   The next example from the product documentation creates three concept rules programmatically that work together as a unit.   The first section is the syntax for creating the three rules: COMPANY, PRODUCT, and TEST_PRED using data step. The first two Classifier rules are used in the third rule which is a PREDICATE_RULE type, also referred to as a “fact”. Notice the property “CASE_INSENSITIVE_MATCH” in the syntax reflecting the default behavior of concept rules not being case sensitive. You can remove this line if you want to make the rule case sensitive.   Each of the three rule blocks start with ENABLE:   options casport=5570 cashost="cloud.example.com"; cas casauto; libname mycas cas; data mycas.concept_rules_long; length config varchar(*); infile datalines delimiter='|' missover; input config$; ruleid=monotonic(); datalines; ENABLE:COMPANY FULLPATH:COMPANY:Top/COMPANY PRIORITY:COMPANY:10 CASE_INSENSITIVE_MATCH:COMPANY CLASSIFIER:COMPANY: Microsoft CLASSIFIER:COMPANY: Amazon CLASSIFIER:COMPANY: Google ENABLE:PRODUCT FULLPATH:PRODUCT:Top/PRODUCT PRIORITY:PRODUCT:10 CASE_INSENSITIVE_MATCH:PRODUCT CLASSIFIER:PRODUCT: xbox CLASSIFIER:PRODUCT: windows CLASSIFIER:PRODUCT: chrome CLASSIFIER:PRODUCT: fire ENABLE:TEST_PRED FULLPATH:TEST_PRED:Top/TEST_PRED PRIORITY:TEST_PRED:10 CASE_INSENSITIVE_MATCH:TEST_PRED PREDICATE_RULE:TEST_PRED(company, product):(DIST_1, "_company{COMPANY}","_product{PRODUCT}") ENABLE:Top FULLPATH:Top:Top PRIORITY:Top:10 CASE_INSENSITIVE_MATCH:Top ; run;   These rules must be compiled before they can be used in the ActionSets. The syntax is shown below. The section starting with textRuleScore.applyConcept about halfway down the following example, takes the compiled rules and extracts the matched documents, placing them in the output table. The documents were read from the table named apply_concept_text. This is useful for quickly applying concept rules to documents that need to be processed on an ad-hoc basis, or for processing large document collections in batch mode.   proc cas; builtins.loadActionSet / actionSet="textRuleDevelop"; builtins.loadActionSet / actionSet="textRuleScore"; textRuleDevelop.compileConcept / casOut={name="outli", replace=TRUE} ruleid="ruleid" config="config" table={name="concept_rules_long"}; run; textRuleScore.applyConcept / casOut={name="out_concept", replace=TRUE} docId="docid" factOut={name="out_fact", replace=TRUE} model={name="outli"} ruleMatchOut={name="out_rule_match", replace=TRUE} table={name="apply_concept_text"} text="text"; run; table.fetch / table={name="out_concept"}; run; table.fetch / table={name="out_fact"}; run; table.fetch / table={name="out_rule_match"}; run; quit;   The terms from the scored documents that are matched by the three rules are listed in the table below. The start and end positions of the matched text are in the table. The _fact_argument_ column refers to the individual supporting rules that found a match which was then used in the main TEST_PRED predicate rule type.     For more details on how to work with concept rules, be sure to sign up for our e-learning or live-web classes on SAS Visual Text Analytics. The previous documentation link above provides lots more information that may be spotlighted in an upcoming post.   Thanks for reading and keep texting!     Find more articles from SAS Global Enablement and Learning here. ... View more

  What if you have to process documents written in different languages? The purpose of this post is to present some tips on using SAS Viya Visual Text Analytics to ensure you get reliable results.   Multi-Lingual text appears in documents in two main ways or categories:   Some documents combine multiple languages. Each document is written in a specific language.   The first category did not seem to pose a problem in the situation where, for example, an English document includes the French phrase “c’est la vie” and Spanish “Muchas felicidades a los dos.” These phrases were detected by the Concepts node of my text analysis project, but when an input document contains multiple languages, the results might not be accurate.   Just to illustrate, here is a Visual Text Analytics Concepts node test of the two previous sentences using the predefined nlpNounGroup concept in an English language-based project. (See my post on creating Visual Text Analytics projects for background on how to use the interface.)   Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   There are 5 matched noun groups spanning 3 languages highlighted in the Test Sample Text document. So, we see that Noun Groups for at least some languages are detected!   Now consider the case where each document is written in a specific language. Multiple languages may appear in document collections related to health care, physicians’ notes, survey responses, social media comments, world news, tourism, or insurance. Documents written in different languages add complexity to the analysis of the document collection because different languages use different linguistic structures.  For example, certain languages follow S-O-V (subject – ob ject – verb) structures while others use S-V-O or V-S-O order (Imagine a Yoda voice saying “Hmm.. away put your weapon”)   Not all languages follow the structure of the English language. English has tenses and verb conjugations, but Chinese and Polynesian languages for example, do not. Subject-verb agreement exists in English but not in Chinese. A mix of languages processed together may present challenges. If you have to analyze many documents in different languages, it makes sense to process them according to their own kind for the best results.   If you are processing text data that has documents in multiple languages, SAS Viya Visual Text Analytics easily identifies languages present in the corpus and helps you create separate pipelines for each language in the collection. You use the Data node to select documents for your project. Starting with SAS Viya 2021.2.4, automatic language detection is available in the Data node!   I created a test document collection of about 90 documents by adding several paragraphs from six different languages related to opera houses, NLP research, news articles, etc. I put these ‘documents’ into a spreadsheet with one document per row and imported it into Viya.   There is a drop-down selection to activate automatic language detection in Model Studio. Click the "Detect Languages" button.     This launches a process that reads the corpus, identifies languages, and flags each document with a language code.     Notice that a new language identifier variable _language_ is added to the original data. A best practice is to set this new _language_ variable as a display variable to see it in subsequent nodes of your project.     By examining the data table, we can check that the detected language code is correctly assigned for each document.     Within a Pipeline, you can choose one language from the list of identified languages to function as the primary language for your project. I do not recommend doing this unless your documents only have occasional multi-language phrases.   By selecting the English language in this example, the entire multi-language document collection is processed from the perspective of English.     If I run this pipeline, the standard predefined nlp concepts (like noun groups) find matches for several languages as shown in the first screen capture.   The same appears true initially for the Text Parsing and Topics nodes. A closer look at the system generated topics for the English pipeline reveals that the discovered topics are comprised of terms that would be considered "stop words" in each language. These generated topics were just based on the frequency of terms that were in the small test document collection. From the English perspective, these terms could have been nothing more than industry-specific terminology such as internal product categories.     Selecting the third topic from the top and displaying the matched documents reveals that although the topic did identify French documents, it was based on uninformative terms (de, la, des, les).     So, using English to process documents from multiple languages is clearly not a best practice!   Now let’s build a language specific pipeline for just the French documents using the properties of the Data node.     The topics generated for the French language contain actual French words rather than the uninformative words detected in the previous English pipeline. A best practice is to create additional pipelines and subset the documents by language for further processing.     An alternate approach:   If you have to process multiple languages, you may want to first translate all documents to English, and then run the NLP (Natural Language Processing). This approach requires that you pre-process the documents with one of many available commercial translation software offerings, but this may yield good overall results. (Be confident that you trust the accuracy of the translation especially if domain-specific terms and phrases exist in the documents).   There is a cas action that can programmatically add the language identifier to an output table. Use this output table to feed select languages into your translation software. Then combine the translated results into a table for your text analytics project.   The following code can be run from SAS Studio and creates an output table with a language identifier for each document stored as the out_language table.   proc cas; /*1*/ session casauto; builtins.loadActionSet / /*2*/ actionSet="textManagement"; run; textManagement.identifyLanguage / /*3*/ casOut={caslib="casuser",name="out_language", replace=TRUE} docId="row" table={name="query"} text="document"; run; table.fetch / /*4*/ table={name="out_language"}; run; quit;   Section 1 establishes a session with cas. Section 2 loads the textManagement action set. Section 3 writes the ISO 639-1 language code in the output table. Section 4 displays language identification information.   The _language_ variable in the following table was created by section 3 of the previous code example:     More details are available in the product documentation   I hope the automatic language detection capability comes in useful for those of you working in multiple languages. Thanks for reading!     Find more articles from SAS Global Enablement and Learning here. ... View more