The purpose of this post is to describe how to enhance concept rules to retrieve better focused information from a collection of documents.   We'll explore concept rules and ideas for refining them in the context of an analyst processing a large number of customer complaints at a large bank.   Better concept matches start with interactive exploration in SAS Visual Text Analytics.   For an introduction to the capabilities in SAS Visual Text Analytics see my post.   Concepts are used to extract relevant information from documents. This analytical journey starts with exploratory analysis. When processing customer complaint documents, concepts help us: identify conditions where someone needs help, identify opportunities to improve customer service, discover social media comments to address before a complaint goes viral, and more.   First some freebie tips.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Predefined concepts are automatically identified by checking a box in the properties panel. These use Natural Language Processing (NLP) techniques to identify concepts such as measures, money, organizations, people, places and more. The nlpMeasures concept for example, returns matches for terms such as (2 years, two days, 5 mg, $250., 8 feet). The nlpMoney concept matches anything related to currency such as ($2, 2.50€, 40 yen, 20 pounds).   You can combine predefined concepts with new custom concepts to easily identify documents having specific information. Keep these freebies in mind to assist you with building your own concepts.   For example, the rule argument (SENT,”_c{cashed}”, ”nlpMoney”}) returns matches for any document that has the term “cashed” and any currency amount in the same sentence. SAS will recognize currency whether it is U.S. dollars, British pounds, etc.   Your mission…   Let’s consider an example scenario.   You work for a bank and one of your responsibilities is to evaluate 10,000 recent customer complaints that came in through various channels and are available in computer readable text. Your mission is to investigate complaints related to the handling of cash and recommend actions your company might take to improve the customer’s experience.   What is the first thing you would try?   Exactly, Visual Text Analytics!   Open the documents in SAS Visual Text Analytics, run the text parsing node and look at the kept terms and list of documents. You might then try searching the documents for the word cash and see if this gives you any ideas. Your search returns 700 documents and you see phrases in the documents like:   Cash a check Cash advance Cash back Cash price Cash deposit / withdrawal     Remember these pointers to structure an effective search:   Placing the "+" in front of a word requires the word to be in a document. A term without a preceding + returns the term if it exists in the document but it does not have to be in the document. A "–" in front of a word finds documents that do not have that term. Place a * at the beginning, middle or end of a search query to get wildcard matches. The query term +*ion returns terms such as exception or action. Quotation marks return the specific content in between such as “fraudulent check”.   You might then shift your attention to the kept terms list at the top and search for cash again and notice the following.      In this display, cash is recognized as a noun, verb, proper noun, and in a noun group by the NLP process. You decide to look at the verb cash in the term map next.     This map shows the terms related to cash based on information gain. One path from this chart suggests that some documents contain (cash, check, issue) and do not contain (offer or deposit) where the ~ sign indicates “not”. These ideas represent new insights to consider as you construct new custom concept rules.   In the parsing node you can click the similarity scores icon, remove “cash” from the kept terms filter, and notice the additional new terms that appear. “Check fraud” and “forged check” didn’t initially come to mind when exploring cash, but you may want to expand your investigation for occurrences of fraud related to cash!     Concepts Node   Next on your journey, you add and run a concepts node to identify predefined concepts in the documents and then create a custom concept looking for various morphological expansions of the term “cash”, with fraud and check. The initial LITI (language interpretation for textual information) CONCEPT and CLASSIFIER rules follow.     These results look promising, but a lot of documents matched. You wonder if there might be some additional ideas for rules. Click the Autogenerate concept rules icon in the concept that has just 3 simple rules.     The fresh new rules were written to the sandbox. The generated rules will need more explanation especially if the LITI syntax is new to you. We plan to cover some of these in the next post, but here is a sample of the autogenerated rules in the meantime.   After building concept rules, the resulting concepts model can score the current or new documents for concept matches. SAS Visual Analytics can also be used to report on the results.   If you can’t wait until the next post to make sense of these rules, reference this documentation, otherwise, I look forward to sharing some more insights with you in the next post! It's your move!       Find more articles from SAS Global Enablement and Learning here. ... View more

  A customer recently posed this question: “I know how to match a concept at the end of a sentence, but how can I match it at the conclusion of a document?” (For a review of concepts and other basics, see my previous post.) The purpose of this post is to match concepts near the end of a document using a technique that leverages multiple nodes. LITI is Language Interpretation for Textual Information, and it is the programming language used to code concept and category rules in SAS Visual Text Analytics.   Customer complaint call transcripts can end up being quite verbose. Sure, for starters, you could run a summary of the documents to get the main ideas being raised, and this would be a good thing to do. (see my post on summarizing documents). But isn’t there something else I could also try?   The complete customer interaction likely makes some progress over its duration. The follow-up actions or conclusions of the dialog can be expected to occur towards the end of the event, after the back-and-forth discussion has become more focused on the real issues. It may not be necessary to read through the entire interaction to gain some actionable insights.   You may be most interested in returning the matches for certain conditions that occur in the last 50 terms or so of an exchange between the customer and the call agent. The text near the close of the conversation likely holds information to help you determine any appropriate followup actions. Designing the structure of your rules thoughtfully can provide meaningful insights. You could also create a dashboard of these conclusions over time for extra credit.   Consider this example where a Concept rule finds matching terms occurring in a document based on a list of conditions and activities that are stored in supporting rules. In the consumer financial services area, you might have a concept rule containing LITI syntax like this:   CallCenterMatches   CONCEPT_RULE:(AND,(DIST_20,"_CONDITION_", "_c{_ACTIVITY_}"))   This rule matches documents that have an item from the Condition rule within 20 tokens of an item from the Activity rule. A match requires that at least one item from each table is in the document due to the use of the AND operator.   Two example tables below use the LITI language syntax for a table of conditions and a table of activities. The items from the columns can appear in any order. The @ symbol indicates morphological expansions of the term (all forms of a term like report, reports, reported, reporting…)   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Here are some of the 635 matches from SAS Visual Text Analytics of the CallCenterMatches rule with the activity name highlighted as the rule requested.     OK, this is great, but we already know how to use the CONCEPT_RULE and all its bells and whistles. What else can I do?   What if you have a long call transcript and only want to know what happens towards the end of the call? This kind of focus may do a better job than our CallCenterMatches rule on its own to identify followup actions with the customer. To do this, we’ll add a category rule to find a concept if it exists at the end of the interaction (call or chat or transcribed audio)   Next, we will create a category rule based on the concept and use an easily overlooked statement available in the Category node.   I can connect a categories node directly to a concepts node as shown in the right-hand branch of this pipeline, since both nodes can do their own parsing.     I added the following Categories rule so I could see only the matches at the end of the document. The syntax of the Categories node is different than the Concepts node, so this syntax is not valid for use in the Concepts node.   I created my new Category rule using this syntax:   (END_40, "[CallCenterMatches]")   The square brackets in double quotation marks tells the Categories node to use a concept rule rather than a term in the rule.   After applying the concept matches to the categories node in the last 40 tokens of these call reports, there are only 190 matches whereas previously there were 635 document matches in the concepts node.   This is the end of one of the matched documents with the highlighted conditions and actions terms within the final 40 tokens!     This is the description of the END_n Categories rule.   END_n (From the end of the document)   Takes a value for n and one or more arguments. Matches if each argument occurs within n tokens from the end of the document. For example, the rule (END_35, "conclusion") produces a match if conclusion is found within 35 words from the last token in the document. Note: Punctuation counts as a token. In some languages, words that include hyphens are counted as one word (for example, merry-go-round is one token).   So, there you have it! You can combine concept rules within categories to get specific matches based on your needs. There are many other hidden treasures in the land of LITI rules and who knows, I may reveal some more in posts yet to come!!!   Thanks for reading and let me know if you use a form of this rule.   BTW,   If you really want to learn this stuff, here is an exercise for you to try. See if you can identify what each of these rules will return. Do some rules not give you what you are really looking for? Do any of these rules return more than the information we are looking for? Read about the DIST_n, AND, OR, and _c operators for Concept rules for hints.   CONCEPT_RULE:(DIST_20, "_c{_CONDITION_}", "_ACTIVITY_") CONCEPT_RULE:(AND, "_c{_CONDITION_}", (OR, "_ACTIVITY_")) CONCEPT_RULE:(OR,(DIST_20, "_c{_CONDITION_}", "_c{_ACTIVITY_}")) CONCEPT_RULE:(AND,(DIST_20,(OR,"_c{_CONDITION_}", "_c{_ACTIVITY_}")))     Find more articles from SAS Global Enablement and Learning here. ... View more

  Hello, and welcome to my blog! The purpose of this blog is to encourage you to experiment with some of the options that are available for taking your Text Topic Modeling beyond the basics. If you are new to Visual Text Analytics, see my article on Getting Started with Text Analytics to get an idea of the capabilities.   Starting with a collection of documents, these can include customer call reports, patients’ comments, product reviews or movie recommendations, SAS provides the Natural Language Processing (NLP) capabilities that can turn the documents into useful insights.   The Topics node is usually inserted near the end of a text processing pipeline. This node provides you with controls that regulate the topic discovery process. Controls include setting cutoff thresholds for terms and documents. A lower threshold captures more items while a higher threshold captures fewer items. The threshold is based on the number of standard deviations above the mean term or document weight.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Note that when running a Topics node, some documents will likely not fit into any of the generated topics, and they will be accounted for in a group designated as “No Matching Topic”. Re-running the node with lower thresholds specified will result in fewer documents falling into the “No Matching Topic” group. These settings provide users with adequate controls for extracting reasonable topics when getting started with a text analysis project. Machine generated topics can be combined or split, and additional user-created topics can also be added.   To illustrate this point, I ran the topics node with the default settings, and it returned 11 topics. The right-most bar represents documents that did not fit into any discovered topic.     I then reduced the document cutoff control setting from 1 to 0.7 standard deviations, and now more documents are accounted for in the generated topics as shown below. The 5th bar from the right in the chart below represents the reduced number of documents with no matching topic. A good approach is to examine the results you get by spot checking the documents in your topics and then make adjustments to the thresholds until you are happy with the settings.     The Latent Dirichlet Allocation (LDA) Topic Model   The results above were generated under the default Singular Value Decomposition (SVD) algorithm also known as latent semantic indexing (LSI). If you wish to experiment with generating topics using a different algorithm, the Latent Dirichlet Allocation (LDA) Topic Model technique is also available. Running LDA for topic generation may produce different topics and document assignments. It utilizes Bayesian and Expectation Maximization techniques. Now, let's discuss how we'd implement it by calling a CAS Action in syntax.   Implementing the Latent Dirichlet Allocation algorithm: ldaTopic.ldaTrain action creates topics from a document collection using just a document ID, input table, and the variable name containing the documents at a minimum. It defaults to creating only 2 topics, but this can be changed using the k= parameter. In my movie description example, I asked for k=5 topics to be created.   From the documentation, the minimum syntax needed to run with default settings is shown here. This code would run in SAS Studio.   /* Connect to CAS and load the data table. */ options casport=5570 cashost="cloud.example.com"; /**/ cas casauto; caslib _all_ assign; proc cas; /**/ session casauto; table.loadTable result=r / /**/ caslib="Library_name_to_use" path="Path_to_my_file" casOut={name="CAS_file_name", replace=true}; run; /* Run the ldaTrain action – this section defines the output table names */ ldaTopic.ldaTrain / /**/ casOut={name="Topics", replace=TRUE } docDistOut={name="DocDist", replace=TRUE } /* This section defines the input table name, id variable and text variable */ docId="D_ID_variable_name" table={name="Table_name"} text={{name="T_variable_name"}}; run; quit; After running code based on this outline, the following output table shows the proportion of terms for each topic that was found in a document. There were 5 topics generated, numbered 0-4, and each document id has a proportion value for each topic.     The following output table shows what topic (0-4) each term was placed into as well as the assigned probability the term will represent the specific topic.     As I mentioned before, I added a parameter k=5 to return 5 topics rather than the default 2 topics when I ran against a movie description document collection. The hyperparameter Alpha with a minimum value of 0 and a default value of .1 specifies the Dirichlet hyperparameter for the document's topic proportion. Beta specifies the Dirichlet hyperparameter for the topic distribution.     Additional optional parameters described in the product documentation let you do additional configuration including adding stop words, modifying the number of iterations, entity identification and stemming.   You can run LDA action sets from CASL, PYTHON, R and Lua, as well as from a SAS STUDIO task for those of you who might prefer to work with a user interface. Let’s take a quick look at SAS Studio for inspiration.     On the left you see the text parsing and topic discovery selection. In the center, the Options tab is selected as is the LDA radio button. On the right you see the generated code calling the ldaTopic.ldaTrain action. Even if you are a ‘coder’, the SAS Studio task is a great way to quickly get a working copy of code showing correct syntax and the placement of additional options.   The SAS Studio tasks are straight forward easy to use and can be an inspiration on your way to becoming a text analytic practitioner!   In a subsequent blog we will explore scoring these models and ways to deploy them in a production environment. Until then, I wish you much success with your analysis and may all your topics be true!     ... View more

Hello, and welcome to my post! The purpose of this post is to showcase the Text Summarization task in SAS Studio (in less than 5000 pages 😒).   The volume of text data seems to increase each day, and we often encounter information overload in our daily activities. It seems that everyone has something to say, whether it is in a video on how to clean a yucky coffee pot, another breaking news report, customer product complaints, or transcribed audio. How do you find meaning in the vast amount of data that is growing every day?    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS Visual Text Analytics (VTA) users certainly have the tools needed to find answers buried in document collections. (See my post on Getting Started with Text Analytics).   The Text Summarization task in SAS Studio is a VTA tool that provides functionality to summarize a document. Ever think “TL;DR” when looking at social media posts? I know that you know this but… TL;DR is an abbreviation for "too long; didn't read."   You probably have found yourself in a situation where you could use some help getting a quick summary of a document collection into a reasonable number of sentences. Especially if the alternative is going through “billions” of pages of text.   The Text Summarization task in SAS Studio is one possible way to summarize documents quickly and easily. It uses natural language processing (NLP) techniques to summarize a document by identifying representative sentences. It can also be run via the TextSummarization action set in code. It computes the summary for each document in the data set, and it can also generate a single summary for the entire data set.   There are various techniques that are used for summarizing text. The purpose of the Text Summarization task is to perform an extractive summary. The software tries to identify meaningful sentences by using NLP (Natural Language Processing) techniques involving terms, entities, and noun groups. Another summarization technique you may have encountered is referred to as an abstractive summary that can use deep learning methods to analyze the text.   A fun fact is that the summarization result will be identical to the original document if it has only 2 sentences and you ask for a summary of 2 sentences. If you try this task on really short documents, you might be disappointed with the findings, but now you know why this would happen. Creating a summary of the entire short-document collection may be more valuable, and that’s what we’ll describe next.   The table in this example has hundreds of historical customer reviews of an electronics product, and I want to condense each review down to 2 sentences. Some of the reviews are several paragraphs in length and others are only a single sentence.   You can find the Text Summarization task in SAS Studio on Viya to get started.     The Text Summarization task in SAS Studio reduces content down to a maximum number of sentences that you can specify. The maximum number is capped at 32. A maximum number this large may be useful if you are summarizing chapters of a novel or scholarly abstracts. In this case, I’m processing consumers’ product reviews, so I chose to represent each review by only 2 sentences. Since I am also interested in the overall ‘sense’ of the reviews, I am also going to ask for a summary of the entire corpus.   Summarizing documents will produce different results depending on what features you generate for the task. You have a choice of parsing terms for the analysis, using entities and noun groups, or selecting both options as I have in this example. Try different combinations of these options with your data and see what options give you the best results for the kind of data you are processing.     This type of text summarization is different than a summary that would be created by a Large Language Model (LLM). Some analysts are wary of using LLMs to summarize documents since they may ‘hallucinate’ or just make stuff up, and that is not what you want. There are situations where LLMs may be useful when they are applied after a summarization is done.   Here is one document from my dataset. At the risk of getting a TL;DR from you, go ahead and read the original review and the subsequent summarized review. Do you think the summary did a good job capturing the important sentences?   Original Review:     Summary of Review:     When you experiment with your own text data, ask for a corpus summary and see what combination of parsing options provides you with the best result. This task is pretty nifty to have in your bag of tricks, but it's not designed for all situations. Remember this is only one of the many resources available to process text. I hope that you enjoyed reading about this task and encourage you to watch for additional articles.   Thanks for reading!     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS® Visual Text Analytics: How you can Create concept rules programmatically.   Hello, and welcome to my post! The purpose of this post is to provide helpful insights for working with text document collections. This post addresses questions students have posed recently that you may also find helpful when working with SAS® Visual Text Analytics.   In this installment we will consider "case sensitivity" in custom concept rules as well as how to create concept rules programmatically.   There are excellent sources of information already available on writing custom concept rules. Some sources are the product documentation or the SAS Support Community articles. These are great starting points if you are interested in learning the basics of writing concept and category rules.   Let’s start with a quick tip that you may not be aware of. Did you know that when you create your own concept rule it is not case sensitive? The following example concept rule identifies several types of desserts.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   It includes some capitalized words, but it will generate matches regardless of the actual case in the documents. There are four matches returned with the default rule behavior.     Suppose now you want to identify terms based on the specific case that is used in a rule.  With the Concepts Node open, right click the rule name that you want to be case sensitive and click “Match Case”. This will place an icon Aa next to the rule name as shown below. This easily distinguishes case-sensitive rules. Notice that now there are fewer matches in the text document with case sensitivity enabled for the rule (only two matches this time).     Using ActionSets   The next example from the product documentation creates three concept rules programmatically that work together as a unit.   The first section is the syntax for creating the three rules: COMPANY, PRODUCT, and TEST_PRED using data step. The first two Classifier rules are used in the third rule which is a PREDICATE_RULE type, also referred to as a “fact”. Notice the property “CASE_INSENSITIVE_MATCH” in the syntax reflecting the default behavior of concept rules not being case sensitive. You can remove this line if you want to make the rule case sensitive.   Each of the three rule blocks start with ENABLE:   options casport=5570 cashost="cloud.example.com"; cas casauto; libname mycas cas; data mycas.concept_rules_long; length config varchar(*); infile datalines delimiter='|' missover; input config$; ruleid=monotonic(); datalines; ENABLE:COMPANY FULLPATH:COMPANY:Top/COMPANY PRIORITY:COMPANY:10 CASE_INSENSITIVE_MATCH:COMPANY CLASSIFIER:COMPANY: Microsoft CLASSIFIER:COMPANY: Amazon CLASSIFIER:COMPANY: Google ENABLE:PRODUCT FULLPATH:PRODUCT:Top/PRODUCT PRIORITY:PRODUCT:10 CASE_INSENSITIVE_MATCH:PRODUCT CLASSIFIER:PRODUCT: xbox CLASSIFIER:PRODUCT: windows CLASSIFIER:PRODUCT: chrome CLASSIFIER:PRODUCT: fire ENABLE:TEST_PRED FULLPATH:TEST_PRED:Top/TEST_PRED PRIORITY:TEST_PRED:10 CASE_INSENSITIVE_MATCH:TEST_PRED PREDICATE_RULE:TEST_PRED(company, product):(DIST_1, "_company{COMPANY}","_product{PRODUCT}") ENABLE:Top FULLPATH:Top:Top PRIORITY:Top:10 CASE_INSENSITIVE_MATCH:Top ; run;   These rules must be compiled before they can be used in the ActionSets. The syntax is shown below. The section starting with textRuleScore.applyConcept about halfway down the following example, takes the compiled rules and extracts the matched documents, placing them in the output table. The documents were read from the table named apply_concept_text. This is useful for quickly applying concept rules to documents that need to be processed on an ad-hoc basis, or for processing large document collections in batch mode.   proc cas; builtins.loadActionSet / actionSet="textRuleDevelop"; builtins.loadActionSet / actionSet="textRuleScore"; textRuleDevelop.compileConcept / casOut={name="outli", replace=TRUE} ruleid="ruleid" config="config" table={name="concept_rules_long"}; run; textRuleScore.applyConcept / casOut={name="out_concept", replace=TRUE} docId="docid" factOut={name="out_fact", replace=TRUE} model={name="outli"} ruleMatchOut={name="out_rule_match", replace=TRUE} table={name="apply_concept_text"} text="text"; run; table.fetch / table={name="out_concept"}; run; table.fetch / table={name="out_fact"}; run; table.fetch / table={name="out_rule_match"}; run; quit;   The terms from the scored documents that are matched by the three rules are listed in the table below. The start and end positions of the matched text are in the table. The _fact_argument_ column refers to the individual supporting rules that found a match which was then used in the main TEST_PRED predicate rule type.     For more details on how to work with concept rules, be sure to sign up for our e-learning or live-web classes on SAS Visual Text Analytics. The previous documentation link above provides lots more information that may be spotlighted in an upcoming post.   Thanks for reading and keep texting!     Find more articles from SAS Global Enablement and Learning here. ... View more

  What if you have to process documents written in different languages? The purpose of this post is to present some tips on using SAS Viya Visual Text Analytics to ensure you get reliable results.   Multi-Lingual text appears in documents in two main ways or categories:   Some documents combine multiple languages. Each document is written in a specific language.   The first category did not seem to pose a problem in the situation where, for example, an English document includes the French phrase “c’est la vie” and Spanish “Muchas felicidades a los dos.” These phrases were detected by the Concepts node of my text analysis project, but when an input document contains multiple languages, the results might not be accurate.   Just to illustrate, here is a Visual Text Analytics Concepts node test of the two previous sentences using the predefined nlpNounGroup concept in an English language-based project. (See my post on creating Visual Text Analytics projects for background on how to use the interface.)   Select any image to see a larger version. Mobile users: If you do not see this image, scroll to the bottom of the page and select the "Full" version of this post.   There are 5 matched noun groups spanning 3 languages highlighted in the Test Sample Text document. So, we see that Noun Groups for at least some languages are detected!   Now consider the case where each document is written in a specific language. Multiple languages may appear in document collections related to health care, physicians’ notes, survey responses, social media comments, world news, tourism, or insurance. Documents written in different languages add complexity to the analysis of the document collection because different languages use different linguistic structures.  For example, certain languages follow S-O-V (subject – ob ject – verb) structures while others use S-V-O or V-S-O order (Imagine a Yoda voice saying “Hmm.. away put your weapon”)   Not all languages follow the structure of the English language. English has tenses and verb conjugations, but Chinese and Polynesian languages for example, do not. Subject-verb agreement exists in English but not in Chinese. A mix of languages processed together may present challenges. If you have to analyze many documents in different languages, it makes sense to process them according to their own kind for the best results.   If you are processing text data that has documents in multiple languages, SAS Viya Visual Text Analytics easily identifies languages present in the corpus and helps you create separate pipelines for each language in the collection. You use the Data node to select documents for your project. Starting with SAS Viya 2021.2.4, automatic language detection is available in the Data node!   I created a test document collection of about 90 documents by adding several paragraphs from six different languages related to opera houses, NLP research, news articles, etc. I put these ‘documents’ into a spreadsheet with one document per row and imported it into Viya.   There is a drop-down selection to activate automatic language detection in Model Studio. Click the "Detect Languages" button.     This launches a process that reads the corpus, identifies languages, and flags each document with a language code.     Notice that a new language identifier variable _language_ is added to the original data. A best practice is to set this new _language_ variable as a display variable to see it in subsequent nodes of your project.     By examining the data table, we can check that the detected language code is correctly assigned for each document.     Within a Pipeline, you can choose one language from the list of identified languages to function as the primary language for your project. I do not recommend doing this unless your documents only have occasional multi-language phrases.   By selecting the English language in this example, the entire multi-language document collection is processed from the perspective of English.     If I run this pipeline, the standard predefined nlp concepts (like noun groups) find matches for several languages as shown in the first screen capture.   The same appears true initially for the Text Parsing and Topics nodes. A closer look at the system generated topics for the English pipeline reveals that the discovered topics are comprised of terms that would be considered "stop words" in each language. These generated topics were just based on the frequency of terms that were in the small test document collection. From the English perspective, these terms could have been nothing more than industry-specific terminology such as internal product categories.     Selecting the third topic from the top and displaying the matched documents reveals that although the topic did identify French documents, it was based on uninformative terms (de, la, des, les).     So, using English to process documents from multiple languages is clearly not a best practice!   Now let’s build a language specific pipeline for just the French documents using the properties of the Data node.     The topics generated for the French language contain actual French words rather than the uninformative words detected in the previous English pipeline. A best practice is to create additional pipelines and subset the documents by language for further processing.     An alternate approach:   If you have to process multiple languages, you may want to first translate all documents to English, and then run the NLP (Natural Language Processing). This approach requires that you pre-process the documents with one of many available commercial translation software offerings, but this may yield good overall results. (Be confident that you trust the accuracy of the translation especially if domain-specific terms and phrases exist in the documents).   There is a cas action that can programmatically add the language identifier to an output table. Use this output table to feed select languages into your translation software. Then combine the translated results into a table for your text analytics project.   The following code can be run from SAS Studio and creates an output table with a language identifier for each document stored as the out_language table.   proc cas; /*1*/ session casauto; builtins.loadActionSet / /*2*/ actionSet="textManagement"; run; textManagement.identifyLanguage / /*3*/ casOut={caslib="casuser",name="out_language", replace=TRUE} docId="row" table={name="query"} text="document"; run; table.fetch / /*4*/ table={name="out_language"}; run; quit;   Section 1 establishes a session with cas. Section 2 loads the textManagement action set. Section 3 writes the ISO 639-1 language code in the output table. Section 4 displays language identification information.   The _language_ variable in the following table was created by section 3 of the previous code example:     More details are available in the product documentation   I hope the automatic language detection capability comes in useful for those of you working in multiple languages. Thanks for reading!     Find more articles from SAS Global Enablement and Learning here. ... View more

Hello, and welcome to my post! The purpose of this post is to introduce a technique that can be used to generate clusters or groupings of documents from a text document collection.   While generating text topics provides insight into topics occurring in documents, clustering documents organizes your document collection in a different way.   In a previous post, I introduced Text Mining concepts and used SAS Visual Text Analytics in a no code or point-and-click environment to extract information from a document collection. Now, let's explore a way of clustering a document collection using code and capabilities available in SAS Viya.   If you are going to run clustering on a document collection, the objective is to have each document placed into the best cluster segment that fits the content of the document. A document will be counted only once and will be found in only one segment. Most documents contain multiple topics, but the clustering of documents attempts to put similar documents together based on the best, most commonly occurring theme.   I am going to run the tmMine action of the textMining action set to represent a document collection as a matrix of numbers. A term by document matrix almost always has more terms than documents. It has many empty cells and is considered to be a sparse matrix.   This transformation is done by creating SVD (singular value decomposition) projections to describe each document by a vector of numbers. The decomposition effectively reduces the dimension of the assignment problem and makes it easier to classify the documents into like groups. SVD is a technique that finds a best, reduced-dimension space to approximate an original matrix, or in our case - the term by document matrix from the document collection.  If you are interested in more details on Singular Value Decomposition refer to SAS Help Center: Text Mining Details   We’ll use the k-means algorithm to put documents into clusters based on the values generated by the projections. More details on singular value decomposition in Text Analysis is given below.   I’ll then use SAS Visual Analytics to combine the original documents with their cluster assignment and assess the results. We’ll use the same drug reports data set of patient's comments on reactions to prescription drugs here that we introduced and described in the previous post.   Note: In Viya, there is a dataSegment action in the Smart Data action set that will cluster numeric or text data. This is an alternate approach you could try on smaller document collections. Since every document collection is different, compare the results of various approaches to see what technique works best for your data.   Process the document collection   Viya gives you a default stop list of words to exclude from text analysis in the ReferenceData caslib if you have a license for Text Analytics. The following code loads this default stop list, and then uses the tmMine action to read the drug_reports data, create 15 topic dimensions (SVD's) and save the numeric projections of the documents in a table named docPro. Topics are stored in a table named topics.   /* Load the documents and create the SVD projections */ cas mySession sessopts=(caslib=casuser timeout=1800 locale="en_US"); cas; caslib _all_ assign; /* load the provided default stoplist */ proc cas; loadtable caslib="ReferenceData" path="en_stoplist.sashdat"; run; quit; /* load the documents into memory */ proc cas; table.fetch / table={caslib="Public", name="Drug_Reports"}; quit; /* run the tmMine action to create 15 text topics */ /* and save the results as docPro for clustering */ proc cas; loadactionset "textMining"; action tmMine; param docId="id" documents={caslib="Public", name="Drug_Reports"} text="drugreport" /* column name of dataset that has the documents */ nounGroups= False tagging = True stemming= True stopList ={ name="en_stoplist"} /* list of terms to exclude from analysis */ parseConfig={name="config", replace=TRUE} parent ={ name="parent",replace=TRUE} offset ={name="offset",replace=TRUE} terms ={ name="terms", replace=TRUE} /* list of terms used in Zipf plot */ reduce=4 /* exclude terms if they are in fewer than 4 documents */ k=15 /* asking for 15 topic dimensions to be computed */ docPro ={ name="docpro", replace=TRUE} /* document projections for clustering*/ topics ={ name="topics", replace=TRUE} /* table of generated topics */ u ={ name="svdu", replace=TRUE} numLabels=5 /* ask for 5 descriptive terms to label each topic */ topicDecision=True ; run; quit;   Table showing derived topics       Document projections from the docPro table to be used to create cluster segments   The document ID is shown on the left and some of the 15 projection columns are shown in the resulting document projection table below.     Cluster the documents   Using the generated document projections, we will cluster the documents into an arbitrary selection of 10 segments using the k-means method available in proc fastclus. K-means is the default clustering method, so we do not have to state it specifically in the code. Other clustering methods are also available. We choose not to include a step to first standardize the projections prior to clustering in this case since they were created using singular value decomposition.   /* create clusters using K-means from the SVD results */ proc fastclus data=CASUSER.DOCPRO maxclusters=10 out=CASUSER.clusters; var _Col1_ _Col2_ _Col3_ _Col4_ _Col5_ _Col6_ _Col7_ _Col8_ _Col9_ _Col10_ _Col11_ _Col12_ _Col13_ _Col14_ _Col15_; run;   Results showing document id at left and the assigned cluster at right.       Merge clusters with original documents.   Now, merge the text with the cluster number to prepare for easy evaluation of the results. Use the "distinct" parameter to match tables on document ID to return one match per document.   /* Use Fedsql to combine the segment numbers with their text documents */ PROC FEDSQL SESSREF=mySession; CREATE TABLE CASUSER."results" AS SELECT DISTINCT t1."CLUSTER", t1."ID", t2."DrugReport" FROM CASUSER."CLUSTERS" t1 INNER JOIN CASUSER."DRUG_REPORTS" t2 ON (t1."ID" = t2."ID") ; QUIT;   Examine the results.   From Visual Analytics, I created a report to show the frequency of documents in clusters. (steps to create the report are not shown). It also shows the text for documents in each cluster from the results table created in the merge. For this report, I combined a text object and a bar chart object to create an interactive report that displays the documents in each cluster. Hovering over a single line of text will display the entire document in the report for convenience. (hovering action is not shown in the example)       Selecting useful terms for document analysis (Zipf's law).   A plot of terms by frequency is called a Zipf plot based on Zipf's law. It can be used to help you decide what terms are useful to include in your analysis. The plot follows a power law where the most frequently occurring term occurs twice as often as the second most frequently occurring term, and three times more often as the next term etc. Terms that occur in every document are not useful in understanding the document collection. Terms that occur in only a few documents are also not useful. The terms appearing in between these extremes are most helpful.   This first display shows a Zipf plot and a linked word cloud. The shape of the plot shown is what we would expect for a typical document collection in the English language. Not all terms appear in the word cloud since multiple terms have the same frequency counts and only one term is displayed. The largest words in the word cloud occur the most frequently and do not provide much useful insight into the collection on patient’s experience with prescribed drugs.         The following report shows sample words that occur in the highlighted frequency range between 60 to 80 or so and may provide more intuition into understanding the document collection. The word cloud now shows terms within the selected range in the Zipf plot.       You can apply the selection in the Zipf plot as a filter, and then use the selected terms as a start list for further detailed analysis with specialized collections. If you use a start list, only the terms in that list are included in the analysis. Except for special circumstances, the best practice is to use a stop list, which excludes terms from analysis.   In the first post, I demonstrated how topic information can be extracted from a collection of documents using the SAS Visual Text Analytics application. In this post, we see how to first create topics using the Text Mining Action Set, then create document clusters and finally use SAS Visual Analytics to explore the results.   In my next post, I plan to show how you can use custom concepts using both SAS Visual Text Analytics and in code. In the meantime, I hope you enjoyed this post and have some fun experimenting with various techniques to cluster documents.  Text analytics can be fun. Keep texting! ... View more

  The purpose of this post is to introduce text analytics to anyone new to the topic who wants to understand what it is and what it does. After reading this post you will have a better idea of what analysts are saying when you hear them talk about text analytics concepts. After all, who doesn’t want to participate intelligently in discussions about interesting analytic topics?   This post is suitable reading for Business Leaders, Analysts from other disciplines, Sales Executives, Systems Engineers and anyone with general interest in this topic. Subsequent post in this series will describe and illustrate applications of text analysis. The concepts presented here provide a foundation that will help you get full value from them and your software.   SAS® Visual Text Analytics analyzes unstructured data, such as text containing customer reviews, social media posts, survey responses, city planning documents, medical records, news feeds, research abstracts, tweets, transcribed phone calls, well you get the idea!   Text is considered “unstructured” in the world of data processing since it doesn’t come in the structured “rows and columns” data format that computers process. It’s easy to analyze basic data in a spreadsheet, but hard to analyze annual reports of all companies in a particular industry sector. You can’t do advanced analytics of free-form text data on a computer without it being represented in a structured format first. This is one of the things that NLP (Natural Language Processing) does. Text represented in this structured format (one example: as a term/document frequency matrix) becomes a building block used by machines to analyze documents.                       Information retrieval, exploratory analysis, topic derivation, research, concept creation, category and predictive modeling are some aspects of analyzing text.   You can build text models using your choice of a graphical interface or code. This post shows output of text models generated by the SAS® Model Studio graphical interface.   Let’s get started   First, the raw readable text data (including combinations of pdf documents collections, spreadsheets, RTF text, etc.) needs to be imported into the software. Viya® streamlines the work of importing your documents from a single data source or from directories into the cloud for processing. Some of the data import options are shown here:                         Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Following this, relevant information is extracted from the text documents by parsing and applying Natural Language Processing (NLP). To understand this idea, imagine that you want to parse this paragraph – what do you notice? It is made up of characters and punctuation marks separated by spaces, and we think of these characters as words. In text analytics, groupings of characters between spaces are called ‘tokens’. It may also make sense to group several tokens together to represent a term that has specific meaning (i.e. body mass index, or northern lights). These are referred to as Noun Groups in the software.   Singular and plural versions of nouns and verbs are grouped together as synonyms along with any misspelled words if that option is enabled.   In text analytics, after tokens are extracted, more sophisticated NLP techniques continue to build the analysis and extract terms, concepts, entities and atomic facts that will be discussed later. Not every word found in the document collection should be included in the analysis. For example, if a word occurs in every document, it provides no useful insight into the document collection and should be dropped. Examples of these kinds of common words are; “and, the, a, some”, but the words that are dropped will differ depending on each document collection. If a word occurs in only one document it should not be used for analysis, but it can still be found in a keyword search of documents. You can add terms to a stop list which is used to exclude uninformative terms from analysis.   Some SAS® Visual Text Analytics capabilities and ideas to be aware of are: Text Parsing. Examples of extracted or parsed parts of speech are nouns, verbs, adjectives, noun groups, determiners (the, a, some). These can give insight into the composition of the document collection. Do you expect higher frequencies of nouns than verbs in the collection? NLP processing determines and assigns the correct part of speech so that a word like “train” is tagged appropriately for each occurrence: - as a verb (train the new employees) - or noun (the train arrived on time) in context. There are 9 Predefined Concepts that can be identified. An example of a predefined concept is a measure that identifies any quantities listed in documents such as 5 mg, 1 foot, 20 liters, 2 weeks, 10 pounds etc. They are used to improve document searches to let you quickly find documents that match specific concepts. The word cloud below generated from one of my projects lists the 9 available predefined concepts found in that document collection. The Topics node lets you to modify and create topics that can be used to generate more effective text analytics models. You can think of a topic as a theme or idea made up of terms and concepts occurring often in a document collection. A single document will likely have multiple topics such as news, weather, sports, each described by several descriptive terms. Topic Discovery functionality in the software will automatically decide how many topics to generate for you based on thresholds and computed weights for each term indicating the influence of the term on a topic, and documents. You can merge topics and create user topics from terms that you select. This is useful where specific language and terms apply to documents from a particular discipline. Businesses in the medical, insurance, manufacturing, legal, finance fields all tend to use industry specific terms that can make up a user topic to help make sense of a document collection. Each document receives a score for each topic that can be used in further explorations or to apply scores to identify topics in new documents. The Sentiment node provides analysts with insight into the authors tone or attitude (positive, negative, neutral). This is often used to analyze customer feedback and reviews on survey results and from social media sites. Next, we’ll see some examples of results commonly generated by the text analysis application.     The bar chart above shows document frequency for the text parsing results for a collection of documents processed by the Text Parsing node. It shows the Natural Language concepts, parts of speech, and the number of terms kept or dropped from analysis.   Text Analysis results are focused and apply to a specific document collection. This is different than running a general web or ChatGPT search where the content is not always accurate and relevant to the subject matter being explored.     The term map above is from the Text Parsing node. It shows the relationship among relevant terms that are connected to the central term you want to explore (in this case the term selected for the map is “medication”).   Information Gain and document frequency are shown. Information Gain is the additional information obtained by adding a conjoined term in the term map to a current rule. The size of the term node indicates the relative number of documents that include that combination of terms. The darker the term node, the more reliable the rule is for predicting that the term of interest will appear in a document. The relative line thickness indicates the strength of the association between terms.   Clicking on a term in the map displays the relevant documents containing the term on the right of the screen.     The chart above shows the number of documents, and Sentiment for each topic in this document collection. It looks like there is a high proportion of Negative sentiment in these patient’s comments on the effectiveness of their prescription drugs.   System generated and user generated topics can also be created. The topics shown below are from a collection of City Planning pdf documents for several North Carolina cities that I decided to explore. I could select the matching documents for a topic to retrieve only those city plans that I’m interested in exploring further.     You can also create your own custom concepts to identify specific structures in your documents. For example, the terminology found in doctors’ reports on patients is probably different than product customer complaint phrases found on social media. Document collections have their own characteristics that would not likely occur in the same way in a different document collection.   Custom concepts return selected text that is meaningful in specific contexts. For example, searching for the single word ‘bad’ may be interpreted out of context. It has a different meaning if the word “not” precedes it, or even if it occurs within a certain distance of ‘bad’. A custom concept can easily be built to take this context into account. Our SAS Visual Analytics training class gets into a lot more detail on creating custom concepts! You can find more information about this class here: SAS Training in the United States -- SAS® Visual Text Analytics in SAS® Viya®   Custom user concepts are written using the LITI (Language Interpretation for Textual Information) language. It is a powerful way to flex your text analytic muscles and fine tune your analysis. Look for more about the LITI language in future posts and in our SAS Visual Text Analytics class.   The Categories node creates Boolean models - another widely used tool in text analytics. They identify documents that fit into specific categories. The categories can be built using existing categorical input variables in your data source, generated topics, and even special category rules you can write. Score code generated from the categories node can be used on new documents to automatically place them into a category. Once again, this topic is covered in our training class.   After you are satisfied with your Text Analytics results, they can be used to automatically assign (or score) new documents.   Model score code created by the concepts, sentiment, topics and categories nodes in a SAS Model Studio Visual Text analytics pipeline can be run against new documents to make predictions in SAS Studio or in batch code. Here are some common useful applications of a completed text analytical project: Determine the sentiment of new product reviews posted on social media channels. Insurance claims adjuster notes or warranty claim notes can be classified into appropriate categories like flagging a claim to be investigated for fraud. Perhaps a claim should be assigned to different departments for specialized processing. Registering your text analysis Concept, Topics, Categories and Sentiment models into SAS Model Manager so new documents can be scored using the ModelOps capabilities provided by SAS. Now that we’ve defined some terms and described some uses of text analysis, let's take a peek at a Text Analytics project that uses the graphical user interface in SAS Model Studio. In subsequent posts in this series, we’ll illustrate performing specific applications of text analyses using this tool.   With the user interface, once the documents are available, you can create and run a pipeline of text analytics tasks from the menu of nodes on the left.   The center section shows one possible arrangement of nodes connected together in a text analytics pipeline.   The nodes have options that control what they do and how they behave. On the right, the options for the Text Parsing node are shown.       This link: SAS Visual Text Analytics Solutions | SAS contains a short 5-minute demo of SAS Visual Text Analytics near the bottom of the page. Look for “Get to Know SAS Visual Text Analytics” to find it quickly.     Output tables created using Text Analytics models are often combined with traditional structured data sources to improve predictions of existing models that do not currently use text scores. By including the additional features (input variables) to these models, previously unknown insights can provide a substantial boost to model precision.   For those who prefer to code, keep on the lookout for another post on how to run text analytics actions using code.   Please leave a comment with additional topics you would like our future posts to highlight.   Thank you for reading this post and be sure to enjoy the video showing SAS Visual Text Analytics in action!     ... View more