Credit cards are a convenient tool to help you acquire goods and services without carrying lots of cash or writing checks. The lure of wanting something and getting it instantly without considering how to pay for it is also a reason that sooo many credit cards exist.   People can easily get into trouble and incur credit card debt at exorbitant interest rates posing a mountain of financial burden that can be extremely difficult to get out of. Would you put your head in the open mouth of a fierce lion? Well, many people have done so without thinking through the consequences as far as using credit is concerned. With credit card agreements often exceeding 15 pages in length, it's little wonder that they are not on top of your reading list.   Still, by looking at the fine print of credit card agreements before signing up for one, you can save money when you do have to carry credit charges over several billing periods.   For this post I downloaded hundreds of credit card agreements from dozens of financial institutions. The data is generally available from the Consumer Financial Protection Bureau.   The documents are PDF files, so to get them into SAS Viya I had to make a connection to the location of the downloaded main folder that had the individual folders and PDFs for each financial institution. There could be multiple credit card agreements for a financial institution. In this example there were over 500 credit card agreements processed. Once the connection to the folder was made in “Manage Data” in SAS Viya, I was able to import all the credit card agreements into one SAS data set with each agreement in its own row.   I created a Visual Text Analytics project in SAS Model Studio using the credit card agreement as the text field. To start, I ran a generic pipeline to see what kind of initial results I would get. I enabled the standard concepts check box for this first pipeline to tag nlp (Natural Language Processing) concepts in the documents. Browsing through the documents to get an idea of what to follow up on, I noticed that the documents had quite varying APR rates. I thought it might be interesting to see what APR rates were available for different types of credit cards. While browsing through the documents in the Concepts node, I noticed that there was a mix of banks and credit unions in the data. To limit unintentional bias in my testing, I wanted to see if I could separate the banks from credit unions in the project.   I ended up accumulating several custom concepts during my effort to try to fine tune the APR information.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I started by searching the documents for “APR” and “rate”. I took one of the predefined concepts “NLP percent” and wrote a concept_rule to match any percentages within six tokens of the terms "APR" or "rate". I then noticed that there were percentages in the documents that related to the terms "margin" and "accrue". These terms also impact interest rate charges, so I ended up creating the following concept that I called _RATES_. The portion of the rule after the _c is the value returned upon a rule match, giving me a list of the desired percentages in each document.   _RATES_ CONCEPT_RULE:(DIST_6,"_c{nlpPercent}",(OR,"APR@", "rate@")) CONCEPT_RULE:(DIST_6,"_c{nlpPercent}",(OR,"margin@", "accrue@"))   I could use the score code from the Concepts node to create a table of matching rates for each document that represents a credit card agreement. This output table could then be used to create a custom report of the frequency or ranges of rates in the document collection in a post processing step.   Here is a word cloud of keywords from the concepts node output table created in SAS Visual Analytics.    Keywords    I thought of a way to separate the credit union documents from the bank documents in Visual Text Analytics without having to pre-process the documents.   To do this I started with the basic classifier rule type to identify documents with the string APR. I then created an empty _CREDIT_UNION_DOCS_ rule to use as a landing place. By cleverly using the export modifier with the string “credit union” (all in square brackets), any document matching APR that also had credit union would magically appear in the previously empty _CREDIT_UNION_DOCS_ rule!   _CREDITUNION_ CLASSIFIER:[export=_CREDIT_UNION_DOCS_: credit union]:APR   The results show 72 matched documents below. Notice this rule doesn’t use any code since its purpose is to only receive the documents from the _CREDITUNION_ rule.     Having this information in a custom concept gives me the ability to do further evaluation for other rule matches for this subset of financial institutions.   In my next rule I wanted to figure out what the APR rates were just for credit unions. Using the concept_rule type, I was able to combine the previous _CREDIT_UNION_DOCS_ “landing pad” rule with the _RATES_ rule to obtain the APR rates for credit unions only. Using the Boolean “and” operator I was able to reference the other two rule types and using the _C operator I was able to list the credit union docs and identify those within this rule.   _RATES_FOR_CUDOCS_ CONCEPT_RULE:(AND,"_c{_CREDIT_UNION_DOCS_}","_RATES_")   Of all 72 credit union documents it turns out that 38 of them matched the rate percentages generated by the _RATES_ rule.  Instead of selecting output from the _CREDIT_UNION_DOCS_ rule to be highlighted (which matched the string "credit union"), I could have written the rule to highlight the actual rates by putting the _RATES_ rule portion first after the _C operator. This would, perhaps, provided more useful results in my rule match.   In the next example I will reconstruct the rule to see what the results look like. I would expect to still have 38 matches.     This is what the rule looks like after changing the context being returned.   CONCEPT_RULE:(AND,"_c{_RATES_}","_CREDIT_UNION_DOCS_")   I like these results better than the previous rule, although some of the APR rates seem to be surprisingly high.     This graph from the results of the Concepts node shows the number of documents matched per concept:     The additional _BINDING_ rule below can be used as a starting point to further explore this document collection on your own. Remember that you can use the concept rule output table in SAS Visual Analytics to create custom reports or score new documents using the rules you build in this node.   I added this final rule to identify language reflecting liability for charges and found some interesting information. For this example, I wanted to capture a larger section of text within a match.   First, the PREDICATE_RULE syntax in the first line below selects all text between matched terms, so if ‘binding’ and ‘heirs’ appear anywhere in the same document, all text between the terms is highlighted. This entire phrase will be selected if you score new documents using this rule.   Secondly, the CONCEPT_RULE syntax below identifies the term ‘notice’ if it falls within 6 tokens of variations of the word demand (demands, demanded, demanding) or the word pay (payer, paid, pays).   _BINDING_ PREDICATE_RULE:(start, end):(AND, "_start{binding}", "_end{heirs}") CONCEPT_RULE:(DIST_6, "_c{notice}", (OR, "demand@", "pay@"))   Some matched text from this rule may reflect surprising and unexpected contract terms, pointing out the importance of reading and understanding agreements before signing.     Technology is changing the way society operates. How many times have you just clicked 'accept’ and continued on to whatever app you are adding. Hopefully this post will encourage you to be a little more curious about the ‘fine print’ you are actually agreeing to, and even ‘opt out’ of agreements you find disagreeable. I hope this post encourages you to experiment with and gain insight into the powerful capabilities of concept rules. Who knows, it may limit some unnecessary hardships down the road. This technique can be applied to all kinds of documents.   Thanks for reading and notice that you didn’t have to accept anything to do so.  😊     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Event Stream Processing Studio is a powerful tool for creating and modifying Internet of things (IoT) event stream processing models. This browser-based visual interface allows users to create projects and modify, version, test or delete existing projects.   In this post I will describe the use of stateful and stateless indexes in SAS Event Stream Processing model windows. I also illustrate the benefit of reading the warning messages from the generated project test log.   Here is an example of an online streaming project that was created using default settings. The project ingests data from pressure sensors in a simulated manufacturing process.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The train window is configured to use the kmeans streaming algorithm to segment the sensor readings into new clusters every 50 events, and the clustering results are computed in the score window.     Source Window Options   In the hierarchy of a SAS Event Stream Processing model, the source window is a crucial component. It's where the data enters the model. It can receive incoming events through connectors, which are configured in the source window itself, as well as through adapters, which run remotely to the project.   The properties of the source and other windows can be configured to optimize the performance of your model. Index type is one of the key properties that impacts performance. The index type determines how events are stored and processed, and indexes can be either stateful or stateless.   Stateful indexes are maintained according to the window's retention setting. The copy, join and aggregate windows use stateful indexes. Since stateful indexes store events in memory (or disk), they use more memory/disk than a stateless index, described next. They are ideal for smaller, ordered data and for huge data that cannot fit into memory, like tables with millions of rows. Pi_RBTREE and pi_HASH are stateful, in-memory index types. Pi_HLEVELDB and pi_HLEVELDB_NC use disk storage. Limitations of disk storage can lead to decreased performance with these index types.   Stateless indexes are not maintained. Events just pass through, and the index does not store events. Since events are not stored in memory (or disk) this leads to better performance. The pi_EMPTY index is the only stateless type. It is best for when we don't need to maintain any state because events are simply passing through the window with no updates or deletes.   Note: If a source window is followed by copy or aggregate windows that manage updates using a stateful index, you can make the source window stateless without any functional impact. Less memory will be required to run the SAS Event Stream Processing model, possibly improving the performance.   The source window is connected to derived windows through directed edges, forming a flowchart-like structure that defines the processing of streaming data.   The thunderbolt icon in the pressures window indicates the window is “stateful”. Event records are going to be accumulated in memory while the model is running. This can have adverse impact when the SAS Event Stream Processing model is deployed on an edge device with limited compute capabilities.   Interpreting Logs When Testing a SAS Event Stream Processing Project   Testing is an integral part of the SAS Event Stream Processing project lifecycle. When running a test, it's important to view the model's test logs. These logs provide valuable insights into the performance and functionality of the model. The logs can be filtered for specific messages, cleared, or exported for further analysis.   When interpreting the logs, look for any error messages or warnings that might indicate issues with the model. Also, pay attention to performance metrics, which can help identify any bottlenecks or inefficiencies in the model.   Here are some key performance metrics:   Connector Current Rate: This metric shows the current rate of events per second for a specific connector. Connector State: This metric indicates the current state of a specific connector, which can be useful for identifying any issues with data input or output. Throttling Time: These metrics indicate the amount of time a window has been throttled. Throttling occurs when a window is temporarily blocked from processing events, often due to resource constraints. There are several related metrics, including the current throttling time for each window, the maximum throttling time across all windows, and the time when maximum throttling occurred. Project State: This metric indicates the current state of the project, which can be useful for tracking the overall status of your SAS Event Stream Processing project.   These metrics can be used to identify potential bottlenecks or inefficiencies in your SAS Event Stream Processing project. For example, a high throttling time might indicate that a window is being overloaded with events, which could slow down the overall processing speed.   The following warning messages indicate a potential problem with unbounded memory growth due to the stateful index in the Pressures source window.     When the source window is set to stateless, the icon and the warning log messages disappear.     The choice between stateful and stateless indexes depends on the specific requirements of your SAS Event Stream Processing project and the nature of the data you are processing. It's a balance between processing events efficiently and meeting the project requirements.   Best Practices for Creating Efficient Streaming Models   Model Design: Start by designing your model at a high level. Identify the source windows, the derived windows, and how they're connected. This will give you a clear picture of the data flow and the processing steps in your model. Window Configuration: Each window in your model has properties that can be configured. Make sure to configure these properties appropriately to optimize the performance of your model. Model Testing: Always test your model before deploying it. Use the testing features in SAS Event Stream Processing Studio to run your model with test data and review the results. This can help you identify any issues or inefficiencies early on. Model Versioning: SAS Event Stream Processing Studio allows you to version your projects. This can be useful for tracking changes and for maintaining different versions of your model for different purposes. Model Deployment: Once your model is tested and ready, it can be deployed for real-time event stream processing. Remember that the model XML can be imported and exported, allowing for flexibility in deployment.   By following these best practices, you can create efficient and effective streaming models using SAS Event Stream Processing Studio. Thanks for reading and watch for stateful source windows!     Find more articles from SAS Global Enablement and Learning here. ... View more

In this post, I’ll outline the steps for adding a Grafana plugin to SAS Event Stream Processing (release 2022.07 or later). We’ll use the plugin to create a dashboard that shows the current location of the International Space Station (ISS) in real time with a Geomap.   Internet of Things (IoT) applications provide new capabilities and benefits that are only limited by our imaginations. For years, SAS and our customers have been extending the limits of what is possible with IoT. SAS Event Stream Processing is a part of the IoT landscape that many use to build new applications that exploit streaming data.   Data dependencies, innovation, experimentation, trying new things are part of bringing businesses into the age of the streaming and connected world of today. Building and testing prototype IoT projects now occurs with increasing frequency. One piece of the analytical toolkit has always been the ability to visualize the stream of data as it flows through the phases of a project. This is where SAS Event Stream Processing Streamviewer fits the bill in my experience.   As of Release 2024.07, SAS Event Stream Processing Streamviewer was no longer supported. Users were instead encouraged to explore the SAS Event Stream Processing Data Source Plug-in for Grafana as a replacement. I don’t know about you, but with changes like this, sometimes it helps to have someone share a few pointers to get started with the transition.     Grafana may be familiar to you as a tool for visualizing logs and traces such as those produced by Kubernetes environments providing alerts to system administrators. Well, we have integrated it into SAS Event Stream Processing as a visualization tool you can use for testing new projects developed in SAS Event Stream Processing Studio.   As you can guess, I happen to be a fan of the now-discontinued Streamviewer when creating new IoT projects with SAS Event Stream Processing. Streamviewer provided a graphic interface you could use to make sure the data flowing through a project matched your expectations as any data transformations, computer vision detection and other advanced analytics were applied. The documentation says that the use of the Grafana plugin is not intended for monitoring IoT projects, but I still find it useful for validating data in new projects. There’s nothing wrong with having a little fun when working on projects and gaining new insights.   There is a nice communities post that mentions Grafana as part of a computer vision project. You can check out this example that mentions the plugin. The post also describes the recent SAS Event Stream Processing project package organization structure that makes it so easy to work with SAS Event Stream Processing projects in a Kubernetes environment without having to separately worry about keeping track of your data in persistent volumes PV and persistent volume claims PVC.   In this post, I’ll show you a few steps of how to add a Grafana dashboard to easily show the current location of the International Space Station (ISS) in real time with a geomap. To recreate the results shown in this post, you must have SAS Event Stream Processing release 2022.07 or later and the Grafana plugin correctly installed on your environment, which is a straightforward task for an experienced systems administrator these days.   For this example, I presume that you have some familiarity with SAS Event Stream Processing Studio, but if you don’t, that’s ok since by continuing to read, you will gain some intuition on SAS Event Stream Processing.   I have a completed SAS Event Stream Processing Studio project saved on my ESP Server. This project has three source windows, one of which makes the connection to the streaming data source having the latitude and longitude coordinates of where the International Space Station is in real-time. The source window uses a SAS Event Stream Processing URL Connector to access an API that returns the location information in JSON format. The name of the source window is “iss”, and this name will be used later in the dashboard.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   I run the SAS Event Stream Processing project in test mode to confirm that I am receiving data from the API. This screen shows that the iss project window is receiving streaming data from the space station. (Note: If you want to run this project in Grafana in your environment, be sure to change the output schema for the ISS source window so that the datetime is set to Type: stamp and not date.)     You can get to Grafana from SAS Viya in the Streaming Analytics application section: Welcome Experience page. (If you are not running release Stable 2024.11 or later, the choice is visible after you first enter one of the Streaming Analytics applications.)       In Grafana, the SAS Event Stream Processing plugin in my environment has been installed and configured with a data source that connects to the SAS Event Stream Processing application. There is a “Build a dashboard” button on this window.     I add a visualization to the new dashboard.     The sasesp-plugin connects to the ESP Server and can access any of the running public projects and any other projects that I have been granted authorization to. To start the dashboard, I use the plugin to find and connect to the running project and then make some selections deciding whether I want to see a table of the data fields, or some other visualization. Multiple visualizations can be added to a single dashboard.     I just had to select the right data source fields at the bottom of the screen to reference the “iss” project and SAS Event Stream Processing window I am interested in viewing. I changed the default Time series view to Geomap and set a few of the listed properties to reference the latitude and longitude fields in the running SAS Event Stream Processing project. I selected location mode > Coords and then added the lat and long fields to the properties panel. I then clicked Apply and Save to add this visualization to the new dashboard.     Here is the Geomap. At the time of writing this post, the space station was heading into the Gulf of Mexico and over the eastern seaboard of the United States.     The SAS Event Stream Processing plugin is easy to use when testing SAS Event Stream Processing projects, and now that I have worked with it for a while, I’m pleased to say that for those of you that liked working with Streamviewer, I can recommend the plugin as a worthwhile replacement.   For anyone interested in trying out this ISS project in your own SAS Event Stream Processing environment, you can get all the details and download a copy of the project from GitHub. This project has more illustrations of SAS Event Stream Processing capabilities and is a good project to run to get some practice with the application. Also, check out this Communities article on Tracking the International Space Station.   Thanks for reading!   Don’t forget that you can always reach out to your local SAS Representative to ask about using your Education Training points to arrange for private online mentoring sessions on SAS Event Stream Processing or any other SAS initiative you are working on.     And remember, with SAS Event Stream Processing the sky’s the limit! 😊     Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this post is to consolidate information available on SAS Viya Workbench in one handy location. First off, what is it?   SAS Viya Workbench is a cloud-based application for Python and SAS developers and data scientists that is accessed through a browser. You can run SAS 9 code and SAS Viya analytics within the same environment without having to install either SAS 9 or SAS Viya.   SAS Viya Workbench is for SAS programmers who want to access SAS in VS (Visual Studio) Code using their choice of running traditional SAS program files (.sas) or using the SAS Notebook (.sasnb) format with features familiar to users of Python notebooks such as markdown cells with inline output and log displays. It is also for Python programmers who use VS Code or Jupyter Notebooks.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In SAS Viya Workbench, users can easily manage their own environment without additional IT support. Users can also choose the compute power to match the needs of the project.   The compute power can be scaled up or down by the user with a simple restart of the workbench to make sure you only use the resources needed for your specific analysis task. This can reduce the cost of running workbench since cloud providers charge by the resources used. No need to be concerned about resizing your environment since all work is saved between restarts of the environment so that nothing is lost. After a restart, you pick up where you left off in the previous session.   Here are some capabilities of SAS Viya Workbench.     You might want to think of SAS Viya Workbench as a convenient standalone cloud-based application that can also be a stepping stone towards a full SAS Viya environment if-and-when enterprise capabilities are needed. Additional Viya enterprise capabilities that you may want are shown in the chart below.     A really good way to get a taste of using Workbench is to view the quick start tutorial youtube videos. I highly recommend you watch the 12-minute Getting Started video from this series of Quick Start Tutorial videos.   The sasviya.ml package   We also provide you with a repository of Python notebooks, SAS programs and sample data for use in SAS Viya Workbench. SAS9 users can use their code and experiment with new Viya procedures that can coexist and run in workbench without requiring a full Viya environment. The SAS procedures run multi-threaded where it makes sense, providing performance benefits that reduce the amount of time and cost required to run your analysis.   Python users can also benefit from using SAS Libraries that offer performance improvements over traditional libraries. SAS wrote the sasviya.ml package for Python users to run in SAS Viya Workbench. This is from a recent post:   The sasviya.ml package is designed to integrate with scikit-learn, so the sasviya.ml model objects have the same syntax and most of the same functionality as the equivalent scikit-learn model objects. The major difference is in the execution, the models in the sasviya.ml package execute using optimized SAS libraries, which takes advantage of multithreading and can yield much faster runtimes than the equivalent models in scikit-learn.   Examples   These examples provide an excellent framework you can modify to quickly run your own data through machine learning algorithms.   In addition to data mining and machine learning procedures, SAS Viya Workbench provides procedures for sampling, data exploration, clustering, dimension reduction, model assessment, computer vision image processing and supervised learning.   Additional Resources are available in the SAS Viya Community where interesting and relevant posts are frequently updated by community members. Examples of community topics for Python and SAS users include tips such as data access, machine learning, and using Git for source control.   E-Learning   In collaboration with my colleagues, we developed a free E-Learning training course on SAS Viya Workbench that is now publicly available on https://learn.sas.com. Simply log into your profile, search for the course Modern Data Science with SAS® Viya® Workbench and Python and activate it to get started.   Product Documentation   The official product documentation has all kinds of useful information including example use cases, data access techniques, programming examples and more.   Why use SAS Viya Workbench?   Cost-efficiency: Combined with cloud providers, you can manage computing costs by user, project and department, scaling resources as needed to only pay for what you use.   Ready-to-use environments using SAS code or Python through Visual Studio Code or Jupyter Notebook interfaces.   Projects can be shared using Git for version control and collaboration.   Improve performance by using multi-thread supported SAS libraries.   There is lots of excitement around SAS Viya Workbench, and I think these links will help guide you to the resources you need to keep you in the know.   Thanks for reading and I hope you enjoy exploring SAS Viya Workbench!     Find more articles from SAS Global Enablement and Learning here. ... View more

Imagine that you have an enormous collection of transcribed phone calls from bank customers as your data source. If you were to ask someone to identify calls related to ‘accounts’ in a bank, what do you think they would say?   Sure, some may simply tell you they’re not interested in whatever you’re selling. Others may indulge your question and provide a thoughtful response possibly suggesting that you:   Look for savings, checking, account balance, and interest rates. Search for overdraft, insufficient funds, closed account. Don’t forget the terms: fraud, check, or any word containing ‘cash’.   With a bit of work, we can come up with convincing search arguments that seem to do a pretty good job of identifying the concept we are looking for. Even with well-crafted arguments, it is possible that some relevant documents remain unidentified.   Is there anything else that can improve our information retrieval searches when we have run out of ideas based on typical human thinking patterns?   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Refer to my previous post where I show how to use SAS Visual Text Analytics to automatically generate concept rules that can build on our research related to ‘accounts’.   In this post, I want to highlight some less-obvious available syntax for constructing various types of concept rules. This screen capture from my previous post shows a handful of generated rules. Think of these as a source of concept rule ideas you can experiment with and use in rules that you have already written.     Let’s consider how we can use these for inspiration to create specialized rules that can be useful in many different scenarios. What do you think of this rule for starters? We can use it to determine what actions have been taken regarding user accounts.   C_CONCEPT:_c{:V} my account   The C_CONCEPT rule returns the _c portion of the rule when a complete rule match is found in a document. This rule looks for a part of speech :V (verb) immediately followed by the term ‘my account’. This rule returns any actions applied to a user’s account. The action will likely be a verb such as opened or closed, but we might discover some unusual or unexpected actions that provide new insight into our customer data.   This information can then be used to identify trends in the number of specific actions that have been taken against user accounts over time. (A SAS Visual Analytics dashboard using the scored concepts is a good way to report on this information).   Different parts of speech can be substituted in this type of rule depending on the information you are looking for. Nouns, verbs, adjectives, prepositions and determiners are examples of parts of speech that can be identified by the Natural Language processing capabilities in SAS Visual Text Analytics. This table lists the available parts of speech for all supported languages. Here are a few examples of additional automatically generated rules to ponder.   C_CONCEPT:_c{:N} from :PN   This rule highlights a noun :N followed by ‘from’ followed by a proper noun :PN (upper case noun). This rule can identify interactions between parties that may be of interest in a business context. Here are some matches for this rule from my data collection of customer complaints.   Matches:   retroactive spousal payments from Department of Veterans Affairs   contacted so many attorneys from New Jersey to Georgia to pursue a case   After hearing from IRS The next example looks for the term fraud in the same sentence as any monetary amount found by the predefined nlpMoney concept. The elements in this rule can appear in any order as long as they are in the same sentence. Combining predefined concepts with specific information you are looking for can be very powerful.   CONCEPT_RULE:(SENT, "_c{Fraud}", "nlpMoney")   Here are three returned matches from this rule.   Bank 's Fraud Department regarding an online purchase of over {$1500.00} in which I stated…   …was victim on fraud in the amount of {$18000.00} on XX/XX/2022…   Overall, I lost over {$13000.00} to this fraud and I really want my money back. Next is a rule that requires items to appear in a specific order for a match to occur. It looks for any word form of either fraud or scam within 5 terms of a proper noun.   CONCEPT_RULE:(ORDDIST_5 , (OR, "fraud@", "scam@"),"_c{:PN}")   Matches:   I was scammed. The FBI is now involved.. I filed a fraud report with local XXXX Sherrif 's police department… Here is a generated rule looking for items that occur in any order in a sentence. This rule combines a predefined concept with a morphological expansion (the @ symbol syntax) of the term advance. (any form of the term 'advance')   CONCEPT_RULE:(SENT, "_c{advance@}", "nlpMeasure")   Matches:   It was not until two days later ( XX/XX/18 ) that I found out that they transferred money from my own credit card as a cash advance to make me believe it was money from somewhere else.   mailing in my payments 1 month in advance   We will send you an advance notice approximately seven days before we reapply the negative balance.   There were 75 cash advances If you are looking for any mention of an action, this rule identifies what’s going on in your documents by looking for a verb before ‘that’ followed by a noun. Here are some examples.   C_CONCEPT::V that _c{:N}   Matches:   lose that amount of money.   stop that retaliation   I sent that money   protect that data or delete that data     I would not have thought of these rules were it not for the automatic generation capability. Consider incorporating some of these ideas in your concept rules to see if it improves your information retrieval success. Who knows, if your current rule ain't broke, you may make it better!   Thanks for reading and I wish you much success with your text analysis adventures.     Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this post is to describe how to enhance concept rules to retrieve better focused information from a collection of documents.   We'll explore concept rules and ideas for refining them in the context of an analyst processing a large number of customer complaints at a large bank.   Better concept matches start with interactive exploration in SAS Visual Text Analytics.   For an introduction to the capabilities in SAS Visual Text Analytics see my post.   Concepts are used to extract relevant information from documents. This analytical journey starts with exploratory analysis. When processing customer complaint documents, concepts help us: identify conditions where someone needs help, identify opportunities to improve customer service, discover social media comments to address before a complaint goes viral, and more.   First some freebie tips.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Predefined concepts are automatically identified by checking a box in the properties panel. These use Natural Language Processing (NLP) techniques to identify concepts such as measures, money, organizations, people, places and more. The nlpMeasures concept for example, returns matches for terms such as (2 years, two days, 5 mg, $250., 8 feet). The nlpMoney concept matches anything related to currency such as ($2, 2.50€, 40 yen, 20 pounds).   You can combine predefined concepts with new custom concepts to easily identify documents having specific information. Keep these freebies in mind to assist you with building your own concepts.   For example, the rule argument (SENT,”_c{cashed}”, ”nlpMoney”}) returns matches for any document that has the term “cashed” and any currency amount in the same sentence. SAS will recognize currency whether it is U.S. dollars, British pounds, etc.   Your mission…   Let’s consider an example scenario.   You work for a bank and one of your responsibilities is to evaluate 10,000 recent customer complaints that came in through various channels and are available in computer readable text. Your mission is to investigate complaints related to the handling of cash and recommend actions your company might take to improve the customer’s experience.   What is the first thing you would try?   Exactly, Visual Text Analytics!   Open the documents in SAS Visual Text Analytics, run the text parsing node and look at the kept terms and list of documents. You might then try searching the documents for the word cash and see if this gives you any ideas. Your search returns 700 documents and you see phrases in the documents like:   Cash a check Cash advance Cash back Cash price Cash deposit / withdrawal     Remember these pointers to structure an effective search:   Placing the "+" in front of a word requires the word to be in a document. A term without a preceding + returns the term if it exists in the document but it does not have to be in the document. A "–" in front of a word finds documents that do not have that term. Place a * at the beginning, middle or end of a search query to get wildcard matches. The query term +*ion returns terms such as exception or action. Quotation marks return the specific content in between such as “fraudulent check”.   You might then shift your attention to the kept terms list at the top and search for cash again and notice the following.      In this display, cash is recognized as a noun, verb, proper noun, and in a noun group by the NLP process. You decide to look at the verb cash in the term map next.     This map shows the terms related to cash based on information gain. One path from this chart suggests that some documents contain (cash, check, issue) and do not contain (offer or deposit) where the ~ sign indicates “not”. These ideas represent new insights to consider as you construct new custom concept rules.   In the parsing node you can click the similarity scores icon, remove “cash” from the kept terms filter, and notice the additional new terms that appear. “Check fraud” and “forged check” didn’t initially come to mind when exploring cash, but you may want to expand your investigation for occurrences of fraud related to cash!     Concepts Node   Next on your journey, you add and run a concepts node to identify predefined concepts in the documents and then create a custom concept looking for various morphological expansions of the term “cash”, with fraud and check. The initial LITI (language interpretation for textual information) CONCEPT and CLASSIFIER rules follow.     These results look promising, but a lot of documents matched. You wonder if there might be some additional ideas for rules. Click the Autogenerate concept rules icon in the concept that has just 3 simple rules.     The fresh new rules were written to the sandbox. The generated rules will need more explanation especially if the LITI syntax is new to you. We plan to cover some of these in the next post, but here is a sample of the autogenerated rules in the meantime.   After building concept rules, the resulting concepts model can score the current or new documents for concept matches. SAS Visual Analytics can also be used to report on the results.   If you can’t wait until the next post to make sense of these rules, reference this documentation, otherwise, I look forward to sharing some more insights with you in the next post! It's your move!       Find more articles from SAS Global Enablement and Learning here. ... View more

  A customer recently posed this question: “I know how to match a concept at the end of a sentence, but how can I match it at the conclusion of a document?” (For a review of concepts and other basics, see my previous post.) The purpose of this post is to match concepts near the end of a document using a technique that leverages multiple nodes. LITI is Language Interpretation for Textual Information, and it is the programming language used to code concept and category rules in SAS Visual Text Analytics.   Customer complaint call transcripts can end up being quite verbose. Sure, for starters, you could run a summary of the documents to get the main ideas being raised, and this would be a good thing to do. (see my post on summarizing documents). But isn’t there something else I could also try?   The complete customer interaction likely makes some progress over its duration. The follow-up actions or conclusions of the dialog can be expected to occur towards the end of the event, after the back-and-forth discussion has become more focused on the real issues. It may not be necessary to read through the entire interaction to gain some actionable insights.   You may be most interested in returning the matches for certain conditions that occur in the last 50 terms or so of an exchange between the customer and the call agent. The text near the close of the conversation likely holds information to help you determine any appropriate followup actions. Designing the structure of your rules thoughtfully can provide meaningful insights. You could also create a dashboard of these conclusions over time for extra credit.   Consider this example where a Concept rule finds matching terms occurring in a document based on a list of conditions and activities that are stored in supporting rules. In the consumer financial services area, you might have a concept rule containing LITI syntax like this:   CallCenterMatches   CONCEPT_RULE:(AND,(DIST_20,"_CONDITION_", "_c{_ACTIVITY_}"))   This rule matches documents that have an item from the Condition rule within 20 tokens of an item from the Activity rule. A match requires that at least one item from each table is in the document due to the use of the AND operator.   Two example tables below use the LITI language syntax for a table of conditions and a table of activities. The items from the columns can appear in any order. The @ symbol indicates morphological expansions of the term (all forms of a term like report, reports, reported, reporting…)   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Here are some of the 635 matches from SAS Visual Text Analytics of the CallCenterMatches rule with the activity name highlighted as the rule requested.     OK, this is great, but we already know how to use the CONCEPT_RULE and all its bells and whistles. What else can I do?   What if you have a long call transcript and only want to know what happens towards the end of the call? This kind of focus may do a better job than our CallCenterMatches rule on its own to identify followup actions with the customer. To do this, we’ll add a category rule to find a concept if it exists at the end of the interaction (call or chat or transcribed audio)   Next, we will create a category rule based on the concept and use an easily overlooked statement available in the Category node.   I can connect a categories node directly to a concepts node as shown in the right-hand branch of this pipeline, since both nodes can do their own parsing.     I added the following Categories rule so I could see only the matches at the end of the document. The syntax of the Categories node is different than the Concepts node, so this syntax is not valid for use in the Concepts node.   I created my new Category rule using this syntax:   (END_40, "[CallCenterMatches]")   The square brackets in double quotation marks tells the Categories node to use a concept rule rather than a term in the rule.   After applying the concept matches to the categories node in the last 40 tokens of these call reports, there are only 190 matches whereas previously there were 635 document matches in the concepts node.   This is the end of one of the matched documents with the highlighted conditions and actions terms within the final 40 tokens!     This is the description of the END_n Categories rule.   END_n (From the end of the document)   Takes a value for n and one or more arguments. Matches if each argument occurs within n tokens from the end of the document. For example, the rule (END_35, "conclusion") produces a match if conclusion is found within 35 words from the last token in the document. Note: Punctuation counts as a token. In some languages, words that include hyphens are counted as one word (for example, merry-go-round is one token).   So, there you have it! You can combine concept rules within categories to get specific matches based on your needs. There are many other hidden treasures in the land of LITI rules and who knows, I may reveal some more in posts yet to come!!!   Thanks for reading and let me know if you use a form of this rule.   BTW,   If you really want to learn this stuff, here is an exercise for you to try. See if you can identify what each of these rules will return. Do some rules not give you what you are really looking for? Do any of these rules return more than the information we are looking for? Read about the DIST_n, AND, OR, and _c operators for Concept rules for hints.   CONCEPT_RULE:(DIST_20, "_c{_CONDITION_}", "_ACTIVITY_") CONCEPT_RULE:(AND, "_c{_CONDITION_}", (OR, "_ACTIVITY_")) CONCEPT_RULE:(OR,(DIST_20, "_c{_CONDITION_}", "_c{_ACTIVITY_}")) CONCEPT_RULE:(AND,(DIST_20,(OR,"_c{_CONDITION_}", "_c{_ACTIVITY_}")))     Find more articles from SAS Global Enablement and Learning here. ... View more

  Hello, and welcome to my blog! The purpose of this blog is to encourage you to experiment with some of the options that are available for taking your Text Topic Modeling beyond the basics. If you are new to Visual Text Analytics, see my article on Getting Started with Text Analytics to get an idea of the capabilities.   Starting with a collection of documents, these can include customer call reports, patients’ comments, product reviews or movie recommendations, SAS provides the Natural Language Processing (NLP) capabilities that can turn the documents into useful insights.   The Topics node is usually inserted near the end of a text processing pipeline. This node provides you with controls that regulate the topic discovery process. Controls include setting cutoff thresholds for terms and documents. A lower threshold captures more items while a higher threshold captures fewer items. The threshold is based on the number of standard deviations above the mean term or document weight.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Note that when running a Topics node, some documents will likely not fit into any of the generated topics, and they will be accounted for in a group designated as “No Matching Topic”. Re-running the node with lower thresholds specified will result in fewer documents falling into the “No Matching Topic” group. These settings provide users with adequate controls for extracting reasonable topics when getting started with a text analysis project. Machine generated topics can be combined or split, and additional user-created topics can also be added.   To illustrate this point, I ran the topics node with the default settings, and it returned 11 topics. The right-most bar represents documents that did not fit into any discovered topic.     I then reduced the document cutoff control setting from 1 to 0.7 standard deviations, and now more documents are accounted for in the generated topics as shown below. The 5th bar from the right in the chart below represents the reduced number of documents with no matching topic. A good approach is to examine the results you get by spot checking the documents in your topics and then make adjustments to the thresholds until you are happy with the settings.     The Latent Dirichlet Allocation (LDA) Topic Model   The results above were generated under the default Singular Value Decomposition (SVD) algorithm also known as latent semantic indexing (LSI). If you wish to experiment with generating topics using a different algorithm, the Latent Dirichlet Allocation (LDA) Topic Model technique is also available. Running LDA for topic generation may produce different topics and document assignments. It utilizes Bayesian and Expectation Maximization techniques. Now, let's discuss how we'd implement it by calling a CAS Action in syntax.   Implementing the Latent Dirichlet Allocation algorithm: ldaTopic.ldaTrain action creates topics from a document collection using just a document ID, input table, and the variable name containing the documents at a minimum. It defaults to creating only 2 topics, but this can be changed using the k= parameter. In my movie description example, I asked for k=5 topics to be created.   From the documentation, the minimum syntax needed to run with default settings is shown here. This code would run in SAS Studio.   /* Connect to CAS and load the data table. */ options casport=5570 cashost="cloud.example.com"; /**/ cas casauto; caslib _all_ assign; proc cas; /**/ session casauto; table.loadTable result=r / /**/ caslib="Library_name_to_use" path="Path_to_my_file" casOut={name="CAS_file_name", replace=true}; run; /* Run the ldaTrain action – this section defines the output table names */ ldaTopic.ldaTrain / /**/ casOut={name="Topics", replace=TRUE } docDistOut={name="DocDist", replace=TRUE } /* This section defines the input table name, id variable and text variable */ docId="D_ID_variable_name" table={name="Table_name"} text={{name="T_variable_name"}}; run; quit; After running code based on this outline, the following output table shows the proportion of terms for each topic that was found in a document. There were 5 topics generated, numbered 0-4, and each document id has a proportion value for each topic.     The following output table shows what topic (0-4) each term was placed into as well as the assigned probability the term will represent the specific topic.     As I mentioned before, I added a parameter k=5 to return 5 topics rather than the default 2 topics when I ran against a movie description document collection. The hyperparameter Alpha with a minimum value of 0 and a default value of .1 specifies the Dirichlet hyperparameter for the document's topic proportion. Beta specifies the Dirichlet hyperparameter for the topic distribution.     Additional optional parameters described in the product documentation let you do additional configuration including adding stop words, modifying the number of iterations, entity identification and stemming.   You can run LDA action sets from CASL, PYTHON, R and Lua, as well as from a SAS STUDIO task for those of you who might prefer to work with a user interface. Let’s take a quick look at SAS Studio for inspiration.     On the left you see the text parsing and topic discovery selection. In the center, the Options tab is selected as is the LDA radio button. On the right you see the generated code calling the ldaTopic.ldaTrain action. Even if you are a ‘coder’, the SAS Studio task is a great way to quickly get a working copy of code showing correct syntax and the placement of additional options.   The SAS Studio tasks are straight forward easy to use and can be an inspiration on your way to becoming a text analytic practitioner!   In a subsequent blog we will explore scoring these models and ways to deploy them in a production environment. Until then, I wish you much success with your analysis and may all your topics be true!     ... View more

Hello, and welcome to my post! The purpose of this post is to showcase the Text Summarization task in SAS Studio (in less than 5000 pages 😒).   The volume of text data seems to increase each day, and we often encounter information overload in our daily activities. It seems that everyone has something to say, whether it is in a video on how to clean a yucky coffee pot, another breaking news report, customer product complaints, or transcribed audio. How do you find meaning in the vast amount of data that is growing every day?    Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   SAS Visual Text Analytics (VTA) users certainly have the tools needed to find answers buried in document collections. (See my post on Getting Started with Text Analytics).   The Text Summarization task in SAS Studio is a VTA tool that provides functionality to summarize a document. Ever think “TL;DR” when looking at social media posts? I know that you know this but… TL;DR is an abbreviation for "too long; didn't read."   You probably have found yourself in a situation where you could use some help getting a quick summary of a document collection into a reasonable number of sentences. Especially if the alternative is going through “billions” of pages of text.   The Text Summarization task in SAS Studio is one possible way to summarize documents quickly and easily. It uses natural language processing (NLP) techniques to summarize a document by identifying representative sentences. It can also be run via the TextSummarization action set in code. It computes the summary for each document in the data set, and it can also generate a single summary for the entire data set.   There are various techniques that are used for summarizing text. The purpose of the Text Summarization task is to perform an extractive summary. The software tries to identify meaningful sentences by using NLP (Natural Language Processing) techniques involving terms, entities, and noun groups. Another summarization technique you may have encountered is referred to as an abstractive summary that can use deep learning methods to analyze the text.   A fun fact is that the summarization result will be identical to the original document if it has only 2 sentences and you ask for a summary of 2 sentences. If you try this task on really short documents, you might be disappointed with the findings, but now you know why this would happen. Creating a summary of the entire short-document collection may be more valuable, and that’s what we’ll describe next.   The table in this example has hundreds of historical customer reviews of an electronics product, and I want to condense each review down to 2 sentences. Some of the reviews are several paragraphs in length and others are only a single sentence.   You can find the Text Summarization task in SAS Studio on Viya to get started.     The Text Summarization task in SAS Studio reduces content down to a maximum number of sentences that you can specify. The maximum number is capped at 32. A maximum number this large may be useful if you are summarizing chapters of a novel or scholarly abstracts. In this case, I’m processing consumers’ product reviews, so I chose to represent each review by only 2 sentences. Since I am also interested in the overall ‘sense’ of the reviews, I am also going to ask for a summary of the entire corpus.   Summarizing documents will produce different results depending on what features you generate for the task. You have a choice of parsing terms for the analysis, using entities and noun groups, or selecting both options as I have in this example. Try different combinations of these options with your data and see what options give you the best results for the kind of data you are processing.     This type of text summarization is different than a summary that would be created by a Large Language Model (LLM). Some analysts are wary of using LLMs to summarize documents since they may ‘hallucinate’ or just make stuff up, and that is not what you want. There are situations where LLMs may be useful when they are applied after a summarization is done.   Here is one document from my dataset. At the risk of getting a TL;DR from you, go ahead and read the original review and the subsequent summarized review. Do you think the summary did a good job capturing the important sentences?   Original Review:     Summary of Review:     When you experiment with your own text data, ask for a corpus summary and see what combination of parsing options provides you with the best result. This task is pretty nifty to have in your bag of tricks, but it's not designed for all situations. Remember this is only one of the many resources available to process text. I hope that you enjoyed reading about this task and encourage you to watch for additional articles.   Thanks for reading!     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS® Visual Text Analytics: How you can Create concept rules programmatically.   Hello, and welcome to my post! The purpose of this post is to provide helpful insights for working with text document collections. This post addresses questions students have posed recently that you may also find helpful when working with SAS® Visual Text Analytics.   In this installment we will consider "case sensitivity" in custom concept rules as well as how to create concept rules programmatically.   There are excellent sources of information already available on writing custom concept rules. Some sources are the product documentation or the SAS Support Community articles. These are great starting points if you are interested in learning the basics of writing concept and category rules.   Let’s start with a quick tip that you may not be aware of. Did you know that when you create your own concept rule it is not case sensitive? The following example concept rule identifies several types of desserts.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   It includes some capitalized words, but it will generate matches regardless of the actual case in the documents. There are four matches returned with the default rule behavior.     Suppose now you want to identify terms based on the specific case that is used in a rule.  With the Concepts Node open, right click the rule name that you want to be case sensitive and click “Match Case”. This will place an icon Aa next to the rule name as shown below. This easily distinguishes case-sensitive rules. Notice that now there are fewer matches in the text document with case sensitivity enabled for the rule (only two matches this time).     Using ActionSets   The next example from the product documentation creates three concept rules programmatically that work together as a unit.   The first section is the syntax for creating the three rules: COMPANY, PRODUCT, and TEST_PRED using data step. The first two Classifier rules are used in the third rule which is a PREDICATE_RULE type, also referred to as a “fact”. Notice the property “CASE_INSENSITIVE_MATCH” in the syntax reflecting the default behavior of concept rules not being case sensitive. You can remove this line if you want to make the rule case sensitive.   Each of the three rule blocks start with ENABLE:   options casport=5570 cashost="cloud.example.com"; cas casauto; libname mycas cas; data mycas.concept_rules_long; length config varchar(*); infile datalines delimiter='|' missover; input config$; ruleid=monotonic(); datalines; ENABLE:COMPANY FULLPATH:COMPANY:Top/COMPANY PRIORITY:COMPANY:10 CASE_INSENSITIVE_MATCH:COMPANY CLASSIFIER:COMPANY: Microsoft CLASSIFIER:COMPANY: Amazon CLASSIFIER:COMPANY: Google ENABLE:PRODUCT FULLPATH:PRODUCT:Top/PRODUCT PRIORITY:PRODUCT:10 CASE_INSENSITIVE_MATCH:PRODUCT CLASSIFIER:PRODUCT: xbox CLASSIFIER:PRODUCT: windows CLASSIFIER:PRODUCT: chrome CLASSIFIER:PRODUCT: fire ENABLE:TEST_PRED FULLPATH:TEST_PRED:Top/TEST_PRED PRIORITY:TEST_PRED:10 CASE_INSENSITIVE_MATCH:TEST_PRED PREDICATE_RULE:TEST_PRED(company, product):(DIST_1, "_company{COMPANY}","_product{PRODUCT}") ENABLE:Top FULLPATH:Top:Top PRIORITY:Top:10 CASE_INSENSITIVE_MATCH:Top ; run;   These rules must be compiled before they can be used in the ActionSets. The syntax is shown below. The section starting with textRuleScore.applyConcept about halfway down the following example, takes the compiled rules and extracts the matched documents, placing them in the output table. The documents were read from the table named apply_concept_text. This is useful for quickly applying concept rules to documents that need to be processed on an ad-hoc basis, or for processing large document collections in batch mode.   proc cas; builtins.loadActionSet / actionSet="textRuleDevelop"; builtins.loadActionSet / actionSet="textRuleScore"; textRuleDevelop.compileConcept / casOut={name="outli", replace=TRUE} ruleid="ruleid" config="config" table={name="concept_rules_long"}; run; textRuleScore.applyConcept / casOut={name="out_concept", replace=TRUE} docId="docid" factOut={name="out_fact", replace=TRUE} model={name="outli"} ruleMatchOut={name="out_rule_match", replace=TRUE} table={name="apply_concept_text"} text="text"; run; table.fetch / table={name="out_concept"}; run; table.fetch / table={name="out_fact"}; run; table.fetch / table={name="out_rule_match"}; run; quit;   The terms from the scored documents that are matched by the three rules are listed in the table below. The start and end positions of the matched text are in the table. The _fact_argument_ column refers to the individual supporting rules that found a match which was then used in the main TEST_PRED predicate rule type.     For more details on how to work with concept rules, be sure to sign up for our e-learning or live-web classes on SAS Visual Text Analytics. The previous documentation link above provides lots more information that may be spotlighted in an upcoming post.   Thanks for reading and keep texting!     Find more articles from SAS Global Enablement and Learning here. ... View more