In this article, I solve the Cutting Stock Problem by implementing a Column Generation algorithm using the action set Optimization in SAS Viya.   Column Generation methods are used successfully in large-scale Mathematical Optimization problems that occur frequently in the airlines, telecommunications, logistics and other industries.   These large-scale problems have thousands of variables (or columns) and enumerating them explicitly is not possible. By using the dual information that is obtained solving the master problem, new variables (or columns) are priced out (or generated). Other examples of column generations methods are the Dantzig-Wolfe Decomposition principle and Branch and Price. See my article on Implementing Column Generation using SAS Optimization Decomposition to Solve Large-Scale Problems about the SAS Decomposition Algorithm, which is based in the Dantzig-Wolfe Decomposition principle and can be implemented to take advantage of the distributed, cloud-based computational environment that SAS Viya provides.   According to George Nemhauser: “Column generation refers to linear programming (LP) algorithms designed to solve problems in which there are a huge number of variables compared to the number of constraints and the simplex algorithm step of determining whether the current basic solution is optimal or finding a variable to enter the basis is done by solving an optimization problem rather than by enumeration”   A quick search on the INFORMS journals for "Column Generation" resulted in 9 articles published between July and August 2019, in the journals Operations Research, Transportation Science, INFORMS Journal on Computing, INFORMS Journal on Optimization. The search returned 851 articles published from 1968 to August 2019. Searching for "Column Generation" in Google shows articles in the Journal of Artificial Intelligence Research, European Journal of Operational Research, IEEE Xplore Digital Library, etc. Cutting Stock Problem The Cutting Stock problem and its mathematical formulation are easy to understand. The problem to solve is: “how to cut a log into smaller stocks of different widths and demands” and it can be solved using Column Generation algorithms.   As an example, let’s assume that we want to cut a 91-inch log into smaller stocks of different widths and demands as shown in this table:     One way to cut the 91” log would be cutting 3 stocks of 25.5” finished-with and the resulting waste would be 14.5”. Another way would be cutting four stocks of 20” finished-width and the resulting waste would be 11”. There are many ways to cut the 91-inch log, some of those ways (or patterns) could result in a considerable amount of waste. The idea is to use the least possible number of 91” logs with minimum waste while satisfying the demand of finished-width logs.   Imagine a logistics company that has many depots, at each depot there are several trailer-size loads that need to be moved to different depots. Moreover, the pick-up and delivery of those trailer-size loads must occur during a specific time-window. The objective then would be to find the minimum number of routes that efficiently connect those depots to timely pick up/distribute the trailer-size loads.   Now imagine that you work for a Fortune 100 Airline company and you must develop its flight schedules. One would like to find the minimum number of flights that connect the cities where that airline flies while satisfying customer demand for all legs in all flights (also within specific time-windows).   There are many ways to cut the 91” log, many ways to create the routes that connect the depots, many ways to develop flight schedules, and Column Generation algorithms are used to solve these problems.   In this article, I will use a very simple version of the Cutting Stock problem. Still, the mathematical formulation and the code provided illustrate the concepts and can be used as basis for more complex problems. Mathematical Formulation of the Cutting Stock Problem   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Solving the Cutting Stock Problem Using SAS Optimization in Viya SAS Optimization is the general-purpose optimization product that takes advantage of the distributed, cloud-based computational environment that SAS Viya provides. L arge-scale problems benefit from running in a VIYA environment and their run time can be further shortened by parallelizing their subproblems.   SAS Optimization has these solvers: LP, MILP, QP, NLP, CLP, LSO and Network. Which are called thru Action Sets (Optimization and Network Optimization), whose actions can be submitted thru a variety of clients such as PROC CAS via SAS Studio, or through similar syntax in Python, Java, or Lua.   In this article, I will use the runOptmodel action in the Optimization Action Set to submit PROC OPTMODEL code to the CAS server.   Pseudo-Code to Solve the Cutting Stock Problem Steps to generate the patterns and solve the cutting stock problem: Initialize LP with a set of trivial patterns N. Solve the Master problem using the LP solver. The LP solution are the x j (number of times pattern j will be used) and the dual values of each constraint Solve the Subproblem: Minimize the reduced cost using a knapsack solver Results will indicate which new column(s) to add to the LP If the reduced cost is negative add column(s) to LP. Repeat steps 2 and 3 until there are no more negative reduced cost patterns. Use the latest LP solution as warm-start for the MILP solver   Code *How to solve the Cutting Stock Problem using Column Generation techniques with SAS Optimization in Viya 3.4; cas mySession1d sessopts=(caslib=CASUSER timeout=1800 locale="en_US"); libname CASUSER cas caslib=CASUSER ; data CASUSER.indata; input type demand width; datalines; 1 78 25.5 2 40 22.5 3 30 20 4 30 15 ; * Using SAS macros facilitates the use of parameters that can be changed easily, as in this example the variable capacity; %macro csp(capacity); /* In this example we use the Action Set Optimization */ proc cas; loadactionset "Optimization"; source pgm; /* declare parameters and sets */ num capacity = &capacity; set ITEMS; num demand {ITEMS}; num width {ITEMS}; num num_patterns init card(ITEMS); set PATTERNS = 1..num_patterns; num a {ITEMS, PATTERNS}; num c {ITEMS} init 0; /* It will hold the dual variables of the demand constraint */ num epsilon = 1E-6; /* read input data */ read data CASUSER.indata into ITEMS=[type] demand width; /* generate trivial initial columns */ for {i in ITEMS, j in PATTERNS} a[i,j] = (if (i = j) then floor(capacity/width[i]) else 0); /* define master problem */ var x {PATTERNS} >= 0 integer; minimize NumberOfRaws = sum {j in PATTERNS} x[j]; con demand_con {i in ITEMS}: sum {j in PATTERNS} a[i,j] * x[j] >= demand[i]; problem Master include x NumberOfRaws demand_con; /* define column generation subproblem */ var y {ITEMS} >= 0 integer; maximize KnapsackObjective = sum {i in ITEMS} c[i] * y[i]; con knapsack_con: sum {i in ITEMS} width[i] * y[i] <= capacity; problem Knapsack include y KnapsackObjective knapsack_con; /* main loop */ do while (1); /* master problem */ /* minimize sum_j x[j] subj. to sum_j a[i,j] * x[j] >= demand[i] x[j] >= 0 and integer */ use problem Master; put "solve master problem"; solve with lp relaxint / presolver=none solver=ps basis=warmstart printfreq=1; print "Master Problem Solution"; print x; print demand_con.dual; for {i in ITEMS} c[i] = demand_con[i].dual; /* knapsack problem */ /* maximize sum_i c[i] * y[i] subj. to sum_i width[i] * y[i] <= capacity y[i] >= 0 and integer */ use problem Knapsack; put "solve column generation subproblem"; print "Column Generation subproblem solution: Finding new pattern"; solve with milp / printfreq=0; for {i in ITEMS} y[i] = round(y[i]); print y; print KnapsackObjective; if KnapsackObjective <= 1 + epsilon then leave; /* include new pattern */ num_patterns = num_patterns + 1; for {i in ITEMS} a[i,num_patterns] = y[i]; end; /* solve IP, using rounded-up LP solution as warm start */ use problem Master; for {j in PATTERNS} x[j] = ceil(x[j].sol); put "solve (restricted) master problem as IP"; solve with milp / primalin; /* cleanup solution and save to output data set */ for {j in PATTERNS} x[j] = round(x[j].sol); create data solution from [pattern]={j in PATTERNS: x[j] > 0} raws=x {i in ITEMS} <col('i'||i)=a[i,j]>; endsource; optimization.runOptmodel / code=pgm; run; quit; %mend csp; %csp(91);   Results and Analysis The Excel table below shows the results. For this simple problem, there are three iterations of the main loop that solves the LP Master and then the Knapsack subproblem until there are no more negative reduced cost patterns.     The first iteration starts with a feasible set of four patterns, the Master objective function is 48.5 which indicates that the pattern X1 is used 26 times, the pattern X2 is used 10 times, the pattern X3 used 7.5 times (non-integer!) and the pattern X4 is used 5 times with a final waste from using these four patterns of 474.5. The subproblem finds a new pattern X5 = (2,0,2,0).   The second iteration starts with 5 patterns, the Master objective function is 46 and the final waste from using these five patterns is 247. The subproblem finds a new pattern X6 = (2,1,0,1).   The third iteration starts with 6 patterns, the Master objective function is 44, and the final waste from using these six patterns is 65. In the subproblem, the reduced costs are positive which indicates that there are not new patterns that would improve the solution and the while loop ends.   The integer solution found is the same as the one found in the last LP. One should use pattern X2 four times, pattern X4 once, pattern X5 15 times and pattern X6 24 times, for a total of using 44 91-in logs.   If you wish, you could run the SAS code provided in this article and analyze the logs which show in detail information on the iterations.   Conclusion Implementing Column Generation algorithms using SAS Optimization allows for a clear Master and Subproblems definitions. In this article, the Subproblem is very simple and in “real-world” situations the Subproblem is more complex. Using SAS Optimization, one can declare and solve complex subproblems while clearly implementing the iterations between Master and Subproblem.   References Dantzig, G. B., and Wolfe, P. (1960). “Decomposition Principle for Linear Programs.” Operations Research 8:101–111.   Barnhart, C., Johnson, E. L., Nemhauser, G. L., Savelsbergh, M. W. P., and Vance, P. H. (1998). “Branch-and-Price: Column Generation for Solving Huge Integer Programs.” Operations Research 46:316–329.   Erwin Walraven, Matthijs T. J. Spaan, "Column Generation Algorithms for Constrained POMDPs," JAIR, Vol 68   A. Pessoa, R. Sadykov , E. Uchoa, F. Vanderbeck, "Automation and Combination of Linear-Programming Based Stabilization Techniques in Column Generation," INFORMS journal on Computing, Vol. 30, No. 2   Capelle, Thomas & Cortés, Cristián E. & Gendreau, Michel & Rey, Pablo A. & Rousseau, Louis-Martin, "A column generation approach for location-routing problems with pickup and delivery," European Journal of Operational Research, Elsevier, vol. 272(1), pages 121-131.   Liyuan Zhang, Zhiguo Gui, Jie Yang, Pengcheng Zhang, "A Column Generation Approach Based on Region Growth," IEEE Access, Vol. 7   Qie He, Stefan Irnich, Yongjia Song, "Branch-and-Cut-and-Price for the Vehicle Routing Problem with Time Windows and Convex Node Costs"   Vahid Zeighami, François Soumis, "Combining Benders’ Decomposition and Column Generation for Integrated Crew Pairing and Personalized Crew Assignment Problems," Transportation Science   Maxence Delorme, Manuel Iori, "Enhanced Pseudo-polynomial Formulations for Bin Packing and Cutting Stock Problems," INFORMS Journal on Computing To Try SAS Optimization If you’d like to find out more, visit the SAS Optimization site which has instructions on how to try the software for free. ... View more

The SAS Optimization decomposition algorithm (DECOMP) is a powerful variant of the column generation methods. The algorithm only requires the original (compact) formulation and the constraint partition. It does not require the explicit formulation of the subproblem nor the dual information.   In this article, I use the well-known Cutting Stock Problem to show how to use the DECOMP algorithm to solve its compact formulation. In my article Implementing Column Generation using SAS Optimization in SAS Viya, I also used the Cutting Stock Problem to show how in SAS Optimization one can formulate a typical Column Generation problem with clear iterations between Master and Subproblem. In this article, I compare both approaches: their mathematical models, their results, and when each of these approaches should be used.   SAS Optimization is the general-purpose optimization product that takes advantage of the distributed, cloud-based computational environment that SAS Viya provides. Large-scale problems benefit from running in a SAS Viya environment and their run time can be further shortened by parallelizing their subproblems. DECOMP can be implemented in distributed mode. DECOMP DECOMP provides an alternative method of solving linear programs (LPs) and mixed integer linear programs (MILPs) by exploiting the ability to efficiently solve a relaxation of the original problem. In our experience, when the problem has a block-angular structure, or a great deal of symmetry, DECOMP often outperforms other algorithms. It automatically detects identical subproblems and uses Ryan-Foster branching if applicable. In Column Generation, one formulates the Master Problem and its subproblem.   The standard linear or mixed integer linear program has the formulation described by the first set of equations below. We can form a relaxation of the preceding mathematical program by removing the master constraints (matrices D and B). The resulting constraint system, defined by the matrix A and vector b, forms the subproblem. If the subproblem can be written in a block-angular form described by the second set of equations below, then DECOMP is often the best algorithm to use because it can achieve dramatic performance improvements over other branch-and-cut formulations.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The master problem is a linear program that is defined over a potentially large number of variables that represent the weights of a convex combination. The points in the convex combination satisfy the constraints that are defined in the subproblem. The master constraints of the original problem are enforced in this reformulated space. In this sense, DECOMP takes the intersection of two polyhedra: one defined by original master constraints and one defined by the subproblem constraints. Since the set of variables needed to define the intersection of the polyhedra can be large, the algorithm works on a restricted subset and generates only those variables (columns) that have good potential with respect to feasibility and optimality. This generation is done by using the dual information that is obtained by solving the master problem to price out new variables. These new variables are generated by solving the subproblems over the appropriate cost vector (the reduced cost in the original space). This generation is similar to the revised simplex method, except that the variable space is exponentially large and therefore is generated implicitly by solving an optimization problem. This idea of generating variables as needed is the reason why this method is often referred to as column generation.   DECOMP is used to solve MILPs, and other complex problems, such as Mixed Integer Nonlinear Programs (MINLP) or Network Problems, if they can be reformulated as MILPS. For examples and more information, view the SAS Optimization documentation for Mathematical Optimization Programming.   For an interesting application of the DECOMP algorithm solving a Network Problem read The Kidney Exchange Problem by Matthew Galati in Blogs on Operations Research with SAS. The Cutting Stock Problem using DECOMP Formulation In Implementing Column Generation using SAS Optimization in SAS Viya, I described how a column generation heuristic works and how to implement it using SAS Optimization. Again, that heuristic can be used to solve complex problems, without the requirement of having a block- angular structure, or a great deal of symmetry. In that article, I used a trivial cutting stock example to illustrate the concepts, and the heuristics.   The image below shows the Column Generation formulation of the Cutting Stock Problem     In the DECOMP algorithm, one must write the compact formulation, and indicate where the partitioning is done by using the .block suffix on constraints (or use one of the built-in methods to automatically identify block structure). The blocks must be disjoint with respect to variables. Writing a clever mathematical formulation of the problem can be the difference between solving the problem, or not solving it at all.   One can formulate the Cutting Stock problem as a problem whose objective is to find the minimum number of raw-stock of fixed capacity (width) that are needed to satisfy the demand of items of varying size. The compact formulation of this problem is shown in the following image     The first constraint ensures that each pattern doesn't exceed the capacity constraint. The second constraint ensures that all items will have demand satisfied, and the third constraint effectively provides an upper bound for the numAssigned variable making the bounds tighter and the problem solves more efficiently. The partitioning is done on the first and the third constraints. In this example, DECOMP recognizes that Ryan-Foster branching can be used. Solution Code: The Cutting Stock Problem using DECOMP The example I solved in my first article on Column Generation assumed that we want to cut a 91-inch log into smaller stocks of different widths and demands as shown in this table     Here is the code I wrote in SAS Studio V to implement the compact formulation of the Cutting Stock Problem using the DECOMP algorithm.   /* Cutting-stock problem solved using the Decomposition algorithm */ cas mySession sessopts=(caslib=CASUSER timeout=1800 locale="en_US"); libname CASUSER cas caslib=CASUSER ; data CASUSER.indata; input items demand width; datalines; 1 78 25.5 2 40 22.5 3 30 20 4 30 15 ; %let capacity=91; proc optmodel sessref="mySession"; /* declare parameters and sets */ num capacity = &capacity; set ITEMS; num demand {ITEMS}; num width {ITEMS}; num num_raws = sum {i in ITEMS} demand[i]; set RAWS = 1..num_raws; /* read input data */ read data CASUSER.indata into ITEMS= [items] demand width; /* define problem */ var UseRaw {RAWS} binary; var NumAssigned {i in ITEMS, j in RAWS} >= 0 <= capacity / width[i] integer; minimize NumberOfRawsUsed = sum {j in RAWS} UseRaw[j]; /* respect capacity of each raw */ con knapsack_con {j in RAWS}: sum {i in ITEMS} width[i] * NumAssigned[i,j] <= capacity suffixes=(block=j); /* satisfy demand of each item */ con demand_con {i in ITEMS}: sum {j in RAWS} NumAssigned[i,j] >= demand[i]; /* if NumAssigned[i,j] > 0 then UseRaw[j] = 1 */ con ifthen_con {i in ITEMS, j in RAWS}: NumAssigned[i,j] <= NumAssigned[i,j].ub * UseRaw[j] suffixes=(block=j); /* Decomposition: */ solve with milp / decomp varsel=ryanfoster; /* These two lines will create the datasets that contain the optimal values for the decision variables */ create data CASUSER.CG_data from [i j] NumAssigned; create data CASUSER.useRaw_data from [j] useRaw; /* We want to extract the information on the patterns that are useful, that is, we want to know how to cut the raw stocks*/ create data CASUSER.CG_data_nonZero from [i j]={i in ITEMS, j in RAWS: NumAssigned[i,j]>0} patterns=NumAssigned; create data CASUSER.useRaw_data_nonZero from [j]={j in RAWS: useRaw[j]>0} use=useRaw; run; quit; /* After you have obtained the results, and you are done don't forget to terminate your CAS session */ /* CAS mySession TERMINATE; */ Analysis of Results As expected, both formulations provide the same result and DECOMP is faster. There are several outputs from the code and one is the Solution Summary, shown below. Here you can see the results indicate the optimal solution uses 44 raw stocks.     Remember, the decision variables are: R j represents the UseRaws, that is the raw stocks that are W-in wide which will have a specific pattern. For example, R 44 represents the 44 th raw stock. k ij represents the number of times the i-th item will be assigned in Rj. For example, if k 44,4 = 6, that means that the 4 th item will be assigned 6 times to the raw stock number 44. The 4 th item is 15 in long. Because 6 * 15 = 90, there will be one-inch waste out of the raw stock #44 because its length 91 is in. There are two important output tables with the result from my code: USERAW_DATA_NONZERO which contains UseRaws=1, the number of raw stocks, it has 44 rows CG_DATA_ZERO which contains the k ij values that are non-zero; it has only 107 rows. I arranged these values into the 44 raw stock thus each row represents one raw stock, and the way it will be cut into the i-th different lengths. In a real-world application, the results are in the thousands or hundred-thousands rows, and they are passed into another custom application to further analyze and implement the results. For the small example I used in this article, I used SAS Visual Analytics (VA) to visualize the results in CG_DATA_ZERO. Notice in the screenshot below that there are: 24 rows with pattern (2,1,0,1) 15 rows with pattern (2,0,2,0) 4 rows with pattern (0,4,0,0) 1 row with pattern (0,0,0,6). Similarly, for the other rows. Again, these results are the same as the ones obtained in the first article.     The 24 rows with pattern (2,1,0,1) indicate that 24 91-in raw stocks will be cut as two 25.5", one 22.5" and one 15" logs. Similarly, for the other rows. Again, these results are the same as the ones obtained in the first article. Using Open Source to execute SAS Optimization’s DECOMP The sasoptpy modeling package gives you the ability to write optimization models in Python and solve them using SAS Optimization solvers. The sasoptpy package provides a convenient modeling framework for programmers who already use Python. This package is available on GitHub, as there are other SAS open source projects.   The SAS Global Forum paper, Optimization Modeling with Python and SAS Viya, provides examples on how to use sasoptpy, and one of them is the Kidney Exchange Problem. This paper provides a clear definition of the problem, data generation, how to define the Mathematical model, call DECOMP, and use Matplotlib and NetworkX to graph the results. Summary Column Generation Techniques are alive and well, they solve a variety of large-scale complex and useful problems. Column Generation can be easily implemented using SAS Optimization and take advantage of distributed environments. If the problem has a block-angular structure, or a great deal of symmetry, our recommendation is to try DECOMP. References Galati, M.V. (2009). “Decomposition in Integer Linear Programming.” Ph.D. diss., Lehigh University. Acknowledgement To Rob Pratt for sharing his knowledge on Mathematical Optimization and for providing feedback on the first draft of this article. ... View more

This is the second part of a two-part series. In Part 1, I focused on the Concept rule types. In this article, I will concentrate on the Fact rule types.   Facts are related pieces of information in the text that are located and matched together.   Concept and Fact rule types are used to address these questions: In a series of documents, how can I find specific information, patterns or relationships between a series of words? In a sentence, can I extract ALL the information that is between two important words, independently of the pattern within those two words? Extracting this type of information requires subject matter expertise and knowledge on how those patterns and relationships are present in the language, and SAS Visual Text Analytics (VTA) is the tool to use. It combines the power of Natural Language Processing (NLP), Machine Learning (ML) and human insight to build linguistic rules.   In Part 1, I wrote about the rule types CLASSIFIER, CONCEPT, C_CONCEPT and CONCEPT_RULE. They are very useful in exploratory analysis. In this article, I write about the Fact rule types which include PREDICATE_RULE and SEQUENCE. The order I addressed these rules corresponds to their complexity. Remember, always use the simplest rule possible to get the job done.   I will continue using the hospitality data and Custom Concepts introduced in Part 1.   Facts can be identified within a Custom Concept. For example, suppose you want to identify fun activities for families with children. First, we build one custom concept that identifies families and a separate custom concept that identifies fun activities. Then we build a Fact named Fact_family_fun using those concepts to find reviews where those two concepts are related.   Another example is seen in Christina Hsiao’s article, Automatically extracting key information from textual data, which illustrates the process on how to find the relationships between patients’ reactions and where on the body they occurred. In that article, the concept BodyParts (muscle, arm, head, etc.) and the concept LocalizedSymptoms (pain, redness, fever, etc.) are used to build the Fact named PatientSymptoms.   Further examples can be found in the SAS published book, SAS Text Analytics for Business Applications (by Jade, Belamaric Wilsey and Wallis). Here the combined authors describe how to conduct an Information Extraction process efficiently. It provides a wealth of information on LITI rules: best practices, tips, how to avoid common mistakes, etc. From its section What does this book Cover “… because real-world examples are essential for increased relevance to users; the book presents best practices from seasoned practitioners through realistic use cases and real data as much as possible. “ Rule Type SEQUENCE SEQUENCE is used when the order of the items in the Fact is important, and you want to extract several terms in highly structured text, where you know that the pattern specified in the rule exists. A sequence rule can detect a structure so that each term in the Fact matches in the order that you specify with no intervening items. Example #1 To illustrate this LITI rule, I will define the concept Sequence_funActivitiesTypes. This concept uses two helper concepts funActivities and funActivitiesType:   Step 1: Define the helper custom concepts   Define the concept funActivities as   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Define the concept funActivitiesType as     Step 2: Now I will use the Sequence rule which combines the two custom concepts I just defined above. This Fact named Sequence_funActivitiesType is defined as     To best illustrate how the sequence rule works, I will use the Test Sample Text tab. This tab is used to test the rule that is defined in the Edit Concept window. Only the text that matches the defined rule will be highlighted.   In the screenshot below, the rule SEQUENCE:(type,activity):_type{funActivitiesType} _activity{funActivities} is defined. Notice that a match is found for the first paragraph only, it is highlighted in blue below. I’ve used yellow to highlight similar texts but notice that they don’t match the SEQUENCE being defined.     To extract “Museum of Arts” or “Museum of Fine Arts”, we must write a different sequence rule from the previous one, Sequence_funActivitiesType. The concept Sequence_FineArts shows how to find those matches     A possible question at this point could be: Should the rules used to define Sequence_FineArts be combined with the concept Sequence_funActivitiesType? Or, more generally, should I build custom concepts with several rule definitions? In this example, yes, you might want to combine these SEQUENCE concepts; however, this might not apply for other projects.   A project which includes concept definitions with many different rule types will be hard to maintain. In general, it is easier to work on developing the LITI rules using the Test Sample Text tab to verify a rule is working as I expect, then I can determine how I need to combine rules after I have everything working. Rule Type PREDICATE_RULE PREDICATE_RULE is used to extract several pieces of information using Boolean, distance, and morphological operators and part of speech tags. If you need a brief review of these items check Part 1. The PREDICATE_RULE type should be used where the CONCEPT_RULE cannot achieve the same results. Order in which the elements to be extracted appear in the sentence is not restrictive for this rule type. This rule extracts the text between the two elements. Example #2   We can use predefined concepts in a PREDICATE_RULE; as in this example, where I am using nlpPlace. In the first paragraph, we can see that in this rule type, the order of the terms extracted is not necessarily the order in which they are written in the rule. Also, notice that all terms are extracted between concept1 and concept2, which is the custom concept funActivities defined earlier in this article. Example #3 We can extract whole sentences, for example, if we wanted to know the room Amenities or hotel Amenities that are mentioned with wonderful hosts. In Part 1, I defined the concepts roomAmenities and hotelAmenities. We can now use these custom concepts to define the concept Fact_host_amenities as   Action Items For further learning about these topics, a helpful reference is the book SAS Text Analytics for Business Applications. Thanks To my SAS Asia Pacific colleagues for attending the VTA workshop and generating the questions that steered this two-part series. ... View more

In part 1 of this two-part series, I will concentrate on the Concept rule types. In the second part of the series, I will focus on the Fact rule types to address these questions: In a series of documents, how can I find specific information, patterns or relationships between a series of words? In a sentence, can I extract ALL the information that is between two important words, independently of the pattern within those two words? These are some examples of those questions applied to a business context: What are the most common complaints/comments customers have on a product? What are the side effects patients report from a group of medical drugs? What is the opinion that people have about a government official? What are the specific times when auto accidents occurred? If you are in the hospitality business, are there a set of features that your customers prefer, and which features are complained about? In a quality control: what were the conditions that caused defective material? What equipment is failing, and what are the types of failure? For Veteran Claims: what is the time the claim was initiated and what is the nature of the claim itself (medical, loans, etc.)? Extracting this type of information requires subject matter expertise and knowledge on how those patterns and relationships are present in the language.  SAS Visual Text Analytics (VTA) is the tool to use to help develop rules that combine the power of Natural Language Processing (NLP), Machine Learning (ML) and human insight to build linguistic rules. LITI is the SAS proprietary programming language thru which these linguistic rules are expressed allowing for highly customized models.   In my previous articles, I have given several examples of LITI rules: Visual Text Analytics identifies urgency over global warming in COP24 tweets How to Extract Key Information from Text Documents How to Quickly Identify Main Themes in Text Documents Movie Topics with the Highest Viewer Ratings: Restructuring Categories in Visual Text Analytics Analysis of Movie Reviews using Visual Text Analytics SAS’s LITI rules are powerful and flexible that one can develop rules for a great variety of business situations. In this article, I will concentrate on the Concept rule types which include CLASSIFIER, CONCEPT, C_CONCEPT and CONCEPT_RULE. In my next article, I will write about the Fact rule types which include PREDICATE_RULE and SEQUENCE. Concepts and Facts in VTA Concepts are useful for analyzing information in context, identifying recurring themes and extracting useful information from documents.  Examples of concepts are:  a book title, last name, city, gender, and so on. VTA provides at least seven* out-of-the-box predefined concepts such as Date (nlpDate), Money (nlpMoney), Organization (nlpOrganization), Percent (nlpPercent), Person (nlpPerson), Place (nlpPlace), Time (nlpTime)  whose LITI rules are already defined in VTA to save development time. These predefined concepts can also be used to create Custom Concepts.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   *Note: There are nine predefined concepts in this screenshot because the selected language is English and there are a few additional supported predefined concepts for English.   Examples of predefined concepts would return matches such as: nlpDate: March 12 nlpMoney: $5.20 nlpOrganization: SAS Institute, Inc nlpPercent: 5% nlpPerson: Martin Luther King Jr. nlpPlace: Seattle nlpTime: 8:20 AM The screenshot below shows some of the matches for the nlpNounGroup predefined concept.     Facts are related pieces of information in text that are located and matched together and there are specific LITI rules to identify them.  I will cover Facts in more detail in the second post.   Custom concepts, on the other hand, are created by writing LITI rules to recognize items in context so that we can match and extract only the pieces of the document that match the rule.  For example, you can specify that the concept kitchen is identified when the terms refrigerator, sink, and countertop are encountered in the document.   LITI syntax includes Boolean and distance operators and can reference part-of-speech tags. It is easier to understand LITI rules by providing examples; for most examples in this article I will use a real-world data set with customer reviews of a hospitality firm.   For the rule types CONCEPT, C_CONCEPT and CONCEPT_RULE one can use morphological expansion operators to return inflected forms of a word     also Part-of-speech (POS) tags which enable you to locate matches by the part of speech that the searched item belongs to, rather than locating a specific term     Add a New Concept This screenshot shows the steps to create a new concept. This example shows the concept LITI syntax for the HotelAmenities custom concept .     Alternatively you could use the New concept button:     Rule Type CLASSIFIER This rule type is the initial building block to defining custom concepts. It identifies single terms or strings that you want matched in context. Example #1 Define the concept myKitchen as     Example #2 Define the concept RoomAmenities as     Rule Type CONCEPT This rule type identifies related information by referencing other concepts. Example # 3 Using the custom concept RoomAmenities defined in Example #2, I define the custom concept NiceRoom as     These rules read: Return matches for the string “hotels with” followed by a match to the HotelAmenities Return matches for the terms “room with” followed by a match to the RoomAmenities concept Example #4 Assume you previously defined the custom concepts HotelAmenities, RoomAmenities, ConvenientLocation. By referring to those previously defined custom concepts, I will define the conceptBestOptions as     The custom concept ConvenientLocation is defined in Example #9 later. Example #5 The hospitality firm dataset had documents where the owner of the facility is referred to as host. A simple rule type CONCEPT can be used to define the custom concept myHost as     Notice the ::PN syntax. This will help match proper nouns.    Matches like the last one shown above indicate we need to refine further the concept myHost since we are getting matches for URL, Nespresso and we want to narrow this down to match for people’s names. Example #6 As I mentioned above, we can further refine the previous Example #5 and find better matches for good hosts, by defining the concept wonderfulHost as     Notice the addition of :PN for proper noun, :V for verb , and :A for adjective. Rule Type C_CONCEPT This rule type is used to extract information that occurs in a specific context or pattern. To extract we use “_c”. As with the rule type CONCEPT one can use morphological expansion operators and POS tags. Example #7 This example doesn’t use the hospitality dataset but illustrates this rule type very well.   Assume these are concepts already defined: concept governmentOfficial that contains words such as president, governor, majority leader, senator, and senators, and concept stateUS that includes names of USA states By defining the concept officialAndState as   C_CONCEPT: governmentOfficial _c{_cap _cap} :Prep stateUS   A match for this concept will be this phrase:   Governor Roy Cooper of North Carolina has proclaimed May 7-11, 2018 as National Teacher Appreciation Week   In this rule with use _cap which represents any capitalized word, and to extract two capitalized words we used _c{_cap _cap}. The string extracted is “Roy Cooper”. Also, notice that we use the POS tag :Prep   However, the following text would not produce a match because the word and is not a preposition: Senators Phineas Craymoor and Garrett Garcia from North Carolina pushed the bill through. Rule Type CONCEPT_RULE This rule type is more powerful than the previous ones, and therefore uses more computational resources. It uses Boolean and proximity operators to determine matches.   The most frequently used Boolean and Proximity Operators are: AND, OR, NOT, SENT (sentence), DIST (distance). Here are useful examples on how to use them. Example #8 NOT is used to find matches if the argument does not occur in the whole document. NOT must be used with the AND operator. Let’s define the concept trainStationInWalkingDistance as     Example #9 SENT_n matches if all arguments occur IN A SENTENCE within n (or fewer) tokens of each other, regardless of their order.   DIST_N matches if all arguments occur within n (or fewer) tokens of each other, regardless of their order.   In Example #4 above, I used the concept convenientLocation which is defined as     This last example shows the importance to catch misspellings … and how challenging and interesting is to do Text Analytics!   I mentioned in my previous post “Analysis of Movie Reviews using Visual Text Analytics” as in all analytical projects, the discovery process in Text Analytics projects requires several iterations where the insights found in one iteration are used in the next iterations. In relationship to the linguistic rules, one must determine if the new rules are an improvement over the ones used in previous iterations and find how many true positives and false positives are matched by the new rules. This process should be repeated until one obtains the precision desired.   My next article will be on rule types for Facts, be sure to check it out! Additional Resources Check out this new book: SAS Text Analytics for Business Applications. Check out these three articles on NLP, Artificial Intelligence and using Text Analytics to analyze descriptions of side effects or adverse events that patients have reported following a vaccination: Natural Language Processing Artificial intelligence, machine learning, deep learning and beyond machine learning, deep learning and beyond ... View more

Unstructured data, in particular text data, is rapidly growing and practically it is everywhere: blogs, customer comments, veteran claims, tweets, notes from doctors and nurses, etc. Including text data in predictive models improves the predictive power of these models and provides richer insights.   In this article I show how to restructure the categories in the VTA pipeline. Also, I illustrate how in Visual Analytics (VA) one can easily: 1)  build a Decision Tree using the unstructured data obtained from the Topics node, and 2) analyze the restructured categories data.   I will use a data set that contains information on 1527 randomly selected movies: their titles, reviews, MPAA Ratings, Main Genre classifications and Viewer Ratings.   With VTA, users can manage the full analytics lifecycle with large volumes of unstructured data; they can access and prepare data, explore document collections, build text models, generate reports, and deploy the models against new unstructured data within their existing systems or processes. Data Preparation and Data Exploration Typically when we import text data, there are character strings that are created from that import process. My movie dataset was originally in excel, and when I explored it in VA I noticed that it had the string ‘_x000D_’ attached to several terms, or at the beginning of each Review. I decided to remove it.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   In Viya 3.3, it is very easy to navigate between different applications. In the SAS Data Studio – Prepare Data application, I used the code shown in the photo below to remove all occurrences of “_x000D_”   Working in the VTA Pipeline In the SAS Data Studio – Build Models application, I assigned variable roles as described in my previous VTA posts.   I used the default Text Analytics Pipeline, and only restructured the categories in the Categories node to as described in the next paragraphs.     Categories are organized in a logical, hierarchical structure, also known as a “taxonomy. The purpose of the next steps is to show how to restructure categories, different people would use different subcategories in the next steps. The general idea is to create a new category by combining several subcategories.   I created three new categories: childrenMovies, ActionMovies and MysteryAndSuspense. After the changes, there are three main categories: mainGenre (which has as subcategory childrenMovies), ActionMovies and MysteryAndSuspense.   childrenMovies category Working in the Categories node, I created the new childrenMovies category using this rule o (OR,(AND,(NOT,(OR,"adults","adult","suitable for children","rated R","strip@","suck@","crude humor","gore","horror","murder","obscenity","drug use@")),"Wizard of Oz"),(AND,"pixar"),(AND,(OR,"animator","animators")),(AND,(OR,"voiced","voices","voicing","voice"),(OR,"cartoon","cartoons")),(AND,(OR,"cartoon characters","cartoon character")),(AND,(OR,"lesson","lessons"),"animated"),(AND,"live action"),(AND,"jeffrey",(OR,"features","feature")),(AND,"3-d")) Deleted the original categories of Family, Animation and Kids because as explained in the previous article the new childrenMovies category selected more appropriate children movies than the original categories. ActionMovies category Created the new Category ActionMovies, and created several sub-categories by moving under it the categories Action, Bond 007,Martial-Arts, Sports, and War MysteryAndSuspense category Created the new Category MysteryAndSuspense, and created several sub-categories by moving under it the categories Crime, Cult, Horror, Mystery and Suspense.     To create subcategories (ex: War) under a category (ex:ActionMovies) these steps were taken: Right-click on ‘All Categories’ and select ‘Add new category’. Name it ‘ActionMovies’ and hit ENTER. Right-click on the “War” category, select Cut and Paste it under ‘ActionMovies’ as a sub category. Repeat step 2 for the subcategories Action, Bond 007,Martial-Arts, and Sports The Categories node must be run for these changes to be applied. Saving Output Datasets in VTA To export the results of the categorization model into a format suitable for visualization and exploration, I exported two output datasets, one from the Topics node and the second data set from the Categories node. The steps are the same, just select the correct node: In the selected node (i.e. Categories), right clicked it and then selected “Save data table”, as shown in the photo below. Save it as, for example, VTA_MovieBlog2_Categories_Data   Build a Decision Tree in VA In this section we build in VA a predictive model using unstructured data using the output data from the Topics node which was saved as described in the section Saving Output Datasets in VTA.   From the SAS Home, select SAS Visual Analytics – Explore and Visualize Data and open the VTA_MovieBlog2_Topics_Data.   Select from the Object panel the Decision Trees object and drag it into the canvas     On the right, assign Roles and select as Response: ViewerRating, then click on + Add to select the Predictors, and select the variables as shown in the photo below. Do not select variables with levels (e.g._1_0_...)   The topic with the highest Viewer Rating is +comedy,+funny,+joke,+laugh,sandler. Interestingly, the topic with the lowest Viewer Rating is +science,+fiction,+alien,+science Fiction.   Visualize Restructured Categories in Visual Analytics Continuing with the same report in Visual Analytics, add a new page to it, and select the VTA_MovieBlog2_Categories_Data save in the section Saving Output Datasets in VTA From the Data pane, select + New data item and select Hierarchy.   Name the new hierarchy:  Movie Category Hierarchy and select all of the category_level data to the Selected items panel.  Click OK.     To analyze categories hierarchy using a bar chart, drag and drop the Movie Category Hierarchy into the canvas which will auto chart a bar chart that shows the different categories.  Explore the bar chart (to go down a level in the hierarchy, double click on a bar.  To go up a level use the bread crumbs at the top left of the canvas).  These are the bar charts produced, notice how they nicely relate to the new categories         Conclusion In SAS Viya one can easily access and prepare data, explore document collections, build text models, and generate reports. In future articles we will see how to deploy the models against new unstructured data. ... View more

Some of us are concerned about Global Warming, pondering on what collective and individual actions we can take to limit it. On October, the United Nations Intergovernmental Panel on Climate Change (IPCC) issued its special report SR1.5 indicating that there are clear benefits to limiting warming to 1.5 °C compared to 2 °C. The SR1.5 report was to be a key scientific input to the COP24 conference. COP24 is the informal name for the 24th Conference of the Parties to the United Nations Framework Convention on Climate Change held in Katowice, Poland from 2-14 December 2018. A main agreement reached at COP24 was on how governments will measure, report and verify their emissions-cutting efforts.   The topic deserves wide-ranging analyses. To start, I analyzed 8,000 tweets issued at COP24 during four different days using Visual Text Analytics (VTA) and Visual Analytics (VA). SAS Visual Text Analytics (VTA) is the SAS offering designed to effectively extract insights from unstructured data in large scale. Currently, VTA supports 32 languages and it has an open architecture supporting 3rd-party programming interfaces. Offered on the SAS Viya architecture, VTA combines the power of Natural Language Processing (NLP), Machine Learning (ML) and Linguistic Rules. Because of these capabilities highly customized models can be developed in VTA. In this article, I use basic linguistic rules to find how renewable energy is mentioned on the tweets.   What is the IPCC? The Intergovernmental Panel on Climate Change (IPCC) is the UN body for assessing the science related to climate change. It was established by the United Nations Environment Programme (UN Environment) and the World Meteorological Organization (WMO) in 1988 to provide policymakers with regular scientific assessments concerning climate change, its implications and potential future risks, as well as to put forward adaptation and mitigation strategies. It has 195-member states. As mentioned above, its October Special Report on Global Warming was to be a key scientific input to the COP24 conference. However, the U.S., Russia, Saudi Arabia, and Kuwait blocked its acceptance at the conference.   This article describes the main steps of this analysis. If you would like to see more detailed instructions, you could check my previous article, Discover Main Topics on #MLKDayofService Tweets Using SAS Visual Text Analytics.   Step One: Bring Twitter Data into Visual Analytics Twitter’s public API is used to import data. There are limitations on what data and on how much data SAS can download using the Twitter public search API. The maximum number of tweets in each download is 2000. Still, this article shows the main steps to follow for this analysis.   In Visual Analytics, you need to specify the search term(s) to import (I used the hashtag #COP24), the maximum tweets to import (up to 2,000) and whether to import retweets.   (select any image to enlarge)   Step Two: Create one table that contains all tweets to analyze I used SAS Studio 5 to create one CAS table containing the 8,000 tweets. The table must contain an ID Column that has unique values. I ran a Profile to make sure the docID column is unique. If it would not have been unique, I would have created a column with unique values.   Step Three: Bring the table into Visual Analytics and Set a Unique Row Identifier for your data From the SAS Home Menu, I selected Visual Analytics (VA). I converted docid from a Measure to a Category variable, and indicated docid was the unique row identifier.   Step Four: Add Text Topics Object and assign Roles Continuing working in VA, I did the steps shown in the photo below: On the left side bar, clicked Objects Selected Text Topics and dropped it into the report page Selected English On the right-side bar, clicked Roles To the Document collection, added “Body”. Selected English and click OK Also under Options, I selected  “Analyze document sentiment”   Note: VTA will auto-generate categories if you add additional categorical variables in this step, but variables added must have less than 400 levels. My twitter data didn’t have any variable that I could utilize at this step. In VTA 8.4 this limit is removed. Also, VTA 8.4 works with text that has emojis.   The photo below shows the out of the box results obtained in Visual Analytics: a word cloud with the most frequent terms, the list of main topics that were automatically extracted as well as their sentiments. Notice the topic with the terms “+la, de, en,+el” and since these term do not add value to the analysis I removed them in later steps in VTA. I saved this VA report.     Step Five: Create a Visual Text Analytics project From the SAS Home Menu, I selected Build Models, that took me to SAS Model Studio, where I selected New Project and entered the following information:     After saving, in the Data tab I assigned to body the role Text, and selected the Pipeline tab to get the default VTA pipeline. In the next steps I will show how to add customizations in the Concepts, Text Parsing and Categories nodes.   Step Six: Develop Customized Concepts Concepts are key pieces of information: energy, renewable energy, solar, food, carbon claim, transportation, etc. Concepts are useful for analyzing information in context and for extracting useful information.   VTA provides 9 predefined concepts such as dates, people, places, measurements, mentions of currency, etc. which are concepts whose rules are already written to save development time. Also in VTA you can write rules for recognizing concepts that are important to you, thereby creating custom concepts. I wanted to create custom concepts to find specific information in the tweets. In later steps, I used one of them “myEnergy” to build a new category.   LITI rules are SAS proprietary linguistic rules, and they are used to developed custom concepts. Basic Boolean operators are used to define custom concepts and categories.   Here is a very brief introduction to them: AND/NOT operators are applied to the whole document. There other operators that search within the same sentence (SENT), the same paragraph (PARA) or a number of terms (DIST) Any line that starts with “#” is a comment Use CLASSIFIER to match a literal sequence Use CONCEPT_RULE to use Boolean and proximity operators. The term extracted should use _c{ } I developed six custom concepts, so I could see the tweets that mention those concepts: myClimateReport, myEnergy, myTransportation, myAction, myForest, and myFood. The photos below show for some of them, their LITI rules and at least one tweet that VTA matched using those rules.   For myClimateReport concept, notice that out of 8,000 tweets there were 162 where VTA found the literal sequences:” Climate Report”, “ClimateReport” or “Intergovernmental Panel on Climate Change”.     I used in later steps myEnergy concept to build a category. The photo below shows some tweets that matched myEnergy concept recommending using bicycles (that I love) or installing solar systems on residential roofs. The complete tweet was: “Bangladesh has been installing 50,000 #solar systems on residential roofs a month”     The photo below shows myForest concept appeared in 56 tweets out of the 8,000 totals, and that when I searched for the word “action”, VTA found 3 tweets out of those 56. I selected the tweet shown because it recommends something I can do: reduce consumption and promote renewables, while still planting some trees in my garden. Yes, I know the solution to this problem is not that simple, but it seems that there is something we all can do.     The concept myFood found 155 matches out of 8,000. The first tweet shown reminded me of a BBC article on Global Warming which listed several “individual actions.” I list them later in this post.     Step Seven: Work in the Text Parsing Node In the Text Parsing Node, unstructured text is parsed and transformed into the structured form of a vector by using NLP and other tools. With the push towards Artificial Intelligence, SAS is applying more machine learning techniques, especially Deep Learning.   At the end of Step Four, I noticed that Visual Analytics listed the topic “+la, de, en,+el” and since these terms do not add value to the analysis I removed them. Also, in this node we can see TermMaps. I was interested on what were the terms that prompted people to retweet. Here is the TermMap for “rt”:     Which made me wonder who is Greta Thunberg? I searched for tweets with that literal string and found this:     I used the VTA default models for Sentiment Analysis which use SAS proprietary rules that identify and analyze terms, phrases, and character strings that imply sentiment. I continued customizations in the Topics Node.   Step Eight: Work in the Topics Node These are the topics that VTA automatically generated:     I selected the topic “urgent appeal, appeal, strong agreement, nece, back” and searched for “greta.” (notice that VTA handles emojis 😊)     and found a retweet to Greta when I selected the topic “climate strike, 14 december, just talk, fridaysf, +strike.”     I decided to promote these two topics to categories as shown in this photo:     Step Nine: Work in the Categories Node It is possible to create new Categories using LITI and Boolean rules. Notice in the photo below four categories: two that I promoted from the Topics node and two that I created using my previously defined custom concepts. The category CleanEnergy uses the custom concept myEnergy. There are 618 tweets in this category. When I searched for the term Pittsburgh, I found three encouraging tweets.     Recommendations on Individual Actions This BBC article has great graphics and lists actions an individual can take recommended by the IPCC SR1.5 report: Buy less meat, milk, cheese and butter and more locally sourced seasonal food - and throw less of it away Drive electric cars but walk or cycle short distances Take trains and buses instead of planes Use videoconferencing instead of business travel Use a washing line instead of a tumble dryer Insulate homes Demand low carbon in every consumer product Conclusion Using VTA, I quickly analyzed COP24 tweets, extracted information on terms of interest to me, and found the main themes discussed in the Conference.   In VTA one can develop highly customized models by combining the power of Natural Language Processing (NLP), Machine Learning (ML) and Linguistic Rules.   References Model Studio: SAS® Visual Text Analytics 8.3 User’s Guide Other articles regarding text analytics BBC’s article Final call to save the world from 'climate catastrophe' ... View more

When one wants to extract useful information from unstructured data, one uses Concepts. A Concept is a key data element such as a book title, last name, city, gender, and so on. Concepts are useful for analyzing information in context and for extracting useful information.   In this article, I will show how to implement Custom Concepts in SAS Visual Text Analytics’ visual and programming interfaces. In the programming interface, I will use Action Sets available in SAS Visual Text Analytics 8.2 in SAS Viya 3.3 and show examples of the LITI rules for CLASSIFIER, CONCEPT_RULE and PREDICATE_RULE.   In SAS Visual Text Analytics, you can write rules for recognizing concepts that are important to you, thereby creating Custom Concepts. For example, if you were planning a vacation and had a series of documents with information on accommodations and their attractions nearby, you could create a Custom Concept called GreatLocation that identifies accommodations in a desirable Location. Also, you could create a Custom Concept called NearToFun which extracts locations near to music events and museums. You could specify that the concept NearToFun is identified when the terms museum, music, band, or festival are encountered in a document.   It is important to mention that Forrester ranked SAS a Leader in The Forrester Wave™: AI-Based Text Analytics Platforms, Q2 2018 where you can read these lines:   “… SAS Visual Text Analytics is fully integrated with SAS Visual Analytics — a self-service BI and discovery tool — both of which run on the highly scalable SAS Viya in-memory grid architecture. SAS's brand speaks for itself as a leader in advanced analytics; as a result, SAS Visual Text Analytics comes with a number of machine learning models.”   Implementing Custom Concepts in the SAS Visual Text Analytics Visual Interface SAS Visual Text Analytics provides nine predefined concepts such as dates, people, places, measurements, mentions of currency which are concepts whose rules are already written to save development time. In the photo below, you can see examples of the nlpNounGroup and nlpMoney predefined concepts.     Custom Concepts are useful for defining a specific concept, or can be referenced as an argument in any of the rules-based definitions. In customized Healthcare or Legal applications, groups of continuous terms and the value of some of those terms might be of ultimate importance.   SAS Visual Text Analytics does context-sensitive matching using complex advanced linguistic rules called LITI (or Language Interpretation for Textual Information). With these rules, concepts are matched in a specific context. LITI rules are SAS proprietary. See in the photos below the process to create a Custom Concept in the SAS Visual Text Analytics visual interface. The first Custom Concept is called HotelAmenities and the second is called testFact.     For HotelAmenities the LITI rules are: CLASSIFIER:Complimentary breakfast CLASSIFIER:Restaurant CLASSIFIER:Free parking CLASSIFIER:Swimming pool CLASSIFIER:Bar CONCEPT_RULE:HOTEL_AMENITIES:(SENT,(OR,"_c{internet}","_c{wifi}","_c{wi-fi}"),"free","lobby")   Some of the matches are “ … a safe and free Swimming pool in the backyard” and “free wifi in lobby”     The LITI rule for Fact extraction is PREDICATE_RULE, for the testFact is: PREDICATE_RULE:TEST_FACT(nlpPlace):(DIST_4,"_nlpPlace{Beacon Hill}",(OR,"Downtown","subway","clean"))   Notice the use of the predefined concept nlpPlace. This rule will match documents that include the term "Beacon Hill" and either of these three terms: "Downtown,” ” subway,” ”clean.”    Notice the documents that are matched with this rule: “… Beacon Hill. Clean, well appointed and convenient” and “… quiet area in Beacon Hill. Very clean. Kitchen is moderately equipped.”   Implementing Custom Concepts in the SAS Visual Text Analytics Programming Interface Previously, I wrote about SAS Viya and Text Mining Action Sets. In this article, I will use Action Sets newly available in SAS Visual Text Analytics 8.3 in SAS Viya 3.4. Action Sets and Actions are important because the same Action Sets and Actions are used no matter the client used to make the request. The examples in this post are worked in CASL, but you could just as easily use Python or Java.   I wrote two short programs. The program ValidateConcept checks that the rule definitions have the correct syntax. It uses the action validateConcept from the action set textRuleDevelop.   There are slight differences in the syntax for LITI rules in the visual and the programming interfaces. The rules with the correct syntax are then used in the program CustomConcept, which uses the action textRuleDevelop from the action set compileConcept to compiles the concept rules and generates an LI binary. This LI binary is used to score a new dataset by the action textRuleScore from the action set applyConcept.   The code for this implementation can be seen below in the Appendix.   Note: If you decide to run the code provided in this article, my recommendation is to copy it into Notepad and then into SAS Studio V. The spaces are key, as well as the quotation marks which should be “.   The main parts of that code are:   ValidateConcept.sas Start a cas session, make caslibs visible in SAS Studio and load data in to CASUSER library The file concept_rule_definition has the rules to be tested. User validate Concept to check the syntax of the concept rules is correct. Notice what is the correct syntax for the LITI rules for CLASSIFIER, CONCEPT_RULE and PREDICATE_RULE.   The output of this program is the table ERROR, and if it doesn’t have any then Number of Rows =0 indicates that the syntax of the rules is correct. Once they are no errors one can continue with the next program.     CustomConcept.sas Start a cas session, make caslibs visible in SAS Studio and load data in to CASUSER library The file concept_rules which contains the concept textual rules The file apply_concept_text to be scored, that is, from which we want to extract the concepts using the concept textual rules defined in the previous file Using the option metrics=true will print in the log the actions executed Use textRuleDevelop to compile the concept rules Use textRuleScore to score the apply_concept_text file The output of the action concept rules is the binary LI file. The output of applyConcept is the second table in the photo below     The results of the textRuleScore are two tables: OUT_CONCEPT and OUT_FACT.   The concept matches are shown in the OUT_CONCEPT table:     The fact matches are shown in the OUT_FACT table:     Conclusion SAS Visual Text Analytics facilitates the development and implementation of Custom Concepts in both its visual and programming interfaces. There are slight differences in the syntax for LITI rules in the visual and the programming interfaces.   References SAS Visual Analytics 8.2: Programming Guide   Thanks to Seung Lee for verifying the syntax of the LITI rule CONCEPT_RULE.   Appendix ValidateConcept.sas /***************************************************************************/ cas mysess sessopts=(caslib=casuser timeout=1800 locale="en_US" metrics=true); caslib _all_ assign; data casuser.concept_rules; length config $300 ; infile datalines delimiter='|' missover; input config$; datalines; ENABLE:COMPANY FULLPATH:COMPANY:Top/COMPANY PRIORITY:COMPANY:10 CASE_INSENSITIVE_MATCH:COMPANY CLASSIFIER:COMPANY: Microsoft CLASSIFIER:COMPANY: Amazon CLASSIFIER:COMPANY: Google ENABLE:HOTEL_AMENITIES FULLPATH:HOTEL_AMENITIES:Top/HOTEL_AMENITIES PRIORITY:HOTEL_AMENITIES:10 CASE_INSENSITIVE_MATCH:HOTEL_AMENITIES CLASSIFIER:HOTEL_AMENITIES: Complimentary breakfast CLASSIFIER:HOTEL_AMENITIES: Restaurant CLASSIFIER:HOTEL_AMENITIES: Swimming pool CLASSIFIER:HOTEL_AMENITIES: Bar CONCEPT_RULE:HOTEL_AMENITIES:(SENT,(OR,"_c{internet}","_c{wifi}","_c{wi-fi}"),"free","lobby") ENABLE:TEST_FACT FULLPATH:TEST_FACT:Top/TEST_FACT PRIORITY:TEST_FACT:15 CASE_INSENSITIVE_MATCH:TEST_FACT PREDICATE_RULE:TEST_FACT(nlpPlace):(DIST_4,"_nlpPlace{Beacon Hill}",(OR,"Downtown","subway","clean")) ENABLE:Top FULLPATH:Top:Top PRIORITY:Top:10 CASE_INSENSITIVE_MATCH:Top ; run; data casuser.apply_concept_text; /* 3 */ length text $300 ; infile datalines delimiter='|' missover; input docid text$; datalines; 1| I just bought an amazon fire tablet 2| microsoft Windows in an operating system 3| In beacon hill location clean studio with easy keypad access 4| a safe and free Swimming pool in the backyard 5| quiet area in Beacon Hill. Very clean 6| free wifi in lobby ; run; proc cas; builtins.loadActionSet / actionSet="textRuleDevelop"; builtins.loadActionSet / actionSet="textRuleScore"; textRuleDevelop.compileConcept / casOut={name="outli", replace=TRUE} config="config" table={name="concept_rules"}; run; textRuleScore.applyConcept / casOut={name="out_concept", replace=TRUE} docId="docid" factOut={name="out_fact", replace=TRUE} model={name="outli"} table={name="apply_concept_text"} text="text"; run; table.fetch / table={name="out_concept"}; run; table.fetch / table={name="out_fact"}; run; quit; CustomConcept.sas /***************************************************************************/ cas mysess sessopts=(caslib=casuser timeout=1800 locale="en_US" metrics=true); caslib _all_ assign; data casuser.concept_rules; length config $120 ; infile datalines delimiter='|' missover; input config$; datalines; ENABLE:HOTEL_AMENITIES FULLPATH:HOTEL_AMENITIES:Top/HOTEL_AMENITIES PRIORITY:HOTEL_AMENITIES:10 CASE_INSENSITIVE_MATCH:HOTEL_AMENITIES CLASSIFIER:HOTEL_AMENITIES: Complimentary breakfast CLASSIFIER:HOTEL_AMENITIES: Restaurant CLASSIFIER:HOTEL_AMENITIES: Swimming pool CLASSIFIER:HOTEL_AMENITIES: Bar CONCEPT_RULE:(SENT,(OR,"_c{internet}","_c{wifi}","_c{wi-fi}"),"free","lobby") ENABLE:CONVENIENT_COST FULLPATH:CONVENIENT_COST:Top/CONVENIENT_COST PRIORITY:CONVENIENT_COST:10 CASE_INSENSITIVE_MATCH:CONVENIENT_COST CONCEPT_RULE: (SENT,(DIST_6,”_c{nlpMoney}”,(OR,”cost”,”reasonable”,”convenient”))) ENABLE:TEST_FACT FULLPATH:TEST_FACT:Top/TEST_FACT PRIORITY:TEST_FACT:10 CASE_INSENSITIVE_MATCH:TEST_FACT PREDICATE_RULE:(nlpPlace):(DIST_4,"_nlpPlace{Beacon Hill}",(OR,"Downtown","subway","clean")) ; run; data casuser.apply_concept_text; length text $200 ; infile datalines delimiter='|' missover; input docid text$; datalines; 1| a safe and free parking at the backyard 2| cost was under $30 including tip and the drivers were great. Everyone in Boston was friendly and helpful 3| Taxi costs between 6-10 dollars for a trip downtown. The home was inviting. 4| Perfect location for Beacon Hill. Clean, well appointed and convenient 5| Also, his 1 bedroom apartment in Beacon Hill is clean, charming, and in the ideal location 6| Safe quiet area in Beacon Hill. Very clean. ; run; proc cas; builtins.loadActionSet / actionSet="textRuleDevelop"; builtins.loadActionSet / actionSet="textRuleScore"; textRuleDevelop.compileConcept / casOut={name="outli", replace=TRUE} config="config" table={name="concept_rules"}; run; textRuleScore.applyConcept / casOut={name="out_concept", replace=TRUE} docId="docid" factOut={name="out_fact", replace=TRUE} model={name="outli"} table={name="apply_concept_text"} text="text"; run; table.fetch / table={name="out_concept"}; run; table.fetch / table={name="out_fact"}; run; quit;   ... View more

Most natural phenomenon and real-world business problems are intrinsically non-linear, for example: Finding the best asset allocation for the fixed amount of money you have for investments The spread of gossip/news in Social Networks, the spread of an infectious disease Equations describing quantum mechanics, electromagnetism, fluid dynamics … the list is endless Often, Linear programming is unable to capture the complex relationships between inputs and outcomes, and as in the examples mentioned above, Nonlinear Optimization must be used and then extended to deal with integer variables becoming a mixed integer nonlinear optimization.   In this article, I will provide examples of problems that can be solved with nonlinear optimization and provide coding examples of new features in SAS Optimization 8.3 that take advantage of the distributed, cloud-based computational environment that SAS Viya provides.   Mixed integer nonlinear optimization is used to determine the best allocation of labor and capital for optimal production of detergents, as well as, for creating trading strategies in the stock market with the purpose of selecting entry and exit points of stocks, to maximize profit and minimize risk.   In 2014, the SAS OR Center of Excellence was a finalist for the Wagner Prize with the entry Statistical & Optimization Techniques for Laundry Portfolio Optimization at P&G. (View the video.) The goal of the project was to set the portfolio strategy for the multi-billion-dollar laundry powder business. This work identified strategies to save $5-20 M and one of the models developed and solved was a Highly Nonconvex Mixed Integer Nonlinear Problem.   The graph below is just in two-dimensions and has two minimum points, one is a local minimum and the second is the global minimum.     Real world problems have many variables and typically a non-linear problem might have many local minima/maxima; imagine an egg carton distorted in multi-dimensions. SAS algorithms can find the global optimum, or a good approximation to it, in a multi-dimensional space.   For the SAS NLP solver it is recommended that the functions describing the objective and the equality and inequality constraints are twice continuously differentiable.   SAS NLP has the option Multistart. When using this option, the local solver solves the problem from different starting points, finding a better local optimum as a result.   The NLP solver implements the following primal-dual methods for finding a local optimum: Interior point trust-region line-search algorithm. It is the default, it uses a primal-dual interior point method. This technique is recommended for both small- and large-scale nonlinear optimization problems. This is the preferred solver if the problem includes many inactive constraints. Active-set trust-region line-search algorithm uses a primal-dual active-set method. This technique is recommended for both small- and large-scale nonlinear optimization problems. This is the preferred solver if the problem includes only bound constraints or if the optimal active set can be quickly determined by the solver. The code below optimizes a function that has many local minima. Without the multistart option the minimum found is 607.03. However, when multistart is used the minimum found is 0.029. A significant difference!   /* Using CAS to solve NLP_Problems that have several local minima */ /* The following SAS statements show how you can solve this problem in a distributed computing environment. After you create a CAS session named mysess, you specify the SESSREF="mySession" option to run the OPTMODEL procedure on CAS. */ cas mySession3 sessopts=(caslib=CASUSER timeout=1800 locale="en_US" metrics=true); proc optmodel sessref="mySession3" nthreads=4; var x{i in 1..5} >= -5 <= 5 init -2; min f=(x[1] - 1)^2 + (x[1] - x[2])^2 + (x[2] - x[3])^3 + (x[3] - x[4])^4 + (x[4] - x[5])^4; con g1: x[1] + x[2]^2 + x[3]^3 = 2 + 3*sqrt(2); con g2: x[2] + x[4] - x[3]^2 = -2 + 2*sqrt(2); con g3: x[1]*x[5] = 2; solve with nlp/ multistart=(maxstarts=20) ; print x; quit;   About this example, note the following: In the CAS declaration: The CAS session is named mySession3, which will be referred by the sessref option in the proc optmodel line. Metrics=true In the proc optmodel: sessref=”mySession3” specifies the name of the CAS session to which you want to connect. It indicates that you want optmodel to interact with that specific cas session. The NLP solver will run in that specific cas session. Within proc optmodel you can reset the current cas session to a different cas session, when for example, you want to access different caslibs from the current cas session. Declarations for decision variables, constraints and objective function to be minimized The line in the code where the option multistart is specified: solve with nlp/multistart=(maxstarts=20) The next example shows how to use the COFOR statement which offers a simple way to exploit parallel processing by solving independent problems concurrently, on either one machine or distributed across multiple machines. You are unlikely to see any benefit from COFOR when each solve is fast.  The best performance improvement from COFOR arises when each solve takes at least several seconds.  Otherwise, the communication overhead swamps any improvement from parallelism.   A key distinction between the COFOR and FOR statements is that in a COFOR loop you are solving independent optimization problems. When you call a solver in a FOR loop, the problems might be independent or not, you might be even calling different solvers.   cas mySession2 sessopts=(caslib=CASUSER timeout=1800 locale="en_US" metrics=true); libname CASUSER cas caslib=CASUSER; Data casuser.Bounds; Input period lower upper; Datalines; 1 0.1 3 2 -3.1 4 3 -6.1 3.5 4 0.1 10 ; proc optmodel sessref="mySession2" ; set Period; num lower{Period}; num upper{Period}; num lowB; num upB ; read data casuser.bounds into PERIOD=[period] lower upper; var x{1..8} >= lowB <= upB; x[1] = 3; x[2] = 3; min f = 0.4*(x[1]/x[7])^0.67 + 0.4*(x[2]/x[8])^0.67 + 10 - x[1] - x[2]; con c1: 1 - 0.0588*x[5]*x[7] - 0.1*x[1] >= 0; con c2: 1 - 0.0588*x[6]*x[8] - 0.1*x[1] - 0.1*x[2] >= 0; con c3: 1 - 4*x[3]/x[5] - 2/(x[3]^0.71*x[5]) - 0.0588*x[7]/x[3]^1.3 >= 0; con c4: 1 - 4*x[4]/x[6] - 2/(x[4]^0.71*x[6]) - 0.0588*x[8]/x[4]^1.3 >= 0; con c5: 0.1 <= f <= 4.2; cofor { p in Period} do; lowB= lower[p]; upB=upper[p]; put lowB= ; put upB= ; solve with nlp / algorithm=activeset multistart=(maxstarts=20) seed=12345 nthreads=4; print x; end; quit; CAS mySession2 TERMINATE;   In this example, COFOR is used to solve four different instances of a highly Nonlinear Optimization problem. For each case, the variables bounds change. In addition to declaring the CAS session and caslibs, other key steps in this code are: How the read data statement uses Period as an index, which is a column in the casuser.bounds cas table. The COFOR loop that is used to change the variables upper and lower bounds For an additional COFOR example, check out the SAS Global Forum paper SAS1758-2018: Using SAS/OR® to Optimize Scheduling and Routing of Service Vehicles by Rob Pratt.   The last example shows how to use the runOptmodel action in the Optimization Action Set. The runOptmodel action provides access to the OPTMODEL modeling language and multiple mathematical programming solvers within the CAS environment, one of them the NLP solver. Moreover, runOptmodel provides optimization modeling access from any programming language that SAS Viya supports, currently R, Python, and Lua.   proc cas; loadactionset "optimization"; source pgm; /*to assign optmodel code to pgm string variable*/ var x{i in 1..5} >= -5 <= 5 init -2; min f=(x[1] - 1)^2 + (x[1] - x[2])^2 + (x[2] - x[3])^3 + (x[3] - x[4])^4 + (x[4] - x[5])^4; con g1: x[1] + x[2]^2 + x[3]^3 = 2 + 3*sqrt(2); con g2: x[2] + x[4] - x[3]^2 = -2 + 2*sqrt(2); con g3: x[1]*x[5] = 2; solve with nlp/ multistart=(maxstarts=20) seed=12345 nthreads=4; print x; endsource; optimization.runOptmodel / code=pgm; run; quit; CAS mySession3 TERMINATE;   Notice the changes that are needed to use runOptmodel: Use proc cas Load the action set Optimization and use the CASL statement source which embeds text in the program and assign it to the pgm string variable Notice that the Action Set Optimization calls runOptModel Conclusion SAS Optimization provides algorithms for Solving Nonlinear Optimization problems which take advantage of the distributed, cloud-based computational environment that SAS Viya provides. As shown in the coding examples, implementing these algorithms is very easy.   References Mathematical Optimization Programming Guide   SAS1758-2018: Using SAS/OR® to Optimize Scheduling and Routing of Service Vehicles by Rob Pratt   SAS OR Center of Excellence’s finalist entry to the 2014 Wagner Prize - Statistical & Optimization Techniques for Laundry Portfolio Optimization at P&G ... View more

In our daily lives, we benefit from the application of Mathematical Optimization algorithms. They are used, for example, by GPS systems, by shipping companies delivering packages to our homes, by financial companies, airline reservations systems, etc. Mathematical Optimization, also known as Mathematical Programming, Operations Research, or simply Optimization, is a discipline that solves a great variety of applied problems in diverse areas: medicine, manufacturing, transportation, supply chain, finance, government, physics, economics, artificial intelligence, etc.   In this article, I provide several examples that will help people identify situations where Optimization can be applied, while avoiding mathematical jargon as much as possible. As you read, think about the implications of combining these algorithms and the vast computational power currently available.   Gartner Analytics Maturity Model proposes four phases: Descriptive (What Happened?), Diagnostic (Why did it happen?), Predictive (What will happen?) and Prescriptive (How can we make it happen). Optimization is at the heart of the Prescriptive stage indicating how to use resources efficiently to achieve the best possible goal under a series of conditions.       In an optimization model, the goal can be to minimize cost in a production system (i.e. oil refinery) where the resources are labor, raw materials, etc. and production targets must be achieved. In a hospital, the goal can be to minimize the wait time for patients in the emergency room before they are seen by a doctor where the resources are the doctors, the nurses, the rooms, the equipment, etc. In Marketing, the goal can be to maximize the profit obtained by targeting the right customers under budget and operational conditions. In a Humanitarian Operation the goal would be to reach as many affected people as quickly as possible to distribute resources water, food, and medical services by designing optimal routes.   More properly (and succinctly), one can say that an optimization problem searches the decision variables which maximize or minimize an objective function under several conditions or constraints. That definition hints that optimization problems are expressed as mathematical models and to solve them requires specialized training. In this article, I will provide several examples describing the problem to be solved. For most of these examples, I will identify their decision variables, objective, and constraints and provide a close description of the meaning of the decision variables and the constraints, since providing the exact definition would involve writing the mathematical models.   I selected most of the examples from the Edelman Award and the Wagner Prize competitions. I have enjoyed serving as the Chair for the Wagner competition in 2017 and 2018, and I have participated in the Edelman Award as a judge and coach. I selected those examples to show the variety of areas where Optimization is applied, and because I believe these examples easily illustrate similar situations where analytics and Optimization could be applied again.   The Edelman Award is given to an implementation of mathematical programming and analytics to an important business problem with a big impact. The Wagner Prize emphasizes mathematics applied to real-world problems, supported by clear and intelligible writing, and verifiable practice success. It was extremely difficult for me to select just a few examples, but the Interfaces Journal devotes its January issue to the Edelman entries and the September issue to the Wagner entries. Most of the links provided below correspond to the journal articles, but you can view additional Edelman videos.   Optimization of the Production Planning and Trade of Lily Flowers at Jan de Wit Company Think of this example as a general production-planning problem. It could refer to other situations such as, producing furniture, textiles, lacteous products, agricultural products, beauty products, etc. These other situations have the same objective, similar constraints and decision variables.   To manage its lily flower business, Jan de Wit Company implemented a decision-support system for the production, planning and trade. In 2001 in its Edelman entry, they reported that between 1999 and 2000, company revenue grew 26 percent, sales increased 14.8 percent for pots of lilies and 29.3 percent for bunches of lilies, costs fell from 87.9 to 84.7 percent of sales, income from operations increased 60 percent, return on owner’s equity went from 15.1 to 22.5 percent, and best quality cut lilies jumped from 11 to 61 percent of the quantities sold. The system also suggested changes in the product mix.   The Linear Programming (LP) goal was to “maximize the farm’s total contribution margin” subject to such constraints as market defined sales limits, market requirements, characteristics of the production cycle duration, technical requirements, bulb inventory, and greenhouse limitations. The main decision variable to be calculated was the number of flower beds in a specific greenhouse, from a specific bulb batch, of a specific variety, for a specific purpose, taking into consideration planting and expected harvesting weeks.   Dell’s Channel Transformation: Leveraging Operations Research to Unleash Potential Across the Value Chain This example illustrates how integrated the four stages of the Garner Analytics Maturity Model are, and the variety of analytical techniques that were used to improve Dell’s processes. Think on how many other industries could benefit from following similar analytical processes as Dell did.   In 2007, Dell launched several strategic initiatives to drive channel transformation by offering fixed configurations through online channels, retailers, distributors, and other channel partners. In 2014 in its Edelman entry, Dell reported that since 2007, Operations Research (OR) had helped Dell solve complex business problems and had facilitated its growth in the FHC business to $15 billion. Since 2010, these OR solutions had delivered a margin impact of more than $140 million by reducing markdown expenditures, improving online conversion rates, increasing ocean shipments, and enhancing customer satisfaction.   In the next two paragraphs notice the two types of mathematical optimization models that Dell used, as wells as, how a variety of analytical techniques produced inputs for these models.   Dell used a variety of analytical and statistical techniques (e.g., A/B testing, multivariate regression, ANOVA, as inputs for a nonlinear mixed integer program) to improve the design of its website and increase the conversion rate of prospective customers to purchasers, leading to increased revenues and customer satisfaction.   Dell’s analytics team created an initial set of fixed hardware configurations (FHCs) based on techniques such as factor and clustering analysis. These initial configurations were upgraded with additional options and features through a quadratic programming model, which considered both costs and maximizing the anticipated revenue coverage.   Operations Research Advances Cancer Therapeutics (2007 Edelman Winner) Using mathematical programming approaches, this winning entry presented sophisticated optimization modeling and computational techniques for real-time (intraoperative) treatment of prostate cancer using brachytherapy (the placement of radioactive “seeds” inside a tumor). The resulting system offered significantly safer and more reliable treatment outcomes.   The goal was to “Minimize the deviations of the dose level at a given tumor point from its target bounds” under the following conditions (or constraints): 1) At each voxel, the radiation dosage should be within specified lower and upper limits, and it can’t deviate from the prescribed dosage by more than a given percentage, 2) Dose-volume coverage constraints for the tumor (for example, that at least 95% of the tumor receives the prescription dose), 3) Total number of radioactive seeds to be used, 4) Total number of needles to be used, and 5) Additional technical constraints.   This Mixed Integer Programming Problem had as main decision variables: 1) The location of the radioactive seeds in the tumor and 2) a variable that indicates if the prescribed radiation was achieved at specific sample points (called voxels) in the tumor.   Obviously, this work can be applied to other types of cancers. View the video.   INDEVAL Develops a New Operating and Settlement System Using Operations Research (2010 Edelman Winner) INDEVAL is the Mexican Central Securities Depository (CSD). A CSD provides the services related to the custody and management of financial securities and their settlement. A settlement occurs when the securities and funds are transferred between the participants of the transaction. In 2010, INDEVAL’s mathematical optimization system, called Dali, reduced the intraday financing costs for market participants by more than USD 150 million annually because of lower daily liquidity requirements of approximately USD 130 billion. Dali made those settlements close to real-time to minimize the difference between the posted price and the actual price in which the financial transactions takes place effectively reducing risk exposure. Lehman Brothers failure can remind us of how important is to reduce this exposure.   Dali’s goal was to “Maximize the number of transactions to settle” while satisfying close to 1,814 security balance constraints (these are related to the number of securities that are in each participant account), and its decision variables were which transactions to settle and their numbers.   This problem is an Integer Problem (IP). However, when cash constraints are removed, its LP formulation is unimodular and it has an optimal solution that is integer. View the video.   There are several similar financial applications, for other countries CSDs and financial brokers. BNY Mellon Optimization Reduces Intraday Credit Risk by $1.4 Trillion describes how the Bank of New York Mellon developed a set of integrated mixed-integer programming models to solve collateral-management challenges involving short-term secured loans. Its objectives were to minimize intraday credit exposure and the liquidity usage of its clients. View the video.   Calibrated Route Finder: Improving the Safety, Environmental Consciousness, and Cost Effectiveness of Truck Routing in Sweden (2016 Wagner Winner) The goal for the 2016 winner was to find routes for heavy logging trucks in Sweden, that is, “minimum cost” routes that are as close as possible to the “best” routes and satisfy the stakeholders constraints.   Routes for heavy logging trucks start at the supply points in the forests and end at the demand points or mills and consider a myriad of constraints due to the many stakeholders in the Swedish forest industry. Some of these considerations are distances, quality of the roads (hilliness, curvature), routes must not pass closed to specially marked places (cities center, hospitals, kindergartens, etc.). Also, routes must include environment considerations.   Traditionally, Swedish heavy logging trucks preferred to use the “best” routes. However, those routes do not have all the considerations that stakeholders asked to be considered, and this project found “minimum cost” routes that included all those considerations. View the video.   Possible Similar Applications: Routing problems for delivery and pickup. For example, see UPS Optimizes Delivery Routes (2016 Edelman Winner). View the video.   Guns or Butter: Decision Support for Determining the Size and Shape of the South African National Defense Force (1996 Edelman Winner) In the absence of a conventional military threat and with a limited budget, what should the size and shape of the South African National Defense Force (SANDF)? The goal for this winner was to maximize the defense value or risk reduction under the conditions for a limited budget and specific army tasks characteristics. Each answer of the mathematical model developed included the tasks, warning periods required, military strategies, force design and cost of the SANDF.   This work was commended highly by South African President Nelson Mandela, and the Edelman judges found it to be excellent.   US Environmental Protection Agency: US Environmental Protection Agency Uses Operations Research to Reduce Contamination Risks in Drinking Water This project competed for the Edelman in 2008. In my opinion, it could be extended to include SAS Event Stream Processing (ESP) that allows you to analyze high-velocity streaming data and take appropriate action instantly.   The US Environmental Protection Agency (EPA) Threat Ensemble Vulnerability Assessment (TEVA-SPOT) Research Program developed contamination warning systems to counter contamination threats against large water-distribution networks throughout the United States that reduce potential public health consequences by thousands of lives and potential economic consequences by billions of dollars.   The goal of the standard TEVA-SPOT optimization model was to minimize the expected impact of the contamination incidents, while considering the overall impact of placing a sensor in a given location and budgets.   This integer program had as decision variables where to locate the sensors in the water-distribution network. View the video.   Non-profit government related projects were the 2017 winners of the Edelman Award and Wagner Prize. The 2017 Wagner winner was The Inmate Assignment and Scheduling Problem and its Application in the Pennsylvania Department of Corrections which solves the complex problem of assigning the inmates to correctional institutions (CIs) and of scheduling their rehabilitation treatment programs while considering about 100 additional factors, such as inmate’s criminal history, demographic characteristics, etc. Inmates who receive timely programming of their rehabilitation treatment programs have a better chance of becoming eligible for parole and leaving the correctional institution earlier, reducing unnecessary overcrowding of the CIs.   This is a complex model, and it considers several goals (or objectives). The multi-objective mixed integer linear optimization problem minimizes the penalties of the weighted sum of all the goals. General factors have the highest priority in assigning inmates to CIs. Minimizing the maximum waiting time for each inmate is second in the hierarchy of objectives. Assigning in the range of the minimum and maximum capacity of each facility has the next highest priority. Additionally, to reduce the population of the CIs, program waiting lists have a high priority in the objective function. Assigning inmates to a facility near their home county is less important compared to the other objectives.   There are several decision variables, some of them are to which facility the inmate is assigned, time when his/her rehabilitation program starts, number of inmates assigned to each CI who start their rehabilitation program at time t, maximum waiting time to start program p, etc. The constraints are related to the 100 factors that affect the inmate assignment, for example the capacity of each CI, the factors that apply to each inmate, the courses that are available at any given time, the distance between the home county of each inmate and the CIs. View the video.   The 2017 Edelman winner was the US Federal Communications Commission for its use of Operations Research and incentive auction to relieve the “spectrum crunch”, with their entry: Unlocking the Beachfront: Using O.R. to Repurpose Wireless Spectrum. For me, the importance of this project is summarized in this paragraph from the article How the FCC won the Edelman Award: “… Asked to sum up the monumental project and its financial impact, Hoffman says, “Forget the $20 billion dollars, forget the $7 billion for deficit reduction. This relates to the future ability of the United States to have the wireless spectrum it needs for 5G and the Internet of Things. This is the wireless economy. It’s the app economy, and if the FCC didn’t do this, we couldn’t move forward.” Watch the presentation.   Conclusion Every day we benefit from the applications of mathematical optimization, from its well-known use in production systems to its role in having wireless spectrum for 5G and the Internet of Things and its use in Artificial Intelligence. I invite you to see many more inspiring applications at the Edelman Award and Wagner Prize web pages. ... View more

In this article, I use SAS Optimization 8.3 to find a tour through the 48 contiguous US state capitals plus Washington, D.C. by solving the well-known Traveling Salesman Problem: Given a set of connected cities, what is the shortest route that visits every city once and returns to the starting place?   The results of the Optimization are easily displayed in a beautiful USA topographical map in Visual Analytics that shows the order each US state capital is visited in the tour.   SAS Optimization 8.3 in Viya 3.4 provides both general optimization and network optimization procedures that take advantage of the distributed, cloud-based computational environment that SAS Viya provides. Examples of those procedures will be the topic of future articles.   The Traveling Salesman Problem In the optimization world, this is a well-known problem, easily stated and difficult to solve: Given a set of connected cities, what is the shortest route that visits every city once and returns to the starting place? There are many applications of this problem, for example: routing vehicles, drilling holes in circuit boards, scheduling tasks, etc.   Imagine you are a salesman who must visit 5 cities, in how many ways you could visit those 5 cities once building a tour? Now imagine that you must visit 10 cities … you can figure out that the number of ways is (n-1)! which means that for 10 cities there are 362,880 permutations, for 15 cities the permutations surpass 87 billion. For this tour example, the correct formula is (n-1)!/2 because the graph is undirected, that is, there is not a definite direction for the tour. For example, from Sacramento, CA, we could travel to either Carson City, NV or to Salem, OR.   Optimization algorithms are used to solve this type of problem.   The mathematical description of the problem is provided in the Appendix.   SAS Viya It is the SAS platform that enables customers to develop, deploy, and manage using a single platform throughout the Analytics Lifecycle. The underlying engine is called CAS; which stands for Cloud Analytic Services.   A CAS Action is the smallest unit of functionality in CAS. You can submit a CAS action though a variety of clients, such as PROC CAS via SAS Studio, or through similar syntax in Python, Java, or Lua. A CAS Action Set is a collection of actions (tasks) that group functionality: for example, session management, table management, Text Mining, Optimization, Network Optimization, etc.   Action Sets and Actions are important because the same Action Sets and Actions are used no matter the client used to make the request. This example is demonstrated using CASL, but you could just as easily use Python or Java.   CAS Procedures are executed from a SAS client, such as SAS Studio 5.1, and provide a wrapper around a CAS action or action set to perform task(s) in the server. I programmed the Network Optimization Actions in SAS Studio 5.1. The code is interesting because it shows how to work with CASL using the new optNetwork action set, in particular the action optNetwork.tsp. The tsp action uses only one core on a single machine.   SAS ® Studio 5.1 is a SAS developer environment that fully integrates with other Viya components (microservices architecture, SAS Drive, SAS Launcher Server/Service and SAS Compute Server/Service, etc.). It is available only with full deployments. For more details, see One, Two, Three SAS Studio! Enter SAS Studio 5.   SAS Optimization 8.3 in Viya 3.4 SAS Optimization 8.3 provides improvements to its procedures, optimization solvers, and CAS actions. Improvements with respect to SAS Optimization 8.2 are significant in performance and memory consumption. Optimization solvers are provided for linear (LP), mixed integer linear (MILP), quadratic (QP), nonlinear (NLP), network, local search (LSO), and constraint logic programming (CLP) problems.   SAS Optimization 8.3 has two Action Sets: Optimization and Network Optimization.   The Network Optimization Action Set “includes a number of graph theory and network optimization algorithms that can augment more generic mathematical optimization approaches. Many practical applications of optimization depend on an underlying network. For example, retailers face the problem of shipping goods from warehouses to stores in a distribution network to satisfy demand at minimum cost. Commuters choose routes in a road network to travel from home to work in the shortest amount of time. Networks also appear explicitly and implicitly in many other application contexts.” See SAS® 9.4 and SAS® Viya® 3.4 Programming Documentation /Network Optimization Programming Guide   In this post, I used the tsp Action, whose formal description is: “the goal is to find a Hamiltonian cycle of minimum total cost, where the total cost is the sum of the costs of the links in the tour.” I use the tsp action to find a tour through the 48 contiguous USA state capitals plus Washington, D.C.   Code   /* Start a cas session */ cas mySession sessopts=(caslib=CASUSER timeout=1800 locale="en_US" metrics=true); caslib _all_ assign; /* Get a list of the state capital cities (with latitude and longitude) */ proc sql; create table Cities as select unique statecode as state, city, lat, long from maps.uscity where capital='Y' and statecode not in ('AK' 'PR' 'HI'); quit; /* Create a list of all the possible pairs of cities */ proc sql; create table CitiesDist as select a.city as from, a.lat as lat1, a.long as long1, b.city as to, b.lat as lat2, b.long as long2, geodist(lat1, long1, lat2, long2, 'DM') as weight from Cities as a, Cities as b where a.city < b.city; quit; data casuser.CitiesDist; set CitiesDist; run; proc cas; loadactionset "optNetwork"; action optNetwork.tsp result=r status=s / indexOffset = 1 links = {name = "CitiesDist"} outNodes = {name= "TSPTour", replace=true}; run; print r.ProblemSummary; run; print r.SolutionSummary; run; action table.fetch / table = "TSPTour" sortBy = "tsp_order"; run; quit; /* Merge latitude and longitude for the cities in the Hamiltonian tour */ proc sql; create table TSPTourLinksAnno_1 as select a.node as from, tsp_order as order, cities.lat as lat_from, -1 * cities.long as long_from from casuser.TSPTour as a left join cities as b on a.node=b.city; proc sql; create table TSPTourLinksAnno_2 as select a.from as to, a.order -1 as new_order, cities.lat as lat_to, -1 * cities.long as long_to from TSPTourLinksAnno_1 as a left join cities as b on a.from=b.city; /* Close the tour, that is, add a row so that the last city in the tour pairs to the first city */ data tempOne; set TSPTourLinksAnno_2; if new_order = 0 then new_order = 49; run; proc sql; create table TSPTourGeoOne as select TSPTourLinksAnno_1.*, tempOne.* from TSPTourLinksAnno_1 as a, tempOne as b where a.order=b.new_order; quit; proc sort data=TSPTourGeoOne out=CASUSER.TSPTourGeo_VA (drop=new_order ); by order; run; /* Promote the table with the Hamiltonian tour to use it in Visual Analytics */ /* These two instructions only need to run once. Once the table is promoted, we don't need to promote it again */ libname ANALYTIC cas caslib="ANALYTICS"; proc casutil outcaslib="ANALYTIC"; promote casdata="TSPTourGeo_VA"; quit;run; proc casutil; list tables incaslib="ANALYTIC"; run;   These are the main steps in the code: Start a cas session, make caslibs visible in SAS Studio 5.1. The option metrics=true will enhance the log to display which actions are executed and other runtime metrics. Create a list of all the possible pairs of capital cities and their respective latitude and longitude. The dataset CitiesDist will have the columns for Origin (and its latitude and longitude) and Destination (and its latitude and Longitude) and the distance from Origin to Destination. This dataset is the input to the next step. The optNetwork Action set will be loaded and the action tsp will use the distance between city-pairs to find the shortest hamiltonian tour for all capital cities. Below is the output of this procedure. It reports that there are 49 nodes (or capital cities) in the graph and 1176 links (or connections) between those cities. The Solution Summary provides information relative to the mathematical problem. Notice that the algorithm found a solution in 0.16 seconds real time and 0.12 seconds of cpu time.       The rest of the code gets the latitude and longitude for each city in the tour and closes the tours, that is, it adds a row so that the last city in the tour pairs to the first city. Promote the session level table to be a global table so that we can use SAS Visual Analytics to map the results.   Visual Analytics in Viya 3.4 These are the steps to plot the tour in a topographical US map: From the “New data items” create two Geography items Origin and Destination as shown for Origin     From Objects select Network Analysis   Assign the Roles: Source = Origin, Target=Destination     In Options, the key features are shown in the photo below     This map is the result     Conclusion This small TSP was solved using SAS Optimization 8.3. The TSP action required a total cpu time of 0.25 seconds in CAS.   The results of the Optimization are easily displayed in a beautiful USA topographical map in Visual Analytics that shows the order each US state capital is visited in the tour.   Given the Traveling Salesman Problem described here, can you see other similar situations in your daily lives that could use this type of optimization?   References Computer Scientists Find New Shortcuts for Infamous Traveling Salesman Problem, Wired 2013 SAS® 9.4 and SAS® Viya® 3.4 Programming Documentation /Network Optimization Programming Guide   Appendix Mathematical Model (SAS/OR® 13.1 User’s Guide)       Thanks to Teri Patsilaras for her guidance on using Geographic Items and Network Analysis objects. ... View more

Imagine you are responsible for monitoring your company’s social media news and for quickly addressing customer complaints. Being able to swiftly identify the main themes in those news and in the customer complaints is key today, and automating that process not only increases productivity but also reduces risk and avoids decreased customer satisfaction.   It is important to determine if previous themes are repeating or if new ones are emerging, and identify which themes are present in each document so they can be resolved by the correct department.   In this article, I will show how 1) to implement several Text Analytics Action Sets in SAS Model Studio to identify the main topics from the customer complaints extracted from the Consumer Financial Protection Bureau (CFPB) in 2015, 2) build a topics scoring model, and 3) apply this scoring model to new customer complaints from January 2018 to Feb 2018.   Visual Text Analytics in Viya 3.3 Visual Text Analytics (VTA) in Viya 3.3 has two interfaces to choose from: Model Studio 8.2 and SAS Studio.   SAS ® Studio is a SAS developer environment that runs in a Web browser, enabling developers to program and interact with SAS. Every SAS product and solution includes SAS Studio automatically. In this interface one programs Text Mining Actions as I will describe in this post.   Model Studio 8.2 on Viya 3.3 is a central, web-based platform that includes a suite of integrated data mining tools. Model Studio can be used to develop models in:   SAS Visual Data Mining and Machine Learning, SAS Visual Forecasting, or SAS Visual Text Analytics. In previous articles, I described the Visual Text Analytics pipeline, and how easy it is to combine text mining, contextual extraction, categorization, sentiment analysis and search.   Note: Click on the images in this post to enlarge them.   SAS Viya and Text Mining Action Sets SAS Viya is the SAS platform that enables customers to develop, deploy, and manage using a single platform throughout the Analytics Lifecycle. The underlying engine is called CAS; which stands for Cloud Analytic Services.   A CAS Action is the smallest unit of functionality in CAS. When you submit a CAS action, which can be done using proc CAS in SAS, or through python, Java, or Lua; it sends a request to the server, parsing the arguments of the request, invoking the action function, returning the results and cleaning up resources.     A CAS Action Set is a collection of actions (tasks) that group functionality: for example, session management, table management, Text Mining, etc.   CAS Procedures are executed from a SAS client, such as SAS Studio, and provide a wrapper around a CAS action or action set to perform task(s) in the server.   Action Sets and Actions are important because the same Action Sets and Actions are used no matter the client used to make the request. In this article the examples are worked in CASL; but you could just as easily use Python or Java.     This post shows how to utilize actions implemented in the Text Mining Action Set, which is used:   to discover the main themes and concepts in the document collection to build a predictive model which uses as input text-variables in the document collection to compare documents or terms for similarity. These are the Actions in the Text Mining Action Set:   tmMine   Derives topics (main themes) from a collection of documents.   It uses other actions from the text mining action set (tpParse, tpAccumulate, tmSvd)   tmScore   Uses the models built using the tmMine action to score new data   tmSvd   Applies a matrix factorization to the output parent table of the accumulation action. It uses the occurrence data from the entire collection to produce a best-fit, low-dimensional representation that can be used to represent documents and terms as a vector of numbers. This representation can be rotated into topics; which provide a more descriptive set of axes for the coordinate representation.   In this article, the Text Mining Action Set is used to   Discover the main themes in the CFPB complaints dataset with dates between March 2015 and June 2015 which contains 2,413 complaints. Score a new data set from the CFPB with complaints collected from January to the 1 st week in February 2018 which contains 2,580 complaints. The code for this implementation can be seen below in the Code window. The main parts of that code are:   Start a cas session, make caslibs visible in SAS Studio and load data in to CASUSER library The customer complaints from 2015 (the training data) The stop list and The new customer complaints from 2018 (data to be scored) Using the option metrics=true will print in the log the actions executed Build a Topics model using the tmMine action. Required parameters are: docId: Text Analytics requires a field with unique values that identifies each document of the collection text:  the field containing the text to be analyzed documents: the input CAS table that contains the input documents to analyze and the stop list loaded in step one Use tmScore to score the new customer complaints from 2018 using the model developed in step two by tmMine Use tmSvd to produce the TopicsSVD table with the topics relevant to the new data using the tables parent, terms and U, which contain the already parsed collection from the scored file.   Output The code produces several output tables which identify the topics from the 2015 documents, and how those topics are present in the new document collection. Also, the Singular Value Decomposition (SVD) produces three main matrices: a term-by-topic matrix, a matrix of topic importance values, and a document-by-topics matrix.   Important: remember that in the text mining action tmMine the parameter K was set to K=3, so we expect to see up to three terms.   In the photo below, we can see which topics were identified in the 2015 customer complaints. Zero or more of these topics will be assigned to each of the 2018 customer complaints.     The table SVDU is term-by-topic matrix, in which each row corresponds to a term, _TermNum_ is the term’s number, and the elements of this matrix can be interpreted as relevance weights—they describe the relationship of each term to each topic—and these relationships help you interpret the derived themes. Each theme is a linear combination of terms, so it is customary to label the topics by using the terms that have the highest weights     The TERMS table is shown below (you can relate each term with its specific _Index_, or _termnum_, if you wish     The DOCPRO table shows the relationship of each document to each topic. It is the document to topic matrix, where the col1 to col3 variables indicate the linear combination of Topics to the document, and which topics are most relevant to each document. For example, the first row indicates that for the complaint with ID 1290183, the first topic (loan, +modification, +mortgage) is the most relevant to it. The 2 nd row indicates that complaint with ID 1290253 can’t be expressed as a linear combination of the 3 themes.     The results of applying the topic scoring model to the new customer complaints is shown in the table scoreDocpro, where we can see the linear combination of Topics to the document and which topics are most relevant to each complaint.     Conclusion One can quickly (and easily) identify the main themes in a document collection using Text Analytics Action Sets in SAS Model Studio.   References SAS Visual Analytics 8.2: Programming Guide   Ray Wright, Temporal Text Mining: A Thematic Exploration of Don Quixote, SAS Global Forum Paper SAS0523-2017   Albright, R. 2004. “Taming Text with the SVD.” Cary, NC: SAS Institute Inc.   Appendix -- Code   /* Score new data set to find its Topics */ /*****************************************************************************/ /* Start a cas session named mysess using the existing CAS server connection */ /* while allowing override of caslib, timeout (in seconds), and locale */ /* defaults. */ /*****************************************************************************/ cas mysess sessopts=(caslib=casuser timeout=1800 locale="en_US" metrics=true); /*****************************************************************************/ /* Create SAS librefs for existing caslibs */ /* so that they are visible in the SAS Studio Libraries tree. */ /*****************************************************************************/ caslib _all_ assign; /* Training data from 2015 is created using the DATA step */ data casuser.reviews; set analytic.codingcomplaints; run; /* Public stop list from SAS HELP is used */ proc casutil; load casdata="engstop.csv" incaslib="ANALYTIC" outcaslib="ANALYTIC" casout="engstop"; data casuser.engstop; set analytic.engstop; run; /* load into cas the data to be scored from 01/01/2018 to 02/07/2018 */ proc casutil; load casdata="scoreComplaints.csv" incaslib="ANALYTIC" outcaslib="ANALYTIC" casout="scoreComplaints"; run; data casuser.scoreComplaints; set analytic.scoreComplaints; run; /* The topics are discovered and document projections */ /* made using the tmMine action */ proc cas; loadactionset "textMining"; action tmMine; param docId="Complaint_ID" documents={ name="reviews"} text="Consumer_complaint_narrative" nounGroups=False tagging = False stopList ={ name="engstop"} parseConfig={name="config", replace=TRUE} parent ={ name="parent",replace=TRUE} offset ={name="offset",replace=TRUE} terms ={ name="terms", replace=TRUE} reduce=1 k=3 docPro ={ name="docpro", replace=TRUE} topics ={ name="topics", replace=TRUE} u ={ name="svdu", replace=TRUE} numLabels=3 topicDecision=True ; action table.fetch /table="topics", orderBy="_TopicID_"; run; action table.fetch /table="docpro", orderBy="Complaint_ID"; run; action table.fetch /table="svdu", orderBy="_TermNum_"; run; run; quit; /* scoring Document made using tmScore based on training data */ proc cas; loadactionset "textMining"; action tmScore; param docId="Complaint ID" documents={name="scoreComplaints"} text="Consumer complaint narrative" terms={name="terms"} parseConfig={name="config"} u={name="svdu"} docPro ={ name="scoreDocpro", replace=TRUE} topics={name="topics"} topicDecision=True ; action table.fetch /table="scoreDocpro"; run; run; quit; /* If you did not calculate the SVD initially, you can do it using the parent table as input */ proc cas; loadactionset "textMining"; action tmSvd; param parent={ name="parent"} terms={name="terms"} k=3 u ={ name="svdu", replace=TRUE} numLabels=3 topics={name="topicsSVD",replace=TRUE} ; action table.fetch /table="topicsSVD"; run; run; quit; /*****************************************************************************/ /* Up to this step the tables are in CAUSERH, */ /* if I want to move them to the analytic library */ /* they most be promoted */ /*****************************************************************************/ proc casutil outcaslib="ANALYTIC"; promote casdata="docpro"; promote casdata="svdu"; promote casdata="scoreDocpro"; promote casdata="topicsSVD"; quit; /*cas mysess terminate; */   ... View more

I used SAS Visual Text Analytics (VTA) to discover the main topics that people tweeted on MLK Day by briefly analyzing tweets on #MLKDayofService. In this blog I will show the steps I took to do that discovery, and other valuable features available in VTA.   Martin Luther King Jr. Day is a US national holiday that honors his legacy in battling for civil rights. Many of us have been moved by his I Have a Dream and his Nobel Peace Prize Acceptance speeches. Many Americans celebrate MLK Day by volunteering, as an answer to one of his famous quotes “Life’s most persistent and urgent question is: What are you doing for others?”   SAS Visual Text Analytics (VTA) is the SAS offering designed to effectively extract insights from unstructured data in large scale. Offered on the SAS Viya architecture, VTA combines the power of Natural Language Processing (NLP), Machine Learning (ML) and Linguistic Rules. Currently, VTA supports 30 languages and it has an open architecture supporting 3rd-party programming interfaces.   Note: Click on the images in this blog to enlarge them in a new tab.   With VTA, users can perform the full analytics lifecycle with large volumes of unstructured data: from accessing and preparing data, to building text models, to analyzing results, to successfully deploying the models against new unstructured data. Since VTA is fully integrated with Visual Analytics (VA) one can visually analyze text data at the beginning and end of the analytics lifecycle, as is shown in this blog. One thing I found fascinating about VTA, is that VTA automatically detects Topics and generates Categories. VTA has integrated capabilities that allow you to do contextual extraction, categorization and sentiment analysis to quickly produce insights. These insights can kick off immediate actions, or you can fine-tune them with subject-matter expertise.     The VTA Pipeline In VTA, a pipeline is a process flow diagram whose nodes represent tasks in the Text Analysis Process. In this blog, I illustrate seven main steps that one can utilize when analyzing twitter data. The first four and final steps are done in VA, while steps five and six are done in VTA. Together they illustrate the nice integration between VA and the VTA pipeline.   Step One: Bring Twitter Data into Visual Analytics Twitter’s public API is used to import data into CAS. Use the Prepare Data application and the menu Import to bring twitter data into VA.   You need to specify the search term(s) to import, the maximum tweets to import and whether or not to import retweets. The maximum number of tweets default value is 2000. There are limitations on what data and on how much data that SAS can download using the Twitter public search API. To change users or remove authorization for the account, select Clear Authorization.           Step Two: Create an ID Column that has unique values Twitter data already has such a column docid. In VA, you can Run Profile and check this fact.     Step Three: Set a Unique Row Identifier for your data Convert docid from a Measure to a Category variable, and indicate docid is the unique row identifier          Step Four: Add Text Topics Object and assign Roles   As shown in the photo below       1. On the left side bar, click Objects 2. Select Text Topics and drop it into the report page 3. Select English 4. On the right side bar, click Roles 5. To the Document collection, add comments. Select English as the Data Source language and click OK 6. On the right side bar, click Options  and select Analyze document sentiment Note One: VTA will auto-generate categories if you add additional categorical variables to Document Details, but variables added must have less than 400 values. My twitter data didn’t have any variable that I could utilize at this step.     Note Two: If your data has emojis it is advisable to preprocess your data to remove them. Version 18W25 will not require this pre-processing.         This photo shows all the out-of-the-box results obtained in the preliminary data analysis done in VA. Save your data.     Step Five: Create a Visual Text Analytics project From the SAS Home menu select the action Build Models that will take you to SAS Model Studio, where you select New Project, and enter the following information.         Once you click Save, you will see two main tabs: Data and Pipelines, and a message indicating “You must assign a variable to the Text role. Assign variable roles”, click in that message and select “body” as your text variable          Step Six: Create a Visual Text Analytics Pipeline and customize nodes   By selecting the Pipeline tab, you see the default VTA pipeline. Working thru these nodes one does contextual extraction, sentiment analysis and categorization. Machine learning is used to generate the rules in the Topics node and also in the Categories node for the automatic rule creation for categories. With the push towards Artificial Intelligence, SAS will be applying more machine learning techniques, especially deep learning, in future VTA releases. One can add customization in the Concepts, Text Parsing and Categories nodes. Sequentially work with each node, right click on it and select Open                               Right click in the Concepts node, and select Open. Concepts are data elements or patterns – such as named entities or fact relationships – that you wish to extract from the larger text field because they match some specific context. VTA provides 9 predefined concepts such as dates, people, places, measurements, mentions of currency which are concepts whose rules are already written to save development time        Below are some examples of the comments matched with nlpDate         And nlpPerson         In VTA, you can write rules for recognizing concepts that are important to you, thereby creating custom concepts. Close the Concept Node and right click the Text Parsing Node. In the Text Parsing Node, unstructured text is parsed and transformed into the structured form of a vector by using NLP and other tools. It is useful to “drop” some terms that appear in the Kept Terms list because they are not useful for analysis. For example, I dropped http://t.co/, https://t.c, https, http, ‘s, 1942-2016, foxnews, ijessewilliams.   After selecting terms to drop, run the node to see the resulting Kept Terms list         These are the terms that are related to mlkdayofservice         This is the TermMap for the term volunteer, it shows other terms that are most commonly found in other tweets with theterm volunteer across the entire tweet collection         Close the Text Parsing Node and right click the Sentiment Analysis Node.   The Sentiment Analysis Node categorizes the sentiment and opinion about documents, a brand(s), person(s), organization(s), etc. VTA uses a set of proprietary rules that identify and analyze terms, phrases, and character strings that imply sentiment. You could specify a custom Sentiment model if you would like to use it instead of the default model.   Close the Sentiment Analysis Node and right click the Topics Node.   In the Topics Node one can see the automatically generated topics, merge topics and promote them as categories if desired.   This photo shows the auto-generated topics. Notice that up to this point, there has not been any customization just “dropping” terms that are not meaningful. Notice the richness of the auto-generated topics. The first two topics are selected and can be merged and then promoted to be used as Category. Merge the topics indicated, promote them and run the node.           Close the Topics Node and right click the Categories Node.   In the Categories Node, VTA assigns each tweet as belonging to one or more categories using linguistic rules, rather than statistical weighting of terms as with Text Topics. As discussed in Note One above, in this twitter project there will not be auto-generated categories. The photo below shows two categories promoted from the Topics Node, and the rules related from the PersistentQuestions category.         It is possible to create new Categories using LITI and Boolean rules. How to create them will be explored in a future VTA blog. In this blog, I want to show how to export the results of your categorization model for exploration. Run the node. Right click on Categories node and select “Save data table”, and save the data.     Step Seven: Visualize VTA Pipeline results in VA From SAS Home open Explore and Visualize Data, open the data saved in the previous step in the Categories node. Create a New data item of type Hierarchy, and add your categories to it and move them into the canvas        For this data, it was useful to add a List Table to the report above in order to see the text reviews that correspond to each category. To add the List Table follow these steps: Click on the List Table, drop it below the _category_ bar chart. Go to Roles at the right side bar and add the “body” column to the table. Go to the Actions menu on the right, select ‘New Action’ and add filter, leave the default options checked and select OK. The photo below shows the “body” for the missing _category_ and by analyzing it, one can select terms that one could use to improve the existing ones, or to develop new customized categories by using LITI and Boolean rules in the Categories Node.        Conclusion Using SAS Visual Text Analytics (VTA) one can discover the main topics on a collection of tweets. In this blog I briefly analyzed tweets on #MLKDayofService. VTA combines the power of Natural Language Processing (NLP), Machine Learning (ML) and Linguistic Rules and it is fully integrated with Visual Analytics. VTA has integrated capabilities that allow you to do contextual extraction, categorization and sentiment analysis to quickly produce insights.   References SAS ® Visual Text Analytics 8.2: User’s Guide      ... View more

UPDATED: Q&A session now available   Hi Customer Intelligence Community,   I’m presenting a live “Ask the Expert” webinar on August 1st, 1 - 2 p.m. EST on the benefits of optimizing campaigns and how to use SAS Marketing Optimization and SAS Marketing Automation together to help you get better results. I hope you’ll join me.   These are the topics I plan to cover:   What marketing optimization is and why it matters. Business and technical benefits that MO brings How objectives, constraints, and contact policies come into play as you optimize. How to identify campaigns worth keeping. Where should you put your money? What is MOMA and how it can take your campaigns even further How to differentiate and select optimization algorithms when using MOMA Additional resources in SAS Communities   Are there any other specific questions you’d like covered? Let me know by responding to this thread.   Be sure to subscribe to the Ask the Expert Community Library to receive follow up Q/A, slides and other related resources from the webinar. From Ask the Expert Library, just click Subscribe from the orange bar underneath the list of the recent articles. Can't join the live event? You can view this and other Ask the Experts sessions on-demand here.        ... View more