Programmatic Manipulation of Concept Rules in SAS® Visual Text Analytics (VTA)   Note: The following article references the use of APIs that are not officially documented by SAS for external use and are not supported by SAS Technical support.   While Model Studio offers a strong interface for developing VTA projects, we identified a need for greater flexibility to support rapid development and collaboration among multiple analysts. In this article, we’ll describe how we built custom processes to enhance the efficiency and transparency of working with taxonomies created within Concept nodes.   The enhanced functionality provides the following benefits: Process to export concept rules from any taxonomy Programmatic merging and editing of concept rules Automated population of an existing Concept node with original or modified rules Exporting Concept Rules Project Details Project Name Project Data Node Details Pipeline Name Node Name Taxonomy UUID Concept Details Concept Name Concept Hierarchical Path Concept Details (continued) Rule Definition (including comments) Rule Priority Case Sensitive Flag Primary / Supporting Flag Concept Metadata Created By Created Timestamp Modified By Modified Timestamp       Figure 1: Exported table for a single VTA project, Color_Project displaying all columns of data. Working with the Exported Concept Rules Exporting concept rules into tabular data opens up numerous opportunities to enhance the development cycle. One example of this increased flexibility and collaboration is the ability to combine multiple taxonomies into a single rule set. Users can also create and maintain a centralized list of commonly used concepts, which can be reused across multiple taxonomies. Additionally, rule sets created by multiple analysts can be merged into a single, unified taxonomy.   The export process also generates a line-by-line view of the rule definitions. This format, combined with the accompanying metadata, makes it easy to identify and track changes to the rule set over time. For example, a single custom concept may include hundreds of individual concept definition rules and associated comments. Comments can be used to describe the purpose of a rule or set of rules within a custom concept. Comparing the output line by line allows users to quickly identify which rules were added, deleted, or modified.   Figure 2: Subset of the exported table in Figure 1 showing multiple observations for a single concept. Using Exported Concept Rules to Create a Structured Directory of Concept Definition Text Files Another potential use for the exported tabular concept definition data is the further processing of this information to create a series of text files containing the concept definition rules (one text file per concept). These text files could then be placed in a version control repository such as Git or SVN to allow the developer to easily archive and/or view changes to the concept’s rules over time, as well as share the concept rule files with other developers.   The code provided in this article, concept_rules_2text_files.sas, contains an example process which automatically builds out the structured directory of text files using the exported concept dataset as the sole input. Once executed the user will have access to a text (.txt) file containing the concept definition rules for each concept, as well as an associated info (.info) file containing metadata about the associated concept (such as last modified date, whether the concept is case sensitive, and other related information). Importing Concept Rules Importing concept rules back into Model Studio requires a blank Concept node within any project, along with the exported concept rule table. The import process loops through all exported rules and adds the definitions to the specified Model Studio project. Once the process is complete, the rule definitions appear within the Concept node. Use import_concept_rules.sas to execute this import process.   Note: For Concept nodes that use predefined concepts, the user must manually enable this option in the Model Studio property panel.   Figure 3: Concepts for the VTA project named Color_Project, as shown in SAS Model Studio user interface, without the Predefined Concepts enabled. Conclusion: Part 2 We hope these tools enhance your experience with SAS Visual Text Analytics and support more agile model development. You can access the programs in our public repository: sas-vta-examples.   Please use these suggestions as you see fit and continue to share any tips and tricks you’ve found useful in your VTA practice in the comments section. In the meantime, stay tuned for our final article of this series, Part 3, focused on tracking your changes to a VTA project.   ... View more

Importing Text Data into SAS Viya If you have worked with SAS and unstructured text before, you are probably familiar with the number 32,767 or the max length of a character field in the SAS 9 engine. This limit often comes into play when dealing with importing text data from various sources such as pdf, word and html files. SAS Viya introduces a new data type to SAS users, the VARCHAR format. The VARCHAR format allows a maximum of 536,870,911 characters (UTF-8 encoding). The VARCHAR data type is important because it allows you to save space when working with fields of varying length. A 32,767 fixed length CHAR field that is populated with the word “N/A” stores 32,764 trailing blanks in the field, a large waste of memory. The VARCHAR format uses only the space that is needed (3 bytes for “N/A”) plus a little overhead (16 bytes). If trying to decide between CHAR and VARCHAR use this guide: CHAR – use a fixed-width CHAR when the sizes of the column data entries are similar. Fixed-width columns are usually accessed faster. VARCHAR(n) – use when the sizes of the column data vary considerably but you are reasonably certain they will not exceed a certain width. VARCHAR(*) – use when the sizes of the column data vary considerably and the column width might exceed any limits you might place on it. In this blog, I will cover three primary objectives: Review the Character Variable Padding (CVP) libname engine, for converting sas7bdat datasets with CHAR variables to VARCHAR variables for SAS Viya Demonstrate code importing a documentation collection programmatically Provide the steps to import a document collection through the UI   Converting CHAR Variables to VARCHAR The CVP libname engine allows you to convert CHAR variables to VARCHAR before loading the data into a CASLIB. The CVPVARCHAR=YES option specified in a CVP libname statement will convert all CHAR variables in the data to VARCHAR data type.   The results of the proc contents show variables with the VARCHAR data type. The example dataset contains 31 variables and 848k observations, with the longest character field being 2048 characters in length. The data loaded into CAS with the VARCHAR data type results in a 0.79 GB table compared to an 8.97 GB without VARCHAR. Importing Documents Programmatically The code below shows an example of importing a document collection programmatically. The first step is to start my CAS sessions and define a caslib that points to document location. Since the documents are split into multiple folders (/PDF and /HTML) it is important to have the “subdirs” option on the caslib. This allows to include any files in subdirectories in the import process. The example imports 7 files into a CAS table with columns for the file path, filename, file type and the content of the imported file. For this example, we import four html pages from Frank Silva’s SAS blog and 3 pdf files with information on different SAS products. Importing Documents with the UI The first step to importing a collection of documents begins with selecting “Documents Directory” as the data import source and choosing the caslib. This will import all potential text files located in the path the caslib is defined.     The document directory import has options for searching recursively in the directories for documents, as well as specifying specific file types of interest. When ready, the files can be imported or a SAS Job created (via the SAS Job icon located next to the Import Item button). Creating a SAS Job, allows for the import process to be scheduled and run regular with the SAS Job Execution’s built in scheduler.  Frequently Asked Questions ·         What type of file format can be converted and imported using the Document Directory import? o   SAS can import the following file types:   ·         What about importing JSON files, since it isn’t included in the list above? o   JSON files can be imported using the JSON libname.   ·         What product do I need to be able to import a document directory? o   SAS Visual Analytics on Viya is all that is needed for importing a document directory.   ·         Once I have the text data imported, how do I analyze it? o   SAS Visual Analytics on Viya provides the ability to explore text topics and sentiment of unstructured text. o   For deeper analysis, additional text analytics capabilities can be found in SAS Visual Text Analytics and SAS Visual Data Mining and Machine Learning ... View more