If you are using or planning on using Model Studio 8.5 and plan to have partitioned data to leverage the method of using training, validation and test data to build your model then read on.     In this article I have 2 topics, namely expanding on the 'automatic data duplication' and how to use some simple code to mitigate it from happening in the first place.    Before I dive in, my thanks to SAS R&D colleagues Fred Burke and Mark Thomas for their help in getting clarity on the points and running some confirmation tests.   In my last article, I briefly mentioned a recent update to the Visual Data Mining and Machine Learning Advanced Topics documentation.  The text reads: Disabling Automatic Data Duplication Model Studio copies the data source when the first Data node is run. This can cause performance issues and can cause you to run out of disk space. The amount of space that is required depends on the number of saved projects and on the size of the data source. To prevent Model Studio from automatically creating copies of your data, ensure that the following conditions are met: A Key variable exists in your data. This can be either a variable named _INDEX_ or a variable that is assigned the role Key. A Partition variable exists in your data. This can be either a variable named _PARTIND_ or a variable that is assigned the role Partition. The data table must be loaded directly from the CASLIB source. You cannot use PROC CASUTIL to load a table into a CASLIB that has a different source path than the table. Some points of clarification.   The sentence “Model Studio copies the data source when the first Data node is run.“, it’s probably more precise to say, “Model Studio copies the data source only on the first run/execution of the Data node in the pipeline“.   Key variables are required within VDMML based projects. If the key variable has a name _DMINDEX_ and this variable is numeric with distinct values, then Viya/CAS will recognize this as being a key variable. If you provide another name for the variable and/or it is not numeric, then you will need to use the Model Studio interface to define the variable as a key variable.   Partitioning of data is optional. It is up to the Model Studio user to deciding whether to select partitioning as part of the Model Studio project properties. The default behaviour of Model Studio is to partition the data. If the user deselects the partition option, then bullet point 2 becomes irrelevant.   To be clear, the data is copied to the CAS Server Controller host machine. The data is only duplicated if the Model Studio user requests the data to be partitioned within the Model Studio interface.   With this in mind I thought it would be worth sharing a visual description of if/when/how the steps should be completed to prevent data duplication.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The overarching recommendation if data duplication is to be avoided, is to ensure the variables be applied prior to the data being used within Model Studio. Readying the data prior to analytical routines is pretty common place and so having a repeatable approach is recommended. Interfaces like Data Studio can be used to generate key and partition variables.   One way tackle this and to implement the 3 bullet points described in the documentation is provided in the example code below. The final table which gets created heart_dmindex_partind is the one which would be used as the input table within Model Studio.   The key points to remember are: Use a sampling scheme option within Proc Partition to create a partition that is appropriate for your analysis. There are some very helpful examples. If you need to create the in 'key' variable, note there are different ways to do it. You can keep it simple by using Stephen Foerster's method. There is also a method described within the SAS 9.4 and SAS Viya 3.5 Programming documentation. Remember to save the updated table back into the CASLIB before then un-loading and then re-loading into CAS My thanks to Fred for writing the code below. Let us know what you think, by adding your comments below.     /* * In this example though we are using the sashelp.HEART data set and create both * the index and partition variables for it. We are using this SAS Data Set * because it is easily accessible by all users. * * However, in your use case your first step would likely be to use the 'Data' * tool in the Administration application 'Manage Environment' to Import your * table (i.e., from your Local Filesystem) into your CASUSER CASLIB and then * add the index and partition variables to that CSA table. * * This example code is expected to be run in SASStudio. * */ OPTIONS MPRINT NOSYMBOLGEN NOSOURCE; options validvarname=ANY validmemname=extend VARLENCHK=NOWARN CASDATALIMIT=ALL; CAS mycas sessopts=(metrics=true messagelevel=all); CASLIB _ALL_ ASSIGN SESSREF=mycas; /* Create the casioca libref so that SAS can use it to access the table in CAS. */ libname mylibref cas caslib="CASUSER"; /* Just to ensure the tables do not exist. */ proc datasets lib=mylibref nolist nowarn; delete heart_dmindex_partind; run; quit; /* Ensure the tables are not already loaded in CAS; show the tableinfo. */ proc cas; table.dropTable caslib="CASUSER" name="heart_dmindex_partind" quiet=TRUE; run; quit; /* * Load the data into CAS so that we can access it. * */ data mylibref.heart_dmindex_partind; set sashelp.heart; run; /* * Create the _dmIndex_ variable. * Model Studio will recognize this variable and automatically assign it a role * of KEY. */ data mylibref.heart_dmindex_partind; length _dmIndex_ 8; _dmIndex_ = _threadid_ * 1E6 + _N_; set mylibref.heart_dmindex_partind; run; /* * Create the partition variable '_PartInd_'. The call below specifies that * the partition variable _PartInd_ be created and that the percentage of * train, validation and test observations wil be 70%, 10%, and 20% respectively. * * For more information see: * https://go.documentation.sas.com/?docsetId=casstat&docsetTarget=casstat_partition_examples.htm&docsetVersion=8.5 */ proc partition data=mylibref.heart_dmindex_partind samppct=10 samppct2=20 seed=10 partind nthreads=3; by STATUS; output out=mylibref.heart_dmindex_partind; run; /* * The above code will generate a Log view, a Results view and an Output Data * view. Select the Ouput Data view and verify that the variables _dmIndex_ and * _PartInd_ have been successfully added to mylibref.heart_dmindex_partind. */ /* * Now write the updated table to the filesystem of the CASUSER CASLIB and * promote it from session scope to global scope so other CAS sesions you * create can see it. */ proc cas; session mycas; table.save / caslib="CASUSER" table="heart_dmindex_partind" name="heart_dmindex_partind.sashdat" replace=TRUE; run; table.fileInfo / caslib="CASUSER" path="heart_dmindex_partind.sashdat"; run; table.promote caslib="CASUSER" name="heart_dmindex_partind"; run; quit; /* * Check the table heart_dmindex_partind is availalbe in CASUSER and * is available in global scope. */ proc casutil ; list tables incaslib="CASUSER" ; quit; CAS _ALL_ clear;   Thanks for reading.   --Simon   ... View more

  For those of you who have read the SAS Viya 3.5 for Linux: Deployment Guide and deployed Viya, you will have come across CASDATADIR references. Interestingly it does not get mentioned much elsewhere, which may seem odd given it's purpose. If you are a technical architect or are working with customers to ensure performance factors beyond the CAS Disk Cache, networking and optimum data loading techniques then read on.   Let's start with some content from SAS Viya 3.5 for Linux: Deployment Guide:   By default, product caslibs are written to /opt/sas/viya/config/data/cas/default, which is often hosted on a single hard disk drive with limited storage. To ensure proper performance of your SAS solutions, SAS recommends that the CASDATADIR option be configured to point to a high-performance storage platform. Examples of high-performance storage platforms include SAN, NVMe, and multiple drive disk arrays. ... Changing the CAS data directory is especially useful for solutions that can be resource-intensive, such as SAS Visual Forecasting, SAS Visual Data Mining and Machine Learning, SAS Visual Text Analytics, and SAS Analytics for IoT. Multiple predefined system caslibs and the Public caslib have a default location for persistent storage: /opt/sas/viya/config/data/cas/instance-name/name-of-Public-caslib. You can specify the instance name when you edit the vars.yml. If you anticipate that many users will use browsers to access the user interfaces and to import data from files, additional space for this file system will be required. SAS recommends monitoring disk usage at /opt/sas/viya/config/data/cas (assuming this the value of CASDATADIR).   Acknowledgements. Before I go any further I wanted to call out this post reflects conversations and information my colleagues here at SAS have shared. My appreciation to them for their input.   Useful data points CASDATADIR - also termed CAS Data Directory By default the directory for CASDATADIR is /opt/sas/viya/config/data/cas/default The CASDATADIR is defined in the CAS_CONFIGURATION block of the vars.yml file When installing and configuring an MPP CAS Server, the CASDATADIR is configured on the CAS Controllers (Primary and Secondary) as well as the CAS Workers In practice/the default action is for applications like SAS Visual Forecasting, SAS Visual Data Mining and Machine Learning, SAS Visual Text Analytics to write results tables to the sub-directories within CASDATADIR only on the CAS Controller i.e. files small or big are written as a block (they are not distributed blocks of data across CAS Workers)   Architecture considerations based on the data points In some customer environments where they have heavy analytical usage (users + volume of data), the CAS Controller could get really busy Ensuring the sufficient bandwidth/throughput between the CAS Workers & CAS Controller becomes really important Writing out data to CASDATADIR (and reading it from) as quickly possible, should be a goal to minimise the impact on CAS Controllers's resources i.e. i/o for the CAS Controller needs to meet the usage patterns Knowing what and when data is getting written to CASDATADIR can help inform customer teams on the performance requirements within architecture design Knowing of methods to prevent unnecessary writing to CASDATADIR location might prove beneficial e.g. have large output tables produced through modelling, to be written to HDFS or DNFS For some customers, backing up the CASDATADIR structure is likely to be a useful recommendation   What's in CASDATADIR? A quick directory listing of the Viya 3.5 environment we use for our teams testing here at SAS shows this, with the root directory being /opt/sas/viya/config/data/cas/default. FYI. 'default' refers to the name of the CAS Server. If an environment has multiple CAS Servers, then something other than 'default' is likely to listed as that directory. Some additional information regarding the predefined Caslibs can be found in the SAS Viya 3.5 Administration: Data document. Sub directories for individual users can be found under 'casuserlibraries' and sub-directories for individuals projects can be found under the 'projects' directory.   drwxr-xr-x.     2      cas sas      4096   Dec  4    17:48    appData drwxr-xr-x.     55     cas sas      4096   Dec  4    17:48    casuserlibraries drwxrwxrwx.     2      cas sas        64   Feb 10    02:30    formats drwxrwxr-x.     2      cas sas        6   Jun 26    2018    modelMonitorLibrary drwxrwxrwx.    2      cas sas       4096  Dec  2    16:34    models drwxr-xr-x.     2      cas sas          6   Jun 26    2018     modelStore drwxr-x---.     16     cas sas      4096    Nov 26  14:13    projects drwxrwxrwx.    2      cas sas      4096    Dec  4   17:48    public drwxr-xr-x.     2      cas sas          6    May 30   2019     qasMartStore drwxr-xr-x.     2      cas sas      4096    Dec  4   17:48    referenceData drwxr-xr-x.     2      cas sas      4096   Jun 26   2018     samples drwxr-xr-x.     2      cas sas        103   Feb  7  15:54    search drwxr-xr-x.     2      cas sas          6   Jun 26   2018     sysData drwxrwxrwx.    2      cas sas          6   Jun 26   2018     vamodels   CASUSER libraries - to write or not to write to CASDATADIR Users of visual applications can choose to write to the CASUSER directory from the visual applications. The decision on whether the CASUSER libraries are written to the CASDATADIR directory structure is based on: the type and interface being used the membership of users to the CASHostRequired custom group. The table below from the SAS® Viya® 3.5 Administration: Identity Management document explains this clearly. See table 1 below.   User Scenario CASUSER Path Location Session Information User starts CAS sessions from visual interfaces (includes all SAS Viya interfaces except SAS Studio 4 and Base SAS or SPRE sessions), and user is not a member of the CASHostAccountRequired custom group. This is the default behavior. /opt/sas/viya/config/data/cas/default/ casuserlibraries/username Sessions run under the CAS server user (cas). The directory and all files within it are owned by the cas user. User starts CAS sessions from visual interfaces (includes all SAS Viya interfaces except SAS Studio 4 and Base SAS or SPRE sessions), and user is a member of the CASHostAccountRequired custom group. $HOME/casuser Sessions run under the user’s host account. User starts CAS sessions from SAS Studio 4, Base SAS, or SPRE, regardless of whether the user is a member of the CASHostAccountRequired custom group. $HOME/casuser SAS Studio 4, Base SAS, and SPRE sessions always run under the user’s host account, and use the $HOME/casuser CASUSER path location. Sessions run under the CAS server user (cas). The directory and all files within it are owned by the cas user.   One item that may be worth mentioning here is when the user is a member of the CASHostAccountRequiredGroup and uses the visual interfaces to store data in the CASUSER caslib. The $HOME directory may result in out-of-space issues as some customer IT teams may restrict the size of the $HOME directory per user.   Large input tables & CASDATADIR For users of VDMML (Model Studio interface) they may be familiar with the fact that on the first run of the Data Node, the source data is copied to the CASDATADIR directory structure e.g. /opt/sas/viya/config/data/cas/default/projects/datamining-0abc3abf-9ad2-477a-89b5-989f1e4cfe9a. The good news is that very recently a method to prevent that happening was documented in the latest version of the Model Studio 8.5: SAS® Visual Data Mining and Machine Learning 8.5: Advanced Topics document. Here is the text:   Model Studio copies the data source when the first Data node is run. This can cause performance issues and can cause you to run out of disk space. The amount of space that is required depends on the number of saved projects and on the size of the data source. To prevent Model Studio from automatically creating copies of your data, ensure that the following conditions are met: A Key variable exists in your data. This can be either a variable named _INDEX_ or a variable that is assigned the role Key. A Partition variable exists in your data. This can be either a variable named _PARTIND_ or a variable that is assigned the role Partition. The data must be persistent on the disk. For 3., there here is some additional clarification. The table must be loaded from the caslib source directly. You can not use proc casutil to load a table into a caslib which has different source path than the table.     Large output tables & CASDATADIR Depending on the type of analytics being done, it may require that the original data be written in full to an output table. The output table will contain additional columns being appended e.g. the columns will contain predicted or forecasted values and the delta between predicted/forecasted values etc. Consider output from SAS Visual Forecasting, where there have been massive input tables due to the nature of what is being forecasted e.g. groceries, values of stocks and shares, etc. Since the forecasted values will need to be used further down the business process, the likelihood is the table will need to converted into another format e.g. CSV, for other applications to leverage. Therefore when working with large output tables, the customer team need to consider when and where to place the output tables, and when to convert them into another format. If a customer team has a mixture of VA users, data scientists and forecasting specialists all using one CAS Server, it may be preferable to initially write the output table into HDFS or DNFS as a SASHDAT file. Then later in the day (outside of normal office hours) write the table out as CSV file and make it available to the business users outside of the Viya environment. Writing very large output tables to the CASDATADIR location (think 10 GB and upwards) during normal office hours may impact the user experience for multiple user groups (assuming they all share one CAS Server.   Summary Knowing the purpose of the CASDATADIR directory and it's sub-directories will help technical architects, users and administrators alike. It will hopefully limit unwanted end-user experiences and contribute to a performant SAS Viya environment. As always comments are welcome, especially if you think there is something which you consider could add value or needs clarifying/correcting.   Thanks, Simon ... View more

Reading Mark Thomas's article, which discussed automating a license update for SAS Intelligence Platform based offerings, it made me think. What information might be useful to contribute to an efficient life cycle management of the SAS platform(s)?   (Below is a visualization of the SAS platform life cycle I like to use when discussing the context of activities to get the best out of the SAS platform.) In discussions with colleagues here at SAS, they highlighted the importance of correctly identifying software orders as part of keeping our SAS platforms updated over time.   For some of SAS's customers, they may recognize this scenario. The customer team installed the SAS platform 12-24 months previously. For one of several reasons during the time that has passed, the customer team ended up downloading different software orders from SAS.   Some of the common use cases for downloading multiple software orders include:   The customer downloaded the maintenance updates of the software; The customer ordered some additional software to add to the existing platform; The SAS and customer team realized that one of the software orders did not contain all of the relevant software so requested a new software order; The customer decided to change their topology to have SAS platform split between 2 operating systems, rather than one operating system   Now let's walk through a simple example using a fictional company and how they would proceed. Let's call them "Thomas Morgan Electrical Widgets Inc." (TMEW).   One day the business and IT teams at TMEW decide to add more software to their existing platform. So what should they do first? A good starting point is for the required stakeholders to review the SAS Administration Guides which discuss the steps to add and update an existing SAS Platform. That helps folks get on the same page and reminds everyone of the high level activities ahead.   As part of these activities they should introduce/re-introduce the term "Software Order Number" to the stakeholders. They can share the useful pages on the SAS communities site and external facing page as part of those discussions. Keeping it simple, the software order number is used to identify the unique combination of:   the SAS software ordered the site IDs associated with the order what software revision number underpins the software order e.g. Rev. 940_15w33 Note. The software revision number is sometimes referred to as the "ship event". It is called out in the "Software Order Emails" sent to customers and is found within the "Internal Reference" field.   The next step for the TMEW team would be to walk through the 'adding, updating and upgrading' concepts which are discussed within the SAS(R) 9.4 Intelligence Platform: Installation and Configuration Guide, Second Edition.   The previous steps are straightforward, especially when only one software depot has been downloaded. But what happens where there are multiple SAS software orders which have been downloaded and no one within the TMEW team know for sure which software order was used to construct the existing operating SAS platform? Without knowing which software order was used to install and configure SAS, the TMEW team run the risk of:   updating the existing installation with the wrong software not updating the existing software to the latest version not being able to add the new software the customer wanted inadvertently damaging the existing SAS platform The initial reaction may be to run a proc setinit command or open the SAS Management Console to look for the information? Both methods help identify the site-IDs, but ... they do not identify the software order used. So how do we get the information?   For Windows based installations you can use the instructions in this SAS Note to get details of the specific software order used.   For Linux based installations you can use these commands:   grep last_order <SASHOME>/deploymntreg/registry.xml grep order <CONFIGDIR>/Lev1/ConfigData/status.xml   This information can then be passed back to SAS, so they can generate a software order which meets the requirements of the TMEW team.    And finally, the TMEW team needs to leverage the previously used deployment plan as a starting point (to add new software, update the software, migrate etc). To identify the last used plan file, take a look at SAS(R) 9.4 Intelligence Platform: Installation and Configuration Guide, Second Edition.   That's it folks! To some of those reading this it will be old news, but hopefully for some folks this will be new and useful when working to maintain your SAS platforms.   Thanks, Simon    ... View more