SAS software always does a great job allowing clients to update the SAS level and know their SAS code will still execute. With Viya and SAS 9.4, we also made sure the Hadoop In-DB libraries (EP) were common allowing each software level to execute with the same capabilities (scoring and parallel data movement). What I found is the SAS In-DB published score codes differ between versions (which is fine) and requires the score execution method to be modified to work with the different published score codes. Note, even with SPRE engine in a Viya deployment, the SAS 9.4 scoring macros are disabled in Viya SPRE engine.
What you need to know about Publishing Score Code in Hadoop:
SAS publishes two types of score files in Hadoop for In-DB scoring:
1) DS2 (.ds2 file) – raw DS2 score code that can prepare and score the data
2) Item Store (.is file) – binary score code that can contain ds2 (prepare and maybe score) and the Analytic Store (astore - binary score file)
When you have a SAS data step score code, the publishing process in SAS 9.4 and Viya will translate the data step code to DS2 and then publishes that into Hadoop.
SAS 9.4 can publish score code as DS2 or as Item Store in Hadoop based on the score code type. SAS 9.4 publishes majority of the score code as DS2 score code unless you have an astore. Only HPFOREST, HPSVM and SAS Function Compiler (FCMP) can create an astore file in SAS 9.4 which means these are the only models in SAS 9.4 that will publish an Item Store in Hadoop.
SAS Viya publishes all score code (data step, DS2 and astore) as an Item Store in Hadoop.
In the “Detail Code examples” sections, you can see the details on how to publish score code for each SAS level.
What you need to know about Executing Score code in Hadoop:
Both SAS 9.4 and Viya use the same In-DB binaries in Hadoop to run score code which means either SAS level can execute published score code from any version. The only catch is the SAS level method to execute the different published score code files (.ds2 or .is).
In the “Detail Code examples” sections, you can see how to execute score code for each SAS level. Here are some high-level comments on executing score code in each version.
SAS 9.4 Execute model:
When you execute a published model using the run model macro on SAS 9.4, you can specify either a DS2 file or an Item Store file stored in Hadoop:
1. To run a model stored as DS2 program in Hadoop, you use the scorepgm= option.
2. To run a model stored as an Item Store om Hadoop, you use the store= option.
Make sure to check Hadoop’s published score code directory for the type of file (.ds2 or .is) to select the correct option when executing score code in SAS 9.4. Remember, if the model was published by Viya, you use the “store” option in the macro because they are published as Item Store.
Viya Execute model:
Remember, the SPRE engine in Viya is similar to SAS 9.4, but the scoring macros are disabled in Viya. When you execute a published model using the SPRE proc in Viya (PROC SCOREACCEL), this can ONLY execute Item Store publish score codes. This works great for all Viya models and only SAS 9.4 astore published models (very limited list with SAS 9.4 - HPFOREST, HPSVM and FCMP). Note, this SPRE proc will convert to call CAS Action set to score the model.
When you execute a published model using the CAS action set (modelPublishing) in Viya, you can choose to execute an Item Store or DS2 published score code. This is the only way in Viya to execute all SAS 9.4 published models and Viya published models in Hadoop.
Detail Code examples
Below are examples on publishing and executing score code in Hadoop for each software level.
Viya Publishing Models (using SPRE vs CAS action):
With Viya, you can choose to use the SPRE procedure or CAS action. Here is an example of using SPRE procedure to publish score code:
proc scoreaccel;
publishmodel
target=hadoop
caslib="caslibhadoop"
modelname="&scorename"
modeltype=DS
filelocation=local
programfile="&scoredir./&scorename..sas"
xmlfile="&scoredir./&scorename..xml"
outdir="&scoredir./out"
modeldir="/sasmodels"
classpath="/hadoop/lib:/hadoop/conf"
;
quit;
Viya Execute Score code
Here is the SPRE procedure to execute publish score code, but this can ONLY execute Item Store published score code (all Viya models and only SAS 9.4 astore models):
proc scoreaccel;
runmodel
target=hadoop
caslib="caslibhadoop"
modelname="&scorename"
modeldir="/sasmodels"
server=" hadoophostname "
intable="datatoscore"
outtable="datascored"
forceoverwrite=yes
;
quit;
Here is the CAS action that can execute any model published in Hadoop (Item Store AND DS2), this is the only way for Viya to execute all of the SAS 9.4 models published:
proc cas;
loadactionset "modelPublishing";
action runModelExternal submit /
modelName="&scorename"
externalOptions={
extType="hadoop",
server="hadoophostname",
modeldir="/&scorename",
classpath="/hadoop/lib:/hadoop/conf",
ds2file="/sasmodels/&scorename./&scorename..ds2",
inTable=" datatoscore",
outTable=" datascored ",
forceOverwrite=TRUE,
trace=TRUE
}
;
run;
SAS 9.4 Publish Scoring:
Here is the SAS 9.4 publish macro to publish DS2 score code in Hadoop:
%INDHD_PUBLISH_MODEL(dir=&scoredir., datastep=&scorename..sas, xml=&scorename..xml,
modeldir=/sasmodels, modelname=&scorename., action=replace);
Note, this assumes you have both the data step code and xml file locally so the macro can convert this to DS2 code in Hadoop.
Here is the SAS 9.4 publish macro to publish Item Store score code in Hadoop:
%INDHD_PUBLISH_MODEL(dir=&scoredir., store=&scorename..sasast, datastep=&scorename..sas,
modeldir=/sasmodels, modelname=&scorename., action=replace);
Note, this assume you have the analytic store and sas code to run that astore so it can convert to an Item Store in Hadoop. Here is example code to generate the sas code for astore:
proc astore;
describe store="&scoredir./&scorename..sasast" epcode="&scoredir./&scorename..sas";
run;
Note, this macro is ONLY usable in SAS 9.4. The SPRE engine in Viya does not have this macro, so you need to use the Viya proc/CAS action.
SAS 9.4 Score:
Here is the SAS 9.4 execute macro to execute DS2 score code in Hadoop:
%INDHD_RUN_MODEL(inputtable=hdplib.datatoscore, outputtable=datascored,
scorepgm=/sasmodels/&scorename./&scorename..ds2, forceoverwrite=true);
Here is the SAS 9.4 execute macro to execute Item Store score code in Hadoop:
%INDHD_RUN_MODEL(inputtable=hdplib.datatoscore, outputtable=datascored,
store=/sasmodels/&scorename./&scorename..is, forceoverwrite=true);
Note, this macro is ONLY usable in SAS 9.4. The SPRE engine in Viya does not have this macro, so you need to use the Viya proc/CAS action.
