topic Integrating Python code in SAS DI Studio Job in SAS Data Management

Integrating Python code in SAS DI Studio Job

Ullsokk — Wed, 24 Oct 2018 11:41:34 GMT

I am building a SAS DI-studio job to update an analytical base table, as well as draw sample data for building predictive models. As part of the flow, I want to run some python code to do a an SGD classifier based on a term frequency–inverse document frequency of some words. The result of this classifier will be a list of words and weights that will produce features to be used in a SAS miner model.

As far as I can tell, I have the following options:

re-write the python program in SAS code (which I fear will be a non-trivial task, possibly requirering lisences we dont have, such as proc IML and/or SAS text miner), then apply this as a user written transformation or user written code
simply run the program manually, and import the results before doing the analysis in SAS MINER
use user-written code\user made transformation to run an x command to execute the python code
Run everything through python instead, using SASpy to execute the SAS code

Are there more options? What would you reccomend?

Re: Integrating Python code in SAS DI Studio Job

LinusH — Wed, 24 Oct 2018 13:03:21 GMT

I don't have any hands on experience of this, just giving my architectural PoV.

If this Python stuff can be seen as part of ETL, I'm pro of having that logic inside DIS.

I don't know which is the best method, but there seem to be an option to call Python from a data step Java object. See the link for details.

Perhaps you could create a custom transformation to package this?

https://github.com/sassoftware/enlighten-integration

Re: Integrating Python code in SAS DI Studio Job

Ullsokk — Wed, 24 Oct 2018 14:18:15 GMT

It is stricktly speaking not part of the daily ETL. But every six months or so, some 20 models will be updated with new data, which requires the python job to run. But will check out the link, thanks!

Re: Integrating Python code in SAS DI Studio Job

AngusLooney — Thu, 25 Oct 2018 11:04:58 GMT

I've certainly done this in the past using the xcmd route.

Essentially you're treating the script as any other OS command. Just need to make sure that the account you run your SAS batch processes as is properly setup to execute the python, e.g. has python in it's path, has permissions to run it, and whatever rights it needs to any files/directories.

Top tip - get the python (or bash, cmd, powershell or whatever) script to echo something to standard out when it finished, like some success or fail or whatever text/messages, and use a pipe to read that, so that your SAS process knows that it's worked, or not. You can't always rely on return codes, or the fact that the thing you called actually ran, to know if it worked or not. Experience with flaky Windows servers has lead me to always try to do this.