topic Re: running SAS in parallel using SLURM job arrays in Administration and Deployment

running SAS in parallel using SLURM job arrays

sfinch — Thu, 05 Oct 2017 18:33:04 GMT

Has anyone ever succeeded in running SAS in parallel on a SLURM-based Linux cluster computer using what are known as "job arrays"?

https://rcc.uchicago.edu/docs/running-jobs/array/index.html

In particular, I need to pass a SLURM environment variable SLURM_ARRAY_TASK_ID from my batch shell script to SAS. This task has completely defeated me.

Google searches haven't helped. I see mention of multi-threading, but that's not quite the same.

I would be so appreciative for some expert help!

Re: running SAS in parallel using SLURM job arrays

SASKiwi — Thu, 05 Oct 2017 19:06:56 GMT

I don't know anything about SLURM but the SYSGET SAS function reads OS environment variables.

Also SAS jobs can be run in parallel using SAS Grid or SAS/CONNECT technology.

Re: running SAS in parallel using SLURM job arrays

sfinch — Thu, 05 Oct 2017 19:15:53 GMT

Unfortunately the SAS command

%let SLURM_ARRAY_TASK_ID=%SYSGET(SLURM_ARRAY_TASK_ID);

does not seem to work. I'm going to guess that SLURM_ARRAY_TASK_ID isn't an "environment variable" in the same sense you mean, but I could be wrong. Thanks for your reply!

Re: running SAS in parallel using SLURM job arrays

SASKiwi — Thu, 05 Oct 2017 19:37:46 GMT

If you could get your shell script to also write it to a "proper" OS environment variable as well that might work. Someone who is more familiar with Unix should be able to help.

Re: running SAS in parallel using SLURM job arrays

Kurt_Bremser — Thu, 05 Oct 2017 20:09:28 GMT

Either add

export SLURM_ARRAY_TASK_ID

to your shell script before starting SAS, or use the -sysparm option to hand SLURM_ARRAY_TASK_ID over on the SAS commandline.

Re: running SAS in parallel using SLURM job arrays

sfinch — Fri, 06 Oct 2017 12:30:52 GMT

Thanks for your suggestions! I've tried many variations, including the lines:

#!/bin/bash

#SBATCH --job-name=job_array
#SBATCH --output=job_array_%a_out.txt
#SBATCH --error=job_array_%a_err.txt
#SBATCH --array=1-3

echo "My SLURM_ARRAY_TASK_ID: " $SLURM_ARRAY_TASK_ID
export SLURM_ARRAY_TASK_ID
sas -noterminal job_array.sas -sysparm $SLURM_ARRAY_TASK_ID

in my shell script and the lines

%let TskID=%scan(&sysparm,1);
%put &TskID;

in my SAS program. Regrettably the value of TskID seems to be empty; the integers 1, 2, 3 seem not be passing into SAS. More help is definitely needed and appreciated!

Re: running SAS in parallel using SLURM job arrays

Kurt_Bremser — Fri, 06 Oct 2017 12:46:19 GMT

With the export in place, did you try

%let tskid=%sysget(SLURM_ARRAY_TASK_ID);

Re: running SAS in parallel using SLURM job arrays

sfinch — Fri, 06 Oct 2017 13:01:49 GMT

Yes. With the export in place, the SAS lines:

%let TskIDa=%SYSGET(SLURM_ARRAY_TASK_ID);
%put &TskIDa;

unfortunately give empty TskIDa. Any other ideas? I'm grateful for your help. If passing the SLURM_ARRAY_TASK_ID parameter from shell script to SAS is impossible, then this is a disappointment. I don't wish to give up quite yet, but I'm at a loss what to try next.

Re: running SAS in parallel using SLURM job arrays

Tom — Fri, 06 Oct 2017 13:26:39 GMT

Are you sure you have a standard SAS command line script and not some local script that is "eating" your command line arguments?

Try just passing in any text to the SYSPARM option. So make a trivial SAS program.

%put &=sysparm;

Then call it with the -sysparm option.

sas -sysparm fred myprogram.sas

In the log you should see

SYSPARM=fred

If that works then replace fred in the command string with the reference to your environment variable.

Re: running SAS in parallel using SLURM job arrays

sfinch — Fri, 06 Oct 2017 13:51:35 GMT

Thank you. The command in my shell script:

sas -noterminal trivial.sas -sysparm $SLURM_ARRAY_TASK_ID

plus your suggested command in my SAS program:

%put &=sysparm;

indeed gives me an integer (1, 2 or 3). Thus something is wrong with my SAS commands:

%let TskID=%scan(&sysparm,1);
%put &TskID;

which give TskID as empty. How should I correct these? I need, of course, to use TskID later in my program.

Re: running SAS in parallel using SLURM job arrays

Tom — Fri, 06 Oct 2017 14:00:35 GMT

If you want to just assume that the caller has passed the proper value in SYSPARM then just remove the %SCAN() function call.
%let tskid=&sysparm;

Re: running SAS in parallel using SLURM job arrays

sfinch — Fri, 06 Oct 2017 14:12:05 GMT

Unfortunately my attempt to memorize the sysparm value:

%let TskID=&sysparm;
%put &TskID;

still appears to be empty. Why should your command:

%put &=sysparm;

work so beautifully but not mine? I appreciate your suggestions!

Re: running SAS in parallel using SLURM job arrays

Tom — Fri, 06 Oct 2017 14:41:25 GMT

This simple program should work.

1    %let TskID=&sysparm;
2    %put &TskID;
fred

There is no reason that %SCAN() shouldn't work either. You could check to see if SYSPARM has gotten any strange binary characters into it that are confusing SAS into thinking it has macro quoting. Try a simple data step and see it if helps. So you might use SYMGET() to pull the value SYSPARM and then use PUT with $HEX format to see if there are any strange characters in the first 25 bytes. You could then use CALL SYMPUTX() to generate your new macro variable.

data _null_;
  length tskid $200 ;
  tskid=symget('sysparm');
  put tskid = / tskid $hex50. ;
  call symputx('tskid',tskid);
run;

If it still doesn't work then either your shell script or your SAS code is messing up somewhere.

Perhaps there is something in your larger shell program that is causing the -sysparm option to not be properly set on the command line. Perhaps somehow the environment variable reference is being deferred to some later time where it evaluates to empty? Or these is some other line in the code that is clearing the value before the call to SAS?

Same thing for the SAS side. Do you have other statements that are changing TSKID or SYSPARM? Are you sure you are not creating TSKID in a local symbol table and then looking for it in the global symbol table?

Re: running SAS in parallel using SLURM job arrays

sfinch — Fri, 06 Oct 2017 15:30:09 GMT

Thanks for all your thoughts. I implemented your data step and obtained the following printout in the log file:

tskid=3
33202020202020202020202020202020202020202020202020

What is meaning of the 50-character string? It looks ominous.

SLURM_ARRAY_TASK_ID is analogous to a do loop variable, covering the integers {1, 2, 3}, however, all at once.

I said the log file, but in fact I have three copies of SAS running in parallel. Does the difficulty reside in the fact that outcomes for three simultaneous processes are being collapsed into just one file? Maybe collisions are making this unworkable.

This is why I need TskID -- to distinguish between the processes -- but I don't know how to introduce it into SAS's naming of log files, etc., to prevent the collisions from taking place.

Re: running SAS in parallel using SLURM job arrays

Tom — Fri, 06 Oct 2017 16:00:27 GMT

Your dump of the hex codes looks fine, '33'x is the digit 3 and '20'x is a space. SAS uses fixed length fields so all variables are padded with spaces.

Perhaps your only issue is that you are overwriting your SAS logs?

Use the -log and -print command line options to re-direct the SAS log and listing outputs to different files.

For example you might add run1, run2, etc to the filenames.

#!/bin/ksh
n=1
fn=myprogram
sas -sysparm $n -sysin ${fn}.sas -log ${fn}_run${n}.log -print ${fn}_run${n}.lst

You might also want to look into the -stdio command line option which will take the program code from standard input and direct the log and listing to standard output and standard error.

Re: running SAS in parallel using SLURM job arrays

sfinch — Fri, 06 Oct 2017 16:15:07 GMT

Thanks again for these very helpful thoughts! The failure of:

%let TskID=&sysparm;
%put &TskID;

persists, despite my avoidance of SAS log overwriting (as of just now). What a mystery! I will return to this issue in a few days, but appreciate your efforts. Maybe alternatives will arise eventually.

Re: running SAS in parallel using SLURM job arrays

sfinch — Wed, 11 Oct 2017 19:59:39 GMT

The following paragraph:

LIBNAME myplace "/home/username/";
DATA myplace.x&TskID;
LENGTH TskID $200 ;
TskID=symget('sysparm');
RETAIN TskID;
RUN;

allows me to save TskID to an appropriately-named sas7bdat file (x1, x2 or x3); the parameter value is henceforth available to me later. This solution seems rather cumbersome. I'm curious whether anyone can offer a more elegant way of doing this. As discussed earlier, for some reason

%let TskID=&sysparm;
%put &TskID;

does not work for me. Thanks very much for your guidance!

Re: running SAS in parallel using SLURM job arrays

sfinch — Thu, 12 Oct 2017 12:39:59 GMT

Interesting. If, in my shell script, I replace

#SBATCH --array=1-3

#SBATCH --array=1993-1995

(actual years rather than small indices), then the SAS commands

%let TskID=&sysparm;
%put &TskID;

suddenly work! This is an unexpected but gratifying resolution.

Might such behavior be a SAS bug? Namely, when the environment variable $SLURM_ARRAY_TASK_ID is small, it fails to be properly passed from shell script to SAS program.

Re: running SAS in parallel using SLURM job arrays

sfinch — Thu, 12 Oct 2017 18:44:24 GMT

-sysparm $SLURM_ARRAY_TASK_ID

was definitely the light at the end of the tunnel: thanks!