topic Re: How to submit the same job 64 times with one submit? in SAS Programming

How to submit the same job 64 times with one submit?

dgower — Wed, 30 Nov 2016 16:32:52 GMT

My preference is to do this interactively versus batch, but it's not necessary. I simply want to submit:

data _null_;
slept= sleep(1*60);
run;

64 times all at once to confirm 64 jobs will run simultaneously. I post this assuming that there's a more clever way than repeating the code 64x.

Thanks,

Dan

Re: How to submit the same job 64 times with one submit?

snoopy369 — Wed, 30 Nov 2016 20:31:34 GMT

How are you submitting this? Are you using SAS/CONNECT or a SAS GRID server or just running sas.exe 64 times?

Re: How to submit the same job 64 times with one submit?

dgower — Wed, 30 Nov 2016 20:35:32 GMT

We have a GRID setup with 7 compute nodes. The interface is SAS Studio 3.5. Each host is setup to handle 32 jobs. I'm increasing that to 64 and need a simple test to confirm 64 jobs will run.

Re: How to submit the same job 64 times with one submit?

snoopy369 — Wed, 30 Nov 2016 20:40:34 GMT

I wonder if you'd be better posting this in https://communities.sas.com/t5/Administration-and-Deployment/bd-p/sas_admin ? (Not sure, i'm not regular enough here to know how topics fall out among the communities.)

Re: How to submit the same job 64 times with one submit?

dgower — Wed, 30 Nov 2016 20:55:01 GMT

I assumed this is more of a programming question than an admin task. You may be right, though.

Re: How to submit the same job 64 times with one submit?

Kurt_Bremser — Thu, 01 Dec 2016 10:18:04 GMT

First of all, your system admin should be able to tell you the maximum number of concurrent processes per user, if such a limit has been set; on the UNIX systems I know, that would need running some kind of workload manager, as the default is that there is no limit.

For all practical purposes, the technical limit of processes in UNIX systems is so large that other limitations come into effect (memory) before one runs out of space in the process table.

To simply check if a certain number of process can run in parallel, I'd write a shell script like that:

COUNT=0
while [[ COUNT -lt 64 ]]
do
  /usr/bin/sleep 10&
  COUNT=$(($COUNT + 1))
done
wait

save it to every node (don't forget to chmod u+x), and run it from SAS with

filename oscmd pipe '/path/script 2>&1';

data _null_;
infile oscmd;
input;
put _infile_;
run;

so you catch all output. The script should be lying dormant with all its children for 10 seconds and then return.

Keep in mind that this only checks the number-of-processes issue; depending on settings like MEMSIZE (and the type of steps run), SAS processes might hit other limits much earlier. Which also might cause a system to become (nearly) unresponsive.

Re: How to submit the same job 64 times with one submit?

Kurt_Bremser — Thu, 01 Dec 2016 10:28:36 GMT

PS running ulimit -u on the nodes should tell you the limit for concurrent processes, if such is set.

Also run ulimit -Hu to see if there is a difference between the soft and hard limits (hard limits are set by the superuser and cannot be changed).

Re: How to submit the same job 64 times with one submit?

dgower — Thu, 01 Dec 2016 13:39:17 GMT

Thank you, Kurt. All our SAS servers are Windows servers and I'm looking for testing SAS jobs, not processes. You can establish a max jobs/queue or host within SAS GRID/LSF. I'm increasing the max jobs or MXJ and need a test to prove that the change was effective.