DATA Step, Macro, Functions and more

How to submit the same job 64 times with one submit?

Reply
Contributor
Posts: 35

How to submit the same job 64 times with one submit?

My preference is to do this interactively versus batch, but it's not necessary.  I simply want to submit:

data _null_;
slept= sleep(1*60);
run;

64 times all at once to confirm 64 jobs will run simultaneously.  I post this assuming that there's a more clever way than repeating the code 64x.

 

Thanks,

 

Dan

Regular Contributor
Posts: 244

Re: How to submit the same job 64 times with one submit?

How are you submitting this? Are you using SAS/CONNECT or a SAS GRID server or just running sas.exe 64 times?
Contributor
Posts: 35

Re: How to submit the same job 64 times with one submit?

We have a GRID setup with 7 compute nodes.  The interface is SAS Studio 3.5.  Each host is setup to handle 32 jobs.  I'm increasing that to 64 and need a simple test to confirm 64 jobs will run.

Regular Contributor
Posts: 244

Re: How to submit the same job 64 times with one submit?

I wonder if you'd be better posting this in https://communities.sas.com/t5/Administration-and-Deployment/bd-p/sas_admin ? (Not sure, i'm not regular enough here to know how topics fall out among the communities.)
Contributor
Posts: 35

Re: How to submit the same job 64 times with one submit?

I assumed this is more of a programming question than an admin task.  You may be right, though. 

Super User
Posts: 6,928

Re: How to submit the same job 64 times with one submit?

First of all, your system admin should be able to tell you the maximum number of concurrent processes per user, if such a limit has been set; on the UNIX systems I know, that would need running some kind of workload manager, as the default is that there is no limit.

For all practical purposes, the technical limit of processes in UNIX systems is so large that other limitations come into effect (memory) before one runs out of space in the process table.

 

To simply check if a certain number of process can run in parallel, I'd write a shell script like that:

COUNT=0
while [[ COUNT -lt 64 ]]
do
  /usr/bin/sleep 10&
  COUNT=$(($COUNT + 1))
done
wait

save it to every node (don't forget to chmod u+x), and run it from SAS with

filename oscmd pipe '/path/script 2>&1';

data _null_;
infile oscmd;
input;
put _infile_;
run;

so you catch all output. The script should be lying dormant with all its children for 10 seconds and then return.

 

Keep in mind that this only checks the number-of-processes issue; depending on settings like MEMSIZE (and the type of steps run), SAS processes might hit other limits much earlier. Which also might cause a system to become (nearly) unresponsive.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 6,928

Re: How to submit the same job 64 times with one submit?

PS running ulimit -u on the nodes should tell you the limit for concurrent processes, if such is set.

Also run ulimit -Hu to see if there is a difference between the soft and hard limits (hard limits are set by the superuser and cannot be changed).

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Contributor
Posts: 35

Re: How to submit the same job 64 times with one submit?

[ Edited ]

Thank you, Kurt.  All our SAS servers are Windows servers and I'm looking for testing SAS jobs, not processes.  You can establish a max jobs/queue or host within SAS GRID/LSF.  I'm increasing the max jobs or MXJ and need a test to prove that the change was effective.

Ask a Question
Discussion stats
  • 7 replies
  • 283 views
  • 0 likes
  • 3 in conversation