topic Re: parallel programming in SAS Programming

parallel programming

makset — Fri, 31 Jul 2020 14:14:27 GMT

Can someone help me? I can't find the error. The program does not wait for the execution of tasks _n1 and _n2

LibName help "C:\test\";

data help.a;
informat datetime  datetime21.;
input datetime	Nr	temperature_1	temperature_2	temperature_3	temperature_4	temperature_5	temperature_6	temperature_7	temperature_8	temperature_9	temperature_10	pressure_1	pressure_2	pressure_3	pressure_4	pressure_5	pressure_6	pressure_7	pressure_8	pressure_9	pressure_10 ; 
format datetime  datetime21.;
cards;
01NOV2012:20:09:00	112113	8.8386586448	9.7382110218	5.0758258603	8.8694050722	6.8936349358	9.432546082	9.2878638301	4.8766398407	7.7988508134	8.7713846681	1	1	1	1	1	1	1	1	1	1
19DEC2013:00:09:00	112113	3.1682274491	3.8016460347	0.0828465261	6.5064271912	7.5012415438	0.2482288703	8.2464989461	4.0366380266	4.2733701062	5.230450104	61	61	61	61	61	61	61	61	61	61
29OCT2014:23:09:00	112113	4.2798770685	7.3606959847	9.6644437662	0.4209582577	6.5941452747	0.6463373546	2.728187507	7.1689701942	3.5560273053	2.1887783287	121	121	121	121	121	121	121	121	121	121
07DEC2017:23:09:00	112113	0.4644698882	7.4807286961	5.35556752	9.3602973665	7.9723406397	1.3072503288	0.65210792	5.4844892211	7.0024573081	6.0352300038	181	181	181	181	181	181	181	181	181	181

run;


data help.b;
informat datetime  datetime21.;
input datetime	Nr	temperature_1	temperature_2	temperature_3	temperature_4	temperature_5	temperature_6	temperature_7	temperature_8	temperature_9	temperature_10	pressure_1	pressure_2	pressure_3	pressure_4	pressure_5	pressure_6	pressure_7	pressure_8	pressure_9	pressure_10 ; 
format datetime  datetime21.;
cards;
28JAN2004:23:09:00	112114	3.4588757018	0.9221725771	8.3306752145	4.5803881995	8.2675979147	9.1477185837	5.7202544645	7.9451231612	0.9673189698	1.8442960829	241	241	241	241	241	241	241	241	241	241
28MAR2006:23:09:00	112114	6.3791716751	2.1213577455	7.9790208535	5.2773255971	8.6945169419	1.6644138051	0.0204887614	5.0024614809	5.4114883579	6.6648952151	301	301	301	301	301	301	301	301	301	301
19OCT2011:00:09:00	112114	6.7945698788	5.6084340392	9.3033694685	0.8323060418	6.897864067	6.0847914289	4.7261788161	4.3651668495	5.3854351002	9.6577573661	361	361	361	361	361	361	361	361	361	361
16DEC2013:19:09:00	112114	3.7592288386	5.5167134176	7.3533652583	5.7666407409	8.0186763406	5.0429561432	4.3662598124	8.0805433588	1.0869284347	3.2223363873	421	421	421	421	421	421	421	421	421	421
23SEP2014:19:09:00	112114	5.7138312096	9.8083063331	0.7149568223	6.3757409411	6.7465888965	1.9086576509	8.1664893031	3.7590556568	7.3485474777	9.9914430827	481	481	481	481	481	481	481	481	481	481
11FEB2019:19:09:00	112114	8.5294209281	1.6220973455	7.9244413553	9.1218567034	7.1427283529	4.5025826711	2.4023903767	9.7427219199	9.1612814669	6.319673683	541	541	541	541	541	541	541	541	541	541

run;

		data _null_;
		file "C:\Test\test1.sas";
		put ;
		put 'LibName help "C:\Test\";';
		put ;
		put 'data help.aa;';
		put 'set help.a;';
		put 'run;	';
		run;


		data _null_;
		file "C:\Test\test2.sas";
		put ;
		put 'LibName help "C:\Test\";';
		put ;
		put 'data help.bb;';
		put 'set help.b;';
		put 'run;	';
		run;


		systask command "START /high ""Batch"" ""C:\Program Files\SAS\SASFoundation\9.2(32-bit)\sas.exe"" 
		-sysin ""C:\Test\test1.sas"" -log ""C:\Test"" -nodms -noterminal -nostatuswin -nosplash -noicon" nowait taskname = _n1; 

		systask command "START /high ""Batch"" ""C:\Program Files\SAS\SASFoundation\9.2(32-bit)\sas.exe"" 
		-sysin ""C:\Test\test2.sas"" -log ""C:\Test"" -nodms -noterminal -nostatuswin -nosplash -noicon" nowait taskname = _n2; 

waitfor _all_  _n1 _n2 ; 

		data help.aabb;
		set help.aa
			help.bb;	
		run;

Best regard

Re: parallel programming

s_lassen — Sat, 01 Aug 2020 15:08:43 GMT

The problem is that you are calling the tasks with the Windows command START, meaning that they will be detached from the calling process. You should simplify the SYSTASK command to just calling SAS. Using SYSTASK, the calling SAS process will take care of the rest.

Something like this (not tested):

systask command """C:\Program Files\SAS\SASFoundation\9.2(32-bit)\sas.exe"" 
		-sysin ""C:\Test\test1.sas"" -log ""C:\Test"" -nodms -noterminal -nostatuswin -nosplash -noicon" nowait taskname = _n1;

Re: parallel programming

ChrisNZ — Mon, 03 Aug 2020 01:53:50 GMT

You launch two OS processes which are detached from the original SAS process.
~~You need to use MPCONNECT to enable the behaviour you want.~~

~~And WAITFOR is a SAS/CONNECT statement, so surely wherever you found this statement, the rest was there too.~~

[Edited: Crossed out the misleading sentences. See below.]

Re: parallel programming

jimbarbour — Sun, 02 Aug 2020 14:45:53 GMT

@ChrisNZ,

With respect (obviously you know WAY more about SAS than I do), but one does not need to use MPCONNECT in order to use WAITFOR or to conduct parallel processing. It's a common misconception that MPCONNECT is required. A very well regarded SAS employee and author told me just that when I was trying to set up a SAS job with parallel processing on a machine without a SAS/CONNECT license. I felt that the aforementioned individual was mistaken, got everything working, and wrote a paper about it the following year. If it were of interest: https://www.lexjansen.com/wuss/2018/15_Final_Paper_PDF.pdf

However, that said, if MPCONNECT were available, it is quite a bit more straightforward to use RSUBMIT to conduct parallel processing than to use SYSTASK. Using SYSTASK to conduct parallel processing on a machine with a SAS/CONNECT license makes little sense.

Regards,

Jim

Re: parallel programming

SASKiwi — Sun, 02 Aug 2020 20:14:36 GMT

AFAIK there is nothing syntactically wrong with your program. If @s_lassen 's suggestion doesn't help then I'd suggest a Tech Support track would be a good idea.

Re: parallel programming

ChrisNZ — Sun, 02 Aug 2020 22:41:18 GMT

@jimbarbour

No offence at all, I stand corrected,it is always good to learn something new, I has no idea using SYSTASK allows SAS to keep tabs on the child processes.Thank you for the link to your paper! 🙂

Re: parallel programming

SASKiwi — Sun, 02 Aug 2020 23:01:50 GMT

I second @ChrisNZ 's comment. I too was unaware of the WAITFOR statement working with SYSTASK, so great to learn this.

I agree with @jimbarbour in that the SAS/CONNECT approach is the better option if you have that product available to you. For example, SYSTASK can't do remote parallel processing or even any remote processing - processing on another SAS server remote from the primary server SAS and SYSTASK is running on.

Re: parallel programming

jimbarbour — Tue, 04 Aug 2020 03:33:16 GMT

@ChrisNZ,

Thank you for taking my comment in the spirit in which it was intended. You've given me a number of good SAS tips, so I'm glad I could perhaps give one in return.

By the way, in my paper I discuss a "raw" file and the use of FIRST OBS and OBS to segment the data for parallelization. With a SAS data set, which has pointer direct read capability, one can dispense with algorithm I wrote to "front load" the data so that the various processes running in parallel end at roughly the same time and just use the pointer feature. Likewise, a continuous variable in a database when used with a WHERE clause is a fairly straightforward way to segment for parallelization (assuming that the variable used has a fairly even distribution). In all cases, I have found significant improvements in run time. However, I suspect I'm "preaching to the choir" here. 🙂

Jim

Re: parallel programming

ChrisNZ — Mon, 03 Aug 2020 01:45:10 GMT

@makset Just a note: Be careful when paralleling jobs if you hit a single storage location. You can easily degrade performance if you hit disks randomly rather than in a more organised manner.

Re: parallel programming

jimbarbour — Mon, 03 Aug 2020 00:06:48 GMT

@SASKiwi,

I've found SYSTASK useful for controlling multiple non-SAS sub-processes from within SAS. I've used it to run hundreds of simultaneous unzips, which would take quite a long time to run serially, all submitted from within SAS. Through use of TASKNAME=, STATUS=, and SYSRC, SAS is aware of their successful or unsuccessful completion and can take actions accordingly.

And, you're quite correct. SYSTASK is great for making use of multiple cores in the CPU that SYSTASK is launched on but is not capable, to my knowledge, of executing on a remote machine.

Jim

Re: parallel programming

ChrisNZ — Mon, 03 Aug 2020 01:52:08 GMT

> Windows command START, meaning that they will be detached from the calling process

Or maybe adding the /wait parameter to the start command would keep the child processes in sync?

Just an idea, if you want to keep controlling the priority.

Re: parallel programming

makset — Mon, 03 Aug 2020 12:40:22 GMT

if I do this it work

		systask command "sas -sysin C:\Test\test1.sas -nolog -nodms -noterminal -nostatuswin -nosplash -noicon" nowait taskname = _n1;

but the previous method should also work according to the article:

https://amadeus.co.uk/tips/systask-the-spawn-of-sas-programmers/

Best regard

Re: parallel programming

s_lassen — Mon, 03 Aug 2020 13:47:30 GMT

Yes, the "START SAS" command works, inasmuch as it starts a process. But your subtask is now START, which will return as soon as the SAS task is started.