Parallel Execution of SAS Macro With %include

[RussellAlmeida]

Hi,

I have a sas program which writes .pdf to the webdav server.

there could be approx 30,000 .pdf files to be written/rewritten in a single day and the job need to be finished before the users come to work that is at 7:00am every morning.

I have a macro program which pipes the files listing and generates a dynamic program which is executed with the help of %include filename.sas.

If i want to optimize my script with the customer requirement, how can i fire %include statement and proceed to the next file without waiting for the %include to get over.

Can i use the function CALL EXECUTE to make programs run in parallel.

Please suggest.

Regards
Russell

[Cynthia_sas]

Hi:
Do you have SAS Data Integration Studio? Are you running a job with the job scheduler in the Enterprise Intelligence Platform?? Are you doing this as a regular batch job without using DI Studio?? Are you trying to write to the webDAV server (Xythos) from a stored process???

Once you are writing programs to execute on the Stored Process Server or the Workspace Server, you sometimes have to change what you do -- it might be different from a regular BASE SAS job. Platform configuration and server configuration might impact how you run the job (whether you are using GRID servers, server pooling, etc).

You probably should work with Tech Support on this question, because the kind of load scheduling/pooling/etc that you have on your servers and how you are submitting the code (via DI Studio job) or via batch SAS job, could impact how you write your program. So before you go down a CALL EXECUTE path -- which will place all the code to be executed into a "stream" of code in one SAS session -- you might really want to work with Tech Support so they can take your specific Platform configuration into account.

cynthia

[RussellAlmeida]

Hi Cynthia,

I have a base sas code which i would like to execute in parallel.

I want to generate several files which will write simeltaneously to the Xythos server.

Regards
Russell

[Cynthia_sas]

Hi:
I understand and I still think you should work with Tech Support. You need to decide which method you're going to use. Just coding a CALL EXECUTE or using a %INCLUDE will not work to achieve parallel processing.

In a single batch job, you would be submitting code to a single SAS session -- and, as you have noted, for 30,000 outputs, this may not be desirable. Unless you code the ability to spawn separate SAS sessions (as described in the papers); in your single batch job, you would be running everything in ONE SAS session. Probably not what you want to do. Much of this functionality (grid computing, load balancing, spawning servers as needed) is built into how the servers work in the Enterprise Intelligence Platform.

These papers define parallel processing basic concepts and implementation in SAS:
http://www2.sas.com/proceedings/forum2008/391-2008.pdf
http://www2.sas.com/proceedings/forum2008/046-2008.pdf
http://www2.sas.com/proceedings/sugi30/224-30.pdf (note that this is an older paper from SUGI 30 in 2005)
http://www2.sas.com/proceedings/sugi29/124-29.pdf (a workshop from 2004)

If you have grid computing enabled on your Platform configuration, you might be able to use grid computing to process your task as you want. Here's some specific GRID computing documentation:
http://support.sas.com/documentation/cdl/en/gridref/62853/HTML/default/n049jf2x471mybn1pn9f6tpz3400....
http://support.sas.com/documentation/cdl/en/gridref/62853/HTML/default/p06nu8c3qifl8vn15ya1ghu97lku....

For specific help with your batch job, you should decide what parallel processing method you're going to use and then work with Tech Support if you need more help to implement your solution.

cynthia

[data_null__]

Perhaps can you achieve the same effect with SYSTASK? You could START a few SAS programs, effectively what want to CALL EXECUTE now, to run synchronously. Wait for them to finish and start a few more.

See documentation for detail on SYSTASK.

[Flip]

I had a very similar task about a year ago. It is very much dependant on your opperating system and scheduler. SAS can generate the jobs and submit them to your batch queue. At the simplest you can uses SAS to generate the jobs, then submit them to the batch proccessor. The OS then has control of the jobs and you will need to monitor the batch queue if you need to get control back to SAS at the end.
By monitoring the queue you may control how many jobs are running concurrently. In my case I needed to limit to 5 or 6 so that I did not impede other systems. When one finishes, then start another.
This is all going to be OS coding executed from within SAS. SAS will not multitask from a single session on its own.