Thank you Jaap. As usual, your comments were helpful. I will be especially mindful of the limit on the number of parallel processes.
I have come up with a partial solution by taking the following steps:
Step 1: Turn on "Allow parallel execution on the same server" at the project level.
- A) Use the EG menu to go to FILE > Project Properties
- B) Select the section "Code Submission"
- C) Check the box next to "Allow parallel execution on the same server"
- D) Save the changes
Step 2: Store the "master" work session path somewhere fixed to be read in by other parallel processes
- A) Create a "program" node in EG that reads in the WORK directory path and writes it out to a 'fixed' location
- a. I used the following code to create a user and project unique location for storing the path as a libname statement in a .sas file to be read in with a %include:
%let projectdir=%sysfunc(compress(&_clientprojectpath,,kn));
data _null_;
FILE "/sas/data/g_research/&sysuserid/paths/&projectdir..sas";
a="libname workshr '%sysfunc(pathname(work))';";
PUT a;
run;
- b. Caution: You want to be sure that this file containing the path isn’t accidentally overwritten in the middle of your process flow. To accomplish this, I used automatic/system macro variables generated by EG to create something that is unique to this project, but common for all SAS workspace sessions spawned by this project. An alternative to using macros for the path is to simply have a fixed path, but this runs a higher risk of being overwritten by another project using the same path. I am trying to make this as portable as possible to be able to use in many different EG projects.
- B) Change this program node so that it does not "Allow parallel execution on the same server"
- a. Go to the properties of the program node
- b. Select the section named “Code Submission”
- c. Click the bullet for the option “Customize code submission options”
- d. Make sure "Allow parallel execution on the same server" is NOT checked
- e. Save the changes
- C) Run this program node before running the rest of your project
- a. Note: Because this program node has had the "Allow parallel execution on the same server" turned off
- b. To automate things a little bit, I created an Autoexec process flow with this code node in it (Good info on Autoexec process flows: You asked for it: the Autoexec process flow - The SAS Dummy)
Step 3: Use this path to store any datasets that are used by other branches/nodes of your EG project
- A) Read in the path from the fixed location
- a. I used the following statements to call the code I had saved previously:
%let projectdir=%sysfunc(compress(&_clientprojectpath,,kn));
%include "/sas/data/g_research/&sysuserid/paths/&projectdir..sas";
- b. To automate things a bit more, I put these two lines of code in the following to option locations in EG
- “Insert custom SAS code before task and query code”
- “Insert custom SAS code before submitted code”
- Note: This will cause the libname to be assigned prior to every task or node in the process flow. A little bit of overkill, but the code is small and the assignment is temporary.
- B) Assign a Library to this path
- a. By using a %include referencing code that contains a libname statement, I have already accomplished this. If another method is used, you may need to have this be a separate statement that is run for every task and node in the EG process flow.
- C) Use the library to read and write any datasets that need to be in common.
- a. I tested this with many program nodes that all looked like the following:
data workshr.temp2;
set sashelp.cars;
run;
- b. I also tested this with a Query Builder task which also worked
- Note: To automate the libname assignment some, I chose the “workshr” directory as the first “Default library for Output Data” option in EG which seems to apply to all tasks, but not programs.
It seems a little contrived, but I chose the method that I did so that I have it set up and available to any EG project that I run. It also doesn’t require any changes for a project that that does not use "Allow parallel execution on the same server" option (i.e. unchecked).
Some drawbacks that I would like to find solutions to:
- 1. I created a “/path/” folder for all of the library paths to be stored in. This “/path/” directory would have to be added to any users home directories who use my project. This can be fixed with a more common or shared path… but that would mean more possibilities of corrupting that filing during process flow running.
- 2. Anyone else who used my project would have to add a %include statement to their “Insert Custom SAS Code…” options unless I put the %include in the program nodes themselves.
- 3. Turning the option "Allow parallel execution on the same server" on and off for the same project causes some weird behavior that looks like errors, but seems to still run correctly.
- 4. I believe that this only works for datasets and that options and macro variables that are assigned in any of these tasks or program nodes would not be able to be used in any other task or program node. Anyone no how to change the storage location of these other values?
- 5. If you are not careful, you could have locking issues or precedence issues where you ty to read a dataset that has not been created yet.
- 6. In order to apply this to existing EG projects, you would have to change the libraries for nearly every step to the shared work library. Has anyone come up with a good way to redirect the WORK library in the middle of running code?
- 7. I am sure there are others…. But I can’t think of them right now.
Any suggestions on how I could improve this?
