08-19-2014 12:56 PM
I have been experimenting with the "Allow parallel execution on the same server" option in Enterprise Guide 5.1 (What was the first version that contained this option?). It seems to me that there should be a way to automatically generate and store the path of the WORK directory that is created when EG connects to the workspace server the first time so that you can call it automatically in code that is running in parallel. Is this possible?
Would I then (using prompts?) be able to insert this stored path in a libname statement so that certain parallel tasks and programs can read and write to the same work directory? I would like all of this to be done in an automatic and repeatable way. I have been trying to use the "Insert SAS Code.." and "Submit SAS Code.." options in EG to make this an automatic and the default method of submitting process flows. I have even considered altering the default output library (for tasks only?) so that all of this happens without having to alter existing EG process flows too much.
Any thoughts or suggestions on a way that I could take full advantage of the capabilities of the "Allow Parallel Execution on the Same Server" but store the data all in the same temporary location?Any good papers or documentation on this option that I have yet to find?
- We do not have SAS/CONNECT which would seem to resolve this issue somewhat.
- We also do not have a grid enabled environment, but that may be coming in the future.
- I also do not want all of these tables to be stored in a "permanent" location which is why I want to send the data to a "shared" WORK directory.
- I am well aware of the potential consequences of allowing users to run MANY SAS workspace sessions with the click of a button, but our user base is small and I am confident that with proper training and monitoring we can take advantage of these features without undue stress on our server.
- We are moving to EG 6.1 soon if that version treats this differently.
** It seems that this would be a useful feature (with proper role permissions) to program as part of EG out-of-the-box functionality to be able to use this option seamlessly in the way I describe for future versions of EG. Any word of this feature being worked on by developers? And since I am wishing, any chance EG could write all of this out to a stored process that takes advantage of parallel processing using multiple "sessions"?
08-19-2014 03:03 PM
EG 5.1 is the first version supporting parallel executing. You can set that parallel execution option at the flow and the node level.
The most easy way is setting it at flow level. This is coming with a disadvantage as parallel nodes do not update the master/Eguide node.
It doe not update library allocations done I the node nor macro vars or other settings. Makes sense as a property of asynchronous processing (parallel gird threading). This note is hidden in the online help of Eguide. You only can find knowing what to look for.
What you can do is:
- define a basic-start node. This node should not be run in parallel (change that option back)
- Define you own libarie is this that should get shared. Be ware that using the same dataset will cause locking problems.
Do not use the saswork for that all sessions are already creating a nested one in some master session.
You can see that as a repetition of the physical filename in work of the EGuide session.
Wanting something with that name you clould use %sysfunc(pathname(work))
- make flow dependicies from this job to the others as needed
- limiting the maximum number of session parallel is possible it is a EGuide profile setting.
With the Installation of EGuide you can set default properties. One of them is the limit of concurrent maximum threads. Choose eg 6.
08-21-2014 03:06 PM
Thank you Jaap. As usual, your comments were helpful. I will be especially mindful of the limit on the number of parallel processes.
I have come up with a partial solution by taking the following steps:
Step 1: Turn on "Allow parallel execution on the same server" at the project level.
Step 2: Store the "master" work session path somewhere fixed to be read in by other parallel processes
a="libname workshr '%sysfunc(pathname(work))';";
Step 3: Use this path to store any datasets that are used by other branches/nodes of your EG project
It seems a little contrived, but I chose the method that I did so that I have it set up and available to any EG project that I run. It also doesn’t require any changes for a project that that does not use "Allow parallel execution on the same server" option (i.e. unchecked).
Some drawbacks that I would like to find solutions to:
Any suggestions on how I could improve this?
08-22-2014 08:50 AM
Wow that is an update.
Adding some figures and some explanations it would almost ready for an paper as presentation.
The drawback questions.
1 It something of a central platform admin role to do preventing unwanted changes. For users read-only
2 That are choices.
3 ?? for SAS TS ?. It should not behave weird. Turning on/off during running not something likely to be usual.
4 I have seen macro-variables (filenames) seen behaving the same way.
Proc optsave / proc optload can be used for saving restoring options. Base SAS(R) 9.3 Procedures Guide, Second Edition
There are also ways to do that with a getoption approach SAS(R) 9.3 System Options: Reference, Second Edition
For macros there must be something like that, when not it is not too difficult to build.
An new proc is coming Presenv Base SAS(R) 9.4 Procedures Guide, Third Edition
5 Yep, that is parallel processing.
6 The work option is not to be changed in a running session. SAS(R) 9.4 Companion for UNIX Environments, Fourth Edition
If you want the default location, one level table names, to another location, That is the User option SAS(R) 9.4 Companion for UNIX Environments, Fourth Edition an can be changed as an option
I think you are looking for this one.
7 Creating directories with new names. SAS(R) 9.4 System Options: Reference, Third Edition DLCreatedir.
8 _clientprojectpath it is a nice variable but not always trustworthy. When possible choose something different?
9 Creating directories