EG Work Library

Accepted Solution Solved
Reply
Valued Guide
Posts: 858
Accepted Solution

EG Work Library

 

Outside of my current project I am in the habit of setting up several projects within one process flow in SAS, it helps me keep my code organized.  I am accustomed to clicking on the process flow and hitting F3 to run all of the projects sequentially. When I do that here project1 will run and project2 will queue as I expected, but when SAS starts to run project2 as scheduled the data from project1 is not available in the work library, Any libnames set up are also not available. It seems like the connection to the server is lost completely between projects so any joins using the work library are invalid. I also noticed that the task status below will say running 1 of 4, rather than running 2 of 5 at the start of the project2 code.

I've never seen anything like this, the easy fix is copy and paste everything into one project which I've done, but I'm wondering if there is a setting in EG or a SAS option that is the root cause of this.  Any help will be greatly appreciated.

Thank You,

Mark


Accepted Solutions
Solution
‎11-07-2014 02:05 AM
Super User
Posts: 7,405

Re: EG Work Library

Keep the standard order of your output library, this won't do you much good. As Jaap said, I think you are a victim of the parallel/multithreading options.

When you connect to a workspace server, a SAS_workxxxxxxxxxxxxx_hostname directory is created, where the x's contain a hex representation of your process number. Within that directory, you may find several SAS_work..... subdirectories which are the WORKs for the threads. Different codes may be assigned to different threads, although they are using the same Appserver.

I suggest the following solutions:

- Either turn off any parallel processing options, or

- Create a dedicated "work" directory with a libname assignment in every code node, and put the data that need to be handed off there; use standard WORK only for data internal to each node. At the end of processing, clean out that "work".

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers

View solution in original post


All Replies
Valued Guide
Posts: 858

Re: EG Work Library

I just noticed that the default output library is EGTASK and not WORK.  I'm not sure if that is related.

Respected Advisor
Posts: 3,156

Re: EG Work Library

Because of the term 'project' , I am a bit confused about your description. As far as I know, one EG instance can ONLY operate on ONE EG project,  While you can have multiple tasks and process flows in ONE project. So here come the questions:

1. Are you actually trying to share WORK library content in between different process flows? if so, make sure your different process flows are connected to the SAME workspace server instance. If not, depending on how you set up your application server, you still maybe able to share it if UNC address being used in the deployment.

2. Or are you trying to share WORK library content in between different and concurrent EG instances? Again, it depends on the whereabout the specific workspace server instance.

3. For a single process flow, EG stores the WORK physical address in a MACRO variable "&sasworklocation",  you maybe able to leverage it for the purpose of sharing (such as to write it down to some UNC address that different SAS instances can access).

Hope this helps,

Haikuo

Valued Guide
Posts: 858

Re: EG Work Library

I apologize for the typo, I meant multiple programs under one process flow.

Respected Advisor
Posts: 3,156

Re: EG Work Library

Actually I need to take back some of my comments as well. Even within the same process flow, different task and different programs CAN be connected to different App server. So you may want to check whether those programs are using the same App server.

Haikuo

Valued Guide
Posts: 858

Re: EG Work Library

I see there are two options under 'selected server' but only one is connected.  Do you know of a way to force that, or set a default so it doesn't change?

Valued Guide
Posts: 858

Re: EG Work Library

Again I found something else.  I do see that where I am selected is the default.  I'm not sure if this is causing the issue or not.    I'm going to play around with the settings and see if I can duplicate the issue.  I'm going to start with changing the default folder to work instead of egtask.  I can't find any good documentation describing what the difference is.

Respected Advisor
Posts: 3,156

Re: EG Work Library

Hmmm, some thoughts: Is the option you mentioned under the 'Tools -- Options -- Tasks -- Output library'? If it is, then it really has nothing to do with your issue. It merely tells EG where to look for tables if not specified. If you ever wonder the library where your tables reside, just check the table properties by right click.

Valued Guide
Posts: 3,208

Re: EG Work Library

Your descried behavior of losing libnames/work etc is normal when you have set parallel processing. This option is available with EG 5.1 and up.

With Grid processing this is also standard.  The reason is that for decreasing runtime response that parallel processing is needed.

Processor speed do not really increase anymore since 2003.  https://www.cs.utexas.edu/~lin/cs380p/Free_Lunch.pdf

Just get used to manipulate this. With SAS code nodes you can set that option back when it is activated at egp project level.

The libname node has too many errors to be really useful. It is missing this setting as it is missing the option to change the appserver.   A code node is more reliable.    

---->-- ja karman --<-----
Valued Guide
Posts: 858

Re: EG Work Library

Jaap, the two programs do not run at the same time, so if this is true I'm running into this issue with no real benefit, I'm not sure I understand.

I am not familiar with manipulating this with a code node, can you expand on that or direct me to some documentation?

Thank You,

Valued Guide
Posts: 3,208

Re: EG Work Library

http://support.sas.com/resources/papers/proceedings12/297-2012.pdf see page 14 and further.

A libname statement is the classic way of coding. SAS(R) 9.4 Statements: Reference, Third Edition

---->-- ja karman --<-----
Valued Guide
Posts: 858

Re: EG Work Library

I'm not sure that this is related to my issue.  I do not have the grid or parallel run selected.

Solution
‎11-07-2014 02:05 AM
Super User
Posts: 7,405

Re: EG Work Library

Keep the standard order of your output library, this won't do you much good. As Jaap said, I think you are a victim of the parallel/multithreading options.

When you connect to a workspace server, a SAS_workxxxxxxxxxxxxx_hostname directory is created, where the x's contain a hex representation of your process number. Within that directory, you may find several SAS_work..... subdirectories which are the WORKs for the threads. Different codes may be assigned to different threads, although they are using the same Appserver.

I suggest the following solutions:

- Either turn off any parallel processing options, or

- Create a dedicated "work" directory with a libname assignment in every code node, and put the data that need to be handed off there; use standard WORK only for data internal to each node. At the end of processing, clean out that "work".

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 12 replies
  • 510 views
  • 3 likes
  • 4 in conversation