topic Re: Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results in SAS Enterprise Guide

Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results

Terho — Tue, 18 Oct 2016 14:27:43 GMT

Hello,

I have 3 unlinked programs inside a SAS EG Project.

I have enabled "Parallel Execution on the Same Server" inside of Project Properties in SAS EG. When I run the project, all 3 programs execute in parallel and complete vith varying time - sometimes program 1 finishes first, sometimes program 2, sometimes program 3.

All three programs output to a report libname: report.data1, report.data2, report.data3

How would I consolidate the results into a single dataset? I want to run this data step below, but only when all 3 programs have finished executing:

data report.combined;

set report.data1 report.data2 report.data3;

run;

I can't link this datastep to the "longest" running program, because their execution times vary. I believe in Base SAS, I was able to use a "waitfor" command after rsubmit to different tasks. However, I cannot find anything equivalent in SAS EG.

My initial thought is to have a fourth program that "listens" in the report libname, and executes only when the 3 datasets have modified date > start time. Not too certain how this syntax would look though. Any help appreciated!

Thanks

Re: Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results

TomKari — Tue, 18 Oct 2016 14:35:01 GMT

I haven't done this for a while, but my recollection is that you can link Program1, Program2, and Program3 to your consolidation program, so it will wait unitl they're all done to run.

Give it a try,

Tom

Re: Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results

Terho — Tue, 18 Oct 2016 14:44:55 GMT

Ah! That was simple.

I created a seperate consolidated program, then "Linked" each of the programs to the consolidated program.

So, Link Program 1 to Consolidated, Link Program 2 to Consolidated, LInk Program 3 to consolidated.

When the project runs, the consolidated program only runs when all 3 are complete!
Thanks!

Re: Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results

Terho — Thu, 01 Dec 2016 14:55:46 GMT

Follow up question to Parallel Execution:

I noticed some unusual behavior when executing this parallel project when first loading the program.

The project does not execute in a parallel on the first run (each program waiting on the previous to finish):

However, if I stop the project, and then run the project again, the project will execute in parallel:

Any tips on why the first execution of the project does not happen in parallel, while the second and future runs do (until I close out of SAS EG)

Thanks

Re: Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results

CaseySmith — Thu, 01 Dec 2016 15:53:14 GMT

I suspect you are running into a bug that was fixed in EG 7.11 HF5, where (when using the parallel execution feature) items would incorrectly run synchronously if the workspace server was not already connected. After you have connected to the workspace server, it properly runs in parallel (hence, why you only see the problem the first time). So, you can workaround by first explicitly connecting to your workspace server (ex. expand it in the Servers view) prior to running the first time. Or you can install 7.11 HF5 or later (7.12, 7.13), where the issue has been addressed.

You can download 7.11 HF5 from this link:

http://ftp.sas.com/techsup/download/hotfix/HF2/U23.html

Or, alternatively, you can upgrade via Help->Check for Updates inside EG *if* your administrator/environment allows.

Casey

Re: Execute Parallel Programs in SAS Enterprise Guide - Consolidate Results

Terho — Thu, 01 Dec 2016 18:16:48 GMT

Interesting.. Thank you for your reply CaseySmith.

If I explicitly connect to the workplace server (by expanding my server view), then the project will run in parallel the first time.

I am currently on 7.11 HF2 (7.100.1.2785), so it seems like I am impacted by this bug. For now, I will be aware to explicitly connect to the server if I am running for the first time (or just never close my EG session)

The decision to upgrade to HF5 will have to be determined by our environment administrators unfortunately. Hopefully sometime soon!