Add BATCH SUBMIT button to EG

[Quentin]

I just came from an EXCELLENT talk on Enterprise Guide for Programmers at Boston Area SAS Users Group, by Chris Bost.

The talk is available at: http://www.basug.org/downloads/2013q2/Bost.pdf

Not surprsingly in a room full of programmers, during the discussion the question came up, "can I do a batch submit?"

And I think currently, the correct answer is "no, you can't, because EG is an interactive environment, in the same (similar) way that DM SAS is an interactive environment."

Chris B. helpfully pointed out that you can schedule an EG project, which accomplishes something like a batch session.

But here's the thing.  When I programming in SAS, I was taught that I should ALWAYS submit my final code as a batch job.  There are two main benefits.  1. You ensure that your code works on it's own, and isn't working (or failing) because you have a lot of temporary datasets / formats / macro variables / whatever assembled junk sitting in your work library.  2.  You get a log file.

During discussion it was suggested by someone that might be nice if EG just had a "batch submit button."  I think it's a grand idea.  If I press "Batch Submit", I want EG to spawn a new SAS session on the server, and execute my program in that new session, then return the results as usual. Getting a log file would also be nice, but that is a separable issue (and PROC PRINTTO is available as a workaround).

Now when using EG, it seems like there are approximations of ways to ensure that you have a clean session to run your code in:

1.  Schedule the flow.

2.  Write a macro that will try to clean up your session (delete temp datasets/filerefs/librefs/macro vars/etc etc)

3.  Disconnect from the server and then reconnect.  (I feel like EG is unhappy with me when I do this, maybe it's because my code is linked instead of embedded in the project, I think EG loses track of the links when I disconnect)

4.  Close EG and restart it.

To me, those are all work arounds for not having a batch submit  button.  Sounded like a grand idea to me, so wanted to throw it out to the group.

--Q.

(edited 12/10/13 to add link to slides from presentation, and make clear that "Chris" is Chris Bost, not Chris H.)

[Patrick]

You can also export the EG code to a .sas file and then batch submit it out of EG using "sysfunc" or the like.

But: Excellent suggestion and something I would like as well. And if EG is also supporsed to be used by developers then please give also an option to batch submit code exactly as created by the developer without adding ANY EG generated code.

[ChrisHemedinger]

Quentin, this is a great idea and one that has been under consideration for a while.  I think the details of your suggestion (and expectations of how it should work) are helpful.

In my work these days (as part of SAS Professional Services) I create and run lots of SAS programs, many in "batch" mode.  I do all of my development in EG.  When it comes to running the code and ensuring it will work as desired, here's a few things that I do:

• To test your program in a new session, right-click on the server icon (ex: SASApp) and select "Disconnect".  That ends your current work session.  You can then rerun your project/program in a "clean" environment to ensure it doesn't rely on recycled WORK tables or options.
• If your program relies on certain SAS options to work, always set those options with an OPTIONS statement.  SAS Enterprise Guide, SAS display manager, and batch SAS all have some different default options that may influence how your program runs.  Eliminate the ambiguity by setting the options explicitly in your programs.
• With a standalone program, change the program properties in EG to suppress automatically generated ODS statements.  You can do that by selecting Properties (top-right icon above the program editor window), then Results tab, then select Customize results, and uncheck all of the results options.
• When developing a standalone program that will ultimately run outside of EG, save the program to the file system and test in batch from a command prompt or remote shell.  I often save SAS programs to my remote UNIX file system.  I edit and test parts of the program in EG as I develop, but I run the program via a SAS command on a remote shell before I roll it into "production".

Your suggestion has merit and could help to make this process easier; I find that these practices help me to stay productive using the current framework of SAS and EG.

Chris

[Quentin]

Thanks very much Chris,

I had heard of the Disconnect/Reconnect approach for getting a clean session.  But when I try this (in 4.3, with a linux server), I end up with problems.  I think it's because my code is out on the file system  (linked to from EG), rather than stored in the EG project.  After I reconnect, if I then try to edit and save the code, I get an error from EG.  And I end up needing to close EG and restart it.

I like your last suggestion.  Since my code should be able to run outside of EG, may try getting in the habit for now of keeping a remote shell window open for the final batch submit.

Thanks again,

--Q.

[ChrisHemedinger]

I think EG 5.1 is better about the remote-file/disconnect thing, but it's always best to Save Program (early and often) and close the program view before you disconnect. 

Chris

[David]

I think it is possible to run a SAS program in Batch if you go down the route of developing a Custom Task.  I have did this a few years ago when I wrote my own "DMS for EG" task, and basically created my own connection and ultimately Workspace Object directly to the server.  One of the various parameters allows you to get the full SAS log (otherwise seen as the Initialisation Log off the properties of SASApp).

The trick would be to get the termination log (i.e. the total time stamps etc. seen at the end of your batch session log).  Again - probably possible by wrapping a SYSTASK statement into the custom task as suggested elsewhere.  Then the problem is security as SYSTASK requires the ability to shell out of SAS to the operating system... so it all depends on the requirements of the organisation using SAS.

I probably got distracted by a real project and left source code in limbo.  I should search through the systems for that and see if we can add a little robustness to it!

Regards,

David.

[Haibo]

Batch submit button in SAS EG is a great idea. This is a feature we have been wanting since day 1 of using SAS EG due to the fact our work in clinical trials is more production oriented - we'd like to develop, test and validate code in EG and do production batch submission without switching to another tool or developing workarounds which we have to maintain/upgrade from version to version.

[PhilC]

Batch programs are suppose to be automatable and therefore, IMHO, not "push button".  Should this function need a batch mode command line option for the Enterprise Guide executable program? If a general EGP file is to be run in a batch mode how would a button know what Process Flow or Program Node to start at unless there was an instruction or a GUI interface? I would expect something like the following command line options be added to the SEGuide executable to implement this. E.g.- using windows:

SEGuide.exe <<current options> 
  (/batch 
    (/batchOptionFile "batchOptionFile.cfg")
    (/outputLog|/saveLogsInFile)
    (/runAutoexecPF)
    (/runPF|/runPN|/runPNbranch) 
    (/PF"processFlow")
    (/PN"Program Node")      
  )  project.egp ...

Sorry for the contrarian view, I see the usefulness of batch functionality, less of a one-button-does-all solution.

[Quentin]

Good points.  I think we're talking about two different kinds of batch submission functionality.

My suggestion for a batch submit button, would be to batch submit the current open program.  So you can hit F3 when a program is open to submit the open program (or highlighted code).  I would want a batch submit button (or function key or whatever) that would batch submit the open program (or highlighted code).  This would be useful for me, because I have all of my logic/macros/includes/etc in SAS code.  And the .cfg could be whatever .cfg is being used for the EG session that spawns the batch job.  I wouldn't need information from the process flow or EG-autoexec or other EG pre-code/post-code constructs.

Seems like what you are suggesting is a way to batch submit an  entire enterprise guide project  or a process flow, not just one program.  Agree, that would likely need more inputs/features, and may be useful to EG-users as well.  I would not need it, because I use EG mostly as a development tool for stored processes/DI jobs etc, I don't have production EG projects that I would want to batch submit.  But certainly make sense to consider such possibilities when considering possibilities for EG batch submits

[ChrisHemedinger]

The function that is describing is just about achievable today using the automation API.  It's not on the command line, as he sketched it out, but you can write VB Script or PowerShell script to run projects, programs, or flows in any combination.  You can also save out results and logs.

Quentin's idea is, in some ways, more basic -- take this code and run it in its own SAS session asynchronous from the EG project session, and then provide a way for me to rendezvous with the results.  At least, that's how I'm hearing it?

[Quentin]

Yep, you've summarized my suggestion correctly.  I'm just a basic guy, I guess.