Submitting a long running job in batch mode from a stored process

[Quentin]

Actually, Looks like I was wrong about 9.3 being the answer to my prayers.

From:

http://support.sas.com/documentation/cdl/en/stpug/62758/HTML/default/viewer.htm#stpugwhatsnew93.htm

Looks like the Workspace Server can't replay images.  Very unfortunate.

•   The SAS Workspace Server supports stored processes with streaming output, except stored processes that use sessions or replay (such as embedded images in streaming output).  

It also mentions a global macro var _METAPASS.  Does that mean that when a user logs into SPWA, their password would be saved in &_METAPASS (in an encrypted or at least encoded form?).

• Reserved global macro variables _ARCHIVE_PATH, _ARCHIVE_NAME, and _GENERATED_NAME have been added for publishing to WebDAV and Sharepoint. The _DEBUG_FILE reserved macro variable has been added for publishing to Sharepoint. Reserved global macro variables _FOLDER_PATH, _METAPASS, and _METAUSER have been added for publishing to subscribers. For more information, see Using Reserved Macro Variables

--Q.

[Yura2301]

Hi Scott,

I’ve just briefly read all posts higher and I probably has easier solution for this task.

So recently I had very similar task that you had, initially I implemented it  using only STP sessions only:

One STP session executes macro that runs for a few hours, another session – observe log file that this macro generates all this long time, and in the end, when log file become not in use by first STP session -   second STP session parse this  log file and send needed info to needed person via sms/email etc.

But during testing I’ve found serious issue of such approach – these two STP sessions locked two sas processes that manages all another STPs, that impact STP response time, it become bigger that is not acceptable.

Then I tried implement variant similar to your approach, but quickly refuse it because of it include changing sessions config files that has hardcoded some servers names etc. , and also probably such approach will be hard maintain , support and promote between servers.

So I decided join “batch processing” with “STP possibilities” to implement the task.

From STP I just create one table(or file, it’s not important) on some server path.

Then I created small flow via SAS management console, this flow has one file event,  - it waits until the file(or table) created by STP arrive on specific path on server.

After this file arrives – Flow runs.

Flow has two jobs, first job run this long time executed macro, in the end this job delete this table(file) created by STP.

After this job finished successfully or fails  - second jobs starts , this job executes parsing logs generated after first job executing, and send needed info to needed person via sms/mail etc.

Off course there are a lot of validations etc things . that I didn’t mentioned in this post, but shortly idea is from STP creates only some table(file) on specific place on server and create small and simple flow that waits for this file occurs(arrive) on this place.

As I said higher flow can be easy created and scheduled via SMC, and testing flow can be done from flow manager, that probably should be installed on your machine, if not -  it’s really easy install and configure it.

Benefits of such solution obviously – no need change any config files(session macros etc.) – easy maintain and support etc.

Minus is the fact that should be run some kind of demon( in my case it’s flow that waits for file arrives) that all time observe some directory on server, but I’m even not sure it is the minus, because it’s usual approach in scheduling process.

Hope this my post will help you a little bit if you would like change smth. In you actual functionaliti

Thanks!

[Quentin]

THanks for sharing, Yura.

I had been playing with scheduled flows in SMC, but hadn't realized I could kick of an event based on the arrival of a file in a directory.    Nifty approach to the problem.

--Q.

[Yura2301]

Hi,

Actually it's very eazy realize event on file: arrives, exists, have some specific size...

Via SMC or Flow editor you should just choose "file event" and then choose needed value from dropdown(arrive, exist etc.) and put the path to needed file on Server.

Thanks!

[Quentin]

Hi SASGeek,

I think the best way to flag errors in SAS is to scan the log.  There are lots of NOTES/WARNINGS that I consider errors which SAS does not.  Tricia Aanderud was kind enough to let me write a guest post-on her blog suggesting an approach to automated log-scanning in a stored process setting.

http://www.bi-notes.com/2012/07/stored-process-ignore-log-your-own-peril/

If you take a look, please let me know if you have any thoughts/suggestions.

--Q.

[sustagens]

Hi Quentin I'd like to look at your blog post but it looks like it's been taken down, can you link me to where it is currently please if it still exists? Thanks!