Running SAS EG Projects sequentially

[Karthikk]

Hi,

I am using SAS EG 7.1 (64-bit) on windows environment.

I have three SAS EG Projects 

1.Sales

2.Marketing

3.Customers

Each EG Project has 5-6 Process flows in it. Each EG project takes  minimum 4-5 hours to run all the process flows. I am not sure about the time because sometimes it takes more than that.

My concern is I want these projects to run automatcially one after the other in sequential order only. And also if somewhere it is breaking or if it throws error it should stop and send a message to the concerned person.

Is there any way to do this. If it is done by coding can someone please share the code with little of explanation.

Regards,

Karthik

[SASKiwi]

If these are business-critical processes you would be far better off running these on your SAS server as scheduled batch jobs given the length of time they take - they would run a lot faster there too. The scheduler can email alerts if there is an error and your IT Operations staff can be available if there are problems.

You are pretty much on your own if you run this from a PC. You would have to export the code from your EG projects to do server-based scheduling though.

[Karthikk]

Hi Kiwi,

Thanks for replying.

Thses projects are running in SAS Server only(Where SAS is installed). I have more EG projects to run this was just sample I cannot adjust time as to run all projects it takes almost 3 days for one time update so you can just imagine if you want a weekly update then when you should schedule

My main concern is I want to write a code which i dont know how to run these projects one after the other without scheduling it with respect to time. 

If not can you suggest me the steps to be followed for what you have mentioned below. It would be a great help if you could tell me what you want to convey in step by step process.

Thanks & Regards,

Karthik

[Kurt_Bremser]

---

@Karthikk wrote:

Hi Kiwi,

Thanks for replying.

Thses projects are running in SAS Server only(Where SAS is installed). I have more EG projects to run this was just sample I cannot adjust time as to run all projects it takes almost 3 days for one time update so you can just imagine if you want a weekly update then when you should schedule

My main concern is I want to write a code which i dont know how to run these projects one after the other without scheduling it with respect to time. 

If not can you suggest me the steps to be followed for what you have mentioned below. It would be a great help if you could tell me what you want to convey in step by step process.

Thanks & Regards,

Karthik

---

What you describe is THE scenario for using a scheduler, period. Stop thinking small, what you are doing now is already bordering on neglicence if those processes have any business significance.

Integration with a scheduler is not a trivial task at first, but once you have mastered that, your future work will become a breeze. This comes from ~ 4 decades in the business, 2 decades with SAS.

Your first step must now be to export your codes so they can run from the commandline in batch mode.

The second step is to identify which parameters need to be supplied to make each run respond dynamically (different infiles, execution date, ....)

Your scheduler can set environment variables for you, and you can import those into SAS with %sysget. This is how our interface to the scheduler works.

[Karthikk]

Hi Kurt,

Thanks a lot for your valuable information. 

It was more like a business understanding for me. I Did not understand it technically as I am new to this.

I am confused on "Your first step must now be to export your codes so they can run from the commandline in batch mode".

1.Did you mean I have to export the whole EG project as SAS Code ??? If yes how am i supposed to do it??

2. How to Run in batch mode??

This has gone over my head. I Could not understand this.

"The second step is to identify which parameters need to be supplied to make each run respond dynamically (different infiles, execution date, ....)

Your scheduler can set environment variables for you, and you can import those into SAS with %sysget. This is how our interface to the scheduler works."

Sorry to say this but I am lot confused. You seems to be very close to my problem but I think there is a communication gap.

Regards,

Karthik

[Kurt_Bremser]

OKOK. We have to start from further up the line 😉

SAS has several modes of execution:

Interactive with Display Manager (what you get when you just enter "sas" on the commandline)

"interactive" through the IOM bridge (this is what you get when you start the workspace server from Enterprise Guide)

batch mode: you start sas with a programname

In the last mode, SAS starts, reads the program, executes it, writes a log (and possibly output), and terminates when the last line of the program (or a termination statement like endsas) is reached.

Batch mode does not have (or need) a user interface, and can be called from any external source (eg the scheduler program).

How is your SAS set up (which type of server are you using - Windows, Linux, AIX, Solaris, Mainframe)? What kind of scheduling does your organization use (you will need to get in touch with your IT people)

Do you have commandline access to your SAS server (using a tool like PuTTY)?

[Karthikk]

Hi Kurt, 

So we have installed SAS in the server and it is Windows 2012 R2. SAS VA 7.3 and SAS EG 7.1

Do you have commandline access to your SAS server (using a tool like PuTTY)?----- NO

Now that all I understood is I need to export all the SAS EG projects into.sas file and run it through a batch file and using windows task scheduler i will need to schedule them. Correct me if I m wrong.

My Concerns are how am I supposed to export the whole EG Project into code and how should i run multiple batch file in order.

Hope you understood my concerns.

If SAS Communities had chat options to share screen shots pdf directly as per the group (eg: SAS EG, SAS VA) it would have been very easy to communicate.

Regards,

Karthik

[Kurt_Bremser]

To test your code in batch mode, you need login access (can be through a graphic interface) on the server. Have that implemented by your IT people. As soon as you can at least start a cmd window on the SAS server, you can start converting to batch.

Also check with your IT people how to schedule jobs from your company's scheduling system on the SAS server. I strongly advise against creating an insular "solution" that is not integrated into the company-wide IT landscape. That would also not go nice with your auditors.

[Kurt_Bremser]

Stop running those projects from EG. NOW.

Export to SAS codes and have the scheduler run the programs on the server.

I can only strongly second what @SASKiwi already said.

And I also suggest to dissect the code into smaller functional units, so if one of the pieces breaks, you don't have to rerun the whole process or a large piece of it. If a 5-hour job fails after 4.5 hours, you lose half of a workday to fix it. If a 1-hour job fails after 30 minutes, you'll be up and running again in 2 hours.

[CaseySmith · Accepted solution]

Hi @Karthikk,

I agree with @SASKiwi and @Kurt_Bremser that long-running business-critical jobs are better off scheduled as .sas code files on a server in batch.  That said, as far as your original question goes, yes, you can run multiple EG projects sequentially using a VBScript file with the following steps:

-In EG, with any project open, click File->Schedule Project and click OK.

(This will create an EGScript1.vbs file in the same directory as your EG project.  Note: It will also create a Windows Scheduled Task, which you can delete, since you'll be running the .vbs file manually.)

-Run the EGScript1.vbs file from a command window and ensure it executes the single project you scheduled.

(After confirming the EGScript1.vbs file works for a single project, we will alter it to run multiple projects...)

-Open the EGScript1.vbs file into a text editor.

-Replace the following line of code:

prjName = "C:\Users\cassmi\Desktop\Project.egp"    'Project Name

...with this code:

Dim prjPaths
prjPaths = Array("C:\Users\cassmi\Desktop\Project1.egp","C:\Users\cassmi\Desktop\Project2.egp")	

For Each prjName in prjPaths

-Edit the paths to point to the .egp project files you wish to run.

-Near the bottom of the EGScript1.vbs file, add this line of code immediately before the "End Sub" line of the dowork Sub:

Next	

-Now when you run the EGScript1.vbs file, it will loop through each project you specified, open it, run it, save it, and close it.

Casey

[Karthikk]

Hi Casey,

First of all I would like to thank you so much for making my life simple as the it is very easy to understand what you said.

I would like to follow the steps you mentioned. 

My doubts are what statetements should I add in my command prompt to Run the EGScript1.vbs file from a command window and ensure it executes the single project you scheduled.

Second this  EGScript1.vbs file after making changes should I be running manually every time I want to run the EG projects sequentially or can I also shcedule this run in Command Prompt or Command window which means can I automate this? 

I think I am very close to what I want now.

I admit that these things have been mentioned by @SASKiwi  and @KurtBremser but I m a layman to understand the admin concepts.

A Kind request from my side would be that if you could elobarate and answer my current doubts it would fulfill my requirement.

Thanks & Regards,

Karthik

[Karthikk]

Hi Casey,

I am sorry Connecting to the the same old question my concern is also about what if the EG Project breaks in between (for an example it has errors in between may be in joins or in creating a dataset) Will I be able to send this Error or Warning to the concerned person automatically when it breaks and It must stop running there so that he can go back to the back end and resolve the problem.

I hope you understand my doubts and concerns.

Regards,

Karthik

[Kurt_Bremser]

---

@Karthikk wrote:

Hi Casey,

I am sorry Connecting to the the same old question my concern is also about what if the EG Project breaks in between (for an example it has errors in between may be in joins or in creating a dataset) Will I be able to send this Error or Warning to the concerned person automatically when it breaks and It must stop running there so that he can go back to the back end and resolve the problem.

I hope you understand my doubts and concerns.

Regards,

Karthik

---

What you describe are core functionalities of any enterprise scheduling / job control system. You try to reinvent the wheel, and at the moment it looks like something from a Flintstones movie, while your corporate scheduler is made by Goodyear or Pirelli.

Talk to your IT people. If you need tips for running SAS in batch from a scheduler, ask here. Many of us have been doing that for decades.

[SASKiwi]

Do you have a SAS administrator you can talk to about your scheduling requirements? Since these are all new concepts for you it would very useful to get help from others who are more knowledgable in your organisation. They may be able to help you set up a better solution rather than trying to do it all on your own.

There are just so many things that can go wrong with PC scheduling, including power cuts, someone else logging on, or borrowing your network cable while you are not there, or IT applying software updates, or a network outage, or.....the list goes on. If your objective is to make it as reliable as possible then get some expert SAS admin help to schedule your application on your SAS server. You may find out you can set server scheduling up in SAS Management Console, a PC tool much like EG:

http://support.sas.com/documentation/cdl/en/scheduleug/68697/HTML/default/viewer.htm#p1st3swwtt231jn...

[Karthikk]

To @CaseySmith @Kurt_Bremser @SASKiwi I really thank you all for all your efforts to make me understand the issue better. 

Cheers!!

Regards,

Karthik