Desktop productivity for business analysts and programmers

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

Reply
Occasional Contributor
Posts: 12

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

Hi,

I have started to work  with SAS/EG a few ago, I have worked a lot with SAS/BASE, but not SAS/EG.

I have a doubt about SAS/EG. I know the main features of the tool, but I don’t know how it works with high volume of data and a lot of number of projects.

I am starting a project with SAS/EG 4.3. We have to migrate JCL-Cobol process to SAS/EG. The process are typical process of reading files, sorting data, merging data, filtering data, aggregating data…, typical ETL steps. The process are typical process yo can do with SAS/EG.

The problem is the volume. The volume of process and the volume of data used in the process.

We have to migrate 500 processes. There are process that move files of 100 Gb.

If I make 500 .egp projects in guide, ¿can we have them in a unique metadata or server?, ¿do I have to share out between several servers/metadata?. ¿How many projects can manage a metadata with good performance?.

If you have to work with 100Gb files, would you use SAS/BASE or SAS/EG. I know, that I can modify code in my SAS/BASE programs in order to have a better performance, shorter processing times (objects like indexes, data in memory, etc..), but in SAS/EG I think I can modify less things…., am I wrong?

Other question: 

I will make the project in a windows server but I have to deploy it in a Unix server, the process will be scheduled in a Unix server and will be called from a .sh Unix shell script. Is it possible?, I think so…., but I prefer to make the question...

As you can see my english is not profiency. I hope my questions are clear…

Any advice will be greatly appreciated.

Thanks in advance

Trusted Advisor
Posts: 2,114

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

I can help with some of the questions. 

First, the conceptual model for EGuide and SAS.  EGuide is a client that interacts with the SAS Server.  The SAS server performs the analyses.  So, whatever performance tweaking you can do with SAS/Base, you can do in a Code Task in EGuide.  Data volume should be no more of an issue for EGuide + SAS server than SAS/Base alone.  You may see some performance changes in going from a mainframe to a unix SAS server, but that is unrelated to the EGuide front end.

Scheduling through EGuide.  Because EGuide only runs on Windows, you must use the Windows Scheduler on the local PC for the the EGuide project; you cannot use the shell scripts or Unix cron.

I can't help on the metadata piece.

Doc Muhlbaier

Duke

Super User
Posts: 3,233

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

There are two ways of using EG:

1) Write and run SAS code just like in good old Base SAS. In this case the performance and processing issues are the same.

2) Use the EG project builder which will automatically generate the SAS code to run.

Since you are more familiar with SAS code and Base SAS you may prefer to use the first option. 

Ask a Question
Discussion stats
  • 2 replies
  • 629 views
  • 0 likes
  • 3 in conversation