topic SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects in SAS Enterprise Guide

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

jvidalg — Thu, 23 Feb 2012 16:27:06 GMT

Hi,

I have started to work with SAS/EG a few ago, I have worked a lot with SAS/BASE, but not SAS/EG.

I have a doubt about SAS/EG. I know the main features of the tool, but I don’t know how it works with high volume of data and a lot of number of projects.

I am starting a project with SAS/EG 4.3. We have to migrate JCL-Cobol process to SAS/EG. The process are typical process of reading files, sorting data, merging data, filtering data, aggregating data…, typical ETL steps. The process are typical process yo can do with SAS/EG.

The problem is the volume. The volume of process and the volume of data used in the process.

We have to migrate 500 processes. There are process that move files of 100 Gb.

If I make 500 .egp projects in guide, ¿can we have them in a unique metadata or server?, ¿do I have to share out between several servers/metadata?. ¿How many projects can manage a metadata with good performance?.

If you have to work with 100Gb files, would you use SAS/BASE or SAS/EG. I know, that I can modify code in my SAS/BASE programs in order to have a better performance, shorter processing times (objects like indexes, data in memory, etc..), but in SAS/EG I think I can modify less things…., am I wrong?

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

Doc_Duke — Thu, 23 Feb 2012 16:42:47 GMT

I can help with some of the questions.

First, the conceptual model for EGuide and SAS. EGuide is a client that interacts with the SAS Server. The SAS server performs the analyses. So, whatever performance tweaking you can do with SAS/Base, you can do in a Code Task in EGuide. Data volume should be no more of an issue for EGuide + SAS server than SAS/Base alone. You may see some performance changes in going from a mainframe to a unix SAS server, but that is unrelated to the EGuide front end.

Scheduling through EGuide. Because EGuide only runs on Windows, you must use the Windows Scheduler on the local PC for the the EGuide project; you cannot use the shell scripts or Unix cron.

I can't help on the metadata piece.

Doc Muhlbaier

Duke

SAS/EG managing high volume of data and SAS/EG metadata managing high number of projects

SASKiwi — Fri, 24 Feb 2012 03:58:26 GMT

There are two ways of using EG:

1) Write and run SAS code just like in good old Base SAS. In this case the performance and processing issues are the same.

2) Use the EG project builder which will automatically generate the SAS code to run.

Since you are more familiar with SAS code and Base SAS you may prefer to use the first option.