Q: Can I use Enterprise Guide to access huge data?

[ChrisHemedinger]

EG works well with large data -- bring it on!

The key is often in how you access the data. It's always best to define the data in terms of the remote server, using a libname and the appropriate engine to get to Oracle, DB2, SPDS, whatever. By using libraries defined in this way, data access performance should be comparable to what you would experience in SAS natively. EG does not retrieve millions of rows to the client machine -- it reads the data in buffered increments as required for display in the data grid.

EG provides other convenient avenues to get to data sources. It has built-in hooks for ODBC, OLEDB, OLEDB for OLAP, and MS Exchange. Typically data access for reading (opening in the grid) is very fast when using these data sources. However, you can have performance issues when you attempt to run a task or a query on one of these "non-SAS" data sources. The reason is that in order for EG to process the task, it must move the data to the server where the code will run. For large volumes of data this can be slow. That's why we always recommend going through the appropriate SAS/ACCESS or database engine on the SAS server to get to your data.

Chris

[deleted_user]

I am interested in how to do this. I recently got EG, and need to pull data from Oracle. I'm told that we have SAS/ACCESS to do this, and I read in the help that I need to create a library, and to reference "SAS Enterprise Guide Explorer 4.1: User's Guide", which I cannot find, for information on how to create the library.

Can someone help?

[ChrisHemedinger]

The book you want is Administering SAS Enterprise Guide 4.1, and it's available at:

http://support.sas.com/documentation/onlinedoc/guide/admin4.pdf

You can also use the Tools->Assign Library wizard to define a library for use just within your Enterprise Guide project.

Chris

[deleted_user]

Thanks for the quick reply.

I was able to create the library in the way you described. It was only accessable to my project, however.

I then went into SAS Management Console to create the library on the server for all the users to access using tips from your post.

ps. The documentation seems to be off, or I may be looking at the wrong thing. I looked at the admin guide (I had seen it before posting here), and went to the "Adding a Library" section, but the menu navigation it described did not match with what I saw in my menu. For example, it said to go to File > New > Library, but I don't have a Library option.

[ChrisHemedinger]

You can also do the same thing via: Tools->SAS Enterprise Guide Explorer. That brings up an administrative application that allows you to do the File->New->Library step.

The end result is the same as if you used SAS Management Console -- a new library is added for all users to access.

[tomrvincent]

'access' is one of those weasel words like 'process' or 'change'. 🙂

I've found that small, targeted explicit pulls (like picking all IDs from a main table) is fast (using indeces when possible) and then I use that pull to do cross-platform joins back to the database from other tables (like detail lines for claims, for example). Especially efficient in a parallel processing environment.

Focus on limiting transfers from the database to SAS as much as possible, doing the bulk of selections, calculations, aggregations and so on back on the database machine itself. Let database engines do what they do best.