A few beginner questions...

[Benjamin_Wang]

Hi, I'm a first time user of SAS, but I am an experienced programmer in other languages. I am using SAS it to access a very large database with measurement readings, but getting very poor results in terms of how fast the system is responding and also navigating the UI.

I tried to collect all of the historical values for a subsection for the past year, but the application crashed.

Now when I try to re-run the application by narrowing it down to the month, the application never completes. It's staying in 'Running...' status but does not seem to be doing anything.

I had a few questions I'd like to ask about SAS EG:

1. Is there any type of console or terminal log I can pull up within SAS EG to see what the program is actually doing in real time to ensure it isn't stuck?

2. The query I initially ran froze the PC, actually used up over 150 GB of storage on my HD completely filling it up and crashing. I deleted the 'results' file to free up that space. Is there any resource someone can point me to that I can use to learn how to pipe out the query output in SAS into smaller files with a maximum of X records in each file? 

[Tom]

So either the data is too large for your machine or you have made some type of coding error. For example perhaps you have coded an infinite loop.  Or are writing millions of line of text to the SAS log.

In general it is best to break the process into steps.  But with large data you don't want to make multiple copies of the same data so that can get difficult.  You might need to see if creating views might help.  If you are subsetting the data to make it smaller make sure to subset as soon as possible.  Either by including a WHERE clause to limit the observations or combination of DROP or KEEP options to limit the number of variables included.

You need to post a more detailed description of what you are doing. 

What is the problem you are trying to solve?  What does your data look like?

Are you writing SAS code and running it? Or just clicking on the tools in EG to have EG generate the SAS code for you?

[TomKari]

One problem is that there are several layers in the stack that are doing things concurrently:

- The DBMS is retrieving the data from its own disk.

- The data is being passed over a network to a SAS server. (Note that EG is actually only a client, which generates and submits SAS code to a SAS process somewhere, which may or may not be on the same machine that EG is running on).

- The SAS server is doing whatever it is supposed to do with the data.

- Results are being passed over a network to your EG session on your machine.

When I need to troubleshoot issues like this, I try to verify that each layer is performing properly:

1. I use a tool like TOAD to submit the SQL query, and see how long the database takes to process it.

2. Same, but on the server that SAS is running on to see if network issues are a problem.

3. Submit the SAS code directly from a SAS session on the SAS server. Again, do problems appear?

4. Are there any communication bottlenecks between the SAS server and your PC? From the comments you're making, it sounds like either your SAS process is on your local PC, or the SAS process is generating a huge amount of output and downloading it to EG, which isn't common.

As you're a new user, I know that some of these steps may be challenging. If you have access to SAS expertise at your organization, it would be a good idea to bring them into the discussion.

There is a SASTRACE system option that can give you some visibility into what is being submitted to your DBMS.