large data management related

[Ruhi]

Hi All

I am working on a large data set for first time, and I have to choose couple variables from the given long list of given thousands of variables . Also dataset has around 850,000 observations.  Every time I run two or three data steps in sas , sas tells me output window full. (NOTE: SAS IN WINDOW ENVIRONMENT), I know unix does better, but don't have that.

Question1: how to deal with this issue. How to clear output window. I tried ODS HTML CLOSE;ODS HTML;

but when sas output window is full it does not allow me to run these commands. If you have other good ideas to share on big data management that would be helpful.

Second question: How to cut short my data set to try to get required data , I mean I want to try my commands on a small subset of data to see if they work properly , so that later I can apply them on entire data set to get my required data to carry out the analysis.

Thanks for your time

Ruhi

[ballardw]

ods listing close; If output is going to window.

to select a limited number of records (observations) you can use the dataset option OBS= anytime you are reading a dataset:

proc freq data=example(obs=100);  to use 100 records from the dataset example.

data want;

     set have (obs=100 ); to read 100 records into the new data set.

If you only want to have a few variables out of a long list then you can use the KEEP= option

data want;

     set have (keep= varname1 anothervarname thatvar);

You must be generating output to fill up the window. What are you doing to write to the output window in a data step?

[Ruhi]

I am not choosing any output destination.

Would writing output in a Libref.dataset_name would help.

[ballardw]

Depending on your install SAS will default to sending some types of output to either the Listing destination (Output window) or HTML (Results window) and possibly both. If you are running DATASTEP code and having the Output window fill up, then your code is being told to write output, possibly with a FILE PRINT statement. You might want to post an example of the code that is causing the issue.

And since datastep code will generate an output data set unless using the special set _NULL_, I really am curious what you're doing currently.

[Ksharp]

ods _all_ close;

OR

use proc printto to redirect the LOG into a external LOG file .

[LearnByMistk]

data want (keep = a b c );

set have (obs=200);

run;

if you want to  create a random sample then

data want(keep=a b c drop=i );

do i =1 to 5000 by 10;

set have;

end;

output;

run ;

[jakarman]

In the ancient time when big data meant you had a lot of cartons with punch cards the interface was designed for humand access.

- sysin      where you put your cards in.

- print       that print on real paper for the results

- saslog   The messages what your program did.

These basics are still there. As it should human readable there has been limits set on the log and Print.

When your ouput is going to a file you can turn the results off. SAS(R) 9.4 Output Delivery System: User's Guide, Third Edition

As you are under a windows license with DMS

There are al lot of windows-specific options. SAS(R) 9.4 Companion for Windows, Third Edition

But the sizing of the log is set to 9999 for all OS types.  SAS(R) 9.4 System Options: Reference, Third Edition  dmslogsize

the output (results) can also be set. SAS(R) 9.4 System Options: Reference, Third Edition DMSOUT 2G of lines is an awful lot

You could cleanup before your are going to run.