topic Re: What are the best ways to read the large data in SAS Programming

What are the best ways to read the large data

vThanu — Mon, 25 Feb 2019 07:22:01 GMT

what are the best ways to read the large data into SAS without creating the dataset to find out the whether the data is allined

Re: What are the best ways to read the large data

Patrick — Mon, 25 Feb 2019 07:29:09 GMT

@vThanu

You will need to provide quite a bit more detail and explanation for us to be able to give you advise.

On a high level: You can use DATA _NULL_; which will allow you to process data without writing anything to a target table (_NULL_).

Re: What are the best ways to read the large data

Satish_Parida — Mon, 25 Feb 2019 09:08:24 GMT

1. If you want to verify your code without runing a large load then you can use options obs=0
2. If you want to validate or check the data then you can control the data load to sas using options obs=<number of records>
3. If you want to validate the code with the wholde data file then you can use data _null_;

4. If you want to read the data efficeintly, use proc copy or proc dataset if both source and target are using similar db engine.

Re: What are the best ways to read the large data

RW9 — Mon, 25 Feb 2019 09:06:04 GMT

I am assuming that by "aligned" you mean the data is valid? I also assume that your using proc import. Switch your programming to use a datastep read - as suggested by @Patrick, use a data _null_ is you don't want to save the file. In the datastep read, specify each variable, informat, length, format, label etc. In this way once the code is executed, anything which doesn't conform to your structure will show problems in the log with the actual line of data which causes the issues (up to what you have set your options level to). There is no way to do this without reading at least header and first observation of the file.

Re: What are the best ways to read the large data

Astounding — Mon, 25 Feb 2019 09:56:47 GMT

It usually helps to examine a few lines of data. That doesn't necessarily involve programming but you can use a program if it would help:

data _null_;
Infile source;
input;
list;
if _n_ > 10 then stop;
run;

That dumps the first 11 lines of data to the log for you to inspect.