08-04-2015 01:38 PM
I have a csv file "abc.csv", it has 5000 variables (var1-var5000, column1 through column5000), and nearly million rows. However I am only interested in var1, var1390,var3782, and var4703. I know I can do it using following code:
filename myfile "abc.csv";
data want(keep=var1 var1390 var3782 var4703);
However the code do waste time inputing those un-wanted columns.
My question is: is there a way to input only wanted columns, i.e. var1, var1390,var3782, and var4703?
08-04-2015 02:29 PM
It is probably not worth the effort. Most of the time will be spent waiting for the disk to move the file into memory. The process of reading in the text version of 5000 variables would be small next to that.
08-04-2015 04:27 PM
IF the file were fixed column then it would be worth determining the columns and reading just those but CSV is going to yield no joy.
Though I would only input variables Var1-var4703;
I suspect you'll need a large value for lrecl as well.
08-04-2015 04:38 PM
If you think it is worth the effort then look into some the unix based tools for subsetting CSV files.
You could wrap your solution inside of a PIPE and read it in SAS.
filename myfile pipe 'csvfilter -f 1,1390,3782,4703 abc.csv' ;
data want ;
infile myfile dsd truncover firstobs=2;
input var1 var1390 var3782 var4703;