BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
rykwong
Quartz | Level 8

Dear SAS community

I am trying to read in a huge txt file (70 GB) using this: but it keeps failing showing "disk full" (please see attached screenshot)Screen Shot 2017-06-08 at 8.18.36 AM.png

 

PROC IMPORT Out=work.gene1_2

DATAFILE

"\\.psf\Home\Dropbox (Partners HealthCare)\BWH Cardiac MR\Grants Application\PIZ\data_analysis\Results\Genotype\Kwong_plate1&2_052217_finalreport.txt"

DBMS=dlm REPLACE;

delimiter='09'X;

RUN;

 

 

any solution would be much appreciated.

 

thanks

Raymond

 

1 ACCEPTED SOLUTION

Accepted Solutions
RW9
Diamond | Level 26 RW9
Diamond | Level 26

Get a bigger hard drive, or allocate more resources to your SAS system.  A large file of that size may well exceed the amount of disk space needed.  Speak with your IT group, not much else I can say really.

Oh, actually, you may want to take the datastep code generated by SAS (in the log) from the proc import and modify it to meet the dataset structure, this is far better than letting proc import guess, and then you have to go in and change a load of things - especially with this amount of data.

View solution in original post

5 REPLIES 5
RW9
Diamond | Level 26 RW9
Diamond | Level 26

Get a bigger hard drive, or allocate more resources to your SAS system.  A large file of that size may well exceed the amount of disk space needed.  Speak with your IT group, not much else I can say really.

Oh, actually, you may want to take the datastep code generated by SAS (in the log) from the proc import and modify it to meet the dataset structure, this is far better than letting proc import guess, and then you have to go in and change a load of things - especially with this amount of data.

rykwong
Quartz | Level 8

sorry could you be more specific to "modify it to meet the dataset structure"?

 

thanks 

 

Kurt_Bremser
Super User

When you read a text file with proc import, it generates a data step and runs it. That datastep can be found in the log (and copied from there).

That data step will tell us a lot about the structure of the resulting dataset, and maybe there are some options that might let you work around your resource problem.

 

RW9
Diamond | Level 26 RW9
Diamond | Level 26

So, proc import should generate some code in the log that looks something like:

data want;
  infile "your data file.txt"...;
  length...;
  informat ...;
  input ...;
run;

This is the code that actually runs, proc import merely scans your data and guesses the best informats/lengths etc. and then generates this code.  Editing this yourself is better as you know the data better.  

Also note that using proc import has overheads of its own, it needs to read in a sample of your data and do processing over it to guess what the data structure is, therefore another good reason to drop the proc import and write the datastep directly.

 

Ksharp
Super User

Do not use WORK library. Try other.

 

libname x v9 'd:\temp\';

PROC IMPORT Out=X.gene1_2

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 5 replies
  • 1447 views
  • 1 like
  • 4 in conversation