ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

[_maldini_]

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.

I am trying to merge `14 datasets (2002 - 2015) w/ a combined number of observations that exceeds 5 million. I am using SAS University Edition on a Mac OS X w/ 70 GB of free storage.

Previous posts on this topic have not solved the problem. 

How much storage space is needed? Is 70 GB not enough?!

Thanks.

Full log: 

1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
55
56 DATA neiss_subset_1;
57 MERGE neiss_merge WORK.NEISS_2002_2015;
58 BY cpsc_case_no trmt_date;
59 RUN;

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 896761 observations read from the data set WORK.NEISS_MERGE.
NOTE: There were 1 observations read from the data set WORK.NEISS_2002_2015.
WARNING: The data set WORK.NEISS_SUBSET_1 may be incomplete. When this step was stopped there were 896760 observations and 28
variables.
WARNING: Data set WORK.NEISS_SUBSET_1 was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
real time 0.90 seconds
cpu time 0.89 seconds


60
61 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
73

[ChrisNZ]

It depends on how you merge.

If you do a cartesian product on 14 tables, I suspect 70 *TB* would not be enough.

[Ksharp · Accepted solution]

You should not put those tables in WORK library, put them in Share Fold.And also make it as WORK library.

libname xx v9 '/folders/myfolders/';
options user=xx;
data NEISS_SUBSET_1;
merge NEISS_SUBSET_1 have;
.....

[rogerjdeangelis]

3TB drives are running at about $90 which means that 70gb is costs about $2.5. You may want to increase tour investment in disk space.

However you do have to determine the nature of your joins, hopefully you are not doing I lot of cartesian joins.

[_maldini_]

@Ksharp I'm not sure I follow. Could you please elaborate? Thanks!

[Ksharp]

I mean take 

'/folders/myfolders/'

as WORK library. All the table will appear in /folders/myfolders/ 

[ballardw]

Please show the log for that code with the error messages.

The fewer the details you provide the more guesses or questions you get before any resolution.

[_maldini_]

 
1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
55
56 DATA neiss_subset_1;
57 MERGE neiss_merge WORK.NEISS_2002_2015;
58 BY cpsc_case_no trmt_date;
59 RUN;

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 896761 observations read from the data set WORK.NEISS_MERGE.
NOTE: There were 1 observations read from the data set WORK.NEISS_2002_2015.
WARNING: The data set WORK.NEISS_SUBSET_1 may be incomplete. When this step was stopped there were 896760 observations and 28
variables.
WARNING: Data set WORK.NEISS_SUBSET_1 was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
real time 0.90 seconds
cpu time 0.89 seconds


60
61 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
73