topic Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete. in SAS Programming

ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

_maldini_ — Tue, 25 Oct 2016 14:45:33 GMT

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.

I am trying to merge `14 datasets (2002 - 2015) w/ a combined number of observations that exceeds 5 million. I am using SAS University Edition on a Mac OS X w/ 70 GB of free storage.

Previous posts on this topic have not solved the problem.

How much storage space is needed? Is 70 GB not enough?!

Thanks.

Full log:

1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
55
56 DATA neiss_subset_1;
57 MERGE neiss_merge WORK.NEISS_2002_2015;
58 BY cpsc_case_no trmt_date;
59 RUN;

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 896761 observations read from the data set WORK.NEISS_MERGE.
NOTE: There were 1 observations read from the data set WORK.NEISS_2002_2015.
WARNING: The data set WORK.NEISS_SUBSET_1 may be incomplete. When this step was stopped there were 896760 observations and 28
variables.
WARNING: Data set WORK.NEISS_SUBSET_1 was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
real time 0.90 seconds
cpu time 0.89 seconds

60
61 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
73

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

ChrisNZ — Tue, 25 Oct 2016 02:33:28 GMT

It depends on how you merge.

If you do a cartesian product on 14 tables, I suspect 70 *TB* would not be enough.

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

Ksharp — Tue, 25 Oct 2016 05:19:54 GMT

You should not put those tables in WORK library, put them in Share Fold.And also make it as WORK library.

libname xx v9 '/folders/myfolders/';
options user=xx;
data NEISS_SUBSET_1;
merge NEISS_SUBSET_1 have;
.....

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

rogerjdeangelis — Tue, 25 Oct 2016 11:57:21 GMT

3TB drives are running at about $90 which means that 70gb is costs about $2.5. You may want to increase tour investment in disk space.

However you do have to determine the nature of your joins, hopefully you are not doing I lot of cartesian joins.

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

ballardw — Tue, 25 Oct 2016 13:56:48 GMT

Please show the log for that code with the error messages.

The fewer the details you provide the more guesses or questions you get before any resolution.

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

_maldini_ — Tue, 25 Oct 2016 14:45:09 GMT

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

_maldini_ — Tue, 25 Oct 2016 14:47:14 GMT

@Ksharp I'm not sure I follow. Could you please elaborate? Thanks!

Re: ERROR: Insufficient space in file. is damaged. I/O processing did not complete.

Ksharp — Wed, 26 Oct 2016 02:08:51 GMT

I mean take

'/folders/myfolders/'

as WORK library. All the table will appear in /folders/myfolders/