BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
_maldini_
Barite | Level 11

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.

 

I am trying to merge `14 datasets (2002 - 2015) w/ a combined number of observations that exceeds 5 million. I am using SAS University Edition on a Mac OS X w/ 70 GB of free storage.

 

Previous posts on this topic have not solved the problem. 

 

How much storage space is needed? Is 70 GB not enough?!

 

Thanks.

 

Full log: 


1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
55
56 DATA neiss_subset_1;
57 MERGE neiss_merge WORK.NEISS_2002_2015;
58 BY cpsc_case_no trmt_date;
59 RUN;

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 896761 observations read from the data set WORK.NEISS_MERGE.
NOTE: There were 1 observations read from the data set WORK.NEISS_2002_2015.
WARNING: The data set WORK.NEISS_SUBSET_1 may be incomplete. When this step was stopped there were 896760 observations and 28
variables.
WARNING: Data set WORK.NEISS_SUBSET_1 was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
real time 0.90 seconds
cpu time 0.89 seconds


60
61 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
73

 

 

1 ACCEPTED SOLUTION

Accepted Solutions
Ksharp
Super User

You should not put those tables in WORK library, put them in Share Fold.And also make it as WORK library.

 

libname xx v9 '/folders/myfolders/';
options user=xx;
data NEISS_SUBSET_1;
merge NEISS_SUBSET_1 have;
.....

View solution in original post

7 REPLIES 7
ChrisNZ
Tourmaline | Level 20

It depends on how you merge.

If you do a cartesian product on 14 tables, I suspect 70 *TB* would not be enough.

Ksharp
Super User

You should not put those tables in WORK library, put them in Share Fold.And also make it as WORK library.

 

libname xx v9 '/folders/myfolders/';
options user=xx;
data NEISS_SUBSET_1;
merge NEISS_SUBSET_1 have;
.....
rogerjdeangelis
Barite | Level 11

3TB drives are running at about $90 which means that 70gb is costs about $2.5. You may want to increase tour investment in disk space.

 

However you do have to determine the nature of your joins, hopefully you are not doing I lot of cartesian joins.

 

_maldini_
Barite | Level 11

@Ksharp I'm not sure I follow. Could you please elaborate? Thanks!

Ksharp
Super User

I mean take 

'/folders/myfolders/'

as WORK library. All the table will appear in /folders/myfolders/ 

ballardw
Super User

Please show the log for that code with the error messages.

The fewer the details you provide the more guesses or questions you get before any resolution.

_maldini_
Barite | Level 11

 
1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
55
56 DATA neiss_subset_1;
57 MERGE neiss_merge WORK.NEISS_2002_2015;
58 BY cpsc_case_no trmt_date;
59 RUN;

ERROR: Insufficient space in file WORK.NEISS_SUBSET_1.DATA.
ERROR: File WORK.NEISS_SUBSET_1.DATA is damaged. I/O processing did not complete.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: There were 896761 observations read from the data set WORK.NEISS_MERGE.
NOTE: There were 1 observations read from the data set WORK.NEISS_2002_2015.
WARNING: The data set WORK.NEISS_SUBSET_1 may be incomplete. When this step was stopped there were 896760 observations and 28
variables.
WARNING: Data set WORK.NEISS_SUBSET_1 was not replaced because this step was stopped.
NOTE: DATA statement used (Total process time):
real time 0.90 seconds
cpu time 0.89 seconds


60
61 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;
73

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 7 replies
  • 8964 views
  • 4 likes
  • 5 in conversation