topic ERROR: Insufficient space in file WORK. in SAS Programming

ERROR: Insufficient space in file WORK.

DougZ — Mon, 16 Nov 2020 03:07:40 GMT

I have 4.6 TB free in my Linux server's filesystem that is used for SASWORK, SAS9.4TSM6 installed, and I tried sorting a char-compressed SAS file that is 20GB into a work dataset. Here's a section of the log with fullstimer option turned on. Why would i get this error and what is the solution?

MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): **********************************************;
MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): * Identify rows that are in one file of the ;
MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): * basefile-compfile pair only ;
MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): **********************************************;
MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): proc sort data=baselib.me_mart_202002 out=work.basefile;
MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): by spm_loan_key;
MPRINT(LOOP_THROUGH_MONTHS.COMPARE_BASEFILE_TO_COMPFILE): run;

NOTE: There were 9760360 observations read from the data set BASELIB.ME_MART_202002.
ERROR: Insufficient space in file WORK.BASEFILE.DATA.
ERROR: Failure while merging sorted runs from utility file 1 to final output.
ERROR: Failure encountered during external sort.
ERROR: File WORK.BASEFILE.DATA is damaged. I/O processing did not complete.
NOTE: The SAS System stopped processing this step because of errors.
NOTE: Due to ERROR(s) above, SAS set option OBS=0, enabling syntax check mode. This prevents execution of subsequent data modification statements.
WARNING: The data set WORK.BASEFILE may be incomplete. When this step was stopped there were 5945247 observations and 624 variables.
NOTE: PROCEDURE SORT used (Total process time):
real time 7:17.58
user cpu time 1:34.11
system cpu time 1:39.85
memory 1056692.82k
OS Memory 1065912.00k
Timestamp 11/15/2020 07:42:42 PM
Step Count 1 Switch Count 7
Page Faults 7
Page Reclaims 3006747
Page Swaps 0
Voluntary Context Switches 1596237
Involuntary Context Switches 282
Block Input Operations 1280
Block Output Operations 148918416

Re: ERROR: Insufficient space in file WORK.

SASKiwi — Mon, 16 Nov 2020 03:21:04 GMT

Try compressing the WORK dataset as well. Unless you have COMPRESS = YES as a session option WORK datasets will not be compressed.

out=work.basefile (compress = yes)

SAS utility files are not compressed so the SORT could be using more space than you think. Also try re-running the program while monitoring your WORK folder usage to see how much space really gets used.

Re: ERROR: Insufficient space in file WORK.

Kurt_Bremser — Mon, 16 Nov 2020 05:55:15 GMT

The fact that the filesystem has a large amount of free space does not automatically mean that a single user has all that space available. A diligent admin will always set user quotas on shared resources where end users have write permission, so a single user can't occupy the whole filesystem.

So you need to talk to your admin to determine what is available for you.

Run a PROC CONTENTS on your source dataset to see if it is compressed. As a test, run this:

data test (compress=yes);
set baselib.me_mart_202002 (obs=1000000);
run;

and look at the compression rate reported in the log. If it is substantial, try using the TAGSORT option in PROC SORT. It will reduce the size of the utility file (utility files are uncompressed!), albeit at a performance cost.

Since the utility file is stored in WORK (unless a special configuration - UTILLOC - is made), it can eat a large part of your quota.