topic ERROR: Insufficient space in file WORK in SAS Studio

ERROR: Insufficient space in file WORK

HSpark — Tue, 24 Jan 2017 03:02:03 GMT

Hi.

I am using SAS University Edition in Windows 10 environment with VM Ware Workstation Player 12. I am now doing logistic regression and propensity score matching. My data consist of 1.5 million observations and 163 variables. The size of data file is about 450 M. Command consists of several macros and 250 lines. When I run the program, it stops and the message pops up.

ERROR: Insufficient space in file WORK.BIRTH.DATA.

ERROR: File WORK.BIRTH.DATA is damaged. I/O processing did not complete.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: There were 1579785 observations read from the data set WORK.BIRTH.

WARNING: The data set WORK.BIRTH may be incomplete. When this step was stopped there were 1579785 observations and 167 variables.

WARNING: Data set WORK.BIRTH was not replaced because this step was stopped.

NOTE: DATA statement used (Total process time):

real time 5.18 seconds

cpu time 2.08 seconds

Part of the code is like this.

PROC IMPORT file='/folders/myfolders/epi271/natl2015_ver11_1_2.dta'
   DBMS=stata
   OUT=WORK.birth;

RUN;
data work.birth;
   set work.birth;

   newmager9=.;

   if mager9=1 then
       newmager9=2;

   if mager9 ne 1 then
       newmager9=mager9;
   id=_N_;
run;

proc logistic descending;
   class newmager9 morace newmar newmeduc para_cat overtdm chrhtn ptdhx cesar_cat;
   model inftr=newmager9 morace newmar newmeduc para_cat overtdm chrhtn ptdhx
       cesar_cat;
   output out=psmodel pred=propscore;
run;

****************************************************************************************************************************************/
data caco1;
   set psmodel;
   propint=propscore;

   if inftr=1 then
       case=1;

    if inftr=0 then
       control=1;


run;

proc sort ;
   by id;
   *id can be modified as wanted;
run;

********* MACRO FOR MATCHING BASED ON PROPENSITY SCORE ******************************************;

%macro caco(s, x, m, l, gd);
   data t3a(keep=id1 propsco1) t3b(keep=id2 propsco2);
       set caco1;

------------------------------------------------------------------------

This message shows even before logistic regression starts. I cannot understand this situation. Image file is the disk setting of the virtual machine.

Is there any way to get around it? Please let me know.

Thank you.

Re: ERROR: Insufficient space in file WORK

Reeza — Tue, 24 Jan 2017 04:52:18 GMT

This may be too big for SAS UE. The original intention was for learning purposes only and there are restrictions built into the software. I would contact SAS Tech Support directly for this and see if there's any work around. Because of the limitations of the software, one way to get around is to divide the data into smaller portions and process individually, if possible.

Re: ERROR: Insufficient space in file WORK

LinusH — Tue, 24 Jan 2017 08:47:58 GMT

Need to chime in with @Reeza: this data set is obviously larger than required for self paced learning purposes. You should be able to use a subset of your input data, and still be able to perform your training.

A thing you could do is to remove any work data set is not needed in your process as soon as it has played out its role.

Re: ERROR: Insufficient space in file WORK

Kurt_Bremser — Tue, 24 Jan 2017 09:29:55 GMT

Although the data file size is "only" 450 MB, this does not mean that a SAS dataset won't be considerably larger.

Let's make a quick calculation:

Assume your variables were all numeric (8 bytes), then a single observation would consume

163 * 8 = 1,304 bytes

Multiply by 1.5 million

1,304 * 1,500,000 = 1,956,000,000 bytes

equates to 1,910,156.25 Kbytes

equates to 1,865.387 Mbytes

equates to 1.8217 GB

Depending on the structure (char variables might be _considerably_ longer), the dataset might be much bigger than that.

UE is simply not designed for this quantity structure.

Inspect your data on the Stata side and reduce the number of observations.

Use the compress=yes option to minimize disk storage consumption.

Re: ERROR: Insufficient space in file WORK

HSpark — Wed, 25 Jan 2017 01:13:56 GMT

Thank you for kind explanation. I will try SAS 9.4 at the school lab. Thank you.