topic Re: MAC OS SAS university edition: Large Dataset and Insufficient space error in SAS Programming

MAC OS SAS university edition: Large Dataset and Insufficient space error

hopeverthino — Mon, 12 Nov 2018 21:46:54 GMT

Dear experts,

I am a MAC OS user of the SAS University Edition. I have a huge dataset with 8 variables and 89,600 rows. I need to conduct a nonparametric mixed model analysis with design of F1-LD-F3. When I entered the insufficient space error, I tried changing the work library into a different library (as follows), yet it didn't work; also, I tried to delete other datasets/codes/programs saved in the libraries/myfolders, not working either. I'm out of solutions. If you could advise me what to try the next, I'd really appreciate it. The change of work library code and mixed model codes are attached as follows for your information, as well as the error code and a sample of my data. Thanks again~

Change the library:

libname ABMDATA v9 '/folders/myfolders/';

optionsuser=ABMDATA;

Mixed Model Code:

proc mixed data=ABMDATA.RANKSUB method=mivque0 anovaf;

CLASS Countr_Type W_rs ProbSh_rs Time SUBJECT;

MODEL rank_Visits =Countr_Type|W_rs|ProbSh_rs|Time/CHISQ;

Repeated W_rs*ProbSh_rs*Time / SUB=SUBJECT TYPE=UN GRP=Countr_Type;

LSMEANS Countr_Type*W_rs*ProbSh_rs*Time;

run;

Error Log:

1 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;

73 libname ABMDATA v9 '/folders/myfolders/';

NOTE: Libref ABMDATA refers to the same physical library as USER.

NOTE: Libref ABMDATA was successfully assigned as follows:

Engine: V9

Physical Name: /folders/myfolders

74 options user=ABMDATA;

76 proc mixed data=ABMDATA.RANKSUB method=mivque0 anovaf;

77 CLASS Countr_Type W_rs ProbSh_rs Time SUBJECT;

78 MODEL rank_Visits = Countr_Type| W_rs| ProbSh_rs| Time/CHISQ;

79 Repeated W_rs*ProbSh_rs*Time / SUB=SUBJECT TYPE=UN GRP=Countr_Type;

80 LSMEANS Countr_Type*W_rs*ProbSh_rs*Time;

81 run;

ERROR:

ERROR: File WORK.UTDATA.UTILITY is damaged. I/O processing did not complete.

NOTE: The SAS System stopped processing this step because of errors.

NOTE: PROCEDURE MIXED used (Total process time):

real time 28.62 seconds

cpu time 20.60 seconds

83 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;

A sample of my data:

Runs	Region	Countr_Type	W_rs	ProbU_rs	ProbSh_rs	Time	Visits
1	SouthEu	HpHd	LrwLsw	LruLsu	LrsLss	1	0
1	SouthEu	HpHd	LrwLsw	LruLsu	LrsLss	2	0
1	SouthEu	HpHd	LrwLsw	LruLsu	LrsLss	3	0.08333333
1	SouthEu	HpHd	LrwLsw	LruLsu	LrsLss	4	0.16666667

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

ChrisNZ — Mon, 12 Nov 2018 22:01:53 GMT

The error

ERROR: File WORK.UTDATA.UTILITY is damaged.

can point to a hardware issue with your disk.

SAS wrote a utility file and could not read it successfully.

This is usually because the medium is damaged, or because SAS ran out of space when trying to write the file.

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

Reeza — Mon, 12 Nov 2018 22:15:46 GMT

@hopeverthino wrote:

Dear experts,

I am a MAC OS user of the SAS University Edition. I have a huge dataset with 8 variables and 89,600 rows. I need to conduct a nonparametric mixed model analysis with design of F1-LD-F3. When I entered the insufficient space error, I tried changing the work library into a different library (as follows), yet it didn't work; also, I tried to delete other datasets/codes/programs saved in the libraries/myfolders, not working either. I'm out of solutions. If you could advise me what to try the next, I'd really appreciate it. The change of work library code and mixed model codes are attached as follows for your information, as well as the error code and a sample of my data. Thanks again~

That is not a huge data set by any standards and should be handled within SAS UE as long as your model is specified correctly.

Have you worked with larger data sets and had SAS crash? If so, SAS UE does not do a good job of cleaning up after itself and cleaning it up after the fact is almost impossible. Sadly the easiest solution is to delete the VM and re-add it, essentially reinstalling it. For a data set of this size you shouldn't have issues. Otherwise, I've posted many times (including in last 24 hours) in the Analytics U forum of how to get around this and you can search there for my recommendations (user library and change RAM settings). DO NOT delete your my folders and then you can just reconnect to it. If you have anything in your auto exec ensure you copy those to re-run them as well.

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

hopeverthino — Tue, 13 Nov 2018 00:22:08 GMT

Thank you, Reeza, for the detailed recommendation of possible solutions. I checked your earlier recommendations regarding enhancing RAM for VM and redirecting the library. I tried both, e.g., enhancing RAM for VM into 5GB, the allowed maximum. Yet the insufficient space error keeps popping up. I also tried to uninstall VM and then reinstall VM and SAS all over again. Yet the error persists. Is there any chance you still have any alternative actions for me to try?
Also, my apologies as I found my data is actually 860,000 rows instead of 86,000 rows. Not sure whether that'd make a difference.

Thanks again and looking forward to any further feedback you may have on this matter~

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

hopeverthino — Tue, 13 Nov 2018 00:23:40 GMT

Thank you for the feedback, ChrisNZ. Would there be anything that I can do to resolve this issue you mentioned?

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

ChrisNZ — Tue, 13 Nov 2018 01:44:38 GMT

I do not see an "insufficient space error" in what you posted.

To resolve the damaged file error:

- Rerun the process just in case this was an unlucky one-off

- If not then make sure there is enough space on the disk. See @Reeza'a post about cleaning up.

Monitor how full your VM disk becomes.

- If the disk is never full, then do a scan for hardware defects of the disk.

- You could also try redirecting the utility files to another disk.

- You could also try turning off multi-threading to see if this reduces the use of utility files

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

hopeverthino — Tue, 13 Nov 2018 17:55:11 GMT

Many thanks for the detailed recommendations, ChrisNZ. My apologies as the error on insufficient space for WORK.UTDATA.UTILITY somehow was not pasted in the question. But this error was right before the error on the damaged file error.

I also tried your recommended list of possible fixes, including closing other tasks and cleaning up the VM memory with reinstallation, etc. Only not sure how to redirect the utility files to another disk as I'm using Mac Pro and it seems to have only one disk...

Not sure whether it is because the data is too long with almost 900,000 rows. Am seriously considering installing the SAS9.4 version in the VMware Fusion/Parallel. How likely do you think that'd resolve the issue? Thanks again for the valuable feedback and any further assistance you could offer~

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

Reeza — Tue, 13 Nov 2018 19:10:37 GMT

You shouldn't have any issues, and the work files are on the VM. The suggestionss I've posted will redirect your work library via setting up a USER library usually gets rid of space issues but even if the PROCs you're using are in memory, with 1 million rows and 5GB it shouldn't be a problem. Something else is the issue here. You can test this easily by using Academics on Demand instead (also free) to see if it works there or not. The RAM doesn't have a maximum limit, or if it does, it's based on your system limitations, not SAS UE. I usually use 2GB with no issue on a 16GB RAM + MacBook Pro.

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

luhawkyesag — Sun, 20 Jan 2019 04:13:45 GMT

Thank you so much for the detailed solution. Sorry that it took me so long to try it and respond. It still cannot be processed. And here attaches the report of errors. I guess it is just too huge for consider the levels of repeated factor to be analyzed. Thank you no matter what for the offering of possible solutions.

NOTE: There were 896282 observations read from the data set ABMDATA.RANK.

NOTE: The data set ABMDATA.SUB has 896282 observations and 10 variables.

NOTE: DATA statement used (Total process time):

real time 0.65 seconds

user cpu time 0.42 seconds

system cpu time 0.07 seconds

memory 3721.75k

OS Memory 35008.00k

Timestamp 01/20/2019 03:26:30 AM

Step Count 226 Switch Count 5

Page Faults 0

Page Reclaims 644

Page Swaps 0

Voluntary Context Switches 2455

Involuntary Context Switches 77

Block Input Operations 126208

Block Output Operations 140552

246

247 proc print data=ABMDATA.SUB(obs=10);

248 title "ABMDATA.RANK WITH SUBJECT";

249 run;

NOTE: There were 10 observations read from the data set ABMDATA.SUB.

NOTE: PROCEDURE PRINT used (Total process time):

real time 0.04 seconds

user cpu time 0.04 seconds

system cpu time 0.01 seconds

memory 2056.96k

OS Memory 32680.00k

Timestamp 01/20/2019 03:26:30 AM

Step Count 227 Switch Count 1

Page Faults 0

Page Reclaims 289

Page Swaps 0

Voluntary Context Switches 14

Involuntary Context Switches 0

Block Input Operations 288

Block Output Operations 32

250

251

252 proc mixed data=ABMDATA.SUB method=mivque0 anovaf;

253 CLASS CT RSUW RSUP Time SUBJECT;

254 MODEL rank_Visits = CT| RSUW| RSUP| Time/CHISQ;

255 Repeated RSUW*RSUP*Time / SUB=SUBJECT TYPE=UN GRP=CT;

256 LSMEANS CT*RSUW*RSUP*Time;

257 run;

NOTE: 282 observations are not included because of missing values.

WARNING: ODS graphics with more than 5000 points have been suppressed. Use the PLOTS(MAXPOINTS= ) option in the PROC MIXED

statement to change or override the cutoff.

ERROR: The SAS System stopped processing this step because of insufficient memory.

NOTE: PROCEDURE MIXED used (Total process time):

real time 23:52.85

user cpu time 2:17.13

system cpu time 1:19.86

memory 1285181.90k

OS Memory 1317204.00k

Timestamp 01/20/2019 03:50:23 AM

Step Count 228 Switch Count 38

Page Faults 3

Page Reclaims 325189

Page Swaps 0

Voluntary Context Switches 157532

Involuntary Context Switches 240

Block Input Operations 151866504

Block Output Operations 343905064

258

259

260 OPTIONS NONOTES NOSTIMER NOSOURCE NOSYNTAXCHECK;

272

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

Reeza — Sun, 20 Jan 2019 04:45:04 GMT

If your data is not confidential you can try and do it on SAS Academics on Demand. Or if you're a student many universities have access to the full version for free or a low price ($30).

Re: MAC OS SAS university edition: Large Dataset and Insufficient space error

luhawkyesag — Sun, 20 Jan 2019 04:50:25 GMT

Thank you! Acturally I just tried on SAS academics on demand online and got the error message of insufficient memory... Thanks again for your instant feedback~ I wonder whether it'd may be because that my within-subject factor of Time has too many levels (>100) hence leads to the impossibility to calculate the three-factor interactions.