topic Re: Is there a way to test if SASFILE will fail before trying to open data set larger than memory. in SAS Programming

Is there a way to test if SASFILE will fail before trying to open data set larger than memory.

RichardDeVen — Fri, 30 Sep 2022 09:51:45 GMT

I run code in a SAS Server that give the session 4GB of RAM (MEMSIZE=4,294,967,296).

A program is being developed that will act on a data set many times and the SASFILE statement is being used to speed up the repeated uses of the data.

SASFILE LIB.XYZ OPEN

The problem is that sometimes the data set is larger than 4GB.

Is there a reliable way to test if SASFILE will fail?

In exploring this question I made a sandbox session with far less memory (128MB) to see logging and macro variables in low memory situations

C:\Program Files\SASHome\SASFoundation\9.4\sas.exe -sasuser c:\temp\sasuser -memsize 128MB

and this program to create an outsized data set for the session and attempt to SASFILE it

data x;
  set sashelp.cars;
  do _n_ = 1 to 2500;
    output;
  end;
run;

sasfile work.x open;
%put _all_;
sasfile work.x close;

A first run of the code logs an out of memory

8    sasfile work.x open;
WARNING: Out of memory.
9    %put &=SYSWARNINGTEXT;
SYSWARNINGTEXT=Out of memory.
10   %put &=SYSERRORTEXT;
SYSERRORTEXT=
11   sasfile work.x close;
NOTE: The file WORK.X.DATA has been closed by the SASFILE statement.

The SYSWARNINGTEXT is just about the only useful info, however, there is no way to reset the warning text for later tests

If the warning goes unheeded and a DATA step involving the oversized file the system fails the step and limps along in a state I would not trust, such as continued steps can't continue because the LOG window is clogged (The LOG window must grab a fair amount of.RAM for itself)

Re: Is there a way to test if SASFILE will fail before trying to open data set larger than memory.

Oligolas — Fri, 30 Sep 2022 10:22:44 GMT

Hi,

none I know of, but a workaround could be to estimate the dataset size before a memory load?

Re: Is there a way to test if SASFILE will fail before trying to open data set larger than memory.

andreas_lds — Fri, 30 Sep 2022 10:55:07 GMT

sashelp.vtable has the size of all datasets available in the session. Try:

data _null_;
   set sashelp.vtable;
   where libname = 'LIB' and memname = 'XYZ';
   memsize = input(getOption('memsize'), best.);
   
   if filesize < memsize then do;
      call execute('sasfile lib.xyz open;');
   end;
run;

Re: Is there a way to test if SASFILE will fail before trying to open data set larger than memory.

Oligolas — Fri, 30 Sep 2022 15:26:39 GMT

N1ce that should do the trick 😊

Re: Is there a way to test if SASFILE will fail before trying to open data set larger than memory.

Quentin — Fri, 30 Sep 2022 15:50:50 GMT

It's hack, but while you can't write to SYSWARNINGTEXT, if you write a WARNING: note to the log, looks like SAS will update the value:

1    %put WARNING: Out of memory ;
WARNING: Out of memory
2    %put &=syswarningtext ;
SYSWARNINGTEXT=Out of memory
3
4    %put WARNING: %str( ) ;
WARNING:
5    %put &=syswarningtext ;
SYSWARNINGTEXT=