WARNING: Out of memory.

[Jems]

Sas is giving 

WARNING: Out of memory.

ERROR: File WORK.COMB.DATA is damaged. I/O processing did not complete.

ERROR: File WORK.MUTNC_6.DATA is damaged. I/O processing did not complete.

ERROR: File WORK.'SASTMP-000000961'n.UTILITY is damaged. I/O processing did not complete.

[Kurt_Bremser · Accepted solution]

Please post the whole log of all failing steps, using the {i} button.

[Jems]

log
-----------------------------------------------------------------------------------------------------
NOTE: The data set WORK.MUTNC_6 has 1623160 observations and 83 variables.
 NOTE: Compressing data set WORK.MUTNC_6 decreased size by 99.32 percent. 
       Compressed is 11062 pages; un-compressed would require 1623160 pages.
 NOTE: DATA statement used (Total process time):
       real time           49.28 seconds
       cpu time            48.95 seconds
       
 
 WARNING: Out of memory.
 ERROR: File WORK.COMB.DATA is damaged. I/O processing did not complete.
 ERROR: File WORK.MUTNC_6.DATA is damaged. I/O processing did not complete.
 
 ERROR: File WORK.'SASTMP-000000961'n.UTILITY is damaged. I/O processing did not complete.
 NOTE: PROCEDURE APPEND used (Total process time):
       real time           0.00 seconds
       cpu time            0.00 seconds
       
 
 NOTE: The file WORK.MUTNC_6.DATA has been closed by the SASFILE statement.
 6
 102        %ty(ua=7);
 
 NOTE: The data set WORK.MUTNC_7 has 6724520 observations and 84 variables.
 NOTE: Compressing data set WORK.MUTNC_7 decreased size by 99.30 percent. 
       Compressed is 46924 pages; un-compressed would require 6724520 pages.
 NOTE: DATA statement used (Total process time):
       real time           3:29.58
       cpu time            3:29.63
       
 
 WARNING: Out of memory.
 ERROR: File WORK.COMB.DATA is damaged. I/O processing did not complete.
 ERROR: File WORK.MUTNC_7.DATA is damaged. I/O processing did not complete.
 
 ERROR: File WORK.'SASTMP-000000964'n.UTILITY is damaged. I/O processing did not complete.
 NOTE: PROCEDURE APPEND used (Total process time):
       real time           0.00 seconds
       cpu time            0.00 seconds
       

code
----------------------------------------------------------------------------------------------------%macro ty(ua=);

%let u=&ua.;
data mutnc_&ua;
  *array x[5] $3 ('ant' 'bee' 'cat' 'dog' 'ewe');
 array x[&mu_n] $500 (&mutatn.);

   array c[&mu_n] $500;
 * do u=1 to 5;
   array i[&u];
   n=dim(x);
   k=dim(i);
   i[1]=0;
   ncomb=comb(n,k);    /* The one extra call goes back */
   do j=1 to ncomb;  /* to the first combination. */
      call allcombi(n, k, of i[*], add, remove);
      do h=1 to k;
         c[h]=x[i[h]];
      end;
      *put @4 j= @10 'i= ' i[*] +3 'c= ' c[*] +3 add= remove=;
   output;

    
end;
run;

sasfile work.mutnc_&ua. open;
    proc append base=comb data=mutnc_&ua force;
    run;
sasfile work.mutnc_&ua. close;

*%put &u.;
%mend;

%ty(ua=1);
%ty(ua=2);
%ty(ua=3);
%ty(ua=4);
%ty(ua=5);
%ty(ua=6);
%ty(ua=7);
%ty(ua=8);
%ty(ua=9);
%ty(ua=10);
%ty(ua=11);
%ty(ua=12);
endsas;

[Kurt_Bremser]

Try it without the sasfile statements.

[Jems]

Thanks KurtBremser ..Now warning and error gone ..run time seems too long still running.

Any suggestions to reduce the running time ?

[FreelanceReinh]

---

@Jems wrote:

run time seems too long still running.

---

I'm not surprised. Your final dataset COMB would have more than 1.5E9 observations with >35000 bytes each, i.e. a file size of approx. 50 TB (uncompressed). What are you going to do with this monster? I guess something which results in a much smaller file, e.g. draw a sample from it or perform some aggregation. In this case I'd try to integrate that second step into the file creation process. Note that COMB would be full of redundancy and thus of limited value anyway.

[Jems]

Absolutely true,this is genetic data involving mutations and we are trying to find all the possible combinations.

[FreelanceReinh]

---

@Jems wrote:

Absolutely true,this is genetic data involving mutations and we are trying to find all the possible combinations.

---

Sure, but isn't everything you'll "find" (in that huge dataset) predictable? For example, there are about 1.1E9 different sequences of 15 letters from the set {A, C, G, T}, but I wouldn't expect too many new insights from a dataset containing all of them. Specific questions like "Is there a sequence that satisfies a certain set of conditions?" usually don't require a brute-force approach scanning all possible sequences.

[LinusH]

Another clue could be that (I belive) compress have no effect on in memory storage, see the concept of PDV.