Insufficient Memory Error

[cstarkey]

Hello.  Thanks in advance to anyone who can help me.  I'm trying to run a psmatch procedure with pscores already in the dataset.  I can run a greedy match fine but if I try to do a varratio or mah distance I get the insufficient memory error.  I've tried different values of memsize, finally setting it to 0, but still get the error.  I have 32G of ram with large amounts of hd space for virtual memory.  The last run of varratio, kmin=1 kmax=2, returned the error with:

ERROR: The SAS System stopped processing this step because of insufficient memory.
WARNING: The data set WORK.MATCH_GREEDY_EVENT may be incomplete. When this step was stopped there were 0 observations and 159
variables.
NOTE: PROCEDURE PSMATCH used (Total process time):
real time 5:16.59
user cpu time 3:08.21
system cpu time 2:08.23
memory 7390631.12k
OS Memory 7410196.00k
Timestamp 12/31/2019 09:29:59 PM
Step Count 2 Switch Count 3

This is despite:

Group=MEMORY
SORTSIZE=1073741824
Specifies the amount of memory that is available to the SORT procedure.
SUMSIZE=0 Specifies a limit on the amount of memory that is available for data summarization procedures when class
variables are active.
MAXMEMQUERY=0 Specifies the maximum amount of memory that is allocated for procedures.
MEMBLKSZ=16777216 Specifies the memory block size for Windows memory-based libraries.
MEMMAXSZ=2147483648
Specifies the maximum amount of memory to allocate for using memory-based libraries.
LOADMEMSIZE=0 Specifies a suggested amount of memory that is needed for executable programs loaded by SAS.
MEMSIZE=22438694400
Specifies the limit on the amount of virtual memory that can be used during a SAS session.
REALMEMSIZE=0 Specifies the amount of real memory SAS can expect to allocate.
NOTE: PROCEDURE OPTIONS used (Total process time):
real time 0.00 seconds
user cpu time 0.00 seconds
system cpu time 0.00 seconds
memory 198.21k
OS Memory 16872.00k
Timestamp 12/31/2019 09:38:27 PM
Step Count 1 Switch Count 0

3
4 %put Free RAM = %sysfunc(putn(%sysfunc(getoption(xmrlmem))/1024**2,comma10.))MB;
Free RAM = 17,106MB

Again, this is with memsize set to 0.  I've even tried upping the amount of virtual memory so I had about 60G total but set it back to let windows manage it. 

If anyone can help with a suggestion, I'd really appreciate it.  Also, if someone can explain with fullstimer stops at 7G when there is still a lot of slack that would really help me understand how SAS and windows are managing my memory.

Thank you.

[SASKiwi]

I suggest SAS Tech Support is probably your best bet here. The PSMATCH procedure may have internal limitations regardless of the amount of memory you have. Have you tried reducing the number of rows to see if that helps?

[cstarkey]

Thanks for the suggestion.  I'll try reducing the amount of rows, though that doesn't really help the analysis long term.  There's about 242k rows now.  Still, it will give me a way to try and figure out where the problem is. 

Greedy match doesn't have a problem with the size so I'm at a loss why any other method stops so abruptly.  The thing that really confuses me is why the fullstimer stats don't reflect what my system has, in terms of memory.

[ballardw]

---

@cstarkey wrote:

Thanks for the suggestion.  I'll try reducing the amount of rows, though that doesn't really help the analysis long term.  There's about 242k rows now.  Still, it will give me a way to try and figure out where the problem is. 

Greedy match doesn't have a problem with the size so I'm at a loss why any other method stops so abruptly.  The thing that really confuses me is why the fullstimer stats don't reflect what my system has, in terms of memory.

---

Show the actual code used that generated the error.

Without actually knowing what/how you attempt to do something we are pretty much in the dark for likely places to look for issues.

[cstarkey]

Thanks for the response.  Here is the code for varratio, which doesn't run:

proc psmatch data=event region=treated;
class HB st;
psdata ps = ps_min treatvar=HB(treated='1');
match distance=ps method=varratio(kmin=1 kmax=2) exact=(st) caliper=.;
assess ps var=(A B C D E F) / plots(orient=vertical);
output out(obs=match)=match_varratio_event matchid=_MatchID;
run;

And the code for greedy, which runs fine:

proc psmatch data=event region=treated;
class HB st;
psdata ps = ps_min treatvar=HB(treated='1');
match distance=ps method=greedy(k=5) exact=(st) caliper=0.25;
assess ps var=(A B C D E F) / plots(orient=vertical);
output out(obs=match)=match_greedy_event matchid=_MatchID;
run;

Thank you.