Is there a limit to the number of observations SAS can analyze?

mkit8 — Thu, 26 Mar 2020 06:23:25 GMT

Hello,

I have been running into issues with SAS completely freezing when I try to run proc lifetest. When I set the number of observations to 100,000 it runs just fine. As I start increasing this number, SAS starts freezing, unfreezing, freezing, etc. When I go above 200,000 observations it freezing entirely. Is there a limit to the number of observations SAS can handle? I need to analyze roughly 600,000 observations. I am using SAS 94 32 through a web browser. I access it through an application gateway through my university.

filename in1 'S:\CRC\export3.txt';
libname save 'S:\CRC\';

proc format;
  value Patient_IDf
    23-77804110 = "* 00000023-77804110"
    ;
  value Age_recode_with_1_year_oldsf
	5-10 = '20-49'
	11-18 = '50+'
	0-4, 29 = 'other'
	;
  value Sexf
    1 = "Male"
    2 = "Female"
    ;
  value Race_recode_White_Black_Otherf
    1 = "White"
    2 = "Black"
    3 = "Other (American Indian/AK Native, Asian/Pacific Islander)"
    9 = "Unknown"
    ;
  value Summary_stage_2000_1998f
    0 = "In situ"
    1 = "Localized"
    2 = "Regional"
    7 = "Distant"
    8 = "N/A"
    9 = "Unknown/unstaged"
    14 = "Blank(s)"
    ;
  value Primary_Site_labeledf
	181 = 'Appendix'
	180, 182-184 = 'Proximal Colon'
	185-187 = 'Distal Colon'
	199,209 = 'Rectum'
	188-189 = 'Colon, NOS'
	0-179,210-809 = 'other'
    ;
  value Gradef
    1 = "Grade I"
    2 = "Grade II"
    3 = "Grade III"
    4 = "Grade IV"
    5 = "T-cell"
    6 = "B-cell; pre-B; B-precursor"
    7 = "Null cell; non T-non B"
    8 = "NK cell; natural killer cell (1995+)"
    9 = "Unknown"
    ;
  value Yost_quintile_registry_basedf
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    4 = "Group 4"
    5 = "Group 5"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Yost_tertile_registry_basedf
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Yost_quintilef
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    4 = "Group 4"
    5 = "Group 5"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Yost_tertilef
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Censusurbanareabasedcategorizaf
    1 = "All Urban"
    2 = "Mostly Urban"
    3 = "Mostly Rural"
    4 = "All Rural"
    7 = "Unable to Calculate"
    9 = "Tract in SEER, no match in Census data"
    ;
  value RUCAbasedcategorizationC2categf
    1 = "Urban"
    2 = "Rural"
    7 = "Unable to Calculate"
    8 = "Not coded"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Survival_monthsf
    0-203 = "* 0000-0203"
    9999 = "Unknown"
    ;
  value Vitalstatusrecodestudycutoffusf
    1 = "Alive"
    0 = "Dead"
    ;
run;

data save.new;
  /*NOTE: The data file was created using the Windows format line delimiter.*/
  /*The TERMSTR=CRLF input option for reading the file in UNIX, requires SAS version 9.*/
  infile in1 LRECL = 32000 delimiter = '09'X TERMSTR = CRLF;

  input Patient_ID
    Age_recode_with_1_year_olds
    Sex
    Race_recode_White_Black_Other
    Summary_stage_2000_1998
    Primary_Site_labeled
    Grade
    Yost_quintile_registry_based
    Yost_tertile_registry_based
    Yost_quintile
    Yost_tertile
    Censusurbanareabasedcategoriza
    RUCAbasedcategorizationC2categ
	Survival_months
    Vitalstatusrecodestudycutoffus
    ;
  label Patient_ID = "Patient ID"
    Age_recode_with_1_year_olds = "Age recode with <1 year olds"
    Sex = "Sex"
    Race_recode_White_Black_Other = "Race recode (White, Black, Other)"
    Summary_stage_2000_1998 = "Summary stage 2000 (1998+)"
    Primary_Site_labeled = "Primary Site - labeled"
    Grade = "Grade"
    Yost_quintile_registry_based = "Yost quintile (registry based)"
    Yost_tertile_registry_based = "Yost tertile (registry based)"
    Yost_quintile = "Yost quintile"
    Yost_tertile = "Yost tertile"
    Censusurbanareabasedcategoriza = "Census urban-area based categorization A"
    RUCAbasedcategorizationC2categ = "RUCA based categorization C (2 categorie"
	Survival_months = "Survival months"
    Vitalstatusrecodestudycutoffus = "Vital status recode (study cutoff used)"
    ;
  format Patient_ID Patient_IDf.
    Age_recode_with_1_year_olds Age_recode_with_1_year_oldsf.
    Sex Sexf.
    Race_recode_White_Black_Other Race_recode_White_Black_Otherf.
    Summary_stage_2000_1998 Summary_stage_2000_1998f.
    Primary_Site_labeled Primary_Site_labeledf.
    Grade Gradef.
    Yost_quintile_registry_based Yost_quintile_registry_basedf.
    Yost_tertile_registry_based Yost_tertile_registry_basedf.
    Yost_quintile Yost_quintilef.
    Yost_tertile Yost_tertilef.
    Censusurbanareabasedcategoriza Censusurbanareabasedcategorizaf.
    RUCAbasedcategorizationC2categ RUCAbasedcategorizationC2categf.
	Survival_months Survival_monthsf.
    Vitalstatusrecodestudycutoffus Vitalstatusrecodestudycutoffusf.
    ;
	run;
data save.new2;
  set save.new;
  if age_recode_with_1_year_olds in (0,1,2,3,4,29) then delete;
  else if 0<=primary_site_labeled<=179 then delete;
  else if 210<=primary_site_labeled<=809 then delete;
  else if Survival_months = 9999 then delete;
run;

proc freq data = save.new2 order=data;
	tables Age_recode_with_1_year_olds 
	Sex*Age_recode_with_1_year_olds
    Race_recode_White_Black_Other*Age_recode_with_1_year_olds
    Summary_stage_2000_1998*Age_recode_with_1_year_olds
    Primary_Site_labeled*Age_recode_with_1_year_olds
    Grade*Age_recode_with_1_year_olds
    Yost_quintile_registry_based*Age_recode_with_1_year_olds
    Yost_tertile_registry_based*Age_recode_with_1_year_olds
    Yost_quintile*Age_recode_with_1_year_olds
    Yost_tertile*Age_recode_with_1_year_olds
    Censusurbanareabasedcategoriza*Age_recode_with_1_year_olds
    RUCAbasedcategorizationC2categ*Age_recode_with_1_year_olds
    ;
	run;

proc lifetest data=save.new2(obs=1000);
strata Age_recode_with_1_year_olds;
time Survival_months*Vitalstatusrecodestudycutoffus(1);
run;

Re: Is there a limit to the number of observations SAS can analyze?

PaigeMiller — Thu, 26 Mar 2020 10:15:27 GMT

There should be no such limit built into the software. The limit is more than likely based on your hardware capabilities.

Re: Is there a limit to the number of observations SAS can analyze?

DavePrinsloo — Thu, 26 Mar 2020 10:16:49 GMT

I am not an expert with this proc. However, it is probably related to the amount of memory allocated to each user session.
You can find out how much is allocated to you by running
Proc options; run;
and look in the results for the option MEMSIZE.

You may need to have the SAS administrator to temporarily allocate you more memory.

Re: Is there a limit to the number of observations SAS can analyze?

Patrick — Thu, 26 Mar 2020 10:21:15 GMT

@mkit8

Please don't ask the same question in two places. That won't help anyone. I believe the the discussion here already moved on to more in-depth analysis.

Re: Is there a limit to the number of observations SAS can analyze?

mkit8 — Thu, 26 Mar 2020 16:32:08 GMT

Thank you, Dave.

When I run proc options, this is what it gives me (see below). Does this seem low? Any idea on how much memory I would need to run about 600,000 observations?

MEMMAXSZ=2147483648
Specifies the maximum amount of memory to allocate for using memory-based
libraries.
MEMSIZE=2147483648
Specifies the limit on the amount of virtual memory that can be used during
a SAS session.

Re: Is there a limit to the number of observations SAS can analyze?

DavePrinsloo — Fri, 27 Mar 2020 10:16:56 GMT

That is 2 GB. That's a pretty standard setting in multiuser environments. The server probably has something like 256 GB total, so 2GB each allows roughly 120 concurrent users. If the administrators gave everyone 4GB, then you would the server would be maxed with only 60 users.
But maybe there is a process at the university for you to justify why you need more space and they may allocate more for a week or so. I bet that if you were a professor researching for a Nobel prize, you would easily get a lot more. I have no idea if the memory usage increase is linear (in which case you would need 6 GB) or if it increases quicker. But I would apply for 8 GB as a start.

topic Is there a limit to the number of observations SAS can analyze? in Statistical Procedures

Is there a limit to the number of observations SAS can analyze?

Re: Is there a limit to the number of observations SAS can analyze?

Re: Is there a limit to the number of observations SAS can analyze?

Re: Is there a limit to the number of observations SAS can analyze?

Re: Is there a limit to the number of observations SAS can analyze?

Re: Is there a limit to the number of observations SAS can analyze?