Why will my proc lifetest code not work?

[mkit8]

Hello,

This is my first time using proc lifetest, and I can't get it to work. Every time I submit my code, SAS just spins and spins and then ultimately freezes (I've tried about 15 times). When I delete my proc lifetest statement, the proc freq statement above it works just fine, so I believe the issue is in the proc lifetest statement.

I am trying to have SAS compare the survival experiences of my two age groups (20-49, 50+).

filename in1 'S:\CRC\export3.txt';
libname save 'S:\CRC\';

proc format;
  value Patient_IDf
    23-77804110 = "* 00000023-77804110"
    ;
  value Age_recode_with_1_year_oldsf
	5-10 = '20-49'
	11-18 = '50+'
	0-4, 29 = 'other'
	;
  value Sexf
    1 = "Male"
    2 = "Female"
    ;
  value Race_recode_White_Black_Otherf
    1 = "White"
    2 = "Black"
    3 = "Other (American Indian/AK Native, Asian/Pacific Islander)"
    9 = "Unknown"
    ;
  value Summary_stage_2000_1998f
    0 = "In situ"
    1 = "Localized"
    2 = "Regional"
    7 = "Distant"
    8 = "N/A"
    9 = "Unknown/unstaged"
    14 = "Blank(s)"
    ;
  value Primary_Site_labeledf
	181 = 'Appendix'
	180, 182-184 = 'Proximal Colon'
	185-187 = 'Distal Colon'
	199,209 = 'Rectum'
	188-189 = 'Colon, NOS'
	0-179,210-809 = 'other'
    ;
  value Gradef
    1 = "Grade I"
    2 = "Grade II"
    3 = "Grade III"
    4 = "Grade IV"
    5 = "T-cell"
    6 = "B-cell; pre-B; B-precursor"
    7 = "Null cell; non T-non B"
    8 = "NK cell; natural killer cell (1995+)"
    9 = "Unknown"
    ;
  value Yost_quintile_registry_basedf
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    4 = "Group 4"
    5 = "Group 5"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Yost_tertile_registry_basedf
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Yost_quintilef
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    4 = "Group 4"
    5 = "Group 5"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Yost_tertilef
    1 = "Group 1"
    2 = "Group 2"
    3 = "Group 3"
    8 = "Missing (unable to calculate)"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Censusurbanareabasedcategorizaf
    1 = "All Urban"
    2 = "Mostly Urban"
    3 = "Mostly Rural"
    4 = "All Rural"
    7 = "Unable to Calculate"
    9 = "Tract in SEER, no match in Census data"
    ;
  value RUCAbasedcategorizationC2categf
    1 = "Urban"
    2 = "Rural"
    7 = "Unable to Calculate"
    8 = "Not coded"
    9 = "Tract in SEER, no match in Census data"
    ;
  value Survival_monthsf
    0-203 = "* 0000-0203"
    9999 = "Unknown"
    ;
  value Vitalstatusrecodestudycutoffusf
    1 = "Alive"
    0 = "Dead"
    ;
run;

data save.new;
  /*NOTE: The data file was created using the Windows format line delimiter.*/
  /*The TERMSTR=CRLF input option for reading the file in UNIX, requires SAS version 9.*/
  infile in1 LRECL = 32000 delimiter = '09'X TERMSTR = CRLF;

  input Patient_ID
    Age_recode_with_1_year_olds
    Sex
    Race_recode_White_Black_Other
    Summary_stage_2000_1998
    Primary_Site_labeled
    Grade
    Yost_quintile_registry_based
    Yost_tertile_registry_based
    Yost_quintile
    Yost_tertile
    Censusurbanareabasedcategoriza
    RUCAbasedcategorizationC2categ
	Survival_months
    Vitalstatusrecodestudycutoffus
    ;
  label Patient_ID = "Patient ID"
    Age_recode_with_1_year_olds = "Age recode with <1 year olds"
    Sex = "Sex"
    Race_recode_White_Black_Other = "Race recode (White, Black, Other)"
    Summary_stage_2000_1998 = "Summary stage 2000 (1998+)"
    Primary_Site_labeled = "Primary Site - labeled"
    Grade = "Grade"
    Yost_quintile_registry_based = "Yost quintile (registry based)"
    Yost_tertile_registry_based = "Yost tertile (registry based)"
    Yost_quintile = "Yost quintile"
    Yost_tertile = "Yost tertile"
    Censusurbanareabasedcategoriza = "Census urban-area based categorization A"
    RUCAbasedcategorizationC2categ = "RUCA based categorization C (2 categorie"
	Survival_months = "Survival months"
    Vitalstatusrecodestudycutoffus = "Vital status recode (study cutoff used)"
    ;
  format Patient_ID Patient_IDf.
    Age_recode_with_1_year_olds Age_recode_with_1_year_oldsf.
    Sex Sexf.
    Race_recode_White_Black_Other Race_recode_White_Black_Otherf.
    Summary_stage_2000_1998 Summary_stage_2000_1998f.
    Primary_Site_labeled Primary_Site_labeledf.
    Grade Gradef.
    Yost_quintile_registry_based Yost_quintile_registry_basedf.
    Yost_tertile_registry_based Yost_tertile_registry_basedf.
    Yost_quintile Yost_quintilef.
    Yost_tertile Yost_tertilef.
    Censusurbanareabasedcategoriza Censusurbanareabasedcategorizaf.
    RUCAbasedcategorizationC2categ RUCAbasedcategorizationC2categf.
	Survival_months Survival_monthsf.
    Vitalstatusrecodestudycutoffus Vitalstatusrecodestudycutoffusf.
    ;
	run;
data save.new2;
  set save.new;
  if age_recode_with_1_year_olds in (0,1,2,3,4,29) then delete;
  else if 0<=primary_site_labeled<=179 then delete;
  else if 210<=primary_site_labeled<=809 then delete;
run;

proc freq data = save.new2 order=data;
	tables Age_recode_with_1_year_olds 
	Sex*Age_recode_with_1_year_olds
    Race_recode_White_Black_Other*Age_recode_with_1_year_olds
    Summary_stage_2000_1998*Age_recode_with_1_year_olds
    Primary_Site_labeled*Age_recode_with_1_year_olds
    Grade*Age_recode_with_1_year_olds
    Yost_quintile_registry_based*Age_recode_with_1_year_olds
    Yost_tertile_registry_based*Age_recode_with_1_year_olds
    Yost_quintile*Age_recode_with_1_year_olds
    Yost_tertile*Age_recode_with_1_year_olds
    Censusurbanareabasedcategoriza*Age_recode_with_1_year_olds
    RUCAbasedcategorizationC2categ*Age_recode_with_1_year_olds
    ;
	run;

proc lifetest data=save.new2 atrisk plots = survival (atrisk cb)outs=save.new2out;
strata Age_recode_with_1_year_olds;
time Survival_months*Vitalstatusrecodestudycutoffus(1);
run; 

[SAS_Cares]

Hi @mkit8 ! Thank you for using and posting your query through our SAS Communities. To understand further your concern, what SAS software version and/or product release are you using? There are certain SAS software for which certain Procedures or PROC are not executable. However, looking at code that you shared here, typically when you use the PROC Lifetest, you need to check whether your ODS Graphic is on. you may refer to this documentation on PROC LifeTest should you need further understanding or guidance on its syntax. Otherwise, if you want, I can assist you by raising a technical support track for your concern. How's that sound?

[mkit8]

Hello,

I believe I am using SAS 94 32 through a web browser. I access it through an application gateway through my university.

I did some reading after what you shared and I see that when I go to Tools>Options>Preferences>Results that the "Use ODS Graphics" is checked off. 

Would this point to there being a different issue with my code then?

Thanks,

Melody

[Tom]

If you are accessing through a web browser does that mean you are using SAS/Studio?  Your system admin has probably put limits on how long your session and remain "idle".  They might also have configured the SAS application server to run with less memory than lifetest needs for your problem.  Ask them if you can run the job as a stand alone batch/background job instead of using the pseudo interactive web interface. 

[mkit8]

Hello, 

It appears that the issue with SAS freezing is due to the number of observations. When I changed my proc lifetest code to this (see below), it successfully ran the program when the observations were small. As I gradually increased the number of observations, SAS started freezing, unfreezing, freezing, etc. Once I went above 200,000 observations, it stopped working altogether. Do you have any idea why this would be? Is there a limit to the number of observations SAS can handle? I need to analyze roughly 600,000 observations. I am using SAS 94 32 through a web browser. I access it through an application gateway through my university.

proc lifetest data=save.new2(obs=200000);

strata Age_recode_with_1_year_olds;

time Survival_months*Vitalstatusrecodestudycutoffus(1);

run;

[ballardw]

Doesn't work is awful vague.

Are there errors in the log?: Post the code and log in a code box opened with the <> to maintain formatting of error messages.

No output? Post any log in a code box.

Unexpected output? Provide input data in the form of data step code pasted into a code box, the actual results and the expected results. Instructions here: https://communities.sas.com/t5/SAS-Communities-Library/How-to-create-a-data-step-version-of-your-dat... will show how to turn an existing SAS data set into data step code that can be pasted into a forum code box using the </> icon or attached as text to show exactly what you have and that we can test code against.

[mkit8]

Hello,

Thank you for your reply. I'm not sure if you saw the body of my message or just the subject line, but every time I submit my code, SAS just spins and spins and then ultimately freezes (I've tried about 15 times). I am unable to access any log or output because it freezes every time. When I delete my proc lifetest statement, the proc freq statement above it works just fine, so I believe the issue is in the proc lifetest statement.

Any ideas what might be wrong with my code?

Thanks,

Melody

[FreelanceReinh]

---

@mkit8 wrote:

This is my first time using proc lifetest, and I can't get it to work.

---

Hello @mkit8,

In this situation I would put my own code and data aside for a moment, start a second SAS session and try to reproduce one of the examples in the procedure documentation or even a simplified version thereof (i.e. without optional statements such as STRATA or advanced options such as plot requests).

As long as this is successful, I would (step by step) add/modify options and optional statements in order to make the example code more and more similar to my intended production code and run each intermediate code version.

If this still worked, I would run my own code on a reasonably small subset of the actual input dataset (e.g. save.new2(where=(enter suitable condition here)) or save.new2(obs=50) in your case), then increase the size/complexity of the subset.

Obviously, the point is to find out the last small change you made before the problem appeared.

[mkit8]

Hello,

Thank you for your response! I took your advice and tried to find the last small change I made before the problem appeared. When I change my proc lifetest code to this (see below) then it works. However, if I go above 200,000 observations, then it freezes. I notice that the bigger the number of observations, the more it freezes and unfreezes; however, past 200,000 is when it stops working altogether. Is there a limit to the number of observations SAS can handle? I'm using SAS 94 32 through a web browser. I access it through an application gateway through my university.  Unfortunately mine has roughly 600,000 observations.

proc lifetest data=save.new2(obs=200000);

strata Age_recode_with_1_year_olds;

time Survival_months*Vitalstatusrecodestudycutoffus(1);

run;

[FreelanceReinh]

Thanks for doing all these checks.

My next step would be to investigate where the bottleneck is (e.g. the amount of memory that is available to SAS). The PROC LIFETEST documentation contains a section Computer Resources where memory requirements are presented in detail. You can also use the FULLSTIMER system option (i.e., submit: options fullstimer;) to obtain a list of computer resources that were used for your PROC LIFETEST step (when using, say, 50,000, 100,000, 150,000, 200,000 observations; then try to extrapolate).

Once you have a rough estimate of the resources your full analysis would require (perhaps omitting the plots in order to save memory), you would need to contact the SAS (server) administrator of your university. An administrator would be able to modify certain system options (such as MEMSIZE) so that temporarily more resources are available to your SAS session. I would be surprised if a university had insufficient computer resources for an analysis of the size you describe.

If there was no way to complete the full analysis, I would resort to splitting the analysis dataset randomly (using PROC SURVEYSELECT) into, say, four subsets of about 150,000 observations and running the PROC LIFETEST step on each of the subsets. I would expect these four partial analyses to produce similar results. If all four log-rank tests (for example) indicate a statistically significant difference between the two survival curves (corresponding to the two strata), this will be strong evidence that the result of the full analysis (with its larger power) would be even more significant. Non-significant results of the partial analyses, however, would be inconclusive.

[Patrick]

200000 or 600000 rows are "nothing" and SAS scales normally really well. This doesn't mean that for the environment you're using the 600'000 rows are not too much and you're hitting some resource constraints.

Having said that: Is the process returning any big report or a large table which the browser based client tries to open automatically? That's what normally causes freezing issues for me besides of long running server side processes where the client stays for too long idle and then disconnects (there are easy fixes for both of these issues). My guess is that this is a client side issue.

Also: In case the help of the very knowledgeable guys here won't let you resolve the issue then take the offer from @SAS_Cares for support from SAS Inst.

[Doc_Duke]

PROC LIFETEST uses lots of memory.  Look for 'memory requirements' in the documentation

https://support.sas.com/documentation/onlinedoc/stat/131/lifetest.pdf