topic Re: How does Proc SurveySelect Deal with Missing values? in Statistical Procedures

How does Proc SurveySelect Deal with Missing values?

Daisy2 — Fri, 09 Oct 2020 21:21:05 GMT

I'm trying to split my data into Train (70%) and Test data sets. My file has 508 rows, but the column I want to split only has about 350 cells filled. The rest are empty. I'm splitting the data through strata (3 cols) with simple random selection which seems to work, but it seems to be splitting the proportions from the 508 rows vs. the ~350 rows with actual data. How do I get it to ignore the missing values in my data column "nce" for the split. Here's my code. Thanks for the help.

/*partitioning data into 2 sets at 70% split with stratification*/
/* new column Selected = 0 is the 30% Test set, Selected = 1 is the 70% Train set */
proc surveyselect data=phrs2cut method=srs samprate= 0.7 out=StratSRS seed=12345 outall;
	samplingunit nce;
	strata loc2 ntrt2 pd2 notsorted / alloc=proportional stats;
	title "After Split";
run;

Re: How does Proc SurveySelect Deal with Missing values?

ballardw — Fri, 09 Oct 2020 21:45:18 GMT

@Daisy2 wrote:

I'm trying to split my data into Train (70%) and Test data sets. My file has 508 rows, but the column I want to split only has about 350 cells filled. The rest are empty. I'm splitting the data through strata (3 cols) with simple random selection which seems to work, but it seems to be splitting the proportions from the 508 rows vs. the ~350 rows with actual data. How do I get it to ignore the missing values in my data column "nce" for the split. Here's my code. Thanks for the help.
/*partitioning data into 2 sets at 70% split with stratification*/
/* new column Selected = 0 is the 30% Test set, Selected = 1 is the 70% Train set */
proc surveyselect data=phrs2cut method=srs samprate= 0.7 out=StratSRS seed=12345 outall;
	samplingunit nce;
	strata loc2 ntrt2 pd2 notsorted / alloc=proportional stats;
	title "After Split";
run;

Please describe what you mean by "ignore the missing values"?

Do you mean not use the record at all, i.e. the source data to the ~350 observations with data for the variable? That could be done with a WHERE statement such as: Where not missing(nce);

Not treat the missing value as level but include the records for selection?

From the documentation for Surveyselect:

PROC SURVEYSELECT treats missing values of STRATA and SAMPLINGUNIT variables like any other STRATA or SAMPLINGUNIT variable value. The missing values form a separate, valid variable level.

It looks like you may need to create an additional variable that has the properties for clustering that you want.

Re: How does Proc SurveySelect Deal with Missing values?

Daisy2 — Fri, 09 Oct 2020 22:56:05 GMT

Thanks for the quick reply. After thinking about your question, I think it's most appropriate to do the former and not the latter - not use the records at all.
Where do I put your suggested code? Before Proc SurveySelect? (I'm pretty new to SAS) such as
Where not missing(nce);
proc surveyselect data=....??

or within the procedure?

Re: How does Proc SurveySelect Deal with Missing values?

ballardw — Sat, 10 Oct 2020 01:05:34 GMT

@Daisy2 wrote:
Thanks for the quick reply. After thinking about your question, I think it's most appropriate to do the former and not the latter - not use the records at all.
Where do I put your suggested code? Before Proc SurveySelect? (I'm pretty new to SAS) such as
Where not missing(nce);
proc surveyselect data=....??

or within the procedure?

In the procedure:

Proc surveyselect data= <other proc options>;

where not missing(nce);

Actual order is not critical but by placing it near the start of the procedure it is easier, IMHO, to remind me I was filtering data for some reason when I come back to the code later. In fact I would likely insert a comment such as

/* Do not want the missing values in this set to be considered as a Cluster for selection*/