topic Re: Predicting a binary response variable in Statistical Procedures

Predicting a binary response variable

GreggB — Sat, 07 Mar 2020 19:16:41 GMT

My objective is to predict if a student will be flagged to attend a summer reading camp that is determined by a test score generated during end-of-year testing in May. The variable used to predict is a reading score earned in the Fall.
I call the response variable camp_flag and the fall score f_read.
My model (I’m assuming) is something like:

model camp_flag = f_read

I have 2 years of data, so I want to use one year to create the model and use the other year to test the accuracy of the model’s ability to predict camp_flag. Camp_flag is 0 or 1.

My online search is a bit overwhelming. I just need a suggestion on which procedure to learn to accomplish this.

Re: Predicting a binary response variable

GreggB — Sat, 07 Mar 2020 19:17:50 GMT

Re: Predicting a binary response variable

Reeza — Sat, 07 Mar 2020 20:08:58 GMT

PROC LOGISTIC models binary outcomes

However you also have time which makes it more complicated.

PROC AUTOREG and ARIMA are probably your starting point.

@GreggB wrote:
My objective is to predict if a student will be flagged to attend a summer reading camp that is determined by a test score generated during end-of-year testing in May. The variable used to predict is a reading score earned in the Fall.
I call the response variable camp_flag and the fall score f_read.
My model (I’m assuming) is something like:

model camp_flag = f_read

I have 2 years of data, so I want to use one year to create the model and use the other year to test the accuracy of the model’s ability to predict camp_flag. Camp_flag is 0 or 1.

My online search is a bit overwhelming. I just need a suggestion on which procedure to learn to accomplish this.

Re: Predicting a binary response variable

GreggB — Sun, 08 Mar 2020 12:36:22 GMT

Is the time issue because the 2 tests are several months apart or because my 2 data sets are from 2 different years?

Re: Predicting a binary response variable

PaigeMiller — Sun, 08 Mar 2020 15:27:07 GMT

I don't see a need for Time Series ARIMA or AUTOREG if there are only two measurements per student.

Simple logistic regression of the measurement in the fall to predict end_of_may test score. The two different years could be used as an additional predictor variable.

In any event, I would combine both years of data, and randomly select individuals to be the training data set, and other randomly selected individuals to be the validation data set.

Re: Predicting a binary response variable

Reeza — Sun, 08 Mar 2020 18:37:02 GMT

For some reason I thought that time would be a factor. Can students be sent to the camp more than once? Does their previous attendance affect their likelihood to attend again?

If not, I totally agree with @PaigeMiller that you should combine both years and take a random sample, BUT make sure to either include or exclude a student entirely or include them entirely. A single student shouldn't have records in both the test and training data set.

Re: Predicting a binary response variable

GreggB — Sun, 08 Mar 2020 19:41:49 GMT

They would attend only once. To be sure I can unduplicate by Student ID to make sure.

I think I read about what you're saying - the data is divided into 2 sets using ranuni. One set is used to create the model and the other half is used for prediction?

Re: Predicting a binary response variable

PaigeMiller — Sun, 08 Mar 2020 20:13:08 GMT

@GreggB wrote:

They would attend only once. To be sure I can unduplicate by Student ID to make sure.

I think I read about what you're saying - the data is divided into 2 sets using ranuni. One set is used to create the model and the other half is used for prediction?

Yes, that's one way to do it.

And make sure that the different years are a categorical predictor variable in the model.

Re: Predicting a binary response variable

GreggB — Sun, 08 Mar 2020 22:00:11 GMT

proc logistic data = twoyears outest=estimates_2yrs;
model camp_flag = RIT;
run;
quit;

twoyears looks like so: (ID is unique; termName has 2 possible values; camp_flag is 0 or 1)

termName ID RIT camp_flag

2016-2017 001 249 0

2017-2018 002 279 1

1. You're saying my model should be camp_flag = termName RIT ?

2. I want to make sure my objective is clear: I have a 3rd data set (termName = 2019-2020) that contains RIT and I want to predict the camp_flag value so that students most likely to have a value of 0 based on their end-of-year test can be identified now and receive academic intervention. My next step?

Re: Predicting a binary response variable

PaigeMiller — Sun, 08 Mar 2020 22:49:26 GMT

What is RIT?

Re: Predicting a binary response variable

GreggB — Sun, 08 Mar 2020 22:58:03 GMT

My mistake. It is the fall reading score I referred to as f_read earlier.

Re: Predicting a binary response variable

PaigeMiller — Sun, 08 Mar 2020 23:06:55 GMT

@GreggB wrote:
proc logistic data = twoyears outest=estimates_2yrs;
model camp_flag = RIT;
run;
quit;
twoyears looks like so: (ID is unique; termName has 2 possible values; camp_flag is 0 or 1)

termName ID RIT camp_flag

2016-2017 001 249 0

2017-2018 002 279 1

1. You're saying my model should be camp_flag = termName RIT ?

2. I want to make sure my objective is clear: I have a 3rd data set (termName = 2019-2020) that contains RIT and I want to predict the camp_flag value so that students most likely to have a value of 0 based on their end-of-year test can be identified now and receive academic intervention. My next step?

Yes
So ... third data set ... this was not mentioned before. If the model finds no difference between the years, then take it out of the model, re-fit the model without year and use the SCORE command in PROC LOGISTIC to predict the results of the individuals in the third data set. If the two years are statistically different in the fitted model, then I don't think you can use the model to predict a different year that was not in the model.

Re: Predicting a binary response variable

GreggB — Sun, 08 Mar 2020 23:20:12 GMT

updated code:

proc logistic data = twoyears outest=estimates_2yrs;
class termname;
model camp_flag = termname rit;
run;
quit;

Since termname is not numeric I used a CLASS statement. Is this correct?

if so, I interpret this as TermName not being signficant.

Analysis of Maximum Likelihood Estimates
Parameter		DF	Estimate	Standard Error	Wald Chi-Square	Pr > ChiSq
Intercept		1	18.7084	1.8919	97.7879	<.0001
TermName	Fall 2016-2017	1	-0.1980	0.1377	2.0676	0.1505
rit		1	-0.1225	0.0113	118.1675	<.0001

Re: Predicting a binary response variable

PaigeMiller — Sun, 08 Mar 2020 23:40:06 GMT

Yes, that's correct.

Re: Predicting a binary response variable

Reeza — Mon, 09 Mar 2020 00:26:51 GMT

So if a student has already attended they cannot attend again or they’re not going to be recommended to attend even if their test scores warrant it?

@GreggB wrote:

They would attend only once. To be sure I can unduplicate by Student ID to make sure.

I think I read about what you're saying - the data is divided into 2 sets using ranuni. One set is used to create the model and the other half is used for prediction?

Re: Predicting a binary response variable

GreggB — Mon, 09 Mar 2020 00:48:46 GMT

The summer camp is for grade 3 only. The only way a student would attend twice would be if they are retained in grade 3 and they score low enough both times to be flagged for attendance at the summer camp. Since all the data sets have a unique student ID I can easily find scenarios like this if they occurred

Re: Predicting a binary response variable

Reeza — Mon, 09 Mar 2020 01:55:46 GMT

So not the same students each year, that's better then. I'd definitely remove those records but you do need to account for them somehow.

Re: Predicting a binary response variable

GreggB — Mon, 09 Mar 2020 21:36:05 GMT

/* split the data randomly with 50/50 split */
data train valid;
set twoyears; /* 2 years of data combined */
if ranuni(7) <= .5 then output train; else output valid;
run;
/*compare the 2 data sets */
proc logistic data = train outest=estimates_train;
model camp_flag = rit;
run;
quit;
proc logistic data = valid outest=estimates_valid;
model camp_flag = rit;
run;
quit;

Based on what I have studied I believe this is the next step. Here is the % concordant for train and valid, respectively. Is PROC SCORE my next step, using "twoyears"? I'm not sure which portion of the output to look at to determine if I have a model that's good for prediction.

Association of Predicted Probabilities and Observed Responses
Percent Concordant	94.3	Somers' D	0.892
Percent Discordant	5.1	Gamma	0.898
Percent Tied	0.6	Tau-a	0.099
Pairs	29455	c	0.946

Association of Predicted Probabilities and Observed Responses
Percent Concordant	89.0	Somers' D	0.788
Percent Discordant	10.1	Gamma	0.795
Percent Tied	0.9	Tau-a	0.063
Pairs	23648	c	0.894

Re: Predicting a binary response variable

PaigeMiller — Tue, 10 Mar 2020 11:43:52 GMT

You want to fit a model to the Training data set, and then apply the fitted model from the training data set to the validation data set. This is not what you have done ... you have fit a whole new model to the validation data set.

Here is an example of how to apply the fitted model to the validation data set: http://support.sas.com/kb/39/724.html

Re: Predicting a binary response variable

Reeza — Tue, 10 Mar 2020 15:34:17 GMT

You use PROC SCORE or PROC PLS to score your new data set. PLS has more options these days as its the 'newest' procedure. Remember to specify the option for logistic regression though otherwise it doesn't exponentiate the estimate.