topic Re: is appropriate to use error bars to account for missing data when you have the full population in Statistical Procedures

is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Mon, 24 Jan 2022 22:07:26 GMT

Hi all, have data on the large number of people who were applying to receive an award. Everyone who applied must fill out a survey and specify their gender (although they can choose "prefer not to say"). My understanding it is NOT appropriate to put "error bars" on this data given that the survey was not a sample of the population, but rather the whole population.

However, say there is missing data (some people did not report their gender) and one argues that error bars are needed to account for the missing data. Hypothetical example: Submissions are 40% female, 50% male, and 10% did not say. Or you could say that success rates for females are 25%, males 26%, unknown is 27% (so conclusion depends on who is in unknown); and one could conceivably create error bars to show what the result would be if ALL unknowns were all women v. ALL the unknowns are men.

My understanding is that the error bars are used to reflect sampling error and NOT missing data. With missing data one would typically impute. Is this correct? Is there any value in making up error bars for missing data?

(Note: cross posted to stack exchange but no responses)

Re: is appropriate to use error bars to account for missing data when you have the full population

PaigeMiller — Mon, 24 Jan 2022 22:14:43 GMT

However, say there is missing data (some people did not report their gender) and one argues that error bars are needed to account for the missing data.

How would you compute these error bars in this case? These would not be based upon some standard error of the estimate that is usually done with sampled date. They would be some other type of error bars ... and that's fine with me as long as you explain that these would not be based on sampling variability. But really, how do you compute the error bars in this case?

Imputing is another way to handle this.

Re: is appropriate to use error bars to account for missing data when you have the full population

Reeza — Mon, 24 Jan 2022 22:53:21 GMT

Prefer not to say is not missing, it's a category on it's own. And from my somewhat biased experience, it's most likely to be women/non-binary who fill in that choice primarily. However, having that option and then attempting imputation seems problematic. If you want to analyze by subpopulation then things get a bit different but I wouldn't be imputing or combining it.

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Tue, 25 Jan 2022 00:27:37 GMT

Thank you! Hypothetical example. Here is the original data:

	Male	Female	PNA
# Applied	900	1800	300
# Selected	330	570	100
Success Rate	37%	32%	33%

2 extremes:

1) all the PNA are male - success rate for male= 36%, female = 32% (note, F does not change)

2) all the PNA are female - success rates for male = 37% (M does not change), female = 32%

I guess it's more that the data would be presented as a range: Male success rate ranges from 36% - 37%

and Female success rate ranges from 32%-32%

(Note this example is made up, real data would show more of a difference)

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Tue, 25 Jan 2022 00:29:20 GMT

I agree imputation is problematic, and who is in the PNA is unknown (I think it could go either way).

Re: is appropriate to use error bars to account for missing data when you have the full population

Reeza — Tue, 25 Jan 2022 01:18:24 GMT

As long as the sample size is large enough I wouldn't recommend consolidating these categories. PNA could be non-binary which would not fit in either group.

Re: is appropriate to use error bars to account for missing data when you have the full population

PaigeMiller — Tue, 25 Jan 2022 11:02:09 GMT

@mjkop56 wrote:

Thank you! Hypothetical example. Here is the original data:

Male Female PNA

# Applied 900 1800 300

# Selected 330 570 100

Success Rate 37% 32%
33%

2 extremes:

1) all the PNA are male - success rate for male= 36%, female = 32% (note, F does not change)

2) all the PNA are female - success rates for male = 37% (M does not change), female = 32%

I guess it's more that the data would be presented as a range: Male success rate ranges from 36% - 37%

and Female success rate ranges from 32%-32%

(Note this example is made up, real data would show more of a difference)

Okay, you can certainly do this, these are not error bars in any standard sense, and you would be advised to make that very very very very clear. As pointed out by @Reeza , also be prepared to have your assumptions challenged.

Re: is appropriate to use error bars to account for missing data when you have the full population

Rick_SAS — Tue, 25 Jan 2022 14:04:33 GMT

It sounds like what you want is confidence limits for the proportion of success for each of the three groups: Male, Female, and PNA. You could also ask for statistical tests, such as "is there a significant difference for the proportions of success between the groups?"

If you only care whether some group is different (but you don't care which one), you can use PROC FREQ and a chi-square test:

data Awards;
length Gender $6 Success $3;
input Gender Success Count;
datalines;
Male    Yes  330
Male    No   570
Female  Yes  570
Female  No  1230
PNA     Yes  100
PNA     No   200
;

proc freq data=Awards;
   weight Count;
   tables Gender*Success / chisq;
run;

The output shows that the observed proportion in some group is different than expected.

If you want a more sophisticated model that enables you to quantitatively compare the proportions and the effect of gender, then you can use the event-trial syntax in PROC LOGISTIC. If you want information about the predicted proportion, you can use the LSMEANS statement to get that information.


data EventTrial;
length Gender $6;
input Gender Applied Success;
datalines;
Male    900 330
Female 1800 570
PNA     300 100
;

proc logistic data=EventTrial;
   class Gender(ref='Male') / param=GLM;  /* do you want to use Male as the ref group? */
   model Success / Applied = Gender;
   lsmeans Gender / ilink diff=anom adjust=bon CL;
run;

The odds ratio table shows that the 95% CI for the OR of Males vs Females does NOT contain 1, so you can conclude that the effect is different. However, the CI for the OR of Males vs PNA is not different from 1.

The LSMEANS table shows the estimated proportions are Female=0.3167, PNA=0.3333, and Male=0.3667. The CI for Males does not include 1/3, but the other CIs do.

Re: is appropriate to use error bars to account for missing data when you have the full population

Quentin — Tue, 25 Jan 2022 15:25:55 GMT

Missing data are hard. There's a whole literature about handling missing data. I think what you're describing is some sort of 'sensitivity analysis.'

Note, in your example case I think the extremes would be the:

PNA category has 100 males that were selected and 200 females that were rejected. That would give you a 43% acceptance for males and 28.5% acceptance for females.
PNA category has 100 females that were selected and 200 males that were rejected. That would give you a 30% acceptance for males and 35% acceptance for females.

Of course it's very unlikely that either of those worst case scenario extremes occurred, which is how you end up in the land of multiple imputation etc.

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Sat, 29 Jan 2022 20:16:57 GMT

thank you! I'm a bit unclear about how you came up with the extremes - could you please explain?

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Sat, 29 Jan 2022 20:18:27 GMT

"Prefer not to say is not missing, it's a category on it's own." => good point, as someone actually chose this option rather than skipping the question. But I'm wondering what the practical difference is between those two situations? In the case of "prefer not to answer", we don't know where they fit, so it seems to be the same thing as missing.

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Sat, 29 Jan 2022 20:18:50 GMT

thank you, this is helpful.

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Sat, 29 Jan 2022 20:19:14 GMT

thank you - your responses have been very helpful.

Re: is appropriate to use error bars to account for missing data when you have the full population

Quentin — Sun, 30 Jan 2022 17:15:40 GMT

@mjkop56 wrote:

thank you! I'm a bit unclear about how you came up with the extremes - could you please explain?

Hi,

Your original data is:

	Male	Female	PNA
# Applied	900	1800	300
# Selected	330	570	100
Success Rate	37%	32%	33%

The extremes are not "all the PNA are male or al the PNA are female." The extremes are (I think) that gender is perfectly associated with selection among the PNA. That would (I think) give you the most extreme estimates you could get for the acceptance rates of males and females.

So if all all males among the PNA are selected and no females among the PNA are selected, your table becomes:

	Male	Female	PNA
# Applied	900+100=1000	1800+200=2000
# Selected	330+100=430	570
Success Rate	43%	28.5%

The other extreme is all females among the PNA are selected and no males among the PNA are accepted:

	Male	Female	PNA
# Applied	900+200=1100	1800+100=1900
# Selected	330	570+100=670
Success Rate	30%	35%

Re: is appropriate to use error bars to account for missing data when you have the full population

Reeza — Tue, 01 Feb 2022 16:09:28 GMT

If you're going to assume a gender after someone has said they don't want to specify one, what was the point of including that option?

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Sat, 05 Feb 2022 23:18:41 GMT

thanks for the explanation!

Re: is appropriate to use error bars to account for missing data when you have the full population

mjkop56 — Sat, 05 Feb 2022 23:21:47 GMT

The point in including it is that providing gender needs to be optional, it can't be required. However, if there is a sizeable portion that did not want to answer, it affects the conclusions you can make.