P-value and 95% CI of OR in weighting survey data

[haoduonge]

Hi all,

I used to report OR and 95% CI in weighting survey data, regardless of p-value.

Based on the paper in the link below, 96%CI is not trusted, p-value may be >0.05 but 95% does not include 1 as it is not taken into account weighting as p-value does.

https://support.sas.com/resources/papers/proceedings17/0970-2017.pdf 

In that case, which one we report? 

Should we report both (OR with 95% CI not include 1) and p-value >0.05, saying that even 95%CI not including 1, the association is not statistically significant?

Thank you for your input.

Hao

[ballardw]

Can you point to the part of the paper, page and lines or similar, that is leading you to that conclusion.

I think you may be paraphrasing or interpreting the author and I am not going to read and reread 10 pages trying to determine how you are getting that interpretation.

[haoduonge]

Hi,

Please have a look at page 3: 

SAS makes it easy to produce point estimates from weighted datasets. All you have to do is include a
WEIGHT statement in your procedure code where you specify the variable holding the weights.
However, the weight statement does not affect standard errors, confidence intervals, or other measures
of variability. This is where you can get into trouble. Standard errors calculated using methods
appropriate to simple random samples will underestimate the standard errors of complex samples. In
other words, if you take the variances or confidence intervals at face value, you will be overconfident
about the estimates and less aware of the expected range in your data. You will be tempted to judge
differences as statistically significant, when they are not.

[ballardw]

---

@haoduonge wrote:

Hi,

Please have a look at page 3: 

SAS makes it easy to produce point estimates from weighted datasets. All you have to do is include a
WEIGHT statement in your procedure code where you specify the variable holding the weights.
However, the weight statement does not affect standard errors, confidence intervals, or other measures
of variability. This is where you can get into trouble. Standard errors calculated using methods
appropriate to simple random samples will underestimate the standard errors of complex samples. In
other words, if you take the variances or confidence intervals at face value, you will be overconfident
about the estimates and less aware of the expected range in your data. You will be tempted to judge
differences as statistically significant, when they are not.

---

It appears that you did not understand what the simple random sample (SRS) included in the discussion above mentions.

Nothing in that says the P-value or OR or limits aren't to be used. It says that you cannot apply the approach typical with a SRS of using a point estimate plus/minus  a test statistic times some values. The test statistic, such as a normal, is using the variance as part of that. However it is going to use a single value of variance if you use the reported summary statistics. Which will generally be smaller than that using the complex sample information. So you use the tools in the survey procedure if the data has a complex sample. Which is why the options are there to provide sample design characteristics and why some options are incompatible with some sample designs. Also the typical introductory statistics class methods assume that your sample is from an infinite population and if your sample design information is done properly it will adjust for finite populations.