Significance tests with weights (and replicate weights?) and producing a summary output table

[Wolverine]

I have very little experience working with weights, so please correct me if my understanding is wrong.

I'm trying to create a summary table of unadjusted rates of quality of care between the TM and MA groups. I was able to produce a table with Proc Ttest and ODS. However, the survey uses a complex design. I need to add a weight variable and, it appears, replicate weight variables. Unfortunately, Proc Ttest can accommodate a weight variable but not replicate weights.

Just to experiment, I tried running Proc Ttest with the weight variable, and the sig score for the variables improved. That confuses me, because the study documentation says "To permit the calculation of random errors due to sampling, a series of replicate weights were computed. Unless the complex nature is taken into account, estimates of the variance of a survey statistic may be biased downward." In other words, not using weights means underestimating the variance. And if the true variance is actually higher, shouldn't that reduce the significance level? One particular variable I looked at has a probt score of 0.0158 when unweighted, and 0.0025 when weighted.

Based on what I found in the study documentation, I'm trying to use Proc Surveyfreq instead. However, this is confusing me as well.  The Pr > ChiSq score is now <.0001 for every variable, even those that were not significant when I used Proc Ttest.

Here is the code, with sample data and Proc TTest commented out. I'm only including 1 of the replicate weights here, but there are actually 100 of them:

data have;
  infile datalines dsd dlm=',' truncover;
  input ACC_HCTROUBL_r ACC_HCDELAY_r ADRD_group TM_group
PUFFWGT PUFF001;
datalines;
3,1,1,0,1310.792231,1957.576268
2,1,1,1,10621.60998,18588.46812
3,2,1,1,3042.093381,5484.728615
3,2,1,1,3166.358963,5497.289892
3,2,1,0,1481.272986,432.6313548
2,2,1,1,6147.605583,9371.965632
2,1,1,1,14001.79093,16689.25322
3,1,1,1,2035.685768,530.211881
2,1,1,1,6356.258972,1899.874476
3,2,1,0,1487.104781,2018.636444
2,1,1,0,5002.553584,1364.125425
2,2,1,1,2493.79145,4039.542597
3,2,1,0,2260.257377,3495.675613
2,2,0,1,9358.048737,2835.543292
3,2,1,1,2978.506348,4932.378916
3,2,1,1,2794.906054,5118.430973
3,1,1,0,1663.418821,519.7549258
3,2,1,0,2083.459361,3067.105973
2,1,1,0,5106.785048,8672.202644
3,1,1,1,3447.574748,854.6276748
3,2,1,1,2819.233426,899.849234
3,2,1,0,4067.38684,6463.15598
3,2,1,1,1249.96647,2053.666234
3,2,1,1,1730.237908,3058.307502
3,2,1,1,4932.936202,1479.55826
; RUN;

/*PROC TTEST plots=none	data=have;
	CLASS 	TM_group;
	VAR ACC_HCTROUBL_r ACC_HCDELAY_r;
	WEIGHT PUFFWGT;
	REPWEIGHT PUFF001; /*REPWEIGHT PUFF001-PUFF100;*/
RUN;*/
PROC SURVEYFREQ data=have VARMETHOD = brr (fay=.30); 
	TABLE ACC_HCTROUBL_r ACC_HCDELAY_r * TM_group / row chisq lrchisq; 
	WEIGHT PUFFWGT; 
		REPWEIGHT PUFF001; /*REPWEIGHT PUFF001-PUFF100;*/
	WHERE ADRD_group ^= 1;
RUN;

[Ksharp]

The more weight the currenct obs have is standing for the correnct obs is more accurate for testing.
Check @Rick_SAS 's blog to see how different bwteen weight and freq.

https://blogs.sas.com/content/iml/2017/10/02/weight-variables-in-statistics-sas.html
https://blogs.sas.com/content/iml/2016/10/05/weighted-regression.html

[Wolverine]

Thank you, that confirms that I do need to use weights here, even for unadjusted summary statistics.

[PaigeMiller]

What do you mean by "replicate weights".

PROC SURVEYMEANS ought to be able to do the survey equivalent of a T-test.

[Wolverine]

---

@PaigeMiller wrote:

What do you mean by "replicate weights".

PROC SURVEYMEANS ought to be able to do the survey equivalent of a T-test.

---

I haven't really worked with data based on a sample before, but from what I've found online, the individual weight adjusts for differences between the sample proportion and the population proportion. So for example, if Hispanics make up 10% of a given population but only 5% of the sample, each Hispanic person in the sample would be given a weight of 2 to give them the proper influence over the sample results.

Replicate weights adjust for primary sampling units, clustering, and other such design effects that make it easier to gather data, but also less representative of the population than a truly random sample.

Thanks for the tip of Proc Surveymeans - I found there are actually several "survey" procedures that take weights into account. The problem is that I'm supposed to make a summary table of the results, using the TableN macro for Table 1 and syntax I wrote myself for other comparisons needed for the manuscript I'm helping with. Changing the procedure changes the output. TableN might be able to accommodate weights, but the syntax I wrote won't be able to 😓

[PaigeMiller]

Does the REPWEIGHTS statement in PROC SURVEYMEANS do what you want?

[Wolverine]

It should -- any proc with "survey" in the name should be able to use sample weights

[Wolverine]

---

@Wolverine wrote:

It should -- any proc with "survey" in the name should be able to use sample weights

---

Generating the proper results is relatively easy. But rearranging those results for dozens of variables and putting them into a well-organized summary table takes a lot of code. The TableN macro does this extremely well, but it doesn't have a provision to add weights. Likewise, the code I wrote for the secondary table uses Proc Ttest, and there is no "SURVEY" equivalent (like Proc Surveyfreq is the equivalent to Proc Freq). Proc Ttest can use a person weight, but not sampling weights.