topic Re: Post hoc z-tests with a Bonferroni correction in SAS? in Statistical Procedures

Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Mon, 22 Feb 2021 14:15:53 GMT

Hi everyone,

I have a 5x3 table, as shown below (letters and numbers are not my actual variable names, my variables are categorical but not ordinal). I have conducted a Fisher's exact test with Monte Carlo estimation and it is significant, but I was wondering if it's possible to apply post hoc z test of proportions with a Bonferroni adjustment? I have not been able to find an answer online/any example of SAS code online for this. Any help is appreciated!

TABLE:

	A	B	C	D	E
1
2
3

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Mon, 22 Feb 2021 14:33:19 GMT

You can use PROC MULTTEST as illustrated in this note. Many multiple testing adjustments are available, with some that are less conservative than the Bonferroni.

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Mon, 22 Feb 2021 17:10:48 GMT

Thank you! I've modified some of the code included in the link you provided but I'm stuck on how to make this work for a Fisher's test.

ODS OUTPUT fishersexact(PERSIST)=fisher(WHERE=(name1="XP2_FISH") rename=(nvalue1=raw_p));
PROC FREQ DATA= winter_t1;
WHERE platform in ('Facebook', 'Twitter');
TABLES platform*account_type;
exact fisher/mc;
RUN;

PROC FREQ DATA= winter_t1;
WHERE platform in ('Facebook', 'Instagram');
TABLES platform*account_type;
exact fisher/mc;
RUN;

PROC FREQ DATA= winter_t1;
WHERE platform in ('Twitter', 'Instagram');
TABLES platform*account_type;
exact fisher/mc;
RUN;
ODS OUTPUT CLEAR;
proc print noobs;
var value raw_p;
run;
proc multtest pdata=fisher bon;
run;

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Mon, 22 Feb 2021 17:22:48 GMT

You didn't say what problem you ran into, but this basic example works - note the newer option INPVALUES= which is a bit easier to work with.

ODS OUTPUT fishersexact(PERSIST)=fisher(WHERE=(name1="XP2_FISH"));
PROC FREQ DATA= test;
TABLES a*b;
exact fisher/mc;
RUN;
PROC FREQ DATA= test;
TABLES b*c;
exact fisher/mc;
RUN;
ODS OUTPUT CLEAR;
proc multtest inpvalues(nvalue1)=fisher bon;
run;

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Mon, 22 Feb 2021 18:49:43 GMT

When I run that code I get this error: ERROR: INPVALUES= must have the NVALUE1 variable.

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Mon, 22 Feb 2021 19:01:26 GMT

You probably still have that RENAME= option in the ODS OUTPUT line. Note that my code removes it.

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Mon, 22 Feb 2021 19:16:18 GMT

This is what my code looks like, I have the rename statement removed.

ODS OUTPUT fishersexact(PERSIST)=winter_t1 (WHERE=(name1="XP2_FISH"));
PROC FREQ DATA= winter_t1;
WHERE platform in ('Facebook', 'Twitter');
TABLES platform*account_type;
exact fisher/mc;
RUN;

PROC FREQ DATA= winter_t1;
WHERE platform in ('Facebook', 'Instagram');
TABLES platform*account_type;
exact fisher/mc;
RUN;

PROC FREQ DATA= winter_t1;
WHERE platform in ('Twitter', 'Instagram');
TABLES platform*account_type;
exact fisher/mc;
RUN;
ODS OUTPUT CLEAR;

proc multtest inpvalues(nvalue1)=winter_t1 bon;
run;

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Mon, 22 Feb 2021 20:10:37 GMT

You should not use the input data set for FREQ (winter_t1) in the ODS OUTPUT statement. The data set name you use in the ODS OUTPUT statement should be a new name for the data set you are creating when you save the table from the Fisher Exact test.

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Mon, 22 Feb 2021 20:41:57 GMT

Okay, I've changed the dataset in the ODS OUTPUT statement, I still seem to be getting an error from the proc multtest statement: ERROR: No valid observations.

And to clarify, I get output for the proc freq statements, so that's why I think the error has to do with the multtest statement.

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Mon, 22 Feb 2021 21:47:59 GMT

Show the contents of the ODS OUTPUT data set. Assuming it is called FISHER, like it was originally:
proc print data=fisher; run;

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Mon, 22 Feb 2021 23:45:12 GMT

Hmmm... even when I do a proc print, it tells me there are no observations in the 'fisher' dataset

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Tue, 23 Feb 2021 00:51:10 GMT

Then either the created data set has zero observations (see the log note following the ODS OUTPUT CLEAR statement) or all the p-values in the data set are missing. Post your input data set (winter_t1) and the code you used that resulted in this error. Run that code to verify that you reproduce the error with that data set before posting.

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Tue, 23 Feb 2021 01:45:52 GMT

Ok I have double checked and I still receive an error message saying there are 0 observations in the fisher dataset. My SAS code is below and my dataset attached. Thank you for all your help @StatDave !!

libname winterd "C:/Feb 2021"; 

proc import datafile="C:/Feb 2021/Winter_TEST.csv" out=table1
dbms=csv replace;
run;


*recoding 6 (formally coded as "other") to 4;
data table1;
set table1;
if account_type=6 then account_type=4; 
run;
*checking to see this worked;
proc freq data=table1;
table account_type;
run;
*Frequency output of ACCOUNT_TYPE should be
1: 37 
2: 198 
3: 7 
4: 40 
5: 450;

*Fisher's exact test with MC. Resulting p value is significant;
proc freq data=table1;
tables platform*account_type;
exact fisher/mc;
run;


*proc multtest with bonferonni correction;
ODS OUTPUT fishersexact(PERSIST)=fisher(WHERE=(name1="XP2_FISH"));
    PROC FREQ DATA= table1;
         WHERE platform in (1,2);
         TABLES platform*account_type;
		 exact fisher/mc;
		 RUN;

	PROC FREQ DATA= table1;
         WHERE platform in (1,3);
         TABLES platform*account_type;
		 exact fisher/mc;
		 RUN;

	PROC FREQ DATA= table1;
         WHERE platform in (2,3);
         TABLES platform*account_type;
		 exact fisher/mc;
		 RUN;
ODS OUTPUT CLEAR;

proc multtest inpvalues(nvalue1)=fisher bon;
	run;
proc print data=fisher;
run;

Re: Post hoc z-tests with a Bonferroni correction in SAS?

StatDave — Tue, 23 Feb 2021 02:34:18 GMT

Anytime you do something like this, you need to run the procedure once with your data to see exactly what tables are produced, which you want to work with, and exactly what its name is. Since these tables are larger than 2x2, you get a FisherExact table and a separate Monte Carlo table named FisherExactMC. Each has a p-value variable. If you save both of those tables with ODS OUTPUT, you will see that neither of them has a p-value variable called "XP2_FISH". That is the name of the variable in the FisherExact table if the table being analyzed is a 2x2 table. Depending on which p-value you want to use, this code does them both, using the correct variable names found by examining the tables that were produced for your data.

ODS OUTPUT fishersexact(PERSIST)=fisher(WHERE=(name1="P_TABLE"))
           fishersexactmc(PERSIST)=fishmc(WHERE=(name1="MCP_FISH"));
    PROC FREQ DATA= table1;
         WHERE platform in (1,2);
         TABLES platform*account_type;
		 exact fisher/mc; 
		 RUN;

	PROC FREQ DATA= table1;
         WHERE platform in (1,3);
         TABLES platform*account_type;
		 exact fisher/mc;
		 RUN;

	PROC FREQ DATA= table1;
         WHERE platform in (2,3);
         TABLES platform*account_type;
		 exact fisher/mc;
		 RUN;
ODS OUTPUT CLEAR;
proc multtest inpvalues(nvalue1)=fisher bon;
	run;
proc multtest inpvalues(nvalue1)=fishmc bon;
	run;

Re: Post hoc z-tests with a Bonferroni correction in SAS?

monsterpie — Tue, 23 Feb 2021 12:28:05 GMT

I should have realized that earlier. Thank you very much for all your help, it's very much appreciated!