A meta analysis for sensitivity comparing two diagnostic tools

[J_Park]

I would like to compare the sensitivity from 4 studies comparing two diagnostic tools using proc glimmix.  Is it possible?   I am familiar with the meta analysis for effect sizes but have never done for sensitivity.   I would appreciate any suggestions from the experts.  Thank you!

data meta_analysis_data;
input study_id sensitivity1 sensitivity12 se_sensitivity1 se_sensitivity2;
datalines;
1 0.85 0.75 0.05 0.04
2 0.90 0.80 0.04 0.03
3 0.80 0.77 0.06 0.05
4 0.88 0.82 0.05 0.04
;
run;

[StatDave]

I believe a model could written to do this assuming
1) that you have the data for each of the studies on each of the diagnostics
2) the two diagnostics are continuous variables
3) for each diagnostic in each study, you fit a model to a binary response (such as a logistic model) that produced predicted probabilities and you selected a cutoff value on those probabilities that classifies all observations as predicted events or nonevents yielding the sensitivity for that cutoff
If the above are all true, then a question is whether an independent set of subjects was used for each study-diagnostic combination or whether subjects were repeatedly measured either within each study or across all studies.

[J_Park]

Thank you for your response!

1) that you have the data for each of the studies on each of the diagnostics  -> Yes, I do.
2) the two diagnostics are continuous variables -> The data I have are binary responses for sensitivity (1 for disease; 0 for no for each device and a gold standard)
3) for each diagnostic in each study, you fit a model to a binary response (such as a logistic model) that produced predicted probabilities and you selected a cutoff value on those probabilities that classifies all observations as predicted events or nonevents yielding the sensitivity for that cutoff -> Since I already have binary data, I can skip this, correct?
If the above are all true, then a question is whether an independent set of subjects was used for each study-diagnostic combination or whether subjects were repeatedly measured either within each study or across all studies. -> Each study has the same patients were tested for all three (gold standard, Device 1, and Device 2). 

I want to do the meta analysis using 4 different studies which device (device 1 or device 2) is better in terms of sensitivity (I will repeat for specificity, PPV, NPC, accuracy).  Is this doable using SAS?  Thank you!

[Ksharp]

And I think missed a very important variable --> sample size of each study id which would effect the variance in model.

[J_Park]

I do have the sample size for each study. I was wondering about that but I thought SE is already taking the sample size into account?

[StatDave]

Following up on my earlier comment, I believe that you can use the original data to do a proper analysis that estimates the common sensitivity for each device across the studies taking into account the correlation caused by subjects providing responses on both devices. The analysis, using a GEE model, can provide the common sensitivities along with standard errors allowing for confidence intervals and a comparison. 

The following DATA step generates some example data. This simple step just generates, for each subject (ID), random binary values on the gold standard and each device. While it doesn't try to build in specific sensitivity values or any correlation, it will serve to illustrate the analysis. It creates data for two devices and three studies, but the proposed analysis could be used for more of both. Note that each subject has two observations created and the variable, DIAG, is created to hold the binary responses from each device (DEV).

data f; 
call streaminit(42633);
do study=1 to 3;
do rep=1 to 20;
  drop rep;
  id+1; gold=rand('bernoulli',.5);
  dev=1; diag=rand('bernoulli',.5); output;
  dev=2; diag=rand('bernoulli',.5); output;
end; end; 
run;

Before doing the analysis, let's find out the sensitivities in the tables from the generated data. The following produces the 2x2 table for each device against the gold standard within each study.

proc sort data=f; 
by study dev; 
run;
proc freq data=f; 
by study dev; 
table diag*gold; 
run;

As shown in this note, the sensitivity in each table is the column percent in the 1,1 cell, so since it is really only necessary to work with the gold=1 column, the following statements again obtain and save the sensitivity values. As such, the sensitivity is just the event probability of the binary, gold=1, variable in each table. The PROC MEANS step then shows the simple averages of the observed sensitivities for each device.

proc freq data=f; 
where gold=1; 
by study dev; 
table diag; 
ods output onewayfreqs=frqs(where=(diag=1));
run;
proc means mean; 
class dev; var percent;
run;

The following fits a logistic GEE model on the sensitivities. Similar to what is done in the second FREQ step above, only the gold=1 data is used.

proc genmod data=f;
where gold=1;
class study dev;
model diag(event='1')=dev / dist=bin;
repeated subject=study;
lsmeans dev / ilink diff cl plots=none;
run;

You can see that the estimated common sensitivity for each device across the studies (provided by the ILINK option in the LSMEANS statement) is similar to the average of its observed sensitivities found by PROC FREQ above. The DIFF and CL options provide confidence intervals and a comparison of the device sensitivities. 

You can see that GENMOD reproduces the observed sensitivities by fitting a saturated model (though that model leaves no variability with which to obtain standard errors):

proc genmod data=f;
where gold=1;
class study dev;
model diag(event='1')=study|dev / dist=bin;
repeated subject=study;
lsmeans study*dev / ilink plots=none;
run;

I believe that a similar approach can be taken to do the same for other 2x2 table statistics like specificity.

[Ksharp]

SteveDave,
I think your model builded on the ODDS Ratio , not Relative Risk, right ?
If compare two ratio, I think your link function should be IDENTITY or LOG ?
model diag(event='1')=dev / dist=bin link=identity;
model diag(event='1')=dev / dist=bin link=log;

[J_Park]

This is the revised data.

data meta_analysis_data;
input study_id sensitivity1 sensitivity2  se_sensitivity1 se_sensitivity2 N;
datalines;
1 0.84 0.59 0.03 0.04 305
2 0.88 0.94 0.09 0.06 35
3 0.94 0.89 0.06 0.08 44
4 0.89 0.94 0.03 0.03 208
;
run;

[Ksharp]

Can't you  directly use sensitivity  in mixed model  to get pooled sensitivity ?

p.s. check attachment paper.

data meta_analysis_data;
input study_id sensitivity specificity se_sensitivity se_specificity;
datalines;
1 0.85 0.75 0.05 0.04
2 0.90 0.80 0.04 0.03
3 0.80 0.77 0.06 0.05
4 0.88 0.82 0.05 0.04
;
run;
data have;
 set meta_analysis_data;
 level=1;count=100*sensitivity;output;
 level=0;count=100*(1-sensitivity);output;
 keep study_id level count;
run;
ods select none;
proc freq data=have;
by study_id;
table level/binomial(level='1' cl=exact);
weight count;
ods output  Binomial= Binomial;
run;
ods select all;
proc transpose data= Binomial out= Binomial2;
by study_id;
var nvalue1;
id name1;
run;
data Binomial3;
 set Binomial2;
 wgt = 1/(E_BIN**2);
run;



proc mixed data=Binomial3 method=ml covtest;
class study_id; *<--indicate that study is a class/factor variable;
weight wgt; *<--weight is inverse of sampling variance per study;
model _BIN_ = / chisq s df=10000; *<--model gives just the global mean (intercept);
random int/subject=study_id; *<--define a random study effect (get variance);
repeated; *<--this and next line force fixed sampling variances;
parms (1) (1) / eqcons=2; *<--forces the sampling-variance to be 1*(1/weight);
estimate 'mean' int 1/ cl df=10000; *<--another way of displaying est. expected effect size;
run;

[J_Park]

This time, I am comparing two sensitivity measures between two different devices.

[J_Park]

The paper is great. What I am not sure is what to use for the effect size when I compare two sensitivity. I wonder if I can use sensitive ratio as effect size as someone suggested before.

[Ksharp · Accepted solution]

If you want to calcuated Effect Size ,check attachement paper:

If you want to compare the difference between two ratios, you could use PROC FREQ + CMH to get POOLED RELRISK. Here is an example:

data meta_analysis_data;
input study_id sensitivity1 sensitivity2  se_sensitivity1 se_sensitivity2 N;
datalines;
1 0.84 0.59 0.03 0.04 305
2 0.88 0.94 0.09 0.06 35
3 0.94 0.89 0.06 0.08 44
4 0.89 0.94 0.03 0.03 208
;
run;
data have;
 set meta_analysis_data;
 trt=0;col=0;count=sensitivity1*N;output;
 trt=0;col=1;count=(1-sensitivity1)*N;output;
 trt=1;col=0;count=sensitivity2*N;output;
 trt=1;col=1;count=(1-sensitivity2)*N;output;
 keep study_id trt col count;
run;
proc freq data=have;
table study_id*trt*col/cmh relrisk(column=1);
weight count;
run;

[J_Park]

Thank you so much for your help!   I really appreciate it.

[J_Park]

Hello Ksharp, 

If you don't mind me asking, what does "Statistics for Table 1 of Device by col Controlling for study_id=1"?  Does this mean it treats the study_id as a variable and control the variable?    How do I interpret the relative risk of 1.147 since the ratio is sensitivity.   The device 1 has a higher sensitivity as RR was computed sen1/sen2?    

Thank you!

[Ksharp]

No. It means like BY statement, like the following code:
proc freq data=have;
where study_id=1;
table trt*col/cmh relrisk(column=1);
weight count;
run;

The common relative risk I showed you is the pooled RR by control the study_id variable.

sensitivity is a just binomial distribution ratio, RR=sen1/sen2 (a.k.a sensitivity1 is 1.147 times sensitivity2) , if RR 's lower CI greater than 1 [1.086,1.2] ,that means sensitivity1 is significant bigger than sensitivity2 at alpha=0.05.