User Friendly SAS application: Enhanced Bland-Altman Method of Agreement
Bland-Altman’s (1983) analysis and plot are the most common methods used to assess the relative agreement between two analytical methods that measure the continuous variables measured on the same scale. Many agreement studies have shown that using the t-test or the Pearson’s product-moment correlation is flawed when measuring the agreement or detecting the bias. The basic concept of Bland-Altman’s approach is the visualization of the difference of the measurements made by the two methods, then plotting the differences (diff) or the bias (Y-axis) versus the mean (mean) of the two readings (X-axis). In addition, additional reference lines such as the zero bias line and 95% upper (0 + 1.96 Sdiff) and 95% lower (0 - 1.96 Sdiff) are also overlaid on the same scatter plot. When there is no systematic bias, it is easy to verify from the plot if the differences are symmetrical around zero. If there is no relationship between the differences and the averages, the agreement between the two methods may be summarized using the means and standard deviations test methods.
However, Fernandez and Fernandez 2009 discussed the flaws of the Bland-Altman’s graphical methods and advocate the use of regression approach especially when the bias distribution shows heterogeneous bias distribution. The problem with the correlation coefficient in assessing the agreement is that the two measures might be highly correlated, yet there could be substantial differences in the two measurements across their range of values. To correct the flaws in the Bland- Altman method, Fernandez and Fernandez 2009 proposed a robust enhanced Bland-Altman’s analysis which combines a superior graphical display of bias distribution supplemented with a regression analysis between the diff and the standard measurement. Furthermore, they proposed a modified regression approach to test the significance of the intercept and the slope in the presence of heteroscedasticity. Using SAS 9.2 they showed that how the enhanced Bland-Altman Method of Agreement can be evaluated under six different scenarios.
To facilitate easy and efficient computation of enhanced Bland-Altman agreement analysis, I have developed a user-friendly SAS macro application called BlandA macro. SAS software 9.4 was used to develop this macro application. By using this macro approach, SAS users can perform the enhanced Bland-Altman analysis and spend more time in data exploration, interpretation of graphs, and output, rather than debugging their program errors. In this posting I will present the application of the Enhanced Bland-Altman macro application under following six different data scenarios:
BlandA User-friendly SAS macro application:
First download and unzip the BlandAltman.zip file specified in this post. The requirements for using this SAS macro are
(1) a valid license to run the SAS software on environment, and
(2) SAS modules such as SAS/BASE, SAS/STAT should be installed to get complete results.
The steps for performing the user-friendly SAS macros are:
Step1: Create a SAS data or an excel sheet like the example data file included with the zip file. This data should contain the following variables:
- An interval variable (accmt) measuring the standard baseline measurement.
- Interval variable(s) (mean) measuring the average of the baseline and the new test measurement.
- Interval variable(s) (Diff) measuring difference between the baseline and the new test measurement.
Step2: Open the BlandAltman macro-call file in your preferred SAS environment. In addition to inputting the dataset name, standard baseline measurement, average of the baseline and the new test measurement, and the difference between the baseline and the new test measurement in the MACRO-CALL file, following options are available to specify in the BlandA macro.
Options for saving the SAS output and SAS graphics files. Users can select the folders to save the SAS output and the graphics files by inputting the folder names in the MACRO-CALL file. Also, the users can select one of the following ODS output file format when saving the output produced by the SAS macro BlandA:
Display: Files are not saved but displayed in the SAS results Window.
PDF: PDF files suitable for PDF format
WORD: RTF files suitable for including in Microsoft products.
WEB: HTML files suitable for including in HTML-based Web documents.
Step 3. Submit the SAS macro call file.
After inputting all required fields (Figure 1), Run the macro-call file. The MACRO-CALL file automatically accesses the SAS BlandA macro from the specified location. After a successful run, this macro will generate following output.
Next using six diverse types of simulated data sets, the proposed enhanced features of robust Bland-Altman’s analysis will be compared with the performances of the original Bland-Altman’s method:
- Scanario1: Simulated data with zero bias
- Scanario2: Simulated data with 16% positive bias with no heteroscedasticity
- Scanario3: Simulated data with zero bias and significant heteroscedasticity
- Scanario4: Simulated data with 16% positive bias with significant heteroscedasticity
- Scanario5: Simulated data with heterogeneous bias and non- significant heteroscedasticity
- Scanario6: Simulated data with heterogeneous bias and significant heteroscedasticity
When there is no systematic bias, it is easy to verify from the plot if the differences are symmetrical around zero. When there is no systematic bias with an estimate of 10% average error, the standard Bland-
Altman plot clearly showed the random nature of the bias around the zero-bias line (scenario 1).
-
When there is a positive systematic bias (16%) with an estimate of 10% average error, the standard Bland-Altman plot clearly showed a random nature of the spread with significant positive bias (majority of the bias points were positive). More than 20% of the biased points fell outside the upper 95% prediction limits (scenario 2).
-
When there is NO systematic bias with significant heteroscedastic error, the standard Bland-Altman plot clearly showed a random nature of the spread with zero bias. Only less than 5% of the points fell outside the 95% prediction limits. Thus, the standard Bland-Altman analysis failed to detect the heteroscedastic error distribution (scenario 3).
-
When there is positive systematic bias with significant heteroscedastic error, the standard Bland-Altman plot showed a random nature of the spread with positive bias. The presence of heteroscedasticity was not evident in the standard Bland-Altman plot. Only less than 5% of the points fell below the zero line (scenario 4).
-
When there is heterogeneous bias with no heteroscedastic error, the standard Bland-Altman plot failed to prominently show the nature of the positive and the negative bias in the scatter plot. Also, only less than 5% of the points fell outside the 95% prediction limits. Thus, the standard Bland-Altman analysis failed to detect the heterogeneous bias (scenario 5).
-
Then there is heterogeneous bias with significant heteroscedastic error, the standard Bland-Altman plot showed a random nature of the spread and failed to detect the heterogeneous bias with significant heteroscedastic error. The presence of heteroscedasticity was not clearly evident in the standard Bland-Altman plot. Thus, the standard Bland-Altman analysis failed to detect the heterogeneous error and heteroscedastic error distribution (scenario 6).
In the enhanced Bland-Altman’s graphical method, I substitute a needle plot for the scatter plot and create a plot between the difference of the readings made by the two methods (diff) or the bias (Y-axis) versus the actual
or standard measurement in the X-axis. Also, I added the regression line, its 95% confidence interval band and an additional zero bias reference lines to this enhanced plot. If the regression line and the zero bias line falls within the 95% CLM band then conclude that the bias trend is statistically NOT different from the zero-bias line.
- If the regression line falls above the 95% CLM band and the slope of this regression line is horizontal, then conclude that a significant homogeneous positive bias exists for the new method or the test method when compared with the standard method.
- If the regression line falls below the 95% CLM band and the slope of this regression line is horizontal, then conclude that a significant homogeneous negative bias exists for the new method or the test method when compared with the standard method.
- If the regression line slope is significant with a positive or negative slope and intersect the zero-bias line, then conclude that significant nonsystematic heterogeneous bias exists. However, verify that this bias is not caused by the heteroscedastic error distribution by performing test for significant heteroscedasticity.
Scanario1:
The enhanced Bland-Altman plot revealed the similar Bland-Altman features in the enhanced display. In addition, the enhanced Bland-Altman plot revealed the following additional features:
- the needle plot clearly shows the direction of the bias more effectively than the scatter plot display.
- Both the regression and the zero bias lines fell within the 95% confidence band indicating that both the intercept and the slope of the regression line are not significantly different from zero.
This study confirms that when there is no systematic bias in the method agreement, the standard Bland-Altman plot and the enhanced robust Bland-Altman were comparable and were effective, making the correct conclusion.
Scanario2:
The enhanced Bland-Altman plot also revealed the similar results in the enhanced display. In addition, the enhanced Bland-Altman plot revealed the following additional features:
- The needle plot clearly shows the positive bias more effectively than the scatter plot display.
- The zero bias lines fell below the lower 95% confidence band indicating that the intercept is significantly greater than zero.
- The estimated slope of the regression line is zero because the regression line fell within the 95% confidence interval band.
This study confirms that when there is homogeneous systematic bias in method agreement, the standard Bland-Altman plot and the enhanced robust Bland-Altman were effective in making correct inferences and detecting the positive bias. However, enhanced robust Bland-Altman method revealed additional features of the positive bias and offers additional capabilities to detect the nature of the bias.
Scanario3:
The enhanced Bland-Altman plot also revealed the zero bias in the enhanced display. In addition, the enhanced Bland-Altman plot revealed the following additional features:
- The needle plot clearly shows the nature of heteroscedastic error (larger degree of error in the high range of test values) more effectively than the scatter plot display.
- The zero bias lines and the regression lines fell within the 95% confidence band confirming that both the intercept and the slopes are NOT significantly different from zero, confirming the zero bias in the estimation.
This study confirms that when there is NO systematic bias in method agreement, the standard Bland-Altman plot and the enhanced robust Bland-Altman were effective in making correct inferences about zero bias. However, standard Bland-Altman method failed to detect the presence of heteroscedasticity whereas enhanced robust Bland-Altman method revealed heteroscedasticity.
Scanario4:
The enhanced Bland-Altman plot revealed the positive bias and the heterogeneous error in the enhanced display. In addition, the enhanced Bland-Altman plot revealed the following additional features:
- The needle plot clearly shows the nature of heteroscedastic error (larger degree of error in the high range of test values) more effectively than the scatter plot display.
- The zero bias line falls outside the 95% confidence band indicating the presence of significant positive intercept, which validates the presence of positive bias.
- The regression line fell within the 95% confidence band confirming that the regression slope is NOT significantly different from zero, confirming a homogeneous bias in the estimation.
This study confirms that when there is systematic bias in the method of agreement, the standard Bland-Altman plot and the enhanced robust Bland-Altman were effective in making correct inferences about zero bias. However, standard Bland-Altman method failed to detect the presence of heteroscedasticity whereas enhanced robust Bland-Altman method revealed heteroscedasticity.
Scanario5:
The enhanced Bland-Altman plot clearly revealed the heterogeneous bias. In addition, the enhanced Bland-Altman plot revealed the following additional features:
- The needle plot clearly shows the nature of both positive and the negative more effectively than the scatter plot display.
- The zero bias lines intersects the 95% confidence band confirming that both the intercept and the slopes are significantly different from zero which confirms the presence of heterogeneous bias in the estimation.
This study confirms that when there is systematic bias in method agreement, the standard Bland-Altman plot failed to detect heterogeneous bias pattern. However, the enhanced robust Bland-Altman analysis is very effective in making correct inferences about heterogeneous bias.
Scanario6:
The enhanced Bland-Altman plot clearly revealed the heterogeneous bias and the heterogeneous error in the enhanced display. In addition, the enhanced Bland-Altman plot revealed the following additional features:
o The needle plot clearly shows the nature of heteroscedastic error (larger degree of error in the upper range of test values) more effectively than the scatter plot display.
o The zero bias line intersects the 95% confidence band indicating the presence of significant regression slope which validates the presence of heterogeneous bias.
This study confirms that when there is heterogeneous bias with significant heteroscedastic error, the standard Bland-Altman plot failed to detect both problems. However, enhanced robust Bland-Altman method clearly revealed heterogeneous bias with significant heteroscedastic error.
Robust heteroscedastic consistent regression model (PROC REG):
Fit the following regression model: diffi = β0 + β1 xi + εi where
- Diffi = (New or test measurement – standard or the actual measurement)
- β0 = Y-intercept, a measure of systematic positive or negative bias
- β1 = Slope, a measure of nonsystematic heterogeneous bias
- xi = standard or the actual measurement
- εi = independent normally distributed homogeneous random error
Test the error distribution for homoscedasticity by using the PROC REG option SPEC.
If the homoscedasticity assumption is satisfied, then test the following hypotheses:
- H0: β0 = 0
- H0: β1 = 0
If the homoscedasticity assumption is NOT satisfied, then test the following hypotheses using ACOV option in the MODEL statement. ACOV option displays the heteroscedastic consistent covariance matrix and adds heteroscedastic -consistent standard errors, also known as White standard errors, to the parameter estimates table.
- H0: β0 = 0
- H0: β1 = 0
Interpretation:
- H0: β0 = 0 and β1 = 0 are not rejected: No bias. Validate the new method
- H0: β0 = 0 is rejected but β1 = 0 not Rejected; Positive are negative homogeneous bias.
- H0: β0 = 0 and β1 = 0 are rejected: Significant bias exists. Presence of significant heterogeneous bias is confirmed.
Scanario1:
The random distribution of the residuals in the residual plot clearly showed that the homoscedasticity assumption is not violated.
- The normal probability plot showed that the residual has a normal distribution
- Both β0 and β1 are not statistically different from zero (P-value > 0.05) confirming the findings from the graphical results
Scanario2:
The random distribution of the residuals in the residual plot clearly showed that the homoscedasticity assumption is not violated.
- The normal probability plot showed that the residual has a normal distribution
- Only β1 is not statistically different from zero (P-value > 0.05) whereas the estimate of β0 was significantly different and confirms the findings from the graphical results of the known simulated data.
Scanario3:
The fan shaped distribution of the residuals in the residual plot clearly showed that the homoscedasticity assumption is violated significantly.
- The normal probability plot showed that the residuals have a normal distribution
- Chi-square test clearly showed significant heteroscedasticity indicating that standard t-tests for regression parameter estimates are not valid. However, valid hypothesis tests can be performed using heteroscedasticity -consistent standard errors.
- Both β0 and β1 were not statistically different from zero (P-value > 0.05) based on heteroscedasticity -consistent standard errors which confirm the findings from the graphical results of the known simulated data.
Scanario4:
The fan shaped distribution of the residuals in the residual plot clearly showed that the homoscedasticity assumption is violated significantly.
- The normal probability plot showed that the residual has a normal distribution
- Chi-square test clearly showed significant heteroscedasticity indicating that standard t-testsfor regression parameter estimates are not valid. However, valid hypothesis tests can be performed using heteroscedasticity -consistent standard errors.
- The regression slope β1 is not statistically different from zero (P-value > 0.05) based on heteroscedasticity -consistent standard errors confirm the findings from the graphical results of the known simulated data.
- The intercept β0 is statistically different from zero (P-value < 0.05) based on heteroscedasticity -consistent standard errors which confirm the findings from the graphical results of the known simulated data.
Scanario5:
- The quadratic nature of the distribution of the residuals in the residual plot clearly showed the heterogeneous nature of the bias. But, as expected there is no evidence of heteroscedastic error.
- The normal probability plot showed that the residual has a normal distribution
- Both β0 and β1 were statistically different from zero ( P-value < 0.05) based on the standard regression output which confirms the findings from the graphical results of the known simulated data.
Scanario6:
The fan shaped distribution of the residuals in the residual plot clearly showed that the homoscedasticity assumption is violated significantly.
- The normal probability plot showed that the residual has a normal distribution
- Chi-square test clearly showed significant heteroscedasticity indicating that standard t-tests for regression parameter estimates are not valid. However, valid hypothesis tests can be performed using heteroscedasticity -consistent standard errors.
- The intercept β0 and the regression slope β1 were statistically different from zero (Pvalue < 0.05) based on heteroscedasticity-consistent standard errors which confirm the findings from the graphical results of the known simulated data.
SUMMARY
The results based on these six simulated studies reveled that the standard Bland-Altman plot was effective only in the presence of zero or homogeneous positive error. When nature of the bias is heterogeneous and the error distribution is heteroscedastic, the standard Bland-Altman plot was ineffective. However, the proposed enhanced Bland-Altman plot and the heteroscedasticity -consistent regression analysis method clearly detected zero bias, homogeneous and heterogeneous bias and the presence of homoscedastic and heteroscedastic error. BlandA SAS macro application (Version 9.4) for performing standard and enhanced Bland-Altman plot and robust heteroscedastic consistent regression model are also included.
REFERENCES
- Bland JM, Altman DG (1986). Statistical methods for assessing agreement between two methods of clinical measurement. Lancet i, 307-310
- 2 Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999; 8: 135–60
- Fernandez R, Fernandez G (2009) Validating the Bland-Altman Method of Agreement Western SAS users conference proceedings https://www.lexjansen.com/wuss/2009/pos/POS-Fernandez.pdf
