Elsevier

Academic Radiology

Volume 16, Issue 2, February 2009, Pages 227-238
Academic Radiology

Review article
Statistical Approaches for Modeling Radiologists' Interpretive Performance1

https://doi.org/10.1016/j.acra.2008.07.022Get rights and content

Although much research has been conducted to understand the influence of interpretive volume on radiologists' performance of mammography interpretation, the published literature has been unable to achieve consensus on the volume standards required for optimal mammography accuracy. One potential contributing factor is that studies have used different statistical approaches to address the same underlying scientific question. Such studies have relied on multiple mammography interpretations from a sample of radiologists; thus, an important statistical issue is appropriately accounting for dependence, or correlation, among interpretations made by (or clustered within) the same radiologist. The aim of this review is to increase awareness about differences between statistical approaches used to analyze clustered data. Statistical frameworks commonly used to model binary measures of interpretive performance are reviewed, focusing on two broad classes of regression frameworks: marginal and conditional models. Although both frameworks account for dependence in clustered data, the interpretations of their parameters differ; hence, the choice of statistical framework may (implicitly) dictate the scientific question being addressed. Additional statistical issues that influence estimation and inference are also discussed, together with their potential impact on the scientific interpretation of the analysis. This work was motivated by ongoing research being conducted by the National Cancer Institute's Breast Cancer Surveillance Consortium; however, the ideas are relevant to a broad range of settings in which researchers seek to identify and understand sources of variability in clustered binary outcomes.

Section snippets

Design considerations

Before we address the implications of differences in statistical methods for achieving consensus across studies with a common scientific goal, we must consider how studies differ in their outcome and predictor definitions. These choices dictate the mechanism a study is trying to understand, that is, the underlying process by which a predictor variable such as interpretive volume influences interpretive performance. We cannot expect to achieve consensus from studies exploring different

Regression approaches for clustered data

A key feature of clustered data is the potential for dependence between interpretations made by the same radiologist. Intuitively, observations for the same radiologist are more “similar” than those for another radiologist. This dependence arises because of heterogeneity across radiologists: differences in skill levels, thresholds for recalling patients, patient populations, and/or practice or facility characteristics (4, 5, 6, 17, 18). One can account for such between-radiologist differences

Additional statistical considerations

A variety of additional statistical issues that can influence estimation and inference, and therefore potentially affect the interpretation of the analysis, should be considered when choosing a regression formulation. In this section, we focus on statistical issues and hence statistical bias in the regression parameter estimates and/or the standard error estimates that is specifically introduced by decisions concerning the statistical analyses. It should be noted, however, that traditional

Recommendations

We have reviewed commonly used approaches for analyzing clustered data and, in particular, for estimating the influence of covariates such as radiologists' interpretive volume on radiologists' performance. We emphasize three key points. First, for the analysis of clustered data, one should consider whether the covariates of interests potentially vary within a radiologist, between radiologists, or both. In settings in which both types of variability occur, one should consider whether the effects

Acknowledgments

We thank the BCSC investigators and the participating mammography facilities and radiologists for providing the data for the examples. A list of the BCSC investigators and procedures for requesting BCSC data for research purposes is available at http://breastscreening.cancer.gov.

References (50)

  • D.L. Miglioretti et al.

    Marginal modeling of nonnested multilevel data using standard software

    Am J Epidemiol

    (2007)
  • I. Theberge et al.

    Volume of screening mammography and performance in the Quebec population-based Breast Cancer Screening Program

    CMAJ

    (2005)
  • L. Kan et al.

    Standardized abnormal interpretation and cancer detection ratios to assess reading volume and reader performance in a breast screening program

    Radiology

    (2000)
  • A.J. Coldman et al.

    Organized breast screening programs in Canada: effect of radiologist reading volumes on outcomes

    Radiology

    (2006)
  • R. Ballard-Barbash et al.

    Breast Cancer Surveillance Consortium: a national mammography screening and outcomes database

    AJR Am J Roentgenol

    (1997)
  • P.A. Carney et al.

    Current medicolegal and confidentiality issues in large, multicenter research programs

    Am J Epidemiol

    (2000)

  • American College of Radiology (ACR) Breast Imaging Reporting and Data System Atlas (BI-RADS Atlas)

    (2003)
  • J.M. Neuhaus et al.

    Between- and within-cluster covariate effects in the analysis of clustered data

    Biometrics

    (1998)
  • R.D. Rosenberg et al.

    Performance benchmarks for screening mammography

    Radiology

    (2006)
  • E.A. Sickles et al.

    Performance benchmarks for diagnostic mammography

    Radiology

    (2005)
  • P.J. Diggle et al.

    Analysis of longitudinal data

    (2002)
  • S.L. Zeger et al.

    Models for longitudinal data: a generalized estimating equation approach

    Biometrics

    (1988)
  • J.M. Neuhaus et al.

    A comparison of cluster-specific and population-averaged approaches for analyzing correlated binary data

    Int Stat Rev

    (1991)
  • P. McCullagh et al.

    Generalized Linear Models

    (1989)
  • N.E. Breslow et al.

    Statistical methods in cancer research, Vol 1: the analysis of case-control studies

    • Cited by (0)

    1

    This work was supported by grants SIRSG-07-271-01, SIRSG-07-272-01, SIRSG-07-273-01, SIRSG-07-274-01, SIRSG-07-275-01, and SIRGS-06-281-01 from the American Cancer Society, Atlanta, GA; grant 1R01 CA107623 from the National Cancer Institute, Bethesda, MD; and grants U01CA63740, U01CA86076, U01CA86082, U01CA63736, U01CA70013, U01CA69976, U01CA63731, and U01CA70040 from the National Cancer Institute Breast Cancer Surveillance Consortium, Bethesda, MD.

    View full text
    Copyright © 2009 AUR. Published by Elsevier Inc. All rights reserved.