Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions

How golden is the 'gold standard'?

R. P. Braun, D. Gutkowicz-Krusin, H. Rabinovitz, A. Cognetta, R. Hofmann-Wellenhof, V. Ahlgrimm-Siess, D. Polsky, M. Oliviero, I. Kolm, Paul Googe, Roy King, V. G. Prieto, L. French, A. Marghoob, M. Mihm

Research output: Contribution to journalArticle

25 Citations (Scopus)

Abstract

Background: The 'gold standard' for the diagnosis of melanocytic lesions is dermatopathology. Although most of the diagnostic criteria are clearly defined, the interpretation of histopathology slides may be subject to interobserver variability. Objectives: The aim of this study was to determine the variability among dermatopathologists in the interpretation of clinically difficult melanocytic lesions. Methods: This study used the database of MelaFind®, a computer-vision system for the diagnosis of melanoma. All lesions were surgically removed and sent for independent evaluation by four dermatopathologists. Agreement was calculated using kappa statistics. Results: A total of 1,249 pigmented melanocytic lesions were included. There was a substantial agreement among expert dermatopathologists: two-category kappa was 0.80 (melanoma vs. non-melanoma) and three-category kappa was 0.62 (malignant vs. borderline vs. benign melanocytic lesions). The agreement was significantly greater for patients ≥40 years (three-category kappa = 0.67) than for younger patients (kappa = 0.49). In addition, the agreement was significantly lower for patients with atypical mole syndrome (AMS) (kappa = 0.31) than for patients without AMS (kappa = 0.76). Limitations: The data were limited by the inclusion/exclusion criteria of the MelaFind® study. This might represent a selection bias. The agreement was evaluated using kappa statistics. This is a standard method for evaluating agreement among pathologists, but might be considered controversial by some statisticians. Conclusions: Expert dermatopathologists have a high level of agreement when diagnosing clinically difficult melanocytic lesions. However, even among expert dermatopathologists, the current 'gold standard' is not perfect. Our results indicate that lesions from younger patients and patients with AMS may be more problematic for the dermatopathologists, suggesting that improved diagnostic criteria are needed for such patients.

Original languageEnglish (US)
Pages (from-to)51-58
Number of pages8
JournalDermatology
Volume224
Issue number1
DOIs
StatePublished - May 1 2012

Fingerprint

Melanoma
Observer Variation
Selection Bias
Artificial Intelligence
Databases
Pathologists

All Science Journal Classification (ASJC) codes

  • Dermatology

Cite this

Braun, R. P., Gutkowicz-Krusin, D., Rabinovitz, H., Cognetta, A., Hofmann-Wellenhof, R., Ahlgrimm-Siess, V., ... Mihm, M. (2012). Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions: How golden is the 'gold standard'? Dermatology, 224(1), 51-58. https://doi.org/10.1159/000336886

Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions : How golden is the 'gold standard'? / Braun, R. P.; Gutkowicz-Krusin, D.; Rabinovitz, H.; Cognetta, A.; Hofmann-Wellenhof, R.; Ahlgrimm-Siess, V.; Polsky, D.; Oliviero, M.; Kolm, I.; Googe, Paul; King, Roy; Prieto, V. G.; French, L.; Marghoob, A.; Mihm, M.

In: Dermatology, Vol. 224, No. 1, 01.05.2012, p. 51-58.

Research output: Contribution to journalArticle

Braun, RP, Gutkowicz-Krusin, D, Rabinovitz, H, Cognetta, A, Hofmann-Wellenhof, R, Ahlgrimm-Siess, V, Polsky, D, Oliviero, M, Kolm, I, Googe, P, King, R, Prieto, VG, French, L, Marghoob, A & Mihm, M 2012, 'Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions: How golden is the 'gold standard'?', Dermatology, vol. 224, no. 1, pp. 51-58. https://doi.org/10.1159/000336886
Braun RP, Gutkowicz-Krusin D, Rabinovitz H, Cognetta A, Hofmann-Wellenhof R, Ahlgrimm-Siess V et al. Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions: How golden is the 'gold standard'? Dermatology. 2012 May 1;224(1):51-58. https://doi.org/10.1159/000336886
Braun, R. P. ; Gutkowicz-Krusin, D. ; Rabinovitz, H. ; Cognetta, A. ; Hofmann-Wellenhof, R. ; Ahlgrimm-Siess, V. ; Polsky, D. ; Oliviero, M. ; Kolm, I. ; Googe, Paul ; King, Roy ; Prieto, V. G. ; French, L. ; Marghoob, A. ; Mihm, M. / Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions : How golden is the 'gold standard'?. In: Dermatology. 2012 ; Vol. 224, No. 1. pp. 51-58.
@article{ecc0502919654de6b67fe466b56dd703,
title = "Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions: How golden is the 'gold standard'?",
abstract = "Background: The 'gold standard' for the diagnosis of melanocytic lesions is dermatopathology. Although most of the diagnostic criteria are clearly defined, the interpretation of histopathology slides may be subject to interobserver variability. Objectives: The aim of this study was to determine the variability among dermatopathologists in the interpretation of clinically difficult melanocytic lesions. Methods: This study used the database of MelaFind{\circledR}, a computer-vision system for the diagnosis of melanoma. All lesions were surgically removed and sent for independent evaluation by four dermatopathologists. Agreement was calculated using kappa statistics. Results: A total of 1,249 pigmented melanocytic lesions were included. There was a substantial agreement among expert dermatopathologists: two-category kappa was 0.80 (melanoma vs. non-melanoma) and three-category kappa was 0.62 (malignant vs. borderline vs. benign melanocytic lesions). The agreement was significantly greater for patients ≥40 years (three-category kappa = 0.67) than for younger patients (kappa = 0.49). In addition, the agreement was significantly lower for patients with atypical mole syndrome (AMS) (kappa = 0.31) than for patients without AMS (kappa = 0.76). Limitations: The data were limited by the inclusion/exclusion criteria of the MelaFind{\circledR} study. This might represent a selection bias. The agreement was evaluated using kappa statistics. This is a standard method for evaluating agreement among pathologists, but might be considered controversial by some statisticians. Conclusions: Expert dermatopathologists have a high level of agreement when diagnosing clinically difficult melanocytic lesions. However, even among expert dermatopathologists, the current 'gold standard' is not perfect. Our results indicate that lesions from younger patients and patients with AMS may be more problematic for the dermatopathologists, suggesting that improved diagnostic criteria are needed for such patients.",
author = "Braun, {R. P.} and D. Gutkowicz-Krusin and H. Rabinovitz and A. Cognetta and R. Hofmann-Wellenhof and V. Ahlgrimm-Siess and D. Polsky and M. Oliviero and I. Kolm and Paul Googe and Roy King and Prieto, {V. G.} and L. French and A. Marghoob and M. Mihm",
year = "2012",
month = "5",
day = "1",
doi = "10.1159/000336886",
language = "English (US)",
volume = "224",
pages = "51--58",
journal = "Dermatology",
issn = "1018-8665",
publisher = "S. Karger AG",
number = "1",

}

TY - JOUR

T1 - Agreement of dermatopathologists in the evaluation of clinically difficult melanocytic lesions

T2 - How golden is the 'gold standard'?

AU - Braun, R. P.

AU - Gutkowicz-Krusin, D.

AU - Rabinovitz, H.

AU - Cognetta, A.

AU - Hofmann-Wellenhof, R.

AU - Ahlgrimm-Siess, V.

AU - Polsky, D.

AU - Oliviero, M.

AU - Kolm, I.

AU - Googe, Paul

AU - King, Roy

AU - Prieto, V. G.

AU - French, L.

AU - Marghoob, A.

AU - Mihm, M.

PY - 2012/5/1

Y1 - 2012/5/1

N2 - Background: The 'gold standard' for the diagnosis of melanocytic lesions is dermatopathology. Although most of the diagnostic criteria are clearly defined, the interpretation of histopathology slides may be subject to interobserver variability. Objectives: The aim of this study was to determine the variability among dermatopathologists in the interpretation of clinically difficult melanocytic lesions. Methods: This study used the database of MelaFind®, a computer-vision system for the diagnosis of melanoma. All lesions were surgically removed and sent for independent evaluation by four dermatopathologists. Agreement was calculated using kappa statistics. Results: A total of 1,249 pigmented melanocytic lesions were included. There was a substantial agreement among expert dermatopathologists: two-category kappa was 0.80 (melanoma vs. non-melanoma) and three-category kappa was 0.62 (malignant vs. borderline vs. benign melanocytic lesions). The agreement was significantly greater for patients ≥40 years (three-category kappa = 0.67) than for younger patients (kappa = 0.49). In addition, the agreement was significantly lower for patients with atypical mole syndrome (AMS) (kappa = 0.31) than for patients without AMS (kappa = 0.76). Limitations: The data were limited by the inclusion/exclusion criteria of the MelaFind® study. This might represent a selection bias. The agreement was evaluated using kappa statistics. This is a standard method for evaluating agreement among pathologists, but might be considered controversial by some statisticians. Conclusions: Expert dermatopathologists have a high level of agreement when diagnosing clinically difficult melanocytic lesions. However, even among expert dermatopathologists, the current 'gold standard' is not perfect. Our results indicate that lesions from younger patients and patients with AMS may be more problematic for the dermatopathologists, suggesting that improved diagnostic criteria are needed for such patients.

AB - Background: The 'gold standard' for the diagnosis of melanocytic lesions is dermatopathology. Although most of the diagnostic criteria are clearly defined, the interpretation of histopathology slides may be subject to interobserver variability. Objectives: The aim of this study was to determine the variability among dermatopathologists in the interpretation of clinically difficult melanocytic lesions. Methods: This study used the database of MelaFind®, a computer-vision system for the diagnosis of melanoma. All lesions were surgically removed and sent for independent evaluation by four dermatopathologists. Agreement was calculated using kappa statistics. Results: A total of 1,249 pigmented melanocytic lesions were included. There was a substantial agreement among expert dermatopathologists: two-category kappa was 0.80 (melanoma vs. non-melanoma) and three-category kappa was 0.62 (malignant vs. borderline vs. benign melanocytic lesions). The agreement was significantly greater for patients ≥40 years (three-category kappa = 0.67) than for younger patients (kappa = 0.49). In addition, the agreement was significantly lower for patients with atypical mole syndrome (AMS) (kappa = 0.31) than for patients without AMS (kappa = 0.76). Limitations: The data were limited by the inclusion/exclusion criteria of the MelaFind® study. This might represent a selection bias. The agreement was evaluated using kappa statistics. This is a standard method for evaluating agreement among pathologists, but might be considered controversial by some statisticians. Conclusions: Expert dermatopathologists have a high level of agreement when diagnosing clinically difficult melanocytic lesions. However, even among expert dermatopathologists, the current 'gold standard' is not perfect. Our results indicate that lesions from younger patients and patients with AMS may be more problematic for the dermatopathologists, suggesting that improved diagnostic criteria are needed for such patients.

UR - http://www.scopus.com/inward/record.url?scp=84861099702&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84861099702&partnerID=8YFLogxK

U2 - 10.1159/000336886

DO - 10.1159/000336886

M3 - Article

VL - 224

SP - 51

EP - 58

JO - Dermatology

JF - Dermatology

SN - 1018-8665

IS - 1

ER -