What does interrater reliability assess?

What does interrater reliability assess?

• A. Consistency of measurements taken by the same examiner across sessions
• B. Accuracy of measurements against a gold standard
• C. Consistency of measurements taken by more than one person
• D. The degree to which a measurement reflects the theoretical construct

Answer: (C)

Interrater reliability measures how consistently different people rate or score the same thing. It captures agreement between observers, which is crucial when more than one clinician or rater is involved. If two clinicians evaluate the same patient with the same instrument and give similar scores, reliability is high; if their scores vary widely, reliability is low, indicating that subjective differences or unclear criteria are affecting the ratings. This is why standardizing the scoring system and training raters can improve interrater reliability. Note that this is not about how accurate a measurement is compared to a gold standard (that’s validity), nor about whether a measurement reflects a particular construct. It’s specifically about consistency across different raters. For continuous scores, intraclass correlation is often used; for categorical judgments, a statistic like Cohen’s kappa is common.

Interrater reliability measures how consistently different people rate or score the same thing. It captures agreement between observers, which is crucial when more than one clinician or rater is involved. If two clinicians evaluate the same patient with the same instrument and give similar scores, reliability is high; if their scores vary widely, reliability is low, indicating that subjective differences or unclear criteria are affecting the ratings. This is why standardizing the scoring system and training raters can improve interrater reliability.

Note that this is not about how accurate a measurement is compared to a gold standard (that’s validity), nor about whether a measurement reflects a particular construct. It’s specifically about consistency across different raters. For continuous scores, intraclass correlation is often used; for categorical judgments, a statistic like Cohen’s kappa is common.