The provision of ‘high-value’ care (HVC) – balancing health outcomes from treatment against financial costs, potential adverse events and the disutility of undergoing treatment – has become increasingly important in a time of austerity and patient-centred care. A recent paper in the Annals of Internal Medicine therefore set out to establish whether a subset of single-best answer questions used as part of a wider knowledge-based examination could be an effective tool for assessing trainees’ knowledge of HVC. Thirty-eight existing questions were identified as assessing domains of HVC and the scores of around 18,000 residents were analysed for evidence of validity. We are not informed of the extent to which any of the measures of HVC used in the study were reliable, although an examination including just 38 questions is unlikely to have sufficient reliability to be used to classify trainees.
The analysis proceeds at the level of the training programme (N=362) and no data on the variability of trainees’ scores within a programme, compared to that between programmes, are provided. We are informed that the HVC subscore correlates positively at programme level with total examination scores, although no quantitative measure of the correlation is provided and any such measure would inevitably be biased upwards by the inclusion of the HVC subscore in the total score. Despite the authors’ statement that their findings “support the importance of the training environment in fostering HVC” (p. 737), there was poor agreement between programme quartiles based on HVC subscores and a measure of hospital care intensity (a quadratic weighted kappa of 0.17 was calculated from data provided). Evidence of validity at trainee level could have been provided, as survey data on self-reported HVC behaviours was also collected, but again analysed at programme level (with no consistent relationship identified across the eight HCV behaviours included in the survey).
Research in medical education – of which assessment is a key domain – is often seen as the poor bedfellow of clinical research. Guidance on reporting and interpreting validity evidence is available  and needs to be followed if medical education research is to raise its profile.
— Celia Taylor, Senior Lecturer
- Ryskina KL, Korenstein D, Weissman A, Masters P, Alguire P, Smith CD. Development of a High-Value Care Subscore on the Internal Medicine In-Training Examination Assessing Residents’ Knowledge of HVC. Ann Intern Med. 2014; 161(10): 733-9.
- Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003; 37(9): 830-7