Another Potential Problem with SMRs

Readers of this CLAHRC WM News Blog will know that the director has pointed out the limitations of standardised mortality ratios (SMRs) for a decade. He has explicated the case-mix adjustment fallacy [1]; the constant risk-adjustment fallacy [2]; and the signal to noise issue.[3] Now another potential problem with case-mix adjustment has come to light – the issue of Simpson’s paradox. This paradox arises when an association found in multiple groups is reversed when these groups are aggregated. This can happen when baseball batters are compared. Consider a scenario where batter 1 receives many more pitches than batter 2 in year one, and vice-versa in year two. In such a scenario, batter 1 can have a better strike rate in both years, but a lower strike rate if these rates are simply aggregated. In a brilliant editorial, Drs Perla Marang-van de Mheen and Kaveh Shojania show how this can happen when outcomes are aggregated over doctors and hospitals.[4] The problem of Simpson’s paradox would also arise in meta-analyses if all the good and bad outcomes were simply added up before applying a simple statistical test. Of course, the standard statistical methods avoid this problem and the Director wonders whether there is a statistical approach that could be used in baseball and comparison of SMRs.

— Richard Lilford, CLAHRC WM Director

References:

  1. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet. 2004; 363(9415): 1147-54.
  2. Mohammed MA, Deeks JJ, Girling A, Rudge G, Carmalt M, Stevens AJ, Lilford RJ. Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals. BMJ. 2009; 338: b780.
  3. Girling A, Hofer TP, Wu J, Chilton P, Nicholl J, Mohammed MA, Lilford RJ. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf. 2012; 21(12):1052-6.
  4. Marang-van de Mheen P & Shojania KG. Simpson’s paradox: how performance measurement can fail even with perfect risk adjustment. BMJ Qual Saf. 2014; 23: 701-5.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s