Tag Archives: Adverse events

Measuring Quality of Care

Measuring quality of care is not a straightforward business:

  1. Routinely collected outcome data tend to be misleading because of very poor ratios of signal to noise.[1]
  2. Clinical process (criterion based) measures require case note review and miss important errors of omission, such as diagnostic errors.
  3. Adverse events also require case note review and are prone to measurement error.[2]

Adverse event review is widely practiced, usually involving a two-stage process:

  1. A screening process (sometimes to look for warning features [triggers]).
  2. A definitive phase to drill down in more detail and refute or confirm (and classify) the event.

A recent HS&DR report [3] is important for two particular reasons:

  1. It shows that a one-stage process is as sensitive as the two-stage process. So triggers are not needed; just as many adverse events can be identified if notes are sampled at random.
  2. In contrast to (other) triggers, deaths really are associated with a high rate of adverse events (apart, of course, from the death itself). In fact not only are adverse events more common among patients who have died than among patients sampled at random (nearly 30% vs. 10%), but the preventability rates (probability that a detected adverse event was preventable) also appeared slightly higher (about 60% vs. 50%).

This paper has clear implications for policy and practice, because if we want a population ‘enriched’ for high adverse event rates (on the ‘canary in the mineshaft’ principle), then deaths provide that enrichment. The widely used trigger tool, however, serves no useful purpose – it does not identify a higher than average risk population, and it is more resource intensive. It should be consigned to history.

Lastly, England and Wales have mandated a process of death review, and the adverse event rate among such cases is clearly of interest. A word of caution is in order here. The reliability (inter-observer agreement) in this study was quite high (Kappa 0.5), but not high enough for comparisons across institutions to be valid. If cross-institutional comparisons are required, then:

  1. A set of reviewers must review case notes across hospitals.
  2. At least three reviewers should examine each case note.
  3. Adjustment must be made for reviewer effects, as well as prognostic factors.

The statistical basis for these requirements are laid out in detail elsewhere.[4] It is clear that reviewers should not review notes from their own hospitals, if any kind of comparison across institutions is required – the results will reflect the reviewers rather than the hospitals.

Richard Lilford, CLAHRC WM Director


  1. Girling AJ, Hofer TP, Wu J, et al. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling studyBMJ Qual Saf. 2012; 21(12): 1052-6.
  2. Lilford R, Mohammed M, Braunholtz D, Hofer T. The measurement of active errors: methodological issues. Qual Saf Health Care. 2003; 12(s2): ii8-12.
  3. Mayor S, Baines E, Vincent C, et al. Measuring harm and informing quality improvement in the Welsh NHS: the longitudinal Welsh national adverse events study. Health Serv Deliv Res. 2017; 5(9).
  4. Manaseki-Holland S, Lilford RJ, Bishop JR, Girling AJ, Chen YF, Chilton PJ, Hofer TP; UK Case Note Review Group. Reviewing deaths in British and US hospitals: a study of two scales for assessing preventability. BMJ Qual Saf. 2016. [ePub].

Electronic Health Record System and Adverse Outcomes

Does introducing an electronic health record system in a hospital result in an increase in adverse outcomes? Seventeen hospitals were tracked over the ‘go live’ date to see if electronic records would affect risk-adjusted mortality or readmission rates.[1] The result was null – care neither improved nor deteriorated on the end-points observed. The CLAHRC WM Director’s comment is that these end-points are not sensitive to change, since the noise to signal ratio is poor. And, of course, the range of outcomes narrow – no mention here of any stress to patients or staff, for example. Nevertheless, these results are broadly reassuring, especially as the NHS is planning a big push on electronic notes following the Wachter review.[2]

— Richard Lilford, CLAHRC WM Director


  1. Barnett ML, Mehrotra A, Jena AB. Adverse inpatient outcomes during the transition to a new electronic health record system: observational study. BMJ. 2016; 354: i3835.
  2. Wachter R. Making IT work: harnessing the power of health information technology to improve care in England. London: Department of Health. 2016.

Patient Safety Really is Improving

Research carried out by CLAHRC WM colleagues showed, mainly on the basis of process measures, that hospital care in the UK became safer over the ‘Blair Decade’.[1] [2] Now an even larger Dutch study, 2005-2013,[3] has produced corroborating findings with respect to adverse events. Both studies were based on case-note review. The Dutch study found an approximately one-third reduction in adverse events on retrospective review of nearly 16,000 case-notes. So, there are now two separate studies that have used a consistent methodology over time and both suggest that care is becoming safer. This is probably the result of national initiatives and diffusion of safety ideas among clinicians. Indeed one of the reasons put forward for failure to find a statistically significant effect from the Safer Patients Initiative in the UK was the system-wide temporal trend, or ‘rising tide’.[4] There are good arguments to conduct a further follow-up of safety in UK hospitals to see if the improvement noted over the first decade of the millennium has been sustained. This might be the last chance, since case-note review may become more difficult as the future case record is fragmented across hospital IT systems.

— Richard Lilford, CLAHRC WM Director


  1. Benning A, Ghaleb M, Suokas A. Large scale organisational intervention to improve patient safety in four UK hospitals: mixed method evaluation. BMJ. 2011; 342:d195.
  2. Benning A, Dixon-Woods M, Nwulu U, et al. Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ. 2011; 342:d199.
  3. Baines R, Langelaan M, de Bruijne M, Spreeuwenberg P, Wagner C. How effective are patient safety initiatives? A retrospective patient record review study of changes to patient safety over time. BMJ Qual Saf. 2015; 24: 561-71.
  4. Chen Y, Hemming K, Stevens AJ, Lilford RJ. Secular trends and evaluation of complex interventions: the rising tide phenomenon. BMJ Qual Saf. 2015. [ePub].

Ranking Hospitals on Preventable Deaths – an Article that Everyone Should Read

The UK government plans to rank hospitals on avoidable mortality based on case reviews of 2,000 deaths in English hospitals each year. They plan to use the method developed by Hogan et al.,[1] which designates a death as preventable when the reviewer concludes that the probability that the death could have been prevented exceeds 50% (P>0.5).

A recent article [2] criticises the use of a single threshold of preventability (e.g. P< or >=0.5) to determine whether or not a death was preventable. CLAHRC WM, in collaboration with Tim Hofer of University of Michigan, has advocated the use of a six-point Likert or a sliding scale to overcome the loss of information from dichotomising preventability.[3]

While preventable mortality rates provide real information on hospital performance, reviewer reliability is rather low (i.e. inter-observer variability is high).[3] This means that the signal can easily be obscured by noise.[4] Further modelling work may shed light on the extent to which this reduces the accuracy of league table approaches to identify outliers.Meanwhile, it is clear that while measuring preventable deaths overcomes some of the problems associated with measuring all deaths,[5] it nevertheless is no panacea. Measurement of preventability should probably be used as a learning tool, rather than as a performance metric.

— Richard Lilford, CLAHRC WM Director


  1. Hogan H, Healey F, Neale G, et al. Preventable deaths due to problems in care in English acute hospitals: a retrospective case record review study. BMJ Qual Saf. 2012; 21(9): 737-45.
  2. Abel G, Lyratzopoulos G. Ranking hospitals on avoidable death rates derived from retrospective case record review: methodological observations and limitations. BMJ Qual Saf. 2015; 24: 554-7.
  3. Manaseki-Holland S, Lilford RJ, Bishop J, et al. Reviewing deaths in British and US hospitals: a study of case-note reviews. 2015. [Submitted].
  4. Girling AJ, Hofer TP, Wu J, et al. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf. 2012; 21(12): 1052-6.
  5. Lilford R & Pronovost P. Using hospital mortality rates to judge hospital performance: a bad idea that just won’t go away. BMJ. 2010; 340: c2016.

High Side-Effect Rates with Statins in Ordinary Practice but not in RCTs – the Enigma is Explained

Why do such a high proportion of patients who take statins complain of muscle pain in ordinary practice, while RCTs find little increase in this unintended effect? Could it be because the RCTs recruited a low-risk population? No – over 170,000 patients have been entered into high-quality RCTs, many had multi-morbidities and this was not a ‘sanitised’ population. Could it be because unintended effects were not assiduously recorded in RCTs? Again, no.

A recent paper in JAMA [1] examines re-introduction studies and find that the great majority of patients are able to tolerate treatment when it is restarted (admittedly, often in a different form). A particularly interesting study was carried out among people who had been ‘intolerant’ of statins. They were randomised to statin and non-statin treatments following a preparation phase where they were given placebo and could withdraw if intolerable symptoms recurred. Muscle pains were no more prevalent among those randomised to receive statins than among those in the non-statin group. It would appear that the very high incidence of muscle pain in standard practice is the result of psychological expectation, not pharmacological action. What do readers think?

— Richard Lilford, CLAHRC WM Director


  1. Newman CB, & Tobert JA. Statin Intolerance: Reconciling Clinical Trials and Clinical Experience. JAMA. 2015. 313(10); 1011-2.

Patient safety in hospitals: errors decline in UK and now in the US

The CLAHRC WM Director and colleagues demonstrated a reduction in error rates over the previous decade by means of in-depth case reviews in 19 UK hospitals. Infection rates, hand washing rates, vital sign observations, the quality of medical history taking, adherence to various tenets of evidence-based practice, and patient satisfaction all showed an improved trend.[1]

Now AHRQ has demonstrated improving safety in US Hospitals in the first half of the current decade, including pressure ulcers, bloodstream infections, and drug errors.[2] [3] The authors think the reasons are multi-faceted, but financial incentives may have played a large part. Increased spending on the NHS over the years of the Blair Government is an obvious candidate explanation for similar, but earlier, improvements in the UK. It would be interesting to see if the improvement in the safety of UK hospitals in the last decade has been sustained or augmented in the current decade.

— Richard Lilford, CLAHRC WM Director


  1. Cohn J. A Picture of Progress on Hospital Errors. Milbank Quarterly. 2015. 93(1); 36-9.
  2. Benning A, Dixon-Woods M, Nwulu U, et al. Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ. 2011; 342:d199.
  3. Agency for Healthcare Research and Quality. Interim update on 2013 annual hospital-acquired condition rate and estimates of cost savings and deaths averted from 2010 to 2013. Rockville, MD: Agency for Healthcare Research and Quality. 2014.

The Weekend Effect

It is well known that the mortality rate of patients admitted to hospitals over the weekend is higher than that for patients admitted during the week. Whether, or to what extent, this ‘weekend effect’ is caused by case-mix factors vs. care quality factors is one of the big unknowns. This is being investigated by a CLAHRC WM-associated HS&DR grant led by Prof Julian Bion with economic support from Sam Watson, the CLAHRC WM Director and Jo Lord. We were thus provoked by a recent article by Meacock at al [1] investigating the health economics of providing increased consultant support over the weekend. The health gain is calculated on the basis of avoiding all of the excess in deaths and this is offset against the cost of providing a seven-day service. Based on their calculation, the authors find that even if the weekend effect could be eliminated, it would not justify the cost of the service at the NICE willingness-to-pay threshold. In other words, the opportunity cost is such that it would be better to leave the money doing what it is currently doing (if no new money), or to allocate it elsewhere (if new money). However, preventable deaths are merely the top of the adverse event severity pyramid and if the adverse events come down roughly in proportion to deaths, then the gains are much greater and the cost much lower than estimated in the paper. CLAHRC WM collaborators have produced a model to estimate the costs and benefits of reducing adverse events.[2] [3] We hope to collaborate with the authors of the Meacock paper in developing this research.

–Richard Lilford, CLAHRC WM Director


  1. Meacock R, Doran T, Sutton M. What are the costs and benefits of providing comprehensive seven-day services for emergency hospital admissions? Health Economics. 2015. [ePub].
  2. Yao GL, Novielli N, Manaseki-Holland S, Chen Y-F, van der Klink M, Barach P, Chilton PJ, Lilford RJ. Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers. BMJ Qual Saf. 2012; 21(s1):i29-38.
  3. Lilford RJ, Girling AJ, Sheikh A, et al. Protocol for evaluation of the cost-effectiveness of ePrescribing systems and candidate prototype for other related health information technologies. BMC Health Serv Res. 2014; 14: 314.

Very Fresh Blood for Transmission: Probably Not Worth the Effort

This is the largest trial (over 1,200 patients per group) to compare extremely fresh (less than a week old on average) with standard (about three weeks old) blood transfusion in sick adult patients.[1] ‘Old’ blood has lower oxygen carrying capacity than fresh blood, and accumulates potentially harmful metabolites. However, there are costs associated with trying to give everyone fresh blood. In the end this study showed a null result with the point estimate favouring old blood. The death rate was high (over one third) so the trial ‘excluded’ the rather large adverse effect of a 16% increased death rate found in observational studies.

— Richard Lilford, CLAHRC WM Director


  1. Lacroix J, Hébert PC, Fergusson DA, et al. Age of Transfused Blood in Critically Ill Adults. New Engl J Med. 2015;372:1410-8.

Bad Apples vs. Bad Systems

The bad apples versus bad systems argument has erupted again. This argument has been put forcibly in the Los Angeles Times by Philip Levitt.[1] He points out that:

  1. Error rates are not declining, despite humongous effort. This is not quite right; they declined quite markedly in England over the last decade,[2] and on many dimensions of safety adherence it was near 100%. Nevertheless, adverse events remain a substantial problem.
  2. Many interventions, such as surgical check-lists [3] and antisepsis bundles,[4] yield positive interventions when first introduced, but these cannot be replicated.[5] [6] [7]
  3. Analysis of the cognitive form of errors put them down mostly to individual failure rather than the system – most are technical errors during procedures, or misdiagnosis.[8] [9]
  4. Many studies show that a small pool of doctors generate a large proportion of complaints (3% of doctors triggering half of all complaints in an Australian study).[10] Arguably this proportion would be reflected among adverse events as well.

So maybe we should re-think our basic safety science premises. Certainly, falls, pressure ulcers, hospital infections, and medication errors can be blamed in large part on the system. However, these are not the major safety issues; over three-quarters of serious adverse events result from misdiagnosis and errors during procedures. While the system may play a part in these failures the CLAHRC WM Director, who practised at various times as physician and surgeon, is not convinced that the main problem lies in the system. No, diagnosis and safe surgery turn on individual skill. So we need to think about selection and improving the performance of individual clinicians – most especially those who make diagnoses and carry out procedures (i.e. doctors). Of course, if the definition of the system is made very broad, then of course selection and training are included, but the solution lies in medical schools and training programmes, rather than individual organisations. Can we identify an error prone phenotype before they end up in court or a complaints tribunal? Identifying such a phenotype is elusive – as work carried out in our pilot CLAHRC discovered.[11]

— Richard Lilford, CLAHRC WM Director


  1. Levitt P. When medical errors kill. Los Angeles Times. 15 March 2014.
  2. Benning A, Dixon-Woods M, Nwulu U, Ghaleb M, Dawson J, Barber N, et al. Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ. 2011; 342: d199.
  3. Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AH, Dellinger EP, et al. A Surgical Checklist to Reduce Morbidity and Mortality in a Global Population. N Engl J Med. 2009; 360: 491-9.
  4. Pronovost P, Needham D, Berenholtz S, Sinopoli D, Chu H, Cosgrove S, et al. An intervention to decrease catheter-related bloodstream infections in the ICU. N Engl J Med. 2006; 355(26): 2725-32.
  5. Urbach DR, Govindarajan A, Saskin R, Wilton AS, Baxter NN. Introduction of Surgical Safety Checklists in Ontario, Canada. N Engl J Med. 2014; 370: 1029-38.
  6. Reames BN, Scally CP, Thumma JR, Dimick JB. Evaluation of the Effectiveness of a Surgical Checklist in Medicare Patients. Med Care. 2015; 53(1): 87-94.
  7. Bion J, Richardson A, Hibbert P, Beer J, Abrusci T, McCutcheon M, et al. ‘Matching Michigan’: a 2-year stepped interventional programme to minimise central venous catheter-blood stream infections in intensive care units in England. BMJ Qual Saf. 2013; 22(2): 110-23.
  8. Brennan TA, Leape LL, Laird NM, Hebert L, Localio AR, Lawthers AG, et al. Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard Medical Practice Study I. N Engl J Med. 1991; 324(6): 370-6.
  9. Fabri PJ, Zayas-Castro JL. Human error, not communication and systems, underlies surgical complications. Surgery. 2008; 144(4): 557-65.
  10. Bismark MM, Spittal MJ, Gurrin LC, Ward M, Studdert DM. Identification of doctors at risk of recurrent complaints: a national study of healthcare complaints in Australia. BMJ Qual Saf. 2013; 22(7): 532-40.
  11. Coleman JJ, Hemming K, Nightingale PG, Clark IR, Dixon-Woods M, Ferner RE, Lilford RJ. Can an electronic prescribing system detect doctors more likely to make a serious prescribing error? J R Soc Med. 2011; 104(5): 208-18.

Are electronic patient records a medico-legal time-bomb?

Litigation lawyers used to chase ambulances; soon they may be trawling patient records.

Electronic records can be an uncompromising record of when care departed from clinical guidelines. Once upon a time the Bolam test – acting in accordance with a body of medical opinion – was sufficient to define the required standard of care. But after 1997, the Bolitho case modified this to allow that a body of medical opinion could be challenged as irrational.[1] More recently, clinical guidelines have begun to inform the required standard of care.[2]

Now both the Medical Defence Union (MDU) and the Medical Protection Society (MPS) advise that doctors must be prepared to justify decisions and actions that depart from nationally recognised guidelines.[3] [4] Alongside this, the General Medical Council regard it as a professional responsibility to be familiar with guidelines.[5] So if a doctor departs from clinical guidelines without recording a reason and their patient suffers a foreseeable adverse outcome as a result, there is a basis for a medical negligence claim.

Do electronic patient records have implications for this? Quite possibly. By retrieving historical records it is relatively easy to identify if a patient who suffered an adverse outcome was previously treated in accordance with guidelines. We can get an idea of the scale of this by looking at an example.

Anticoagulants halve the risk of stroke in atrial fibrillation, and guidelines recommend anticoagulant use in most patients with atrial fibrillation.[6] But in the UK half are untreated.[7] By contrast, 87% are treated in Germany and 92% in Switzerland.[8] [9] So it is not a question of contraindications. There are about 100,000 strokes a year in the UK, with about 80,000 ischaemic strokes.[10] Data from Sweden – with a similar undertreatment problem – indicates that about 29% of these will have atrial fibrillation.[11] So the UK has roughly 23,000 strokes a year in atrial fibrillation patients, half of whom were not on anticoagulants – 11,500 litigation opportunities. A case would be hard to defend without a documented rationale for withholding treatment.

From the ambulance-chasing perspective this means one in every nine stroke patients represents a business opportunity: certainly worth a letter requesting the records if only the patient will accede. The archive of patient data is easy to search. It is a treasure trove for medical negligence lawyers. It is just surprising it has not already happened.

— Tom Marshall, Deputy Director CLAHRC WM, Prevention and Detection of Diseases


  1. Brazier M, Miola J. Bye-bye Bolam: a medical litigation revolution? Med Law Rev. 2000; 8: 85–114.
  2. Samanta A, Mello MM, Foster C, Tingle J, Samanta J. The role of clinical guidelines in medical negligence litigation: a shift from the Bolam standard? Med Law Rev. 2006; 14(3): 321-66.
  3. Medical Defence Union. Must Doctors Comply With Guidelines? 2010. [Online]. [Last accessed 11 July 2014].
  4. Medical Protection Society. Ignoring the guidelines Case Reports. 2013; 21(1). [Online]. [Last accessed 11 July 2014].
  5. General Medical Council. Good Medical Practice. 2013.[Online]. [Last accessed 11 July 2014].
  6. National Institute for Health and Care Excellence. NICE Guidelines CG180. Atrial fibrillation: the management of atrial fibrillation. 2014. [Online]. [Last accessed 11 July 2014].
  7. Holt TA, Hunter TD, Gunnarsson C, Khan N, Cload P, Lip GY. Risk of stroke and oral anticoagulant use in atrial fibrillation: a cross-sectional survey. Br J Gen Pract. 2012; 62(603): e710-7.
  8. Meiltz A, Zimmermann M, Urban P, Bloch A, Association of Cardiologists of the Canton of Geneva. Atrial fibrillation management by practice cardiologists: a prospective survey on the adherence to guidelines in the real world. Europace. 2008; 10(6): 674-80.
  9. Meinertz T, Kirch W, Rosin L, Pittrow D, Willich SN, Kirchhof P, ATRIUM investigators. Management of atrial fibrillation by primary care physicians in Germany: baseline results of the ATRIUM registry. Clin Res Cardiol. 2011; 100(10): 897-905.
  10. British Heart Foundation. Stroke Statistics 2009. 2009. [Online]. [Last accessed 11 July 2014].
  11. Björck S, Palaszewski B, Friberg L, Bergfeldt L. Atrial fibrillation, stroke risk, and warfarin therapy revisited: a population-based study. Stroke. 2013; 44(11): 3103-8.