Predicting Success (or Failure) of Interventions to Improve Care

CLAHRCs are concerned with improving care, but many initiatives fail. This article seeks a method to predict success or failure in advance. The CLAHRC WM Director recently came across a pair of papers that used Bayes theorem to predict successful organisational change interventions.[1] [2] The work proceeded as follows:

  • Twenty factors that might alter the chance of a successful intervention were identified by expert consultation and literature review; for example, whether monitoring and feedback takes place during the implementation.
  • Prior probabilities of success were assigned – 5% for a culture change initiative and 16% for process change.
  • Experts were asked to assign the likelihood ratio (probability factor present given success / probability factor present given failure) for each of the above 20 factors.
  • 221 projects were classified as either a success or failure by managers who had been involved in the organisational change.
  • Then Bayes’ formula was used to see if success or failure could have been predicted for each of the 221 projects. In each case the appropriate prior (5% or 16%) was updated by the likelihoods.
  • Bayesian predictions were compared with the verdict (success or failure) given by the managers. The predictions were good covering 84% of the area under an ROC curve.

This is an interesting application of binary updating of a prior through subjective likelihood ratios to predict the outcomes of organisation change initiatives. It shows that managers can get a pretty good idea of the likely success or failure of improvement initiatives before they implement them. The CLAHRC WM Director thanks Shaun Leamon of the Health Foundation for drawing his attention to these papers.

— Richard Lilford, CLAHRC WM Director

References

  1. Gustafson DH, Sainfort F, Eichler M, Adams L, Bisognano M, Steudel H. Developing and Testing a Model to Predict Outcomes of Organisational Change. Health Services Research. Health Serv Res. 2003;38(2): 751-76.
  2. Molfenter T, Gustafson D, Kilo C, Bhattacharya A, Olsson J. Prospective Evaluation of a Bayesian Model to Predict Organizational Change. Health Care Manage Rev. 2005;30(3):270-9.
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s