Tag Archives: Director & Co-Directors’ Blog

Estimating Mortality Due to Low-Quality Care

A recent paper by Kruk and colleagues attempts to estimate the number of deaths caused by sub-optimal care in low- and middle-income countries (LMICs).[1] They do so by selecting 61 conditions that are highly amenable to healthcare. They estimate deaths from these conditions from the global burden of disease studies. The proportion of deaths attributed to differences in health systems is estimated from the difference in deaths between LMICs and high-income countries (HICs). So if the death rate from stroke in people aged 70 to 75 is ten per thousand in HICs and 20 per thousand in LMICs, then ten deaths per 1000 are preventable. This ‘subtractive method’ to estimate deaths that could be prevented by improved health services simply answers the otiose question: “what would happen if low-income countries and their populations could be converted, by the wave of a wand, into high-income countries complete with populations enjoying high income from conception?” Such a reductionist approach simply replicates the well-known association between per capita GDP and life expectancy.[2]

The authors of the above paper do try to isolate the effect of institutional care from access to facilities. To make their distinction they need to estimate utilisation of services. This they do from various household surveys, conducted at selected sites around the world. These surveys contain questions about service use. So a further subtraction is performed; if half of all people deemed to be having a stroke utilise care, then half of the difference in stroke mortality can be attributed to quality of care.

Based on this methodology the authors find that the lion’s share of deaths are caused by poor quality care not failure to get care. This conclusion is flawed because:

  1. The link between the databases is at a very coarse level – there is no individual linkage.
  2. As a result risk-adjustment is not possible.
  3. Further to the above, the method is crucially unable to account for delays in presentation and access to care preceding presentation that will inevitably result in large differences in prognosis at presentation.
  4. Socio-economic status and deprivation over a lifetime is associated with recovery from a condition, so differences in outcome are not due only to differences in care quality.[3]
  5. There are measurement problems at every turn. For example, Global Burden of Disease is measured in very different ways across HICs and LMICs – the latter rely heavily on verbal autopsy.
  6. Quality, as measured by crude subtractive methodologies, includes survival achieved by means of expensive high technology care. However, because of opportunity costs, introduction of effective but expensive treatments will do more harm than good in LMICs (until they are no longer LMICs).

The issue of delay in presentation is crucial. Take, for example, cancer of the cervix. In HICs the great majority of cases are diagnosed at an early, if not at a pre-invasive, stage. However, in low-income countries almost all cases were already far advanced when they present. To attribute the death rate difference to the quality of care is inappropriate. Deep in the discussion the authors state ‘comorbidity and disease history could be different between low and high income countries which can result in some bias.’ This is an understatement and the problem cannot be addressed by a passing mention of it. Later they also assert that all sensitivity analyses support the conclusion that poor healthcare is a larger driver of amenable mortality than utilisation of services. But it is really difficult to believe such a sensitivity analyses when this bias is treated so lightly.

Let us be clear, there is tons of evidence that care is, in many respects, very sub-optimal in LMICs. We care about trying to improve it. But we think such dramatic results based on excessively reductionist analyses are simply not justifiable and in seeking attention in this way risk undermining broader support for the important goal of improving care in LMICs. In areas from global warming to mortality during the Iraq war we have seen the harm that marketing with unreliable methods and generalizing beyond the evidence can do to a good cause by giving fodder to those who don’t want to believe that there is a problem. What is needed are careful observations and direct measurements of care quality itself, along with evaluations of the cost-effectiveness of methods to improve care. Mortality is a crude measure of care quality.[4][5] Moreover, the extent to which healthcare reduces mortality is quite modest among older adults. The type of paper reported here topples over into marketing – it is as unsatisfying as a scientific endeavour as it is sensational.

— Richard Lilford, CLAHRC WM Director

— Timothy Hofer, Professor in Division of General Medicine, University of Michigan


  1. Kruk ME, Gage AD, Joseph NT, Danaei G, García-Saisó S, Salomon JA. Mortality due to low-quality health systems in the universal health coverage era: a systematic analysis of amenable deaths in 137 countries. Lancet. 2018.
  2. Rosling H. How Does Income Relate to Life Expectancy. Gap Minder. 2015.
  3. Pagano D, Freemantle N, Bridgewater B, et al. Social deprivation and prognostic benefits of cardiac surgery: observational study of 44,902 patients from five hospitals over 10 years. BMJ. 2009; 338: b902.
  4. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet. 2004; 363: 1147-54.
  5. Girling AJ, Hofer TP, Wu J, et al. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling studyBMJ Qual Saf. 2012; 21(12): 1052-6.

A framework for implementation science: organisational and psychological approaches

Damschroder and colleagues present a meta-analytic approach to development of a framework to guide implementation of service interventions.[1] They call their framework a “consolidated framework for implementation research”. Their approach is based on a review of published theories concerning implementation of service interventions. Since two-thirds of interventions to improve care fail, this is an important activity. They offer on over-arching typology of constructs that deal with barriers to effective implementation, and build on Greenhalgh’s monumental study [2] of factors determining the diffusion, dissemination and implementation of innovations in health service delivery. These frameworks are useful because they take an organisation-wide perspective and so psychological frameworks of individual behaviour change, such as the trans-theoretical [3] or COM-B [4] frameworks are subsumed within these frameworks. I proposed something similar with my “framework of frameworks”.[5]

In any event, the framework produced seems sensible enough. In effect it is an elaboration of the essential interactive dimensions of intervention, context and the process of implementation. Context can be divided into the external setting and the internal setting. This particular study goes further and ends up with five major domains, each broken up into a number of constructs – eight relating to the intervention itself.

This paper is carefully written and well researched, and is an excellent source of references to some of icons of the organisational research literature. But is it useful? And will it be the last such framework? I rather think the answer to these two questions is no. I once had a boss who said the important thing about science was ‘knowing what to leave out’! I think a much simpler framework would have sufficed in this case. Maybe I should have a go at producing one!

— Richard Lilford, CLAHRC WM Director


  1. Damschroder LJ, Aron DC, Keith RE, Kirsch SR, Alexander JA, Lowrey JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009; 4: 50.
  2. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004; 82: 581-629.
  3. Prochaska JO, Velicer WF. The transtheoretical model of health behaviour change. Am J Health Promot. 1997; 12(1): 38-48.
  4. Michie S, van Stralen M, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci. 2011; 6: 42.
  5. Lilford RJ. A Theory of Everything! Towards a Unifying Framework for Psychological and Organisational Change Models. NIHR CLAHRC West Midlands News Blog. 28 August 2015.

How Theories Inform our Work in Service Delivery Practice and Research

We have often written about theory in this News Blog.[1] [2] For instance, the ‘iron law’ of incentives – never use reward or sanction unless the manager concerned believes that they can influence the odds of success in reaching the required target.[3] [4] This law, sometimes called ‘expectancy theory’ was articulated by Victor Vroom back in 1964.[5] Here we review some of the theories that we have described, refined or enlarged over the last CLAHRC, and which we shall include among those we will pursue if we are successful in our Applied Research Collaborations (ARC) application. In each case we begin with the theory, then say how we have explicated it, and then describe how we plan to further develop theory through ongoing empirical work. Needless to say, our summaries are an impoverished simulacrum of the full articles:

  1. The theory of ‘hybrid managers’. It is well known that many professionals develop hybrid roles so that they toggle between their professional and managerial duties, and it is also known that tension can arise when the roles conflict. In our work we found that organisational factors can determine the extent to which nurses retain strong professional ethos when fulfilling managerial roles.[6] Simply put, the data seem to show that nurses working in more successful healthcare institutions tend to hew closer to their professional ethos than nurses in less successful units. It is reasonable to infer that an environment that can accommodate a strong professional orientation among hybrid managers is more likely to encompass the checks and balances conducive of safe care, than one that does not accommodate such a range of perspectives; most of us would choose to be treated in the environment where professional ethos is allowed a fair degree of expression. However, whether such a climate reflects better managers or a more difficult external environment is harder to discern. We now plan to examine this issue across many environments – for example, midwife hybrid managers balancing the need to expand choices of place of delivery with logistical limitations on doing so. Similarly, improving care for people with learning difficulties will require clinical managers to have freedom to innovate in order to improve services. Note that working with Warwick Business School enables us to locate our enquiries and theory development in the context of management in general, rather than just the management of health services. For example, the above study of nurse managers encompasses tax inspectors who now have to balance their traditional role in enforcing the tax code with one of helping the likes of us to make accurate declarations.
  2. Hybrid managers as knowledge brokers. Hybrid managers, it is known, act as a conduit between senior managers and frontline professionals, in mediating adoption of effective practice – i.e. knowledge brokering. It is also known that effecting change means overcoming structural, social and motivational barriers. The task of implementing state-of-the-art care practices is a delicate one and, prior to our research, the social dynamic of effecting change was poorly understood. In particular, the CLAHRC WM team wanted to study the role of status and perceived legitimacy in facilitating or inhibiting the knowledge brokers task. We found that hierarchies are critically important – safe care is more than following rules, but requires a degree of initiative (sometimes called discretional energy) by multiple actors across the hierarchy.[7] Nurses were often severely inhibited in using such personal initiative. The attitude of more senior staff is thus crucial in permitting, indeed encouraging, the use of initiative within a broader system of checks and balances. If the hierarchy within nursing is a barrier to progress, then that between doctors and nurses is a much bigger obstacle to uptake of knowledge. Moreover, there was also evidence of a difference barrier across different medical specialities with clinicians at the most action-oriented end of the spectrum (such as surgeons) showing lower levels of team-working than those with more reflective daily tasks (such as geriatricians). The work pointed towards the effectiveness of creating opportunities for purposeful interaction across these various types of hierarchical barriers – what the researchers called “dyadic relationships between hybrid middle managers with clinical governance responsibility and doctors through engagement and participation in medical-oriented meetings”; Elinor Ostrom would call this opportunities for ‘cheap talk’.[8] This work is crucial for laying the foundation for our work on the integration of care covering management of patients at the acute hospital / ambulance / community interface; care of patients with multiple diseases; care of the elderly; and the care of people with rare diseases, to mention but a few. Clearly, such opportunities for structured interaction are only parts of the story, and other factors that have been shown to be important (e.g. job design, performance management, education, patient empowerment, and data sharing) must be included in service improvement initiatives.
  3. Logics. Our third example concerns the unwritten assumptions that underpin what a person should do in their domain of work, and why they do it – so called ‘logics’. In a place like a hospital or university, many professions must co-exist, yet each will have a different ‘logic’. This idea applies across society, but CLAHRC WM investigator Graeme Currie wanted to examine how the professional logic and policy logic interact in a hospital setting.[9] The background to this study is the finding that policy logic has constrained and limited professional logic over the last few decades – doctors are no longer in charge of performance improvement, the management of waiting lists, etc. The researchers used the introduction of a new evidence-based, multi-component guideline as a lens through which to explore the interactions of different ‘logics’ in hospital practice. The implementation of a multi-component guideline is not a simple thing, and some intuitive cost-benefit calculations could justify, at least intellectually, massaging some aspects of the guideline to fit management practices rather than the reverse. However, the way this played out was not the same across contexts. As before, doctors were generally (but not invariably) less amenable to change than nurse practitioners with managerial responsibility. This study, published in a premier management journal,[9] identifies contingencies that will provide depth to our evaluations of different ways to reshape services. We will build on these insights when we examine a proposed service to use Patient Reported Outcome Measures, rather than simply elapsed time, to determine when patients should be seen in the outpatient department. An understanding of ‘logics’ is likely to come into play when we empower community and ambulance staff to elicit patient preferences and respect them even when to do so flies in the face of guidelines. At the level of the system, change is best viewed as an institutional problem of professional power and policy, around which change needs to orientate. It is not that systems and organisations can’t be changed, but subtle tactics and work may be required.[10] [11]
  4. Health care organisations viewed as political grouping and the need to do ‘political work’ when implementing interventions. Trish Greenhalgh has recently provided an evidence-based framework which unpicks reasons why IT implementations so often disappoint.[12] She points out that managers consistently underestimate the size of the task and the sheer difficulty of implementing IT systems so that they reach even some of their potential. Likewise, work conducted under an NIHR Programme grant that developed out of CLAHRC WM showed how new IT systems could introduce serious new hazards.[13] One of the methods to avoid failure in any large initiative, such as a large IT system, comes from a study of Italian hospitals conducted by the CLAHRC WM team,[14] advocating an iterative process, time and careful preparation of the ground by doing ‘political work’ to win hearts and minds and adapt interventions to context.[15] This type of approach will be critical to the development of complex interventions, such as those widening access to homebirth, and integrating patient feedback (including Patient Reported Outcome Measures) into patient care pathways.
  5. Absorptive capacity. Many CLAHRCs have relied on a knowledge brokering model to underpin translation of research, through which key individuals ensure knowledge gets to the right people at the right time to benefit patient care.[16] However, such an approach may have a limited effect and we need to consider how organisations and systems can be developed to ensure the efforts of knowledge brokers are leveraged and evidence informs patient care more widely. This is a matter of developing organisation and system ‘absorptive capacity’. Many of the implementation studies under our current CLAHRC have sought to develop co-ordination capability of organisations and systems to translate evidence into practice. For example, public and patient involvement, GP involvement, better business intelligence processes and structures is highlighted as ensuring clinical commissioning groups make evidence-informed decisions.[17] We have taken our work further to develop a ‘tool’ to assess the Absorptive Capacity of organisations.[18]

In this short review we have described how theoretical work, based on the development and evaluation of service interventions, can help understand the reasons why an intervention may succeed or fail, and how this may vary from place to place. Increasingly we are applying Elinor Ostrom’s work on collaboration between managers when the incentives are not aligned to the problems of integrated care in the NHS.[19] Our work represents successful collaboration between management and medical schools and, indeed, a difference in ‘logics’ between these organisations. This collaboration has taken time to mature, as have those between the services and academia more broadly. The essential point is that consideration of wider organisational and systems context will prove crucial to our efforts to continue broadening, accelerating and deepening translation of evidence into practice in our proposed ARC.

— Richard Lilford, CLAHRC WM Director

— Graeme Currie, Professor of Public Management, CLAHRC WM Deputy Director


  1. Lilford RJ. A Theory of Everything! Towards a Unifying Framework for Psychological and Organisational Change Models. NIHR CLAHRC West Midlands News Blog. 28 August 2015.
  2. Lilford RJ. Demystifying Theory. NIHR CLAHRC West Midlands News Blog. 10 April 2015.
  3. Lilford RJ. Financial Incentives for Providers of Health Care: The Baggage Handler and the Intensive Care Physician. NIHR CLAHRC West Midlands News Blog. 25 July 2015.
  4. Lilford RJ. Two Things to Remember About Human Nature When Designing Incentives. NIHR CLAHRC West Midlands News Blog. 27 January 2017.
  5. Vroom VH. Work and motivation. Oxford, England: Wiley. 1964.
  6. Croft C, Currie G, Lockett A. The impact of emotionally important social identities on the construction of managerial leader identity: A challenge for nurses in the English NHS. Organ Stud. 2015; 36(1): 113-31.
  7. Currie G, Burgess N, Hayton JC. HR Practices and Knowledge Brokering by Hybrid Middle Managers in Hospital Settings: The Influence of Professional Hierarchy. Hum Res Manage. 2015; 54(5): 793-812.
  8. Lilford RJ. Polycentric Organisations. NIHR CLAHRC West Midlands News Blog. 25 July 2014.
  9. Currie G & Spyridonidis D. Interpretation of Multiple Institutional Logics on the Ground: Actors’ Position, their Agency and Situational Constraints in Professionalized Contexts. Organ Stud. 2016; 37(1): 77-97.
  10. Currie G, Lockett A, Finn R, Martin G, Waring J. Institutional work to maintain professional power: Recreating the model of medical professionalism. Organ Stud. 2012; 33(7): 937-62.
  11. Lockett A, Currie G, Waring J, Finn R, Martin G. The influence of social position on sensemaking about organizational change. Acad Manage J. 2014; 57(4): 1102-29.
  12. Lilford RJ. New Framework to Guide the Evaluation of Technology-Supported Services. NIHR CLAHRC West Midlands News Blog. 12 January 2018.
  13. Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. W. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Qual Saf. 2017; 26: 542-51.
  14. Radaelli G, Currie G, Frattini F, Lettieri E. The Role of Managers in Enacting Two-Step Institutional Work for Radical Innovation in Professional Organizations. J Prod Innov Manag, 2017; 34(4): 450-70.
  15. Lilford RJ. Implementation Science at the Crossroads. BMJ Qual Saf. 2017; 27: 331-2.
  16. Rowley E, Morriss R, Currie G, Schneider J. Research into practice: Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for Nottinghamshire, Derbyshire and Lincolnshire (NDL). Implement Sci. 2012; 7:
  17. Croft C & Currie G. ‘Enhancing absorptive capacity of healthcare organizations: The case of commissioning service interventions to avoid undesirable older people’s admissions to hospitals’. In: Swan J, Nicolini D, et al., Knowledge Mobilization in Healthcare. Oxford: Oxford University Press; 2016.
  18. Currie G, Croft C, Chen Y, Kiefer T, Staniszewska S, Lilford RJ. The capacity of health service commissioners to use evidence: a case study. Health Serv Del Res. 2018; 6(12).
  19. Lilford RJ. Evaluating Interventions to Improve the Integration of Care (Among Multiple Providers and Across Multiple Sites). NIHR CLAHRC West Midlands News Blog. 10 February 2017.

Stunted Child Growth: How Good a Marker of Nutrition?

Around the world children, adjusted for age, are getting taller. When a child falls below two standard deviations of the reference mean height for a given age they are labelled as ‘stunted’. Given that children have been growing faster, even in poor countries, the incidence of stunting has decreased over the last four decades when compared to an unchanging reference standard. Using the WHO reference standard, for example, the prevalence of stunting in Asia decreased from 49% to 28% over the years 1990 to 2010. In Africa stunting rates remain stubbornly high at ~40%.[1] However, classifying 40% of any population as ‘abnormal’ should always raise a scientist’s suspicion.

The WHO reference is based on measurements across a mixture of rich and poor countries. So it is tempting to use it as a measure of infant nutrition; as nutrition improves so the prevalence of stunting should decline. It is tempting to infer that a well-nourished infant population would attain a stunting rate of 2.5%, even if judged against high-income norms. However, this argument is flawed – growth rates reflect not just each individual child’s nutrition, but the nutrition available to at least two preceding generations of the child’s family.[2] Targets should be based on what is achievable, and a statistically defined 2.5 percent ‘stunting’ rate is an unachievable target; the high-income threshold is unachievable within one generation. However, a poor country’s own 2.5% threshold is too low. In fact, there is no perfect external standard; they are all arbitrary.

We propose a ‘risk-adjusted’ stunting rate. One method would be based on the evidence of what can be achieved within one generation by examining the variances in height across generations, such as those in Japan, Taiwan and the Gulf states over a period of rapid economic progress and the years that followed. For instance, if two-thirds of the variance in growth rates is attributable to nutrition in a given generation, and one-third to inter-generational effects, then thresholds could be adjusted accordingly. A more refined method still could adjust for the height of the mother, since that also imposes a limit on growth rates. We think that the world should move to an empirically supported method of monitoring, rather than stunting rates based on an arbitrary standard that ignore intergenerational effects, which should henceforth be regarded as discredited.

— Richard Lilford, CLAHRC WM Director


  1. de Onis M, Blössner M, Borghi E. Prevalence and trends of stunting among pre-school children, 1990–2020. Public Health Nutr. 2012; 15(1): 142-8.
  2. Kaati G, Bygren LO, Pembrey M, Sjöström M. Transgenerational response to nutrition, early life circumstances and longevity. Eur J Hum Genet. 2007; 15: 784-90.

Our CLAHRC’s Unique Approach to Public and Community Involvement Engagement and Participation (PCIEP)

All NIHR-funded research is required to involve the public/patients at all stages of the research process. Here in CLAHRC WM we are ardent supporters of this principle, and we hew to the INVOLVE guidelines in doing so. We are keen to improve our ways of involving patients and the public in our research and have used the recently-published Standards for involvement to reflect on our activities and develop better ways of working.

CLAHRC WM is a Service Delivery Research Organisation. Because we are in the business of shaping the way health services are designed and delivered, we have given careful consideration to our approach to public and community involvement in research.  Our unique approach and rationale is described below.

Let’s start with a basic point. Service delivery research is, in most instances, best conducted prospectively. This is for three reasons:

  1. Prospective involvement of researchers provides access to the world’s literature, along with critical appraisal of that literature, to help inform the selection and design/adaptation of interventions.
  2. Researchers can assist in co-design and alpha-testing of proposed changes, deploying disciplines such as behavioural economics, operations research, and organisational theory.
  3. Prospective evaluations are generally more powerful (valid) than purely retrospective studies – for example, providing baseline data and information on both mediating, clinical processes and outcome variables.[1]

This takes us to the next basic point – service interventions are in the purvey of managers who control the purse strings, not the researchers. Yes, researchers can influence intervention selection and deployment, but they do not have the final say.

A third basic point is that service managers have a duty to consult patients and the public, just as researchers do.

We could have a model for public involvement with one set of patient/public advisors advising on research, and another set advising on interventions, as in in Figure 1.

Figure 1: Separate Involvement of PPI (PCIEP) in Research and in Service

108 DCB - Our CLAHRC Fig 1However, such a plan seems an opportunity missed. It could result, for instance, in conflicting advice, with patients/public in the research sector advocating evaluation of an intervention that their counterparts in the service have not prioritised.

We are not advocating combining PCIEP in research with patient and public involvement in the service to create just one monolithic structure. There are many research issues that are not relevant in a purely service context. We do, however, advocate an integrated approach, as represented in Figure 2. By involving patient and public contributors that are also involved in advising the service, we generate a group of people to champion our research and help ensure evidence is used in practice.

Figure 2: A system that integrates patient and the public across the service and research domains

108 DCB - Our CLAHRC Fig 2

So what can we do to achieve this level of integration? We do not have all the answers, as this is an evolving idea. However, here is what we do in CLAHRC WM:

  1. We try to recruit public contributors who also have (or have had) a role in advising the service. We target them and give some preference to such people in our competitive selection process.
  2. We hold joint consultative sessions with service managers, our public contributors, and (when possible) those who advise the service. Such was the situation, for example, in the ‘Dragon’s Den’ events we held to select priorities for our forthcoming Applied Research Collaboration application.
  3. Working with PCIEP and service partners, we create structures where research PCIEP, service PCIEP (say from Healthwatch) and CLAHRC WM researchers work together. We have worked with Sustainability and Transformation Partnerships (STPs) and our local Academic Health Science Network (AHSN) to create these structures.

Our strategy has evolved over considerable discussion in the CLAHRC WM and we have ‘market tested’ our approach with Simon Denegri, the past head of INVOLVE. However, we welcome feedback, advice, and opinions from readers.

Those who wish to read more on our work and/or thoughts on patient and public involvement can do so by clicking here.

— Richard Lilford, CLAHRC WM Director

— Magdalena Skrybant, PPIE Lead


  1. Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end pointsBMJ2010; 341: c4413.

The Slow March of Epidemiology: From Disease Causation to Treatment to Service Delivery

Traditional epidemiology was concerned with the causes of disease – many of the great medical discoveries, from malaria to the effects of smoking, can be credited to classical epidemiology. The subject continues to make great strides thanks to modern developments, such as genome-wide association studies and Mendelian randomisation. Approximately 70 years ago Austin Bradford Hill ushered in the days of clinical epidemiology.[1] Epidemiological methods were used to study the diagnosis and treatment of disease, rather than simply the causes and prognosis. Randomised trials and systematic reviews became the ‘stock in trade’ of the clinical epidemiologist.

As more and more effective treatments were discovered, people started to worry about large variations in practice and in the quality of care. Service delivery health research and the ‘quality movement’ were born. Researchers naturally felt the need to measure quality. Progress was slow, however. First, quality improvement was initially dominated by management research; a subject that does not have a strong tradition of measurement, as I have reported elsewhere.[2] Second, the constructs that quality researchers were dealing with were much harder to measure than clinical outcomes. For example, an attempt was made to correlate the safety culture with standardised mortality rates across intensive care units. The result was null, but this might have resulted entirely from measurement error; mortality rates suffer from unavoidable signal to noise problems,[3] while the active ingredient in culture is hard to capture in a measurement.[4] As the subject of the quality of care seemed to become bogged down with measurement issues, the patient safety movement became dominant. Initially people focused on psychology and organisational science. However, no science can mature without, at some point, making its central concepts quantifiable. As Galileo (allegedly) said, “Measure what can be measured, and make measurable what cannot be measured.” So it became necessary to try to measure safety, and all the problems of quality measurement re-surfaced.

Most sensible people now realise that impatience does more harm than good; shortcuts lead nowhere and we simply have to work away, measuring and mitigating measurement error as bast we can. As stated, and as I have argued elsewhere,[5] clinical outcomes are insensitive to many service interventions. This is a lesson that those of us with a background in classical or clinical epidemiology have been slow to learn. Trying to copy clinical epidemiology, and to rely entirely on clinical endpoints, has driven service delivery research into two camps – qualitative researchers who eschew quantification, and quantitative researchers who want to apply rules of evidence that served them well in clinical research. However, there really is a third way. This method is based on observations across the causal chain linking intervention to clinical outcome. I have long argued that it is the pattern of data (qualitative and quantitative) across a causal chain that should be analysed.[5] Since then, people have started to pay attention, not just to the outcome at the human level, but also to mediating variables. More recently still, I have argued for the use of Bayesian networks to synthesise information from the causal chain in a particular study, along with evidence from reviews in salient topics.[6] Note that while coming from the same, realist, epistemology as ‘mixed-methods’ research, mediator variable analysis and Bayesian networking developed mixed-methods to another level, since they enable data of different sorts to be captured in a clinical outcome of sufficient importance to populate a decision model. The use of proxy outcomes acquired a bad reputation in clinical epidemiology. However, carrying this idea over into service delivery research is extremely limiting. It is also unscientific, since science is dependent on induction, and induction can only be carried out if the causal mechanisms behind the results obtained are understood.

— Richard Lilford, CLAHRC WM Director


  1. Hill AB. The environment and disease: Association or causation? Proc R Soc Med. 1965; 58(5): 295-300.
  2. Lilford RJ, Dobbie F, Warren R, Braunholtz D, Boaden R. Top-rated British business research: Has the emperor got any clothes? Health Serv Manage Res. 2003; 16(3): 147-54.
  3. Girling AJ, Hofer TP, Wu J, Chilton PJ, Nicholl JP, Mohammed MA, Lilford RJ. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf. 2012; 21(12): 1052-6.
  4. Mannion R, Davies H, Konteh H, Jung T, Scott T, Bower P, Whalley D, McNally R, McMurray R. Measuring and Assessing Organisational Culture in the NHS (OC1). 2008.
  5. Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end points. BMJ 2010; 341: c4413.
  6. Watson SI & Lilford RJ. Essay 1: Integrating Multiple Sources of Evidence: a Bayesian Perspective. In: Challenges, solutions and future directions in the evaluation of service innovations in health care and public health. Southampton (UK): NIHR Journals Library, 2016.

Sustainability and Transformation Partnerships: Why they are so Very Interesting

There is a strong international, national and local initiative to develop services generically by integrating care across multiple providers and many diseases, rather than to focus exclusively on disease ‘silos’. However, integrating care across providers runs into immediate problems because the interests of these different providers are seldom aligned. For instance, providing care in the community may reduce earnings in a hospital where money follows patients.

Integrating care across multiple providers can take different forms, which might play out in different ways. The least radical solution would consist of informal alliances to help plan services. At the other end of the scale organisations merge into common legal entities with consolidated budgets (so-called Responsible Care organisations). Between these two extremes lie formal structures, but where the budgets and legal responsibility remain with local providers.

The Sustainability and Transformation Partnerships (STPs) in England are a good example of the intermediate arrangement. They are part of official government policy, have some funding, and have generated considerable local buy-in.

However, the interests of local providers cannot be overridden by the STP. It is tempting to say that they are unlikely to be very successful given that, inevitably, the interests of the different organisations are not the same. However, there is some evidence that this might not be the inevitable, dismal outcome. The evidence comes from Elinor Ostrom, Nobel Prize winner for economics. We have cited her work previously within this news blog.[1][2] She describes the conditions under which collaboration can take place, even when the interests of the collaborating organisations are imperfectly aligned:

  1. Clearly defined boundaries.
  2. Congruence between appropriation/provision rules and local conditions.
  3. Collective choice arrangements.
  4. Monitoring.
  5. Graduated sanctions.
  6. Conflict-resolution mechanisms.
  7. Local autonomy.
  8. Nested enterprises (polycentric governance).

Ostrom’s work was carried out in the context of protection of the environment; fisheries, farms, oceans, forests and the like. So, it would be extremely interesting to examine STP using Ostrom’s findings as an investigative lens. Working with CLAHRC London we plan to conduct numerous case studies of STPs that exhibit different features or philosophies. We expect that we will uncover differences in structure and culture that play out differently in different places. Among other things, we will see whether we can replicate Ostrom’s findings in a health care context. On this basis, we may be able to develop a tool that could help predict how well an organisation, such as an STP, is working. In the long-term we would examine (any) correlation in adherence with Ostrom’s criteria and the overall success of an STP.

Of course, this is not an easy topic for study. That is precisely why we think it is a good topic for a capacity development centre, such as a CLAHRC, to tackle. There is an inverse relationship between the importance of a topic, and its tractability. This is where various tools that we have developed, such as Bayesian networks, come into their own. These tools make intractable subjects, such as evaluating the success of STPs, a little more tractable.

— Richard Lilford, CLAHRC WM Director


  1. Lilford RJ. Polycentric Organisations. NIHR CLAHRC West Midlands News Blog. 25 July 2014.
  2. Ostrom E. Governing the Commons: The Evolution of Institutions for Collective Actions. Cambridge University Press: Cambridge, UK; 1990.

Health Economics and Access to Care: Are We Using the Wrong Model?

I woke one morning, many years ago, to the voice of a famous economist sounding off on my bedside radio. He spoiled the equanimity of my morning with his argument that the value of primary care should be evaluated by comparing the costs of the service with the health gain achieved by the service (in terms of quality adjusted life years [QALYs]). That is cobblers! Quite apart from the facile idea that the health gain from primary care can be calibrated with any kind of accuracy, the economist’s health economic model bypasses much of the purpose of healthcare. In this model, health care is simply as an instrument to improve health status. But a little thought will immediately show that health gain is a very incomplete understanding of the reason that people consult doctors. Health care serves a deep psychological need; human beings have turned to healers from the time that we became human beings. Health practitioners are not only valuable for the health gain they can now achieve, but also because they provide human warmth and support. The need for comfort, information, magic, and cure are all entangled. Not only do we need someone to turn out at times of mental or physical distress, but crucially, we also need to know that someone will be there for us when our time comes. And we need this assurance, even when we are perfectly healthy. We could perhaps wrap in the avoidance of catastrophic loss and call this the ‘insurance value’. Nor should the value of information – news about your own body – be underestimated. Berwick & Weinstein found that half of the benefit of an antenatal scan was simply to get a picture of the baby and had nothing to do with its medical purpose.[1]

The classical health economic model of cost utility analysis is well adapted to rationing demand once the patient’s condition has been defined. At that point calculating the relative value of different treatment options is a relatively straightforward issue (Figure 1). However, calculating the return-on-investment from simply providing access to healthcare is a different matter altogether. First, there is the extraordinarily difficult instrumental question of how to hypothecate the treatment effect over the full range of health conditions (Figure 2). Second, there is a need to factor in the value of:

  1. Information.
  2. Solace, comfort, support.
  3. Knowing that access will be available when required – the insurance value.

At the very least, it should recognise that cost-utility analysis for a calculation of a QALY or a DALY is not up to the task. The topic of access is one area where health economics raises many unsolved problems. In a recent news blog we discussed another issue that exposes some of the deep philosophical conundrums at the heart of health economics – the thorny issue of infertility.[2]

Fig 1. Management of a Specific Condition: a task for standard health economics

105 DCB - Health economics Fig 1

Fig 2. Providing Access to Healthcare: Health benefits are diffuse and hence hard to capture

105 DCB - Health economics Fig 2 

— Richard Lilford, CLAHRC WM Director


  1. Berwick DM, Weinstein MC. What do patients value? Willingness to pay for ultrasound in normal pregnancy. Med Care. 1985; 23(7): 881-93.
  2. Lilford RJ. The Health Economics of Infertility Treatment. NIHR CLAHRC West Midlands News Blog. 9 March 2018.

A Framework to Improve Access to Acute Care in Low- and Middle-Income Countries

Universal healthcare is an important goal in global health, as described in previous News Blogs.[1] Key to the concept of universal healthcare is the question of access to health. I lead one of three work packages in the NIHR Global Health Unit for Surgery, directed by Dion Morton. Along with Dr Dmitri Nepogodiev, I have recently returned from a series of meetings with influential doctors, policy-makers and community leaders in Oyo and Osun states in Nigeria. Our host was Dr Wally Adisa of Ife University, to whom we extend our sincere thanks.

Dmitri has reviewed the literature on how barriers may be overcome and access to healthcare facilitated. In Figure 1 we have attempted to synthesise the barriers from the literature and meetings in Nigeria. We use the famous Grossman model,[2] which recognises four phases on the pathway linking symptoms to effective treatment: recognition of the need for help; seeking help; transport to a place where appropriate care can be delivered; and then obtaining care in the healthcare institution. Many of the barriers we have identified could have been discerned though intuition: lack of money; poor understanding of disease and how it can be remedied; reliance on traditional healers; etc. However our investigations have identified certain factors we had not anticipated. For instance, many people are reluctant to call an ambulance, even when available, because they are superstitious of entering such a vehicle.[3]

While poverty is an important factor, limiting access, people who need acute care usually make it to hospital eventually. When we probe the reasons for delay among people who eventually did make it to hospital, we find that delay occurs because resources are not in the right place at the right time. In countries with a low tax base, more use should be made of existing networks and community ‘assets’ , to short-cut the barriers.[4] [5]

Figure 1. Barriers and Facilitators on the Pathway to Acute Care

104 DCB - A Framework to Improve Access to Care in Low - Figure - v2

During our meetings, and in the literature, there is much agreement about what the barriers are: poverty, superstitious beliefs, perceptions (not always erroneous) of poor care in hospitals, lack of facilities with need for further transfer, and so on. Where there is much less agreement, between stakeholders and within the literature, is on the relative importance of the various factors or how they may vary by clinical scenario – obstetric emergency, acute abdomen, trauma, childhood illness, and so on. While Desmond (personal communication) found severe constraints in access due to inadequate transport or ability to pay for transport (phase 3 in Grossman’s model) in a paediatric context in Malawi, Orji found that all the problems were in phase 1 and 2 or 4 in an obstetric context in Nigeria.[6] It is also known that no emergency transport systems are available in 33 (61%) of 54 African countries that answered a recent survey, and many only covered trauma or obstetric emergencies, while few were country-wide. Overall, only 8.7% of the population need could be met.[7] Nor should it be assumed that transport costs are negligible compared to health costs.[8] It is lack of facilities for transport that is the most important problem, not poor roads or hospitals too widely dispersed. Sheer distance can be a problem in some countries, such as Sudan, and lack of roads in other places, such as Ethiopia, but over two-thirds of Africa’s population live within two hours of hospital.[9]

We have produced a list of possible measures to improve access, classified according to whether they stimulate demand or supply. None of these interventions are easy to implement or evaluate. For this reason we plan to engage stakeholders to see what might be feasible, review the literature on what has been tried before, and then develop a health economic model to evaluate the cost-effectiveness of different potential solutions. In a future News Blog we will describe our approach to health economic modelling of this complex, but important, topic.

Factors that may be Tackled by Interventions to Improve Access

Demand for transfer Supply of the means for transfer
Knowledge of treatable illness, such as meningitis, typhoid, snakebite. Tackled through education. For example, messages targeted at misconceptions were massively influential in eliminating the Ebola epidemic. Many superstitions and beliefs are cultural, so different messages will be needed in different places. People can be influenced through local community and religious leaders, as well as through feedback from people who have experienced services.[10] Since most people do reach services, promotion of risk-sharing community schemes or ‘electronic’ wallets to provide resources when and where needed. Women’s participatory groups can also encourage autonomy, making women less reliant on husbands for money or permission.
Use of eHealth in general and eConsulting in particular to help translate awareness of symptoms to the intention to seek help. Public / NGO provision of inexpensive motorcycle taxis, successfully used for labour care in Sierra Leone [11] and Malawi.[12]
Interaction with traditional healers to recognise illnesses responsive to ‘modern medicine’. Encouraging / investing in small enterprises to promote transport, e.g. Uber-taxi style ambulances deployed in Nairobi.[13]

— Richard Lilford, CLAHRC WM Director & Dmitri Nepogodiev, Doctoral Research Fellow in Public Health.


  1. Lilford RJ. A Heretical Suggestion! NIHR CLAHRC West Midlands News Blog. 9 February 2018.
  2. Grossman M. The demand for health: A theoretical and empirical investigation. Cambridge, MA: NBER Books; 1972.
  3. Wilson A, Hillman S, Rosato M, Skelton J, Costello A, Hussein J, MacArthur C, Coomarasamy A. A systematic review and thematic synthesis of qualitative studies on maternal emergency transport in low- and middle-income countries. Int J Gynecol Obstet. 2013; 122: 192-201.
  4. Nielson K, Mock C, Joshipura M, Rubiano AM, Rivara F. Assessment of the Status of Prehospital Care in 13 Low- and Middle-Income Countries. Prehosp Emerg Care. 2012; 16(3): 381-9.
  5. Lilford RJ. Pre-Payment Systems for Access to Healthcare. NIHR CLAHRC West Midlands News Blog. 18 May 2018.
  6. Orji EO, Ogunlola IO, Onwudiegwu U. Brought-in maternal deaths in south-west Nigeria. J Obstet Gynaecol. 2002; 22:4:385-8.
  7. Mould-Millman N-K, Dixon JM, Sefa N, Yancey A, Hollong BG, Hagahmend M, Ginde AA, Wallis LA. The State of Emergency Medical Services (EMS) Systems in AfricaPrehosp Disaster Med. 2017; 32(3):273-83.
  8. Jan S, Laba T-L, Essue BM, Gheorghe A, Muhunthan J, Engelgau M, Mahal A, Griffiths U, McIntyre D, Meng Q, Nugent R, Atun R. Action to address the household economic burden of non-communicable diseases. Lancet. 2018; 391:2047-58.
  9. Ouma PO, Maina J, Thuranira PN, Macharia PM, Alegana VA, English M, Okiro EA, Snow RW. Access to emergency hospital care provided by the public sector in sub-Saharan Africa in 2015: a geocoded inventory and spatial analysis. Lancet Glob Health. 2018; 6:e342-50.
  10. Mould-Millman N-K, Rominski SD, Bogus J, Ginde AA, Zalariah AN, Boatemaah CA, Yancey AH, Akoriyea SK, Campbell TB. Barriers to Accessing Emergency Medical Services in Accra, Ghana: Development of a Survey Instrument and Initial Application in Ghana. Glob Health. 2015; 3(4):577-90.
  11. Bhopal SS, Halpin SJ, Gerein N. Emergency Obstetric Referral in Rural Sierra Leone: What Can Motorbike Ambulances Contribute? A Mixed-Methods Study. Matern Child Health J. 2013; 17: 1038-43.
  12. Hofman JJ, Dzimadzi C, Lungu K, Ratsma EY, Hussein J. Motorcycle ambulances for referral of obstetric emergencies in rural Malawi: Do they reduce delay and what do they cost? Int J Gynecol Obstet. 2008; 102: 191-7.
  13. Moh C. How a speedy emergency services app is saving lives. BBC News. 24 November 2017.

Interim Guidelines for Studies of the Uptake of New Knowledge Based on Routinely Collected Data

CLAHRC West Midlands and CLAHRC East Midlands use Hospital Episode Statistics (HES) to track the effect of new knowledge from effectiveness studies on implementation of the findings from those studies. Acting on behalf of CLAHRCs we have studied uptake of findings from the HTA programme over a five year period (2011-15). We use the HES database to track uptake of study treatments where the use of that treatment is recorded on the HES database – most often these are studies of surgical procedures. We conduct time series analyses to examine the relationship between publication of apparently clear-cut findings and the implementation (or not) of those findings. We have encountered some bear traps in this apparently simple task, which must be carried out with an eye to detail. Our work is ongoing, but here we alert practitioners to some things to look out for based on the literature and our experience. First, note that the use of time series to study clinical practice based on routine data is both similar and different from the use of control charts in statistical process control. For the latter purpose, News Blog readers are referred to the American National Standard (2018).[1] Here are some bear-traps/issues to consider when using databases for the former purpose – namely to scrutinise databases for changes in treatment for a given condition:

  1. Codes. By a long way, the biggest problem you will encounter is the selection of codes. The HTA RCT on treatment of ankle fractures [2] described the type of fracture in completely different language to that used in the HES data. We did the best we could, seeking expert help from an orthopaedic surgeon specialising in the lower limb. Some thoughts:
    1. State the codes or code combinations used. In a recent paper, Costa and colleagues did not state all the codes used in the denominator for their statistics on uptake of treatment for fractures of the lower radius.[3] This makes it impossible to replicate their findings.
    2. Give the reader a comprehensive list of relevant codes highlighting those that you selected. This increases transparency and comparability, and can be included as an appendix.
    3. When uncertain, start with a narrow set of codes that seem to correspond most closely to indications for treatment in the research studies, but also provide results for a wider range – these may reflect ‘spill-over’ effects of study findings or miscoding. Again, the wider search can be included as an appendix, and serves as a kind of sensitivity analysis.
    4. If possible, examine coding practice by examining local databases that may contain detailed clinical information with the routine codes generated by that same institution. This provides empirical information on coding accuracy. We did this with respect to use of tight-fitting casts to treat unstable ankle fracture (found to be non-inferior to more invasive surgical plates [4]) and found that the procedure was coded in different ways. We combined these three codes in our study, although this increases measurement error (reducing the signal) on the assumption that these codes are not specific.
  2. Denominators.
    1. In some cases denominators cannot be ascertained. We encountered this problem in our analysis of surgery for oesophageal reflux, where surgery was found more effective than medical treatment.[5] The counterfactual here is medical therapy that can be delivered in various settings and that is not specific for the index condition. Here we simply had to examine the effects of the trial results on the number of operations carried out country-wide. Seasonal effects are a potential problem with denominator-free data.
    2. For surgical procedures, the procedure should be combined with the counterfactual procedure from the trial to create a denominator. The denominator can also be expanded to include other procedures for the same operation if this makes sense clinically.
  3. Data-interval. The more frequent the index procedure, then the shorter the appropriate interval. If the number of observations falls below a certain threshold, then the data cannot be reported to protect patient privacy, and a wider interval must be used. A six month interval seemed suitable for many surgical procedures.
  4. Of protocols and hypotheses. We have found that the detailed protocol must emerge as an iterative process including discussion with clinical experts. But we think there should be a ‘general’ prior hypothesis for this kind of work. So we specified the dates of publication of the HTA report as our pre-set time point – the equivalent of the primary hypothesis. We applied this date line for all of the procedures examined. However, solipsistic focus on this data line would obviously lead to an impoverished understanding, so we follow a three phase process inspired by Fichte’s thesis-antithesis-synthesis-thesis model [6]:
    1. We test the hypothesis that a linear model fits the data using a CUSUM (cumulative sum) test. The null hypothesis is that the cumulative sum of recursive residuals has an expected value of 0. If it wanders outside the 95% confidence band at any point in time, this indicates that the coefficients have changed and a single linear model does not fit the data.
    2. If the above test indicates a change in the coefficients, we use a Wald test to identify the point at which the model has a break. We estimate two separate models before and after the break data and the slopes/intercepts are compared.
    3. Last we ‘check by members’ and discuss with experts who can fill us in on when guidelines emerged and when other trials may have been published – ideally a literature review would complement this process.
  5. Interpretation. In the absence of contemporaneous controls, cause and effect inference must be cautious.

This is an initial iteration of our thoughts on this topic. However, increasing amounts of data are being captured in routine systems, and databases are increasingly constructed in real time since they are used primarily as a clinical tool. So we thought it would be helpful to start laying down some procedural rules for retrospective use of data to determine long-term trends. We invite readers to comment, enhance and extended this analysis.

— Richard Lilford, CLAHRC WM Director

— Katherine Reeves, Statistical Intelligence Analyst at UHBFT Health Informatics Centre


  1. ASTM International. Standard Practice for Use of Control Charts in Statistical Process Control. Active Standard ASTM E2587. West Conshohocken, PA: ASTM International; 2018.
  2. Keene DJ, Mistry D, Nam J, et al. The Ankle Injury Management (AIM) trial: a pragmatic, multicentre, equivalence randomised controlled trial and economic evaluation comparing close contact casting with open surgical reduction and internal fixation in the treatment of unstable ankle fractures in patients aged over 60 years. Health Technol Assess. 20(75): 1-158.
  3. Costa ML, Jameson SS, Reed MR. Do large pragmatic randomised trials change clinical practice? Assessing the impact of the Distal Radius Acute Fracture Fixation Trial (DRAFFT). Bone Joint J. 2016; 98-B: 410-3.
  4. Willett K, Keene DJ, Mistry D, et al. Close Contact Casting vs Surgery for Initial Treatment of Unstable Ankle Fractures in Older Adults. A Randomized Clinical Trial. JAMA. 2016; 316(14): 1455-63.
  5. Grant A, Wileman S, Ramsay C, et al. The effectiveness and cost-effectiveness of minimal access surgery amongst people with gastro-oesophageal reflux disease – a UK collaborative study. The REFLUX trial. Health Technol Assess. 2008; 12(31): 1–214.
  6. Fichte J. Early Philosophical Writings. Trans. and ed. Breazeale D. Ithaca, NY: Cornell University Press, 1988