How Effective Are Management Consultants?

CLAHRC WM collaborator Ian Kirkpatrick (University of Warwick) reports an interesting article on the effectiveness of management consultants in the NHS hospitals in England.[1] Across all such hospitals the mean yearly spend is £1.2m. I know of only one RCT of use of management consultants.[2] This was a study of garment manufacturing companies in India where the use of management consultants was associated with an upturn in productivity. The retrospective study of Kirkpatrick, et al. reaches the opposite conclusion. Their explanatory variable is deployment of a management consultant, and their outcome variable is a change in efficiency before and after the intervention. They also make use of the fact that different hospitals have deployed management consultants at different times, which strongly mitigates against a temporal trend in the intervention effectiveness. As I understand it each hospital acts as its own control, and these differences are then amalgamated across all hospitals. This mitigates (but does not eliminate) selection bias.[3] The authors are careful to allow for autocorrelation, that is lack of independence between the outcome variable within hospitals, and they adjust for all the expected covariates, such as hospital size and teaching status. The efficiency measure was derived from a publically available database comparing the average unit cost for providing diagnosis and treatment of the trust to the national average.

This is a unique and extremely provocative study. However, we need to be very careful in jumping to a cause and effect conclusion. Firstly, large regression-based studies should be interpreted cautiously, since important confounding variables may be omitted, and it is impossible to take into account all interactions (and first and higher orders). Second, we also need to consider reverse causality; it is possible that deployment of management consultants was prompted by managers’ pre-emptive response to challenges. All of that being said, I have not always been persuaded by the value of management consultants during the various director and non-executive director roles I have occupied. The management consultant model is rather different to the CLAHRC model. CLAHRCs make sure all relevant literature is taken into account, we explicate the causal pathways that may lead to both good and bad outcomes (pre-implementation testing / prospective evaluation), and we conduct proof of principle studies as a prelude to evaluation of larger interventions. In short, our approach is more sceptical.

— Richard Lilford, CLAHRC WM Director

References:

1. Kirkpatrick I, Sturdy AJ, Alvarado N, Blanco-Oliver A, Veronesi G. The impact of management consultants on pubic service efficiency. Policy & Politics. 2018.
2. Bloom N, Eifert B, Mahajan A, McKenzie D, Roberts J. Does Management Matter? Evidence from India. Q J Econ. 2013; 128(1):1-51.
3. Brown C, Hofer T, Johal A, Thomson R, Nicholl J, Franklin BD, Lilford RJ. An epistemology of patient safety research: a framework for study design and interpretation. Part 2. Study design. Qual Saf Health Care. 2008. 17;162-9.

Future Trends in NHS

The future of health care is often conceptualised in terms of improved treatments emerging from the bio-medical science base – for instance increasing the precision with which particular therapies can be targeted. Many of these advances in the effectiveness of care will have supply side consequences in terms of cost and some will require service re-configuration – regenerative medicine and bed-side diagnostics, for example. However the larger challenges are likely to originate from increased demand. The service will have to adapt to these supply and demand side changes. This blog considers the role of applied research in informing these adaptations in order to improve the overall effectiveness and efficiency of services.

We discern three trends which, absent a major perturbation such as international conflict, will alter demand over the medium to long term. The time horizon for our analysis is the next quarter century, given that the longer the time horizon the wider the variance in any predictions.

The trends are as follows:

1. The population demographic will continue towards higher proportions of elderly people.
2. The dependency ratio (ratio of working age to young and retired people) will become increasingly adverse.
3. Demand for services per capita will increase.

None of these assumptions is unarguable as they involve outcomes that have not yet been observed. They are ordered from least to most contentious.

1. That the population will continue to age is almost a given, but the rate at which it will do is less certain. Some predict that over a third of children alive now will reach a century. However, the rate of increase in life expectancy may slow as the large reductions in smoking related deaths are absorbed into the base-line. Immigration could affect population projections in ways that are hard to predict. The recent sudden increase in mortality among white middle-aged males in the USA,[1] but improvement in survival of low socio-economic group children in the same country,[2] shows how difficult projections can be. A recent demonstration of trends over two decades suggests that age-specific prevalence of dementias are reducing, arguably because risk factors for cardiovascular disease are also risk factors for dementia. This will not reduce the total prevalence of dementia, of course, if life expectancy continues to increase.[3] [4]
2. The worsening of the dependency ratio is almost a corollary of an ageing society, but again the extent to which this happens is less certain as the work force gradually internalises the notion that 65 years of age is not a biological watershed but a social convention.[5] But delayed retirement will not solve the problem of a deteriorating dependency ratio; absent a method to delay ageing, many types of work, such as aviation and mining, are simply not suitable for older people. In addition, as people work longer at the end of life; so policies are encouraging longer leaves of absence from work outside the home to care for young children. So, all things considered, the dependency ratio will become more adverse as a function of increased longevity. Note, Britain appears to be at an earlier stage in this transition than many other high-income countries, such as Japan and Germany, and the opportunity for immigration to mitigate the tendency is likely to be accentuated given recent events.
3. Demand for services contingent on an ageing population is somewhat controversial. A reasonable planning assumption is that people will be healthier at a given age but this will not completely mitigate the frailty of older people at a given age. In that case we must assume a rise in demand as the population ages, even if age-specific morbidity declines to some extent.

Implications for the NHS flow from the above. Demand for services will increase relative to resources. That is to say there will be more old people relative to working age people and there will be more frail people relative to the population and demand will outpace economic growth. All of this may be compounded by a tendency for old people to live in remote areas at a distance from major conurbations where health services are concentrated. However, this problem will be less acute than in most other countries.

There are many possible mitigations and the NIHR has a role in all of them; these are listed in the table below.

Factors to help the service cope with increasing demand.

Mitigating factor	How it might work	Caveats	Potential impact
Major technical advances that might affect demand.	A ‘cure’ or prevention for dementia would both improve the economy (and hence supply) while supressing demand.	Probably lies outside our 25 year time horizon. Will prolong life and hence increase the proportion of frail elderly people.	Potentially very high but out of scope. Medical advances more generally likely to increase demand by increasing longevity.
Self-care	An ‘extreme’ form of skill substitution.	Unlike other mitigations there is an extensive research literature. Beneficial for capable patients minimal impact on global demand.	The correct answer to improving care, reducing demand will require development of interventions and further research.
Information technology	Can make care safer and supply more efficient.	Full electronic notes disrupt patient communication in their current form. A lot more needs to be learned about the design and implementation of this deceptively complex technology.	Huge benefits in prospect but the socio-technical aspects require extensive development and research.
Robotics	May substitute for expensive/scarce human resources.[6]	Humans require the care and attention of other humans.	Moderate. Likely to assist rather than replace clinical input.
Skill substitution	Less expensive staff (physician’s assistants) substitute for more expensive (doctors). Increasingly feasible as health care increasingly codified.	Limited by the complexity of decision making in patients with many diseases.	Very hard to say without more research. May be modest.
Pro-active community services	Prevent deterioration to improve health and decrease admissions.	Existing research disappointing – may actually increase demand by identifying self- correcting illness.	Potentially great but we are in the foothills of discovery.

Mitigating demand is not easy in the face of the demographic factors mentioned above. It is often argued, even in official enquiries, that prevention is the key to reducing demand. While prevention may reduce demand arising from particular diseases, such as diabetes, survivors go on to develop further diseases on their trajectory to death.[7] It is therefore not at all clear that prevention will reduce total demand and it may even be the case that deferred demand is augmented demand. There are some potential mitigating possibilities. A prevention or cure for Alzheimer’s disease would make a massive difference. Less distant is an ‘artificial pancreas’ that might massively simplify diabetes care. Methods to make people independent, such as home telemetry, have had nugatory impact on demand to date,[8] but this may change in the future. Patient self-care is beneficial in improving healthcare and satisfaction,[9] but effects on total demand have been modest.

If supply side measures might help services cope with the consequences and demand continues to rise, then two points should be noticed. First, efficiency gains are notoriously difficult to achieve in service industries. Second, the likely increasingly adverse dependency ratio is likely to limit expansion in skilled staff. Partial solutions may lie in manufacturing, including robotics and information technology. Skill substitution is a future area where it may be possible to improve efficiency.[10] In particular, physicians assistants may reduce costs overall.[11] The research for skills or system substitution is not entirely positive – for example, substituting nurses for doctors may not improve efficiency because consultation times had to increase.[12] There is an international trend to provide more care at ‘grass roots’ by means of Community Health Workers (CHWs) – an area where high-income countries are learning from low- and middle-income countries.[13] CHWs have a large potential role in improving care – helping patients to adhere to medications, providing preventative services, identifying deteriorating patients. Their effect on reducing demand is less certain, and on occasion they may actually increase it.[14]

Readers may think that the CLAHRC WM Director can be rather pessimistic, even nihilistic. Not so, CLAHRC WM has recently conducted an overview (umbrella review) across 50 systematic reviews of different methods to integrate care across hospitals and communities.[15] Discharge planning with post-discharge support is highly effective. Multi-skill teams are much more effective if they include hospital outreach than if they are entirely community-based. Self-management is effective but mainly for single diseases. Case management is of minimal value. Across all intervention types, length of stay was reduced in over half, emergency admissions were reduced in half, and readmissions were reduced in nearly half. In almost no case did the intervention make any of the above outcomes worse. Costs to the service were reduced in over a third of intervention types, but the quality of evidence is poor on this point – a topic that is being addressed across all CLAHRCs. And here is the CLAHRC WM Director’s point; there are no quick wins and no silver bullets. And the solutions are not self-evident. Only by patiently trying out new things and evaluating them methodologically can things improve. It may sound self-serving, but that does not mean it is incorrect – CLAHRCs have an immense contribution to make to improve the effectiveness and cost-effectiveness of health services.

— Richard Lilford, CLAHRC WM Director

I acknowledge advice from Prof Peter Jones (University of Cambridge), Director of CLAHRC East of England, but the views expressed are entirely my own.

References:

1. Deaton A, Lubotsky D. Mortality, inequality and race in American cities and states. Soc Sci Med. 2003;56(6):1139-53.
2. Chetty R HN, Katz LF. The Effects of Exposure to Better Neighbourhoods on Children: New Evidence from the Moving to Opportunity Experiment. Am Econ Rev. 2016.
3. Matthews FE, Stephan BC, Robinson L, Jagger C, Barnes LE, Arthur A, Brayne C; Cognitive Function and Ageing Studies (CFAS) Collaboration. A two decade dementia incidence comparison from the Cognitive Function and Ageing Studies I and II. Nat Commun. 2016; 7: 11398.
4. Matthews FE, Arthur A, Barnes LE, Bond J, Jagger C, Robinson L, Brayne C; Medical Research Council Cognitive Function and Ageing Collaboration. A two-decade comparison of prevalence of dementia in individuals aged 65 years and older from three geographical areas of England: results of the Cognitive Function and Ageing Study I and II. Lancet. 2013; 382(9902): 1405-12.
5. Lilford R. Robotic hotels today – nursing homes tomorrow? NIHR CLAHRC West Midlands News Blog. March 6 2015.
6. Lilford R. Medical Technology – Separating the Wheat from the Chaff. NIHR CLAHRC West Midlands News Blog. February 26 2016.
7. Lilford R. Improving Diabetes Care. NIHR CLAHRC West Midlands News Blog. November 11 2016.
8. Henderson C, Knapp M, Fernández J-L, Beecham J, Hirani SP, Cartwright M, et al. Cost effectiveness of telehealth for patients with long term conditions (Whole Systems Demonstrator telehealth questionnaire study): nested economic evaluation in a pragmatic, cluster randomised controlled trial. BMJ. 2013; 346: f1035.
9. Tricco AC, Ivers NM, Grimshaw JM, Moher D, Turner L, Galipeau J, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet. 2012; 379: 2252–61.
10. Lilford R. The Future of Medicine. NIHR CLAHRC West Midlands News Blog. October 23 2015.
11. Lilford R. Improving Hospital Care: Not easy when budgets are pressed. NIHR CLAHRC West Midlands News Blog. January 23 2015.
12. Laurant M, Reeves D, Hermens R, Braspenning J, Grol R, Sibbald B. Substitution of doctors by nurses in primary care. Cochrane Database Syst Rev. 2005; 2(2).
13. Lilford R. Lay Community Health Workers. NIHR CLAHRC West Midlands News Blog. April 10 2015.
14. Roland M, Abel G. Reducing emergency admissions: are we on the right track? BMJ. 2012; 345: e6017.
15. Damery S, Flanagan S, Combes G. Does integrated care reduce hospital activity for patients with chronic diseases? An umbrella review of systematic reviews. BMJ Open. 2016; 6: e011952.

Mega- and Ultra-Trials

Randomised controlled trials (RCTs) are getting larger. Increased sample sizes enable researchers to achieve greater statistical precision and detect increasingly smaller effect sizes against diminishing baseline rates of the primary outcomes of interest. Thus over time, we are seeing an increase in the sample sizes of RCTs, leading to what may be termed mega-trials (>1,000 participants) and even ultra-trials (>10,000 participants). The below figure shows the minimum detectable effect size (in terms of difference from baseline) for a trial versus its sample size (with α = 0.05, β = 0.8 and a control group baseline of 0.1, i.e. 10%), along with a small selection of non-cluster RCTs from the New England Journal of Medicine published in the last three years. What this figure illustrates is that there is diminishing returns, in terms of statistical power, from larger sample sizes.

Figure 1. Minimum detectable effect size vs. sample size

Nevertheless, with great statistical power comes great responsibility. Assuming that the sample size is large enough that observation of a p-value greater than 0.05 is evidence that no (clinically significant) effect exists may lead to perhaps erroneous conclusions. For example, Fox et al. (2014) enrolled 19,102 participants to examine whether ivabradine improved clinical outcomes of patients with stable coronary artery disease.[1] The estimated hazard ratio for death from cardiovascular causes or acute myocardial infarction with the treatment was 1.08 but with a p-value of 0.2, and so it was concluded that ivabradine did not improve outcomes. However, we might see this as evidence that ivabradine worsens outcomes. A crude calculation suggests that the minimum detectable hazard ratio in this study was 1.14, and, for a sample of this size, the results suggest that almost 50 more patients died (against a baseline of 6.3%) in the treatment group. One might therefore actually see this as clinically significant.

Similarly, Roe et al. (2012) enrolled 7,243 patients to compare prasugrel and clopidogrel for acute coronary syndromes without revascularisation.[2] The hazard ratio for death with prasugrel was 0.91 with a p-value of 0.21. The authors concluded that prasugrel did not “significantly” reduce the risk of death. Yet, with the death rate in the clopidogrel group at 16%, a hazard ratio of 0.91, with a sample size this large, represents approximately 50 fewer deaths in the prasugrel group. Again, some may argue that this is clinically significant. Importantly, a quick calculation reveals that the minimum detectable effect size in this study was 0.89.

Many authors have warned against using p-values to decide on whether an intervention has an effect or not. Mega- and ultra-trials do not reduce the folly of using p-values in this way and may even exacerbate the problem by providing a false sense of confidence.

— Samuel Watson, Research Fellow

References:

1. Fox K, Ford I, Steg PG, Tardif J-C, Tendera M, Ferrari R. Ivabradine in Stable Coronary Artery Disease without Clinical Heart Failure. New Engl J Med. 2014; 371(12): 1091-99.
2. Roe MT, Armstrong PW, Fox KAA, et al. Prasugrel versus Clopidogrel for Acute Coronary Syndromes without Revascularization. New Engl J Med. 2012; 367: 1297-1309.

…And While Talking about Culture and Misbehaviour

A recent Lancet editorial addresses the retraction by BioMed Central of 42 articles published by medical researchers in China.[1] The fraudulent articles emanate from prestigious centres in many parts of the country. This information furnishes a possible explanation for the finding that effectiveness studies that provide null results in North America often provide positive results in China.[2]

— Richard Lilford, CLAHRC WM Director

References:

1. The Lancet. China’s medical research integrity questioned. Lancet. 2015; 385:1365.
2. Hartley LC, Girling AJ, Bowater RJ, Lilford RJ. A multi-study analysis investigating systematic differences in cardiovascular trial results between Europe and Asia. J Epidemiol Comm H. [ePub].