Numbers and the Doctor/Patient Relationship

I have always been interested in communicating scientific information and probability. A paper co-authored by CLAHRC WM colleague Eivor Oborn [1] therefore caught my eye. The paper concerns numbers and their ‘performativity’, by which the authors mean how the numbers affect doctors, patients, and the interaction between doctors and patients. They use medical consultations in a Swedish rheumatology clinic to explore the issue, since this is a ‘data-rich’ environment. By this I mean charts are used to plot long-run numerical data relating to patient-reported outcomes, medical assessments, and laboratory data. The study shows that the numbers have high salience for patients who generally find graphical representation of long-run data useful. Doctors also find graphical display of trends useful in spotting threats to patient health. However, patients sometimes feel that the data on display take precedence over how they actually feel. That is to say, the doctor tends to focus on the numbers while the patient’s main symptom might not be captured in the numbers. Of course, there is no counterfactual, so how much of this dissatisfaction is caused by availability of numbers is uncertain. Also I felt that more could be said about the extent to which patients, and indeed doctors, really understand the meaning of the numbers they were seeing. Many people have poor numeracy skills and draw erroneous inferences from data. For instance, people tend to over-interpret improving trends following a run of high-values – the issue of regression to the mean, covered in the Method Matters section of a previous News Blog.

— Richard Lilford, CLAHRC WM Director

Reference:

  1. Essén A & Oborn E. The performativity of numbers in illness management: The case of Swedish Rheumatology. Soc Sci Med. 2017; 184: 134-43.

The Problem with ‘Red, Amber, Green’

I have always thought that the so-called traffic light system, which classifies service quality as ‘red, amber, green’ is so crude as to be otiose. A recent article in BMJ Quality and Safety explicates the logic behind my intuition.[1] There are two problems with the so-called traffic light system. First, it focuses just on one episode in time and says nothing about trends. Second, it (usually) relies on thresholds set externally. For both of these reasons, the authors argue (and I agree) that the traffic light system is inimical to safety. It does not distinguish between common cause variation (play of chance) and special cause variation (likely to be due to some specific, potentially remediable factor). The point is made that tackling common cause variation, even if it comes up red against some externally set threshold, is likely to lead nowhere. If one wants to improve outcomes from systems that are fluctuating randomly, then it is necessary to look for a common cause, not a cause specific to a particular time and space. Control charts analyse trends and hence distinguish between common cause variation and special course variation; the latter requires a focussed approach. For instance most English Accident and Emergency departments would show up red if judged against the four hour waiting time target. A red rating does not therefore suggest a problem specific to a particular hospital, but failure across the hospital system. It therefore needs a systemic approach across hospitals. This would be self-evident if a funnel plot were used. Such a chart would distinguish outliers where a targeted diagnosis and intervention would be appropriate from the generality of hospitals where a more systematic approach is more likely to bear fruit. CLAHRC WM is trying to enhance uptake of control charts by hospitals based on our previous work that shows they are seldom used.[2]

— Richard Lilford, CLAHRC WM Director

References:

  1. Anhøj J, Hellesøe A-MB. The problem with red, amber, green: the need to avoid distraction by random variation in organisational performance measures. BMJ Qual Saf. 2017; 26: 81-4.
  2. Schmidtke KA, Poots AJ, Carpio J, Vlaev I, Kandala N-B, Lilford RJ. Considering chance in quality and safety performance measures: an analysis of performance reports by boards in English NHS trusts. BMJ Qual Saf. 2017; 26: 61-9.

The Beneficial Effects of Taking Part in International Research: an Old Chestnut Revisited

Two recent and well-written articles grapple with this question of whether or not clinical trials are beneficial, net of any benefit conferred by the therapeutic modalities evaluated in those trials.[1] [2]

The first study from the Netherlands concerns the effect of taking part in clinical trials where controls are made up of people not participating in trials (presumably because they were not offered entry in the trial).[1] This is the topic of a rather extensive literature, including a study to which I contributed.[3] The latter study found that the putative ‘trial effect’ applied only in circumstances where care given to control patients was not protocol-directed. In other words, our results suggested that the ‘trial effect’ was really a ‘protocol effect’. In that case the effect should be ephemeral and disappear as greater proportions of care become protocolised. And that is what appears to have happened – Lin, et al.[1] report no benefit to trial participants versus non-trial patients for the highly protocolised disease Hodgkin lymphoma. They speculate that while participation in trials does not affect individual patient care in the short-term, hosting trials does sensitise clinicians at an institutional level, so that they are more likely than clinicians from non-participating hospitals to practice evidence-based care. However, they offer no direct evidence for this assertion. Such evidence is, however, provided by the next study.

The effects of high participation rates in clinical trials at the hospital level is evaluated in an elegant study recently published in the prestigious journal ‘Gut’.[2] The team of authors (that includes prominent civil servants and many distinguished cancer specialists and statisticians) compared outcomes from colon cancer according to the extent to which the hospital providing treatment participated in trials. This ingenious study was accomplished by linking the NIHR’s data on clinical trials participation to cancer registry data and Hospital Episode Statistics. It turned out that risk-adjusted survival was significantly better in the high participation hospitals than in lower participation hospitals, even after substantial risk-adjustment. “Residual confounding” do I hear you say? Perhaps, but the authors have two further lines of evidence for the causal explanation. First, they documented a dose-response; the greater the level of participation, the greater the improvement in survival. Of course, an unknown confounder that was correlated with participation rates would produce just such a finding. The second line of evidence is more impressive – the longer the duration over which a hospital had sustained high participation rates, the greater the effect. Again, of course, this argument is not impregnable – duration might not serve as a good Instrumental Variable. How might the case be further strengthened (or refuted)? By unravelling the theoretical pathway between explanatory and outcome variables.[4] Since this is a database study, the process variables that might mediate the putative effect were not available to the authors. However, separate studies have indeed found an association between improved processes of care and trial participation.[5] Taken in the round, I think that a cause/effect explanation holds (>90% of my probability density favours the causal explanation).

— Richard Lilford, CLAHRC WM Director

References:

  1. Liu L, Giusti F, Schaapveld M, et al. Survival differences between patients with Hodgkin lymphoma treated inside and outside clinical trials. A study based on the EORTC-Netherlands Cancer Registry linked data with 20 years of follow-up. Br J Haematol. 2017; 176: 65-75.
  2. Downing A, Morris EJA, Corrigan N, et al. High hospital research participation and improved colorectal cancer survival outcomes: a population-based study. Gut. 2017; 66: 89-96.
  3. Braunholtz DA, Edwards SJ, Lilford RJ. Are randomized clinical trials good for us (in the short term)? Evidence for a “trial effect”. J Clin Epidemiol. 2001; 54(3): 217-24.
  4. Lilford RJ, Chilton PJ, Hemming K, Girling AJ, Taylor CA, Barach P. Evaluating policy and service interventions: framework to guide selection and interpretation of study end pointsBMJ. 2010; 341: c4413.
  5. Selby P. The impact of the process of clinical research on health service outcomes. Ann Oncol. 2011; 22(s7): vii2-4.

Payment by Results – a Null Result!

Reimbursement levels for medical care in large US hospitals are reduced by up to 2% if compliance with evidence-based clinical care standards falls below threshold levels. Does this result in improved care compared to control hospitals not exposed to the financial incentive? To find out, intervention hospitals were compared to control hospitals.[1] The ‘value based purchasing’ schemes were not introduced in a prospective experiment, and the controls (small rural hospitals) are very different in nature to those larger hospitals to whom the incentive applies. To mitigate potential bias, difference-in-difference approaches were used; hospitals were matched for previous performance; and the usual statistical adjustments were made. Adherence to appropriate clinical processes was increasing among both control and intervention hospitals before the intervention was implemented. Rates of adherence did not differ between intervention and control hospitals post-intervention. The clinical indicators related to three tracer conditions frequently used in studies of adherence to clinical standards – pneumonia, heart attack or heart failure. Patient experience measures also did not differ over intervention and controls, and while mortality was improved for pneumonia, it did not do so for the other conditions. The effect on pneumonia deaths was regarded as a chance finding (alpha error), given the null result on mediating variables (i.e. clinical process variables). Arguably these results were null because the incentive was low (only 2% of total reimbursement) and distributed over a large number of outcomes. Alternatively, doctors are largely intrinsically motivated and do not need financial incentives to moderate their performance. We will pick up on this issue in our next News Blog.

— Richard Lilford, CLAHRC WM Director

Reference:

  1. Ryan AM, Krinsky S, Maurer KA, Dimick JB. Changes in Hospital Quality Associated with Hospital Value-Based Purchasing. N Engl J Med. 2017; 376: 2358-66.

It is Really True: Detailed Analysis Shows that Social Media Really do Lead to Silo Thinking

It is popular to claim that social media sites (SMSs), such as Twitter and Facebook, democratise knowledge. This is true in one sense – it places information within the easy reach of the population at large. Individuals certainly benefit.[1] But what about society at large? Here the story is bleak. Far from bringing people into contact with ideas that might challenge their precepts, SMSs increase intellectual isolation. Walter Quattrociocchi, writing in Scientific American,[2] summarises the mountain of data that has been painstakingly collated by himself and others; they analysed data from two million Facebook users in one study.[3] People eschew views they find challenging and isolate their attention in online groups, which reinforces their pre-existing beliefs. It gets worse – the less educated a person, the more isolationist they tend to be. Such people hew to conspiracy theories, which grow like a snowball among online communities. Scientific analysis is shut out so that detailed analysis of data on topics such as climate change are less widely disseminated. Conspiracy theories proliferate, for example, saying that climate change is a hoax perpetuated to further academic careers and earnings of alternative energy suppliers. The very worst news is that campaigns aimed at debunking these myths actually reinforce belief in conspiracy theories; there is no antidote to the myths perpetuated down social media.

SMSs are here to stay, but as the author says, the Information Revolution is fostering an Age of Credulity not an Age of Enlightenment.

— Richard Lilford, CLAHRC WM Director

References:

  1. Lilford RJ. The Second Machine Age. NIHR CLAHRC West Midlands. 5 May 2017.
  2. Quattrociocchi W. Why Social Media Became the Perfect Incubator for Hoaxes and Misinformation. Scientific American. April 2017.
  3. Del Vicario M, Bessi A, Zollo F, et al. The Spreading of Misinformation Online. Proc Natl Acad Sci USA. 2016; 113(3): 554-9.

Providing Care at Less Cost – the Great Skill-mix Debate

Health care professionals do not all receive the same emoluments. In all countries doctors are paid the most. They carry the greatest responsibility for making decisions that affect people and they are the most likely to be sued – so their differential pay seems fair. But the other side of the coin is that non-doctor health professionals can do many things equally well, or perhaps better. Similarly, there are things that Community Health Workers can do as well or better than nurses, and again at lower unit cost. There are many types of skill mix initiative, and the most widely used classification emanated from Bonnie Sibbald,[1] herself a previous CLAHRC director:

Sibbald’s Skill-Mix Classification

Changing roles

  • Extending roles
  • Substituting – exchanging one type of worker for another
  • Delegation
  • Innovation – creating new jobs

Changing structures at the interface between services

  • Transferring service from one setting to another
  • Relocation
  • Liaison

There are a number of systematic reviews on skill-mix summarising a great many articles. However, review authors agree that there is little clear evidence on effectiveness or cost-effectiveness. Many studies concentrate on skill substitution, usually comparing doctors and nurses.[2] However, the subject is hard to study, and deriving generalisable conclusions is always going to be difficult because of differences in context – especially training. One cadre that has received a lot of attention over the last two decades involves innovation more than substitution – the use of Community Health Workers. They have a valuable role in prevention (e.g. malnutrition/vaccination), maintenance of therapy (e.g. HIV, TB and hypertension), and frontline care (e.g. rehydration therapy), as discussed in previous News Blogs.[3-5]

— Richard Lilford, CLAHRC WM Director

References:

  1. Sibbald B, Shen J, McBride A. Changing the skill-mix of the health care workforce. J Health Serv Res Policy. 2004; 9(s1):28-38.
  2. Antunes V & Moreira JP. Skill mix in healthcare: An international update for the management debate. Int J Healthc Man. 2013; 6(1): 12-7.
  3. Lilford RJ. Lay Community Health Workers. NIHR CLAHRC West Midlands News Blog. 10 April 2015.
  4. Lilford RJ. An Intervention so Big You Can See it from Space. NIHR CLAHRC West Midlands News Blog. 4 December 2015.
  5. Lilford RJ. Between Policy and Practice – the Importance of Health Service Research in Low- and Middle-Income Countries. NIHR CLAHRC West Midlands News Blog. 27 January 2017.

Government vs. Private Schools

CLAHRC WM is not just interested in health care since the methods we use are equally relevant to decision-makers in education, social services, industrial policy, criminology, and so on. We should all be learning from each other. In a previous blog I reported on the (mostly positive) results of the ‘Moving to Opportunity’ experiment in the USA, where families were given an opportunity to move from a deprived neighbourhood to a more salubrious one. So I was interested to spot an RCTs of vouchers that allowed children (over a wide age range) from government schools to attend private schools (also in the USA).[1] The experiment was recent (last five years) and we have outcomes at one year only. Seventy percent of pupils allocated a voucher to attend a private school took up their offer; so both intention to treat and per protocol analyses are reported. The educational outcomes were lower in the intervention group, and were statistically significantly lower for mathematics. This negative effect was greater if the voucher was taken up than if it was not. The negative effect was greater if the child came from a school that was not rated as poor performing than if the previous school was rated satisfactory or good. The negative effect was greatest if the child was in elementary school, and non-significantly positive if they were already in high school.

What caused the negative effect on educational outcomes? Simply moving school does not seem to explain the results, since a proportion of control children moved school with little or no apparent effect. However, private schools provide less instructional time than government schools, especially in elementary school. Other studies have also noted negative effects of moving children to private school on educational outcomes in the short term. But it is far too early to declare the intervention a failure. There is a limit to how much an elementary school child can assimilate, and it is the long-term effects that are important. However, I was surprised by this result – educational interventions have a habit of producing results different to those intended. Full marks to the US Congress, which had the wisdom to evaluate its own policies. The UK Cabinet Office has published a document arguing for more RCTs of policy,[2] and I expect to be able to report the results of further RCTs of educational interventions in the News Blog.

— Richard Lilford, CLAHRC WM Director

References:

  1. Dynarski M, Rui N, Webber A, Gutmann B, Bachman M. Evaluation of the DC Opportunity Scholarship Program. Impacts After One Year. Alexandria, VA: Institute of Education Sciences, 2017.
  2. Haynes L, Service O, Goldacre B, Torgerson D. Test, Learn, Adapt: Developing Public Policy with Randomised Controlled Trials. London: UK Cabinet Office, 2012.

Bariatric Surgery – Improve Five-Year Outcomes

The short-term (2-3 year) outcomes of bariatric surgery have been extensively studied in RCTs, and the (mainly) positive outcomes documented. Now we can use unbiased information to look a little further into the future as longer-term outcomes of an RCT of bariatric surgery have been published.[1] Outcomes were assessed at the age of five years among 150 overweight, type 2 diabetic people randomised to intensive medical therapy alone versus such therapy accompanied by bariatric surgery (by-pass or gastric size reduction). At five years the differences in HbA1c were massive – 2.1 points vs. 0.3 points improvement over baseline. The surgical group were thinner, had improved blood fats and reported better quality of life.

So what are the service implications of this iconic study? We need to ‘industrialise’ surgery, so that more operations can be done at a given cost. That means teams of technicians operating under consultant (anaesthetist and surgeon) supervision. Here is an artist’s representation of such a ‘Taylorised’ process:

An Operations Director oversees a Surgeon and two Anaesthetists who oversee six technicians operating on six patients

Such a process was established to improve access to cataract surgery in Moscow over three decades ago.[2] Similar processes have been used with respect to open heart surgery in the US. I propose we should conduct simulations and then move gradually and incrementally to a safe, but efficient, method of implementing high-throughput bariatric surgery. Evaluation of the roll-out would be essential. Barriers will need to be overcome, but one or two effective demonstration sites will speak more than volumes of words. My only real concern is that some new ‘technology’ will come along and sweep away bariatric surgery and all its (protesting) practitioners.

— Richard Lilford, CLAHRC WM Director

References:

  1. Schaeur PR, Bhatt DL, Kirwan JP, et al. Bariatric Surgery versus Intensive Medical Therapy for Diabetes – 5-Year Outcomes. New Engl J Med. 2017; 376: 641-51.
  2. Schmemann S. Moscow Eye Doctor Hails Assembly-Line Surgery at Clinic. The New York Times. 2 July 1985.

Wait Until Your Symptoms are Really Bad Before Having an Osteoarthritic Joint Replaced

Early studies of hip replacement showed that the operation was associated with large and sustained QALY gains, and hence that it was cost-effective. The same applied to knee replacement. The patients in these studies all had severe disease, as shown by their quality of life scores at baseline. But indications for both operations have gradually expanded so that the majority of current patients who receive the operation are atypical of those in the original studies. That is to say, their disease is less severe. The long-term benefits of hip replacement have been found to be modest in recent studies, doubtless because patients with less severe disease have less capacity to benefit.[1] Now a large American study [2] has reported similar findings with respect to knee replacement, confirming that the operation is only cost-effective in people with severe pain and  limitation of movement at baseline. As in many medical/surgical treatments it is important to stratify by baseline severity. Absolute (and sometimes relative) benefits fall as baseline severity decreases, thereby radically shifting the harm to benefit ratio.

— Richard Lilford, CLAHRC WM Director

References:

  1. Beswick AD, Wylde V, Gooberman-Hill R, Blom A, Dieppe P. What proportion of patients report long-term pain after total hip or knee replacement for osteoarthritis? A systematic review of prospective studies in unselected patients. BMJ Open. 2012; 2: e000435.
  2. Ferket BS, Feldman Z, Zhou J, et al. Impact of total knee replacement practice: cost effectiveness analysis of data from the Osteoarthritis Initiative. BMJ. 2017; 356: j1131.

Theory of Mind in Tennis

On Tuesday nights I play tennis with my friends; Zac, Boris and Sergei (not their real names). CLAHRC interventions are often heavily based on behaviour change, and so I have had to brush up on my psychology. But is this knowledge of any use in tennis? In the following few editions of the News Blog I shall explore the modern psychological theory in the context of my feeble attempts at tennis. I shall start with the idea of a “theory of mind”, most often related to Tomasello.[1] [2]

Zac hates it when I intercept his powerful return of serve. So I know that when my partner, Sergei, next serves to the seething Zac, he will be predisposed to punish my impecuniousness by hammering his return down my tram line. To reduce this risk I should position myself towards the edge of the net. But Zac knows that I know that he is seething, and therefore that I will anticipate the tramline shot. He will anticipate my preventive action, which would open up the centre of the court, enabling him to pass me and place the ball on my partner’s backhand. So, I must anticipate his anticipation… This is an archetypal example of a theory of mind – the ability of humans to anticipate the effects of their actions in the mind of other humans. If Zac and I could undergo functional MRI on the court, then the complementary parts of our brains would brighten up in a kind of dance. So, I position myself if an intermediate position – I do not cower over the tramline, nor do I move aggressively to mid-court. Zac has three options – risk the tramline shot, go for mid-court, or play a standard cross-court shot. If he has read my reading of his reading of my mind, he will go cross-court. But what if he has anticipated I have anticipated that he has anticipated!

Tennis court showing three possible shots - tramline shot, shot to backhand, or 'default' cross-court shot

— Richard Lilford, CLAHRC WM Director

References:

  1. Carpenter M, Nagell K, Tomasello M. Social cognition, joint attention, and communicative competence from 9 to 15 months of age. Monogr Soc Res Child Dev. 1998; 63(4): 1–143.
  2. Tomasello M, Carpenter M, Call J, Behne T, Moll H. Understanding and sharing intentions: The origins of human social cognition. Behav Brain Sci. 2005; 28: 675–735.