School-based suicide prevention – prepare to be impressed

A recent cluster trial [1] covered over 11,000 pupils in 168 schools randomised to three different types of intervention or control. The interventions were:

  1. Educating teachers to spot children at risk;
  2. A programme delivered by outside instructors, which included role play and other measures aimed at enhancing resilience and reducing negative thoughts;
  3. A screening service.

All interventions were associated with a reduced risk of attempted suicide, but this was substantial and statistically significant only for the second intervention targeting adolescents directly. CLAHRC WM has programmes of youth psycho-prophylaxis and the Director was mega-impressed by this study.

— Richard Lilford, CLAHRC WM Director


  1. Wasserman D, Hoven C, Wasserman C, et al. School-based suicide prevention programmes: the SEYLE cluster-randomised, controlled trial. Lancet. 2015; 385; 1536-44.

The Weekend Effect

It is well known that the mortality rate of patients admitted to hospitals over the weekend is higher than that for patients admitted during the week. Whether, or to what extent, this ‘weekend effect’ is caused by case-mix factors vs. care quality factors is one of the big unknowns. This is being investigated by a CLAHRC WM-associated HS&DR grant led by Prof Julian Bion with economic support from Sam Watson, the CLAHRC WM Director and Jo Lord. We were thus provoked by a recent article by Meacock at al [1] investigating the health economics of providing increased consultant support over the weekend. The health gain is calculated on the basis of avoiding all of the excess in deaths and this is offset against the cost of providing a seven-day service. Based on their calculation, the authors find that even if the weekend effect could be eliminated, it would not justify the cost of the service at the NICE willingness-to-pay threshold. In other words, the opportunity cost is such that it would be better to leave the money doing what it is currently doing (if no new money), or to allocate it elsewhere (if new money). However, preventable deaths are merely the top of the adverse event severity pyramid and if the adverse events come down roughly in proportion to deaths, then the gains are much greater and the cost much lower than estimated in the paper. CLAHRC WM collaborators have produced a model to estimate the costs and benefits of reducing adverse events.[2] [3] We hope to collaborate with the authors of the Meacock paper in developing this research.

–Richard Lilford, CLAHRC WM Director


  1. Meacock R, Doran T, Sutton M. What are the costs and benefits of providing comprehensive seven-day services for emergency hospital admissions? Health Economics. 2015. [ePub].
  2. Yao GL, Novielli N, Manaseki-Holland S, Chen Y-F, van der Klink M, Barach P, Chilton PJ, Lilford RJ. Evaluation of a predevelopment service delivery intervention: an application to improve clinical handovers. BMJ Qual Saf. 2012; 21(s1):i29-38.
  3. Lilford RJ, Girling AJ, Sheikh A, et al. Protocol for evaluation of the cost-effectiveness of ePrescribing systems and candidate prototype for other related health information technologies. BMC Health Serv Res. 2014; 14: 314.

Objectivity in Service Delivery Research

The CLAHRC WM Director gave two recent talks about methodology and causal modelling in the evaluation of service delivery / quality improvement initiatives.

In both he received push back from a section of the audience. The remarks could be divided into two categories:

  1. Objectivity may be useful in physics and biomedical research, but cannot be usefully applied to service delivery research.
  2. Service delivery is too complex to be evaluated by standard scientific tools and the CLAHRC WM Director just doesn’t get it.

So, in this blog I shall tackle the question of objectivity, leaving the issue of complexity for a forthcoming post.

Concerning objectivity, I am a Bayesian and therefore need no convincing that science cannot be shorn of subjectivity. After all, the posterior probability is a function, not just of the data, but the ‘prior’. And the prior is subjective since it is constructed mentally (except in rare cases, such as genetics when it can be calculated from Mendel’s laws). Therefore, I regard subjectivity as an ineluctable part of science. But that does not follow that objectivity must be extirpated from health service evaluations. The fact that science cannot be totally objective does not mean that it is all subjective, any more than the fact that it being partly subjective excludes a role for objectivity. No, in forming a subjective view of the world (for example, in calibrating a parameter of interest to a decision-maker) the observations that are made should be as objective as we can make them. Why should they be objective? The answer is simple – to reduce the risk of error. Why is there a risk of error? Again the answer is simple – the human mind is prone to cognitive illusions. We favour observations that fit our preconceptions,[1] as discussed in a previous post. We anchor our minds on more recent experience or evidence encountered early in a chain of evidence. We are poorly calibrated over probability estimates, especially contingent probabilities.[2] The list of cognitive biases to which the human mind is prone is extensive and has been the subject of considerable research – try Daniel Kahneman’s “Thinking Fast and Slow”, for a summary.[3] It flies in the face of accumulated evidence to reify ‘lived experience’ at the expense of gathering objective evidence in the search for scientific understanding.

It should be understood that neither subjectivism nor objectivism need to ‘win’ – they are both in play. This idea that subjectivity is inherent in science, but objectivity has an important part to play, is clearly counter-intuitive to many people. So it may help to think metaphorically, and regard science as a journey, and objectivity as sign-posts along the way. The journey has to start with a question originating in human creativity and imagination – clearly a subjective process. But creativity yields theories to test and parameters to estimate. It is in collecting and making the initial analysis of such data that objectivity should be sought. The degree to which objectivity can be achieved will, of course, vary from one situation to another. In some cases, the observer can distance herself, as when a statistician is blinded to the intervention and control group in estimating an effectiveness parameter from RCT data. In other cases such separation is not possible, as when an ethnographer makes field notes. But objectivity is still the aim, just as a (good) teacher strives for objectivity in marking a piece of work. Once the analysis is complete, then meaning must be ascribed, guidelines formulated, etc. Here personal and social factors interact, as Bandura so elegantly describes in social cognitive theory,[4] and Bruno Latour equally elegantly explicates in the specific context of scientific understanding.[5] It is failure to appreciate that science is not just one thing that seems to cause people to trip up in understanding the interplay between subjectivity and objectivity in scientific achievement. To further help explain this concept I provide the following mind-line:

Conception of the idea - creativity and imagination; to Design  study; to Collect data; to Analyse data; to Interpret data; to Determine action.

Lastly, I encounter the objection that this is as may be in physics or life sciences, but does not apply to the social sciences. That’s cobblers – if there were no general statements we could make about personal and collective behaviour, then there would be no such thing as psychology or sociology. People who argue that human volition vitiates scientific inference confuse heterogeneity (it is hard, maybe impossible, to predict how an individual will behave) from a general tendency (women will accept a lower return than men in the ultimatum game; demand for health care is elastic on price). For a sure-footed philosophical account of this issue of objectivism and subjectivism in scientific reasoning I recommend John Searle.[6]

— Richard Lilford, CLAHRC WM Director


  1. Lord CG, Ross L, Lepper MR. Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence. J Pers Soc Psychol. 1979; 37(11): 2098-109.
  2. Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumeracy to insight. BMJ. 2003; 327: 741-4.
  3. Kahneman D. Thinking Fast and Slow. New York, NY: Farrar, Strauss and Giroux. 2011.
  4. Bandura A. Social Learning Theory. Englewood Cliffs, NJ: Prentice Hall. 1977.
  5. Latour B. Science in Action: How to Follow Scientists and Engineers Through Society. Milton Keynes: Open University Press. 1987.
  6. Searle JR. The Construction of Social Reality. London: Penguin Books. 1996.


The Payback from Improving Availability of Donor Human Milk for Premature Babies

CLAHRC WM is collaborating with the African Population Health Research Centre (APHRC) in the evaluation of donor milk banks in slums (informal settlements) in Kenya. The initiative is led by PATH,[1] which has had considerable success in establishing an altruistic donor service in South Africa. The donor milk is donated to hospital wards caring for premature infants.

There is excellent evidence that donor human milk is superior to ‘formula’ in babies whose mothers are unable to express breast milk. As a result of passive immunity, and also because it has nutritional properties that formula is not able to replicate, donor human milk reduces the risk of neonatal infection.[2] In particular, it reduces the dangerous condition of necrotising enterocolitis (NEC).[3][4] NEC can be fatal and may also require surgery that may have permanent consequences – particularly the ‘short bowel syndrome’. The decreased infection risk resulting from use of donor milk is associated with a measurable decrease in mean length of stay.[5]

One concern is that the mothers of infants who receive donor milk may be less likely to initiate breast feeding at a later date for psychological or physiological reasons. The evidence does not bear out this concern and, if anything, these mothers, perhaps inspired by the altruism of the donors, are more likely to breastfeed.[6][7] If so, this may be expected to augment the benefits of donor milk and also reduce the mother’s risk of developing breast cancer later in life.[8]

The benefits do not seem to end there. There is observational evidence, recently reinforced by a substantial study from Brazil,[9] that cognitive ability in later life is improved by human milk. There is a dose-response effect and the results remain after extensive statistical adjustment for confounders. There is also some experimental (RCT) evidence for a beneficial effect on IQ.[10] Improved IQ is correlated with earning power [11] and, we must assume, payback to society.[12]

To summarise the benefits of breastfeeding we offer the following Influence Diagram (Causal Pathway: Model):

CI - Improving Availability of Donor Human Milk Fig 1

A health economic analysis of promotion of breastfeeding for older children (not premature infants specifically) found that the intervention ‘dominated’ – reduced short-term benefits (less infection) and the contingent cost savings (reduced hospital stays) meant that interventions to promote breastfeeding are cost-saving, not just beneficial for health.[12][13]

There have been two studies of the cost-effectiveness of a donor milk service for premature babies. Both found that the service was cost-effective. The first study was based on a hypothetical baby who was very premature (28 weeks gestational age), rather than an observed mean intervention effect observed at the group level.[14] The calculated benefits might therefore be exaggerated. The second study was based on only 175 propensity scored low birth weight infants.[5] The risk of sepsis decreased with increasing dose of human milk, and total costs obtained from the hospital billing system were lower in proportion to the amount of human milk consumed. However, most infants received some human milk, so the infants could not be divided into a control and intervention population, and the above correlation between outcome and volume of donor milk consumed may have been confounded by factors that determine both access to human milk and sepsis, notwithstanding propensity scoring. Both the above studies were American.

Working with colleagues above, we propose a comprehensive health economic model that takes account of long-term outcomes and that can be populated with country-specific data. The base-case model will be populated with evidence from systematic reviews,[12][13] and we propose to use Bayesian techniques to ‘down weight’ observational evidence using the Turner and Spiegelhalter method.[15]

— Richard Lilford, CLAHRC WM Director
— Celia Taylor, Senior Lecturer


  1. PATH. Models of milk banking in South Africa. Seattle, WA: PATH, 2011.
  2. Arslanoglu S, Ziegler EE, Moro GE. Donor human milk in preterm infant feeding: evidence and recommendations. J Perinat Med. 2010; 38: 347-51.
  3. Lucas A, Cole TJ. Breast milk and neonatal necrotising enterocolitis. Lancet. 1990; 336: 1519-23.
  4. Quigley M, McGuire W. Formula versus donor milk for feeding preterm or low birth weight infants. Cochrane Database Sys Revs. 2014; 4: CD002971.
  5. Patel AL, Johnson TJ, Engstrom JL, Fogg LF, Jegier BJ, Bigger HR, Meier PP. Impact of early human milk on sepsis and health-care costs in very low birth weight infants. J Perinatol. 2013; 33: 514-9.
  6. Arslanoglu S, Moro GE, Bellù R, Turoli D, De Nisi G, Tonetto P, Bertino E. Presence of human milk bank is associated with elevated rate of exclusive breastfeeding in VLBW infants. J Perinat Med. 2013; 41(2): 129-31.
  7. Vázquez-Román S, Bustos-Lozano G, López-Maestro M, et al. Clinical impact of opening a human milk bank in a neonatal unit. An Pediatr (Barc). 2014; 81(3): 155-60.
  8. Collaborative Group on Hormonal Factors in Breast Cancer. Breast cancer and breastfeeding: collaborative reanalysis of individual data from 47 epidemiological studies in 30 countries, including 50 302 women with breast cancer and 96 973 women without the disease. Lancet. 2002; 360: 187-95.
  9. Victora CG, Horta BL, Loret de Mola C, Quevedo L, Pinheiro RT, Gigante DP, Gonçalves H, Barros FC. Association between breastfeeding and intelligence, educational attainment, and income at 30 years of age: a prospective birth cohort study from Brazil. Lancet Glob Health. 2015; 3(4): e199-205.
  10. Horta BL, Victora CG. Long-term effects of breastfeeding: a systematic review. Geneva: World Health Organization. 2013
  11. US Environmental Protection Agency. The benefits and costs of the clean air act, 1970 to 1990, appendix G, lead benefits analysis. Washington, DC: Environmental Protection Agency, 1997.
  12. Renfrew MJ, Pokhrel S, Quigley M, et al. Preventing disease and saving resources: the potential contribution of increasing breastfeeding rates in the UK. UNICEF. 2012.
  13. Kramer MS & Kakuma R. Optimal duration of exclusive breastfeeding. Cochrane Database Sys Revs. 2012; 8: CD003517.
  14. Arnold LDW. The Cost-effectiveness of Using Banked Donor Milk in the Neonatal Intensive Care Unit: Prevention of Necrotizing Enterocolitis. J Hum Lact. 2002; 18(2): 172-7.
  15. Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modeling in evidence synthesis. J R Stat Soc Ser A. 2009; 172: 21–47.

Biology never ceases to surprise: preventing cancer of the ovary by tubal ligation

One might have thought that ovarian cancer can be prevented by removing the ovaries – say at the time of hysterectomy – and that obstructing or removing the fallopian tubes would not, by itself, reduce the risk of ovarian cancer. These hypotheses are based on the plausible assumption that cancer arises in the ovaries, just as breast cancer arises in the breast. However, it now appears that ovarian cancer may arise in the fallopian tubes, at least in a substantial proportion of cases. A Swedish record linkage study shows that fallopian tube ligation is associated with a 30% reduction in the incidence of ovarian cancer.[1] Confounding by high fertility do I hear you say? Apparently not as this has been adjusted for. Caused by preventing access of carcinogens ascending the reproductive tract? Probably not, since removal of the fallopian tube provides even stronger protection against ovarian cancer, than does ligation which leaves the ovarian end of the tube in situ. Implausible hypothesis reminiscent of transubstantiation? Apparently not, since cells with the appearance of ovarian cancer have been harvested from fallopian tubes, and the molecular signature of many ovarian cancers suggests a fallopian tube provenance.

— Richard Lilford, CLAHRC WM Director


  1. Falconer H, Yin L, Grönberg H, Altman D. Ovarian cancer risk after salpingectomy: a nationwide population-based study. J Natl Cancer Inst. 2015; 107(2): dju410.

Integrated Care

A recent BMJ paper [1] from Stephen Shortell and collaborators from the King’s Fund discusses the design of models of care to improve integration between hospital and community, and between health and social care – an old chestnut. They have a taxonomy of integrated care models that I represent like this:

DC - Integrated care Fig1

Model 1 seems to locate responsibility for integration mostly with community providers, while model 2a evokes a structural solution in which hospital and community providers work in an organisation straddling hospital and community. The article comes down in favour of model 2b and gives successful examples from America (where quality has improved at reduced cost).[2][3][4][5] The article emphasises the importance of integrated computer care records, almost saying this is a necessary ingredient.

On this latter point, the CLAHRC WM Director begs to differ – as argued in a previous post, a patient-held paper record has considerable advantages over ‘all singing and all dancing’ IT systems. He does agree, however, with the idea of an integrated record (not necessarily computer-based), the authors’ emphasis on ‘clinical integration’, and the need to win the hearts and minds of service providers.[6] CLAHRC WM is involved with two grant applications to help develop the tacit skills needed to care for patients with multiple morbidities, and different needs and preferences, across multiple types of care provider in different locations.

— Richard Lilford, CLAHRC WM Director


  1. Shortell SM, Addicott R, Walsh N, Ham C. The NHS five year forward view: lessons from the United States in developing new care models. BMJ. 2015; 350: h2005.
  2. Centers for Medicare and Medicaid Services. Factsheet: Medicare ACOs continue to succeed in improving care, lowering cost growth. 2014.
  3. McWilliams JM, Chernew ME, Landon BE, Schwartz AL. Performance differences in year 1 of pioneer accountable care organizations. N Engl J Med. 2015. [ePub].
  4. Song Z, Rose S, Safran DG, Landon BE, Day MP, Chernew ME. Changes in healthcare spending and quality four years into global payment. N Engl J Med. 2014; 37: 1704-32.
  5. Markovich P. A global budget pilot project among provider partners and Blue Shield of California led to savings in first two years. Health Aff. 2012; 31: 1969-76.
  6. Curry N, Ham C. Clinical and service integration. The route to improved outcomes. London: King’s Fund, 2010.

Calling All Systematic Reviewers

The CLAHRC WM Director is provoked by the ever increasing – indeed, exponentially increasing – number of articles returned by standard literature searches. At this rate, screening all the articles identified by a typical search will be all but impossible within two decades. Some form of systematisation is necessary.

A start has been made in clinical research through the creation of the McMaster Premium LiteratUre Service (PLUS) database of pre-appraised clinical studies. PLUS is generated by manually reviewing 120 clinical journals for high-quality articles using a reproducible selection process. A paper comparing PLUS with 89 recent Cochrane reviews,[1] found that while PLUS contained fewer articles, restricting searches to PLUS did not change the conclusions of any of the Cochrane reviews included in the sample.

The PLUS database is a start, but:

  1. it still relies on manual review;
  2. it is confined to clinical research.

A method is urgently required to:

  1. improve coding of topic and study type;
  2. automate compilation of bibliographies;
  3. cover health and social care as a whole.

Literature retrieval processes will be radically different in two decades – they will have to be.

— Richard Lilford, CLAHRC WM Director


  1. Hemens BJ, Haynes RB. McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews. J Clin Epidemiol. 2012; 65(1): 62-72.


Use of WHO Surgical Check List

We thank Mary Dixon-Woods for drawing our attention to an interesting article on the use of the fabled WHO surgical checklist.[1] Interesting because the topic is important and because the authors used a step wedge, cluster, experimental design as they introduced the intervention across different surgical specialities in two Norwegian hospitals. Step wedge designs need to avoid pitfalls of all cluster studies related to interaction between intervention and willingness to be recruited. The neat way out of this conundrum is to use routinely collected data and enter everyone in the cluster. That was done here. It is important to control for systematically later time periods in the intervention ‘cells’ of the step wedge and, again, the authors did so. So what did they find in this procedurally satisfactory study? A large and statistically significant intervention effect was observed. This is in keeping with many, but not all, previous studies of the conventional checklist.

My problem lies in the underlying hypothesis; as a previous surgeon I find the theoretical basis for the checklist unconvincing. In other words I start from a sceptical prior that is reluctantly being pulled towards a more optimistic estimate. Also, I fret over publication bias in the social science/service delivery literature, as discussed in a previous post. All the same a sceptic like me cannot ignore these positive results. So how may they work. Firstly, the word “checklist” may be a misnomer. It may just be a convenient focus around which to engender a positive and professional personal and team approach. This could explain why it sometimes works and sometimes does not. The idea here would be that it can’t work when: 1) attitudes are totally hostile, or 2) practice is already very good so there is little headroom for improvement. In that case it would be like any behavioural intervention – it will work among those who are receptive to improvement, but not yet improved. It is also possible that use of the checklist, even in a tokenistic way, will be effective in the very long term. Here I rely on the theory of cognitive dissonance.[2] People who start with ritualistic tokens of compliance are inclined to either stop complying or move their attitudes towards their outward actions, if I understand correctly. Comments welcome.

— Richard Lilford, CLAHRC WM Director


  1. Haugen AS, Søfteland E, Almeland S, et al. Effect of the World Health Organization Checklist on Patient Outcomes: A Stepped Wedge Cluster Randomized Controlled Trial. Ann Surg. 2015; 261 (5): 821-8.
  2. Festinger L. A Theory of cognitive dissonance. Stanford, CA: Stanford University Press, 1957.

Health care in a parallel world: the Birmingham screwdriver

Imagining health care in a parallel world can reveal a lot about the health care system we enjoy in this one.

Counterfactual narratives have long been popular. Livy speculated about a confrontation between Rome and Alexander the Great, had the latter chosen to expand his empire westwards instead of eastwards.[1] Kingsley Amis wrote about a world where the reformation failed and Roman Catholicism continued to dominate Europe for centuries.[2] An alternate health service seems a minor alteration in comparison. What would it look like?

I recently took part in a workshop on type 2 diabetes in adolescents and young adults. Most of the speakers were medical researchers, and the audience clinicians. I was the public health afterthought. The talks focused on pathophysiology of type 2 diabetes, speculating on whether South Asians might exhibit a distinct illness trajectory to Europids. This effortlessly morphed into a speculative discussion of genetics. The medical academics were fascinated, leaning forward on their seats, vying with each other to interject. A single question about whether South Asians and White British might possibly have different lifestyles was brushed aside. The essential genetic homogeneity of the human species compared to its great ape cousins was ignored,[3] (see also our previous blog).

Although irrelevant to patients, pathophysiology and genetics fascinate doctors because they are the core of our undergraduate professional training. Pharmacological treatments predominate our therapeutic thinking because they are the logical response to pathophysiology. Doctors enjoy a near monopoly on prescribing and it is the defining and distinguishing feature of the profession. As the profession is a key influencer of the health services and research agendas, the ability to deliver the right drugs to the right patients is a central preoccupation of the health care system and the understanding of pathophysiology in order to develop and test drugs dominates the research agenda. From my background reading on the public health aspects of diabetes I learned that only 16.7% of newly diagnosed type 2 diabetics are offered structured diabetes education and only 3.6% attend.[4] How could an important and effective intervention be afforded such a low priority?

In another world a profession of health educators is in the ascendant. The profession dominates the provision of health care. Clearly the most important intervention for anyone developing a chronic disease is structured education. This conveys factual information about prognosis, life skills, confidence, and self-efficacy. The first intervention follows diagnosis. It serves an anthropological, as well as an educational, purpose, marking a life transition into a new state. Ongoing education reinforces skills, builds knowledge, and addresses the disease progression. Alongside service delivery, a vigorous research agenda constantly refines the educational interventions. New educational materials are developed. Innovative modes of delivery test new communication technologies, gamification (the use of game thinking and mechanics in non-game contexts to engage users), and virtual learning communities. Patients become co-producers of educational interventions. Stratified education is emerging where psychometric testing and preference elicitation allows patients to be matched to the most appropriate educational intervention. The primary outcomes of health care are the same: quality of life and length of life. The process measures by which we mark our progress are very different: self-efficacy, knowledge, and measurable skills replace physiological parameters. Even the typology of disease might change, with categories defined by the type of educational intervention as much as by pathophysiology.

What does this tell us? Sometimes they are so ingrained, we can’t see our own assumptions. The French call this déformation professionnelle. To a man with a hammer, everything looks like a nail. Which is why a hammer was referred to as the Birmingham screwdriver.

— Tom Marshall, Co-Director CLAHRC WM, Prevention and Detection of Diseases


  1. LiviuS T. The History of Rome (book IX, sections 17–19). English Translation by Rev. Canon Roberts. New York, NY: E.P. Dutton and Co. 1912.
  2. Amis K. The Alteration. London: Jonathan Cape. 1976.
  3. Prado-Martinez J, Sudmant PH, Kidd JM, et al. Great Ape Genetic Diversity and Population History. Nature. 2013; 499: 471-5.
  4. The Healthcare Quality Improvement Partnership (HQIP). National Diabetes Audit 2012 – 2013. Report 1: Care Processes and Treatment Targets. Leeds: Health & Social Care Information Centre. 2014

Assessing Publication Bias in Social Sciences – a Critically Important Paper from Science

Publication bias means that null results do not make it into the public domain. Assessing publication bias is straightforward in subjects where all studies have to be registered in advance – clinical trials for example. But there is little evidence on publication bias in service delivery / health services research. The CLAHRC WM Director suspects that this lack of evidence arises because much social science literature is observational rather than experimental, and it is so hard to collect convincing evidence on publication bias among such studies. There is no registry of studies; the original hypothesis may not correspond to comparisons reported; many studies might not be written up; and the investigators may evaluate a large number of associations so that results do not neatly dichotomise into significant or null. In addition, the famous funnel plot may be less likely to signal bias than is the case for much clinical research. This is because the association between sample size and risk of publication bias is less likely to hold when the size of the sample is limited more by the size of the database than the cost of recruiting individual participants. These problems were overcome in an interesting article that studied the destiny of 249 grant-funded (peer review) studies conducted within a single ongoing data collection survey over a ten year period.[1] Most of the studies consisted of an evaluation of modifications of the survey instrument (questionnaire) used to populate the survey database. The results show a massive effect. Studies with a positive result (as judged by the author) were much more likely to be written up and, if written up, much more likely to be published. The fact that the source studies were all based on a single database removes (or at least strongly mitigates) bias due to interaction between study topic and probability of a positive result.

These results reinforce the CLAHRC WM Director’s weariness to accept positive results of association studies, such as those that relate patient perception of care to standardised morality rates. Such results feed into the prevailing meta-narrative, in this case that organisational culture determines the quality of the full range of front line services. A null result is less likely to survive peer review under such circumstances. The paper cited here interviewed holders of grants based in the database, and found that they were disheartened by null results and often did not bother to submit them, anticipating that they would be rejected. They are right to be pessimistic since null results were less likely to be accepted when submitted, in keeping with the natural human tendency to reject studies that do not fit with prevailing or preconceived ideas.[2] [3]

What do we recommend? Only studies where the protocol has been published should be considered for publication, and they should all be published provided the protocol was adhered to. The clinical research world has tightened up its act. It is high time for the service delivery world to stop claiming scientific exceptionalism and adhere to the standard tenets of good scientific practice that hark back to Francis Bacon.[4]

— Richard Lilford, CLAHRC WM Director


  1. Franco A, Malhotra N, Simonovits G. Publication bias in the social science. Science. 2014; 345(6203): 1502-5.
  2. Lord CG, Ross L, Lepper MR. Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence. J Pers Soc Psychol. 1979; 37(11): 2098-109.
  3. Kaptchuk TJ. Effect of interpretive bias on research evidence. BMJ. 2003; 326: 1453-5.
  4. Bacon F. Novum Organum. 1620.