The WISDAM* of Rupert Fawdry

* With each Individual; Social Demographic And Medical patent-held and patient-owned unified health and social care record.

The retired gynaecologist Rupert Fawdry has had a lifetime interest in medical computing. He recently published a paper in the BMJ,[1] which made an argument that is out of sync with prevailing beliefs and attitudes. He pointed out that the idea of an integrated paperless record for patients with complicated needs is quixotic at the current time. His insight is that elderly people, such as his centenarian mother, have so many care providers (social services, primary care, ambulance service, charities, district nurses, hospital departments and more) that an integrated record is a near impossible achievement. The nub of the problem lies in synthesising a consolidated record from an eclectic group of component records designed with no unifying framework. The problem is not that computers cannot be connected to each other electronically. This is easily achieved and in this way they can share data as a kind of electronic mail. However, that is a far cry from the logical problem of bringing all these disparate sources of information together into a single integrated record. Maybe this will happen one day, but in the meantime Fawdry has an important practical suggestion, which is as unglamorous as it is useful – he suggests a handheld paper record. Of course there are many examples where this has worked extremely well, not least the handheld maternity record. Fawdry says the multi-sector handheld record should become the norm for people with multiple diseases and complicated needs. Iain Chalmers, founding editor of the James Lind Library, points out that such a record could ensure that a person’s wishes for resuscitation and organ donation would be readily available.

Some people may think that Fawdry is a Luddite, but failure to actually achieve an integrated record, despite decades of trying, now convinces many that “heaven must wait” and we need to get ahead with developing, alpha testing and piloting a paper-based shared record for people with multiple conditions. There is, of course, no reason why the paper record should not include computer print-outs. As the years go by an increasing proportion of the record will be computer-generated – at some point most output may be computer-provided. At that point we will be moving towards an integrated care record, but we still won’t be there.

Many I.T. experts understand the theory of integrating disparate information, and wax eloquent about ontologies and so on, but don’t seem to be able to achieve an integrated record in practice. Most people are just ignoramuses who don’t understand the difficulties of logical integration of disparate information sources, where the meaning of a term is not just determined by its definition, but where it sits temporally in relation to other terms. Fawdry is a lover of computers and can bore for Africa on the topic, but he is a realist. For people like his mother there is no time to wait for an all-singing, all-dancing, universally applicable, inter-operative, electronic health and social care record.

— Richard Lilford, CLAHRC WM Director


  1. Fawdry R. Paperless records are not in the best interest of every patient. BMJ. 2013; 346: f2064.

Vaccinations and Anti-retroviral Therapy

When I was at medical school, we were told that BCG vaccination reduced the risk of serious tuberculosis (TB) by a greater proportion than the reduction in infection of a milder or innocuous nature. However, until recently, it has been difficult to distinguish between infection and the effects of an earlier infection, previous BCG vaccination, or non-TB mycobacterial infection. A recent paper, using newer tests that can distinguish between these events, has confirmed what my teachers said at medical school – BCG protects against infection and protects even more against active disease.[1]

Meanwhile, a cluster RCT from CLAHRC Africa collaborators shows that initiating anti-retroviral therapy at home, rather than in hospital, increases uptake of therapy nearly three-fold. This practice was ‘dominant’, promising greater effectiveness at lower upfront and net costs.[2] A lot of research may be a waste of money, but not this study.

–Richard Lilford, CLAHRC WM Director


  1. Roy A, Eisenhut M, Harris RJ, Rodrigues LC, Sridhar S, Habermann S, Snell L, Mangtani P, Adetifa I, Lalvani A, Abubakar I. Effect of BCG vaccination against Mycobacterium tuberculosis infection in children: systematic review and meta-analysis. BMJ. 2014; 349: g4643.
  2. MacPherson P, Lalloo DG, Webb EL, Maheswaran H, Choko AT, Makombe SD, Butterworth AE, van Oosterhout JJ, Desmond N, Thindwa D, Squire SB, Hayes RJ, Corbett EL. Effect of Optional Home Initiation of HIV Care Following HIV Self-testing on Antiretroviral Therapy Initiation Among Adults in Malawi. A Randomized Clinical Trial. JAMA. 2014; 312(4): 372-9.

Another Potential Problem with SMRs

Readers of this CLAHRC WM News Blog will know that the director has pointed out the limitations of standardised mortality ratios (SMRs) for a decade. He has explicated the case-mix adjustment fallacy [1]; the constant risk-adjustment fallacy [2]; and the signal to noise issue.[3] Now another potential problem with case-mix adjustment has come to light – the issue of Simpson’s paradox. This paradox arises when an association found in multiple groups is reversed when these groups are aggregated. This can happen when baseball batters are compared. Consider a scenario where batter 1 receives many more pitches than batter 2 in year one, and vice-versa in year two. In such a scenario, batter 1 can have a better strike rate in both years, but a lower strike rate if these rates are simply aggregated. In a brilliant editorial, Drs Perla Marang-van de Mheen and Kaveh Shojania show how this can happen when outcomes are aggregated over doctors and hospitals.[4] The problem of Simpson’s paradox would also arise in meta-analyses if all the good and bad outcomes were simply added up before applying a simple statistical test. Of course, the standard statistical methods avoid this problem and the Director wonders whether there is a statistical approach that could be used in baseball and comparison of SMRs.

— Richard Lilford, CLAHRC WM Director


  1. Lilford R, Mohammed MA, Spiegelhalter D, Thomson R. Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma. Lancet. 2004; 363(9415): 1147-54.
  2. Mohammed MA, Deeks JJ, Girling A, Rudge G, Carmalt M, Stevens AJ, Lilford RJ. Evidence of methodological bias in hospital standardised mortality ratios: retrospective database study of English hospitals. BMJ. 2009; 338: b780.
  3. Girling A, Hofer TP, Wu J, Chilton P, Nicholl J, Mohammed MA, Lilford RJ. Case-mix adjusted hospital mortality is a poor proxy for preventable mortality: a modelling study. BMJ Qual Saf. 2012; 21(12):1052-6.
  4. Marang-van de Mheen P & Shojania KG. Simpson’s paradox: how performance measurement can fail even with perfect risk adjustment. BMJ Qual Saf. 2014; 23: 701-5.

Excess Winter Deaths

CLAHRC West Midlands has a long-standing interest in excess winter deaths. We have charted the reduction in winter deaths over the last decade and have carried out an intervention study on home improvement in the Sandwell district – results of the latter will be reported soon. In a recent fascinating study,[1] Staddon et al. showed that excess winter deaths have not only declined over the last half-century, but they have also become less variable and no longer respond to yearly differences in winter temperature. This paper argues that we may be approaching something of an asymptote, such that it will be very difficult to reduce excess winter deaths further. Improvements in housing quality and reductions in the relative costs of heating a home have contributed to the reduction in winter deaths. Temperatures have risen slightly over England and Wales over the last few decades, but this has probably been a minor factor behind the decrease in winter mortality. In the meantime we are also probing Staddon’s finding in more detail, since the nihilistic conclusion, if true, would have large implications for local authority attempts to improve public health and cut mortality by keeping people warmer.

— Richard Lilford, CLAHRC WM Director


  1. Staddon PL, Montgomery HE, Depledge MH. Climate warming will not decrease winter mortality. Nature Climate Change. 2014; 4: 190-4.

Poking Fun at Service Re-organisations

Tim Jones (University Hospital Birmingham) recently drew the CLAHRC WM Director’s attention to a 2005 paper by Oxman et al. published in the Journal of the Royal Society of Medicine,[1] which he feels you might enjoy. Unlike previous Director’s Choices, this paper does not reveal a counter-intuitive result or refute a long-held theory; instead it shows how revealing humour can be, poking fun at evidence-free management theory and the jargon that covers up this “empirical vacuum”. Some examples:

“We discovered that the literature is almost impenetrable due to creative jargon and the meaningless terminology generated by a variety of cults adhering to different beliefs and led by competing gurus.”

“We identified several over-lapping reasons for reorganizations, including money, revenge, money, elections, money… and no apparent reason at all.”

Of course the Director was reminded of the old refrain:

“We trained hard, but it seemed that every time we were beginning to form up into teams, we would be reorganised. Presumably the plans for our employment were being changed. I was to learn later in life that, perhaps because we are so good at organising, we tend as a nation to meet any new situation by reorganising; and a wonderful method it can be for creating the illusion of progress while producing confusion, inefficiency and demoralization.” – Charlton Ogburn (1957)

— Richard Lilford, CLAHRC WM Director


  1. Oxman AD, Sackett DL, Chalmers I, Prescott TE. A surrealistic mega-analysis of redisorganization theories. J R Soc Med. 2005; 98: 563-8.

The middle-management myth in healthcare

The value of middle managers in large organisations has been questioned for decades. When times get tough, the knives inevitably come out for the ‘men in grey suits’. There are few places where this cynicism towards the strategic importance of middle managers has been more evident in recent years than in the NHS. New research, however, suggests that the NHS may have underestimated the importance of a particular breed of managers – those with a clinical or professional background, referred to here as ‘hybrid’ middle managers.[1]

Identifying the ‘hybrid’ middle manager
A hybrid middle manager is anyone whose professional background enables them to act as a ‘two-way mirror’ [1] – capable not just of assimilating top-down management knowledge, but also of translating and transmitting ideas belonging to clinical practice back up into their organisation. Hybrid middle managers may have various professional backgrounds and may be located at different levels of an organisation – from ward manager to clinical director. Their strategic value does not come by virtue of their role, but rather the level of influence they are able to exert downwards to their teams and upwards, for example, to the wider clinical governance agenda. A ward manager, for example, may have deputy ward managers and team leaders below them, to whom they can broker knowledge cascaded through internal management channels. At the same time, they may offer a credible voice at departmental or divisional management meetings, a role which they can use to share practical knowledge and experience gained from day-to-day clinical practice. This type of hybrid middle manager has been calculated to represent around a third of all staffing in a traditional hospital, compared to just three per cent of ‘pure’ general managers.[2]

The strategic importance of the hybrid middle manager
Studies into private sector corporations commonly highlight the importance of middle managers as ‘knowledge engineers’ – capable of combining visionary concepts emanating from the top of an organisation with practical knowledge from the shop floor.[3] The same has been found to be true in the healthcare setting, where hybrid middle managers are uniquely placed to translate strategic management initiatives into practical applications in a clinical setting. However, the influence of these hybrids in the NHS is more complex and important than the mere ability to bridge the knowledge gap between the top and bottom layers of an organization.[4]

Knowledge brokering in service improvement
Studies into organisational behaviour indicate that hybrid middle managers have an almost unrivalled ability to broker knowledge within and between healthcare organisations. These managers operate at the frontline of service delivery and enjoy a credibility and legitimacy within their clinical communities that is not afforded to more generalist managers. They do not just understand the importance of accumulating knowledge, but also what it can be used for. The nature of clinical practice, where knowledge is constantly used alongside individual judgements, means hybrid middle managers are well equipped to act as brokers, connecting the subjective knowledge used in day-to-day clinical decision making with the more specific managerial information used in strategic service planning.  In effect, they are able to apply their professional ‘mindlines’ to more explicit organisational ‘guidelines’. This knowledge-brokering role has been identified as a key component in service improvement. In relation to the clinical governance agenda, their fusing of patient safety knowledge from clinical governance systems and the frontline of clinical practice is crucial to ensuring high-quality care for older patients in hospitals.[4] [5]

Contingencies framing the influence of hybrid middle managers
It important to acknowledge that not all hybrid middle managers are equally important knowledge brokers. The levels of influence they are able to – and, in some cases, are prepared to – exert are dependent on a number of personal and professional circumstances.[4] [5]

Inter-professional standing
The hierarchical nature of healthcare means some professionals have more perceived legitimacy than others. Nurses, for example, have legitimacy with their peers, but this can dissipate when trying to broker knowledge with and between doctors.

Intra-professional standing
Hierarchies also exist within professions, with certain clinical specialities perceived as enjoying a higher status than others, which can have an adverse impact on their ability to influence.

Professional credibility
Concerned at being seen as a manager first and a clinician second, hybrids have been found to use a number of different tactics to try and maintain a level of professional credibility. Some argue that a managerial position allows them to deliver better care. Others position themselves as a ‘representative’ of their profession or take up administrative positions within their professional bodies, which they use as a type of shield to perceived management encroachment.[6] [7]

Personal disposition
The extent to which hybrid middle managers engage with the potential of their role is governed in many cases by their overall perception of general management. Often this view is formulated early in an individual’s career, but the effects, in terms of being reluctant to embrace a knowledge-brokering role, can be long-lasting.[8]

Social capital
Social capital – an individual’s understanding, trust and reciprocity with others [9] – has been identified as a key factor in helping lower-level hybrid middle managers to break down professional boundaries, to broker their unique knowledge and thereby exert strategic influence. Hierarchies are widespread in healthcare, but in organisations where teams had developed a collective identity, there was evidence of effective knowledge brokering that crossed status and inter-disciplinary divides.[10]

— Graeme Currie, Deputy Director CLAHRC WM, Implementation & Organisation Studies Lead


  1. Llewellyn S. Two-way windows’: Clinicians as medical managers. Organ Stud. 2001; 22(4): 593-623.
  2. Walshe K, & Smith L. The NHS management workforce. 2011. London, UK: The King’s Fund.
  3. Nonaka I. Towards middle up/down management: Accelerating information creation. Sloan Manage Rev. 1988; 29: 9-18.
  4. Burgess N, & Currie G. The knowledge brokering role of the hybrid middle manager: The case of healthcare. Br J Manage. 2013; 24(s1): s132-s142.
  5. Currie G, Burgess N, Hayton J. HR practices and knowledge brokering by hybrid middle managers in hospital settings: the influence of professional hierarchy. Hum Resource Manage. 2015. [In Press].
  6. McGivern G, Currie G, Ferlie E, Fitzgerald L, Waring, J. Hybrid manager-professionals’ identity work: The maintenance and hybridization of professionalism in managerial contexts. Public Admin. 2014. [In Press].
  7. Croft C, Currie G, Lockett A. Broken ‘two way windows’? An exploration of professional hybrids. Public Admin. 2014. [In Press].
  8. Croft C, Currie G, Lockett A. The impact of emotionally important social identities on the construction of managerial leader identity: A challenge for nurses in the English NHS. Organ Stud. 2015. [In Press].
  9. Nahapiet J, & Ghoshal S. Social capital, intellectual capital, and the organizational advantage. Acad Manage Rev. 1998; 23(2): 242-66.
  10. Currie G, & White L. Inter-professional barriers and knowledge brokering in an organizational context: the case of healthcare. Organ Stud. 2012; 33(9): 1333-61.

Let’s Stop this Hysteria about Ebola

Readers of this News Blog will know that the CLAHRC WM Director is a fervent supporter of the principles of the Enlightenment – the self-consciously rational and scientific approach adopted by reflective people in the 17th century. Such principles do not characterise responses to the Ebola epidemic either locally or internationally. Locally, people succumb to conspiracy theories and fail to attend medical facilities when symptoms erupt, thereby ensuring perpetuation of the epidemic. Internationally, there are frenzied calls to ban travel and effectively quarantine whole nations – Korea airlines suspending flights to Africa and a group of faint-hearted Brazilian business people cancelling a trip to Namibia – nearly 3,000 miles from the nearest known Ebola victim.

So let’s look at the facts. Ebola virus has low infectivity because:

  1. It is spread through contact with body fluids, not through the air, like measles, SARS and influenza.
  2. It is not infective during the incubation period.

Now look at the harms. Africa’s economic growth is fragile and relies heavily on trade, aid, and connections to the rest of the world. Health and wealth are highly correlated toward the bottom of the wealth scale, but not at the top. So loss of GDP means loss of life in Malawi and Ethiopia. The damage done to Africa by poorly calibrated risk assessment is potentially large. As my colleague Jayne Parry points out, epidemics can do more harm through their irrational effects on human behaviour than through direct biological action.

— Richard Lilford, CLAHRC WM Director

Controversy in Modern Clinical Research

Tony Blair’s memoirs [1] are an interesting read for people who lived through the years of his rise to prominence and premiership. In them he makes the interesting point that he found a lack of empirical evidence to guide decisions once he had been handed the reins of power – choosing which objectives to pursue is a political choice he said, but how to reach a selected objective should be an instrumental issue to be resolved by the evidence. Aziz Sheikh and colleagues give an interesting account of the evaluation of the National Programme for Information Technology,[2] which started while Blair was Prime Minister, and which continues to produce output to this day.[3] The evaluation programme was commissioned by the CLAHRC WM Director on behalf of the Department of Health, and was the brain-child of Sir Muir Gray.

But evidence is not always conclusive, not even in the world of clinical evidence. Controversies are rife – clot-busting drugs for stroke, Tamiflu® and now statins.[4] A disheartening feature of these debates concerns the amount of personal vindictive that is manifest. It is natural for human beings to seek turpitude in the actions of others. That is why the ‘ad hominem’ fallacy – discrediting an argument by discrediting the person behind the argument – is so ubiquitous. While this practice comes naturally to us humans, scientists should be educated to rise above it and ‘play the ball, not the person’. Prof Sir Rory Collins seems to have fallen at this fence in making a connection between those who claim that statins have frequent side-effects and those who perpetrated the MMR fraud.[5] From a purely technical point of view, the CLAHRC WM Director sides with Sir Collins, who gives much more weight to the RCT evidence on side-effects than the observational and anecdotal evidence. However, he thinks Collins is in danger of exceeding the bounds of propriety in his attack on those with a different opinion and personally interceding with Fiona Godlee, editor of the BMJ. Meanwhile, other scientists are thinking up imaginative ways to collect more evidence – Iain Chalmers suggests a randomised trial of treatment withdrawal, while CLAHRC WM Deputy Director, Tom Marshall, proposes a series of n=1 trials.

I understand that we will have a chance to witness further controversy concerning clot-busting medicines in acute stroke, when Roger Shinton and the MHRA take opposing views in this weekend’s Lancet.

–Richard Lilford, CLAHRC WM Director


  1. Blair T. A Journey. London, UK: Arrow Books. 2010.
  2. Sheikh A, Atun R, Bates DW. The need for independent evaluations of government-led health information technology initiatives. BMJ Qual Saf. 2014; 23: 611-3.
  3. Petrakaki D, Waring J, Barber N. Technological affordances of Risk and Blame: the Case of the Electronic Prescription Service in England. Social Health Ill. 2014; 36(5): 703-18.
  4. Hawkes N. Risks in the balance: the statins row. BMJ. 2014; 349: g5007.
  5. Deer B. How the case against the MMR vaccine was fixed. BMJ. 2011; 342: c5347.

Operative versus Non-operative Treatment for Heel Fracture

Professor Damian Griffin of University of Warwick has recently completed one of the largest randomised controlled trials to be performed in orthopaedic surgery, looking into operative vs. non-operative treatment for heel fractures in the UK.[1] This study is singled out because it was so hard to do – surgeons have strong opinions and the condition occurs infrequently; only around 2% of all fractures are to the heel bone (calcaneus). Many are severe, high-energy fractures, resulting from, for example, falling from a height or a road traffic incident. Surgical treatment began to be widely used following the advent of computerised topography (allowing for better understanding of fractures), and new techniques that allowed bone fragments to be realigned, fixed and restored.

The RCT randomly allocated 151 patients to receive either operative or non-operative treatment. The entry criterion was based on equipoise between surgery and a conservative option.[2] At two year follow-up there were no symptomatic or functional advantages in patients who had undergone operative treatment – there was no significant difference in patient-reported pain and function scores, or in any secondary outcome (complications, hindfoot pain and function, general health, quality of life, clinical examination, walking speed, and gait symmetry). There were, however, significantly more complications and re-operations in the group that received operative treatment. Again, a treatment that makes sense intuitively turns out to do more harm than good.

— Richard Lilford, CLAHRC WM Director


  1. Griffen D, Parsons N, Kulikov Y, Hutchinson C, Thorogood M, Lamb SE. Operative versus non-operative treatment for closed, displaced, intra-articular fractures of the calcaneus: randomised controlled trial. BMJ. 2014; 349: g4483.
  2. Lilford RJ. Ethics of clinical trials from a bayesian and decision analytic perspective: whose equipoise is it anyway? BMJ. 2003; 326: 980-1.

From CAHRD to Campbell

After two days of an intensive consultation meeting at the Collaboration for Applied Health Research and Delivery (CAHRD), where the focus was on learning from stakeholders about the future direction of applied health research in low- and middle-income countries, I set off to Belfast to attend the 2014 Campbell Collaboration Colloquium. Having been a traditional ‘Cochraner’ for some time, it was a bizarre experience for me to meet so many people who, while doing the same type of work (systematic reviews) for the same purpose (informing policy and practice), are doing it in quite different contexts (education, crime and justice, and international development). It is somewhat like travelling to a different country in the modern world – you see people doing the same thing, such as going to a restaurant, but they have quite different menus and speak a different language.

Talking of language, one common issue that emerged from both meetings is terminology. In the CAHRD meeting we talked about the need for a standardised terminology in health service delivery research. As an example, the term “health system” means different things for different people and is often used when people want to describe something about health care, but know relatively little about it. In the Campbell conference I joined a session of the Knowledge Translation and Implementation (KTI) group where we were tasked with consolidating the definition of ‘knowledge translation’. The group leaders presented no less than 15 related terms (such as knowledge mobilisation and technical assistance) and identified 61 different frameworks or models of KTI through preliminary research. The tasks of resolving differences and reaching a consensus seem daunting.

While differences appear to be ubiquitous, many of them need not be a cause of concern so long as they do not lead to misunderstanding and ignorance. In the world of Campbell I soon got used to the term “moderator analysis,” which had only been known to me in the context of subgroup analysis and meta-regression for exploring potential sources of heterogeneity; and “impact evaluation for a development programme,” which appears somewhat similar to health technology assessments for new drugs, with which I am more familiar. I realised that although the names may be different and the techniques and emphasis may (quite rightly) vary to some extent to suit a different context, the principles are the same.

With my unease dissipated, I quickly started to enjoy exploring the new territory – as expected at such a conference there are many interesting things to be uncovered. For example, Professor Paul Connolly talked about how randomised controlled trials (RCTs) are depicted negatively in research methods textbooks as an unrealistic method advocated by positivists ignorant of the complex world of teaching and learning. He also detailed how the team at the Centre for Effective Education, based in the Queen’s University Belfast, have managed to conduct more than 30 RCTs in education settings since 2007. My recent task of sifting through nearly 10,000 records for a systematic review is easily dwarfed by the efforts of international colleagues who have trawled through over 60,000 records for a review of youth crime and violence. However, against the rather gloomy prospect of soon getting lost in the ever expanding sea of information, comes the welcome news that the Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre (a major player in the field of evidence synthesis in education and social policy) has developed smart software that utilises text mining and machine learning to automatically ‘prioritise’ references that are most likely to be relevant for a review based on the input of a few key words.

One of the most inspiring talks was delivered by Dr Howard White, who illustrated that the lack of permanent changes backed up by solid evidence has rendered education and social policy vulnerable to the influence of short-term political cycles. The example he quoted is the resurfacing of the debate on the merit of pay-for-performance based on exam results in school settings – an issue that was claimed to be resolved in a book concerning the education system in West Africa in the 1920s.

For people like me who have mainly been involved in evidence synthesis and evaluation in health care, but are curious about their application in the wider world, the International Initiative for Impact Evaluation (3ie), of which Dr White is the Executive Director, is well worth looking into. They are a US-based, not-for-profit organisation that commissions and carries out in-house systematic reviews and impact evaluations of development programmes for developing countries. They have offices in Washington, New Delhi and London and have commissioned or carried out more than 130 impact evaluations and 30 systematic reviews since 2009. Topics have been diverse, ranging from the more familiar, such as a systematic review of community-based intervention packages for reducing maternal morbidity and mortality and improving neonatal outcomes, to the less familiar, for example, impact evaluation of export processing zones on employment, wages and labour conditions in developing countries. All reports are available from their website, which also includes a wealth of other resources such as evidence ‘Gap Maps’, methodological working papers, a prospective registry for international development impact evaluations, and a searchable database of evaluation experts.

My final reflections upon the journey through both meetings is that to achieve the common aspiration of evidence-informed policy and practice, we need to break any boundary of disciplines and ideologies; and understand and embrace differences rather than exclude or ignore them, so that the diverse strength from individual persons and organisations can be harvested to the greatest extent to expedite the progress. Perhaps science has its own cycles, just like politics, and after a period of phenomenal advances in increasingly divided subject areas, the time has come to focus on how to integrate and synergise specialised knowledge.

Yen-Fu Chen with Martina Vojtkova, Evaluation Specialist from the 3ie, at the Campbell Collaboration Colloquium 2014.
Yen-Fu Chen with Martina Vojtkova, Evaluation Specialist from the 3ie, at the Campbell Collaboration Colloquium 2014.

— Yen-Fu Chen, Senior Research Fellow