Objectivity in Service Delivery Research

The CLAHRC WM Director gave two recent talks about methodology and causal modelling in the evaluation of service delivery / quality improvement initiatives.

In both he received push back from a section of the audience. The remarks could be divided into two categories:

1. Objectivity may be useful in physics and biomedical research, but cannot be usefully applied to service delivery research.
2. Service delivery is too complex to be evaluated by standard scientific tools and the CLAHRC WM Director just doesn’t get it.

So, in this blog I shall tackle the question of objectivity, leaving the issue of complexity for a forthcoming post.

Concerning objectivity, I am a Bayesian and therefore need no convincing that science cannot be shorn of subjectivity. After all, the posterior probability is a function, not just of the data, but the ‘prior’. And the prior is subjective since it is constructed mentally (except in rare cases, such as genetics when it can be calculated from Mendel’s laws). Therefore, I regard subjectivity as an ineluctable part of science. But that does not follow that objectivity must be extirpated from health service evaluations. The fact that science cannot be totally objective does not mean that it is all subjective, any more than the fact that it being partly subjective excludes a role for objectivity. No, in forming a subjective view of the world (for example, in calibrating a parameter of interest to a decision-maker) the observations that are made should be as objective as we can make them. Why should they be objective? The answer is simple – to reduce the risk of error. Why is there a risk of error? Again the answer is simple – the human mind is prone to cognitive illusions. We favour observations that fit our preconceptions,[1] as discussed in a previous post. We anchor our minds on more recent experience or evidence encountered early in a chain of evidence. We are poorly calibrated over probability estimates, especially contingent probabilities.[2] The list of cognitive biases to which the human mind is prone is extensive and has been the subject of considerable research – try Daniel Kahneman’s “Thinking Fast and Slow”, for a summary.[3] It flies in the face of accumulated evidence to reify ‘lived experience’ at the expense of gathering objective evidence in the search for scientific understanding.

It should be understood that neither subjectivism nor objectivism need to ‘win’ – they are both in play. This idea that subjectivity is inherent in science, but objectivity has an important part to play, is clearly counter-intuitive to many people. So it may help to think metaphorically, and regard science as a journey, and objectivity as sign-posts along the way. The journey has to start with a question originating in human creativity and imagination – clearly a subjective process. But creativity yields theories to test and parameters to estimate. It is in collecting and making the initial analysis of such data that objectivity should be sought. The degree to which objectivity can be achieved will, of course, vary from one situation to another. In some cases, the observer can distance herself, as when a statistician is blinded to the intervention and control group in estimating an effectiveness parameter from RCT data. In other cases such separation is not possible, as when an ethnographer makes field notes. But objectivity is still the aim, just as a (good) teacher strives for objectivity in marking a piece of work. Once the analysis is complete, then meaning must be ascribed, guidelines formulated, etc. Here personal and social factors interact, as Bandura so elegantly describes in social cognitive theory,[4] and Bruno Latour equally elegantly explicates in the specific context of scientific understanding.[5] It is failure to appreciate that science is not just one thing that seems to cause people to trip up in understanding the interplay between subjectivity and objectivity in scientific achievement. To further help explain this concept I provide the following mind-line:

Lastly, I encounter the objection that this is as may be in physics or life sciences, but does not apply to the social sciences. That’s cobblers – if there were no general statements we could make about personal and collective behaviour, then there would be no such thing as psychology or sociology. People who argue that human volition vitiates scientific inference confuse heterogeneity (it is hard, maybe impossible, to predict how an individual will behave) from a general tendency (women will accept a lower return than men in the ultimatum game; demand for health care is elastic on price). For a sure-footed philosophical account of this issue of objectivism and subjectivism in scientific reasoning I recommend John Searle.[6]

— Richard Lilford, CLAHRC WM Director

References:

1. Lord CG, Ross L, Lepper MR. Biased Assimilation and Attitude Polarization: The Effects of Prior Theories on Subsequently Considered Evidence. J Pers Soc Psychol. 1979; 37(11): 2098-109.
2. Gigerenzer G, Edwards A. Simple tools for understanding risks: from innumeracy to insight. BMJ. 2003; 327: 741-4.
3. Kahneman D. Thinking Fast and Slow. New York, NY: Farrar, Strauss and Giroux. 2011.
4. Bandura A. Social Learning Theory. Englewood Cliffs, NJ: Prentice Hall. 1977.
5. Latour B. Science in Action: How to Follow Scientists and Engineers Through Society. Milton Keynes: Open University Press. 1987.
6. Searle JR. The Construction of Social Reality. London: Penguin Books. 1996.