The CLAHRC WM Director was recently sent a transcript of the Richard Davies QC Memorial Lecture 2015, “Standards of Proof in Law and Science: Distinctions without a Difference“. The transcript was dispatched by Dr Martin Quinn, an old friend from his gynaecology days, and the speech was given by prominent High Court Judge, Justice Jay. Such experience as the CLAHRC WM Director has of High Court Judges is that they are a cognitively astute bunch, but not necessarily highly numerate. If he is right about that, then Jay is something of an exception. His theme was the similarity and differences between the scientific and legal intellectual frameworks. He first makes a parody of their different epistemologies, but soon comes round to cogent arguments that they are more united by their similarities than divided by their differences. After all they both have to analyse evidence, work out what it means, and make judgements under uncertainty. Individual cases come down to probabilities in both areas; the balance of probabilities in cases of tort, probabilities sufficient to put the matter ‘beyond reasonable doubt’ in criminal cases, and the relative probabilities of benefit and harm in medical cases.
This all means that both professions need quite sophisticated notions of probability with which to work. Doctors fall over their feet on probability, but Justice Jay has a clear understanding of frequentist and Bayesian notions of probability. As CLAHRC WM News Blog readers know only too well, frequentist statistics cannot tell you the probability that something is true (given the data), but only the probability of the data, given that something (typically the null hypothesis) is true. Given only the latter (i.e. a frequentist calculation of the probability of the data under a null hypothesis), then the probability of some alternative hypothesis can only be calculated given a prior probability. This is obviously a crucial concept in both law and medicine.
Consider two scenarios – a judge deciding on a case of homicide, and a doctor considering the diagnosis of Duchenne muscular dystrophy. The judge has blood group and inconclusive alibi – what was the probability that the accused was at the scene of the crime? The doctor has the result of a blood enzyme test and family history – what is the probability of the diagnosis? They both use Bayes theorem:
The difference between judge and doctor lies not in the axiomatic method that normatively underpins the requisite probabilities, but what to do with them. The judge interprets a given probability with reference to a legal framework – reasonable doubt. That might correspond to posterior odds of, say, 99:1. The doctor must make his interpretation with reference to the balance of benefits and harms. Since benefits and harms are not all equivalent, the decision turns on a ‘loss function’. The loss function is derived under expected utility theory and weights probabilities by preferences.
Both doctors and lawyers must understand notions of contingent probability. Failure to understand this idea leads to erroneous thinking, for example the famous ‘prosecutor’s fallacy’. This is exemplified in the case of Sally Clark, where an expert, Roy Meadow, argued that guilt was likely on the grounds that two cases of infant death in one family are very rare; one in many thousands. However, that consideration of the frequency of a certain scenario is quite beside the point once the scenario has been observed. In that case, the salient probability is a contingent probability – namely that of malfeasance versus that of natural causes given the observed outcome.
Cases of tort frequently turn on evidence of effectiveness. For example, the observed relative risk reduction in a meta-analysis of high quality RCTs may observe a statistically ‘significant’ 55% reduction in relative risk of outcome x if treatment y was administered. Given a particular case of tort where failure to administer y (in the absence of a contra-indication) was followed by x, it might be tempting to argue that causality can be established on the balance of probabilities. But not so fast:
- This is to conflate the probability of the effect and the probability of the data, and ‘well-brought-up people do not do that’; the prior must be brought into play.
- As in medical care, the particular features of the case must be taken into account – there may be good grounds to argue that the typical effect would be greater or smaller among people resembling the case under consideration.
In the end, ‘evidence-based medicine’ may have relatively little effect on outcomes in cases under tort. This is because most interventions examined by RCTs, the standard tool of evidence-based medicine, are not so powerful as to halve relative risks – relative risk ratios of around 20% are more typical. Furthermore, the magnitude of effect is generally smaller for less serious outcomes (such as admission to hospital with angina) than for more serious outcomes (such as cardiac death) that drive compensation quanta in claims. The situation is different with diagnostic errors, procedural errors, and failure to rescue. The CLAHRC WM Director favours a change in the law, whereby compensation is weighted by the (Bayesian) probability of causality rather than the (illogical?) balance threshold.
— Richard Lilford, CLAHRC WM Director
- Thornton JG, Lilford RJ, Johnson N. Decision analysis in medicine. BMJ. 1992 ; 304(6834): 1099-103.
- Bowater RJ, Hartley LC, Lilford RJ. Are cardiovascular trial results systematically different between North America and Europe? A study based on intra-meta-analysis comparisons. Arch Cardiovasc Dis. 2015; 108(1): 23-38.