An Extremely Fascinating Debate in JAMA

You really should read this debate – Steven Goodman, a statistician for whom I have the utmost regard, wrote a brilliant paper in which he and colleagues show the importance of ‘design thinking’ in observational research.[1] The essence of their argument is that in designing and interpreting observational studies one should think about how the corresponding RCT would look. This way one can spot survivorship bias, which arises when the intervention group has been depleted of the most susceptible cases. This way of thinking encourages a comparison between new users of an intervention with new users of the comparator. Of course, it is not always possible to identify ‘new users’, but at least thinking in such a ‘design way’ can alert the reader to the danger of false inference.
One of the examples mentioned concerns hormone replacement therapy (HRT) where the largest RCT (Women’s Health Initiative trial) gave a very different result to the largest observational study (Nurses’ Health Study). The latter suggests a protective effect for HRT, while the former suggest the opposite. It looks as though this might not have been a very good example because, as Bhupathiraju and colleagues point out, there is a much simpler and more convincing explanation for the difference in the observed effects of HRT across the two studies.[2] The hormone replacement was given to much younger women in the observational study than in the trial. Subsequent meta-analysis of subgroups across all RCTs confirms that HRT is only protective in younger woman (who do not have established coronary artery disease). Thus, HRT is probably effective if started sufficiently early after the menopause.

This does not mean, of course, that Goodman and colleagues are wrong in principle; they may simply have selected a bad example. This is an extremely interesting exchange conducted politely between scholars and is interesting from both of the methodological and the substantive points of view.

— Richard Lilford, CLAHRC WM Director


  1. Goodman SN, Schneeweiss S, Baiocchi M. Using design thinking to differentiate useful from misleading evidence in observational research. JAMA. 2017; 317(7): 705-7.
  2. Bhupathiraju SN, Stampfer MJ, Manson JE. Posing Causal Questions When Analyzing Observational Data. JAMA. 2017; 318(2): 201.

One thought on “An Extremely Fascinating Debate in JAMA”

  1. The concept of design thinking should be correctly attributed to Hernan and Taubman and rightly cited by Goodman et al. As Hernan clarifies in his obesity question one can reduce BMI by amputating a limb but this is unlikely to have the same causal effect as increasing physical activity or reducing calorie consumption which in turn may have additional cardio-metabolic protective over and above a reduction in BMI. By forcing one to consider the RCT equivalent it is clear BMI reduction is an intermediary variable to assess whether the intervention is active just like serum cholesterol would be measured in a RCT of a lipid lowering agent. This approach to causal thinking is indeed very useful but also can be criticised as it has been argued this may negate the value of looking at variables that cannot be modified e.g. chronological age, sex (one could argue that there are medical methods to alter hormones), race though clearly these are important public health variables.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s