Another Study of Studies: Effectiveness using Routinely Collected Health Data vs. RCT Data

News Blog readers will be familiar with previous meta-epidemiological studies comparing effectiveness of the same treatment when evaluated using Routinely Collected Data (RCD) vs. prospective experiments (RCTs).[1] [2] Here is another such study from John Ioannidis, the world’s number one clinical epidemiologist – masterful.[3]
The RCD studies all:

  1. Preceded their complimentary RCTs
  2. Used prediction score modelling, and
  3. Had mortality as their main outcome.

Sixteen primary routine database studies were complimented by a mean of just over two subsequent RCTs examining the same clinical question. The findings here are not as sanguine regarding database studies as those cited in previous posts. The direction of effect was different in five (30%) of the 16 studies; confidence intervals in nine (59%) of the database studies did not include the observed effect in complimentary RCTs and, where they differed, the database studies tended to over-estimate treatment effects relative to the RCT estimate by a substantial 30%. This re-informs the perceived wisdom – experimental studies are the gold standard and database studies should not supplant them.

— Richard Lilford, CLAHRC WM Director


  1. Lilford RJ. Yet Again RCTs Fail to Confirm Promising Effects of Dietary Factors in Observational Studies. CLAHRC WM News Blog. 25 September 2015.
  2. Lilford RJ. Very Different Results from RCT and Observational Studies? CLAHRC WM News Blog. 25 September 2015.
  3. Hemkens LG, Contopoulos-Ioannidis DG, Ioannidis JPA. Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey. BMJ. 2016; 352: i493.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s