Discontinuities in Data – a Neat Statistical Method to Detect Distorted Reporting in Response to Incentives

Discontinuities can be very revealing in Service Delivery and Policy Research – they provide a statistical method to detect the distorting effects of incentives. For example, the statistical test for p-hacking reported previously in your News Blog,[1] is based around the p<0.05 threshold for statistical significance. While the p-value is easily ‘hacked’ by selectively reporting ‘significant’ results, other data may be harder – death rates for example.

The great American economist Raymond Fisman (he of the New York traffic violations fame)[2] and Yongxiang Wang examined industrial deaths in China.[3] A threshold for such deaths was set at national level, with a penalty for Provincial administrations who failed to reach the target threshold. The distribution of deaths across provinces looked like this before the incentives went live:

085 DCii Discontinuities in Data Fig 1

After the incentive, it looked like this:

085 DCii Discontinuities in Data Fig 2

Not only that, but this discontinuity is found exclusively in reports from the fourth quarter of the year. This makes a compelling case – if you provide a target and managers do not think it is fair, then they will manipulate it, even if it is something that, on the face of it, is hard to manipulate. You would not succumb to such a temptation, do I hear you say? But you would, oh yes, you would!

— Richard Lilford, CLAHRC WM Director

References:

  1. Lilford RJ. Look out for ‘p-hacking’. NIHR CLAHRC West Midlands News Blog. 11 September 2015.
  2. Fisman R, Miguel E. Cultures of Corruption: Evidence from Diplomatic Parking Tickets. NBER Working Paper No. 12312. 2006.
  3. Fisman R & Wang Y. The Distortionary Effects of Incentives in Government: Evidence from China’s “Death Ceiling” Program. Am Econ J Appl Econ. 2017; 9(2): 202-18.
Advertisements

2 thoughts on “Discontinuities in Data – a Neat Statistical Method to Detect Distorted Reporting in Response to Incentives”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s