“It’s as though campus physics departments have been taken over by teams of frightfully useful engineers.”
Randomised controlled trials are increasingly trendy in social science and development economics. Based on clinical drug trials in medicine, RCTs randomly assign people to treatment and control groups. The treatment group might, for example, have access to savings accounts, while the control group doesn’t. The technique avoids selection bias: the economist might distort results by picking a group that really wants to take part or where he knows the chances of success are high.
I’m sceptical, as I wrote here. RCTs ignore the role of power in policy, imagining that researchers and policymakers are engaged in an objective and noble hunt for the truth rather than being swayed by convention and corporate interests. It’s no surprise that the State Bank of India, Standard Chartered and Citi all support microfinance. Microfinance gets so much attention — for good or ill — because it’s so prevalent in development policy.
RCTs are unambitious, tinkering only with what can be experimented with and tending to overlook bigger issues. Giving some of the 2.4 billion people who live on US$2 a day savings accounts just isn’t alone going to propel them toward acceptable living conditions.
Testing discrete policy ideas tends to focus on what’s been already done, shifting focus away from the search for radical or sweeping new ideas. Economists now seen as mainstream — like Smith and Hume — were at first considered heretics because they espoused off-the-wall theories would have been untestable.
I’m also wary when natural scientists start doing social science. People aren’t atoms, and because human society is changeable and unpredictable the results of experiments may work in one place for a short time but can’t really be taken as hard fact for ever in the same way as findings in natural science can. So most social-science findings are only provisional and context-dependent. Most natural scientists are aware of these differences, but some aren’t.
A great article in the Boston Review by Pranab Bardhan sums up the criticisms:
First, it is very hard to ensure true randomness in setting up treatment and control groups. So even within the domain of an RCT, impurities emanate from design, participation, and implementation problems.
Second, RCTs face serious challenges to their generalizability or “external validity.” Because an intervention is examined in a microcosm of a purposively selected population, and not usually in a randomly sampled population for any region, the results do not generalize beyond the boundaries of the study.
Third, for many important policy issues, RCTs are not very useful. You cannot run experiments in order to decide where to put power plants or ports. You cannot do a controlled test on the advisability of tight money, fiscal austerity, or deregulation. Moreover, even if you can show convincingly that a policy intervention works in a small-scale trial, policymakers still have to worry about the economic and political spillover effects of a policy when it is implemented regionally or nationally. What will be its impact on other markets and the macro economy? And what happens when a policy once handled experimentally by a local NGO is taken up for large-scale implementation by a national bureaucracy, even a well-functioning one?
Fourth, RCTs show only the average impact: a policy intervention may be very helpful for some people and not at all for others, just as a clinical drug trial may show that a particular drug works well for the average person, but it may not work at all for you. One of the standard questions of political economy, however, concerns who gains and who loses from a given policy. RCTs cannot answer that distributional question.
Finally, even when an RCT shows quite cleanly that A causes B, we do not quite know the mechanism through which it works. In interpreting many experimental results, [authors] give plausible accounts of the processes that may be at work, but these are at best their informed guesses. They are usually not rigorous derivations from the experiments themselves. In understanding alternative mechanisms through which A may have caused B, theory has to play a more important role in empirical economics than the experimentalists have assigned to it.
“It’s as though campus physics departments have been taken over by teams of frightfully useful engineers,” writes Bardhan.
None of this means that social scientists shouldn’t do controlled trials or that policymakers shouldn’t pay any attention to RCTs, but it does mean their findings should be taken with a hefty pinch of salt and that they shouldn’t sideline other techniques.