“It’s as though campus physics departments have been taken over by teams of frightfully useful engineers.”
Randomised controlled trials are increasingly trendy in social science and development economics. Based on clinical drug trials in medicine, RCTs randomly assign people to treatment and control groups. The treatment group might, for example, have access to savings accounts, while the control group doesn’t. The technique avoids selection bias: the economist might distort results by picking a group that really wants to take part or where he knows the chances of success are high.
Popular science writers like Ben Goldacre have promoted their use in public policy.
I’m sceptical, as I wrote here. RCTs ignore the role of power in policy, imagining that researchers and policymakers are engaged in an objective and noble hunt for the truth rather than being swayed by convention and corporate interests. It’s no surprise that the State Bank of India, Standard Chartered and Citi all support microfinance. Microfinance gets so much attention — for good or ill — because it’s so prevalent in development policy.
RCTs are unambitious, tinkering only with what can be experimented with and tending to overlook bigger issues. Giving some of the 2.4 billion people who live on US$2 a day savings accounts just isn’t alone going to propel them toward acceptable living conditions.
Testing discrete policy ideas tends to focus on what’s been already done, shifting focus away from the search for radical or sweeping new ideas. Economists now seen as mainstream — like Smith and Hume — were at first considered heretics because they espoused off-the-wall theories would have been untestable.
I’m also wary when natural scientists start doing social science. People aren’t atoms, and because human society is changeable and unpredictable the results of experiments may work in one place for a short time but can’t really be taken as hard fact for ever in the same way as findings in natural science can. So most social-science findings are only provisional and context-dependent. Most natural scientists are aware of these differences, but some aren’t.
A great article in the Boston Review by Pranab Bardhan sums up the criticisms:
First, it is very hard to ensure true randomness in setting up treatment and control groups. So even within the domain of an RCT, impurities emanate from design, participation, and implementation problems.
Second, RCTs face serious challenges to their generalizability or “external validity.” Because an intervention is examined in a microcosm of a purposively selected population, and not usually in a randomly sampled population for any region, the results do not generalize beyond the boundaries of the study.
Third, for many important policy issues, RCTs are not very useful. You cannot run experiments in order to decide where to put power plants or ports. You cannot do a controlled test on the advisability of tight money, fiscal austerity, or deregulation. Moreover, even if you can show convincingly that a policy intervention works in a small-scale trial, policymakers still have to worry about the economic and political spillover effects of a policy when it is implemented regionally or nationally. What will be its impact on other markets and the macro economy? And what happens when a policy once handled experimentally by a local NGO is taken up for large-scale implementation by a national bureaucracy, even a well-functioning one?
Fourth, RCTs show only the average impact: a policy intervention may be very helpful for some people and not at all for others, just as a clinical drug trial may show that a particular drug works well for the average person, but it may not work at all for you. One of the standard questions of political economy, however, concerns who gains and who loses from a given policy. RCTs cannot answer that distributional question.
Finally, even when an RCT shows quite cleanly that A causes B, we do not quite know the mechanism through which it works. In interpreting many experimental results, [authors] give plausible accounts of the processes that may be at work, but these are at best their informed guesses. They are usually not rigorous derivations from the experiments themselves. In understanding alternative mechanisms through which A may have caused B, theory has to play a more important role in empirical economics than the experimentalists have assigned to it.
“It’s as though campus physics departments have been taken over by teams of frightfully useful engineers,” writes Bardhan.
None of this means that social scientists shouldn’t do controlled trials or that policymakers shouldn’t pay any attention to RCTs, but it does mean their findings should be taken with a hefty pinch of salt and that they shouldn’t sideline other techniques.
Emergent Economics 
After a justified query from a scientist friend on Facebook regarding my suspicions about natural scientists doing social science, i feel i should say that in no way was my post directed against natural scientists. My friend points out that ‘natural science’ is broad and nuanced, and that it’s important not to be too black-and-white about it.
I know that scientists do all sort of other things apart from RCTs. I’m saying that these sort of trials are limited in scope and have tended to assume too much importance in public policy, sidelining efforts to think big. Some of the findings are reasonably interesting and shouldn’t be discounted, such as the finding that savings accounts tend to be better than credit, but they’re not the only way of doing things and they don’t amount to ‘proof’ like drug trials do.
Apparently some American development economics departments focus almost entirely on experimental approaches, excluding the study of the history of ideas, reducing focus on theory and downgrading the importance of other empirical techniques. Countries simply become data to prove or disprove a hypothesis instead of being important in themselves. Cross-sectional data studies have tended to obscure country studies.
As Bardhan says later in the article, “none of the authors speak to the need for in-depth country or regional studies of political and economic processes, which provide deeper insights than those gleaned from cross-country standardized data, case studies, or micro experiments. Meticulous country and regional investigations do not answer “hot” global questions with over-arching judgments, which seems to preoccupy many prominent development economists today.”
I also feel that social reality has different characteristics to physical reality, notably the fact that its architects (us) are also its subjects, that societies differ radically according to time and place, and that repeatability — a key principle of scientific experiments — is very difficult if not impossible because of the huge number of random and unknowable elements that feature in social experiments. You could do the same experiment in two different villages and get different results. In some cases I suspect that you could do the same experiment in the same village and get a different result. That’s not to criticise “science” or anything like that, it’s to say that science needs to be done differently in social studies.
I think that Bardhan’s third point is particularly convincing: “You cannot do a controlled test on the advisability of tight money, fiscal austerity, or deregulation.” These things just can’t be tested for using RCTs. They need theories, possibly models, and empirical testing work.
He’s also quite right that “One of the standard questions of political economy, however, concerns who gains and who loses from a given policy. RCTs cannot answer that distributional question.” Right-wingers and neoliberals tend to ignore distributional issues, so RCTs fit well with their worldview. This tendency to ignore the gainers and losers and to focus on averages is particularly unhelpful.
Lastly, on the issue of whether I take a view of science: I don’t, at least not above. I take a view on what the social world is like, which is a different thing to the appropriate techniques for studying it. It’s a the difference between methods and substance — or if you prefer, between the nature of reality and how you find out about it.