Poor economics
The latest trend to arouse development economists is the randomised controlled trial. Borrowed from biology and healthcare, it’s when a researcher picks two or more different groups according to the roll of a dice, giving each group a different drug or treatment. The scientist can be fairly sure that the best-responding group has been given the intervention that works.
In development economics the researcher tests policies or techniques instead of drugs. One group, randomly chosen, receives the policy to be tested and one doesn’t, with an attempt made to keep all other circumstances equal. This avoids the problem of selection bias: the economist might distort the results by choosing a group which is particularly willing to take part or where she knows a technique will work well. It’s why the results of those pop surveys after you’ve visited a website aren’t to be trusted. The answers only come from the type of person who can be bothered to fill them in.
In a 2007 paper Benjamin Oken tried to measure ways of reducing corruption in Indonesia. He chose 608 villages in Indonesia where roads were to be built, splitting them into groups: ones without audit vs. those with audit; those featuring invitations to accountability meetings vs. those without; and ones with invitations to accountability meetings along with anonymous comment forms. Oken found that less money went missing in the villages with government audits than in the ones featuring grassroots participation in monitoring. The conclusion: audits cut corruption better than the other methods.
Oken works at MIT’s poverty action lab, of which Esther Duflo and Abhijit Banerjee are directors. They’re the authors of a recent bestseller, Poor Economics, which uses experimental evidence to discover what methods best lower poverty. Over years they did more than 240 experiments in 40 countries.
Some of their conclusions run against received wisdom. Political corruption isn’t too bad. Microfinance (tiny loans for poor people) is no magic bullet.
Fans of randomised trials are right that there’s no big answer. Lots of development fads boil down to the researcher’s arbitrary preferences rather than to hard evidence. Development sometimes seems like a Roman arena in which big, shouty men try out different ways of stabbing each other. The actual subjects — poor people — are often forgotten.
Experimentalists are also right to deny that everything’s down to corruption. That’s often just a sloppy byword for blaming the poor for their own problems. If you didn’t tolerate kleptocrats, the argument goes, you’d be richer. But lots of corrupt countries aren’t democracies, and plenty of corrupt countries got rich (China springs to mind). Some reasonably graft-free nations continue to toil.
But i’m a bit wary of these sort of experimental approaches, which amount to tinkering rather than really changing underlying conditions. There’s no understanding of capitalism as a system and very little about the exploitation of labour by the wealthy. You don’t need to be a raving socialist to understand that what’s good for the poor often isn’t good for the rich. We get cheap Iphones because Foxconn holds wages so low.
As this review says, there’s little real understanding of power in development. It’s as if the world is gradually tiptoeing toward a better understanding of poverty, and by carefully applying a few well-founded scientific results we’ll eventually live as one happy tribe. Call me a cynic, but I think power relations play a bigger role than most people admit. It’s actually beneficial to have a multi-million strong pool of underpaid or unemployed people clamouring for jobs because it keeps wages down and lets us have cheap stuff.
Experiments may work in medicine, but they can be methodologically dubious in social science. The gist of philosopher Nancy Cartwright’s presentation at INET is that the sample size needed for a fully objective study can be so enormous as to be impossible. Researchers often draw completely unwarranted conclusions from these studies.As pointed out by Marx and others, ’empirical facts’ are often distorted by their proponents. One person’s common sense is another’s ideology. I’d imagine that the sorts of questions asked by most middle-class academic Americans aren’t always the same as the sorts of things that many poor people themselves would ask.
Most of these trials also seem to fail to take enough account of context. What works in one culture might not be generalisable elsewhere. Cash incentives don’t work as well in the communitarian cultures of the South Pacific as they do in, say, Kenya. When people have plenty of food and are generally happy, they’re less likely to work hard for more pay.
Because humans are at the same time the architects of their own theories and its subjects, the problem of reflexivity emerges. The results of experiments may be valid in one situation for a short period of time but they can’t really be taken as hard fact for eternity in the same way as findings in natural science can. So most findings are only provisional and context-dependent. Such is the eternal dismalness of social science. This isn’t a body-blow to experimentalism in economics, but it does reduce its importance.
Emergent Economics 
Very interesting review, Dan. If I may say so, I find it to better than the one provided by the Guardian on ‘Poor Economics’. Few things spring to mind reading the above (in addition to what you’ve said):
– I find the very act of ‘controlled trials’ in the social field very problematic on a number of levels. Firstly it is very difficult to control anything in any society, unless one is Kim Jong(s) (even then?). Secondly, what is ‘controlled for’ might be all the interesting bits. And thirdly, a social science approach based one that goes around ‘controlling things’ is undesirable, particularly in terms of knowledge that is drawn from such exercises and policies designed thereafter. Here one is reminded of Hayek’s ‘Road to Serfdom’.
– On the specifics such as ‘micro-finance’ (MFs) , the limitations of which Poor Economics has presented, maybe all too apparent if one were to spend time in the communities with such schemes and critically observe. My own observation has been:
(1) MFs (many which are based on the principle of reaching financial sustainability) merely tap into existing skills (capabilities) rather than add/create new ones. Consequently, there exists a inverted U-shaped diminishing returns on MFs over time; i.e. high returns in the beginning but it diminishes as existing capabilities are exhausted and new ones are not added. The latter is constrained because Banks (which MFs principally are based on) are interested in lending and accruing interests, not building capacities.
(2) MFs merely allow households to move along existing production possibility frontiers, rather than shift it outwards. The latter has as much to do with individual/household capacities as it has with the (production) capacities of country itself. This is why despite of MFs, workers in my country (Nepal) choose to work in the Middle East and East Asia to take advantage of the production processes there (because there are few back home) and accrue higher returns on their labour.
I agree with you in that the issue of reflexivity is central to how we go about ‘researching’ as well as ‘theorising’ the problems and possibilities of development.
Thanks. You could say a lot more about the book — I was really only trying to make a few comments about experimental economics. Some of the best bits of the book are when the authors actually ask poor people what they think; when they try to think from the perspective of the Indonesian labourer or the Indian farmer. In this sense it’s almost the opposite of what passes for economics nowadays. Neoclassical economists would tend to make up a theory and gather abstract numerical data to support it. They would be unlikely to modify their results on the basis of ground-up empirical observation.
You’re right that philosophically there are problems with the notion of controlled experiments in social sciences. Economies or economic situations are open systems in which it is very difficult to achieve closure by holding one variable constant.
As the authors themselves are quick to point out, it’s doubtful whether the findings actually say much about development in general. For instance the last chapter is titled ‘In place of a sweeping conclusion’.
Even when you think that you’ve ‘discovered’ something, interpreting your ‘discovery’ always involves subjectivity, and there’s nothing wrong with that. It’s in the nature of social science. I never ceased to be amazed at the different ways in which people can interpret data. But the problem comes when you pretend that your research is grounded in irrefutable truth because of your own expert scientific techniques.
Funnily enough, and as William Easterly tweeted, few of the assertions in the Guardian interview are actually based on experiments. Just like most other celebrity economists, the authors are implicitly trying to legitimise their comments with an aura of scientism. It’s as if they’re saying “We’ve done hard science, so our ideas must be true”. Social science is definitely possible, but most findings are provisional, subject to time and place, and need to be accompanied by a healthy dose of humility.