Syndicate content

Watermelons vs. Sesame Seeds

Justin Yifu Lin's picture

The English cartoonist Ashleigh Brilliant once offered the following piece of advice to strategists of all sorts who are concerned with their reputation: “To be sure of hitting the target, shoot first, and call whatever you hit the target…” With little time and fewer resources than elsewhere to battle the burning issues of poverty, insecurity and sociopolitical instability, economists and policymakers in developing countries may not be in the position to benefit from such cynical wisdom. Rather than listening to Ashleigh Brilliant, they should always keep in mind the constraints they face and the urgency of the situation in poor countries, and reflect on the maxim that recommends to “always aim before shooting.

A policy and research domain where there is a serious deficit of strategic thinking and prioritization is that of evaluation, which is traditionally defined as the systematic assessment of the worth or merit of some project, program or policy. The importance of evaluation cannot be underestimated: first, in a world where ideas compete constantly for funding, it is essential to ensure that value for money is at the core of public policy. Second, only by assessing the pertinence and efficiency of development initiatives can we get a full picture of their outcomes, and ensure accountability. Third and perhaps even more importantly, evaluation helps define the criteria for decision-making on new initiatives, and chart the course of future action. It highlights what works and what does not. It is therefore not surprising that evaluation has become a hot area of research and policy.

For good reasons, randomized controlled trials (RCT) are among the most popular new approaches and methodologies for impact evaluation. They suggest applying the simplicity and robustness of RCT techniques--the same approach used by the medical industry to determine if a drug or treatment does what it was designed to do--to poverty interventions to determine whether or not a program is effective. A noble goal, indeed, and a clever idea. However, as my colleague Martin Ravallion has pointed out one can wonder whether RCTs are really delivering on their main goal and promise. (See Martin’s articles “Should the Randomistas Rule?” and “Evaluation in the Practice of Development.”) Assessing the impact of specific projects, no more than one at a time, without taking into consideration the many sources of heterogeneity relevant to behavior and the interaction effects--the fact that each such project is only one component of a development portfolio that often cuts across sectors--is a recipe for biases. The very paradox of RCTs is that they do not allow for a random selection of projects and programs. While they certainly provide some useful “micro” feedback to a variety of audiences including sponsors, donors, client-groups, administrators, staff, and other relevant constituencies, they have few secrets to yield to policymakers facing big strategic “macro” decisions. As the Chinese would say, RCT researchers may be spending most of their time trying to pick up sesame seeds but neglecting or even throwing away watermelons. In other words, they may be concentrating on small matters to the expense of some important ones because the small matters happen to be amenable to RCTs.

To be sure, some questions asked by the “randomistas” are interesting and relevant. But one constraint with this approach is that it is not suitable for and may exclude other learning opportunities that are at least equally important for improving our understanding of how to reduce poverty. After all, not too long ago, most countries in the world were poor. Yet some have managed to break the poverty trap, start a sustained dynamic growth and become middle-income or even high-income economies, sometimes in a matter of just one or two generations. Outside the Western Europe and North America, Japan was the first economy in Asia to start this dynamic transition process after the Meiji Restoration in the late 19th century, followed by Korea, Taiwan-China, Hong Kong-China, Singapore, Malaysia, Indonesia, and Thailand in the 1960s, and China, Vietnam, Cambodia and Laos in the 1980s. This dynamic process has spread beyond the East Asia to Mauritius in the 1970s, India in the 1990s, and even African countries in the 2000s. In fact, among the 29 countries that grew annually at 7 and above percent in 2000-2008, 11 of them are in Sub-Saharan Africa. In terms of the impact of a country’s sustained, dynamic growth on poverty reduction, 600 million people in China alone have been lifted out of extreme poverty since 1980.

The governments in the above mentioned successful countries must have drawn valuable policy lessons or at least inspiration from the successful countries before them in the formulation of policies that unleashed their growth potentials. For economists engaged in research on poverty reduction, one of the “watermelon” questions should be why and how some countries succeeded and others failed to make it out of poverty, so that the countries that remain trapped into poverty can derive policy insights from their experiences to avoid mistakes and start successfully a sustained, dynamic growth in their countries.

RCTs and other evaluation techniques are not designed to analyze these big success and failure experiences as results of policy “experiments” of many governments. By focusing on individual development projects or programs—even in a “radiographic” fashion—they explore relevant but “sesame seed” type of questions.  They may provide interesting information about particular trees but they do not really help uncover the secrets of the forest. I hope that the enthusiasm about RCTs will not cause the development research community to “throw away watermelons for picking up sesame seeds”.

Development economists should ignore Ashleigh Brilliant’s humoristic suggestion of considering as legitimate targets for their investigations and good subjects of research any topic can be studied with currently fashionable methodologies—regardless of whether they actually generate useful new knowledge or not. After all, the ultimate test of relevance is not the desirability of a certain research methodology, but the importance of the question that the research asks and the policy insights that such research generates. The greater the extent to which the policy insights generated by a research help a country start a sustained, dynamic, and inclusive growth, the more important is the research.

Comments

Submitted by Nachiket Mor on
Dear Dr. Lin, I am entirely in agreement with you that learning about the things that reduce poverty on a national scale and set it on rapid and inclusive growth path are very important but I wonder if indeed the work on RCTs is taking away resources from such a learning effort or if instead this work is infact providing the microfoundations that can actually help a more cogent theory of what actually distinguishes countries which is at least as valuable if not more so than the multi-country econometric analyses. In my work I am somewhat engaged at both the micro level and the policy level in India in the fields of financial systems and healthcare systems design. I find myself turning more and more to these RCTs to try and separate Signal from Noise so that any policy-edifice that is built is founded on some sound underpinnings on key questions such as: What is the impact of user fees on the usage of health services, what are the transission paths through which financial access produces impact, what incentives work / do not work when one is trying to build human resource policies for the health sector. So while I am entirely with you that the jump from RCT to policy needs to be made with a great deal of caution, I feel that if one combines the lessons that learns from various tools and uses the RCT as a gold standard wherever feasible for unambigous (even if context specific and narrow) signal extraction, it would be possible to develop policies that have a much better chance of being effective post-implementation. Sincerely, Nachiket Mor

Thanks for your comment regarding 'Separating Signal from Noise'. There is undoubtedly value in randomized control trials, since they do indeed provide us some answers about practical questions such as whether free mosquito net, microcredit, schooling, cash transfers, or other programs generate results. You might want to check out David McKenzie's post on Dean Karlan's recent book, 'More than Good Intentions: How a new economics is helping to solve global poverty' [url: - http://blogs.worldbank.org/impactevaluations/dean-karlan-s-new-book-rcts-this-time-it-s-personal ] for a flavor of how RCTs are being deployed in the fight against poverty. However, I should clarify that in my post I am not only advocating multi-country econometric analysis. My point is that, by carefully studying lessons from successes and failures of countries in achieving high growth and dynamic transformation, we might be able to help lower-income countries map out a path to growth. Fundamentally a sustained and inclusive growth is the best way to achieve poverty reduction. I don't think one particular avenue of research crowds out another, and, in fact the World Bank's Development Impact Evaluation Initiative [link here to DIME site], or DIME team, is engaged actively with researchers from J-PAL and their expertise is highly relevant to the work we do. I just hope that the enthusiasm about RCTs will not cause researchers to neglect other important questions and lessons which are not suitable for using the methodology of RCTs.