In our last posting we talked about six techniques to make our questions more precise so as to get the best answers from the Web. In this blog, we look at the other side of the equation: how can we be reasonably confident that the answers we get from an online resource are correct? How can we know that the web has given us the right answer when we do not have the subject matter expertise ourselves?
Path to “Confucian” wisdom
How to know what you don’t know
The adage “True wisdom is knowing what you don't know” has been attributed to Confucius. While addressing this philosophical statement is beyond the scope of this blog, it is appropriate to title a pragmatic article borrowing from ancient wisdom. Knowing what you do not know is the essential problem of learning in the modern era. Legacy learning depends on teachers and textbooks who you can rely on to be correct. However, for contemporary learning - how can you tell the correct from the incorrect if you don’t have sufficient knowledge of a domain?
We describe a four step process one can use to eliminate the really bad answers and get a decent idea of which ones are very good.
The process may not be able guarantee the answers we got are absolutely correct, but the level of accuracy of the answers we will get by following the process will be useful in most cases.
In this blog, we advocate the importance of in-depth reporting on implementation processes, evaluation processes, and relevant contextual details of interventions and linked evaluations. This will facilitate research transparency, as well as assessments of both learning and the potential for generalizability beyond the original study setting (learning lessons from ‘there’ for ‘here,’ but not necessarily promoting the strict and exact duplication of a program from one setting to another, in line with an understanding of external validity that is appropriate for the social sciences in development).
We start with a hypothetical scenario of an intervention and associated evaluation, based on too-frequent experiences in the impact evaluation space. We hope that it doesn’t sound familiar to those of you who have been involved in evaluation or have tried to make sense of evaluation results -- but suspect that it will.
A research team, connected to a larger research and evaluation organization, ran a study on an intervention. For reasons of statistical and political significance, they have deemed it sufficiently successful and worthy of scaling up, at least in a very specific new setting.
The intervention sought to overcome the following problem, for which there are supply-side and demand-side issues. People in malarious areas may procure a bednet (whether for free or for a positive price), but they do not always follow-through with maintenance (re-treatment or replacement).
For supply, the private sector only sporadically offers retreatment and replacement, and it is expensive, while the public sector does not always have supplies available. The intervention, therefore, concentrates provision of this service at a specific time and place through temporary service centers.
For demand, people with nets often don’t understand the need for retreatment and, even if they do, continuously put off doing so. The intervention, therefore, included a non-monetary incentive for which there is local demand (in this case, soap) to be picked up at the time of net retreatment.
Over the past five years, the Agence Française de Développement (AFD) and the World Bank Group have coproduced 20 volumes on various dimensions of development in Africa. The Africa Development Forum (ADF) book series has addressed subjects including the agricultural, demographic, climatic, and environmental challenges facing African countries, as well as the various methods of financing infrastructure, cities, and social safety nets. In-depth research brings to light specific and diverse situations encountered around the continent. Moving beyond the results of such endeavors, the question remains of how to conduct research that can make a pertinent and meaningful contribution to public policy. Two fundamental tools are required: robust, and often times original, data and cutting-edge research. This research must not only be connected to international realities; it must be firmly anchored in African realities and geared toward public policy making.
In this final post, Deval Desai and Rebecca Tapscott respond to comments by Lisa Denney and Pilar Domingo, Michael Woolcock, Morten Jerven, Alex de Waal, and Holly Porter.
Our paper, Tomayto Tomahto, is in essence an exhortation and an ethical question. The exhortation: treat and unpack fragility research (for we limit our observations to research conducted for policy-making about fragile and conflict-affected places) as an institution of global governance, a set of complex social processes and knowledge practices that produce evidence as part of policy-making. The ethical question: all institutions contain struggles over the language and rules by which they allocate responsibility between individual actors (ethics) and structural factors (politics) for their effects—this might be law, democratic process, religious dictate. In light of the trends of saturation and professionalization that we identify (and as Jerven astutely points out in his response, a profound intensification of research), is it still sufficient to allocate responsibility for the effects of fragility research using the language and rules of method?
The five responses to our piece enthusiastically take up the exhortation. A series of positions are represented: the anthropologist (Porter), the applied development researcher (Denney and Domingo), the anthropologist/practitioner (DeWaal), the practitioner/sociologist (Woolcock), and the economist (Jerven). They unpack the profoundly socio-political nature of the relationship between research and policy from a number of different perspectives: Porter’s intimate view from the field, Jerven’s sympathetic ear in the statistics office, Woolcock and Denney and Domingo’s feel for the alchemic moments when research turns into policy at the global level, and de Waal’s distaste for the global laboratories in which those moments occur, preferring the local re-embedding of research. These all, of course, spatialize the research-policy nexus, just as we do; however, all then ask us to privilege one space over the others.
How to avoid “We saw the evidence and made a decision…and that decision was: since the evidence didn’t confirm our priors, to try to downplay the evidence”
Before we dig into that statement (based-on-a-true-story-involving-people-like-us), we start with a simpler, obvious one: many people are involved in evaluations. We use the word ‘involved’ rather broadly. Our central focus for this post is people who may block the honest presentation of evaluation results.
In any given evaluation, there are several groups of organizations and people with stake in an evaluation of a program or policy. Most obviously, there are researchers and implementers. There are also participants. And, for much of the global development ecosystem, there are funders of the program, who may be separate from the funders of the evaluation. Both of these may work through sub-contractors and consultants, bringing yet others on board.
Our contention is that not all of these actors are currently, explicitly acknowledged in the current transparency movement in social science evaluation, with implications for the later acceptance and use of the results. The current focus is often on a contract between researchers and evidence consumers as a sign that, in Ben Olken’s terms, researchers are not nefarious and power (statistically speaking) -hungry (2015). To achieve its objectives, the transparency movement requires more than committing to a core set of analyses ex ante (through pre-analysis or commitment to analysis plans) and study registration.
To make sure that research is conducted openly at all phases, transparency must include engaging all stakeholders — perhaps particularly those that can block the honest sharing of results. This is in line with, for example, EGAP’s third research principle on rights to review and publish results. We return to some ideas of how to encourage this at the end of the blog.
This post is by Holly Porter, a visiting fellow at the Department of International Development of the London School for Economics and Political Science and lead researcher for northern Uganda for the Justice and Security Research Programme. It is a contribution to an online symposium on the changing nature of knowledge production in fragile states. Be sure to read other entries by Deval Desai and Rebecca Tapscott, Lisa Denney and Pilar Domingo, Michael Woolcock, Morten Jerven, and Alex de Waal.
The piece is a welcome provocation to discussion, even if ultimately I am left with the thought: there is a rather fundamental difference between tomatoes on a supply chain and the pursuit of understanding human experience. I show that here, intentionally choosing to write from a personal perspective, rather than in more academic prose.
Two main responses spring to mind in light of my own (anthropological) work:
1) Knowledge is not an “alienable” commodity.2) The complexity of human relationships in the research process are not best captured with reference to market forces.
The piece raises an underlying question about the production of “knowledge:” is knowledge a kind of raw material –- is it an “alienable commodity”? The idea that data is a commodity implies that it is something; that it is a thing which exists independently and apart from the intentionality of human relationships. Perhaps some information is similar to a raw material that can be extracted in crude form but the kind of “knowledge” which interests me is born of shared experiences and long-term relationships. Knowledge appears to me less of a raw material to be processed and packed, and more the stuff of human interaction.
This post, written by Michael Woolcock, is a contribution to an online symposium on the changing nature of knowledge production in fragile states. Be sure to read other entries by Deval Desai and Rebecca Tapscott and Lisa Denney and Pilar Domingo.
My nomination for development’s ‘Most Insightful, Least Cited’ paper is Ariel Heryanto’s “The development of ‘development.'” Originally written in Indonesian in the mid-1980s, Heryanto’s gem has been cited a mere 79 times (according to Google Scholar), even in its carefully-translated English incarnation. For me, this paper is so wonderful because it makes, in clear and clever ways, two key points that bear endless repetition, especially to today’s junior scholars. The first point is that inference from evidence is never self-evident: significance must always be interpreted through theory. Consider the seemingly obvious fact that the sun rises in the east every morning, he writes. What could be more universally and unambiguously true? The problem, of course, is that the sun does not rise in the east; instead, despite every piece of sensory evidence to the contrary, the earth rotates counterclockwise on its axis and revolves around a stationary sun, making it appear as ifthe sun rises in the east. But we only know this – or, more accurately, claim to know this – because today we happen to have a theory, itself based on more complex forms of observation and theory, that helps us interpret the prevailing evidence, to reconcile it with evidence from analyses of other cosmic phenomena, and thus draw broadly coherent conclusions and inferences.
Heryanto’s second key point is that we are all captives of language, of the limits of any given tongue to convey the subtleties of complex issues. From this premise he proceeds to unpack the clumsy, alluring yet powerful word that in English we call ‘development’, noting that in Indonesian there are at least two very different interpretations of its meaning, and with this, two very different words – perkembangan and pembangunan – connoting two very different teleologies and policy agendas: the former a natural, ‘organic’ process akin to flowers blooming (“software”); the latter to an overt, intentional and ‘constructed’ political project of nation building (“hardware”). When translated into English, however, both perkembangan and pembangunan are typically rendered simply as “development,” thereby collapsing into a singular popular conception what in Indonesian discourse is a distinctly pluralist one. In the opening week of my class at the Kennedy School, which typically has 50 students who between them speak around 30 languages, we begin with a lively discussion of what “development” means in Arabic, Hindi, French, Turkish, Spanish, Swahili, Swedish… It turns out to mean all sorts of things.
I open this way because I think the next article we need in this “genre” – though hopefully one that quickly transcends it because it is both highly insightful and highly cited! – is something akin to what Desai and Tapscott have begun with their ‘Tomayto Tomahto’ paper. In short, echoing Heryanto, we need more development research on development research. Such scholarship, however, would go beyond providing a mere chronology of changing professional styles, methodological emphases and funding characteristics (scale, sources, time horizons, expectations) to explanations of how and why such changes have occurred. Such explanations would be grounded in analyses of the shifting historical experiences and geo-political imperatives different generations of researchers have sought to accommodate, the particular ideas these experiences and imperatives rendered normative, and the concomitant gains and losses these changes have entailed for those finding themselves managing the “trade-offs” (such as they are) between scholarly independence and public utility.
This post by Lisa Denney and Pilar Domingo is a contribution to an online symposium from Humanity Journal on the changing nature of knowledge production in fragile states. Be sure to read other entries, beginning with Deval Desai and Rebecca Tapscott's piece.
While researchers (ourselves included) now consistently underline the importance of understanding the political economy of developing countries and donors that support them in order to achieve better aid outcomes, the research industry remains largely ambivalent about questions of our own political economy. Desai and Tapscott’s paper is therefore a refreshing attempt to start unpacking this and the ways in which ‘evidence’ is produced within the development industry.
Here, we offer reflections on three stages of this process: building evidence, translating evidence and dislodging evidence. But a word of caution is also merited upfront. The fact that we are talking about “evidence,” rather than research, is itself telling and underscores a shift in the development industry in the last ten years. Speaking about ‘evidence’ rather than about “research” suggests something much more concrete and indisputable. Evidence is taken as proof. But surely research is also debate. While there are, of course, things for which largely indisputable evidence can be found (the effects of vaccines on disease, for instance), the use of this terminology, particularly in the social sciences where little is concrete or universal, suggests that final answers are discoverable. It can, thus, be used to close down debate, as much as to encourage it. Research, on the other hand, recognizes that most findings are contributions to knowledge that helpfully allow to move us towards deeper understanding and greater awareness but do not claim to be the final word on a given topic.
This post is the first in a symposium from Humanity Journal on the changing nature of knowledge production in fragile states. It was written by Deval Desai, a Research Associate at ODI, and Rebecca Tapscott, a PhD Candidate at the Fletcher School at Tufts University.
Aid in the 21st century is increasingly evidence-driven. Between 2000 and 2006, the World Bank spent a total of $630 million on research. By 2011 the World Bank was spending $606 million per year, or about a quarter of its country budgets. In September of this year, by signing up to the Sustainable Development Goals, the global community enshrined a commitment to “increase significantly” a range of high-quality data over the next 15 years, to facilitate qualitative as well as quantitative understandings of growth and progress.
As the international community seeks to tackle the “hard problems” of development—fragility, conflict, endemic poverty—qualitative research is ever-more important. These problems are not amenable to best-practice solutions but must be tackled through deep contextual understanding of their drivers. Or so the policy story goes. As a result, conducting qualitative research today is different from the days when Geertz set out for Bali. Gone are the intrepid individuals setting off to explore and explain an untouched environment, unaware of the demands of policymakers.
We argue that while practice has changed, the ideology of qualitative research has not. Qualitative research is generally understood as the individual exercise of research methods to produce knowledge about the world, knowledge that can then be taken up by governance actors of all stripes. By contrast, we believe that today we must understand research as asystemic intervention, within the broader context of globalization and international development. Therefore, we should start with the political economy of contemporary research—an iterative, professionalized and increasingly saturated practice—to rethink the political and ethical implications of the research that we do.
As a first step to this end, we contrast two stylized frameworks for understanding qualitative research in fragile contexts: The “fragility research” framework, which we argue dominates the current debate; and the “research supply chain” framework, which we offer as a new framework and a provocation to discussion. We discuss each in turn, first considering how fragility research frames knowledge production in fragile or conflicted-affected states, identifying some assumptions the fragility research framework rests on, and critiquing some of its key conclusions. We then discuss the research supply chain as an alternative framework to explore the relationship between knowledge generation and policy. Finally, we raise some questions based on the new framework’s implications.