Syndicate content

Does it matter who pays?

Markus Goldstein's picture

The 6 foot 6 inch man looked me in the eye.

“And if we don’t like the results, I’ll break your kneecaps,” he said, without smiling.

This encounter, on my first impact evaluation, made me wonder about the impartiality of the whole exercise…and I am still wondering.

Recently I’ve read a bunch of papers where the authors acknowledge funding from a company which clearly would be interested in the results and, in one case, in a particular slant to the results.   And part of the work I do is directly funded by governments which clearly have a political interest in the results saying something good. And other parts of my work (as well as my salary) are funded by the World Bank, which is owned by those governments. 

Early on in my career, I worked on a project where we were looking at the effects of antiretroviral medicines on a set of economic outcomes.   This was the early days of treatment in the developing world (indeed, we worked with the only significant scale (around 200 patients) rural site in Africa at the time) and we had to cobble together funding from a range of places.   We ended up getting partial funding from a big pharmaceutical company – but one which did not have an antiretroviral on the market (they do now). The three of us working on this project had a long conversation about whether or not to take this money – but in the end we decided that as long as we were clear they would get a presentation of the results, but have no role in research design or any say over what we were to say, we were ok with it. And this was how it played out. 

Speaking with a colleague about this – this seems to be key (even if it might be obvious). She indicated that she always lays out the ground rules before she takes money from someone with an interest in the results – this work will be published no matter what, you will have no say in anything that could compromise the results of this work.   She did indicate that in at least one case she shared the results with the funders before making them public because the funders were also the implementers, and their reactions and comments might lead to greater understanding/other dimensions to look at. I have also done this with funder-implementers and found it fruitful because they know a heck of a lot more about many dimensions of the technicalities of the implementation than I do.  

But this relationship can go wrong. My colleague had a depressing tale of one NGO funder-implementer she and a collaborator worked with.   Despite the rules being explained to them, the NGO proceeded to violate them big time.   They selected higher ability participants for their training program, specifically told the participants to do well because they were being evaluated, and then asked the researchers to rewrite the report (4 times!). In the end, the researchers had to walk away. So this points to another potential lesson – when the implementer also funds the evaluation, the stakes are higher and they may go more out of their way to make sure things come out their way.    

What’s worrying about this kind of problem is that it points to some problems that the safeguards we think about – making your data public, peer review, IRB reviews – aren’t going to catch. This highlights the importance of knowing what is going on with your evaluation in significant detail – doing independent qualitative work, understanding the realities of the selection process in ways that the program implementers may not tell you, and understanding the responses of beneficiaries in some depth.   And having the courage to walk away.  

In the end, the 6 foot 6 man didn’t break my knees. I guess I am lucky the results were what they were.   Or maybe it had something to do with his Hippocratic oath.   And he even offered to help us find funding for follow up work...  

Comments

While researchers sometimes get criticized for being too interested in publications when working on impact evaluations, I think having the impact evaluation being done by someone who has an interest in publishing the results regardless of what those results are matters a lot here - whether it be the World Bank researcher evaluating a World Bank project or the academic researcher funded by the funding organization that has an interest in particular results. Then one hopes peer review and researcher integrity play a role in ensuring the results are presented accurately. This is also where having an evaluation registry of some form will help too - so I look forward to seeing what emerges from recent discussions on this issue.

Submitted by Catherine on
Don't you think part of our work as evaluators should also be to create an understanding and an ownership among the project implementers of the results we produce? The funding source is of course relevant for the validity of the evaluation, but what matters equally much is that the results are understood and taken into account to improve the project design and its implementation. If project implementers are also funding the evaluation, we should maybe let them know that there are cheaper options to communicate how well a project is doing.. in case they are reluctant of accepting and using somewhat negative results. Letting aside the interest of the researcher (e.g. publishing the results), we have to promote a critical self-evaluation and a demand among the project managers for the results, otherwise they rest an academic exercise, discussed among peers. Impact evaluations should be conducted to improve indivudal projects' outcomes, not only to understand development interventions at large.

Submitted by Anonymous on
It is common practice for one WBG agency in particular to fund the outside evaluation of its own programs - using trust fund money. I have personally been involved in two of these evaluations and in both cases the operational department had full access to the evaluation as it proceeded and commented extensively on the results before they were finalized. Sometimes it is a good idea to check results in advance with the operational deprtments to ensure that there are no obvious misunderstandings or errors, - but where do you draw thew line on such involvement??

Submitted by Dalta on
Unfortunately, many researchers themselves are not detached from their results. A New Yorker article nicely summarizes how a researchers' desire to be published may similarly compromise a study's integrity, as well as the potential prevalence of this tendency: http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer.

Submitted by Anonymous on
I've also had the experience that your friend describes where a study is supposedly "randomized" (at least that is what the poor researcher is told by the managers) but what is actually happening is that the treatment assignment is not random at all (from what the poor researcher learned from talking with the actual field staff...). On the other hand I sympathize with the NGO since for them good research and press legitimizes their work and continues the flow of $$ for their work. There is a lot of underlying fear about the negative consequences of research.