Does it matter who pays?

The 6 foot 6 inch man looked me in the eye.

“And if we don’t like the results, I’ll break your kneecaps,” he said, without smiling.

This encounter, on my first impact evaluation, made me wonder about the impartiality of the whole exercise…and I am still wondering.

Recently I’ve read a bunch of papers where the authors acknowledge funding from a company which clearly would be interested in the results and, in one case, in a particular slant to the results.   And part of the work I do is directly funded by governments which clearly have a political interest in the results saying something good. And other parts of my work (as well as my salary) are funded by the World Bank, which is owned by those governments. 

Early on in my career, I worked on a project where we were looking at the effects of antiretroviral medicines on a set of economic outcomes.   This was the early days of treatment in the developing world (indeed, we worked with the only significant scale (around 200 patients) rural site in Africa at the time) and we had to cobble together funding from a range of places.   We ended up getting partial funding from a big pharmaceutical company – but one which did not have an antiretroviral on the market (they do now). The three of us working on this project had a long conversation about whether or not to take this money – but in the end we decided that as long as we were clear they would get a presentation of the results, but have no role in research design or any say over what we were to say, we were ok with it. And this was how it played out. 

Speaking with a colleague about this – this seems to be key (even if it might be obvious). She indicated that she always lays out the ground rules before she takes money from someone with an interest in the results – this work will be published no matter what, you will have no say in anything that could compromise the results of this work.   She did indicate that in at least one case she shared the results with the funders before making them public because the funders were also the implementers, and their reactions and comments might lead to greater understanding/other dimensions to look at. I have also done this with funder-implementers and found it fruitful because they know a heck of a lot more about many dimensions of the technicalities of the implementation than I do.  

But this relationship can go wrong. My colleague had a depressing tale of one NGO funder-implementer she and a collaborator worked with.   Despite the rules being explained to them, the NGO proceeded to violate them big time.   They selected higher ability participants for their training program, specifically told the participants to do well because they were being evaluated, and then asked the researchers to rewrite the report (4 times!). In the end, the researchers had to walk away. So this points to another potential lesson – when the implementer also funds the evaluation, the stakes are higher and they may go more out of their way to make sure things come out their way.    

What’s worrying about this kind of problem is that it points to some problems that the safeguards we think about – making your data public, peer review, IRB reviews – aren’t going to catch. This highlights the importance of knowing what is going on with your evaluation in significant detail – doing independent qualitative work, understanding the realities of the selection process in ways that the program implementers may not tell you, and understanding the responses of beneficiaries in some depth.   And having the courage to walk away.  

In the end, the 6 foot 6 man didn’t break my knees. I guess I am lucky the results were what they were.   Or maybe it had something to do with his Hippocratic oath.   And he even offered to help us find funding for follow up work...