Notes from the field: October edition

In our continued series on experiences in implementing impact evaluations in the field, here are a couple of observations from my recent experiences in the field on some enterprise related impact evaluations I am working on:

• The case of the fired up implementers.One of the evaluations we are working on is comparing two different types of business training – with both being delivered by the same service providers.Apparently the training of the trainers worked too well; in at least one location the trainers were so entrepreneurially energized by the training that they developed their own hybrid model that combines the two (yes, there already was a third arm where folks get both).This reminded me the importance of always knowing (at multiple points in time), as best you can, what is actually being implemented.
• Branding your program.One thing we discovered here – but also in many countries throughout Africa, is that many (if not most) entrepreneurs in urban areas have been through some kind of business training.And that means they aren’t so keen to do another one (maybe they read David’s review paper with Chris Woodruff).So an important thing we’re grappling with now is how to convince folks this is worth their while, this time it is different.Time to find someone who knows marketing.
• Working within a larger program issues #1.In one of our projects, we’re working with government.We’re trying to incentivize folks to show up for training and one idea on the table is to provide an additional allowance for transport (as some governments do for job training, for example).However, there aren’t the resources to extend this to the non-experimental trainees – of which there are many.So, understandably, the implementation team is clearly worried about the inequality this would generate among trainees and the headaches they will face.We will have to find some other way to incentivize trainees.
• Working within a larger program issues #2.One of the big risk points of contamination of the control group comes after all of the treatment folks have been treated and the evaluation team has retreated, waiting for the intervention to show an impact.Well, what does a motivated implementation team do during this time?Yup, they will look for more beneficiaries, and who better to treat than the control group.This turns out to be particularly acute when there are performance targets (e.g. a certain number of beneficiaries to cover in a very limited time) facing the implementation team.This is the time when it’s key that they understand that the control list (see below) is really people they shouldn’t visit…at least not yet.
• Do you give a control list to the recruiters or not?One question we are collectively grappling with is should we give the list of control people to the people who lead the recruitment of new clients.The management of the implementation agency is fully on board with the work (in fact they raised this issue). But they’re concerned that when the initial treatment group is done, the field team will look for new clients and, presto, here’s a nice list of people with contact details (i.e. the controls).I’ve blogged about this in the past, but this time the solution is different. Our conclusion was to go into the project (computerized) management information system and tag these people as red – plus someone will check in as per the comment before.This will help the implementers identify who not to go to…the folks who are waiting for later.There were a bunch of pros and cons for both approaches (make it obvious vs do not give the list), but in this case, my take away lesson is different from last time (the computerized project records makes a difference).
• Learning through meetings.   One thing I’ve done this week is sit through a lot of long, very detailed meetings on various aspects of program implementation.    In general, I have more than enough meetings in my life.    But these are not only interesting, but important.   I got a better understanding of the different motivation of many people and teams who matter for the intervention.  I also got a better understanding of the (evolving) politics amongst these different actors and the political constraints they face from others.  Finally, I got to see where there might be room to evaluate and/or propose some new interventions.   And I met some cool people.