I recently shared five failures from some of my impact evaluations. Since this is just scratching the surface of all the many ways I’ve experienced failures in attempting to conduct impact evaluations, I thought I’d share a second batch now too.
Case 4: working with a private bank in Uganda to offer business training to their clients, written up as a note here.
Failure 6: We worked with a local bank who had a business training program they wanted to offer to female clients. The training program was a 12-day course, which cost approximately US$800 per firm trained. The bank had identified 200 clients who they considered eligible for the study, and wanted to train 50 of them (Failure right here, even thinking there was a hope of doing anything with this size sample). We dropped 25 of the largest firms to reduce the variance, and then formed groups of 7, where 2 were assigned to treatment, 1 to a reserve group, and 4 to control. This gave a reserve group of 25 who they could call if not all those in the treatment wanted to attend (see my post on adding a waitlist). The partner bank insisted on full control of the invitation process to their clients, and didn’t want anyone else involved. So they called to invite people to training, found lots of the treatment group couldn’t attend, ran through the reserve list, and still only had 34 out of the 75 in the treatment and reserve list attend. Since they had contracted the training provider to train 50 people, they quickly started calling the control group and inviting them to the training to fill the remaining spaces.
Lessons: This is an evaluation we should probably have never started in the first place given the sample sizes. But then with no control over the process, and no ability to help ensure those invited to training could be helped to attend, it was impossible. Having a plan B (propensity score matching) at least helped provide some evidence to give to the bank, but this was one where we surrendered too much control.
Case 5: working with the government in Egypt to evaluate a microfinance expansion (abandoned).
Failure 7: A World Bank loan to Egypt was intended to help expand access to finance in the country. One of the components of this was to be an expansion of microfinance to the poorest areas. At the time (2009/10) of starting this there wasn’t much in the way of microfinance evaluations, and I was particularly excited about the possibilities of seeing the impacts of microfinance in a MENA country and of the project being large enough scale to measure impacts on other financial institutions in these villages. The ultimate plan was to study at least 100 villages. As is typical in many projects, the set-up phase was long, and we had someone on the ground in Egypt for over a year from the time we thought something was going to happen. Then we were finally given a list of the first 25 villages that a baseline survey could be done in, and so went ahead and did listing surveys of 13,000 households, surveyed 2,500 households and 2,500 enterprises, surveyed the financial institutions in these villages, and got ready to monitor the implementation of the intervention, while gearing up for surveying in the other 75 villages. But the instability in the wake of the Egyptian revolution meant microfinance organizations didn’t want to expand to poor areas, and the intervention never took place. The data are available if anyone can use them.
Lesson: While much of this was beyond our control, in retrospect we should have probably opted to wait and try and do a baseline once the intervention was underway. Another alternative would be to skip the baseline altogether. But given the delays and amount of work getting to the stage where we finally had a greenlight to interview, along with grant funding that there is always pressure to show you are using, we were less patient than we should have been.
Case 6: evaluating soft skills training for young women in Jordan, published here.
Failure 8: We tested a 2-week soft skills training program on the employment outcomes of community college graduates in Jordan. We carefully vetted providers of such training and had input and long discussions over the content of the classes. At the end of the classes, the provider fielded a satisfaction survey and 98% of participants rated each component as excellent. However, we found no impacts on employment. The failure was that we failed to measure the direct desired outcome of the training, which was soft skills. As a result, we couldn’t say whether the lack of employment impact was because the soft skills training failed to improve soft skills, or whether the employment return to soft skills is in fact low. Soft skills are tough to measure well – in a follow up project I think we got some good measures, but these required interactive group exercises, and may be challenging to do in a standard survey.
Lesson: this is a general issue with all training programs (e.g. financial literacy, business training, vocational training, soft skills training) - if they have no impact, is it because the thing they are trying to teach doesn’t matter that much, or because they just aren’t very good at getting people to learn more of this skill. Measuring the skills directly and not just the end outcomes like employment is thus important.
Remember we want your failures too! To date we haven’t received any failure posts from anyone! You can read about some of the other failures we’ve posted about here, and we’d love to have others share their experiences – we can collate several small examples from several contributors if you just have something specific that doesn’t seem enough for a post by itself, or happy to have you share in more detail a specific case as in Dean and Jake’s post : just email them to developmentimpactblog@gmail.com (see the end of the book review for more details). If you need to remain anonymous in sharing these failures, we can also do that.
Join the Conversation