A while back someone poked me to do more on the blog about qualitative methods. So this week, I sat down (virtually) with Rachael Pierotti, a sociologist who leads the qualitative work at the Africa Gender Innovation Lab.
MG: So sometimes I say to myself, we need some qualitative work here. But then, I am not so sure if that would actually make sense. Looking at the experiences you have had, what are some common misperceptions the economists you’ve met have about qualitative work?
RP: It seems to me that economists often think of qualitative research as surveys with small samples and open-ended questions. Based on that picture, they assume that qualitative research will be quick and cheap, which is not usually the case for in-depth qualitative research. I also find that some economists think only of focus group discussions when they think of qualitative research. Why is that? There are many different qualitative, ethnographic, and participatory methods of data collection.
MG: Good question, I am definitely guilty of that. You’ve done some cool work at the Gender Lab. What’s your favorite example of where qual and quant worked really well together?
RP: The project where the scope of what we were able to learn was most dramatically expanded because we used both quantitative and qualitative research methods was the impact evaluation of IRC’s Engaging Men through Accountable Practice (EMAP) program in eastern DRC. We evaluated the effectiveness of men’s participatory discussion groups for achieving short-term reductions in intimate partner violence and improvements in gender attitudes and household relationships. The quantitative cluster randomized control trial was complemented by extensive qualitative research that captured baseline gender attitudes through in-depth interviews and examined processes of change using a variety of qualitative and ethnographic research methods throughout program implementation. The quantitative research showed significant improvements in the quality of the couples’ relationships, and in men’s gender attitudes, but no average treatment effects on prevalence of intimate partner violence (Vaillant et al 2020). Through observation of the men’s discussion groups, the qualitative research team had documented how men collectively evaluated the ideas debated during the program. We found that men were willing to change their day-to-day practices, like participation in housework, as long as they maintained control of any changes in their household—indicating a lack of willingness to relinquish authority (Pierotti et al 2018) (gated) (ungated). Together, these results provide a much fuller picture of what and how the program worked, which IRC is now considering in their program redesign.
MG: In general, when quant researchers are starting a new project, for what kind of questions does it make sense to use qual methods?
RP: As shorthand, I like to say that qualitative research methods are important for investigating meaning and motivations. Questions about meaning and motivation examine how a particular behavior or action is understood, or how people make sense of their circumstances. Akerlof (2020) attempts to sell economists on the value of qualitative research, explaining that “…people are motivated through the stories they are telling themselves at the time they make their decisions” and to understand people’s choices, you must examine their stories. A recent paper (2017) from Margaret Frye, a sociologist who studies adolescents in Malawi (among other things), provides an example of why it matters to understand the perspectives of the people who participate in our studies. This is a mixed methods study of school dropout that combines longitudinal survey data with data from in-depth interviews with teachers, school administrators, and in and out-of-school youth. The survey data confirm the statistical association between sexual activity and girls school dropout. The qualitative study also finds that there is a widely shared belief that sexual activity is antithetical to girls’ success in school because girls will become distracted and perform poorly, or get pregnant and be forced to withdraw from school. The survey data, on the other hand, do not confirm these causal processes. Instead, Frye finds that teachers and parents act as though these processes are inevitable and harshly punish or disinvest in girls who they believe are sexually active. It is these punitive actions that lead to dropout. The policy recommendations for reducing school dropout are substantively different when the additional insights provided by the qualitative data are considered.
Another example of how understanding people’s motivations can inform how we think about constraints to development comes from work with my colleague Sophia Friedson-Ridenour (2018) (gated) (earlier working paper, ungated). Through repeated in-depth qualitative interviews with women microentrepreneurs in Accra, combined with observation of their saving and investment behavior for six months, we examined why some women were not investing in ways consistent with profit maximization. We found that because of the structure of their intrahousehold relationships, some women have competing motivations. For women who fear that visible business growth will lead their husband to withdraw support from the household, or for women who are unsure of the long-term survival of their marital relationship, their investment choices reflect desires to reinforce their husband’s role as provider, save for emergencies and petty cash, and prioritize independent long-term investments.
Quantitative researchers associate qualitative research with open-ended questions, and therefore conclude that qualitative research is primarily about capturing unanticipated consequences. That is a good reason for qualitative research, but I hope to help quantitative researchers see that qualitative research is also valuable for developing a deeper understanding of the patterns that they do anticipate and observe.
MG: And indeed – this might lead them to a very different economic model afterwards – like your Ghana example above.
MG: When is the best time in a quantitative impact evaluation to do qualitative work? Can you give examples of cases of each point in the evaluation?
RP: There are ways that qualitative research can be useful at all stages of an impact evaluation. In Republic of Congo, for an impact evaluation of a vocational training program, we did light-touch qualitative research to help us contextualize theories for why men and women tend to concentrate in different sectors of the economy, which we used to develop baseline survey measures. In Cote d’Ivoire, we conducted brief qualitative exit interviews with people who had just attended an information session on a new savings product that we intended to test. The exit interviews indicated that almost no one had understood the product description—so we modified our approach.
Qualitative research during the implementation of the intervention that is being evaluated can be extremely valuable for investigating mechanisms and documenting operational challenges. The EMAP qualitative research that I described before falls into this category. We are planning to do something similar for an impact evaluation of a new component of the Social Safety Net program in Cameroon that will integrate better training of project personnel for providing psychosocial support and referrals for survivors of gender-based violence, plus some programming for couples from beneficiary households. The project team leaders are keen to use the qualitative research to understand responses to the program, and also to document operational challenges and successes to inform similar efforts by other social protection programs.
After quantitative results are in, it can make sense to use qualitative research to answer specific puzzles that arise. For example, GIL is conducting an impact evaluation in southwest Uganda to test whether nudges or conditions can increase the number of land titles that include both the husband and wife’s name. After rollout, we found that many more men than anticipated agreed to joint land titles, even in the control condition. We designed an in-depth qualitative study to examine the meaning of joint titles in this context. If the title conferred additional bargaining power to the woman, what would motivate her husband to include her? The answer is forthcoming…
One word of caution, however. I am always hesitant about asking people to reconstruct the past, so I recommend restraint when considering retrospective qualitative research to understand why the impact estimates look the way they do. Some Gender Innovation Lab colleagues found an unexpected and unhappy result in a recent impact evaluation and suspected that there may have been implementation failures. We conducted qualitative interviews with program personnel and asked retrospective questions about implementation. They identified implementation challenges, but all interventions have implementation challenges. After the fact, it is not possible to know the extent to which the impact evaluation results were driven by the implementation quality. In general, a more narrow research question about how participants make sense of some aspect of a program will produce more meaningful results than qualitative research that broadly tries to explain an impact evaluation result.
Stay tuned for part II of this discussion and more on qualitative research methods on the blog tomorrow.
Join the Conversation