Some of the most pressing policy problems involve outcomes that have a low incidence rate in most populations. For example, take a random sample of the population, and the proportion you will find committing a crime, experiencing homelessness, being a victim of violence or making a risky irregular migration journey in a given time period will be very low. This poses a challenge for impact evaluations that aim to reduce these problems, since even with more targeting of the interventions to a subsample of the population, the ultimate outcomes of interest may be relatively rare, making any treatment effects small in absolute terms, even if they are large in relative terms.
This was the challenge faced in an experiment designed to test different interventions for reducing irregular migration from The Gambia to Europe and providing alternative livelihood options. In a new working paper with Tijan Bah, Catia Batista, and Flore Gubert, we provide the results of the experiment, with a 2-page impact note summarizing the key results and drawing out some of the policy lessons. For this blog, I wanted to draw out some of the broader issues involved in an impact evaluation of this type.
Interventions designed to reduce irregular migration and provide meaningful alternatives
Our experiment consists of a clustered randomized trial that worked with 3,641 young men aged 18-30 in 391 settlements (villages) in the Eastern part of The Gambia. Settlements were randomized to a control group, and three treatment groups:
· Information and deterrence: this follows the most common policy option of an information campaign, designed to increase knowledge and awareness of the risks and dangers of irregular migration through the desert, across Libya, and across the Mediterranean Sea. Youth were shown an information video that featured interviews with youth from an NGO formed by returned and deported migrants discussing their experiences, along with animations showing the statistics behind these risks.
· Information and a regional migration alternative: in addition to the information video above, and an extra video profiling Gambians working in Dakar in neighboring Senegal, youth in these treatment villages were given logistical help and a labeled cash transfer of around 20 euros to pay for the cost of travelling to Dakar. This aimed to provide a safer visa-free regional migration alternative that would still offer the chance to earn more, but with less cost and risk, and with legal migration.
· Information and vocational training: as well as the information video, youth in these treatment villages were offered vocational training in skills like construction, welding, plumbing, small engine maintenance and electrical repair that could open up access to urban jobs elsewhere in The Gambia or outside it – providing more alternative livelihood options.
Designing an impact evaluation for a rare outcome
Targeting: According to the European Border Agency (FRONTEX), almost 3 million irregular migrants were detected crossing European borders between 2015 and 2019. This sounds like a lot, but this includes migrants from all around the world, and once one starts thinking about annual per capita rates from a particular country, the rates are incredibly low. So our first form of targeting was to choose the African country with the highest per-capita rate of irregular migration to Europe in 2016 and 2017: The Gambia had 12,927 irregular migrants recorded by FRONTEX in 2016 and 8,522 in 2017. But this is still only 0.38 percent of the population.
We then employed geographic and demographic targeting to try to select a target group for the intervention that would have a higher incidence of migration. We used Gambian surveys to select the administrative regions and then districts with the highest rates of irregular migration. We then decided to focus on young men aged 18-30, since they are the most likely to migrate this way, and did a door-to-door listing to get this sample. During the listing in March/April 2019, we further screened for those most likely to migrate by asking the household for which male aged 18-30 was most likely to migrate in the next five years.
Outcome measurement: apart from trying to target the interventions towards the most at-risk population, and taking a relatively large sample size, the other thing that can be done with a rare outcome is to expand the set of measures one is interested in, to measure other outcomes that are part of the causal chain but more likely to occur, or that might take place more quickly. This is a common approach in medical trials, where a surrogate outcome that is considered an early predictor of later illnesses is often used. In the migration domain, this involved us pre-specifying that we would also look at knowledge and intentions about migration, as well as at other more common types of migration: regional migration to Senegal and internal migration to the capital city of Banjul.
The other key issue with a rare outcome like migration is that you want to really work hard to reduce attrition, since the people who are hardest to locate may be precisely the ones who have the outcome of interest (you cannot find them because they moved). Here we tried a range of different approaches, including getting multiple contact details, using both face-to-face and phone surveys, asking household and community members to provide proxy reports, and having youth share their location using Whatsapp. In our 18-month follow-up survey between October and December 2020, we were able to obtain the location and migration outcome of all but 1 person out of our 3,641 sample!
Tackling a problem that (temporarily?) disappeared by itself
We designed this study in 2017-18, when The Gambia had the highest irregular migration rates in Europe, and launched the study in early 2019. However, in complete surprise to us, irregular migration from The Gambia to Europe fell dramatically during the period in which we designed and implemented our interventions, and particularly during our follow-up period. Figure 1 shows this using Frontex data. This drop is driven by several factors. First, after the end of the Jammeh government in 2016, and the country transitioning from a dictatorship to a democratic government, European governments became much less likely to grant asylum status to Gambians on the basis of political repression, and increased deportations (along with providing more aid to projects in The Gambia designed to increase local opportunities). Then, in 2020, border closures and fears around the COVID-19 pandemic further reduced both the ability and desire to migrate. The extent to which this dropped was unanticipated by us, and not well-known in the country, even ex-post. In June 2021, we held a dissemination event at the University of The Gambia, and showed 30 local researchers, aid officials, and journalists the Frontex numbers for each year from 2014-2017, and asked them what they thought rates had been in 2018, 2019, and 2020. The lines p10, p25, p50, p75, and p90 report the 10th, 25th, 50th, 75th, and 90th percentiles of their beliefs on these numbers. The median belief was for the migration numbers to have fallen from 8,522 in 2017 to 7,532 in 2018, and down to 6,000 in 2020. That is, most locals believed there had been a much more gradual decline than recorded.
So, if we had just done a before-after analysis, we would have found much less irregular migration after our interventions than before and naively concluded we were incredibly successful. Despite all our screening efforts, we ended up having only 1.1 percent of control group individuals making a backway migration attempt, and only 0.6 percent making it to Europe during the 18-month period – leaving very little scope for our interventions to have any impact on this outcome. The is not unique to our study – a couple of other recent evaluations of information campaigns in Guinea and Nigeria also have this problem of hardly any of the control group migrating irregularly over their follow-up period.
This is where our other measures of interest come in – we do find that intentions to migrate irregularly remain high, with 28% of the control group said they will surely move in the next five years and 52% said they are sure or likely to migrate. The vocational training treatment lowered intention to migrate to Europe by 5-7 percentage points. The Senegal treatment increased intentions to migrate to Senegal 3 to 8 percentage points. Although we did not change actual migration to Europe, our interventions did change regional and internal migration: The Senegal and vocational training interventions increased the likelihood of residing somewhere in Senegal at the time of the follow-up survey by, respectively, 2.2 and 2.6 percentage points, which more than doubled the rate in the control group. Temporary internal migration to the capital city of Banjul was reduced in these groups.
We used the Social Science Prediction Platform to see how these treatment effects compared to the predictions of migration experts, asking them for both their point estimates and 90 percent confidence intervals of treatment effects, as well as of attrition. Our ability to track the location of migrants was far greater than expected by migration experts. Our estimated treatment effects on steps towards the backway and attempted backway migration are smaller than expected by academic experts, but within their 90 percent confidence intervals. The impacts on migration to Senegal are smaller than predicted for the Senegal treatment (experts predicted a 5-7 percentage point effect, compared to our effect of around 2 p.p.), and slightly larger than predicted for the vocational training treatment
A good potential use case for registered reports: In describing the JDE registered reports, Andrew Foster noted that they see this process as potentially encouraging people to take on riskier projects where there are states of the world in which the interventions might not work, or where there is a chance that someone else might scope the paper by doing something similar before you are done. We realized this type of project is expensive, time-consuming and risky, and did submit the paper as a JDE stage 1 registered report before collecting any follow-up data. Having this acceptance did provide us with some comfort when we saw our results and when we learned of other information campaign evaluations taking place around the same time.
Join the Conversation