Guest post by Arthur Alik-Lagrange and Lea Rouanet
Impact evaluations of interventions aiming at reducing intra-partner and sexual and gender-based violence (IPV-SGBV) have mostly failed at detecting statistically significant impacts.
Changing behaviors at the individual, couple and community level, these interventions are usually holistic. They tackle multiple related issues at the same time such as violence and controlling behavior within households and gender norms at the community level. This implies high intervention costs and a decrease in the number of communities that can be reached ( Abramsky et al., 2014).
There are good reasons to expect low power in this context. In fact, out of 13 papers based on RCTs listed in Bourey et al. (2015), nine explicitly mention power issues. IPV-SGBV outcomes show low means as they are typically (i) less prevalent than other health outcomes and (ii) under-reported. In addition, IPV-SGBV prevalence being often driven by community characteristics, the intracluster correlation (ICC) of such outcomes is generally high. Both low prevalence and high ICC will reduce power, increasing the number of clusters needed to detect an impact on IPV-SGBV prevalence.
Getting the power right on these interventions is particularly important. First, given the multi-faceted/holistic approaches to many of these programs, they can be expensive. Second, given the volatile and unpredictable nature of the problem these interventions are designed to address, it’s particularly important that they be powered to rule out unintended impacts.
In order to take a look at how serious an issue this is, we take 32 DHS (Demographic and Health Surveys) surveys from sub-Saharan Africa and we compute the number of clusters needed to identify a 20% decrease in IPV-SGBV outcomes, women’s disempowerment outcomes, as well as 9 other key health variables, at given power, confidence level and cluster size using Hemming at al. (2011) formula.
As shown in Figure 1, we find that IPV-SGBV outcomes are associated with a very high number of clusters. Other measures of partner’s control (justification of beating, control over wife’s life, etc.) also require a high number of clusters to be considered as outcomes of interest in an impact evaluation. Several robustness checks not reported here show that these differences:
Figure 2 reports the mean and 95% C.I. of these two statistics. We can observe that (i) ICCs of IPV-SGBV outcomes are in the same range as ICCs of other outcomes measured in the DHS and (ii) prevalence is typically lower for IPV-SGBV outcomes. We also find that controlling behaviors on the part of the husband are reported at higher means, with ICCs on the higher end.
These results suggest that the low power documented for IPV-SGBV outcomes is mostly due to low means. For gender norm related outcomes, such as accepting violence and controlling behavior, low power is also driven by high ICCs.
Figure 2. Mean prevalence and ICC across DHS
Clustered RCTs of IPV-SGBV pilot interventions are not destined to fail in providing significant impact estimates. The structural low power documented here does mean these evaluations need to be approached with a fair amount of planning and some caution. Here is a non-exhaustive list of avenues to explore:
Impact evaluations of interventions aiming at reducing intra-partner and sexual and gender-based violence (IPV-SGBV) have mostly failed at detecting statistically significant impacts.
Changing behaviors at the individual, couple and community level, these interventions are usually holistic. They tackle multiple related issues at the same time such as violence and controlling behavior within households and gender norms at the community level. This implies high intervention costs and a decrease in the number of communities that can be reached ( Abramsky et al., 2014).
There are good reasons to expect low power in this context. In fact, out of 13 papers based on RCTs listed in Bourey et al. (2015), nine explicitly mention power issues. IPV-SGBV outcomes show low means as they are typically (i) less prevalent than other health outcomes and (ii) under-reported. In addition, IPV-SGBV prevalence being often driven by community characteristics, the intracluster correlation (ICC) of such outcomes is generally high. Both low prevalence and high ICC will reduce power, increasing the number of clusters needed to detect an impact on IPV-SGBV prevalence.
Getting the power right on these interventions is particularly important. First, given the multi-faceted/holistic approaches to many of these programs, they can be expensive. Second, given the volatile and unpredictable nature of the problem these interventions are designed to address, it’s particularly important that they be powered to rule out unintended impacts.
In order to take a look at how serious an issue this is, we take 32 DHS (Demographic and Health Surveys) surveys from sub-Saharan Africa and we compute the number of clusters needed to identify a 20% decrease in IPV-SGBV outcomes, women’s disempowerment outcomes, as well as 9 other key health variables, at given power, confidence level and cluster size using Hemming at al. (2011) formula.
As shown in Figure 1, we find that IPV-SGBV outcomes are associated with a very high number of clusters. Other measures of partner’s control (justification of beating, control over wife’s life, etc.) also require a high number of clusters to be considered as outcomes of interest in an impact evaluation. Several robustness checks not reported here show that these differences:
- Are statistically significant: a regression analysis shows that within DHS surveys (holding sampling constant), IPV-SGBV outcomes require significantly more clusters than a reference variable such as anemia. We find that the only variable showing the same kind of pattern in the data used is piped water access, which is very clustered geographically, by construction, and expected to suffer less underreporting.
- These results hold when multiplying the prevalence of IPV-SGBV outcomes by two (making the conservative assumption that violence is not reported in half of the cases).
Figure 1. Number of clusters needed to detect a 20% decrease
(Power=80%,=5%, cluster size=60)
Figure 2 reports the mean and 95% C.I. of these two statistics. We can observe that (i) ICCs of IPV-SGBV outcomes are in the same range as ICCs of other outcomes measured in the DHS and (ii) prevalence is typically lower for IPV-SGBV outcomes. We also find that controlling behaviors on the part of the husband are reported at higher means, with ICCs on the higher end.
These results suggest that the low power documented for IPV-SGBV outcomes is mostly due to low means. For gender norm related outcomes, such as accepting violence and controlling behavior, low power is also driven by high ICCs.
Figure 2. Mean prevalence and ICC across DHS
Clustered RCTs of IPV-SGBV pilot interventions are not destined to fail in providing significant impact estimates. The structural low power documented here does mean these evaluations need to be approached with a fair amount of planning and some caution. Here is a non-exhaustive list of avenues to explore:
- Increase the number of clusters to improve the precision of your estimates.
- Use IPV-SGBV outcomes with higher prevalence. IPV-SGBV events are less prevalent – but not less dramatic - than other health and socioeconomic bad events. However, a sampling strategy focusing on socioeconomic groups more affected by SGBV will push prevalence up. In addition, careful attention to how to measure these outcomes well (and reduce underreporting) is important. There is a fairly large literature in public health and from sources like the WHO on how to measure this well.
- Stratify the randomization as much as possible. If you first sort clusters in homogeneous groups and then randomize within these groups, then your ICC will go down and this will buy you extra power. For instance, pair matched randomization has been used in Abramsky et al. (2014). The caveat here is that this often requires access to very sensitive information, such as reliable, disaggregated IPV-SGBV prevalence statistics.
Join the Conversation