A new study was recently carried out by the Water and Sanitation Program (WSP) of the World Bank on how to unlock the potential of Information and Communications Technology (ICTs) to improve Water and Sanitation Services in Africa. According to a Groupe Speciale Mobile Association (GSMA) report, in 2014 52% of all global mobile money deployments were in Sub Saharan Africa and 82% of Africans had access to GSM coverage. Comparatively, only 63% had access to improved water and 32% had access to electricity. This early adoption of mobile-to-web technologies in Africa provides a unique opportunity for the region to bridge the gap between the lack of data and information on existing water and sanitation assets and their current management — a barrier for the extension of the services to the poor.
Despite the differences in various methodological and data handling choices, which I discussed below in my original post, it is clear that the interpretation of whether one believes the results of Miguel and Kremer are robust really rests on whether one splits the data or not. Therefore it is important to solely focus on this point and think about which choice is more justified and whether the issue can be dealt with another way. A good starting point is the explanation of DAHH in their pre-analysis plan as to why they decided to split the data into years and analyze it cross-sectionally rather than the difference-in-difference method in the original MK (2004):
The data from a stepped wedge trial can be thought of as a one-way cross-over, and treated as such, by comparing before and after in the cross-over schools (group 2) and accounting for the secular trend using the non-crossing schools (groups 1 and 3). However, such an approach requires assumptions about the uniformity of the trend and the ability of the model to capture the secular change, and as such loses the advantage of randomization.
However, let's accept for a second DAHH's argument that there's something strange with Group 2 and we're wary of it. Them it seems to me that the solution is simple: why not look at the two clean groups that never change treatment status the whole study period of 1998-1999. In other words, exclude Group 2, pool all the data for 1998 and 1999 and compare the effects between Group 1 and Group 3. Sure, we lose power from throwing out a whole study arm, but if the results stand we're done! Thankfully, Joan Hamory Hicks was able to run this analysis and send me the table below, which is akin to their Table 3 in their original response:
As you can see, all effect sizes on school participation are about 6 percentage points (pp), which is remarkably close to the effect size of 7 pp in the original study. The p-values went up from <0.01 to <0.05, but that is fully expected having shed a third of the sample. So, even if you think that there is something strange going on with Group 2, for which the visual inspection presented by DAHH in Figure 3 is really not sufficient, you still have similarly-sized and statistically significant effects when making the cleaner comparison of Groups 1 & 3. Problem solved?
I want to conclude by making a bigger picture point about replications. They are really a really expanded version of robustness checks that are conducted for almost any paper. It's just that the incentives are different: authors want robustness and replicators might be tempted to find a hole or two to poke in the evidence and "debunk" the paper (if I had a dime yesterday for every deworming debunked tweet...). But, when that happens, I start worrying about multiple hypothesis testing. We now know and have tools for how to deal with multiple inference corrections, when the worry is Type I errors (false rejections of a correct null). But, what about Type 2 errors? After all this is exactly what a replicator would be after: finding a manner of handling the data/analysis that makes the results go away. But, how do we know whether that is a true "failure to reject" or a Type 2 error? Even in studies with 80% power, there is a 20%chance that each independent test will fail to reject under the null of a positive effect. The more of these you try, the more likely you'll come across one or two estimates that are insignificant. What to do about that?
To be fair to the authors, they were at least aware of this issue, mentioned on page 7 of the PAP:
But, then this is where it would have been really important to have a very clear PAP, describing only a very few, carefully methodologically justified, analyses proposed and sticking very strictly to it. But, every step of the way when the authors decide to weight or not weight the data (cluster summaries), splitting the data by year, adjusted/unadjusted estimates, alternative treatment definitions dropping large numbers of observations, etc. there is a fork and the fork opens up more roads to Type 2 errors. We need replications of studies that are decently powered themselves, where the replicators are careful to hoard all the power that is there and not scatter it along the way.
We aim to deal with this problem by making a small number of analyses using as much of the original data as possible at each stage and concentrating initially on the direct intervention effects on the major study outcomes.
I hope that this update has brought some clarity to the key issues that are surrounding the debate about the publication of the replication results and the accompanying flurry of articles. I was an unwitting and unwilling participant of the Twitter storm that ensued, only because many of you were responsible for repeatedly pointing out the fact that I had written the blog post below six months ago and linking to it incessantly throughout the day. I remain indebted to our readers who are a smart and thoughtful bunch...
This post follows directly from the previous one, which is my response to Brown and Wood’s (B&W) response to “How Scientific Are Scientific Replications?” It will likely be easier for you to digest what follows if you have at least read B&W’s post and my response to it. The title of this post refers to this tweet by @brettkeller, the responses to which kindly demanded that I follow through with my promise of reviewing this replication when it got published online.
“You cannot solve a problem you haven’t fully understood.” – Chief Justice Mutunga, April 15, 2015
It’s difficult to know whether you’re succeeding in any institution – public or private – if you don’t set targets and collect data to measure progress against them. Courts are no different.
The Kenyan Judiciary has been making great strides in performance management. A ceremony at the Supreme Court in Nairobi last month was the latest step. Chief Justice Willy Mutunga signed “Performance Measurement and Monitoring Understandings” with the heads of Kenya’s courts.
These commit each court to targets such as hearing a case within 360 days, delivering judgments within 60 days of the end of a trial, and delivering a minimum number of 20 rulings a month.
This is the second in our series of posts by students on the job market this year.
Relaxing supply-side constraints is not always sufficient to ensure delivery of public services to poor and remote communities. It may be necessary to stimulate demand by exploiting local agents who can link the relevant parties. We thus see the use of intermediaries in a variety of sectors in development; for example through the use of agricultural extension agents (Anderson 2004), loan officers for microfinance (Siwale 2011), and referral incentive programs – like that used by the British colonial army in Ghana (Fafchamps 2013). My job market paper studies the use of intermediaries in the maternal health sector in the Western Province of Kenya. I use an RCT to evaluate the efficacy of financial incentives for Traditional Birth Attendants (TBAs). The program provides payments for TBAs to encourage pregnant women to attend antenatal care (ANC) visits at a local health facility. In this way, TBAs link pregnant women with health facilities, the TBAs’ rivals. This potential competition, which is absent from most intermediary relationships, is a noteworthy feature of this program as it creates a nontrivial incentive problem for the TBA.
Take a moment and think of the women who inspire you. Make a list. Who are the top 11 women? Would you include a construction worker from Jamaica? How about a midwife in Sudan or a jewelry maker in Costa Rica? What about a student from India or a small business owner in Egypt?
When most of us think about people who inspire us, we consider world leaders, celebrities, or those who’ve changed the course of world history. Or we might think of individuals who have had a significant influence in our lives—our role models or people we strive to emulate. The people who make it to our “inspiration list” are there because we relate to them, regardless if we’re man or woman.
As we celebrate International Women’s Day this week, we present 11 stories of women around the world who’ve made amazing strides to achieve their goals and make long-lasting impacts on the lives of their children, families and communities.
As we reflect on the promise of the New Year in Africa, the irrefutable link between peace and development has never been clearer after my recent travels.
Earlier this month, I joined leaders from 53 African nations, the United Nations, and the African and European Unions at the Elysee Summit for Peace and Security in Africa to talk candidly about how our countries can work together to maintain and enhance peace.
We talked about what this would mean in practice. For example, we must curb drug trafficking on the continent, increase financing for African peacekeeping operations, fight terrorism, manage borders more securely, include women fully in the political and economic decision-making process, and condemn the intolerable persistence of sexual violence when conflicts do occur. This last measure was strongly endorsed by the First Ladies of the Summit who also met to discuss issues of gender, development, and women’s rights.
The African leaders recognize that for many of these measures to work, economic development must be twinned with public and private investment in business, technology, agriculture, climate-smart policies, and in young people who are fast becoming Africa’s driving force and future. Africa is now the world’s youngest continent and how well we meet the skills needs of our young people will greatly determine the continent’s future.
When Jane Otai said there are flying toilets in slums of Nairobi, most of her audience, like me, was trying to figure out what she meant.
A few others laughed softly. Because there are no toilets, she said, “people just do it [in bags] and throw it on the rooftops.” And it is really difficult for women and girls, she added.
New developments and curiosities from a changing global media landscape: People, Spaces, Deliberation brings trends and events to your attention that illustrate that tomorrow's media environment will look very different from today's, and will have little resemblance to yesterday's.
This week's Media (R)evolutions: How Many Kenyans Use Mobile Money?
These are some of the views and reports relevant to our readers that caught our attention this week.
Mobile Phones and Violent Conflict - Is there a Connection?
“Over the past several years, a significant body of research has examined how communication technologies are transforming social, political, and economic dynamics in societies around the world. Much of this work has observed the positive effects of these technologies on improving civic engagement, increasing transparency, supporting free and fair elections, fostering economic development, and preventing violent conflict. We at NDI have developed numerous programs using communication technologies to improve democracy and good governance across borders and issue areas.
A new report, “Technology and Collective Action: The Effect of Cell Phone Coverage on Political Violence in Africa,” sheds light on the less beneficial aspects of communications technologies.” READ MORE
These are some of the views and reports relevant to our readers that caught our attention this week.
Keeping online newsrooms sustainable in the developing world
“Independent news websites in the developing world tend to be on shaky ground, as they often oppose a corrupt regime or report in a censored environment. Their work attracts hacking attempts from the government and sends advertisers fleeing.
Offering a solution to this two-pronged problem of sustainability for these sites is Media Frontiers, a social-purpose enterprise of International Media Support, a nonprofit, Danish press freedom organization.” READ MORE