Transparency remains the sine qua non of the international development sector. We preach its value to others; we see open records laws, for example, as key indicators of good governance. But what we rarely discuss in the context of access to information, is the value not just of the data itself, but of transparency about how the data is analyzed.
Lots of studies come across each of our desks everyday. Some come directly from the folks conducting the studies; the Pew Research Center, for example, sends me a weekly email of their work. Some studies we learn about via the media: a news outlet itself or a pollster has completed a survey, and a news story summarizes the major takeaways. And some studies come to us another step removed: we pick up a book by Malcolm Gladwell or Ori Brafman, for example, and the author précis a study to argue his own insights.
In some of those instances, we can click on a link or go to an appendix and find some specifics about the study: perhaps if the study is a poll we might be able to learn the exact wording of the questions that were asked. But it is generally impossible to get to the raw data and be able to make sense of it ourselves. We aren’t given the tools (or we don’t have the experience) that would make it possible for us to “check up” on the conclusions of the study.
That may be now changing.
My center, the International Center for Media & the Public Agenda, launched a study this week that assessed how well 13 media outlets in the United States, Britain, and the Middle East covered five recent elections in the “Muslim world”: the election just past in Iraq—and the earlier one in 2005, the elections in 2009 in Afghanistan and Iran, and the 2008 election in Pakistan.
Our study surfaced some terribly interesting insights about what has changed AND what has remained the same about US and UK media coverage of elections in Muslim-dominated countries. But the point of this post, actually, is not to highlight the study’s conclusions but to note how we got to those conclusions - and even more specifically to observe the much greater transparency of our analysis as a result. (If you are interested in the results of the study, you can read the findings here.)
- Bar Graphs – that tracked the frequency of terms used by each news outlet
- Stacked Graphs – that illustrated the aggregated use by all news outlets of various terms of reference.
- Word Trees – that collated every article/transcript used in the study allowing an analysis of word selection by each news outlet
In addition, for a further evaluation of the coverage of individual news outlets – specifically the use of specific clusters of terms – the research teams used a fourth kind of ManyEyes visualization.
- Phrase Nets – that diagramed the relationships between different words used in the stories
These data visualization made our analysis considerably more efficient and neutral. And our decision to link to the ManyEyes visualizations (which themselves link to our raw data) means that our research is more transparent than any comparable study that we are aware of.
One thing that’s very cool about the data visualizations, housed on the ManyEyes site that we link to, are that they are all dynamic; they can be manipulated online.
Ok, well, that may be “cool,” but so what?
Well, visitors to our site can confirm our research teams’ evaluations of the data. And, in addition visitors to our site can use our open data sets to create their own studies. If, for example, a visitor wanted to evaluate more specifically how the media we studied spoke about gender and elections or wanted to investigate how these media outlets sourced their stories, those new studies could easily build on our current one.
It is our hope that other researchers do build on what we have done – and further it is our hope that this transparency in research becomes the standard not only in academe, but from governments, think tanks and international organizations. In an era when many are advocating for more Open Source practices in IT development and production, we believe that research should be openly sourced and shared as well.
Photo Credits: http://newsinsite.wordpress.com/.
- Visualization 1: Al Jazeera Afghanistan. Wordtree of Al Jazeera's coverage graphically demonstrates how common a topic "fraud" was in framing the Afghan election.
- Visualization 2: Afghanistan Politics & Protests stack graph