Syndicate content

Open Data at the World Bank: 2 years old today

Shaida Badiee's picture

Today is the second birthday of the Bank's Open Data Initiative—announced by the Bank's President, Robert Zoellick on April 20th, 2010:

 

"It's important to make the data and knowledge of the World Bank available to everyone. Statistics tell the story of people in developing and emerging countries and can play an important part in helping to overcome poverty."

 

To mark the occasion, we've created a new blog specifically aimed at discussing data and open data issues related to development. And I'm going to use the opportunity of this first post to briefly look back and recap what's been achieved, and—more importantly—to outline some of the plans we have for moving ahead.

When we started out, we focused on making existing public datasets freely available and as accessible as possible, building on the new Access to Information Policy. To make sure our data was usable, we "ate our own dog food;" that is, we developed our own website using the same data and tools (an application programming interface, or API) that we provide to the public. And we developed an "open" Terms of Use for World Bank datasets, together with a single catalog listing of all our open data resources.

Since then, we've been adding datasets to the catalog at a rapid rate. There are now many more development indicators, from around 2,000 in the initial release to over 8,000 today; a new interface and API for accessing data on the Bank's projects and operations—with project activities geo-coded and shown on maps; datasets on the financial aspects of the Bank's business (we're a Bank, after all); and a library of raw data from household and other surveys—an enormously popular resource for development researchers. This week we've updated the World Development Indicators database—the open data source behind the open data website—with the 2012 edition.

Being more open has helped us provide World Bank data in multiple ways: we've published to the International Aid Transparency Initiative's registry, we've mashed-up World Bank projects together with indicators of development progress, and we've launched interactive applications to better analyze key datasets—such as poverty, climate change, jobs, financial inclusion, and aid flows. We've developed applications for mobile devices, with free applications for all major platforms. And, earlier this month, the Bank launched a new Open Knowledge Repository with over 2,100 research and knowledge reports available under a Creative Commons (CC-BY) license—with more reports being added every week.

That's an impressive list. But we've also learnt that others can often do things better than we can. Software developers have been quick to build applications using the datasets. We had overwhelming responses to the Apps for Development competition we ran last year, and the Apps for Climate competition which closed last month. To solve development challenges, you need to understand them; statisticians and designers have developed impressive, innovative visualizations, and story-tellers like Hans Rosling have brought the numbers to life to help explain and illuminate key development policy issues.

So what's next? How can we manage the growth of online data properties? How do we make sure that the data we provide, which often describe complex phenomena, can be interpreted and used correctly? How can open data have an impact on our mission to reduce poverty?

In the last few months, we've been thinking about these and other questions, and we've concluded that some of the priorities and principles are:

To build better links between open data resources. Integrate open data resources, but be careful not to constrain growth—the principle should be "small pieces loosely joined," using open metadata standards. An improved data catalog, bringing datasets from different data repositories together, is a key priority. We've made a start this week, with a new design for the data website country pages, bringing together resources from several data repositories.

To provide better metadata. Data is not enough—good quality, accessible context and documentation (metadata) is often just as important. Part of this should be about World Bank data specialists providing documentation in searchable, accessible, and usable ways; but it's also important to provide online, community mechanisms for sharing the knowledge of other data producers and users.

To promote interaction and use. We plan to continue to improve the tools available for accessing and visualizing our datasets. As an example, we've recently upgraded our DataBank application, so that you can save and share reports, charts, and maps. We aim to develop and share new and innovative ways to help you explore data.

To be more local. Data are often most powerful when used to address pressing local concerns. Meeting this demand means finding ways to provide more datasets at sub-national or transactional levels, with higher-frequency updates and with improved support for local languages. We're working on "curating" key development indicators at sub-national levels, and we're working with countries to support their efforts to publish datasets openly.

To listen more, and to support the global community. We think it should be easier to ask questions, receive answers, provide feedback—and we want to be responsive to what we hear and to support the global open data community. We're taking some actions: we're planning to support on-line discussions about the datasets we publish, through blogs and community forums, and in partnership with the US Government we're hosting the next International Open Government Data Conference here at World Bank headquarters, on July 10-12, 2012.

We're starting to implement some of these plans now, but we're very eager to hear from you. What do you think of these ideas? What are we still missing? I'm looking forward to your thoughts.

Comments

Submitted by Jiesheng on
Could you release all IDA-12 docuemts especially discussions papers and meetings minutes (if there were any?) Thank you very much

All of the World Bank's public documents that are currently available can be found on the World Bank's Documents and Reports page (http://documents.worldbank.org/curated/en/home). Just search for IDA 12. I saw one document, a retrospective report on IDA 12 which might be of interest to you ( http://documents.worldbank.org/curated/en/2003/03/2176358/ida-12-2000-2002-retrospective-overview). If there are documents that are not currently available online, a request for them can be submitted from the Access to Information page (http://go.worldbank.org/TRCDVYJ440).

Submitted by Anonymous on
I would implement a plan to use the website to benefit the needy. Meaning offer a plan to (hypothetically thinking) house homeless families and post on all websites if you can get so many likes you will donate X amount of dollars to the homeless.

Submitted by EllaRHagen on
Congrats on the two years! As someone who uses the data, it would be really wonderful if there was a staff person to ask about the gaps in the data and perhaps even someone to fill them! Sending not-very-complicated requests into the help email address has gotten me no where. Would love to see some more support to make the data truly useful!

Submitted by azita on
Thanks for the kind words! We've been learning and growing at a fantastic rate over the last 2 years and supporting others to use our data is a priority for us. To help with growing demand, we'll be launching new on-line tools to support users in the coming months, which among other things will make it easier for you to see frequently asked questions and to learn from other users' experiences. I lead the Client Services team in the Data Group, and I've dropped you an email separately to see if I can help with your query. A key issue that affects data coverage and availability is the constraint on the statistical and data capacity of countries, and we do work with the global community to support efforts to make improvements. A good place to learn more about this issue is the PARIS21 partnership at http://www.paris21.org we also monitor statistical capacity at http://data.worldbank.org/data-catalog/bulletin-board-on-statistical-capacity.

Submitted by Azar on
Congrat to Sheida and her very able data team. It is with much delight to have such a wealth of country data at your finger tips. May I suggest to add a compilation of links to countries official web sites, or perhaps some specific web sites such as those of ministries of finance, central banks, from where one can get the latest data.

Submitted by Tariq Khokhar on
Azar - great minds clearly think alike, we're working on exactly this! As you know, many of the countries we feature have national open data initiatives - so for countries Kenya we could provide a link to https://opendata.go.ke/ and there are primary sources (like statistics offices) in most countries I think we should link back to. It's a bit tricky to implement in practice - our country pages are built automatically from APIs - there's little or no manual editing of content on each page, we just suck in content from other (internal) sources. Right now, I don't know of a well-curated, API-available list of public (ideally "open") primary government data sources that we could sensibly pull in from on most of our country pages. If anybody knows of a one with good coverage, please drop us a note or leave a comment. If not, let's make one!

Submitted by Azar on
Tariq, Thanks for the quick reply and the info of plans in this respect. The usefulness of the proposed set of links, which I can appropriately call it "Linkstan" [watch, this term may have already been used elsewhere], cannot be exaggerated. In my work in many countries where I have had to prepare presentation on the fly for public debt management and other topics in public sector management, I have had to search, often with not much success, for country data links. The links that you might want to include should be to "official" websites, like those maintained by national statistics offices, the ministries of finance, the central banks, plan & budget organizations ... In some cases, you might have to either inform them and/or obtain permission. Best, Azarbayejani

Excuse my english: Congratulations with a great concept - may the child live and develop for many years ahead. You consider: " How can we manage the growth of online data properties? How do we make sure that the data we provide, which often describe complex phenomena, can be interpreted and used correctly? How can open data have an impact on our mission to reduce poverty?" Good and relevant questions! And you might add: How can you strengthen the use of the statistics in education? Consider some "small-facts" were you extract relevant figures and link up to further studies. Pupils and teachers do get lost in the maze of numbers on the site now! And consider improving the possibility of autoupdating of usermade tables - like I have used it in the graphs-module. Super if possible.. And: put some colours/markers on the graphs - they disappear on my screen and when printing - too bad for a great facility! Hi Karsten

Karsten, Thanks for your comments. You may want to see the Atlas of Global Development at http://data.worldbank.org/news/eatlas-of-global-development-released. Its electronic version is accessible from at http://www.app.collinsindicate.com/worldbankatlas-global/en which is a great resource to be used as a teaching tool. In addition, you can access our DataBank where you can access various databases; create your own queries which can be saved and used at a later time; and view results in table, chart and map formats. The DataBank can be accessed from http://databank.worldbank.org/ddp/home.do?Step=1&id=4. You can always send us an e-mail to data@worldbank.org in case you have any questions.

Submitted by James Gorman on
As a frequent user of such data, I remember how great it was two years ago when this new initiative was undertaken. I now collect data from your databank more than I do anywhere else. However if I may ask, my current research has hit a wall do to the lack of data regarding the World Bank's Small Grants Program (now called Civil Society Fund). I've sent requests, but haven't had any luck. Would you have any advice on how to find this data?

Thank you for the question and the feedback. We are glad the Databank has made your data collection easier. On question of Civil Society Fund, we need to look into it and get back to you in a separate e-mail with our findings. Regards. Azita

These are profound and important questions. Our Poverty & Equity open data dashboard for Indonesia will provide you with an overview of some relevant indicators (see http://povertydata.worldbank.org/poverty/country/IDN). Over the past decade, poverty incidence in Indonesia has declined steadily and the country is currently one of Asia Pacific's most vibrant democracies and has emerged as a confident middle-income country. For much more information about the World Bank's work and programs in Indonesia, please visit: http://www.worldbank.org/en/country/indonesia

Submitted by Anonymous on
The World Bank Open Data blog is uniquely positioned as a central resource for access to the World Bank's wealth of knowledge. It'd be (dare I say) "fun" if this blog could identify a project, rally support and coordinate a crowdsourced project (where everyone is working together). I, for one, would be keen to assist a project that has deep, meaningful and measured impact. This concepts speaks directly to promoting interaction and being local. Keep it in the back of your mind as the blog takes a life of its own.

Submitted by Sam Laird on
Open Data - which I use for teaching in Africa - is a great initiative, especially for students in developing countries who do not now have to buy WDI on CD-ROM. Thanks also to the really helphul support team! Keep up the good work! Sam

Sam - thanks for your kind worlds, the support team will be pleased to hear them. We still get plenty of demand for the WDI CD-ROMs - lots of people are just used to the interface and if you've got no internet access, they're a useful resource. May I ask where in Africa you are? How do you find the online resources perform on your internet connection? tk

Submitted by Anonymous on
i was checking the data for africa and middle east and the latest update is for 2009 ... is there any updates for 2010 or estimation for the last two years? when is the paln to update the data ? thanks regards,

Thank you for using our data. We have updated country aggregates with 2010 data for many of the indicators. Which specific series you have been looking for? Below, you will find an example with 2010 data for Middle East and North Africa region. GDP (current US$) 1,206,982,758,169 GDP growth (annual %) 4.3 Population, total 331,263,000 please let us know specifics of your request, so we can be of more help. Regards, Azita

Submitted by Anonymous on
How do you confirm the validity of the records? Example, Tanzania - the map, is it an error? there are no islands Zanzibar and Pemba.

Thanks for your question. It happens that Zanzibar and Pemba are on the map but they're just obscured by the project location dots. In terms of validity, our data sets in general undergo quality assurance procedures but the extent of such work will vary depending on the original source of the data and the nature of the indicators. If you have questions on specific data, we will be happy to follow up.

Submitted by Anonymous on
congratulations on your 2nd Bday! I just have one observation. Any data available on the site is only as good/accurate as the data collected in the respective countries. So how much suport is availble to ensure quality/reliable data is provided to the system from the respective countries? I know mnay countries have from very adhoc to very advanced mechanisms on data collection/compilation. I think it would be very useful for all of us who use the data to know this when we use this in our research papers etc.

Thanks for your comment - I agree with your observation and you can see some useful measures in the Bulletin Board on Statistical Capacity (BBSC) here: http://data.worldbank.org/data-catalog/bulletin-board-on-statistical-capacity . The World Bank is trying to help countries improve their data quality through financial and technical assistance. In many cases for national statistics, improving quality involves improving sample frames, survey questionnaires and quality assurance processes and as such, improvements will only be realised in the medium term.

Submitted by Ephraim on
The current database is hard to search. I tried the beta version today and i spent a lot of time to build a small database of three countries. The world bank tries to analyze data for its users and give cool graphs. for me, these cool summarized graphs are not useful as i need to get the raw data and analyze it myself. the best data organization is the FAOSTAT, where i am allowed to choose a country, the data type & period of coverage. This is not easy to do in the worldbank database. Please simplify the data and let users do the analysis.

Add new comment