The author is grateful to Nandini Krishnan, Umar Serajuddin and Sebastian Trenner for their very useful suggestions.
Something new and important is happening in the Arab world, and it has so far gone largely unnoticed. Since the beginning of the 2011 revolutions, statistical agencies in the North Africa and the Middle East have started to open up access to their raw data and sharing it not only with selected individuals and institutions but also with the public at large. This amounts to a cultural revolution the implications of which are exciting and wide ranging.
National statistical agencies worldwide are in charge of gathering data to determine key social statistics on such things as poverty, unemployment, health or education. Some of these statistics are produced from data collected by state administrations such as the Ministry of Health or the Ministry of Education while other statistics are produced by means of household or individual surveys implemented by statistical agencies. This is the case for example of statistics on poverty or unemployment which are compiled using Household Income and Consumption Surveys (HICS) or Labor Force Surveys (LFS). These last surveys rely on questionnaires that are administered by trained interviewers at the household level and that collect private information on income, consumption or employment. This sort of detailed data is invaluable for research, and statistical agencies worldwide usually provide researches with access to individual and household level data (micro data for short) but with key identifiers such as names and addresses of interviewed households and individuals removed. It is a standard practice that allows researchers to produce a wide range of studies that can inform public debates while complying with privacy and data protection laws.
For a very long time, statistical agencies in the Arab world have refrained from providing access to individual or household records to anyone requesting them, with only a few exceptions. This refusal was typically grounded in the argument that private and sensitive information could not be shared with the public. The Arab world was not alone in this practice. Several statistical agencies worldwide in rich and poor countries refuse access to micro data usually citing privacy and data protection laws. In some specific cases, legislation may be an impediment to the sharing of micro data but in most cases the refusal is based on fear; fear that the data will be used in improper ways, fear of someone finding mistakes in the official statistics that are published or fear of public disclosure of information that would embarrass the government. Two countries better known for their fragility than their openness were notable exceptions to this trend in the MENA region. The Palestinian Central Bureau of Statistics (PCBS) was perhaps the first agency in the region to provide open access to micro data even before the Arab spring and the Central Organization for Statistics and Information Technology (COSIT) of Iraq had granted access to its 2006-07 HICS data through one of the World Bank data repositories.
With the advent of the Arab spring, attitudes towards data openness have begun changing in a number of countries across the region. The National Statistical Institute (NSI) of Tunisia has now posted the HICSs and LFSs on their website becoming the first agency in the region to offer access to data in such a comprehensive fashion. The Central Statistical Organization (CSO) of Yemen approved a decree in July 2012 to allow full access to its data to the public. The Egyptian Agency for Public Mobilization and Statistics (CAPMAS) has recently posted on their website 50% of the HICS individual records, departing from their old practice of providing 25% of the data on demand only and on a case by case basis. The Direction of Statistics in Morocco has started since 2011 to provide access to data to various organizations based on bilateral agreements. And countries like Libya, Djibouti and Jordan have granted restricted access to micro data to selected individuals and organizations with unprecedented openness, despite keeping various restrictions in place. These changes amount to nothing less than a revolution when compared with the universal restrictions on access to data of only two years ago.
But the biggest surprise for many observers is that the old fears associated with increased openness have not materialized. The first tests conducted by the World Bank on the newly available data revealed that much of the micro data are of good quality, had not been manipulated and compare very well with world standards. A recent study [1] combined HICS and LFS data to produce quarterly poverty statistics for Morocco. It turned out that the quality of the HICS and LFS data in Morocco is very good and allows for the building of very consistent and reliable statistics. Another study on inequality in Egypt [2] that used four rounds of the Egyptian HICS has shown that these data are of very good quality even when compared to hundreds of household surveys available in the World Bank repository. A study on subsidies that used the HICS of Tunisia shows that the Tunisian data are of good quality. [3] The much criticized and suspicious poverty rate of four percent that Tunisia used to publish before the revolution turned out to be a problem with where the poverty line was set, not with data manipulation on the part of statisticians. These new openness initiatives are showing that the data produced by statistical agencies in the Arab world are of good quality. In fact, data openness is both removing the old fears of government institutions and increasing trust in government institutions on the part of citizens and data users. This is a win-win situation.
There are wide ranging implications to this new openness. Free access to micro data will allow local universities, NGOs and research institutions to access and use this vital data for research. Locally produced analyses and reports will grow exponentially as seen in countries where this process has already begun. International organizations will be able to access and use these data to produce country specific or cross-country comparative analyses and so will any researcher worldwide who has an interest in the region. There will be an unprecedented growth of research on the region that will be made available to the public at large. It will be possible to compare results across studies and stimulate lively debates on issues such as unemployment, poverty or growth based on hard evidence. This process will encourage the emergence of new researchers from the region and will allow these researchers to grow rapidly and contribute to informing policy. It will also provide journalists with a broader range of studies and factual evidence on which to base their reporting. This is a data revolution that - by ushering in an Arab Spring of the mind - will create the conditions for a facts based debate on the key issues confronting the Arab world.
Something new and important is happening in the Arab world, and it has so far gone largely unnoticed. Since the beginning of the 2011 revolutions, statistical agencies in the North Africa and the Middle East have started to open up access to their raw data and sharing it not only with selected individuals and institutions but also with the public at large. This amounts to a cultural revolution the implications of which are exciting and wide ranging.
National statistical agencies worldwide are in charge of gathering data to determine key social statistics on such things as poverty, unemployment, health or education. Some of these statistics are produced from data collected by state administrations such as the Ministry of Health or the Ministry of Education while other statistics are produced by means of household or individual surveys implemented by statistical agencies. This is the case for example of statistics on poverty or unemployment which are compiled using Household Income and Consumption Surveys (HICS) or Labor Force Surveys (LFS). These last surveys rely on questionnaires that are administered by trained interviewers at the household level and that collect private information on income, consumption or employment. This sort of detailed data is invaluable for research, and statistical agencies worldwide usually provide researches with access to individual and household level data (micro data for short) but with key identifiers such as names and addresses of interviewed households and individuals removed. It is a standard practice that allows researchers to produce a wide range of studies that can inform public debates while complying with privacy and data protection laws.
For a very long time, statistical agencies in the Arab world have refrained from providing access to individual or household records to anyone requesting them, with only a few exceptions. This refusal was typically grounded in the argument that private and sensitive information could not be shared with the public. The Arab world was not alone in this practice. Several statistical agencies worldwide in rich and poor countries refuse access to micro data usually citing privacy and data protection laws. In some specific cases, legislation may be an impediment to the sharing of micro data but in most cases the refusal is based on fear; fear that the data will be used in improper ways, fear of someone finding mistakes in the official statistics that are published or fear of public disclosure of information that would embarrass the government. Two countries better known for their fragility than their openness were notable exceptions to this trend in the MENA region. The Palestinian Central Bureau of Statistics (PCBS) was perhaps the first agency in the region to provide open access to micro data even before the Arab spring and the Central Organization for Statistics and Information Technology (COSIT) of Iraq had granted access to its 2006-07 HICS data through one of the World Bank data repositories.
With the advent of the Arab spring, attitudes towards data openness have begun changing in a number of countries across the region. The National Statistical Institute (NSI) of Tunisia has now posted the HICSs and LFSs on their website becoming the first agency in the region to offer access to data in such a comprehensive fashion. The Central Statistical Organization (CSO) of Yemen approved a decree in July 2012 to allow full access to its data to the public. The Egyptian Agency for Public Mobilization and Statistics (CAPMAS) has recently posted on their website 50% of the HICS individual records, departing from their old practice of providing 25% of the data on demand only and on a case by case basis. The Direction of Statistics in Morocco has started since 2011 to provide access to data to various organizations based on bilateral agreements. And countries like Libya, Djibouti and Jordan have granted restricted access to micro data to selected individuals and organizations with unprecedented openness, despite keeping various restrictions in place. These changes amount to nothing less than a revolution when compared with the universal restrictions on access to data of only two years ago.
But the biggest surprise for many observers is that the old fears associated with increased openness have not materialized. The first tests conducted by the World Bank on the newly available data revealed that much of the micro data are of good quality, had not been manipulated and compare very well with world standards. A recent study [1] combined HICS and LFS data to produce quarterly poverty statistics for Morocco. It turned out that the quality of the HICS and LFS data in Morocco is very good and allows for the building of very consistent and reliable statistics. Another study on inequality in Egypt [2] that used four rounds of the Egyptian HICS has shown that these data are of very good quality even when compared to hundreds of household surveys available in the World Bank repository. A study on subsidies that used the HICS of Tunisia shows that the Tunisian data are of good quality. [3] The much criticized and suspicious poverty rate of four percent that Tunisia used to publish before the revolution turned out to be a problem with where the poverty line was set, not with data manipulation on the part of statisticians. These new openness initiatives are showing that the data produced by statistical agencies in the Arab world are of good quality. In fact, data openness is both removing the old fears of government institutions and increasing trust in government institutions on the part of citizens and data users. This is a win-win situation.
There are wide ranging implications to this new openness. Free access to micro data will allow local universities, NGOs and research institutions to access and use this vital data for research. Locally produced analyses and reports will grow exponentially as seen in countries where this process has already begun. International organizations will be able to access and use these data to produce country specific or cross-country comparative analyses and so will any researcher worldwide who has an interest in the region. There will be an unprecedented growth of research on the region that will be made available to the public at large. It will be possible to compare results across studies and stimulate lively debates on issues such as unemployment, poverty or growth based on hard evidence. This process will encourage the emergence of new researchers from the region and will allow these researchers to grow rapidly and contribute to informing policy. It will also provide journalists with a broader range of studies and factual evidence on which to base their reporting. This is a data revolution that - by ushering in an Arab Spring of the mind - will create the conditions for a facts based debate on the key issues confronting the Arab world.
[1]Douidich et al. (2013) “Estimating Quarterly Poverty Rates Using Labor Force Surveys: A Primer”, World Bank Policy Research Working Paper No.6466
[2]Hlasny and Verme (2013) “Top Incomes and the Measurement of Inequality in Egypt”, mimeo
[3]World Bank (2013) Evaluation de la Compensation Energetique et la Protection Sociale en Tunisie, mimeo
Join the Conversation