About a year and a half ago, I wrote a blog about the Organisation for Economic Co-operation and Development’s (OECD) PISA for Development initiative.
Most of us are already familiar with the OECD’s PISA exercise, which is a test that assesses the reading, mathematics, and science competencies of 15-year olds around the world. The aim of PISA for Development is to identify how PISA can support evidence-based policy making in developing countries that, until now, have been unable or unwilling to participate. The expected outcome is to produce a set of enhanced student assessment instruments that are tailored to the needs of these countries, but which also produce reading, mathematics, and science scores on the same scale as the main PISA survey. In that earlier blog, I made three wishes for the initiative.
Have any of my wishes been fulfilled?
It’s too early to tell because project instrument development has just begun, data collection is due to take place between 2015 and 2016, and reporting will only occur in 2017. Nonetheless, there has been sufficient progress to allow me to make a few observations about where things seem to be headed and to flag some issues that may need more focused attention and discussion by the participating countries and other stakeholders, including development partners, donors, and civil society.
Wish #1 - A test with questions that 15-year-olds in emerging and developing economies can actually answer.
The OECD’s original plan was to draw solely on their existing pool of 337 PISA questions to create the PISA for Development test. The reasoning was that this would, among other things, help reduce costs. In addition, if the current question pool proved too difficult, new questions could be developed in future rounds. In my earlier blog, I foresaw problems, given that even the middle-income countries participating in the regular PISA surveys can end up with so many zero scores on the test that they are unable to do much with the data.
Eighteen months on, it seems that the OECD has moved away from its initial stance. It’s seriously exploring the possibility of using suitable questions from other regional and international assessments to supplement their existing PISA questions and better flesh out the lower end of the PISA scale. The challenges to obtaining and incorporating these questions into the PISA for Development test include the need to adhere to the other assessment programs’ guidelines for keeping their test questions secure so that they can be reused in future rounds, which is feasible. As of today, it’s looking a bit more likely that the PISA for Development test will be something that 15-year-olds in emerging and developing economies can actually answer. Wish #1 is on the way to being granted.
Wish #2 - A test that emerging and developing economies can afford.
Based on current budget figures, the PISA for Development pilot is about twice as expensive as the regular PISA exercise. It involves a considerable amount of developmental work (such work is mainly one-off and costs would be expected to go down in future rounds). For example, the background questionnaires need to be adapted to better fit the contexts found in developing countries and a methodology needs to be developed to include out-of-school students in the assessment. None of these issues have prevented countries from signing up for the pilot – Cambodia, Ecuador, Guatemala, Paraguay, Senegal, and Zambia. Nor has it prevented several donors and development partners (including the World Bank) from providing significant financial support to make this participation possible. Such enthusiasm and support is firmly based on the anticipated project benefits for the participating countries, including the likely payoff from the project’s strong focus on technical capacity building.
Looking beyond the developmental phase, however, it will be important to take a hard look at the long-term sustainability of PISA for Development (whether as stand-alone or folded into the regular PISA survey), particularly if countries are ultimately expected to handle the costs of participation on their own. Until such issues have been addressed, it remains unclear whether this is a test that emerging and developing economies can afford in the long term. Wish #2 still not granted.
Wish #3 - A test that contributes to learning for all.
The PISA for Development initiative is meant to contribute to the global learning agenda that will emerge from the next round of global education goals. A global learning goal requires a way to monitor the learning of all children and youth, whether in or out of school. This is a need that’s not being addressed by the current fractured landscape of assessment initiatives.
The work being done under PISA for Development- which includes partnerships with other assessment programs- is a step towards creating a more cohesive framework for global monitoring work.
At the same time, the pilot faces challenges in becoming a truly useful tool for the global learning agenda in that it must demonstrate the ability to collect meaningful learning data on the entire 15-year-old cohort in a country, including those who are out of school.
The technical and logistical challenges are daunting. How do you collect learning data from 15-year-olds who are illiterate or semi-literate? What about those with disabilities? Is it okay to use oral or pictorial assessments and, if so, what are the implications for mapping results to the PISA scale?
The PISA for Development pilot aims low in this area by promising to test and validate different sampling approaches and techniques for identifying informants, and also to try out different test administration practices. The aim is that the lessons learned will allow future PISA cycles to incorporate nationally representative samples of the entire cohort of 15-year-olds, including those who are out-of-school.
Kudos to the PISA for Development team for taking on this challenge, but there is a long way to go before it’s clear whether it’s a game changer when it comes to creating a test that truly contributes to learning for all, in the most fundamental sense. Wish #3 partially granted?
So, to sum up, it seems that there are several things to like about the way PISA for Development is shaping up 18 months in. However, there are also several areas that raise concerns and warrant greater discussion. Some of my wishes show signs of being fulfilled and others are still pending. I’d be curious to know how others feel about this initiative. What do you hope it will achieve? What are your concerns? What else should we be doing in this area? Please share in the comment section below.
Follow the World Bank Group Education team on Twitter @wbg_education
Join the Conversation