The PISA for Development initiative moves forward: Have my wishes been fulfilled?

|

This page in:



About a year and a half ago, I wrote a blog about the Organisation for Economic Co-operation and Development’s (OECD) PISA for Development initiative.
 
Most of us are already familiar with the OECD’s PISA exercise, which is a test that assesses the reading, mathematics, and science competencies of 15-year olds around the world. The aim of PISA for Development is to identify how PISA can support evidence-based policy making in developing countries that, until now, have been unable or unwilling to participate. The expected outcome is to produce a set of enhanced student assessment instruments that are tailored to the needs of these countries, but which also produce reading, mathematics, and science scores on the same scale as the main PISA survey. In that earlier blog, I made three wishes for the initiative. 

Have any of my wishes been fulfilled?

It’s too early to tell because project instrument development has just begun, data collection is due to take place between 2015 and 2016, and reporting will only occur in 2017. Nonetheless, there has been sufficient progress to allow me to make a few observations about where things seem to be headed and to flag some issues that may need more focused attention and discussion by the participating countries and other stakeholders, including development partners, donors, and civil society.
 
Wish #1 - A test with questions that 15-year-olds in emerging and developing economies can actually answer.
 
The OECD’s original plan was to draw solely on their existing pool of 337 PISA questions to create the PISA for Development test. The reasoning was that this would, among other things, help reduce costs. In addition, if the current question pool proved too difficult, new questions could be developed in future rounds. In my earlier blog, I foresaw problems, given that even the middle-income countries participating in the regular PISA surveys can end up with so many zero scores on the test that they are unable to do much with the data.
 
Eighteen months on, it seems that the OECD has moved away from its initial stance. It’s seriously exploring the possibility of using suitable questions from other regional and international assessments to supplement their existing PISA questions and better flesh out the lower end of the PISA scale. The challenges to obtaining and incorporating these questions into the PISA for Development test include the need to adhere to the other assessment programs’ guidelines for keeping their test questions secure so that they can be reused in future rounds, which is feasible. As of today, it’s looking a bit more likely that the PISA for Development test will be something that 15-year-olds in emerging and developing economies can actually answer. Wish #1 is on the way to being granted.
 
Wish #2 - A test that emerging and developing economies can afford.

Based on current budget figures, the PISA for Development pilot is about twice as expensive as the regular PISA exercise. It involves a considerable amount of developmental work (such work is mainly one-off and costs would be expected to go down in future rounds). For example, the background questionnaires need to be adapted to better fit the contexts found in developing countries and a methodology needs to be developed to include out-of-school students in the assessment. None of these issues have prevented countries from signing up for the pilot – Cambodia, Ecuador, Guatemala, Paraguay, Senegal, and Zambia. Nor has it prevented several donors and development partners (including the World Bank) from providing significant financial support to make this participation possible. Such enthusiasm and support is firmly based on the anticipated project benefits for the participating countries, including the likely payoff from the project’s strong focus on technical capacity building.

Looking beyond the developmental phase, however, it will be important to take a hard look at the long-term sustainability of PISA for Development (whether as stand-alone or folded into the regular PISA survey), particularly if countries are ultimately expected to handle the costs of participation on their own. Until such issues have been addressed, it remains unclear whether this is a test that emerging and developing economies can afford in the long term. Wish #2 still not granted.

Wish #3 - A test that contributes to learning for all.

The PISA for Development initiative is meant to contribute to the global learning agenda that will emerge from the next round of global education goals.  A global learning goal requires a way to monitor the learning of all children and youth, whether in or out of school. This is a need that’s not being addressed by the current fractured landscape of assessment initiatives.

The work being done under PISA for Development- which includes partnerships with other assessment programs- is a step towards creating a more cohesive framework for global monitoring work.
At the same time, the pilot faces challenges in becoming a truly useful tool for the global learning agenda in that it must demonstrate the ability to collect meaningful learning data on the entire 15-year-old cohort in a country, including those who are out of school.

The technical and logistical challenges are daunting. How do you collect learning data from 15-year-olds who are illiterate or semi-literate? What about those with disabilities? Is it okay to use oral or pictorial assessments and, if so, what are the implications for mapping results to the PISA scale?

The PISA for Development pilot aims low in this area by promising to test and validate different sampling approaches and techniques for identifying informants, and also to try out different test administration practices. The aim is that the lessons learned will allow future PISA cycles to incorporate nationally representative samples of the entire cohort of 15-year-olds, including those who are out-of-school.

Kudos to the PISA for Development team for taking on this challenge, but there is a long way to go before it’s clear whether it’s a game changer when it comes to creating a test that truly contributes to learning for all, in the most fundamental sense. Wish #3 partially granted?

So, to sum up, it seems that there are several things to like about the way PISA for Development is shaping up 18 months in. However, there are also several areas that raise concerns and warrant greater discussion. Some of my wishes show signs of being fulfilled and others are still pending. I’d be curious to know how others feel about this initiative. What do you hope it will achieve? What are your concerns? What else should we be doing in this area? Please share in the comment section below.
 
Follow the World Bank Group Education team on Twitter @wbg_education

Topics
Sigamoney
February 13, 2015

Testing is undoubtedly a very important aspect of any teaching and learning situation. The diversity in the student population presents challenges to test designers and implementers. I note reference has been made to children with disabilities. It is important that disabilities is taken seriously and other forms of barriers to learning. At an international level much focus has been directed to inclusive education. We know that inclusive education and the ideology that underpins it, is very relevant and appropriate to developing contexts. Developing contexts can benefit from an inclusive philosophy. Assessment and assessment instruments should take very seriously the issue of barriers to learning.How test instruments can be modified, adapted or developed to facilitate the inclusion of all children is the step in the right direction.

Marguerite
February 19, 2015

Thanks for your comments. I agree that the issue of disabilities needs to be taken very seriously in any assessment program that purports to contribute to monitoring and achieving global learning. The fact is that many (most?) assessment programs do not offer decent options for measuring the learning of children with disabilities, even assessments carried in so-called developed education systems. The reasons range from cost to technical challenges, but none warrant excluding these students. Assessment should not prevent students with disabilities or other learning barriers from participating in the learning process or from demonstrating the learning they have achieved as a result of that process. Inclusive education -- and inclusive assessment -- has to be part of the post-2015 global agenda in education. Otherwise, we cannot truly say that it is a global learning agenda. I really hope that PISA for Development can identify some ways that assessments can be adapted to better capture the learning of the ‘non-typical’ student in a variety of contexts, both developed and developing.

Fekadu Bogale
February 14, 2015

Dear Marguerite how are you? I am pleased with what you have stated above and I wish your wishers become reality.recently,I was discussing with my friends as to how we can involve Ethiopia in SACMEQ and PISA for development to test our relative post ion before joining on international assessment by 2017.

Marguerite
February 19, 2015

Thanks for your comments. For countries that have never participated in an international assessment, it can be a good idea to first try participating in a regional assessment exercise. In your case, SACMEQ could indeed be a good option. PISA for Development is already in progress, but you may want to investigate some of the other options out there that allow countries to ‘ease into’ participating in an international assessment. For example, the IEA offers PIRLS-Literacy and TIMSS-Numeracy, in addition to its regular PIRLS and TIMSS exercises (http://timssandpirls.bc.edu/). The important thing is for a country to be clear on what they want to get out of one of these international assessment exercises. Do they want to have a sense of their international standing? Or are they more interested in their regional position? Are they interested in building technical capacity in doing a particular type of assessment? Do they want to ensure that the data generated by the exercise are rich enough to allow for in-depth policy analyses? Depending on the answers, some assessments may be more suitable than others. Good luck and keep me posted on what you decide to do!

Alejandro Ganimian
February 15, 2015

Great post, Marguerite!
I agree with your three wishes and hope they are granted.
My own wish is that it would be useful if the P4D results were comparable to the PISA results. This would imply slightly altering your wish #1 to include link items in P4D that are also tested in PISA. The way I see it, this would avoid creating perverse incentives for countries to take P4D so as not to be compared with other countries. I believe that comparisons between countries are partly why PISA is so useful. My understanding is that this is what the OECD already plans to do, so my wish is also on the way of being granted.
PS: Posting to this blog was a bit of a hassle. When I tried to do so, I saw multiple copies of your blog entry with a "verification" field somewhere in between these copies.

Marguerite
February 19, 2015

Thanks for your comments (and for your feedback on the experience of posting a response to this blog, which I will pass along). The ultimate aim is that the PISA for Development results would be placed on the same scale as the regular PISA so hopefully your wish will be granted! In addition, the PISA for Development ‘option’ is meant to be folded into the regular PISA program so that it all becomes one assessment exercise that includes different options for countries in terms of the types of test booklets, questionnaires, and sampling approaches they use to gauge the achievement levels of their 15-year-olds. All countries' results will be placed on the same scale. You’re right that comparisons among countries is one reason why PISA is so useful and that’s something that the PISA for Development initiative wants to preserve (in part by drawing on existing items from the secure PISA item pool that have been used in earlier PISA exercises – this helps create the link that you refer to in your comments). Whether it can achieve this is another issue.