Technology Use and Educational Performance in PISA
This page in:
Every three years, students around the world participate in an international assessment of their competencies in reading, mathematical and scientific literacy as part of the Programme for International Student Assessment, more commonly known as PISA. In 2006, schools from 58 countries were randomly selected to take part in the effort, overseen by the OECD, to test how well students can apply the knowledge and skills they have learned at school to real-life challenges. (When you read a press report about a given country being highly ranked -- or doing poorly -- in comparison to other countries on how its students do in reading, math, or science, quite often this a reference to the so-called 'league tables' that are published by the OECD in this regard.)
PISA provides a goldmine of data for researchers interested in many topics, and the OECD has just its analysis of Technology Use and Educational Performance in PISA, which notes that "OECD countries [here's the list of them] have undertaken significant investments to enhance the role of technology in education. What are the results of these investments? Are they fulfilling expectations? PISA 2006 provides a wealth of comparative data to begin answering these questions ..."
While the full report, Are New Millennium Learners Making the Grade?, is not available for free download on-line at this time, you can read an 11-page summary from here (in PDF) or a related earlier paper here (pdf, 2.3MB). This work, part of the OECD's initiative looking at 'New Millennium Learners (NML), is a follow-up to a 20006 study, Are Students Ready for a Technology-Rich World? What PISA Studies Tell Us.
(Side note: The recent event on '1-to-1 computing in education', featured on this blog here and here, was also part of the NML project.)
There is much food for thought in the 205 pages of charts, graphs, data tables and analysis contained in this eagerly anticipated report (even if, as the authors state, "data availability remains one of the main handicaps for understanding the role of ICT in education."). If there is a 'headline message' here, it is probably that:
One of the most striking findings of this study is that the digital divide in education goes beyond the issue of access to technology. A second digital divide separates those with the competencies and skills to benefit from computer use from those without.
More concretely,
even accounting for a student's socio-economic status, there is a significant correlation between computer use at home and educational performance, a correlation that does not appear for computer use at school. Some analysts have rightly pointed out that in a school setting what matters is the use of computer in the wider context of a particular educational strategy. According to this view, gains in educational performance would only appear in the presence of a successful educational strategy. Therefore, the amount of use, i.e. the time a computer is used, would not matter at all. This certainly makes sense from a strictly educational perspective, but fails to explain why substantial gains in educational performance are correlated with the frequency of computer use at home. This is even more striking in view of the mostly leisure or entertainment-oriented nature of computer activities performed by students at home.
Are New Millennium Learners Making the Grade? concludes by examining a set of policy implications, which appear (to me at least) relevant for all countries, not just those in the OECD:
- Raise awareness among educators, parents and policy makers of the consequences of increasing ICT familiarity
- Identify and foster the development of 21st century skills and competences
- Address the second digital divide
- Adopt holistic approaches to ICT in education
- Adapt school learning environments as computer ratios improve and digital learning resources increase
- Promote greater computer use at school and experimental research on its effects
This report provides a valuable service in providing hard data against which we can test the many hypotheses and claims put forward in this area -- even if, as with so much research into the impact of technology use in education, the deeper one delves, more questions are often raised than answered. I expect that the release of this valuable report will generate a lot of comment and discussion in the coming years, as its findings are circulated widely.
Visitors to this blog may also be interested in an OECD publication that came out late last year, the culmination of a multi-year OECD project on digital learning resources:
A few years ago, I put together a quick guide for infoDev cataloguing what the OECD was doing related to ICT and education. While a little out of date now, it still might be of interest:
Please note: The image used at the top of this blog entry of the Italian city of Pisa (seen from its famous Leaning Tower) comes from the Wikipedian Blorg via Wikimedia Commons and is used according to the terms of its Creative Commons Attribution ShareAlike 3.0 License.
Mike,
Am I misreading, or did the PISA report only measure children's Science and ICT? If it was only Science scores they used, then to extrapolate from that to make the huge conclusions they do re things like the second digital divide, seems to me to be at best, ambitious and at worst not exactly good research practice.
Even allowing for translation difficulties, I found the whole Second digital divide line confusing and even obvious because in essence it argues that the lack of stable, parental, cultural support in the home negates any possible educational gain made using computers. Surely, children from such home conditions, when compared with children with a firmer educational foundation, would produce the same result for anything you chose to measure: modern languages, music, history.
Joe,
Thanks for your comments.
Indeed, I expect that this work from the OECD will be hotly discussed and debated. I have loaned my hard copy of the publication out to colleagues here (and I can't yet find a downloadable version on the OECD site), so I am loathe to comment on methodological issues or to characterize the OECD findings at geater length unless I have the actual text in front of me (for fear of conflating my opinions/comments on the report findings with what the OECD has actually written).
That said, and taking a step back for a moment:
As for your comments about the items (excerpted in my blog post above) about the 'second digital divide' being "confusing" or "even obvious":
1. re: "confusing": It does appear, based on the exerpt above, that the OECD researchers believe that this finding raises many additional questions.
2. re: "obvious": One goal of research is to discover what we do not yet know; another is to attempt to confirm what we may already believe is obvious, but which we do not have hard data to support. Especially in the area of the use of ICT use in education, there are many things that people claim to "know", even though they have no reliable data in support of such "knowledge"; many policy decisions are then taken that "naturally follow" from such "knowledge". (In addition: sometimes people "know" things that are directly contradicted by historical data, but argue that "things are different now because of ___".)
When I make comments like this (#2), I am routinely criticized by people in certain quarters who say that, by pointing out that some of the core beliefs driving investments in ICTs in education are not (or at least not yet) supported by the data at hand, I am (variously) guilty of small-minded thinking, a lack of vision, or a bureaucratic mindset. Setting aside any cognitive deficiencies I may have, I do think there is value in questioning some core assumptions and beliefs that are at the heart of many policy decisions related to the use of ICTs to aid a variety of educational objectives. The New Millennium Learner project that the OECD has been leading for the past few years has performed a valuable service in this regard, even if some of its conclusions in this report are 'obvious', some raise more additional questions than the answers they provide, and still others are open to debate. Indeed, I think one of the goals of the NML project is to provoke such debate, informed by both opinions and beliefs about the role(s) of ICTs in education, and by the available data at hand.
Personally, I find it shocking in many cases how little reliable data we have to guide decisionmaking in this area. This is not to say that there are not imperatives of many sorts compelling such decisions, nor that 'we need to wait until all the data is in' before we can responsibly make policy in this area -- we are talking about technology after all, where new advances regularly upend established practices and our thinking about what is possible.
As people delve more deeply into the data and analysis from the 'Are New Millennium Learners Making the Grade?' report and start to publish their own findings and comments (and once I get my copy back!), I expect we'll be looking at this OECD report in greater detail on this blog in the months ahead.
-M
ps Responding to the question that led your post: Yes, the major domain for PISA 2006 was indeed science (in 2003 it was mathematics, in 2000 it was reading).
Mike,
Many thanks for clarifying and developing the underlying concern I had so perceptively. I'm just about to publish a piece of research here in the UK which looks at what has been driving this unprecedented investment, and indeed arguably still is, and since I'm also currently directing a project in the Caribbean which invests tremendous faith in what ICT can achieve educationally, it's a major issue for me... no less than for other taxpayers.