Syndicate content

November 2016

Open data, closed algorithms, and the Black Box of Education

Michael Trucano's picture
hey, what's going on in there?
hey, what's going on in there?
Education is a ‘black box’ -- or so a prevailing view among many education policymakers and researchers goes.

For all of the recent explosion in data related to learning -- as a result of standardized tests, etc. -- remarkably little is known at scale about what exactly happens in classrooms around the world, and outside of them, when it comes to learning, and what the impact of this has.

This isn't to say that we know nothing, of course:

The World Bank (to cite an example from within my own institution) has been using standardized classroom observation techniques to help document what is happening in many classrooms around the world (see, for example, reports based on modified Stallings Method classroom observations across Latin America which seek to identify how much time is actually spent on instruction during school hours; in many cases, the resulting data generated are rather appalling).

Common sense holds various tenets dear when it comes to education, and to learning; many educators profess to know intuitively what works, based on their individual (and hard won) experience, even in the absence of rigorously gathered, statistically significant 'hard' data; the impact of various socioeconomic factors is increasingly acknowledged (even if many policymakers remain impervious to them); and cognitive neuroscience is providing many interesting insights.

But in many important ways, education policymaking and processes of teaching and learning are constrained by the fact that we don't have sufficient, useful, actionable data about what is actually happening with learners at a large scale across an education system -- and what impact this might have. Without data, as Andreas Schleicher likes to say, you are just another person with an opinion. (Of course, with data you might be a person with an ill-considered or poorly argued opinion, but that’s another issue.)
 
side observation: Echoing many teachers (but, in contrast to teaching professionals, usually with little or no formal teaching experience themselves), I find that many parents and politicians also profess to know intuitively ‘what works’ when it comes to teaching. When it comes to education, most everyone is an ‘expert’, because, well, after all, everyone was at one time a student. While not seeking to denigrate the ‘wisdom of the crowd’, or downplay the value of common sense, I do find it interesting that many leaders profess to have ready prescriptions at hand for what ‘ails education’ in ways that differ markedly from the ways in which they approach making decisions when it comes to healthcare policy, for example, or finance – even though they themselves have also been patients and make spending decisions in their daily lives.

One of the great attractions of educational technologies for many people is their potential to help open up and peer inside this so-called black box. For example:
  • When teachers talk in front of a class, there are only imperfect records of what transpired (teacher and student notes, memories of participants, what's left on the blackboard -- until that's erased). When lectures are recorded, on the other hand, there is a data trail that can be examined and potentially mined for related insights.
  • When students are asked to read in their paper textbook, there is no record of whether the book was actually opened, let along whether or not to the correct page, how long a page was viewed, etc. Not so when using e-readers or reading on the web.
  • Facts, figures and questions scribbled on the blackboard disappear once the class bell rings; when this information is entered into, say,  Blackboard TM (or any other digital learning management system, for that matter), they can potentially live on forever. 
And because these data are, at their essence, just a collection of ones and zeroes, it is easy to share them quickly and widely using the various connected technology devices we increasingly have at our disposal.
 
A few years ago I worked on a large project where a government was planning to introduce lots of new technologies into classrooms across its education system. Policymakers were not primarily seeking to do this in order to ‘transform teaching and learning’ (although of course the project was marketed this way), but rather so that they could better understand what was actually happening in classrooms. If students were scoring poorly on their national end-of-year assessments, policymakers were wondering: Is this because the quality of instruction was insufficient? Because the learning materials used were inadequate? Or might it be because the teachers never got to that part of the syllabus, and so students were being assessed on things they hadn’t been taught? If technology use was mandated, at least they might get some sense about what material was being covered in schools – and what wasn’t. Or so the thinking went ....

Yes, such digital trails are admittedly incomplete, and can obscure as much as they illuminate, especially if the limitations of such data are poorly understood and data are investigated and analyzed incompletely, poorly, or with bias (or malicious intent). They also carry with them all sorts of very important and thorny considerations related to privacy, security, intellectual property and many other issues.

That said, used well, the addition of additional data points holds out the tantalizing promise of potentially new and/or deeper insights than has been currently possible within 'analogue' classrooms.

But there is another 'black box of education' worth considering.

In many countries, there have been serious and expansive efforts underway to compel governments make available more ‘open data’ about what is happening in their societies, and to utilize more ‘open educational resources’ for learning – including in schools. Many international donor and aid agencies support related efforts in key ways. The World Bank is a big promoter of many of these so-called ‘open data’ initiatives, for example. UNESCO has long been a big proponent of ‘open education resources’ (OERs). To some degree, pretty much all international donor agencies are involved in such activities in some way.

There is no doubt that increased ‘openness’ of various sorts can help make many processes and decisions in the education sector more transparent, as well as have other benefits (by allowing the re-use and ‘re-mixing’ of OERs, teachers and students can themselves help create new teaching and learning materials; civil society groups and private firms can utilize open data to help build new products and services; etc.).

That said:
  • What happens when governments promote the use of open education data and open education resources but, at the same time, refuse to make openly available the algorithms (formulas) that are utilized to draw insights from, and make key decisions based on, these open data and resources?
     
  • Are we in danger of opening up one black box, only to place another, more inscrutable back box inside of it?

How to climb the Open Data maturity ladder: lessons from Europe

David Mariano's picture

Editor’s note: This is a guest blog from Jorn Berends and Wendy Carrara on behalf of the European Data Portal. The indicator “Open Data readiness” mentioned in the analysis below is unrelated to the Open Data Readiness Assessment tool developed by the World Bank.

The majority of European countries are improving their Open Data maturity

Open Data is taking off in Europe. Open Data – which refers to publicly available data that is free for all to use – is set to have a monumental impact on societies in the upcoming years, and has a potential market size for the European Union of 75.7 bn EUR by 2020. European countries seem to be on track to reap the potential of Open Data, although a lot of work remains to be done and substantial differences persist between countries. In 2016, with a 28.6% increase compared to 2015, the EU28 and Norway, Switzerland and Liechtenstein – referred to as EU28+ –  completed over 55% of their Open Data journey. This shows that, by 2016, a majority of the EU28+ countries have successfully developed a basic approach to address Open Data. This is just one of the conclusions of the 2016 Open Data Maturity in Europe report, prepared by the European Data Portal. What can this report tell us on the countries that are doing particularly well in this regard, what is it that makes them European champions? Furthermore, what can other countries learn from them?

Spain and France are leading the way in Europe, but substantial differences persist

The measurement is built on two key indicators ‘Open Data Readiness’ and ‘Portal Maturity’, thereby covering the level of development of national activities promoting Open Data as well as the level of development of the country’s national data portal. Based on these indicators, the European countries are clustered into four different levels of Open Data maturity: Beginners, Followers, Fast Trackers and Trend Setters.

 

Benchmarking rural water systems by a simple score

Kristoffer Welsien's picture
 
Improved water supply in Kipanduka village, Tanzania
Photo: Alessandra Argenti/ World Bank

Can the ability to sustain rural water systems be captured by a simple score? A new multipurpose four-page tool seeks to measure the likelihood of sustainability by assessing the capacity of a village water committee. Previously, such tools were often either too lengthy and academic or the assessment was left to the discretion of local officials with the risk of omitting critical components. Now a more practical model has been developed that aims to be user-friendly but detailed enough to detect gaps and prioritize interventions for village water committees.
 

This journey started in a rural village in the district of Karatu, Tanzania where a new water system had just been commissioned. The district water engineer was about to inaugurate the water scheme after a five-day training of the village water committee, giving them full ownership of their water scheme. During the opening ceremony, one question kept puzzling her: “Was the village water committee fully equipped to manage, operate and maintain their newly installed water scheme?”

Employees and Government Ministry Win in Reform Project in Afghanistan

Shahenshah Sherzai's picture
Rumi Consultancy
Students studying at Dunya University, supported by the Public Financial Management Reform Program (PFMR). Rumi Consultancy/World Bank

Armed with only a high school certificate, Daoud Shah Noor, 42, started working at the Ministry of Finance in 2012. The sole supporter of his family, he was unable to attend university because of prohibitively high tuition prices. Just four years on, Daoud is studying for his Master’s degree at the Dunya University, where he had graduated with a Bachelor’s degree in Business Administration.
 
“Before university I was not professional in my work. Now I am doing the job more professionally and in a better way,” says Daoud, who comes from Parwan Province. Daoud is a beneficiary of the Public Financial Management Reform (PFMR), a project that aims to strengthen public financial management through effective procurement, treasury and audit structures, and high standards of financial monitoring, reporting, and control.

Committed to sustainability: Five World Bank corporate actions that support the SDGs

Shaolin Yang's picture
World Bank Corporate Sustainability


At the World Bank, we look to lead by example to achieve our goals of ending extreme poverty and boosting shared prosperity in a sustainable way.  This is a longstanding promise recently reconfirmed and reenergized by President Kim in October 2014.

And with the adoption of the Sustainable Development Goals (SDG) – 17 goals to be achieved by 2030 – we are linking our own corporate sustainability activities to many of the SDG indicators.

Are tablets the best way to increase digital literacy in African countries?

Edward Amartey-Tagoe's picture
Credit: Arne Hoel


A good number of African governments have shown how technologically-forward thinking they are by announcing one-tablet-per-child initiatives in their countries. President John recently announced that tablets for Ghana’s schoolchildren were at the center of his campaign to improve academic standards. Last year, President Kenyatta of Kenya abandoned a laptop project for tablets.

E Pluribus Unum (out of many one)

Patrick Field's picture

Consider editing a major planning document with 5 federal agencies, 3 agencies each in 6 states, 15 non-profit organizations, three to four layers each. That equals ninety commenters and thousands of comments over multiple drafts. That’s any author’s nightmare! Comments come in late. Multiple commenters from a single agency contradict one another. A new high-level commenter suddenly demands a host of changes without any context, history, or understanding of why you are where you are. This is the reality for many planners, coordinators, and technical writers in multi-stakeholder processes. How in the world do you manage wide-ranging opinions on topics from common usage to fundamental substance, from multiple commenters, and get a product out and done?

Given our experience engaging with talented (and overworked and sometimes frustrated) convenors and coordinators working on issues from oceans planning to government transparency, we at CBI wanted to offer some good practices for such a challenging task. How do you ensure transparency and create legitimacy? How do you provide reasonable procedures for coordination and bring the process to a decisive end? How can you be thorough and collaborative without collapsing under complexity and confusion? In short, what to do?

Establish norms and expectations A coordinator*’s job, first and foremost, is to help establish norms and expectations for the process. The basic expectations and norms should include roles and responsibilities, the process or procedures for how comments are collected and considered, schedules and milestones, and how decisions will be made. It’s best if the group builds norms and expectations together rather than if they are imposed from above or the side. Then, when the coordinator has to “bring the hammer down,” she can remind the offending party of the process the group jointly established. Lastly, whatever the norms, expectations, and process established, the participants will likely need reminders all along the way, and sometimes not just via email, but in direct conversation, be that face-to-face or over the phone. Yes, picking up the phone can help!

Ensure transparency There’s nothing worse than your comments being ignored or the final product seemingly almost unrelated to earlier drafts. The coordinator’s job is to ensure transparency in multiple ways. First, the process should be transparent: this is how and when decisions will be made, and by whom. Second, comments need to be transparent to all who are participating. This involves providing mechanisms so that every commenter can see the comments of others and not wonder who said what. Third, the disposition of those comments needs to be transparent.

In a fast-moving, complex environment, it may be too much to ask for a detailed responsive summary often provided by federal agencies in federal rulemaking. But, a coordinator can deploy any number of techniques. A coordinator can provide a concise summary of key comments or comment themes and how she addressed them. A coordinator can provide a section-by-section redline document (which may or may not include all the comments, depending on how many and how messy it makes the document). A coordinator can take key issues and build a comment matrix of original text, comments, changes, and reasons why comments were not accepted.

Lastly, a coordinator may need to reconcile the comments through group process and meetings so she can rely on the participants themselves to reconcile differences and not hope she gets it “right enough.” Technologies like Google Docs, WebEx, Zoom, and others allow you to share your screen and even share the document for joint, simultaneous editing.

Centralize tracking The mechanics of keeping track of multiple commenters, comments, and versions of the document can be daunting. But here, logistical and technical expertise can be very helpful. The coordinator needs to establish a tool or tools to track the changing nature of the document. First, providing a procedure for version control and a nomenclature is essential. Second, utilizing Google Docs, DropBox, or other on-line tools can keep versions, comments, and responses all in one place and accessible to those who need them. Third, ensuring there is one or only a few coordinators, who can be the contact person as well as keep the whole in her head as she is buffeted by contradictory comments, needs, and expectations from all directions, is very important.

Provide comment guidance To help guide the process, a coordinator should provide clarity on what to comment on. It’s one thing to say, here’s the document and comment away. It’s another thing to say: 1) please focus on the introduction, key findings, and draft recommendations; 2) don’t focus on style or visuals at this time; and, 3) leave copy editing for later drafts. Of course, some participants will not be able to help themselves. They’ll fix commas, semi-colons, and provide lots of visual ideas. But most will appreciate the direction. They are busy people too with too many things to do.

Finance numérique : quoi de neuf en Afrique de l’Ouest ?

Estelle Lahaye's picture
Cette page en : English
Photo: Philippe Lissac, 2011 CGAP Photo Contest


Même si la route reste encore longue, l'Union économique et monétaire ouest-africaine (UEMOA) a accompli des progrès intéressants en matière d’inclusion financière en développant ses services financiers numériques (SFN).


Pages