Assessing education with computers in Georgia

One of the fascinating benefits of working at a place like the World Bank is the exposure it offers to interesting people doing interesting things in interesting places that many other folks know little about.  Small countries like Uruguay and Portugal, for example, are beginning to attract the attention of educational reform communities from around the world due to their ambitious plans for the use of educational technologies.  Much is happening in other parts of the world as well, of course, especially in many countries of Eastern Europe and Central Asia.  The largest stand-alone World Bank education project to date that focused on educational technologies, for example, was the Russia E-Learning Support Project.  Macedonia gained renown in many corners as the first 'wireless country', with all of that Balkan country's primary and secondary schools online since the middle of the last decade -- although other countries, like Estonia and the tiny Pacific island nation of Niue, also lay claim to versions of this title. (If you are looking for more information on the Macedonian experience, you can find it here and here [pdf]). Much less well known, however, is the related experience of the small country of Georgia, located at the crossroads of Eastern Europe and Western Asia, where small laptops are being distributed to primary school students and where school leaving exams are now conducted via online computer-adaptive testing.

Beginning with a pilot program that distributed 2500 netbooks to first graders and their teachers in fall 2010, the government has now distributed over 60,000 'bukis', a localized version of the Intel Classmate laptop which is produced by a Georgian computer firm.  A national education portal with Georgian language content is also available, and digital textbooks and other educational content is also pre-loaded on the devices themselves. These materials (and other, like 'Art' and fifty other education games developed by the Ministry of Education) are meant to supplement traditional paper-based texts.  (When people think of cooperation between Europe and South America around educational laptops, they often cite links between Portugal and Venezuela, smaller connections between Georgia and Suriname are less well known).

While the buki is aimed at students at the lower primary school level, the so-called buki generation, an even more ambitious program has been underway targeting students finishing their secondary education.

Last year Georgia had nationwide school leaving exams for the first time.  While this is notable in and of itself in the Georgian context (corruption in university placement is a challenge for many countries in this part of the world, and school leaving exams of this sort are one attempt to help address this issue), what is more interesting for international audiences may be the fact that this was done entirely using computer adaptive testing.

The costs for a national paper-based examination system were seen to be prohibitive in Georgia, and so the decision was made to take advantage of the investments in school ICT infrastructure to do everything online.

Over 1500 schools and 44,000 students participated in this ambitious exercise, sitting for proctored, school-based examinations over eight days (one subject per day).  Results were available immediately at the end of each exam, both for individuals and consolidated across the system (centrally, the team responsible for this process say they were able to monitor progress in real time).  Sitting in school classrooms, student logged in en masse using their national identification number, and, for security purposes,  the exam proctor in the room entered a related code to enable access to the online test itself.

An official from the Ministry of Education in Slovenia explained to me that this sort of centralized, 'Big Bang' approach (which he referred to as the 'Lithuanian model') can be attractive and cost-effective to countries like his and Georgia, given their small size.  (The Uruguayans are also exploring how the ICT infrastructure in schools there can be utilized across the country to help teachers with formative assessments in informal, low stakes ways.)  As with the buki, the technology at the heart of this had a decidedly Georgian flavor, and it was told to me that "everything used here was developed in Georgia, the software, the assessment engine, individual test items and the test item data bank: everything".)

While I am always skeptical of 100% reported success rates, officials say that the process went off without a hitch.  Students -- almost all of whom had never taken an official computer-adaptive test before and who have been taking paper-based assessments their entire academic lives -- were able to practice on the system in prior weeks before logging in to take the actual test, which also helped officials test load across the system.  (I would like to learn about how they were able to ensure adequate bandwidth across the system and that the testing servers were robust enough not to buckle under what must have been considerable strain.) Those who would like to learn more about this Georgian experience may be interested in an upcoming international event organized by the country's national examinations center. The Third Black Sea Conference on Assessment in Education: New Technologies in Educational Assessment will take place in the Georgian resort town of Batumi this spring, just a few weeks after the national exams take place in May. Come this June, officials in countries considering similar things might do well to have Georgia on their minds.

 
note: The image used at the top of this blog post ('the buki generation') comes from the buki portal and is used with the permission of the Ministry of Education in Georgia, which holds its rights.