Who Trains the Best Computer Scientists?

Computing is perhaps one of the hottest professions today. And as technology becomes a ubiquitous part of life, the demand for computer professionals will only increase. In the United States alone, over 500,000 ICT jobs will be created within the next decade, and by 2024 almost three-quarters of STEM job growth will be in computer-related occupations. Higher education systems have responded swiftly, with a rapid increase in the quantity of computer science (CS) undergraduate programs. While the quantitative expansion in programs has been quite visible, what about their quality? What skills do computer science graduates bring with them? Are some countries doing a better job equipping their computer graduates with core skills?

To answer these questions, a cross-national team, including the World Bank, embarked on an ambitious exercise: to identify the key computing skills of the entering and graduating class of computer science engineers in four major economies: China, India, Russia and the United States. Together, these four countries train nearly half the CS professionals in the world.

Led by Stanford University’s Professor Prashant Loyalka, we first identified all undergraduate CS majors from China, India, Russia that had similar course requirements and content with undergraduate CS majors in the United States. We then took nationally representative samples of institutions including elite and non-elite programs in each country. We next randomly sampled smaller administrative units (departments and classes) within each of the sampled programs in China, India, and Russia and selected all seniors in those administrative units. Sampled seniors in the three countries took a two-hour, computer-based, standardized CS exam from the Major Field Test^® suite of assessments designed by Educational Testing Service (ETS), which assesses how well CS seniors master CS-related concepts, principles, and knowledge. For the United States, we used ETS data from the same test.

Based on this work, a paper released in the Proceedings of the National Academy of Sciences earlier this week, provides interesting insights.

• The United States trains the best computer graduates ― by far: We find that CS seniors in the United States substantially outperform seniors in China, India, and Russia. The average computer science student in the United States ranks higher than about 80 percent of students tested in China, India, and Russia. Seniors in elite institutions in the United States similarly outperform seniors in elite institutions in China, India, and Russia by approximately 0.85 Standard Deviations (SDs). Importantly, the skills advantage of the United States is not because it has a large proportion of high-scoring international students.

Figure 1: Computer Science Skills across China, India, Russia, and the United States

Notes: Mean estimates for China, India, and Russia are each statistically lower than the mean estimate for the United States (P = 0.000). Mean estimates are not statistically different between China and India (P = 0.435), China and Russia (0.914), and India and Russia (P = 0.509). Estimates are reported as effect sizes (in standard deviation units). Scaled CS exam scores were converted into z-scores using the mean and SD of the entire cross-national sample of exam takers. As such, the overall mean of the standardized score across all four countries is zero. Standard errors are adjusted for clustering at the institution (university/college) level.

• Women students lag behind their male counterparts in all countries: We find consistent but moderate differences in CS skills between female and male students within all four countries. Males score 0.15 SDs higher than females in China, 0.24 SDs higher in India, 0.25 SDs higher in Russia, and 0.41 SDs higher in the United States.
   
• Colleges’ “value-added” may differ in China, India and Russia: While the skill levels of entering computer science freshmen are much higher in China than in India and Russia by the end of college, Chinese, Indian, and Russian students have comparable skill levels. This suggests that CS program quality and rigor may be lowest in China and highest in India.

 The study has important policy implications.

First, it underscores the need to understand what college students are learning, something not captured in current international comparative assessment studies or even college rankings.

Second, it emphasizes the need to understand what is happening in settings that boost student learning, such as the United States, or in settings where students may be learning at a faster pace, such as India.

Third, the gender gap in skills suggests that more effort may be needed to attract and retain higher achieving female students into computer science and ensure that they have equal opportunities to receive a quality education.