Which countries punch above their weight in education rankings?

This article is published in collaboration with The Conversation. Read the original article.

Rankings of countries based on how well their students perform in the Programme for International Student Assessment (PISA) always receive a great deal of attention from the media and politicians. But PISA rankings, produced by the Organisation for Economic Cooperation and Development, are limited when it comes to evaluating the quality of education systems and their efforts to improve children’s lives.

The rankings tell us, for example, that 15-year-olds in Germany and the US perform better in mathematics than students in Peru or Indonesia. Yet we also know that income per capita in Peru and Indonesia is way less than that in Germany and the US.

So is it fair to compare education systems operating in such different socio-economic conditions? How would these education systems perform if they served students and operated in countries with more or less similar socio-economic characteristics?

In studies of how well individual children do and the “effectiveness” of individual schools, pupils’ socio-economic characteristics are now ritually taken into account. The same reasoning can be extended when comparing countries’ education systems.

Our research has measured the effectiveness of education systems by adjusting the performance of students who take the PISA tests for the socio-economic context. The effectiveness measure is obtained by calculating the difference between how well students in the education systems ranked by the OECD actually perform, and how they should be expected to perform due to the socio-economic characteristics of students, schools, and countries.

In the graph below, those countries with values higher than zero – towards the right-hand side – have 15-year-olds who perform above expectations in mathematics, meaning the education system is effective. Those below zero, to the left, indicate ineffective performance.

PISA 2012, effectiveness of education systems in mathematics performance. Note: Albania and Lichtenstein were removed for missing data in the economic, social and cultural status index and sample size restrictions.
Author provided

The results show a different configuration of countries’ performance once the socio-economic context is considered. For example, Turkey, Thailand, and Indonesia are effective systems, once results are adjusted in this way, although their absolute performance in the PISA tests is below average. Conversely, the US, Sweden, and Norway are among the least effective systems if socio-economic context is taken into account, but exhibit higher absolute performance scores.

There are also systems that perform highly according to both absolute scores and for effectiveness once their performance is adjusted for socio-economic context – such as Hong Kong, Korea, and Chinese Taipei. Others have both low absolute performance scores and effectiveness measures, such as Argentina, Jordan, and Qatar. And there are also systems which perform within their expected range, such as Mexico, Spain, Finland, and New Zealand.

Taking wealth into account

The second graph below shows the relationship between absolute mathematics scores in PISA 2012 and effectiveness measures, adjusted for socio-economic context. Those education systems that perform well in absolute terms do also tend to be more effective, but the relationship is not perfect and there are considerable differences between the two.

For example, absolute mathematics performance in Norway and the US is similar to performance in Portugal – but Portugal is effective and Norway and the US are not. For their socio-economic context, Norway and the US perform below what can be expected, whereas Portugal exceeds its expected performance. You could argue that for their advantaged socio-economic conditions, the US and Norway could perform considerably higher than they do.

Which countries are most effective? PISA 2012, math absolute scores versus effectiveness scores, adjusted for socio-economic context. The horizontal line at 0 separates effective from ineffective systems and the vertical line at 500 score points indicates the OECD average. Albania and Lichtenstein removed for missing data.
PISA 2012 data., Author provided

It’s clear that some countries perform very differently when their socio-economic situation is taken into account, whereas others perform more or less the same. For example, performance in Mexico, Spain, Finland, and New Zealand practically does not change after the socio-economic context is considered.

Better than expected

But performance is much higher than expected in Thailand, Turkey, and Portugal. Put differently, these systems would score higher in PISA rankings if it took into account the socio-economic context of the countries.

There are, on the other hand, systems such as Norway, Sweden, the US, Israel, Greece, Jordan, and Qatar that perform worse than expected. It seems that comparatively, these countries could perform better considering the available economic resources and the socio-economic characteristics of the student populations they serve.

But there might be other cultural and economic factors, such as the distribution of economic resources, that condition the results of effectiveness and are difficult to change through reforms to education policy.

The PISA results tell us how countries perform in absolute terms. But
looking at how effective education systems are according to their socio-economic context offers a complementary perspective.

Viewed this way, it is possible for an education system operating in relatively disadvantaged conditions, such as Indonesia, Thailand, Turkey, and Vietnam, to perform better than one with higher performance according to PISA rankings but operating under more favourable socioeconomic conditions, such as Iceland, Norway, the UK, the US, and Sweden.

Publication does not imply endorsement of views by the World Economic Forum.

To keep up with the Agenda subscribe to our weekly newsletter.

Author: Daniel Caro, Research Fellow, University of Oxford and Jenny Lenkeit, Research Fellow, Oxford University Centre for Educational Assessment, University of Oxford

Image: Caroline Hunt, 8, places a weight in position to balance a puzzle in the Math Midway exhibit at the Boonshoft Museum of Discovery in Dayton, Ohio.  REUTERS/Skip Peterson