Researchers demonstrate poor methodology behind PISA studies

The Programme for International Student Assessment (PISA) is a worldwide study by OECD in member and non-member nations of 15-year-old school pupils' scholastic performance on mathematics, science, and reading.

Inge Henningsen, associate professor emeritus at Department of Mathematical Sciences, has, together with Peter Allerup, professor emeritus at DPU, Aarhus University, scrutinised PISA. The result is the book 'PISA - matematik'.

Both researchers are statisticians and have monitored the Danish PISA mathematics studies, since they were first launched in 2001.

"We have used our statistical background to take the readers with us down into the engine room for a closer look at what kind of mathematical models, they use in the PISA studies. If you don't know the techniques, you can't recognise that the analyses are often not in accordance with the principles, PISA is officially based on," says Peter Allerup.

PISA concludes too much

According to Henningsen  and Allerup PISA concludes beyond what the studies actually can say. The mathematical models and principles for comparison of pupil performances, that form the basis for the actual test design, are setting some limitations for the conclusions.

For example: The PISA model requires that all tasks in relation to all pupils must measure 'the same'.

"Now, most probably have a perception that mathematics and math tasks has many different dimensions, and that you can be good or bad for various types of tasks. Some are good at calculation assignments and bad at geometry tasks, while others have it the other way around," says Inge Henningsen.

But in PISA, the easy tasks must be easy for everyone. And the same applies to the difficult tasks. They must be difficult for all pupils in the world.

"This means that tasks that are difficult in Denmark but easy in another country, cannot be included in the test. And what types of tasks are we then left with?" asks Inge Henningsen rhetorically.

Random measurement

"We actually do not know which competencies, PISA tests. The statistical models they have chosen, can formally deliver ranking but they give no nuances," says Inge Henningsen.

PISA does not show anything about the details behind the pupils' understanding and knowledge. And it is thus very randomly, what the tests are measuring. And therefore also which results will appear.

"The PISA studies show that it doesn't matter how much money you spend at the school. Nor does class sizes have any effect, and involving teaching leads to poorer results - because you compare totally heterogeneous groups. But all other experience say the opposite," says Inge Henningsen.

Read more about the book PISA-matematik (in Danish)