On the properties of PISA performance indicators

Members of MODUL University Vienna (Ponocny, Bauernfeind, Zins) are involved in a methodological work on the OECD PISA study. This study is an international assessment of students’ performance which takes place every three years in many countries all over the world. A special focus of the methodological analyses are dimensionality issues, that is, whether the strategy of building scales and publishing results on scale level but not on the level of individual items is empirically justified. With respect to competences, PISA compares countries based on Mathematics, Science and Reading performances of students who are around 15 years old.

PISA evaluations involve highly sophisticated methodology, such as item response theory models and multiple imputations (cf. OECD, 2005). A claim that a student’s performance on one of the three competences can be theoretically and sufficiently expressed by one single value is put into a statistical model which can be tested empirically. In the preparatory phase of the PISA studies, test questions (items) were given to a pre-sample and those items selected for the main study for which no deviations from the model assumptions were found.

Nevertheless, full consistence with the model is not guaranteed by the model and needs to be checked carefully. This may be done by so-called goodness-of-fit-tests which test whether some response patterns occur more frequently than is to be expected by the PISA model. The tests investigate the structure of correlations between the item responses (factor analyses) or whether all items behave similarly (for example, when observing over time or with respect to possible differences between subpopulations).

At the moment, a corresponding analysis of PISA 2006 is being undertaken ,with results to be published in Fall. Similar work was carried out for PISA 2003 three years ago (Hambleton, Gonzalez, Plake & Ponocny, 2005, Ponocny, 2006). This analysis detected some deviations from the model’s unidimensionality assumptions.

As the main result regarding the 2003 data analysis, the science scale has to be considered to be at least two-dimensional. A goodness-of-fit-test showed that both items with open verbal and closed responses are more similar to each other than predicted by the model.

The same is true for a total of 34 out of 37 participating countries. Therefore, freely responding to an item in words involves, to some extent, other abilities than responding to multiple choice items or items which are solved by just inserting a correct word.

The results of the goodness-of-fit-test were also validated by investigating the trends between 2000 and 2003. There was no marked trend with respect to closed response items, but a significant negative trend was noted in items with open verbal responses. Austria is the participating country with the largest negative trend on that type of item. Nevertheless, the trend differences between the two types of items are also significant on an international level.

Therefore, a more detailed evaluation of the science performance was recommended.

Relevant Literature

• Hambleton, R.K., Gonzalez, E., Plake, B., Ponocny, I. (2005). Technical Review of PISA. Published in OECD (2006), Call for Tender for an international contractor for the implementation of the PISA 2009 cycle, http://www.pisa.oecd.org/dataoecd/37/8/35696755.pdf.
• OECD (2005a). PISA 2003 Technical Report. Paris: OECD Publications.
• Ponocny, I. (2006). Leistungsabfall bei den Naturwissenschaften: Misst man die Leistung wirklich eindimensional? In: E. Neuwirth, I. Ponocny & W. Grossmann, PISA 2000 und PISA 2003: Vertiefende Analysen und Beiträge zur Methodik (S. 71-90). Graz: Leykam.

Contact
Dr. Ivo Ponocny, MODUL University Vienna, Department of Tourism and Hospitality Management, Am Kahlenberg 1, 1190 Vienna, Austria
Ivo.Ponocny@modul.ac.at | www.modul.ac.at