Share

Completed RDI Projects

.................................................................................

Increasing efficiency through multi-stage adaptive testing

 

RDI MSAT project icon

About 

Starting with the 2018 cycle, PISA uses a multi-stage adaptive testing (MSAT) algorithm to assign different test forms to students with different abilities. This helped PISA address test-fairness concerns and reduced measurement error, especially for students with exceptionally low or high performance.

This project explored PISA’s current approach to adaptive testing, identified other adaptive designs that can be incorporated into the instrument, and provided recommendations regarding future test designs.

Related outputs
  • Adaptive Testing Recommendations for future tests (endorsed by the PISA Governing Board in March 2023)
  • Invited Symposium
  • Buchholz J., Shin H. J. and M. Bolsinova (forthcoming); “Test engagement in multistage adaptive testing - A causal analysis based on PISA 2018”
  • Chang H. and Y. Zheng (forthcoming); “On-the-fly Multistage Testing: An Alternative Design for PISA”
  • Frey, A., König, C. and Fink, A. (2024), A Highly Adaptive Testing Design for PISA. Journal of Educational Measurement. https://doi.org/10.1111/jedm.12382
  • Shin H. J., C. König, F. Robin, K. Yamamoto and A. Frey (forthcoming); “Robustness of Multistage Adaptive Testing Designs in Educational Large-Scale Assessments”
  • Van Rijn P. , U. Ali , H. J. Shin  and F. Robin (forthcoming); “Stepwise Assembly for Multistage Adaptive Testing: An Application to PISA 2022 Mathematics”

Developing measures of engagement


RDI engagement project icon

About 

Because performance in PISA has limited consequences for students who sit the test, their motivation to show what they know and can do might not always conform to the expectations of test developers and policy makers who rely on PISA data for decision making. Student motivation, effort, and engagement might influence not only their performance on the PISA test, but also the reliability of the responses in the student questionnaire. Systematic differences in student engagement between groups of students thus hold the potential to distort valid comparisons between these groups.

This project aimed to enhance our understanding about the phenomenon based on previous cycles of PISA, and to inform the development and validation of PISA instruments as well as the use and communication of PISA data in the future.

Related outputs
  • "Item Characteristics and Test-Taker Disengagement in PISA". OECD Education Working Papers, No. 312, OECD Publishing, Paris, https://doi.org/10.1787/7abea67b-en. 
  • Buchholz J. , M. Cignetti and M. Piacentini (2022); “Developing measures of engagement in PISA”. A subset of the results was also published in the PISA in Focus series (no. 119).
  • Buchholz J., H.J. Shin and M. Bolsinova (forthcoming); “Test engagement in multistage adaptive testing - A causal analysis based on PISA 2018”
  • Uliztzch E., J. Buchhoz, H.J. Shin, O. Lüdtke, J. Bertling (forthcoming);“Scale format and careless responding"

Using process data to augment measures of student performance

 

RDI process data project icon

About 

The transition to technology-based assessments has introduced new opportunities to incorporate more open, interactive and engaging tasks that require students to engage in complex response processes. These opportunities come with the possibility to digitally capture fine-grained data on students’ actions throughout the tasks, thus broadening the possibilities to make inferences on students’ thinking processes and behaviours.

This project aimed to illustrate the different possible uses of process data in PISA and to demonstrate best practices in the design of tasks and scoring models that integrate process data as a source of evidence.

Related outputs

Contact us

For more information, reach out to the PISA innovative assessments team at [email protected]