FAQ

Background and Basics

What does the term "PISA" mean?

PISA is an acronym taken from the "Programme for International Student Assessment".

 

What is the history of PISA?

Responding to member countries’ demands for regular and reliable data on the knowledge and skills of their students and the performance of their education systems, the OECD began work on PISA in the mid-1990s. PISA was officially launched in 1997, with the first survey taking place in 2000, the second in 2003, the third in 2006, the fourth in 2009, the fifth in 2012 and the sixth in 2015. Future surveys are planned in 2018 and beyond…

 

What makes PISA unique?

PISA benefits from its worldwide scope and its regularity. More than 70 countries and economies have taken part in PISA so far and the surveys, which are given every three years, allow them to track their progress in meeting key learning goals. PISA is the only international education survey to measure the knowledge and skills of 15-year-olds, an age at which students in most countries are nearing the end of their compulsory time in school.

 

PISA is also unique in the way it looks at:

  • Public policy issues Governments, principals, teachers and parents all want answers to questions such as "Are our schools adequately preparing young people for the challenges of adult life?", "Are some kinds of teaching and schools more effective than others?" and "Can schools contribute to improving the futures of students from immigrant or disadvantaged backgrounds?"
  • Literacy Rather than examine mastery of specific school curricula, PISA looks at students’ ability to apply knowledge and skills in key subject areas and to analyse, reason and communicate effectively as they examine, interpret and solve problems.
  • Lifelong learning Students cannot learn everything they need to know in school. In order to be effective lifelong learners, young people need not only knowledge and skills, but also an awareness of why and how they learn. PISA both measures student performance in reading, mathematics and science literacy and also asks students about their motivations, beliefs about themselves and learning strategies.

 

Which countries/economies participate in PISA?

All OECD member countries participated in the first three PISA surveys, along with certain partner countries and economies. In total, 43 countries took part in PISA 2000, 41 in PISA 2003, 58 in PISA 2006 and 74 in PISA 2009. For PISA 2009, 65 countries/economies implemented the assessment in 2009 (with results published on 7 December 2010). A further 9 implemented the same assessment in 2010, the results of which were published by the Australian Council for Educational Research in December 2011. Sixty-five economies participated in 2012 and 71 signed up to participate in 2015.


Click here to obtain the full list of PISA participants to date and information on each country’s contacts, website, national report and more.

 

How are countries/economies chosen to participate in PISA?

Countries/economies interested in participating in PISA contact the OECD Secretariat. The PISA Governing Board then approves membership according to certain criteria. Participants must have the technical expertise necessary to administer an international assessment and must be able to meet the full costs of participation. To take part in a cycle of PISA, participants must join two years before the survey takes place. For example, PISA 2012 participants will have joined before March 2010.

 

Who pays for PISA?

PISA is financed exclusively through direct contributions from the participants' government authorities, typically education ministries.

 

Does PISA tell participants how to run their schools?

No. The data collected by PISA shows the successes of some participants' schools and the challenges being faced in other countries/economies. It allows countries and economies to compare best practices and to further develop their own improvements, ones appropriate for their school systems.

 

Are for-profit companies involved in PISA?

All work relating to the development, implementation, reporting and follow-up of PISA is carried out under the sole responsibility of the OECD, under the guidance of the PISA Governing Board which represents the participating countries. The OECD does, of course, contract specific technical services out to individual academics, institutions or companies. All such contracts are awarded through a transparent and open competitive process that ensures that each task is carried out by the best qualified agencies which provide the best value for money, regardless of whether they are for-profit or not-for-profit organisations. No individual academic, institution or company gains any commercial advantage from this since the results of all PISA-related work are placed in the public domain.

Statement by the PISA Governing Board on the development, implementation and reporting of PISA

 

How is PISA governed?

PISA is developed and implemented under the responsibility of the Ministries of Education through PISA’s decision-making body, the PISA Governing Board. The Board has representatives from all member countries plus partner countries with Associate status, currently only Brazil. Countries appoint representatives to the Board who are knowledgeable about large-scale student assessments and their interface with educational policy and practice. Representatives comprise a mix of government officials and staff of research and academic institutions. Such representation is very similar to the composition of the boards that oversee studies by the IEA, for instance.

The Board determines the policy priorities for PISA and oversees adherence to these priorities during implementation. This includes the setting of priorities and standards for data development, analysis and reporting as well as the determination of the scope of work that will then form the basis for the implementation of PISA.

To ensure the technical robustness of PISA, a Technical Advisory Group (TAG) is appointed by the OECD comprising independent, world-renowned experts in the fields that underpin the PISA methodology, such as sampling, survey design, scaling and analysis. The TAG is regularly called upon to adjudicate the PISA methods and the results of individual countries to ensure that what is published from PISA is robust and internationally comparable.

Click here for the names and contact details of the PISA Governing Board.

Who's Who in PISA

Who are the institutions and teams behind PISA?

  • Education authorities: PISA would not be possible without the support and guidance of the education ministries in the participating countries. Click here for links to education ministries in OECD countries.

  • The OECD Secretariat: the OECD Secretariat is responsible for the day-to-day management of PISA. This means that the PISA team monitors the survey’s implementation, manages administrative matters for the PISA Governing Board, builds consensus among countries and serves as a go-between for the PISA Governing Board and the PISA Consortium. Click here for the OECD Secretariat contact list for PISA.

  • The PISA Governing Board: The PISA Governing Board is composed of representatives of OECD members and PISA associates*. Economies that participate in PISA but do not have associate status are welcome to participate in PGB meetings as observers. Representatives are appointed by their education ministries, and the PGB Chair is chosen by the Board itself. Guided by the OECD’s education objectives, the Board determines the policy priorities for PISA and makes sure that these are respected during the implementation of each PISA survey. Click here for the names and contact details of the PISA Governing Board.
*Associates are economies that are not OECD members but have membership rights and obligations in regard to specific OECD bodies and programmes.
  • The international contractors (the "PISA Consortium"): For each PISA survey, international contractors (usually made up of testing and assessment agencies) are  responsible for the design and implementation of the surveys. The contractors are chosen by the PISA Governing Board through an international call for tender. The contractors are typically referred to as the PISA Consortium.

  • The PISA National Project Managers: Working with the OECD Secretariat, the PISA Governing Board and the international contractor, the PISA National Project Managers oversee the implementation of PISA in each participating country/economy. The PISA National Project Managers are appointed by their governments.

  • The PISA Subject Matter Expert Groups: PISA has Subject Matter Expert Groups for its three key areas of testing – reading, mathematics and science literacy – as well as for other subjects when appropriate (problem solving in PISA 2003, for example). These groups are made up of world experts in each area. They design the theoretical framework for each PISA survey.

  • The PISA Questionnaire Expert Group: The Questionnaire Expert Group provides leadership and guidance in the construction of the PISA context questionnaires. The members of the Questionnaire Expert Group are selected by the PISA Governing Board.

The Test and Questionnaires

Who takes the PISA tests?

Schools in each country are randomly selected by the international contractor for participation in PISA. At these schools, the test is given to students who are between age 15 years 3 months and age 16 years 2 months at the time of the test, rather than to students in a specific year of school. This average age of 15 was chosen because at this age young people in most OECD countries are nearing the end of compulsory education. The selection of schools and students is kept as inclusive as possible, so that the sample of students comes from a broad range of backgrounds and abilities.

 

What does PISA test?

Every PISA survey tests reading, mathematical and scientific literacy in terms of general competencies, that is, how well students can apply the knowledge and skills they have learned at school to real-life challenges. PISA does not test how well a student has mastered a school’s specific curriculum.

 

How does PISA test this?

To date, PISA has used pencil-and-paper tests. The tests are made up of both multiple-choice questions and questions requiring students to construct their own responses. The material is organised around texts and sometimes includes pictures, graphs or tables setting out real-life situations. Each PISA survey includes about seven hours of test material. From this, each student takes a two-hour test, with the actual combination of test materials different for every student. In 2009 and 2012, some countries chose to devote additional testing time to a computer-based assessment. In 2015 the default mode of testing was computer.

 

Who creates the test questions?

All PISA participants are invited to submit questions to the international contractors; in addition, the international contractors write some questions. The questions are reviewed by the international contractors and by participants and are carefully checked for cultural bias. Only those questions that are unanimously approved are used in PISA. Further, before the real test there is a trial run in all participants. If any test questions prove to have been too easy or too hard in certain countries/economies, they are dropped from the real test in all countries and economies.

 

What are the PISA context questionnaires? What are they used for?

Students answer a background questionnaire, providing information about themselves, their attitudes to learning and their homes. It takes 20-30 minutes to complete. In addition, school principals are given a 20-minute questionnaire about their schools. Countries can also choose to administer several optional PISA questionnaires: the computer familiarity questionnaire, the educational career questionnaire and the parent background questionnaire. In addition, many countries choose to gather further information through national questionnaires.

The information collected from these questionnaires helps countries to explore connections between how students perform in PISA and factors such as migration, gender and students’ socio-economic background, as well as students’ attitudes about school and their approaches to learning.

 

Are the PISA tests and questionnaires available to the general public?

Yes, for each assessment cylcle a selection of PISA test materials is made available to the general public. Click here to access all publshed PISA test questions and questionnaires to date. You can consult the publication Take the Test, try questions online and take a look at PISA's computer-based questions.

Click here for an interactive selection of PISA test questions.

 

Why aren’t all the PISA test materials available?

In order to allow countries to follow their performance over time, many questions are used in more than one PISA survey. These questions cannot be made public as long as they are in use.

Understanding the Results

How are the tests corrected?

Each participant has its own group of test correctors, overseen by its National Project Manager. They mark the PISA tests using a guide developed by the international contractors and the PISA Subject Experts (with input from all participating countries). The corrections are cross-checked by other experts. The final results are then sent to the international contractors, who in turn transmit the final data to the OECD Secretariat.

 

What do the test scores mean?

PISA scores can be located along specific scales developed for each subject area, designed to show the general competencies tested by PISA. These scales are divided into levels that represent groups of PISA test questions, beginning at Level 1 with questions that require only the most basic skills to complete and increasing in difficulty with each level.

Once a student’s test has been corrected, his or her score in reading, mathematics and science (plus problem solving in PISA 2003) can be located on the appropriate scale. For example, a student who is likely to lack the skills need to correctly complete easiest questions on a PISA test would be classified as below Level 1, while a student who is likely to have many of these skills need to correctly complete the test questions would be at a higher level.

In each test subject, the score for each participating country is the average of all student scores in that country. The average score among OECD countries is 500 points and the standard deviation is 100 points. About two-thirds of students across OECD countries score between 400 and 600 points.

 

Where can I find more information on the PISA scales?

These are described in detail in the PISA assessment framework publications and the PISA technical reports. Free PDFs of these books are available on our Data page.

 

How are countries/economies ranked in PISA?

PISA ranks participants according to their performance in reading, mathematics and science, as well as problem solving in PISA 2003. PISA does not give a collective score for all subjects combined; rather it gives a score for each subject area and countries are ranked by their mean score in each area. However, it is not possible to assign a single exact rank in each subject to each country. This is because PISA tests only a sample of students from each country and this result is then adjusted to reflect the whole population of 15-year-old students in that country. The scores thus reflect a small measure of statistical uncertainty and it is therefore only possible to report the range of positions (upper rank and lower rank) within which a country can be placed. For example, in PISA 2003 Finland and Korea were widely reported as ranking 1st and 2nd in PISA, when in fact we can only say that, among OECD countries, Finland’s rank was between 1st and 3rd and Korea’s was between 1st and 4th.

 

Can results from PISA be compared across the surveys or does each survey stand alone?

Yes, student performance can be compared across the surveys, as can some of the background questionnaire items.

 

Why doesn’t the whole of the People’s Republic of China participate in PISA?

The national ministry has carried out piloting of PISA in several provinces in previous cycles as preparation for fuller participation of China. In addition, Shanghai participated fully in PISA 2009 and PISA 2012. China is now expanding its participation in PISA and the provinces of Beijing, Jiangsu and Guangdong took part in the 2015 survey, in addition to Shanghai. This is a further step towards full participation of the whole of China in PISA and a welcome sign of increased co-operation between the OECD and China.

 

Are the data from the PISA surveys publicly available?

Yes. Click here for the PISA databases.

 

Are the results of every participant's performance in PISA published?

Generally, yes – the PISA reports published after each survey include data from all participants as long as the data and the amount of students tested (the sample size) meet certain standards. If the standards are not attained, depending on the reason, the data are either not published at all or published but set apart from data of other countries with an explanatory note. See below.

 

Anomalies in PISA data

The following lists cases in which the OECD, on the basis of technical advice from the PISA Consortium, removed or annotated national data in the report because of technical anomalies or because the data did not meet the OECD technical standards for PISA.

  • Austria: As noted in the PISA 2000 Technical Report (OECD, 2002), the Austrian sample for the PISA 2000 assessment did not adequately cover students enrolled in combined school and work-based vocational programmes as required by the technical standards for PISA. The published PISA 2000 estimates for Austria were therefore biased (OECD, 2001). This non-conformity was corrected in the PISA 2003 assessment. To allow reliable comparisons, adjustments and modified student weights were developed which make the PISA 2000 estimates comparable to those obtained in PISA 2003 (OECD Working Paper No. 5 “PISA 2000: Sample Weight Problems in Austria”).

    For the PISA 2009 assessment, a dispute between teacher unions and the education minister in Austrial led to the announcement of a boycott of PISA which was withdrawn after the first week of testing. The boycott required the OECD to remove identifiable cases from the dataset. Although the Austrian dataset met the PISA 2009 technical standards after the removal of these cases, the negative atmosphere in relation to education assessments affected the conditions under which the assessment was administered and could have adversely affected student motivation to respond to the PISA tasks. The comparability of the 2009 data with data from earlier PISA assessments can therefore not be ensured and data for Austria have therefore been excluded from trend comparisons.

  • The Netherlands: As noted in the PISA 2000 Technical Report (OECD, 2002), the response rate of schools for the Netherlands for PISA 2000 was insufficient to warrant inclusion in the PISA 2000 database. Therefore, the Netherlands is excluded from trend analysis relating to PISA 2000.

  • Luxembourg: For Luxembourg changes were implemented in the assessment conditions between PISA 2000 and PISA 2003 with regard to organisational and linguistic aspects in order to improve compliance with OECD standards and to better reflect the national characteristics of the school system. In PISA 2000, students in Luxembourg had been given one assessment booklet, with the languages of testing chosen by each student one week prior to the assessment. In practice, however, familiarity with the language of assessment became an important barrier for a significant proportion of students in Luxembourg in PISA 2000. In PISA 2003 and PISA 2006, therefore, students were each given two assessment booklets – one in each of the two languages of instruction – and could choose their preferred language immediately prior to the assessment. This provided for assessment conditions that are more comparable with those in countries that have only one language of instruction and resulted in a fairer assessment of the performance of students in mathematics, science, reading and problem solving. As a result of this change in procedures, the assessment conditions and hence the assessment results for Luxembourg cannot be compared between PISA 2000 and PISA 2003. Assessment conditions between PISA 2003 and PISA 2006 have not been changed and therefore results can be compared.

  • United Kingdom: In PISA 2000, the initial response rate for the United Kingdom fell 3.7% short of the minimum requirement. At that time, the United Kingdom provided evidence to the PISA Consortium that permitted an assessment of the expected performance of the non-participating schools and on the basis of which the PISA Consortium concluded that the response-bias was likely negligible and the results were therefore nevertheless included in the international report. In PISA 2003, the United Kingdom’s response rate was such that required sampling standards were not met and further investigation by the PISA Consortium did not confirm that the resulting response bias was negligible. Therefore, these data were not deemed internationally comparable and were not included in most types of comparisons. For PISA 2006, the more stringent standards are being applied and PISA 2000 and PISA 2003 data for the United Kingdom are therefore not included in the comparisons of this chapter.

  • United States: In PISA 2006, in the United States an error in printing the test booklets, in which the pagination was changed and instructions for some reading items directed students to the wrong page, may have affected student performance. The potential impact of the printing error on student performance was estimated by examining the relative performance of students in the United States on the item set that was common between PISA 2006 and PISA 2003, after controlling for performance on the items that were not likely to be affected by the printing error.

    The predicted effect of the printing error and the wrong directions on student mean performance on the reading test was up to 6 score points, and thus exceeds one standard error of sampling. Reading performance data for the United States are therefore excluded from this publication and the PISA database.

    The predicted effect of the printing error on student mean performance on the mathematics and science tests was one score point. Mathematics and science performance data for the United States, therefore, have been retained.

Technical Questions

Does PISA data fit the underlying theoretical scaling model?

PISA is an international test with items to which students have been exposed to a different extent in different schools, different countries and different curricular contexts. The idea of PISA is not to reduce the assessment to the lowest common denominator of tasks that are taught in all countries in identical ways, but rather to reflect the range of competencies that students across the world need to master to be successful in the different subject areas. As a result, international tests like PISA do not perfectly fit any mathematical item response model that assumes identical student behaviour with respect to all tasks. The PISA assessments have been designed such that perfect model fit is not a necessary attribute of the test to allow for robust comparisons across countries. PISA aims to investigate aspects of a subject area in breadth and depth and also needs to select items suitable for students from a wide range of geographical and cultural contexts.

 

Could alternative scaling procedures, produce different outcomes – more particularly the rankings

There have been several studies which have investigated possible changes in the statistical model used in PISA, and indeed, there have been some modifications over time and there were further modifications in PISA 2015. However, investigations of alternative models do not show significant changes in PISA outcomes. For example, the two-parameter model which Kreiner mentions has been applied to PISA data and no effects have been found on outcomes (Macaskill, 2008). In his own analysis Kreiner explored a different approach, but as he himself acknowledges, this had a less optimal fit to the data than the model already used in PISA. In short, the PISA tests meet the statistical properties of the Rasch model to the extent needed to allow for robust international comparisons

 

How reliable are the PISA rankings?

Any assessment of the skills of people, whether it is a high school exam, a driving test, or an international sample –based assessment like PISA, will have some uncertainty – because the results depend on the tasks that are chosen for the assessment, on variations of the ways in which the assessment was administered, the fact that the assessment is sample-based or even on the disposition of the person taking the test on the day of the assessment. So the goal of PISA is not to eliminate uncertainty, but to design instruments that provide robust comparisons of the performance of education systems in ways that reflect remaining uncertainty. For that reason, for example, each country is assigned a range of ranks in the PISA reports, rather than given a precise rank order.

 

Do the outcomes of PISA – more particularly the rankings – depend on the test items chosen?

The pool of test items which form the PISA assessment are chosen to give a robust and representative international assessment of the respective domains (reading, mathematics and science) across the different characteristics of the domains. If only a sub-set of the items is chosen then the comparative picture may well change.  This is because while, for example, all children learn mathematics in school, teachers, schools and education systems vary in the emphasis which they give to different mathematical topics and in the ways in which they teach these topics. Thus, students in schools or countries where teachers place greater emphasis on algebra relative to other mathematical aspects are likely to perform relatively better in tasks relating to algebra. Similarly, students whose teachers tend to expose them more to open-ended rather than multiple-choice tasks, can be expected to perform better on the former rather than the latter. Any test based on a few tasks that would not show such variability, would have to be constructed around things that are taught in all schools in exactly the same ways. It would simply reflect the lowest common denominator of national curricula and thus be much less relevant for policy or instructional purposes.

 

What steps are taken to ensure the PISA tests and the results from it are robust?

Confidence in the robustness of PISA is based on the rigour which is applied to all technical aspects of the survey design, implementation and analysis, not just on the nature of the statistical model, which has  developed over time and will continue to do so. Specifically on test development, the robustness of the assessment lies in the rigour of the procedures used in item development, trialling, analysis, review and selection.

The task for the experts developing the assessment is to ensure that all these aspects are taken into account, and to use their expert judgment to select a set of test items such that there is a sufficient balance across all these aspects. In PISA this is done by assessment specialists who work with advisory groups made up of international experts. Participating countries also take part in this item selection process.

The criteria used to select items from the available pool which are first trialled, are typically as follows:

  • The psychometric properties of all selected items had to be satisfactory;
  • Items that generated coding problems had to be avoided unless those problems could be properly addressed through modifications to the coding guides; and
  • Items given high priority ratings by national centres were to be preferred, and items with lower ratings were to be avoided.
  • The major framework categories had to be populated as specified in the reading literacy framework; and
  • There had to be an appropriate distribution of item difficulties.

(See: OECD 2012, PISA 2009 Technical Report Paris, OECD Publishing.: page 40)

 In practice, this involves examination of the following item and test statistics:

  • Missing responses
  • Test targeting
  • Scale reliability
  • Dimensionality
  • Local dependency within units
  • Classical item statistics for pooled OECD data in tabular form
  • International summary of dodgy items
  • Country-by-item interaction (country DIF)
  • Gender and language DIF
  • Inter-coder reliability

 

How comparable are the samples of students in PISA?

PISA applies strict technical standards including for the sampling of schools and students within schools. The sampling procedures are quality assured and the achieved samples and corresponding response rates are subject to an adjudication process that verifies that they have complied with the standards set or not. The outcomes of the sampling are described in Annex A2 of Volume I of the PISA 2012 results and in more detail in the PISA 2012 Technical Report.

 

What about bias in the samples arising from some students being deliberately excluded?

All countries attempt to maximise the coverage of 15-year-olds enrolled in education in their national samples. The sampling standards permit countries to exclude up to a total of 5% of the relevant population. Permissible exclusions include if the school is geographically inaccessible or if the students has a disability. In PISA 2012, only a handful of countries exceeded this limit and even then only by a small margin.

 

Why don’t all the students answer all the same test questions?

The test design in PISA is determined by the aim of providing an assessment of performance at the system (or country) level. It is not designed to produce individual student scores, so it is not necessary for each student to receive exactly the same set of test items. Thus, PISA adopts an efficient design in which the full set of test material is distributed among 13 different test booklets, which are randomly assigned to the randomly sampled students who participate in the test.

 

The PISA Technical Reports provide a detailed account of the methodology for PISA’s design, implementation and analysis. Click here to view the PISA 2012 Techincal Report

Responses to Criticism

The OECD makes no effort to make its methodology and processes open and transparent.

The OECD through its website and publications makes available to both the public and specialists all the key information on the methods and processes associated with the PISA surveys.

These include:

  • ­   The assessment frameworks that explain what is to be assessed, why and how
  • ­   Examples of the test and questionnaire items
  • ­   A comprehensive technical report for every cycle that includes detailed technical information on every aspect of the assessment and analysis
  • ­   The data from the assessment
  • ­   The data analysis manuals that enable researchers to further analyse and interrogate the data and
  • ­   The technical manuals for administration and language quality assurance

 

The survey results show only correlations, and do not expose the real reasons behind country performance.

Every country has its own unique features and context and they are the most qualified to understand the implications of the data for their circumstances.

The OECD presents the findings of PISA based on country performance and the information provided by students and principals to the background questionnaires. It does secondary analyses on the findings to show (i) correlations between the data sets and (ii) the characteristic features of high performing countries. It offers these analyses and information to countries so they might then consider them in relation to their own policies and practices.

 

The official OECD analysis of the survey is seriously flawed. The survey results show only correlations and are insufficiently informed as regards country context and policy development over time. The analysis overreaches the conclusions that may be drawn from looking at correlations alone and encourages simplistic policy-borrowing.

The OECD analysis of the survey is undertaken by world-leading analysts and measurement specialists. An international Technical Advisory Group oversees all aspects of sampling, test construction, analysis and reporting.

The OECD publishes the data and analyses from the assessment and encourages countries to inform their policy and practice from the findings. After seven cycles of PISA, there certainly is a rich history of analysis and development.

 

How do we know whether all the effort and money devoted to PISA actually improves education?

In 2018, 80 countries will participate in the PISA survey. The number joining the program grows every cycle. This is because countries know just how valuable the data are.

Many countries have introduced policies and practices based on the PISA data and findings that have clearly led to improved student and country performance.

 

PISA drives curriculum and pedagogy to improve test performance

PISA is an international survey that focuses on how 15 year old students apply their mathematics and science literacy, and reading skills to everyday situations and post school expectations. It draws content that can be found in curriculums across the world, but does not prescribe or promote any one curriculum.

The OECD also highlights the features of quality pedagogy to inform countries, systems and schools of practices that appear to work well. The focus is not on test performance, but better education through high quality teaching and learning.

 

What is the OECD's response to the open letter from Heinz-Dieter Meyer?

First Claim: PISA has-promoted over reliance on standardised testing in many countries. Leading to narrowing of curriculum to measureable outcomes at the expense of other important educational focuses such as physical, moral, civic, and artistic developments.

There is no scientific or significant evidence to show that such programs narrow the curriculum for students. These assessments support and complement curriculum. They do not compete with it.

International and national assessments continue to provide an evidence base to inform decision making in education. This promotes efficient and effective policies and practices.

 

Second Claim: PISA ranking of countries has encouraged governments to resort to short-term fixes designed to help a country quickly climb the rankings.

There is no sound evidence that shows governments implement short-term policies and programs to impact a country's performance in PISA.

On the contrary, the evidence shows that countries carefully consider PISA data and findings together with a range of other indicators and evidence when they make policy decisions. The complex nature of education does not allow for so-called 'quick fixes'.

 

Third Claim: PISA is not administered through proper democratic governance mechanisms ensuring transparency and democratic participation.

Every significant and important decision relating to the program (such as content, technical requirements, assessment and questionnaire items, and analysis, reporting and secondary analyses) is made by the PISA Governing Board (PGB).

The PGB is comprised of every country and economy that participates in the program (80 countries for 2018). The process for decision making and managing the program could not be more democratic.

 

Fourth Claim: PISA has been hijacked by global for-profit assessment and testing companies.

PISA uses a management model that enables it to draw on the highest levels of expertise and experience available globally. The OECD contracts to the best possible service providers irrespective of whether they are a commercial or not-for-profit organisation.

All contractors are responsible to the OECD Secretariat who manages the program on behalf of the PISA Governing Board that is comprised of all participating countries and economies. PISA uses extremely robust procurement processes and is governed by the regulations of the OECD. Both commercial and not-for-profit organisations have provided services to the program, with the majority over the last 16 years being not-for-profit organisations.

 

Fifth Claim: PISA promotes more multiple-choice testing and more scripted lessons and less autonomy for teachers - reducing student and teacher well-being.

PISA uses multiple-choice testing as a primary feature of its assessments because it is highly reliable, efficient, and cost-effective and supports robust and scientific analyses. The results of multiple-choice testing in PISA are extremely valid and reliable.

While PISA provides very valuable information to support evidence-based policy making, it does not prescribe practice for teachers or schools. The secondary analyses of PISA clearly aim to support quality teaching practice and effective learning in the classroom - but neither of these will reduce teacher or student well-being, they are in fact likely to do the opposite.

 

What is the OECD's response to the criticisms by Professor Kreiner?

While respecting Professor Kreiner's criticisms of PISA, the OECD considers these criticisms are not robust or scientifically convincing. 

The OECD's first concern is that while PISA has a very large bank of items, Professor Kreiner focuses his attention and investigations on an extremely small number of items. This very restricted sample is not a basis on which to try to discredit PISA country rankings.  Professor Kreiner has also failed to take advantage of relevant research in relation to the matters he investigated. His claim that countries 'manipulate' the choice of items for their students cannot be substantiated. Finally, while different experts prefer different methods for analysing the data, Professor Kreiner's critique of the use of the highly-regarded Rasch analysis model in PISA cycles is simply not credible.

 

Doesn't the high performance of countries such as Shanghai suggest that the sample has been biased?

The PISA program has rigorous specifications and scientifically robust requirements that regulate the sample that each country uses in the assessment.

Exactly the same requirements are enforced for all participants and the expert international contractor managing the sampling works closely with every country to ensure they understand and comply with these stringent standards. An independent international expert acts as the sampling referee if there is any uncertainty about a sample, and if a country's data do not comply with the sampling requirements that data will not be reported with the results of all other countries.

 

Is there any truth in the accusations that some responses in PISA questionnaires are fabricated?

PISA data - both from the test and the questionnaires that are given to students, school, principals and parents- are validated to high standards. This includes analysis to detect response biases in the questionnaires.  Some reports from an unpublished research paper have claimed that response patterns in the school principal questionnaire in some countries are indicative of the results having been falsified or are otherwise dubious.  Our assessment is that the response patterns highlighted are in fact quite plausible and do not present evidence of falsification, cheating or other improper practices by respondents. Commonality in responses between different respondents to the school questionnaire in a given country can arise because of how schools or programmes are organised, which can result in the same school principal legitimately responding for a number of schools or programmes that share many of the same characteristics. Individual countries have made their own specific responses to these accusations.

It is also important to note that the school principal questionnaire responses are used in the analysis of the PISA test results; they do not have any bearing on the test results themselves.