Seven Questions about Creativity and Creative Thinking

Annex A.2. Scoring methods and coding design in the PISA CT Rescoring project

Introduction

The PISA 2022 Creative Thinking assessment framework describes the scoring rules applied in the PISA 2022 test (OECD, 2023[1]). The international scoring rules focused on evaluating students’ capacity to generate “sufficiently” original or different ideas (depending on the item type) across task contexts, and accordingly, student responses were awarded either full credit, partial credit (in some cases), or no credit. This approach was adopted to maximise the scalability and reliability of the human coding effort in each country/economy and to facilitate the cross-cultural comparability of scores by focusing on scoring criteria that could be evaluated most objectively across diverse contexts (Foster and Piacentini, 2025[2]). However, the trade-off of the PISA 2022 scoring approach was that it could not identify the most creative responses within each participating country/economy, nor did it evaluate the overall “creative value” of students’ responses (Foster and Piacentini, forthcoming[3]).

The three scoring methods applied in the PISA CT Rescoring project conceptualise and prioritise creative value in different ways to each other and to the international scoring rules applied in the PISA 2022 Creative Thinking assessment. The three scoring methods applied to the PISA CT Rescoring project data sample were:

• The Holistic Judgement method: Student responses were given a single “creativity” score reflecting the overall creative quality of the response, considering all relevant elements together.

• The Criteria-Based method: Each idea within a student response was scored for its appropriateness, originality and value, and responses to tasks requiring multiple ideas were scored for idea flexibility.

• The Conceptual Knowledge Tree method: Each idea within a student response was mapped onto a branch and node of a conceptual knowledge tree that defines all possible solution path ideas in a problem space.

  The Holistic Judgement method and Criteria-Based method were applied to all student responses in the PISA CT Rescoring project sample. The Conceptual Knowledge Tree method was applied to two problem-solving tasks only, Task 5 – Food Waste and Task 7 – The Exhibit (see Figures F and H in the Introduction).

Holistic Judgement Method

Objective of scoring method

The Holistic Judgement scoring method involves attributing valid student responses with a single score between 1 (least creative) and 7 (most creative), based on judges’ holistic judgement of the creativity of the response. Scores should reflect judges’ subjective appraisal of the response, taking into account all aspects of its appropriateness, originality and usefulness, as well as idea flexibility in responses that include multiple ideas per response. The main objective of the holistic judgement method is to establish a relative ordering of the overall creative quality of responses within each country-language-task sub-sample, and to be able to identify the most creative responses within each national-language context.

Common rubric for attributing scores

Judges attributed scores with respect to within-country comparisons. Therefore, a holistic creativity score of 7 denotes that a response is considered to be amongst the most creative observed by the judge compared to all other responses in the country-language-task dataset. Likewise, a score of 4 denotes a response is considered of average quality with respect to the country-language-task sample, and a score of 1 denotes that a response is amongst the least creative observed within the country-language-task dataset.

To help judges interpret holistic creativity scores in a consistent way across country-language groups (i.e. to minimise culturally sensitive grading styles), a high-level rubric was provided that divided the available holistic creativity scores into three “score bands” and that described, in general terms, the overall quality of responses corresponding to each score band.

Judges were instructed to use the rubric as a guideline rather than a strict prescription, given that scores attributed according to this scoring method are ultimately subjective in nature. Judges were also instructed to approximate a normal distribution in their holistic creativity scores to facilitate relative ranking comparisons across country-language groups.

Score band 0 points (non-appropriate or missing responses)

Responses that are non-appropriate (i.e. are not meaningful or do not clearly connect to the task prompt) or that are missing should be coded as 0 score points.

Score band 1-2 points (“low creativity” responses)

Responses in this score band may reflect a limited effort to engage seriously with the task, may not be wholly relevant or appropriate considering the task constraints, and may lack original, thoughtful, interesting or noteworthy elaboration. Responses show minimal imagination and are conventional in their approach, with little personal flair.

• Around 10-20% of all responses in the country-language-task sample should be scored within this band.

• Note: For ‘generate diverse ideas’ tasks that require multiple ideas per student response: if students provide only one appropriate solution idea, they should be awarded a maximum score of 2 points.

Score band 3-5 points (“average creativity” responses)

Responses in this score band may reflect a reasonable level of engagement with the task but tend to rely on relatively common or obvious idea associations that lack further original, thoughtful, interesting or noteworthy elaboration. Responses show some signs of imagination but in a relatively conventional approach to the task that respects expected rules and conventions, albeit with some personal flair.

• Around 60-80% of all responses in the country-language-task sample should be scored within this band.

• Note: For ‘generate diverse ideas’ tasks that require three ideas per student response: if students provide only two appropriate solution ideas, they should be awarded a maximum score of 5 points.

Score band 6-7 points (“high creativity” responses)

Responses in this score band may reflect a significant engagement with the task, may propose relatively novel or original idea associations, and may add significant elaboration or detail that has some interesting, emotive, reflective, humorous or otherwise noteworthy impact on the judge. The response follows an innovative or original approach and demonstrates attention to detail. It effectively blends personal flair with technical knowledge.

• Around 10-20% of all responses in the country-language-task sample should be scored within this band.

Coding process

Coding design

For each task, at least two judges per country-language group were involved in scoring student responses so that inter-rater reliability metrics could be calculated and verified (see ‘Inter-rater reliability’ sub-section for more information). Multiple judges scored a sub-sample of around 100 responses per task, but not all student responses for each task were coded by multiple judges in all country-language groups.

In most country-language groups, a designated lead judge scored around 100 responses in the task sub-sample together with a second judge (i.e. the double coded sub-sample). The lead judge was responsible for verifying inter-rater reliability and implementing recalibration (if needed). The second judge scored the entirety of the task response sample. In Brazil and Portugal, coding teams implemented a balanced Incomplete Block Design, where each response in the task sample was coded by rotating pairs of judges. In this coding design, all possible combinations of two judges are represented in the coding of each task, with judges scoring batches of around 50 responses per pair of judges. A designated lead coder in each team was still responsible for verifying inter-rater reliability across judge pairs and implementing recalibration (if needed).

Coding steps

The Holistic Judgement method was the first of the three scoring methods applied by country-language teams in the PISA CT Rescoring project. Judges were instructed to follow the steps below when scoring responses according to the Holistic Judgement method:

1. As a first step, both judges assigned to a given task should familiarise themselves with all the responses in their country-language-task sample as well as the key pointers for scoring described in the PISA CT Rescoring project coding guide. As part of this familiarisation process, judges should identify and code any clearly aberrant, off-topic or missing responses (code 0).

2. Next, the designated lead judge should code 100 responses in the task sample (i.e. responses 1 to 100). The designated second judge should also code the same 100 responses in the task sample in reverse order (i.e. responses 100 to 1).

3. The two judges should then review their inter-rater reliability metrics for the double-coded sample of 100 responses and discuss their scores if flagged as requiring calibration attention (i.e. where one or more of the inter-reliability metric thresholds is exceeded). Judges should adjust their scores accordingly following calibration discussions until the inter-rater reliability metrics are within the acceptable range.

4. After this review (and where needed, calibration exercise), the second judge should proceed to score the entire response sample for the task.

Inter-rater reliability metrics

For the holistic judgement method, the main metrics used to evaluate inter-rater reliability across the double-coded sample were:

• Correlation coefficient: r > 0.75

• Mean score difference between judges: x̄ difference < 1

If these thresholds were breached after completing the double-coded sample, then judges were instructed to meet to discuss and recalibrate their scores to try and reach, or at least approximate, the acceptable range(s). Across all tasks, all coding teams maintained a mean score difference between judges of less than one score point. In Brazil, the average correlation coefficient between judge pairs was below 0.75 in five tasks (Tasks 2, 4, 5, 6 and 7), although in three of these tasks (Tasks 5-7) the average correlation coefficient remained above 0.7. In Tasks 2 and 4, the average correlation coefficient between judge pairs was 0.57 and 0.58, respectively.

Criterion-Based Method

Objective of scoring method

The Criteria-Based scoring method involves attributing student ideas within a response separate scores based on a series of individual criteria: appropriateness, originality, and value; and, where multiple ideas are involved, idea flexibility (i.e. for ‘generate diverse ideas’ tasks). In this scoring method, each idea within a student response receives multiple scores. The main objective of the Criteria-Based scoring method is to facilitate inter-rater agreement based on a set of common criteria recognised by the literature on creativity measurement as being important for defining creative ideas. A secondary objective is to identify cross-cultural differences in the relative weight attributed to these different criteria in the holistic appraisal of students’ responses.

General description of criteria

To help judges understand and evaluate the scoring criteria in a consistent way across country-language groups, and to help judges assign scores consistently (i.e. to minimise culturally sensitive grading styles), country-language teams were provided with a high-level description of the scoring criteria (see below) that described the available score points for each criterion. In addition, teams were also provided with an item-specific contextualisation of the scoring criteria for each task. The item-specific criteria were developed and standardised based on the inputs of the country-language teams participating in the PISA CT Rescoring project.

Appropriateness (0-2 score points)

Appropriateness refers to the extent to which a response respects the task instructions and constraints, is in the required format, and is relevant to the task content. Scores for appropriateness range from 0 points (not appropriate) to 2 points (high appropriateness).

• Note: Ideas awarded a score of 0 points for appropriateness should not be further scored for originality, value or (where relevant) flexibility.

Originality (0-3 score points)

Originality refers to the extent to which a response presents a relatively uncommon, unusual, different, imaginative or innovative idea, or includes novel or imaginative details, with respect to other responses in the country-language sample. Scores for originality range from 0 points (not at all original) to 3 points (very original).

Value (0-3 score points)

Value refers to the extent to which a response is useful and impactful for its stated purpose. Scores for value range from 0 points (no value) to 3 points (very high value).

Flexibility (variable score points depending on number of ideas required)

Flexibility refers to the extent to which ideas within the same student response are different from each other. Scores for flexibility range from 0 point (not different) to 1 point (both ideas different), or 0 point (not different) to 2 points (all three ideas different), depending on the number of ideas required by the task.

• Note: for ‘generate diverse ideas’ tasks, all ideas within the response should be considered together when attributing a response flexibility score. There is therefore only one flexibility score per response.

Coding process

Coding design

For each task, at least two judges per country-language group were involved in scoring student responses so that inter-rater reliability metrics could be calculated and verified (see ‘Inter-rater reliability’ sub-section for more information). Multiple judges scored a sub-sample of around 100 responses per task, but not all student responses for each task were coded by multiple judges in all country-language groups.

In most country-language groups, a designated lead judge scored around 100 responses in the task sub-sample together with a second judge (i.e. the double coded sub-sample). The lead judge was responsible for verifying inter-rater reliability and implementing recalibration (if needed). The second judge scored the entirety of the task response sample. In Brazil and Portugal, coding teams implemented a balanced Incomplete Block Design, where each response in the task sample was coded by rotating pairs of judges. In this coding design, all possible combinations of two judges are represented in the coding of each task, with judges scoring batches of around 50 responses per pair of judges. A designated lead coder in each team was still responsible for verifying inter-rater reliability across judge pairs and implementing recalibration (if needed).

Coding steps

The Criteria-Based scoring method was applied after the Holistic Judgement scoring method. Judges were instructed to follow the steps below when scoring responses according to the Criteria-Based method:

1. As a first step, both judges assigned to a task should (re)-familiarise themselves with all the responses in their country-language-task sample as well as the key pointers for scoring described in the PISA CT Rescoring project coding guide.

2. Next, the designated lead judge should code 100 responses in the task sample (i.e. responses 1 to 100) according to the item-specific appropriateness criteria (score code 0, 1 or 2). The second judge should also code the same 100 responses in the sample according to the item-specific appropriateness criteria in reverse order to the lead judge (i.e. responses 100 to 1).

3. Then, the two judges should review their inter-rater reliability metrics for appropriateness for the double-coded sample of 100 responses and discuss their scores if flagged as requiring calibration attention (i.e. where one or more of the inter-reliability metric thresholds has been exceeded). Judges should adjust their scores accordingly following any calibration discussions until the inter-rater reliability metrics are within the acceptable range.

• Note that any ideas that receive a score code of 0 for appropriateness should subsequently be treated as missing for the rest of the scoring exercise and should therefore not be scored for originality and value. Ideas that receive a score code of 0 for appropriateness should also not contribute towards the response flexibility score.

4. After this review (and where needed, calibration exercise), the lead judge should code the same 100 responses in the task sample in reverse order to Step 2 (i.e. responses 100 to 1) according to the item-specific originality criteria (score code 0, 1, 2 or 3). The second judge should also code the same 100 responses in the task sample in reverse order (i.e. responses 1 to 100).

5. Next, the lead judge and second judge should code the same 100 responses in the task sample in reverse order again (i.e. responses 1 to 100 for the lead judge, or responses 100 to 1 for the second judge) according to the item-specific value criteria (score code 0, 1, 2 or 3).

6. Finally, the lead judge and second judge should code the same 100 responses in the task sample in reverse order to the previous step according to the item-specific flexibility criteria (score code 0 or 1, or score code 0, 1 or 2, depending on the number of ideas required by the task).

• Note that appropriate ideas in a student response should contribute to its flexibility score.

7. The two judges should then review their inter-rater reliability metrics for the double-coded sample of 100 responses for originality, value and flexibility scores. Judges should discuss their scores if flagged as requiring calibration attention (i.e. where one or more of the inter-reliability metric thresholds has been exceeded) for any of the score criteria. Judges should adjust their scores accordingly following any calibration discussions until the inter-rater reliability metrics are within the acceptable range for each score criteria.

8. After this review (and where needed, calibration exercise), the second judge should proceed to score the entire task sample for idea appropriateness, followed by idea originality, followed by idea value, and finally response flexibility. The second judge should continue to score all responses for each criterion before moving onto the next criterion.

Inter-rater reliability metrics

For the Criteria-Based scoring method, the main metrics used to evaluate inter-rater reliability across the double-coded sample for the four criteria were the same as for the Holistic Judgement method:

• Correlation coefficient: r > 0.75

• Mean score difference between judges: x̄ difference < 1

If these thresholds were breached after completing the double-coded sample, then judges were instructed to meet to discuss and recalibrate their scores to try and reach, or at least approximate, the acceptable range(s). Across all tasks, all coding teams maintained a mean score difference between judges of less than one score point in all four of the score criteria. However, some teams did not meet the targeted correlation coefficient for all criteria in all tasks: in Brazil, the average correlation coefficient between judge pairs was below 0.75 for appropriateness in three tasks, for both originality and value in five tasks, and flexibility in three tasks; in Portugal, the average correlation coefficient between judge pairs was below the target threshold for appropriateness and originality in four tasks, and for value and flexibility in three tasks; in Belgium (French community), the correlation coefficient was not achieved for appropriateness and flexibility in one task, for originality and value in two tasks; and in Hong Kong (China), the correlation coefficient was not achieved for value in one task.

Conceptual Knowledge (C-K) Tree Method

Objective of scoring method

The Conceptual-Knowledge (C-K) Tree scoring method has emerged in recent years for scoring problem-solving items in divergent thinking tasks (Hatchuel and Weil, 2009[4]; Agogué et al., 2014[5]). This approach makes it possible to generate, before obtaining any response data, an exhaustive mapping of relevant concepts and knowledge within a given problem and solution space. This objective mapping, often conceptualised as a “tree” with branches, sub-branches and nodes, allows judges to associate ideas within a solution response onto a well-defined schema from which ideational pathways can be inferred, as well as more granular idea flexibility and originality metrics.

Coding materials

To code responses according to this method, judges use two coding materials:

• C-K Tree schema: The C-K Tree schema represents all possible solution paths and is presented in the form of a decision tree, the ends of whose branches represent major categories of solutions (i.e. all solutions within a given sub-branch). Each major category (sub-branch) is labelled with a given letter (e.g. “A”, “B”, “C”, etc.)

• List of nodes: Each sub-branch (i.e. major category) contains several nodes, which represent different, detailed types of responses. Each idea within a student response can be assigned to a particular node. Each node therefore has its own code, with the first letter of the code corresponding to the relevant sub-branch (e.g. for nodes within sub-branch “A”, available node codes may be “Aa”, “Ab”, “Ac”, and so on).

The full C-K Tree schema and comprehensive lists of nodes for Task 5 – Food Waste and Task 7 – The Exhibit can be found in Annexes C1-C4.

Coding process

The C-K Tree method was the last of the three scoring methods applied by country-language teams in the PISA CT Rescoring project.

Like the holistic judgement method and the criteria-based scoring method, two judges per country-language group were involved in scoring student responses so that inter-rater reliability metrics could be calculated and verified (see ‘Inter-rater reliability’ sub-section for more information). Multiple judges scored a sub-sample of at least 100 responses per task, but not all student responses for each task were coded by multiple judges in all country-language groups.

For each idea within a student response, judges were asked to assign ideas to the node of the C-K Tree schema that best represented the idea. In cases where an idea could be assigned to multiple nodes, judges could assign multiple codes (though the node that most closely maps to the idea should be assigned in the “main code” space).

Judges were instructed to follow the steps below when scoring responses according to the C-K Tree method:

1. As a first step, both judges assigned to a given task should familiarise themselves with the C-K Tree schema and the different major categories of responses. Note that in this scoring method, all ideas within a student response should be mapped onto a node of the C-K Tree schema, including ideas that might otherwise be considered non-appropriate (i.e. ideas that scored 0 points in the Holistic Judgement scoring method or that score 0 points for appropriateness in the Criteria-Based method).

2. Next, the designated lead judge should code 100 responses in the task sample (i.e. responses 1 to 100). The designated second judge should also code the same 100 responses in the task sample in reverse order (i.e. responses 100 to 1).

3. The two judges should then review their inter-rater reliability metrics for the double-coded sample of 100 responses and discuss their scores if flagged as requiring calibration attention (i.e. where one or more of the inter-reliability metric thresholds is exceeded). Judges should adjust their scores accordingly following calibration discussions until the inter-rater reliability metrics are within the acceptable range.

4. After this review (and where needed, calibration exercise), the second judge should proceed to score the entire response sample for the task.

References

[5] Agogué, M. et al. (2014), “The impact of type of examples on originality: Explaining fixation and stimulation effects”, The Journal of Creative Behavior, Vol. 48/1, pp. 1-12.

[2] Foster, N. and M. Piacentini (2025), “Assessing creative thinking at scale and across culture”, in Katz-Buonincontro, J. and T. Kettler (eds.), Oxford Handbook of Creativity and Education, Oxford Handbooks, Oxford University Press.

[3] Foster, N. and M. Piacentini (forthcoming), “PISA 2022 Creative Thinking: What it is, what it is not, and what next?”, Journal of Creative Behavior.

[4] Hatchuel, A. and B. Weil (2009), “CK design theory: An advanced formulation”, Research in Engineering Design, Vol. 19, pp. 181-192.

[1] OECD (2023), PISA 2022 Creative Thinking Framework, PISA, OECD Publishing, https://doi.org/10.1787/dfe0bf9c-en.