Seven Questions about Creativity and Creative Thinking

4. Are there cross-cultural differences in creative thinking?

Creativity is generally understood as resulting from the interaction among an individual’s aptitudes, process and their environment, with the outcome being defined as both novel and useful within a given social context (Plucker, Beghetto and Dow, 2004[1]). Any creative product is therefore embedded within a particular social context, that in turn is inherently shaped by cultural norms and expectations as well as other socio-environmental factors.

Cultural norms and expectations can influence the skills that individuals choose to develop, the values that shape their personality development, and differences in performance expectations within societies (Niu and Sternberg, 2003[2]; Wong and Niu, 2013[3]; Lubart, 1998[4]; Shao et al., 2019[5]). For example, Rudowicz (2003[6]) analysed differences in creativity perceptions between “Western” and “Eastern” cultures noting that, in the West, creativity often implies a break with tradition and is primarily valued for solving problems or achieving personal success, whereas in the East, an individual’s creativity is primarily valued for its social and moral contribution. Some studies have also investigated how cultural differences affect national measures of creativity and innovation, with differences along the individualism-collectivism spectrum significantly shaping how creative products can be defined and valued (Rinne, Steel and Fairweather, 2013[7]; Ng, 2003[8]).

Confluence approaches of creativity also emphasise the role that socio-environmental factors play in influencing an individual’s capacity to engage in creative work (Amabile, 1983[9]; 2012[10]; Amabile and Pratt, 2016[11]; Sternberg and Lubart, 1991[12]; 1995[13]; Sternberg, 2006[14]). Findings from organisational research have demonstrated that practices and extrinsic motivators like receiving informal feedback, appropriate recognition and encouragement, engaging in goal setting, working as a team, and enjoying task autonomy all enable individuals to develop creative ideas (Amabile, 2012[10]; Zhou and Su, 2010[15]).

From an education perspective, cultural norms will inevitably influence the outcomes an education system values for its students and the content it prioritises in the curriculum. In some cases, educational approaches might actively discourage creative thinking and achievement at school (Wong and Niu, 2013[3]). For example, the pressures of standardisation and accountability in educational testing systems might reduce opportunities for creative thinking in schoolwork (DeCoker, 2000[16]). Classroom practices will also be affected by their school environment. For example, school leaders and teachers can cultivate environments that help students learn when and how creative thinking is appropriate, encourage students to set their own goals and identify promising ideas, and take responsibility for contributing to creative teamwork (Beghetto and Kaufman, 2010[17]; 2014[18]). Certain classroom practices might also stifle creative thinking, for example by perpetuating the idea that there is only one way to learn or solve problems, by cultivating attitudes of submission and fear of authority, by promoting beliefs that creativity is a rare innate quality, or by discouraging students’ curiosity and inquisitiveness in formal learning contexts (Nickerson, 2010[19]).

In Chapter 1 of this report, we explored what makes ideas creative and how the different score criteria used in this study (appropriateness, originality and value) relate to judges’ holistic creativity scores. In this chapter, we focus on differences in how creativity is understood, manifested and evaluated across the different socio-linguistic groups included in the PISA CT Rescoring study. For example, what are the differences in terms of the quality of students’ ideas across country-language groups? Is the extent to which the criteria of appropriateness, originality and value determine holistic creativity scores similar across socio-linguistic contexts?

The goal of this chapter is not to “re-rank” country-language groups by students’ creative thinking performance. For a variety of reasons – including the limited subset of country-language groups, tasks and student data included in the study, as well as differences between the scoring methods involved – direct comparisons of student scores and the relationships between the different scoring methods across country-language groups should be interpreted with caution (see Box 4.1). Nonetheless, these data can still provide interesting insights that, when considered together as a whole, demonstrate that students and/or judges may differ in how they interpret “creative quality” across socio-linguistic contexts.

Box 4.1. Caution on interpreting the data in this chapter

Results reported in this chapter should be interpreted bearing in mind three key points: (1) the potential effects of judge bias in human scoring approaches; (2) the principles of the two scoring methods involved (holistic and criteria-based methods); and (3) the analytical approach used to compare criteria-based and holistic creativity scores.

Human scoring approaches: addressing rater bias and inter-rater reliability

Any human scoring approach is susceptible to rater bias. Several studies have demonstrated the effects of rater expertise (Kaufman et al., 2008[20]; Kaufman, Baer and Cole, 2009[21]; Long, 2014[22]) and subjective and socio-cultural bias (Lloyd-Cox, Pickering and Bhattcharya, 2022[23]; Kaufman et al., 2010[24]) on scoring creative tasks. The PISA CT Rescoring project implemented several measures intended to reduce rater bias, including anchor training exercises, inter-rater reliability checks, and using task-specific rubrics for attributing the criteria-based scores (see Annex A2). Nonetheless, differences in student scores across country-language groups may reflect differences in judge leniency and the relative importance they assign to different elements of the response (e.g. originality vs relevance), as much as they reflect differences in students’ relative strengths or weaknesses or capacity to generate creative ideas.

Principles of the holistic and criteria-based scoring methods: relative ranking versus absolute scores

The two scoring methods referred to in this chapter differ in how judges assign scores (Box 4.2). The holistic creativity scores reflect a relative ranking of student responses in each country-language-task group, whereas the criteria-based scores reflect the quality of student ideas according to a centralised, task-specific rubric (see Annex A2). Given these key differences, ideas in a holistic score category in one country-language-task group may not be equivalent in terms of quality to ideas in the same holistic score category in another country-language-task group. These fundamental differences between the two scoring approaches should therefore be kept in mind when comparing the relative relationship between the two types of scores, both within and across country-language groups.

Analytical approach: averaging criteria-based scores across ideas in ‘generate diverse ideas’ tasks

The analytical approach adopted in this chapter makes an important assumption that each idea within a student response contributes equally to a judge’s holistic creativity score for that response. For tasks that require students to propose multiple ideas (i.e. tasks classified as ‘generate diverse ideas’ items), idea criteria scores are first averaged within each response before being associated to the response holistic creativity score. However, it may be that some ideas within a response are more important than others for determining judges’ holistic creativity score.

4.1. Appropriateness, originality and value of ideas across country-language groups

Examining the contribution of appropriateness, originality and value scores to the average student idea score in each country-language group can tell us about cross-cultural differences in relative idea quality. Figure 4.1 shows the composition of the mean idea sum score across tasks in the study in each country-language group, divided into shares that account for the mean appropriateness, originality and value score of the average idea (Box 4.2). On average across country-language groups, close to 40% of the mean idea sum score is derived from idea appropriateness, with the relative shares of value and originality scores contributing a smaller proportion towards the mean idea sum score (32% and 29%, respectively).

These differences in relative contribution are notable given that the range of scores for appropriateness is smaller (0-2 points) than for originality and value (0-3 points), respectively. In real terms, students tend to score higher in appropriateness than they do in originality and value, on average, which suggests that they find it comparatively easier to come up with appropriate ideas than novel or useful ideas (Table B2.2). This is probably as expected, as coming up with an appropriate idea is arguably easier than thinking of an original or useful one. Some learning progressions for creative thinking also typically consider generating appropriate ideas as the first stage of proficiency in creative thinking (Krstic, 2024[25]). These results may also be a function of the tasks included in the study: students are expected to come up with appropriate ideas in all tasks, but not necessarily creative, original or effective ideas in all of them (e.g. ‘generate diverse tasks’ only require students to think of relevant different ideas).

Box 4.2. Which score data are used in this chapter?

The analyses in this chapter draw upon two types of rescored data:

• Criteria scores: Each idea within a student response was scored for its appropriateness, originality and value. Appropriateness scores range from 0 to 2 points, whereas originality and value scores range from 0 to 3 points, respectively. A cumulative idea sum score was also produced by aggregating the scores across these three criteria. Tasks classified as ‘generate diverse ideas’ tasks required students to submit multiple ideas per response, and each idea was scored separately for appropriateness, originality and value.

• Holistic creativity scores: Valid student responses were given a score of between 1 (lowest) and 7 points (highest), based on judges’ evaluations of the overall creative quality of the response. The response score reflects a relative ranking of creative quality compared to all other responses in that country-language-task group.

The analyses in this chapter focus on differences in the contribution to and importance of idea appropriateness, originality and value for creative ideas within and across country-language groups. For more information on the scoring methods, see Annex A2.

There are some differences in the general composition of the average idea across tasks across the country-language groups in the study. In Chile, students’ appropriateness scores represent a much larger share of the mean idea sum score (56%) (Table B4.1), and in Greece, Colombia and Saudi Arabia, the appropriateness scores represent a relatively larger-than-average share of the mean idea sum score (43-44%). In the case of Colombia and Chile, this might be explained by the overall lower mean idea scores achieved by students compared to those in other country-language groups: while students in these two countries achieve similar mean appropriateness scores to all other students, their mean idea sum score is around 0.9 score points lower than the average idea sum score (Table B2.2). Notably though, students in Brazil scored similarly (and even marginally lower) than students in Colombia and Chile overall, yet appropriateness scores contributed relatively far less to their mean idea sum score. These results imply two things: (1) that students in Chile, Greece, Colombia and Saudi Arabia directed relatively more of their efforts to generating appropriate ideas than original or valuable ones; and/or (2) that judges in these country-language contexts were relatively more generous in their appropriateness scores compared to their scores for originality and value than in other country-language contexts.

Figure 4.1. Appropriateness scores contribute most to the average idea score, followed by value then originality scores

Mean criteria scores as a relative share of the mean idea sum score (on average across tasks), by country-language group

Note: Each bar represents 100% of the mean idea sum score on average across tasks in each country-language group. Country-language groups are ordered by mean idea sum score (highest to lowest). Mean percentages are rounded to the nearest integer.

Source: Table B2.2 and Table B4.1

On average across country-language groups, there was a modest difference in the relative share of appropriateness and value scores (7 percentage points) (Table B4.1). In many of the country-language groups where appropriateness contributed a relatively large proportion of the mean idea sum score, the relative difference between appropriate and value scores were larger than average – for example in Greece, Saudi Arabia but also France (10-11 percentage points), and particularly in Colombia (17 percentage points) and Chile (32 percentage points). These large differences observed in Chile and Colombia might imply that students found it relatively harder to come up with ideas of added value beyond simply providing an appropriate idea for the task. Alternatively, again, it may be that judges in these country-language groups were relatively less lenient when interpreting the value criterion compared to judges in other country-language groups, or that they incorporated elements in their appropriateness evaluations considered by other judges to better reflect value.

In most country-language groups, originality scores represented the smallest contribution to students’ mean idea sum score (29% on average). This suggests that most students demonstrated a relative weakness in coming up with original ideas compared to appropriate or valuable ideas – especially as some tasks explicitly asked students to focus on idea originality – or it may be that judges were consistently less generous with awarding points for originality compared to the other score criteria across country-language groups. The smallest originality score shares were observed in Chile (20%), Greece (22%), Canada (French provinces, 23%), Saudi Arabia (26%) and Brazil (27%). In only a handful of country-language groups were the relative score shares for originality marginally yet statistically significantly larger than either the appropriateness score share (Italy) or the value score share on average across tasks (Belgium [French community], Colombia, France, Hong Kong [China] and Korea).

4.1.1. Differences by domain context

Are there differences in these general patterns when grouping tasks by their domain context? For example, do value scores contribute relatively more to students’ overall scores in the problem-solving tasks compared to the creative expression tasks, and vice versa? Do students tend to focus relatively more on originality and relatively less on appropriateness in the expressive tasks? On average across country-language groups, we do observe differences in the relative shares of the criteria scores along these lines, but the differences between task groups are small. In the problem-solving tasks, value contributes only 2 percentage points more to the mean idea sum score than on average across all tasks, at the expense of both appropriateness and originality scores, on average across country-language groups, while in the creative expression tasks, originality scores contribute relatively slightly more and value scores slightly less (both 30%) to the mean idea sum score than on average across all tasks (Table B4.1).

In some country-language groups, these differences by domain context are more pronounced. For example, in Brazil and Saudi Arabia, originality contributes to only 23-24% of the mean idea sum score in problem-solving tasks compared to around 30% of the mean idea sum score in creative expression tasks. An even larger difference in the relative share of originality scores across the two task types is observed in Germany (8 percentage points). In both Colombia and Hong Kong (China), appropriateness scores represent a relatively larger proportion of the average idea score in the creative expression tasks compared to problem-solving tasks (a difference of 8-9 percentage points). And in Chile, Colombia, Germany and Greece, value scores constituted around 7-8 percentage points more of the mean idea sum score in problem-solving tasks than for the expression tasks.

Figure 4.2 shows the difference in the relative shares of originality and value scores, as a proportion of the mean idea sum scores, across tasks in the two broad domain contexts in the PISA CT Rescoring study: creative expression tasks and creative problem-solving tasks. In Canada (English and French provinces), Greece and Brazil, value scores represent a larger share of the total idea sum score than originality scores in both types of task context. Conversely, in France, Korea and Hong Kong (China), originality scores represent a larger share of the idea sum score than value scores in both the expressive and problem-solving task types. In Germany, Saudi Arabia, Chile, Slovenia, Italy and Portugal, as well as on average across country-language groups, originality scores represented a larger share of the mean idea sum score in creative expression tasks than value, but value scores represented a larger share of the mean idea sum score in problem-solving tasks. Interestingly, in no country-language group was the opposite observed.

Figure 4.2. Ideas in creative expression tasks tend to be more original than valuable, whereas ideas in problem solving tasks tend to be the opposite

Mean difference in the relative share of originality and value scores as a proportion of the mean idea sum score, by broad domain grouping and country-language group

Note: Significant differences for creative expression tasks are shown by the shape of the marker (circle/diamond) and for creative problem-solving tasks by the colour of the marker.

Source: Table B4.1

4.2. What makes an idea creative? The relative importance of appropriateness, originality and value within creative ideas across cultural-linguistic contexts

4.2.1. Are creative ideas highly appropriate, original and valuable in all country-language groups?

Chapter 1 explored the characteristics of creative ideas, in general, by examining the interactions between response appropriateness, originality and value scores, with response holistic creativity scores. Our results show that response appropriateness scores are strongly positively associated with judges’ holistic creativity scores, on average across tasks and country-language groups (r=0.7), though the strength of this association varies by country-language group – ranging from 0.5 in Canada (French and English provinces) to 0.8 in Colombia, Greece, Korea, Portugal and Slovenia (Table B1.2). Response originality and response value scores were also similarly correlated with response holistic scores (r=0.7), on average, with the strength of these associations ranging from 0.6 to 0.9 by country-language group (Table B1.9 and Table B1.10). These relatively large variations in correlation strength between the different scoring criteria and holistic creativity scores suggest that what matters most in determining creative ideas varies by country-language group.

Figure 4.3 shows the percentage of relatively creative responses – i.e. those that achieved a holistic creativity score of between 5 and 7 score points – that also scored highly for appropriateness (rounded average response idea appropriateness score of 2 points), with notable differences across country-language groups. In Portugal, Greece and Korea, above 95% of relatively creative responses were highly appropriate, whereas in Canada (both French and English provinces), Hong Kong (China) and Italy, less than 80% of relatively creative responses were highly appropriate.

Figure 4.3. Most relatively creative responses are highly appropriate in all country-language groups, though the share varies

Percentage of highly appropriate responses amongst relatively creative responses (on average across tasks), by country-language group

Note: Country-language groups are ordered by percentage of highly appropriate responses amongst relatively creative responses (lowest to highest). Results are computed by task first, then averaged across tasks within each country-language group. “Relatively creative responses” are those that achieve a holistic creativity score of 5, 6 or 7. “Highly appropriate” responses are those whose average idea appropriateness score is 2 (rounded). For responses to items classified as 'generate diverse ideas' items, appropriateness scores for all ideas within the response were first averaged to give a single appropriateness score per student response. The averaged appropriateness scores were then rounded to the nearest integer.

Source: Table B4.2

It may be that differences between the holistic and criteria-based scoring methods account for some of these country-language differences, with the quality of ideas in each holistic creativity score category sensitive to the overall performance of students in the country-language group (see Box 4.1). Yet students in all four country-language groups that have less than 80% highly appropriate responses amongst relatively creative responses – Canada (English and French provinces), Hong Kong (China) and Italy – score around or above the average in terms of their mean idea sum scores (Table B2.2). In other words, it is not because students in these countries struggle to think of good ideas that there are less highly appropriate, relatively creative responses. Instead, students and judges in these country-language groups appear more willing to push the boundaries of appropriateness in the pursuit of creativity, or it may simply be that in these country-language groups, appropriateness is not as important as originality and/or value for determining whether a response is creative or not.

Chapter 1 also investigated the relationship between holistic creativity scores and originality and value scores, respectively. The results showed that both criteria are important for defining creative ideas, in general, and that responses at the very highest holistic creativity scores (scores 6 and 7) tended to have a higher mean originality score than value score, on average across tasks and country-language groups. Are there differences in these general observations across country-language groups? Figure 4.4 shows the difference in the shares of highly original and highly valuable responses (i.e. those with a mean response idea originality or value score of at least 2 points, respectively) within relatively creative responses, by country-language group. In around half of the country-language groups included in the study, the shares of highly original and highly valuable responses amongst relatively creative responses are similar and not statistically significantly different – although on average, there is a small statistically significant difference in favour of highly valuable responses amongst relatively creative responses (difference of 3 percentage points). This result may be explained by the larger number of student responses in holistic score category 5 compared to score categories 6 and 7, with responses in score category 5 scoring higher in value than originality on average across country-language groups (Table B1.7 and Table B1.8). These results generally support the idea that both criteria are important for defining creative ideas across socio-cultural contexts.

Two country-language groups showed a much larger difference in the relative shares of highly valuable and highly original responses amongst responses that achieved a holistic creativity score of at least 5: Greece (difference of 32 percentage points) and Brazil (difference of 20 percentage points). In these two countries, there were far more highly valuable responses amongst relatively creative responses than highly original ones. This is likely because students in both Greece and Brazil struggled to come up with original ideas in general, with the mean idea originality score in both countries around 0.3 score points lower than on average across country-language groups. In contrast their mean value scores were around (Greece) or only slightly below (Brazil) the country-language group average (Table B2.2). Canada (French provinces) also recorded a statistically significant larger share of highly valuable responses amongst relatively creative responses (difference of 10 percentage points). Only in Belgium (French community), France and Korea were there significantly more highly original responses than highly valuable ones amongst student responses that achieved a score of at least 5 points on the holistic creativity scale.

Figure 4.4. Relatively creative responses tend to be more highly valuable than original, though not in all country-language groups

Difference in the relative share of highly original/valuable responses amongst relatively creative responses (on average across tasks), by country-language group

Note: Country-language groups are ordered by difference in relative share of highly original/valuable responses (original-value) amongst relatively creative responses. Values that are statistically significant are indicated by a filled marker. Results are computed by task first, then averaged across tasks within each country-language group. “Relatively creative responses” are those that achieve a holistic creativity score of 5, 6 or 7. For 'generate diverse ideas' items that contain multiple ideas per response, the criteria (originality/value) scores for all ideas were averaged to give a single criteria score per student response. “Highly original/valuable” responses are those with an average response criteria score of at least 2 score points.

Source: Table B4.2

4.2.2. Variation in holistic creativity scores explained by appropriateness, originality and value across country-language groups

Given that the findings shown in Figure 4.3 and Figure 4.4 are sensitive to the overall success of students in each country-language group, another way of examining the relative importance of appropriateness, originality and value in defining creative responses overall is to examine and compare the holistic scores of students with similar criteria scores. Figure 4.5 shows the unique variation in judges’ holistic creativity scores explained by appropriateness, originality and value, respectively, in each country-language group. It shows the additional variation in holistic creativity scores explained by students’ response appropriateness scores, for example, when added to a linear regression model that already includes originality and value scores.

Figure 4.5. The relationship between holistic creativity scores and response appropriateness, originality and value scores varies significantly by country-language group

Proportion of the unique variation in holistic creativity scores explained by mean response appropriate, original and value scores (all tasks), by country-language group

Note: Country-language groups are ordered by the proportion of unique variation in holistic creativity scores explained by appropriate scores (from highest to lowest). The proportion of unique variation in holistic creativity score describes the proportion of variance additionally explained by a given score criterion beyond the variation explained by the other score criteria. These results are likely attenuated given that judges were also instructed to consider idea flexibility when assigning holistic creativity scores to responses to ‘generate diverse ideas’ (GDI) items. Results were computed on the pooled responses across all tasks in each country-language group. For GDI items that contain multiple ideas per response, the criteria (appropriate/originality/value) scores for all ideas were first averaged to give a single criteria score per student response.

Source: Table B4.3

Overall, the results show that there is significant variation across country-language groups in the extent to which response appropriateness, originality and value scores explain holistic creativity scores. These results also do not seem to be highly linked to student success or judge leniency in the scoring. On average, the largest proportion of unique variation in holistic creativity scores is explained by originality scores (around 15%), followed by appropriateness (around 9%) and then value (around 7%). In Canada (French provinces) originality scores explain well over 30% of the unique variation in holistic creativity scores, and in Hong Kong (China), Belgium (French community) and Germany, originality scores explain between 25-30%. In contrast, in Korea and Saudi Arabia, originality scores explain only 5% or less additional variation in holistic creativity scores in a model that already includes response appropriateness and value scores.

The importance of appropriateness scores in explaining holistic creativity scores appears to be the most variable score criteria across the country-language groups included in the PISA CT Rescoring study, with response appropriateness explaining the largest relative proportion of unique variation in holistic creativity scores in Greece (39%), Chile (21%), Colombia (20%) and Korea (13%) yet only marginal amounts (less than 2%) in Italy and Canada (French and English provinces). Conversely, value scores seem to explain a relatively modest but consistent proportion of the unique variation in holistic creativity scores across country-language groups. Only in Greece do value scores seem to explain a relatively large proportion of the unique variation in holistic creativity scores (around 25%) (Table B4.3).

Country-language groups also vary in the extent to which appropriateness, originality and value scores collectively explain the total variation in holistic creativity scores. As shown in Figure 4.6, in Portugal, Greece, Slovenia, Hong Kong (China) and Colombia, over 75% of the total variation in holistic creativity scores can be explained by judges’ response criteria scores, while less than half of the variation in holistic creativity scores in Canada (English provinces) can be explained by these three criteria scores. One additional criteria that would clearly influence holistic creativity scores for responses to some items (‘generate diverse ideas’ items) is idea flexibility, given that responses that contained very different ideas may have achieved a higher holistic score and judges were given guidelines regarding the maximum holistic score to award in cases where some ideas were missing from a response (see Annex A2). However, even when looking at the total variation explained in ‘generate creative ideas’ tasks only (see markers in Figure 4.6), there remains unexplained variation in students’ holistic creativity scores across all country-language groups.

Figure 4.6. Appropriateness, originality and value explain most – but not all – of the variation in holistic creativity scores

Total variation in response holistic creativity score explained by mean response appropriate, original and value scores (all tasks and ‘generate creative ideas’ tasks only), by country-language group

Note: Country-language groups are ordered by proportion of total variation explained across all tasks (highest to lowest). Bars show the total variation in holistic creativity scores explained by criteria (appropriate, originality and value) scores across all tasks (pooled responses), markers show the total variation explained in holistic creativity scores by these factors across ‘generate creative ideas’ items only (pooled responses). For 'generate diverse ideas' items that contain multiple ideas per response, the criteria scores for all ideas were first averaged to give a single criteria score per student response.

Source: Table B1.18 and Table B4.4

Once again, as discussed in Chapter 1 of this report, these results point towards the influence of other important factors that are not captured by the score criteria of appropriateness, originality and value in determining the overall creativity of a student response – for example, the inclusion of domain-relevant elements within a response, or other response characteristics that might implicitly or explicitly influence judges’ holistic creativity evaluations (e.g. sophistication of vocabulary, response length, etc.).

4.3. Measuring creativity across cultures: a final word

Beyond the specific findings of the PISA CT Rescoring project, the results in this chapter highlight the complexity of measuring creativity – both in general, and specifically in a cross-culturally valid and reliable way. Although coding teams in our study completed small anchor training exercises and were required to achieve a certain level of within-country inter-rater reliability in their coding (see Annex A2), it is not easy to disentangle whether the cross-cultural differences observed in this chapter stem from differences in the preferences and priorities of students or differences in how judges understand and implement the evaluation criteria. Prior researchers have noted not only differences in performance expectations across cultures more broadly when it comes to creative work (Shao et al., 2019[5]; Lubart, 1998[4]; Rudowicz, 2003[6]; Ng, 2003[8]; Niu and Sternberg, 2003[2]), but the effects of socio-cultural rater bias in creativity assessments (Kaufman et al., 2010[24]; McCarthy, Chen and McNamee, 2018[26]).

The country-language groups included in the PISA CT Rescoring study represent a diverse set of socio-linguistic and cultural groups across North America, Latin America, Europe, the Middle East and South-East Asia. Despite efforts to reduce cross-cultural bias in the design of the PISA tasks throughout the test development process (see (Foster and Piacentini, 2025[27]; OECD, 2024[28]) for more), task and content familiarity will influence both students’ capacity to generate creative ideas and judges’ capacity to score students’ ideas in a cross-culturally consistent way. In other words, both familiarity with the test materials and culture-related task misinterpretations resulting from different norms or social conventions will influence the accuracy of creativity assessments and the comparability of their results across student groups (Shao et al., 2019[5]). Designing and administering culturally appropriate tasks is therefore important for addressing performance bias across student groups and improving the validity of assessment results. In the context of education specifically, this would mean developing tasks that are more clearly connected to the curriculum and typical classroom experiences of different student populations.

The scoring method used in the PISA 2022 Creative Thinking assessment was developed in such a way to optimise the comparability of the results across diverse socio-cultural and linguistic contexts and to facilitate the implementation of a large-scale, human-scoring operation in 64 countries and economies. This was namely achieved by using statistical infrequency as a more objective proxy for originality, by not including any evaluation of response value, and by developing a relatively simple, atomised scoring procedure with a limited number of score codes. The complex trade-offs resulting from these design choices, including the granularity, sensitivity and interpretation of the PISA scores, are discussed in detail elsewhere (Foster and Piacentini, forthcoming[29]), but the PISA scoring method resulted in a sufficiently reliable approach to scoring and scaling the PISA data.

The scoring methods used in the PISA CT Rescoring study differ significantly by design (see Box 4.1) and in how they either allow for or mitigate potential cross-cultural differences in creativity. The holistic scoring method attempts to rank student responses within a country-language-task group based on a variety of factors, deemed important by each country-language coding team, that contribute to a response’s overall creative quality. This ranking exercise, and the relative weighting of the factors influencing such a ranking, are inherently socio-culturally variable and intended to capture and reflect the cultural specificity of creativity in each context. Therefore, in the holistic scoring method, differences in widespread knowledge and social norms mean that responses that are considered creative in one country-language context may not be considered creative in another (Shao et al., 2019[5]). In contrast, the criteria-based method was developed as a more analytic and standardised approach to scoring creative ideas across country-language teams, but that nonetheless addressed one of the major omissions of the PISA scoring approach (the lack of value criteria). The OECD developed standardised guidelines on how to interpret the appropriateness, originality and value criteria for each task included in the study, given the form and response type varied from task to task. The aim of these guidelines was to establish a shared understanding of the respective scoring criteria for each task and promote their consistent application, both within and across coding teams. While the analyses in this chapter focus on understanding the relationships between the two methods – precisely to understand how the core criteria of appropriateness, originality and value interact with judges’ holistic evaluations – choosing an appropriate scoring method for any measurement context is ultimately a question of the assessment goals, and the extent to which cross-cultural differences in how creativity is understood and manifests need to be mitigated or accommodated.

Key findings and implications

• The criteria of appropriateness, originality and value remain useful and relevant for evaluating creative ideas across diverse country-language groups, but their relative importance may differ across socio-cultural contexts.

  • Cross-cultural differences in the interpretation of appropriateness, originality and value might stem from different cultural expectations of judges or different preferences of student groups.

• Students find it relatively easier to come up with appropriate ideas than original or valuable ideas – with appropriateness representing 39% of students’ mean idea sum scores, on average. However, the proportion of highly appropriate, relatively creative responses varied significantly across country-language groups – from 65% to 100% of creative ideas.

  • The extent to which appropriateness defines creative ideas may vary significantly amongst students and judges from different socio-linguistic and cultural backgrounds. The appropriateness of creative ideas in each country-language group does not seem strongly linked to the overall mean scores of students.

• Originality explained around 15% (the largest share) of the unique variation in holistic creativity scores, on average. However, students demonstrate a relative weakness in coming up with original ideas, which was particularly evident in some countries – possibly because judges are more demanding in their evaluations of what it means to be original in such tasks.

  • Both students and those evaluating their work (e.g. educators) might benefit from support in understanding how originality can be expressed in appropriate, task-relevant ways. In an educational context, Bahar and Maker (2025[30]) suggest linking originality to uncommon ways of demonstrating specific learning objectives in the curriculum, rather than broader norm-based expectations of originality.

• In a measurement context, socio-cultural norms and expectations matter: tasks must be culturally relevant, and evaluation criteria should be contextualised in culturally appropriate ways.

  • Contextualised scoring criteria should also be clearly communicated to students and judges, so that both the creators and evaluators of creative work share an understanding of what constitutes a culturally-relevant, creative idea for a given context.

References

[10] Amabile, T. (2012), Componential theory of creativity, No. 12-096, Harvard Business School, http://www.hbs.edu/faculty/Publication%20Files/12-096.pdf.

[9] Amabile, T. (1983), “The social psychology of creativity: A componential conceptualization”, Journal of Personality and Social Psychology, Vol. 45/2, pp. 357-376, https://doi.org/10.1037/0022-3514.45.2.357.

[11] Amabile, T. and M. Pratt (2016), The dynamic componential model of creativity and innovation in organizations: Making progress, making meaning, https://doi.org/10.1016/j.riob.2016.10.001.

[30] Bahar, K. and C. Maker (2025), “A new perspective on scoring children’s originality: a standards-based criterion-referenced assessment approach”, Frontiers in Psychology, Vol. 16, https://doi.org/10.3389/fpsyg.2025.1545396.

[18] Beghetto, R. and J. Kaufman (2014), “Classroom contexts for creativity”, High Ability Studies, Vol. 25/1, pp. 53-69, https://doi.org/10.1080/13598139.2014.905247.

[19] Beghetto, R. and J. Kaufman (eds.) (2010), How to discourage creative thinking in the classroom, Cambridge University Press, https://doi.org/10.1017/CBO9780511781629.002.

[17] Beghetto, R. and J. Kaufman (2010), Nurturing creativity in the classroom, Cambridge University Press.

[16] DeCoker, G. (2000), “Looking at U.S. education through the eyes of Japanese teachers”, Phi Delta Kappan, Vol. 81.

[27] Foster, N. and M. Piacentini (2025), “Assessing creative thinking at scale and across culture”, in Katz-Buonincontro, J. and T. Kettler (eds.), Oxford Handbook of Creativity and Education, Oxford Handbooks, Oxford University Press.

[29] Foster, N. and M. Piacentini (forthcoming), “PISA 2022 creative thinking: What it is, what it is not, and what next?”, The Journal of Creative Behavior, https://doi.org/10.1002/jocb.70068.

[21] Kaufman, J., J. Baer and J. Cole (2009), “Expertise, domains, and the consensual assessment technique”, Journal of Creative Behavior, Vol. 43/4, pp. 223-233, https://doi.org/10.1002/j.2162-6057.2009.tb01316.x.

[20] Kaufman, J. et al. (2008), “A comparison of expert and nonexpert raters using the consensual assessment technique”, Creativity Research Journal, Vol. 20/2, pp. 171-178, https://doi.org/10.1080/10400410802059929.

[24] Kaufman, J. et al. (2010), “In the eye of the beholder: Differences across ethnicity and gender in evaluating creative work”, Journal of Applied Social Psychology, Vol. 40/2, pp. 496-511, https://doi.org/10.1111/j.1559-1816.2009.00584.x.

[25] Krstic, S. (2024), Putting creative thinking at the core of the English school curriculum: An exploratory study., Australian Council for Educational Research, https://research.acer.edu.au/cgi/viewcontent.cgi?article=1077&context=ar_misc.

[23] Lloyd-Cox, J., A. Pickering and J. Bhattcharya (2022), “Evaluating creativity: How idea context and rater personality affect considerations of novelty and usefulness”, Creativity Research Journal, Vol. 34/4, pp. 373-390.

[22] Long, H. (2014), “More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks”, Thinking Skills and Creativity, Vol. 13, pp. 183-194, https://doi.org/10.1016/j.tsc.2014.05.002.

[26] McCarthy, M., C. Chen and R. McNamee (2018), “Novelty and usefulness trade-off: Cultural cognitive differences and creative idea evaluation”, Journal of Cross-Cultural Psychology, Vol. 49/2, pp. 171-198, https://doi.org/10.1177/0022022116680479.

[8] Ng, A. (2003), “A cultural model of creative and conforming behavior”, Creativity Research Journal, Vol. 15/2&3, pp. 223-233, https://doi.org/10.1080/10400419.2003.9651414.

[2] Niu, W. and R. Sternberg (2003), “Societal and school influences on student creativity: the case of China”, Psychology in the Schools, Vol. 40/1, pp. 103-114, https://doi.org/10.1002/pits.10072.

[28] OECD (2024), PISA 2022 Technical Report, PISA, OECD Publishing, https://doi.org/10.1787/01820d6d-en.

[1] Plucker, J., R. Beghetto and G. Dow (2004), “Why isn’t creativity more important to educational psychologists? Potentials, pitfalls, and future directions in creativity research”, Educational Psychologist, Vol. 39/2, pp. 83-96.

[7] Rinne, T., G. Steel and J. Fairweather (2013), “The role of Hofstede’s individualism in national-level creativity”, Creativity Research Journal, Vol. 25/1, pp. 126-136, https://doi.org/10.1080/10400419.2013.752293.

[6] Rudowicz, E. (2003), “Creativity and culture: A two-way interaction”, Scandinavian Journal of Educational Research, Vol. 47/3, pp. 273-290, https://doi.org/10.1080/00313830308602.

[5] Shao, Y. et al. (2019), “How does culture shape creativity? A mini-review”, Frontiers of Psychology, Vol. 10, https://doi.org/10.3389/fpsyg.2019.01219.

[14] Sternberg, R. (2006), “The nature of creativity”, Creativity Research Journal., Vol. 18/1, pp. 87-98, https://www.cc.gatech.edu/classes/AY2013/cs7601_spring/papers/Sternberg_Nature-of-creativity.pdf.

[4] Sternberg, R. (ed.) (1998), Creativity across cultures, Cambridge University Press, https://doi.org/10.1017/CBO9780511807916.019.

[13] Sternberg, R. and T. Lubart (1995), Defying the crowd: Cultivating creativity in a culture of conformity, Free Press.

[12] Sternberg, R. and T. Lubart (1991), “An investment theory of creativity and its development”, Human Development, Vol. 34/1, pp. 1-31, https://doi.org/10.1159/000277029.

[3] Wong, R. and W. Niu (2013), “Cultural difference in stereotype perceptions and performances in nonverbal deductive reasoning and creativity”, The Journal of Creative Behavior, Vol. 47, pp. 41-59, https://doi.org/10.1002/jocb.22.

[15] Zhou, J. and Y. Su (2010), “A missing piece of the puzzle: The organizational context in cultural patterns of creativity”, Management and Organization Review, Vol. 6/3, pp. 391-413, https://doi.org/10.1111/j.1740-8784.2010.00192.x.