Seven Questions about Creativity and Creative Thinking

1. What makes “creative ideas” creative?

Researchers in the field of creativity have long debated how to define creativity and how to evaluate the creative quality of ideas or products. In general, creative work is judged by the twin criteria of 1) novelty and 2) appropriateness and/or value (Runco and Jaeger, 2012[1]; Mumford, Lonergan and Scott, 2002[2]). While novelty refers to being unusual, uncommon, original, or outside established norms, appropriateness and value refer to qualities of relevance, usefulness, feasibility and efficiency. While there is relative consensus that each of these criteria are important, what is less clear is the extent to which they determine what counts as “creative”. Research has found that the criteria tend to interact with each other when judges are asked to evaluate the overall creativity of an idea (Diedrich et al., 2015[3]) and that these interactions may differ across cultures and contexts (McCarthy, Chen and McNamee, 2018[4]).

From an educational perspective, understanding what makes ideas creative can help educators recognise creativity and creative thinking in different subject areas, as well as being able to teach students how to incubate creative ideas. For instance, some research findings have suggested that it is difficult for students to think both originally and appropriately at the same time (Runco, Illies and Eisenman, 2005[5]). Others have suggested that an effective way to manage this tension, as reported by professionals in creative industries, is to focus primarily on one dimension of creativity and allow the other to emerge subsequently (Harvey and Cronin, 2020[6]). Ivcevic and Kaufman (2024[7]) suggest that the meaning of the appropriateness and value criteria in defining creativity should be adapted to the level of creativity required for a given context, with meaningfulness and relevance being the most apt interpretation of these criteria in educational contexts whereas effectiveness and impact are most important in professional and eminent contexts. For contextualising originality in educational settings, Bahar and Maker (2025[8]) suggest adopting a standards-based approach: teachers can identify specific criteria linked to learning objectives in the curriculum to evaluate the originality of students’ work, without needing to compare their students’ work to a norm group or relying on expert consensus.

What do the PISA CT Rescoring project data tell us about what makes a creative idea creative? Is every creative idea appropriate? Is it more important for students to practise strategies for coming up with original ideas or is it better for them to focus on generating ideas that have value? Do these things change depending on the context?

1.1. Are creative ideas always appropriate?

Many have argued that appropriateness is a fundamental criteria for any idea to be creative. Logically speaking, it is reasonable to expect this to be the case: an idea that is highly original but is completely inappropriate does not serve much purpose. Yet research on the relationship between appropriateness and originality has produced mixed results, with some finding an inverse relationship between the two criteria (Runco, Illies and Eisenman, 2005[5]; Runco and Charles, 1993[9]) and others noting a dialectical tension between them that must be effectively balanced to produce creative outcomes (Rastelli et al., 2022[10]; Harvey and Cronin, 2020[6]). While these studies have focused on examining this relationship based on a relatively small sample size of adult or children subjects (all N<200) in specific contexts, we can investigate this question using a much larger and representative sub-sample of 15-year-olds using the PISA CT Rescoring project data (Box 1.1).

Box 1.1. Which score data are used in this chapter?

The analyses in this chapter draw upon two types of rescored data:

• Holistic creativity scores: Valid student responses are given a score of between 1 (lowest) and 7 (highest), based on judges’ evaluations of the overall creative quality of the response. The response score reflects a relative ranking of creative quality compared to all other responses in that country-language-task group. Non-valid responses include those where all ideas are missing, aberrant or clearly inappropriate.

• Criteria scores: Each idea within a student response was scored for its appropriateness, originality and value. A cumulative idea sum score was also produced by aggregating the scores across these three criteria. Tasks classified as ‘generate diverse ideas’ tasks required students to submit multiple ideas per response.

The analyses in this chapter focus on the relative relationship between the two scoring methods – namely, which criteria drive increases in the holistic creativity scores. For more information on the scoring methods, see Annex A2.

Our findings show that, in general, as judges’ holistic creativity scores increase so does the share of responses in that score category that are appropriate (Figure 1.1) – with this pattern exhibited more acutely at the lower end of the holistic score scale and plateauing towards the upper end. At the same time, appropriate answers do not automatically translate into high creativity scores: around 15% percent of responses that received a holistic creativity score of 1 (i.e. the lowest “valid” holistic score) were fully appropriate as were almost 80% of the responses in the mid-point of the scale (score of 4). Although almost no completely inappropriate responses were considered highly creative, defined in this chapter as scoring 5 or above in holistic score, a non-negligeable proportion of these were only partially appropriate. While these results may be due, in part, to the scoring and analytical methodology applied (see Box 1.1 and Box 1.2), it also suggests that deficiencies in idea appropriateness can in some circumstances be overcome by other factors (e.g. a high originality or value factor) and still be considered highly creative. It therefore seems to confirm from these initial findings that creative work is usually appropriate – at least, partially so – but that appropriateness alone is insufficient for producing creative outcomes.

Figure 1.1. High-scoring holistic creativity responses tend to be highly appropriate

Share of responses with each appropriateness score amongst all responses (across tasks and country-language groups), per holistic score category

Note: Percentages were computed on the pooled responses across all tasks within each country-language group and then averaged across country-language groups. The percentages shown in the Figure labels refer to the share of responses in each holistic score category that scored 0 points in appropriateness. For responses to items classified as 'generate diverse ideas' items that require students to submit multiple ideas, the appropriateness scores for all ideas within the response were first averaged to give a single appropriateness score per student response. The averaged appropriateness scores were then rounded to the nearest integer (0, 1 or 2).

Source: Table B1.1

Box 1.2. Caution on interpreting the data in this chapter

Results reported in this chapter should be interpreted bearing in mind two points: (1) the principles of the two scoring methods involved; and (2) the analytical approach used in the chapter (i.e. averaging idea criteria scores in divergent thinking tasks).

Principles of the holistic and criteria-based scoring methods: relative ranking versus absolute scores

The two scoring methods compared in this chapter differ in how judges assign scores. The holistic judgement method, from which the holistic creativity scores are derived, involves ranking student responses in each country-language-task group relative to other responses within the same country-language-task group. In contrast, the criteria-based method assigns scores to student ideas independently of the quality of other ideas in the sample. These differences between the two scoring approaches may mean that, in country-language groups that have relatively more high performing students in general, it may be more likely that fully appropriate responses are assigned a low holistic creativity score than in jurisdictions where students struggle to think creatively. In other words, if most ideas in the country-language-task sample are appropriate, then some low creativity responses will likely be appropriate. Likewise, in country-language groups where students perform more poorly overall compared to other jurisdictions, it may be more likely that high holistic creativity responses are only partially appropriate. This feature should be kept in mind when directly comparing the relative proportions of appropriate responses per holistic score category across country-language groups. The same caution applies to direct comparisons involving the other score criteria (originality, value) described in this chapter.

Analytical approach: averaging criteria-based scores across ideas in divergent thinking tasks

The analytical approach employed in this chapter assumes that each idea within a student response in divergent thinking tasks contributes equally to a judge’s holistic creativity score. The criteria scores are therefore averaged across all ideas in a response before then being associated to the response holistic creativity scores. However, it may be that some ideas are more important than others for determining judges’ overall holistic response score.

The role of appropriateness in influencing whether responses are creative also seems to vary across tasks (Figure 1.2). Although there is a strong association (r=0.68) between appropriateness and holistic creativity scores, on average across country-language-task groups, this correlation is only moderate (r=0.45) for Task 2 – Illustration Titles (see Insert in the Introduction of this report for more information on the tasks included in the PISA CT Rescoring project). In some country-language groups, there is even no significant association between the two scores for Task 2. In general, the problem-solving tasks (Tasks 4-7, Figures E to H in the Introduction) tend to show a greater correlation between holistic creativity and appropriateness scores than the expressive tasks in the study (Tasks 1-3, Figures B to D in the Introduction).

Figure 1.2. Holistic creativity scores and appropriateness scores are strongly correlated

Average correlation across country-language groups, by task

Note: All correlations are statistically significant at the average country-language level. The average task results were computed by task and then averaged across the 7 tasks. For responses to items classified as 'generate diverse ideas' items, the appropriateness scores for all ideas in a response were first averaged to give a single appropriateness score per student response.

Source: Table B1.2

Why might these differences across tasks exist? Certain tasks may elicit more abstract ideas whose appropriateness are evaluated more subjectively by judges. For example, in the Illustration Titles task, students were asked to suggest three different title ideas for a fantasy countryside scene (see Figure C in Introduction). In this context, relatively abstract idea associations may be perceived as creative even if not wholly appropriate given the task places a premium on imagination. It is also reasonable that what constitutes an appropriate answer for this task – that asks students to make idea associations to elements of an already abstract and whimsical image – is more a matter of subjective interpretation than in other more objective task contexts (e.g. problem-solving tasks). That is not to say that all creative expression tasks have low appropriateness requirements simply because they require imagination: in fact, both the remaining written and visual expression tasks included in study, respectively, demonstrated a strong correlation between appropriateness scores and holistic scores (r > 0.7). Unlike the Illustration Titles tasks, both 2983 (Figure D) and Science Fair Poster (Figure B) tasks have well-defined constraints that might lead to a stricter interpretation of appropriateness: for example, in 2983, students must suggest a story idea that concretely connects to the number 2983, and in Science Fair Poster, students must design a poster for a school science fair focused on a specific theme (‘Life in Deep Space’).

1.2. Originality and value: which is more important for determining creative ideas?

In addition to being appropriate, creative ideas tend to be original and valuable. But which factor is more important, and does this change across tasks? When comparing the distribution of original and valuable responses in each holistic score category, respectively, the results are very similar (Figure 1.3). Broadly speaking, as judges’ evaluations of response originality and value increase, so do their overall holistic creativity scores. However, this trend is far less pronounced for responses with a relatively low holistic creativity score (e.g. scores 1 to 3). Contrast this pattern with the one observed in Figure 1.1: almost 61% of responses with a holistic creativity score of 3 are highly appropriate, compared to just 17% of responses in the same holistic score category that are highly original (Table B1.1 and Table B1.3). Conversely, the relative proportions of highly original or highly valuable responses increase dramatically as the holistic score increases from scores 4 to 7.

Figure 1.3. High-scoring holistic creativity responses also tend to have high originality and value

Share of responses with low, medium and high original/value scores amongst all responses (across tasks and country-language groups), per holistic score category

Note: Percentages were computed on the pooled responses across all tasks within each country-language group and then averaged across country-language groups. The percentages shown in the Figure labels refer to the share of responses in each holistic score category that were low original or value responses, respectively. 'Low’ originality/value responses are those with an average idea criteria score of less than 1 point, 'medium’ originality/value responses are those with an average idea criteria score of at least 1 point but less than 2 points, and 'high' originality/value responses are those with an average idea criteria score of 2 or more points. For 'generate diverse ideas' items that contain multiple ideas per response, the originality/value scores for all ideas were first averaged to give a single criteria score per student response. All originality/value scores for non-appropriate ideas were recoded to 0 for these analyses.

Source: Table B1.3 and Table B1.4

How should these results be interpreted? While appropriateness might act as an “entry barrier” for coming up with a creative idea, once this criterion is partially achieved it seems to add relatively little to a response’s holistic creativity – and indeed, many non-creative ideas can be highly appropriate. In contrast, relatively few low creativity ideas are either highly original or highly valuable. For the minority that are, their relatively low holistic creativity score can likely be explained by the absence of the other factor. For example, amongst highly original responses, those that overall have a low holistic creativity score record significantly lower value scores on average across country-language groups (mean difference of 0.7 score points) (Table B1.5). While the same is true for appropriateness, the differential is much smaller (close to 0.2 score points). A similar mean difference in originality scores is observed amongst high value responses that have a low holistic creativity score (mean gap of 0.7 score points), with an even smaller gap to the mean appropriateness scores of those responses (difference of 0.1 score points). While the latter is still significant on average across country-language groups, this only holds in a third of the individual country-language groups (Table B1.6).

Despite their similarities, can we identify differences in the patterns of original and valuable responses at different holistic creativity scores? When highly and moderately original responses are considered together, the increase in their relative proportion within each holistic creativity score increases quite consistently as the holistic scores increase (see the slope of the line in Figure 1.3). When looking at the same pattern amongst at least moderately valuable responses, the relationship is slightly less linear: while the share of at least somewhat valuable responses increases at a larger rate amongst low holistic creativity responses (an increase of around 50 percentage points from holistic score category 1 to 3), the proportion plateaus around the mid-range with over 93% of all responses at holistic score 4 or above considered at least somewhat valuable, and increasing by just 5 percentage points to 98% at holistic score 7 (Table B1.3). In comparison, only 85% of responses at holistic score 4 are at least moderately original, but this rises by 14 percentage points to 99% of all responses awarded the top holistic score of 7 (Table B1.4).

Overall, the PISA CT Rescoring study findings seem to confirm that both originality and value criteria are important for defining whether an idea is creative but with two caveats: 1) that the holistic creativity scores are more sensitive to response value at the lower end of the score spectrum than at the upper end; and 2) that it is more likely that the most creative responses are highly original than they are highly valuable. These findings are also implied when looking at the mean originality and value scores of responses in each holistic category: response value scores are greater amongst those with holistic creativity scores 5 or below, but response originality scores are greater amongst those with holistic creativity scores of 6 or 7 (Figure 1.4). This result might also be explained by the relative difficulty of achieving each criteria – in other words, it might be easier to suggest an effective idea than an original one – but it also suggests that originality matters slightly more for distinguishing somewhat creative ideas from very creative ones.

Figure 1.4. The most creative answers are more original than they are valuable

Mean response originality and value scores (across tasks and country-language groups), by holistic score category

Note: Mean original and value scores were computed on the pooled responses across all tasks within each country-language group and then averaged across country-language groups. For 'generate diverse ideas' items that contain multiple ideas per response, the originality/value scores for all ideas were averaged to give a single criteria score per student response. All originality/value scores for non-appropriate ideas were coded as 0.

Source: Table B1.7 and Table B1.8

1.2.1. Differences across tasks

Are originality and value similarly associated with creative outcomes in different types of task? As expected, both response originality and value scores are significantly and strongly correlated with response holistic creativity scores on average across tasks and country-language groups (r = 0.73 for originality and r = 0.72 for value; Table B1.9 and Table B1.10). Within country-language groups, the two criteria tend to demonstrate an equally strong association with the holistic creativity scores – though there are some exceptions. For example, in Canada (French provinces), originality is more strongly correlated with holistic creativity scores (difference of 0.2 compared to the equivalent correlation for value) but in Greece the opposite is observed.

Across the seven different tasks included in the PISA CT Rescoring project, on average across country-language groups, these relationships are significant and moderate to strong (range of 0.65 to 0.79). The task that records the weakest correlation between holistic score and originality score is Illustration Titles (a written expression task; see Figure C in the Introduction) (r=0.66) and the task that records the strongest correlation between the two scores is The Exhibit (an engineering problem-solving task; see Figure H in the Introduction) (r=0.78). At first glance, these results might seem unexpected: it would be reasonable to expect originality to be more influential in determining creativity scores in more imaginative task contexts and less so in narrower problem-solving tasks; yet it may be precisely for that reason that original solutions in The Exhibit task are perceived as more creative. It is relatively easier to come up an uncommon or imaginative idea association in an open, abstract task compared to one that requires a functional solution. That is not to say that originality is not important in expressive tasks, but rather, it may be more common for student responses to be imaginative, and therefore originality is less of a feature that separates poor ideas from average or highly creative ideas. We can infer the relative difficulty of the tasks with respect to each criteria by looking at the mean criteria-based scores per task (Tables B1.11-B1.17). The mean sum score and originality score for Illustration Titles is significantly higher than for all other tasks included in the study, on average across countries, with the opposite being true for The Exhibit. For example, students score over 1.5 points more in total and over 0.5 points more in originality in the former, than they do in the latter (Table B1.12 and Table B1.17).

A similar effect of task difficulty could explain the relationship between value and overall holistic scores. In our study, response value scores were least correlated with holistic creativity scores in the Food Waste (social problem-solving) task (r=0.6) and most strongly correlated in The Exhibit task (r=0.79). Again, it may be intuitive to expect value would be most strongly correlated with problem-solving tasks regardless of task difficulty, but this is not what we observe. The Food Waste item asks students to think about an everyday problem that is likely to be a familiar context and one for which students may already have some awareness or lived experience; as such, students may find it easier to suggest ideas that would be functional and effective. In fact, out of the four problem-solving items included in the PISA CT Rescoring project, students are most successful overall, and in coming up with valuable ideas, in the Food Waste task (Figure F in the Introduction) (Tables B1.14-B1.17). In contrast, The Exhibit task is less familiar and potentially more demanding from a knowledge perspective (e.g. knowledge of basic physics, engineering or biological principles); as a result, we observe that it is the problem-solving task in which students are least successful overall.

1.3. Creative ideas: greater than the sum of their parts?

While most researchers agree that appropriateness, originality and value are all fundamental characteristics of creative ideas, these criteria alone do not fully explain what makes creative ideas creative. When considered together, these criteria scores only explain around two-thirds (66%) of the total variance in holistic creativity scores, on average across country-language-task groups (Table B1.18). These findings align with previous research that has demonstrated that creativity scores are influenced by more factors than originality and quality (Long, 2014[11]), such that the creative value of the whole might be considered greater than the sum of its parts. While criteria-based (sometimes referred to as analytic) methods can provide a more easily understood and replicable formula for evaluating creative ideas in a measurement context, they may fail to capture the full creative quality of ideas in the way that more holistic scoring methods are able to.

There are many other factors – individual-, task- and judge-related – that might determine how creative ideas are formed, perceived and evaluated. For instance, domain-specific theories of creativity assert that domain-relevant knowledge and skills are important determinants of successful creative thinking processes. While domain readiness may certainly help individuals to come up with ideas of value, and perhaps even original ideas, these scoring criteria may fail to fully capture the role of relevant knowledge and skills in supporting creative thinking. Runco, Illies and Eisenman (2005[5]) also demonstrated how task instructions and context can differentially influence individuals’ success in generating original and flexible ideas, and the analysis in this chapter also suggests that task difficulty plays a mediating role in how judges differentially perceive the importance of the same criteria in different tasks – even within the same broader domain context. Several studies have also demonstrated that rater expertise (Kaufman et al., 2008[12]; Kaufman, Baer and Cole, 2009[13]; Long, 2014[11]), subjective and socio-cultural rater bias (Lloyd-Cox, Pickering and Bhattcharya, 2022[14]; Kaufman et al., 2010[15]), and ambiguity in how the scoring criteria are defined for given tasks (Reiter-Palmon et al., 2009[16]) can influence how judges understand and evaluate creative ideas. Assessment choices, including how to operationalise scoring approaches, should be mindful of these factors and take into account the nature of the assessment tasks and the dimension(s) of interest of the construct (Reiter-Palmon, Forthmann and Barbot, 2019[17]).

Key findings and implications

• Creative ideas are not always highly appropriate, highly original and highly valuable – but they very likely are.

  • Over 88% of highly creative ideas are appropriate, and over 75% are highly original and highly valuable.

• The PISA CT Rescoring project shows that it is possible to evaluate differences in student idea quality across the criteria of appropriateness, originality and value in a variety of everyday imaginative and problem-solving tasks.

  • Expectations of what appropriateness, originality and value mean in educational contexts should be set at developmentally appropriate levels and should focus on identifying evidence of “little-c” creativity rather than the expression of creative genius. For example, focusing on relevance and meaningfulness in students’ ideas might be the most apt interpretation of the appropriateness and value criteria (Ivcevic and Kaufman, 2024[7]). For originality, focusing on uncommon ways that students demonstrate their understanding of specific learning objectives grounded in curricular expectations may be the most instructionally relevant approach (Bahar and Maker, 2025[8]).

• The relative importance of appropriateness, originality and value varies across task types, though they remain important in all contexts.

  • Task demands, constraints and difficulty appear to affect how much each criteria determines the overall creativity of a response.

• Appropriateness, originality and value do not fully explain what makes creative ideas creative, accounting for around two-thirds of the variation in holistic creativity scores on average. Other factors also play a role in determining the extent to which individuals successfully achieve creative outcomes across different tasks. These factors might include:

  • Domain-relevant knowledge and skills that support students’ contextualisation of appropriate, original and valuable ideas;

  • Students’ background, characteristics, or attitudes and traits that might support or hinder the development of creative thinking skills. Some of these are explored further in subsequent chapters in this report (e.g. attitudes and traits in Chapter 3; gender in Chapter 6; and socio-economic status in Chapter 7).

References

[8] Bahar, K. and C. Maker (2025), “A new perspective on scoring children’s originality: a standards-based criterion-referenced assessment approach”, Frontiers in Psychology, Vol. 16, https://doi.org/10.3389/fpsyg.2025.1545396.

[3] Diedrich, J. et al. (2015), “Are creative ideas novel and useful?”, Psychology of Aesthetics, Creativity, and the Arts, Vol. 9/1, pp. 35-40, https://doi.org/10.1037/a0038688.

[6] Harvey, S. and M. Cronin (2020), Make it work: The importance of usefulness and appropriateness for creativity, https://doi.org/10.5465/AMBPP.2020.20251abstract.

[7] Ivcevic, Z. and J. Kaufman (2024), “Inside the box: Considering the role of appropriateness for creativity”, Psychology of Aesthetics, Creativity, and the Arts, https://doi.org/10.1037/aca0000703.

[13] Kaufman, J., J. Baer and J. Cole (2009), “Expertise, domains, and the consensual assessment technique”, Journal of Creative Behavior, Vol. 43/4, pp. 223-233, https://doi.org/10.1002/j.2162-6057.2009.tb01316.x.

[12] Kaufman, J. et al. (2008), “A comparison of expert and nonexpert raters using the consensual assessment technique”, Creativity Research Journal, Vol. 20/2, pp. 171-178, https://doi.org/10.1080/10400410802059929.

[15] Kaufman, J. et al. (2010), “In the eye of the beholder: Differences across ethnicity and gender in evaluating creative work”, Journal of Applied Social Psychology, Vol. 40/2, pp. 496-511, https://doi.org/10.1111/j.1559-1816.2009.00584.x.

[14] Lloyd-Cox, J., A. Pickering and J. Bhattcharya (2022), “Evaluating creativity: How idea context and rater personality affect considerations of novelty and usefulness”, Creativity Research Journal, Vol. 34/4, pp. 373-390.

[11] Long, H. (2014), “More than appropriateness and novelty: Judges’ criteria of assessing creative products in science tasks”, Thinking Skills and Creativity, Vol. 13, pp. 183-194, https://doi.org/10.1016/j.tsc.2014.05.002.

[4] McCarthy, M., C. Chen and R. McNamee (2018), “Novelty and usefulness trade-off: Cultural cognitive differences and creative idea evaluation”, Journal of Cross-Cultural Psychology, Vol. 49/2, pp. 171-198, https://doi.org/10.1177/0022022116680479.

[2] Mumford, M., D. Lonergan and G. Scott (2002), “Evaluating creative ideas: Processes, standards and context”, Inquiry: Critical Thinking Across Disciplines, Vol. 22/1, pp. 21-30, https://doi.org/10.5840/inquiryctnews20022213.

[10] Rastelli, C. et al. (2022), “Balancing novelty and appropriateness leads to creative associations in children.”, PNAS Nexus, Vol. 1/5, https://doi.org/10.1093/pnasnexus/pgac273.

[17] Reiter-Palmon, R., B. Forthmann and B. Barbot (2019), “Scoring divergent thinking tests: A review and systematic framework”, The Psychology of Aesthetics, Creativity, and the Arts, Vol. 13/2, pp. 144-152, https://doi.org/10.1037/aca0000227.

[16] Reiter-Palmon, R. et al. (2009), “Creativity and domain specificity: The effect of task type on multiple indexes of creative problem-solving.”, Psychology of Aesthetics, Creativity, and the Arts, Vol. 3/2, pp. 73-80, https://doi.org/10.1037/a0013410.

[9] Runco, M. and R. Charles (1993), “Judgements of originality and appropriateness as predictors of creativity”, Personality and Individual Differences, Vol. 15/4, pp. 537-546, https://doi.org/10.1016/0191-8869(93)90337-3.

[5] Runco, M., J. Illies and R. Eisenman (2005), “Creativity, originality and appropriateness: What do explicit instructions tell us about their relationships?”, Journal of Creative Behavior, Vol. 39/2, https://doi.org/10.1002/j.2162-6057.2005.tb01255.x.

[1] Runco, M. and G. Jaeger (2012), “The standard definition of creativity”, Creativity Research Journal, Vol. 24/1, pp. 92-96, https://doi.org/10.1080/10400419.2012.650092.