This chapter describes the proficiency levels used to report the results of the 2023 Survey of Adult Skills. It also provides information on the reporting of results for participating countries and economies that conducted the survey in more than one language.
Table of contents
Share
Facebook
Twitter
LinkedIn
3. Reporting the results of the 2023 Survey of Adult Skills
Copy link to 3. Reporting the results of the 2023 Survey of Adult SkillsAbstract
This chapter describes how the results from the 2023 Survey of Adult Skills are reported. It shows how the literacy, numeracy and adaptive problem solving items used in the assessment are categorised according to difficulty; the cognitive processes required of adults to answer the questions; the real-life contexts in which such problems/questions may arise; and the way in which items are delivered to respondents, as described in their respective assessment frameworks (OECD, 2021[1]).
Proficiency levels for each domain describe in detail what adults can do when their score is within a certain range. This chapter presents how the proficiency levels were constructed and how they relate to proficiency scores. It concludes with information about the languages in which the survey was conducted and the approach to reporting in countries and economies where the assessment was delivered in more than one language.
Proficiency scales
Copy link to Proficiency scalesIn each of the three domains assessed, proficiency is considered a continuum of ability involving the mastery of information-processing skills that allow tasks of increasing complexity to be solved. The proficiency of respondents (their assessment results) and the difficulty of the assessment items are expressed on the same 500-point scale. At each point on the scale, an individual with a proficiency score of that value has a 67% chance of successfully completing test items given that difficulty value.1 This individual will also have a lower likelihood of being able to complete more difficult items (those with higher values on the scale), and a greater likelihood of successfully completing easier items (those with lower values on the scale).
For instance, Table 3.1 shows the probability of a person who scores 300 points on the literacy scale completing tasks of the same, greater and lesser difficulty. This person will successfully complete items whose difficulty matches their proficiency (300 points on the literacy scale) 67% of the time; their chances of completing easier items (with a difficulty value of 250) are as high as 95%, but the chances of correctly answering more difficult items (with a difficulty value of 350) drop to 28%.
Table 3.1. Probability of successfully completing items of varying difficulty for a person scoring 300 on the literacy scale
Copy link to Table 3.1. Probability of successfully completing items of varying difficulty for a person scoring 300 on the literacy scale|
Item difficulty (literacy scale) |
||||
|---|---|---|---|---|
|
200 |
250 |
300 |
350 |
|
|
Probability of success |
0.97 |
0.95 |
0.67 |
0.28 |
Proficiency levels
Copy link to Proficiency levelsThe proficiency scale in each of the domains assessed can be described in relation to the items that are located at the different points on the scale according to their difficulty. Annex A presents the characteristics of all items used in the 2023 Survey of Adults Skills to assess literacy, numeracy and adaptive problem solving. These characteristics include the item’s difficulty as well as other key features, such as the cognitive processes needed to engage with it and the context in which the task was framed.
The scales have been divided into “proficiency levels” defined by particular score-point thresholds. Six proficiency levels are defined for literacy and numeracy (below Level 1 and Levels 1 to 5) and five for adaptive problem solving (below Level 1 and Levels 1 to 4). To help interpret of the results, the experts who designed the assessments have developed a summary description of the characteristics of the types of tasks that adults at a particular level can successfully complete. In other words, they offer a summary of what adults with proficiency scores within a range in a particular domain can do.
Except for the lowest level (below Level 1), a person scoring at the bottom of the range defining a given level has a 50% chance of successfully completing the tasks located at that level. This probability will, of course, vary with the difficulty of each particular item, as illustrated in Table 3.1. For example, a person with a score at the bottom of Level 2 would correctly answer about half of the items in a hypothetical assessment containing only items of Level 2 difficulty, while a person whose proficiency is at the top of Level 2 could correctly answer many more. An individual with a proficiency score around the mid-point of Level 2 will successfully answer about 67% of the items in this hypothetical assessment.
Literacy and numeracy
Table 3.2 and Table 3.3 contain the score-point thresholds for each of the six levels for literacy, and numeracy, and the descriptors of what respondents located at each of the levels are able to do. Box 3.1 and Box 3.2 provide illustrative item maps for literacy and numeracy for one item in each proficiency level.
In the case of literacy and numeracy, the score-point ranges associated with each level are the same as those used in the first cycle of the Survey of Adult Skills. These in turn were based on the International Adult Literacy Survey (IALS) and the Adult Literacy and Life Skills Survey (ALL) for document and prose literacy and ALL for numeracy. However, the descriptors of the proficiency levels in the 2023 Survey of Adult Skills differ from those of previous surveys because the underlying frameworks have evolved. This evolution does not prevent results from being compared over time, as strong links (both conceptually and through the presence of many common assessment items) remain between the different surveys (see Chapter 6).
Table 3.2. Proficiency levels: Literacy
Copy link to Table 3.2. Proficiency levels: Literacy|
Level |
Score range |
Literacy |
|---|---|---|
|
5 |
Equal to or higher than 376 points |
At Level 5, the assessment provides no direct information on what adults can do. This is mostly because feasibility concerns (especially with respect to testing time) precluded the inclusion of highly difficult tasks involving complex interrelated goal structures, very long or complex document sets, or tools containing highly complex texts (e.g. extensive catalogues, complex menu structures, or lists of unstructured results from search engines), which require advanced skills to access and process the information they contain. These tasks, however, form part of the construct of literacy in today's world, and future assessments aiming at a better coverage of the upper end of the proficiency scale may seek to include testing units tapping into literacy skills at Level 5. The characteristics of the most difficult tasks at Level 4 offer some insight into what might constitute proficiency at Level 5. Adults at Level 5 may be able to reason about the task itself, setting up reading goals based on complex and implicit requests. They can presumably search for and integrate information across multiple, dense texts containing distracting information in prominent positions. They are able to construct syntheses of similar and contrasting ideas or points of view; or evaluate evidence-based arguments and the reliability of unfamiliar information sources. Tasks at Level 5 may also require the application and evaluation of abstract ideas and relationships. Evaluating the reliability of evidentiary sources and selecting information which is not just topically relevant but also trustworthy may be key to achievement. |
|
4 |
326 to less than 376 points |
At Level 4, adults can read long and dense texts presented on multiple pages in order to complete tasks that involve accessing, understanding, evaluating and reflecting on the text(s) contents and sources across multiple processing cycles. Adults at this level can infer what the task is asking for based on complex or implicit statements. Successful task completion often requires the production of knowledge-based inferences. Texts and tasks at Level 4 may deal with abstract and unfamiliar situations. They often feature both lengthy content and a large amount of distracting information, which is sometimes as prominent as the information required to complete the task. At this level, adults are able to reason based on intrinsically complex questions that share only indirect matches with the text content, and/or require taking into consideration several pieces of information dispersed throughout the materials. Tasks may require evaluating subtle evidence claims or persuasive discourse relationships. Conditional information is frequently present in tasks at this level and must be taken into consideration by the respondent. Response modes may involve assessing or sorting complex assertions. |
|
3 |
276 to less than 326 points |
Adults at Level 3 are able to construct meaning across larger chunks of text or perform multi-step operations in order to identify and formulate responses. They can identify, interpret or evaluate one or more pieces of information, often employing varying levels of inference. They can combine various processes (accessing, understanding and evaluating) if the task requires. Adults at this level can compare and evaluate multiple pieces of information from the text(s) based on their relevance or credibility. Texts at this level are often dense or lengthy, including continuous, non-continuous and mixed texts. Information may be distributed across multiple pages, sometimes arising from multiple sources that provide conflicting information. Understanding rhetorical structures and text signals becomes more central to successfully completing tasks, especially when dealing with complex digital texts that require navigation. The texts may include specific, possibly unfamiliar, vocabulary and argumentative structures. Competing information is often present and sometimes salient, though no more than the target information. Tasks require the respondent to identify, interpret or evaluate one or more pieces of information, and often require varying levels of inference. Tasks at Level 3 also often demand that the respondent disregard irrelevant or inappropriate text content to answer accurately. The most complex tasks at this level include lengthy or complex questions requiring the identification of multiple criteria, without clear guidance regarding what has to be done. |
|
2 |
226 to less than 276 points |
At Level 2, adults are able to access and understand information in longer texts with some distracting information. They can navigate within simple multi-page digital texts to access and identify target information from various parts of the text. They can understand by paraphrasing or making inferences, based on single or adjacent pieces of information. Adults at Level 2 can consider more than one criterion or constraint in selecting or generating a response. The texts at this level can include multiple paragraphs distributed over one long or a few short pages, including simple websites. Non-continuous texts may feature a two-dimension table or a simple flow diagram. Access to target information may require the use of signalling or navigation devices typical of longer print or digital texts. The texts may include some distracting information. Tasks and texts at this level sometimes deal with specific, possibly unfamiliar, situations. Tasks require respondents to perform indirect matches between the text and content information, sometimes based on lengthy instructions. Some task statements provide little guidance about how to perform the task. Task achievement often requires the test taker to either reason about one piece of information or to gather information across multiple processing cycles. |
|
1 |
176 to less than 226 points |
Adults at Level 1 are able to locate information on a page of text, find a relevant link from a website, and identify relevant text among multiple options when the relevant information is explicitly cued. They can understand the meaning of short texts, as well as the organisation of lists or multiple sections within a single page. The texts at Level 1 may be continuous, non-continuous or mixed and pertain to printed or digital environments. They typically include a single page with up to a few hundred words and little or no distracting information. Non-continuous texts may have a list structure (such as a web search engine results page) or include a small number of independent sections, possibly with pictorial illustrations or simple diagrams. Tasks at Level 1 involve simple questions providing some guidance as to what needs to be done and a single processing step. There is a direct, fairly obvious match between the question and target information in the text, although some tasks may require the examination of more than one piece of information. |
|
Below Level 1 |
Less than 176 points |
Most adults below Level 1 are able to process meaning at the sentence level. Given a series of sentences that increase in complexity, they can tell if a sentence does or does not make sense either in terms of plausibility in the real world (i.e. sentences describing events that can versus those that cannot happen), or in terms of the internal logic of the sentence (i.e. sentences that are meaningful versus those that are not). Most adults at this level are also able to read short, simple paragraphs and, at certain points in the text, tell which of two words makes the sentence meaningful and consistent with the rest of the passage. Finally, they can access single words or numbers in very short texts in order to answer simple and explicit questions. The texts below Level 1 are very short and include no or just a few familiar structuring devices such as titles or paragraph headers. They do not include any distracting information nor navigation devices specific to digital texts (e.g. menus, links or tabs). Tasks below Level 1 are simple and very explicit regarding what to do and how to do it. These tasks only require understanding at the sentence level or across two simple adjacent sentences. When the text involves more than one sentence, the task merely requires dealing with target information in the form of a single word or phrase. |
Box 3.1. Illustrative item map: Literacy
Copy link to Box 3.1. Illustrative item map: LiteracyIn addition to the released items (available at https://www.oecd.org/en/about/programmes/piaac/piaac-released-items.html), the selected items below illustrate the types of tasks within each level by describing non-released items and mapping them to the literacy framework.
Banking (C503P001)
difficulty: below Level 1 (161 points)
cognitive process: accessing text (locate information within texts)
source: single
pages: single
context: personal
Based on an image of a very brief text message from a bank, respondents are asked to identify how long a provided identification code is valid. There is some distracting information in the message, with “code” or “identification code” being present in three locations. However, the very limited amount of text plus the fact that a time is listed in only a single location, makes the correct response easy to locate.
BiciMAD (C509P003)
difficulty: Level 1 (215 points)
cognitive process: accessing text (locate information within texts)
source: single
pages: single
context: community and citizenship
This task is at the higher end of Level 1 and is based on a short section of a brochure with information about an electric bicycle rental programme. The brochure explains that bicycles are stored at stations around the city, with a number of bases at each station. Respondents are asked to identify the number of bases in the city. That information is explicitly listed in the brochure, but locating the information is a bit more complex for several reasons: the text is longer and the context may be unfamiliar to some respondents, there is no explicit structure (such as headings) to signal where the requested information is located, and there are several other numbers in the text that present distracting information.
App Comparison (C511P004)
difficulty: Level 2 (230 points)
cognitive process: evaluating
source: multiple
pages: single
context: personal
The stimulus for this task consists of a web page with reviews of several apps for storing and organising photos posted by five individuals. Respondents are asked how one of those reviews could be made more credible. This item is the easiest “evaluate” item in the assessment. The task requires respondents to go beyond understanding the content of the specified review and to think about how each of four plausible options would or would not improve credibility.
Recycling Guide (C517P004)
difficulty: Level 2 (270 points)
cognitive process: accessing text (identify relevant text)
source: multiple
pages: multiple
context: community and citizenship
This access task is situated in a simulated website consisting of six pages. The number of pages and the dynamic nature of the display adds to the complexity of the task because the respondent must identify the relevant page to locate the requested information. The task opens on the home page of a community recycling site and respondents are asked how residents can order a recycling cart. Respondents need to navigate to the Frequently Asked Questions page, where the information is provided. The names of the pages do not provide strong clues about where to look. Once the correct page is located, there are several mentions of recycling carts, creating distracting information. Together, these features make this one of the most difficult items in Level 2.
Online Learning (C516P004)
difficulty: Level 3 (325 points)
cognitive process: evaluating
source: multiple
pages: multiple
context: education and training
This task includes two text sources. The first is a multi-paragraph narrative written by an employee at a manufacturing company describing his experience taking an online training course provided by the company. The second is an advertisement for online worker training programmes. The advertisement includes a list of several advantages of online learning. Respondents are asked to identify the advantages that are supported by the training experiences described by the employee in his narrative. There are no direct statements in the narrative that exactly match the items in the list of advantages in the advertisement, so making those connections requires a deeper understanding of the employee’s statements. This factor, along with the need to integrate and compare information across two texts, makes this the most difficult Level 3 item in the assessment.
Desk Cycling (C514P004)
difficulty: Level 4 (362 points)
cognitive process: understanding (inferential)
source: multiple
pages: multiple
context: work and occupation
The stimulus for this task includes two text sources: an article about desk cycling (i.e. using a small pedalling device that fits under a desk and can be used for exercise while working) and a blog where four users share their experience using this device. Respondents are asked to identify a statement on which one of the bloggers (identified in the item) and the study director quoted in the article would agree. Some of the provided options are true for one individual but not both, or are true for a different blogger. Respondents must integrate and compare information across the two sources, making this one of the more difficult items in the assessment.
Table 3.3. Proficiency levels: Numeracy
Copy link to Table 3.3. Proficiency levels: Numeracy|
Level |
Score range |
Numeracy |
|---|---|---|
|
5 |
Equal to or higher than 376 points |
Adults at Level 5 can use and apply problem-solving strategies to analyse, evaluate, reason and critically reflect on complex and formal mathematical information, including dynamic representations. They demonstrate an understanding of statistical concepts and can critically reflect on whether a data set can be used to support or refute a claim. Adults at this level can determine the most appropriate graphical representation for relational data sets. |
|
4 |
326 to less than 376 points |
Adults at Level 4 can use and apply a range of problem-solving strategies to access, analyse, reason and critically reflect on and evaluate a broad range of mathematical information that is often presented in unfamiliar contexts. Such information may not be presented in an explicit manner. Adults at this level can devise and implement strategies to solve multi-step problems. This may involve reasoning about how to integrate concepts from different mathematical content areas or applying more complex and formal mathematical procedures. Adults at this level can:
|
|
3 |
276 to less than 326 points |
Adults at Level 3 can access, act on, use, reflect on and evaluate authentic mathematical contexts. This requires making judgements about how to use the given information when developing a solution to a problem. The mathematical information may be less explicit, embedded in contexts that are not always commonplace, and use representations and terminology that are more formal and involve greater complexity. Adults at this level can complete tasks where mathematical processes require the application of two or more steps and where multiple conditions need to be satisfied. Tasks may also require the use, integration or manipulation of multiple data sources in order to undertake the mathematical analyses necessary for the specific task. Adults at this level can:
|
|
2 |
226 to less than 276 points |
Adults at Level 2 can access, act on and use mathematical information and evaluate simple claims, in tasks set in a variety of authentic contexts. They are able to interpret and use information presented in slightly more complex forms (e.g. doughnut charts, stacked bar graphs or linear scales) that includes more formal terminology and more distracting information. Adults at this level can carry out multi-step mathematical processes. Adults at this level can:
|
|
1 |
176 to less than 226 points |
Adults at Level 1 demonstrate number sense involving whole numbers, decimals, and common fractions and percentages. They can access, act on and use mathematical information located in slightly more complex representations set in authentic contexts where the mathematical content is explicit and uses informal mathematical terminology with little text and minimal distracting information. They can devise simple strategies using one or two steps to determine the solution. Adults at this level can:
|
|
Below Level 1 |
Less than 176 points |
Adults performing below Level 1 demonstrate elementary whole number sense and can access and use mathematical knowledge to solve single-step problems, where the information is presented using images or simple structured information set in authentic, commonplace contexts with little or no text or distracting information. The mathematical content is non-formal and explicit. Adults at this level can:
|
Box 3.2. Illustrative item map: Numeracy
Copy link to Box 3.2. Illustrative item map: NumeracyIn addition to the released items (available at https://www.oecd.org/en/about/programmes/piaac/piaac-released-items.html), the selected set of items below illustrates the types of tasks within each level by describing non-released items and mapping them to the numeracy framework.
Moving (C815P002)
difficulty: below Level 1 (174 points)
cognitive process: accessing and assessing situations mathematically
representation: images of objects
content: quantity and number
context: work
In this task, a note containing the number of moving boxes needed for each of five rooms is shown, and respondents are asked to calculate the total number of boxes needed. The required operation of addition is signalled by the use of “total” in the question, and there are no extraneous numbers or distracting information included in the stimulus. The numbers that must be added are all whole numbers less than or equal to ten. These features of the stimulus and question make this one of the easiest tasks on the PIAAC numeracy scale.
Bike Tour (C801P001)
difficulty: Level 1 (198 points)
cognitive process: acting on and using mathematics
representation: text or symbols
content: quantity and number
context: personal
In this task, an advertisement for a bike tour is presented that shows the number of kilometres that bikers will ride each day on a three-day tour. Respondents are asked to determine the length, in kilometres, of the “complete tour”. However, nowhere in the advertisement is the phrase “complete tour” used, nor is a total distance given. Hence, respondents need to act on the information provided and decide how to use the advertisement to answer the question. The numbers in the problem are all two-digit whole numbers.
Expenses (C811P002)
difficulty: Level 2 (229 points)
cognitive process: accessing and assessing situations mathematically
representation: dynamic applications
content: data and chance
context: personal
This task is based on a simulated app used to monitor monthly expenses in six categories. At the start of the task, the only visible information in the app is buttons for two specific months. Based on the information in the question, respondents must select the correct month to view a doughnut chart of the expenses for that month. Tapping on any segment of the doughnut chart displays the amount spent in that category as well as the percentage of the monthly expenses that the category represents. Respondents are asked to identify the three categories with the highest spending and then use the drag-and-drop function to put them in order from highest to third highest in terms of spending.
Zoo Visitors (C833P002)
difficulty: Level 3 (280 points)
cognitive process: evaluating, critically reflecting and making judgements
representation: structured information
content: quantity and number
context: work
The stimulus for this task is a table showing the average number of visitors that arrive at each of the four entrance gates during each hour that a zoo is open. Respondents are asked to determine in which time slot it would be most helpful to have additional staff members in order to reduce waiting time at the entrances. Actual waiting times are not given in the table, so respondents need to understand how these data are related to wait times and then devise a strategy that can be used to complete the task. There is a lot of information to process in the table, and the total number of visitors during several of the time periods is very similar, which adds to the complexity of choosing the best strategy in this task.
Electric Cars (C810P002)
difficulty: Level 4 (348 points)
cognitive process: evaluating, critically reflecting and making judgements
representation: structured information
content: change and relationships
context: social/community
The stimulus for this task includes two bar graphs: one showing the purchase price for three cars (one diesel, one petrol and one electric), and another graph showing the average cost per kilometre to drive each of these car types. Respondents are asked to calculate how long it would take to recoup the higher purchase price of an electric car as compared to the lower-priced diesel car, based on the different fuel costs for each type of car and the average number of kilometres she drives each year. Respondents must devise a strategy for integrating the information, and several computational steps are required to complete the task. The data in each graph are monetary amounts while the answer needs to be in terms of years and months, which adds to the complexity of the task.
Flying Hours (C812P003)
difficulty: Level 5 (496 points)
cognitive process: accessing and assessing situations mathematically
representation: dynamic application
content: data and chance
context: social/community
The stimulus for this task is a table of data showing the average number of flying hours and average monthly salary per pilot for ten different airlines. Respondents are asked to identify the best type of graph to display the relationship between these variables. Tapping on an image of each of the four types of graphs given as the options shows the data displayed in that type of graph. However, only one type of graph shows the correct relational nature of these data, so the item requires a relatively sophisticated understanding of statistical representations.
Table 3.4 and Table 3.5 expand on the simple example made in Table 3.1. They show how the probability of correctly answering literacy and numeracy items at different levels of difficulty varies with the proficiency of respondents. An adult with a proficiency score of 300 in literacy (i.e. the mid-point of Level 3) has a 68% chance of successfully completing items of Level 3 difficulty, but only has a 29% chance of providing a correct answer to items of Level 4 difficulty. The probability of successfully completing items of Level 2 difficulty, on the other hand, is as high as 90%.
Table 3.4. Probability of successfully completing items of varying difficulty levels by proficiency score: Literacy
Copy link to Table 3.4. Probability of successfully completing items of varying difficulty levels by proficiency score: Literacy|
Item difficulty |
Proficiency score |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
150 |
175 |
200 |
225 |
250 |
275 |
300 |
325 |
350 |
375 |
400 |
425 |
|
|
Level 4 |
0.00 |
0.01 |
0.02 |
0.04 |
0.07 |
0.14 |
0.26 |
0.43 |
0.61 |
0.77 |
0.88 |
0.94 |
|
Level 3 |
0.07 |
0.12 |
0.19 |
0.29 |
0.40 |
0.54 |
0.66 |
0.77 |
0.85 |
0.91 |
0.94 |
0.97 |
|
Level 2 |
0.24 |
0.34 |
0.45 |
0.56 |
0.67 |
0.76 |
0.83 |
0.89 |
0.93 |
0.95 |
0.97 |
0.98 |
|
Level 1 |
0.32 |
0.49 |
0.67 |
0.81 |
0.90 |
0.95 |
0.97 |
0.99 |
0.99 |
1.00 |
1.00 |
1.00 |
Table 3.5. Probability of successfully completing items of varying difficulty levels by proficiency score: Numeracy
Copy link to Table 3.5. Probability of successfully completing items of varying difficulty levels by proficiency score: Numeracy|
Item difficulty |
Proficiency score |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
150 |
175 |
200 |
225 |
250 |
275 |
300 |
325 |
350 |
375 |
400 |
425 |
|
|
Level 4 |
0.04 |
0.07 |
0.10 |
0.15 |
0.23 |
0.32 |
0.43 |
0.55 |
0.66 |
0.76 |
0.84 |
0.89 |
|
Level 3 |
0.08 |
0.13 |
0.20 |
0.29 |
0.41 |
0.54 |
0.67 |
0.77 |
0.85 |
0.91 |
0.94 |
0.96 |
|
Level 2 |
0.05 |
0.12 |
0.26 |
0.46 |
0.68 |
0.84 |
0.93 |
0.97 |
0.99 |
1.00 |
1.00 |
1.00 |
|
Level 1 |
0.50 |
0.59 |
0.67 |
0.75 |
0.81 |
0.86 |
0.90 |
0.93 |
0.95 |
0.96 |
0.97 |
0.98 |
Adaptive problem solving
The adaptive problem solving scale was divided into four proficiency levels. Table 3.6 presents the score-point ranges defining each level and the descriptors of what respondents located at each of the levels are able to do. Box 3.3 provides an illustrative item map for adaptive problem solving.
Table 3.6. Proficiency levels: Adaptive problem solving
Copy link to Table 3.6. Proficiency levels: Adaptive problem solving|
Level |
Score range |
The types of tasks completed successfully at each level of proficiency |
|---|---|---|
|
Level 4 |
Equal to or higher than 326 points |
Adults at Level 4 are able to define the nature of problems in ill-structured and information-rich contexts. They can integrate multiple sources of information and their interactions, identify and disregard irrelevant information, and formulate relevant cues. Adults at this level can identify and apply multi-step solutions to meet one or more complex goals. They adapt the problem-solving process to changes, even if these changes are not obvious, occur unexpectedly or require a major re-evaluation of the problem. They are able to distinguish between relevant and irrelevant changes, predict future developments of the problem situation, and consider multiple criteria simultaneously to judge whether the solution process is likely to lead to success. Adults at Level 4 engage in the following cognitive processes:
Adults at this level engage in the following metacognitive processes:
|
|
Level 3 |
276 to less than 326 points |
Adults at Level 3 understand problems that are either static but more complex, or have an average to high level of dynamics. They can solve problems with multiple constraints or problems that require the attainment of several goals in parallel. In problems that change and require them to adapt, they can deal with frequent and, to some extent, continuous changes. They discriminate between changes that are relevant and those that are less relevant or unrelated to the problem. Adults at this level can identify and apply multi-step solutions that integrate several important variables simultaneously and consider the impact of several problem elements on each other. In dynamically changing problems, they predict future developments in the problem situation based on information collected from past developments. They adapt their behaviour according to the predicted change. Adults at Level 3 engage in the following cognitive processes:
Adults at this level engage in the following metacognitive processes:
|
|
Level 2 |
226 to less than 276 points |
Adults at Level 2 can identify and apply solutions that consist of several steps to problems that require one target variable to be considered in order to judge whether the problem has been solved. In dynamic problems that exhibit change, adults at this level can identify relevant information if they are prompted about specific aspects of the change or if changes are transparent, occur only one at a time, relate to a single problem feature and are easily accessible. Problems at this level are presented in well-structured environments and contain only a few information elements with direct relevance to the problem. Minor impasses may be introduced but these can be resolved easily by adjusting the initial problem-solving procedure. Adults at Level 2 engage in the following cognitive processes:
Adults at this level engage in the following metacognitive processes:
|
|
Level 1 |
176 to less than 226 points |
Adults at Level 1 are able to understand simple problems, and develop and implement solutions to solve them. Problems contain a limited number of elements and little to no irrelevant information. Solutions at this level are simple and consist of a limited number of steps. Problems are embedded in a context that includes one or two sources of information and presents a single, explicitly defined goal. Adults at Level 1 engage in the following cognitive processes:
|
|
Below Level 1 |
Less than 176 points |
Adults performing below Level 1 understand very simple static problems situated within a clearly structured environment. Problems contain no invisible elements, no irrelevant information that might distract from the core of the problem, and typically only require a single step to solve the problem. Adults at this proficiency level are able to engage in the basic cognitive processes required to solve problems if explicit support is given and if they are prompted to do so. |
Box 3.3. Illustrative item map: Adaptive problem solving
Copy link to Box 3.3. Illustrative item map: Adaptive problem solvingIn addition to the released items (available at https://www.oecd.org/en/about/programmes/piaac/piaac-released-items.html), the selected set of items below illustrates the types of tasks within each level by describing non-released items and mapping them to the APS framework.
Product Return (C107P001)
difficulty: below Level 1 (170 points)
cognitive process: searching for information relevant to the solution of the problem
metacognitive process: not applicable
information environment: digital
context: personal
This is the first task in a unit situated in a simulated website environment. The environment is simple and well structured, consisting of five pages that contain only images and limited text. Each page can be accessed by clicking on a tab that has a name summarising the content on the page.
The task begins on the home page of the website, and the respondent is asked to follow a process for returning an item that was ordered on line. The wording of the problem is clearly defined and uses the keyword “return”, which exactly matches the label on one of the web page tabs (“Returns”). The direct match between the task and the relevant web page makes the solution very noticeable. Once the respondent navigates to the Returns page, there is only one action that can be taken on that page, which is to tap on a button to begin the return process. These features make this the easiest task on the PIAAC APS scale.
Transport (C101P001)
difficulty: Level 1 (188 points)
cognitive process: defining the problem
metacognitive process: not applicable
information environment: physical
context: work
This task is situated in a simple, well-structured environment with a limited number of elements, consisting of a graphic of a delivery truck and three packages. Respondents are asked to drag and drop the packages onto the truck, given a limited set of constraints (e.g. the truck can only be loaded from the back to the front and the delivery order of the three packages). There is a single problem-solving goal, and operators are visible and arranged logically. To solve this problem, respondents must create a mental model of loading and then delivering the packages and co-ordinate this model with the images of the truck and packages.
Travel Planning (C109P003)
difficulty: Level 2 (239 points)
cognitive process: applying a solution
metacognitive process: searching for information relevant to the solution of the problem
information environment: digital
context: social/community
To complete this task, respondents must review four travel options and select the one that best meets a set of requirements. A list of five requirements is provided, four of which are relevant to this task. The resources needed to solve the problem are well structured and consist of a table with several types of information: departure/arrival time (represented graphically), travel mode/duration of trip and round-trip cost. Respondents need to consider the four travel options and match the details with the constraints provided in the travel requirements. Although there are several elements to consider, the difficulty of this task is manageable because the elements can be considered independently when assessing each travel option. Each of the incorrect options can be eliminated by identifying the one travel requirement that it does not meet. This process of elimination requires respondents to use metacognitive processes related to searching for the solution by evaluating alternative solutions.
Garden Time (C102P005)
difficulty: Level 3 (298 points)
cognitive process: searching for information relevant to the solution of the problem
metacognitive process: applying a solution
information environment: digital
context: personal
The overall task is to select a combination of three fertilisers that meet a set of optimal criteria for growing trees. This and the previous item in the unit share the same complex environment, which consists of an interactive tool for selecting amounts of fertilisers and two tables where the results of those selections can be viewed. In this environment, the critical information is unknown until respondents begin making their selections.
In this item, there is a change in the problem configuration that requires respondents to adapt their solution strategies. A new fertiliser is introduced, requiring respondents to reconsider the selections from the previous item. The change is explicitly mentioned in the problem statement, but the characteristics of the new fertiliser are only discoverable through the interactive tool. Respondents must derive the composition of the selected fertilisers (the two previous fertilisers and the new fertiliser) and monitor progress to evaluate when their selections best match the target. To complete this task correctly, respondents must generate a complex mental model by iteratively implementing solutions in the interactive tool. They must monitor and evaluate their progress as they work towards a final application of their solution.
Ships Ahoy (C114P005)
difficulty: Level 4 (350 points)
cognitive process: applying a solution
metacognitive process: applying a solution
information environment: physical
context: personal
This is an example of a highly dynamic problem where respondents have to continuously adapt their solution to the latest evolution in the problem environment. The environment consists of an interactive simulation where respondents control the speed and direction of a boat as it travels around islands and under varying wind conditions to reach a harbour. The entire route is not visible from the start but is revealed as the boat progresses. Respondents must continuously monitor their progress and adapt the position and speed of the boat as they encounter obstacles while travelling to the harbour.
Table 3.7 shows the probability of adults with particular proficiency in adaptive problem solving items of different levels of difficulty.
Table 3.7. Probability of successfully completing items of varying difficulty levels by proficiency score: Adaptive problem solving
Copy link to Table 3.7. Probability of successfully completing items of varying difficulty levels by proficiency score: Adaptive problem solving|
Item difficulty |
Proficiency score |
|||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
150 |
175 |
200 |
225 |
250 |
275 |
300 |
325 |
350 |
375 |
400 |
425 |
|
|
Level 4 |
0.12 |
0.17 |
0.22 |
0.29 |
0.37 |
0.46 |
0.55 |
0.63 |
0.71 |
0.78 |
0.84 |
0.88 |
|
Level 3 |
0.04 |
0.07 |
0.14 |
0.24 |
0.39 |
0.57 |
0.73 |
0.84 |
0.92 |
0.96 |
0.98 |
0.99 |
|
Level 2 |
0.05 |
0.11 |
0.23 |
0.44 |
0.66 |
0.83 |
0.93 |
0.97 |
0.99 |
0.99 |
1.00 |
1.00 |
|
Level 1 |
0.24 |
0.42 |
0.62 |
0.79 |
0.89 |
0.95 |
0.98 |
0.99 |
1.00 |
1.00 |
1.00 |
1.00 |
Test languages and reporting
Copy link to Test languages and reportingEach country or economy participating in the 2023 Survey of Adult Skills had to choose which language or languages they would use to administer the assessment. Most frequently, they chose their official national language or languages. In some cases, they also chose a widely used language in addition to the national language. Participating countries and economies chose the most relevant and/or commonly used languages in economic and civic life (e.g. in interaction with public bodies and institutions, such as educational institutions). Table 3.8 shows which languages were used in which countries and economies.
Table 3.8. Test languages by country
Copy link to Table 3.8. Test languages by country|
OECD countries and economies |
Language(s) of the direct assessment |
Languages of the background questionnaire |
|---|---|---|
|
OECD countries |
||
|
Austria |
German |
German |
|
Canada |
English, French |
English, French |
|
Chile |
Spanish |
Spanish |
|
Czechia |
Czech |
Czech |
|
Denmark |
Danish |
Danish |
|
Estonia |
Estonian, Russian |
Estonian, Russian |
|
Finland |
Finnish, Swedish |
Finnish, Swedish |
|
France |
French |
French |
|
Germany |
German |
German |
|
Hungary |
Hungarian |
Hungarian |
|
Ireland |
English |
English |
|
Israel |
Hebrew, Arabic |
Hebrew, Arabic |
|
Italy |
Italian |
Italian |
|
Japan |
Japanese |
Japanese |
|
Korea |
Korean |
Korean |
|
Latvia |
Latvian, Russian |
Latvian, Russian |
|
Lithuania |
Lithuanian |
Lithuanian |
|
Netherlands |
Dutch |
Dutch |
|
New Zealand |
English |
English |
|
Norway |
Norwegian (Bokmål) |
Norwegian (Bokmål), English |
|
Poland |
Polish |
Polish |
|
Portugal |
Portuguese |
Portuguese |
|
Slovak Republic |
Slovak, Hungarian |
Slovak, Hungarian |
|
Spain |
Castilian, Catalan, Basque, Galician, Valencian |
Castilian, Catalan, Basque, Galician, Valencian |
|
Sweden |
Swedish |
Swedish |
|
Switzerland |
French, Italian, German (standard), Swiss German |
French, Italian, German (standard), Swiss German |
|
United States |
English |
English, Spanish |
|
Subnational entities |
||
|
Flemish Region (Belgium) |
Dutch |
Dutch |
|
England (UK) |
English |
English |
|
Partner countries |
||
|
Croatia |
Croatian |
Croatian |
|
Singapore |
English |
English |
It should always be borne in mind that proficiency in literacy, numeracy and adaptive problem solving can’t be disentangled from proficiency in the language in which the assessment was administered. This is particularly important for interpreting the results of adults whose native language is different from the one used for the assessment. Their relatively poor performance in the assessment is not necessarily an indication of the lack of (abstract) “skills”. A British citizen living and working in France may display poor literacy skills when taking the assessment in French but be a proficient reader with high literacy skills when working and interacting in English.
While most foreign-born adults living abroad would have sufficient proficiency in the language of the host country to participate in the survey and take the assessment, a small minority may lack the basic language proficiency to do so. In the first cycle of the Survey of Adult Skills, these adults were classified as “literacy-related non-respondents”. As they were not even able to answer the background questionnaire, it was not possible to provide reasonable estimates of their literacy, numeracy and adaptive problem solving proficiency. In the first cycle of the survey, across the three data collection rounds between 2012 and 2018, 1.5% of adults were not able to take part in the survey because of insufficient proficiency in the questionnaire and assessment language; in the Flemish Region (Belgium) and the United States the share of adults in this situation exceeded 5%.
In order to collect more information on these adults, the 2023 Survey of Adult Skills introduced a new instrument called the doorstep interview. This is a short, self-administered questionnaire offered in 43 languages (the official languages of all 31 countries and economies participating in the survey, as well as the languages of the most important linguistic minorities in each country). Through the doorstep interview, adults with poor language proficiency in the assessment language were able to report key personal background information such as gender, age, level of education, employment status, country of birth and duration of residence in the survey country. The information collected through the doorstep interview was used to estimate the skills of such respondents, thus obtaining a more accurate picture of the distribution of skills in the overall adult population.
For those countries and economies that administered the assessment in more than one language, results are normally aggregated across the different languages, and a single measure of proficiency is reported. In other words, the average literacy proficiency in Estonia is computed as the average result of Estonian adults who took the assessment in Estonian or Russian. Only in Canada was the sample designed to allow proficiency to be reliably estimated separately for both English and French. OECD publications do not present separate results for English and French speakers in Canada although they can be computed from the available microdata and can be reported in national reports or other publications.
References
[1] OECD (2021), The Assessment Frameworks for Cycle 2 of the Programme for the International Assessment of Adult Competencies, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/4bc2342d-en.
Note
Copy link to Note← 1. This is consistent with the first cycle of the survey but differs from the approach used in IALS and ALL in which a value of 0.80 was used to locate items and test takers on the relevant scales (see Chapter 6 for more details).