The Survey of Adult Skills

Annex A. Relationship between the level of descriptors used in the Survey of Adult Skills (PIAAC) and other skills surveys

 

In presenting the results of the Survey of Adult Skills (PIAAC), the descriptors used to describe the characteristics of the tasks at each proficiency level in literacy and numeracy differ from those used when presenting the results of the International Adult Literacy Survey (IALS) and the Adult Literacy and Life Skills Survey (ALL). This is the result of:

• The introduction of the domain of literacy, which replaces the previously separate domains of prose and document literacy used in IALS and ALL.

• A change in the way in which the “proficiency” of individuals and the “difficulty” of items are defined in the Survey of Adult Skills compared to the IALS and ALL.

 

A single literacy scale

The construct of “literacy” measured in the Survey of Adult Skills encompasses prose and document literacy, which were reported on separate scales in previous international adult literacy surveys, and also incorporates the reading of digital texts. Irrespective of any change to the definition of proficiency levels, the development of a new, single literacy scale necessitated a review of the descriptors of the proficiency levels used for reporting results.

The definition of proficiency levels

The Survey of Adult Skills locates items and individuals on the three proficiency scales using a response probability (RP) value of 0.67. In other words, individuals are located on the scale at the point at which he or she has a 67% probability of successfully completing a random set of items representing the construct measured. Items are located on the scale at the point at which they have a 67% probability of being successfully completed by a random sample of the adult population. This differs from the approach used in IALS and ALL in which a response probability of 0.80 was used. This change was made so that the approach used to define what it means for a person to be at a certain proficiency level was similar to that used in PISA (see OECD, 2010, p. 48).

The change in response probability has no consequences for either the estimation of the proficiency or the precision of the scales. The estimation of proficiency is independent of the selection of an RP value, as it is a function of the level of correct response to the test items. The precision of the scale is a function of the number of items in the scale, which is again independent of the choice of RP value. What the change in RP value does affect is the way proficiency is defined and described. In effect, “proficiency” is defined in terms of a different probability of successfully completing tasks. In the case of the shift from an RP value of 0.80 to one of 0.67, the result is that proficiency is described in terms of more difficult items that are completed with a lower probability of success.

This can be seen in the Table A.1 below, which presents item maps for literacy and numeracy when response probabilities of 0.67 and 0.80 are used. For example, the literacy item “Summer Streets” is located at 350 on the scale when a response probability of 0.67 is used as opposed to 369 when 0.80 is used. Similarly, the numeracy item “TV” moves from 279 to 260 when the response probability changes from 0.67 to 0.80.

Table A.1. Location of items on the literacy scale using RP67 and RP80

Score	RP67	RP80
400		Baltic Stock Market C308A116
398		Library Search C323P005
397		CANCO 306B111
389		Work-related Stress C329P003
386		Apples P317P001
376	Library Search C323P005	Work-related Stress C329P002
374	Work-related Stress C329P003
372	CANCO C306B111
371	Baltic Stock Market C308A116
369		Summer Streets C327P004
368		Milk Label P324P002
364		Library Search C323P002
359	Apples P317P001
358		Baltic Stock Market C308A118
357		Generic Medicines C309A322
350	Summer Streets C327P004
349	Work-related Stress C329P002
348	Library Search C323P002
347	Milk Label P324P002
346		Distances-Mexican Cities C315B512
343		Library Search C323P004
342		Summer Streets C327P003
341		International Calls C313A410
337	Baltic Stock Market C308A118
336		Milk Label P324P003
333		Civil Engineering C318P003
331		Contact Employer C304B711
330		Summer Streets C327P002
329	Generic Medicines C309A322 Library Search C323P004	International Calls C313A411 Memory Training C310A407 TMN Anti-theft C305A218
324	International Calls C313A410
321		Summer Streets C327P001
320	Summer Streets C327P003	Civil Engineering C318P001
318	Distances-Mexican Cities C315B512
316	Civil Engineering C318P003
315	International Calls C313A411
314		Baltic Stock Market C308A119 Lakeside Fun Run C322P003
312	Memory Training C310A407
312	Milk Label P324P003
309	TMN Anti-theft C305A218
308		Lakeside Fun Run C322P004 MEDCO Aspirin C307B402
306	Summer Streets C327P002	Lakeside Fun Run C322P001
305		Library Search C323P003 International Calls C313A413
304	Contact Employer C304B711
303	Civil Engineering C318P001
301		Discussion forum C320P003 Discussion forum C320P004
298	Summer Streets C327P001	Contact Employer C304B710
297	Baltic Stock Market C308A119
295		Baltic Stock Market C308A121
294	Lakeside Fun Run C322P003
293	Lakeside Fun Run C322P004	Discussion forum C320P001
292		International Calls C313A414
291		Generic Medicines C309A319
289	Library Search C323P003
288	MEDCO Aspirin C307B402
287		Apples P317P003
286	Discussion forum C320P003 International Calls C313A413 Contact Employer C304B710	Memory Training C310A406
285	Discussion forum C320P004
283	Lakeside Fun Run C322P001	Apples P317P002
281	Discussion forum C320P001
280		International Calls C313A412
280		Internet Poll C321P002
279	Baltic Stock Market C308A121	TMN Anti-theft C305A215
272	Memory Training C310A406 Generic Medicines C309A319 International Calls C313A414	Internet Poll C321P001
271		Baltic Stock Market C308A120
265	Apples P317P003	Lakeside Fun Run C322P002
264		Lakeside Fun Run C322P005
262	Apples P317P002
261		CANCO C306B110
260	TMN Anti-theft C305A215
259		Baltic Stock Market C308A117
258		Generic Medicines C309A320
257	International Calls C313A412
254	Baltic Stock Market C308A120
251	Internet Poll C321P001
244	CANCO C306B110 Lakeside Fun Run C322P005
240	Lakeside Fun Run C322P002	Generic Medicines C309A321
239	Baltic Stock Market C308A117
239	Generic Medicines C309A320
238	Internet Poll C321P002
234		Guadeloupe P330P001
231		Dutch Women C311B701
219	Generic Medicines C309A321
207	Guadeloupe P330P001
203		Election Results C302BC02
201	Dutch Women C311B701
190		MEDCO Aspirin C30B7401
169	MEDCO Aspirin C30B7401
163		Employment Ad C300AC02
162	Election Results C302BC02
136	Employment Ad C300AC02
117		SGIH C301AC05
75	SGIH C301AC05

Table A.2. Location of items on the numeracy scale using RP67 and RP80

Score	RP67		RP80
397			Dioxin (MOD)	C612A518
388			Educational Level	C632P001
375	Dioxin (MOD)	C612A518
361			Compound Interest	P610A515
359			Weight History	C660P004
357			Wine	P623A618
354	Educational Level	C632P001
349			Package	C657P001
348	Compound Interest	P610A515
343			Cooper Test Amoeba	C665P002 C641P001
341	Wine	P623A618
335			BMI	C624A620
334			Study Fees	C661P002
333			Inflation	C620A612
332	Weight History	C660P004
331			Peanuts	C634P002
330			NZ Exports	C644P002
328			Fertilizer	C651P002
327			Classified	C622A615
326	Cooper Test	C665P002
324	Amoeba	C641P001	Study Fees Peanuts Orchestra Tickets	C661P001 C664P001 C634P001
323			Map	C617A605
322			Temp Scale	C611A517
320	BMI	C624A620
319			Six Pack 1 Lab Report	C618A608 C636P001
318	Peanuts	C634P002
317	NZ Exports	C644P002
315	Study Fees Package	C661P002 C657P001	Map	C617A606
314	Fertilizer	C651P002
308	Study Fees	C661P001
308	Inflation	C620A612
307	Orchestra Tickets	C664P001
305	Peanuts	C634P001
303	Map	C617A605
302			Tiles	C619A609
301	Classified	C622A615
299			Weight History Tree	C660P003 C608A513
297	Six Pack 1	C618A608
296	Temp Scale	C611A517
294	Lab Report	C636P001	Solution	C606A509
292			Wine	C623A617
289			Educational Level	C632P002
287	Map	C617A606	Urban Population	C650P001
285			Temp Scale	C611A516
284			Photo	C605A506
283			Inflation	C620A610
282	Tiles	C619A609
280			Wine	C623A616
278			Price Tag Rope	C602A503 P666P001
277			Rug Production	C646P002
276	Wine Weight history	C623A617 C660P003
273	Solution	C606A509
271			PriceTag	C602A502
270			Logbook	C613A520
267	Inflation	C620A610
267			Path	C655P001
266	Educational Level	C632P002
263			Airport Timetable	C645P001
262			Photo	C605A507
261	Temp Scale	C611A516
260	Urban Population Tree	C650P001 C608A513	TV	C607A510
259	Photo Price Tag	C605A506 C602A503
258	Wine	C623A616	Cooper Test	C665P001
256	Rug Production	C646P002
255			Candles	C615A603
252			Gas Gauge	C604A505
250	Logbook	C613A520	BMI Candles	C624A619 C615A602
249	Path	C655P001	Photo Six Pack 1	C605A508 C618A607
242	Photo	C605A507
240	Rope	P666P001
239	TV	C607A510
238	Price Tag	C602A502
234	Cooper Test	C665P001
231	Candles Airport Timetable	C615A603 C645P001
228	Gas Gauge	C604A505
227	Photo	C605A508
221	BMI Candles	C624A619 C615A602
219			Odometer	P640P001
217	Six Pack 1	C618A607
212			Watch	C614A601
201			Price Tag	C602A501
200			Parking Map	C635P001
195	Odometer	P640P001
185	Watch	C614A601
183			Election Results	C600AC04
179	Parking Map	C635P001
168	Price Tag	C602A501
167			Bottles	C601AC06
155	Election Results	C600AC04
129	Bottles	C601AC06

As the score point ranges defining the proficiency levels for literacy and numeracy have not changed between IALS and ALL and the Survey of Adult Skills, the group of items used to describe each proficiency level – i.e. those that are located in the score-point range that defines a proficiency level – changes. This necessitated revising the descriptors of the proficiency levels. Tables A.3 and A.4 present the descriptors used in the Survey of Adult Skills and the previous surveys.

Table A.3. Descriptors of literacy proficiency levels

Level	Score range	Survey of Adult Skills (PIAAC) Literacy (RP67)	ALL/IALS Prose literacy (RP80)	ALL/IALS Document literacy (RP80)
1	Lower than 225	Most of the tasks at this level require the respondent to read relatively short digital or print continuous, non-continuous, or mixed texts to locate a single piece of information which is identical to or synonymous with the information given in the question or directive. Some tasks may require the respondent to enter personal information onto a document, in the case of some non-continuous texts. Little, if any, competing information is present. Some tasks may require simple cycling through more than one piece of information. Knowledge and skill in recognising basic vocabulary, evaluating the meaning of sentences, and reading of paragraph text is expected.	Most of the tasks at this level require the respondent to read relatively short text to locate a single piece of information which is identical to or synonymous with the information given in the question or directive. If plausible but incorrect information is present in the text, it tends not to be located near the correct information.	Tasks at this level tend to require the respondent either to locate a piece of information based on a literal match or to enter information from personal knowledge onto a document. Little, if any, distracting information is present.
2	226-275	At this level the complexity of text increases. The medium of texts may be digital or printed, and texts may be comprised of continuous, non-continuous, or mixed types. Tasks in this level require respondents to make matches between the text and information, and may require paraphrase or low-level inferences. Some competing pieces of information may be present. Some tasks require the respondent to: cycle through or integrate two or more pieces of information based on criteria, compare and contrast or reason about information requested in the question, or navigate within digital texts to access-and-identify information from various parts of a document.	Some tasks at this level require respondents to locate a single piece of information in the text; however, several distractors or plausible but incorrect pieces of information may be present, or low-level inferences may be required. Other tasks require the respondent to integrate two or more pieces of information or to compare and contrast easily identifiable information based on a criterion provided in the question or directive.	Tasks at this level are more varied than those in Level 1. Some require the respondents to match a single piece of information; however, several distractors may be present, or the match may require low-level inferences. Tasks in this level may also ask the respondent to cycle through information in a document or to integrate information from various parts of a document.
3	276-325	Texts at this level are often dense or lengthy, including continuous, non-continuous, mixed, or multiple pages. Understanding text and rhetorical structures become more central to successfully completing tasks, especially in navigation of complex digital texts. Tasks require the respondent to identify, interpret, or evaluate one or more pieces of information, and often require varying levels of inference. Many tasks require the respondent construct meaning across larger chunks of text or perform multi-step operations in order to identify and formulate responses. Often tasks also demand that the respondent disregard irrelevant or inappropriate text content to answer accurately. Competing information is often present, but it is not more prominent than the correct information.	Tasks at this level tend to require respondents to make literal or synonymous matches between the text and information given in the task, or to make matches that require low-level inferences. Other tasks ask respondents to integrate information from dense or lengthy text that contains no organisational aids such as headings. Respondents may also be asked to generate a response based on information that can be easily identified in the text. Distracting information is present, but is not located near the correct information.	Some tasks at this level require the respondent to integrate multiple pieces of information from one or more documents. Others ask respondents to cycle through rather complex tables or graphs which contain information that is irrelevant or inappropriate to the task.
4	326-375	Tasks at this level often require respondents to perform multiple-step operations to integrate, interpret, or synthesise information from complex or lengthy continuous, non-continuous, mixed, or multiple type texts. Complex inferences and application of background knowledge may be needed to perform successfully. Many tasks require identifying and understanding one or more specific, non-central ideas in the text in order to interpret or evaluate subtle evidence-claim or persuasive discourse relationships. Conditional information is frequently present in tasks at this level and must be taken into consideration by the respondent. Competing information is present and sometimes seemingly as prominent as correct information.	These tasks require respondents to perform multiple-feature matches and to integrate or synthesise information from complex or lengthy passages. More complex inferences are needed to perform successfully. Conditional information is frequently present in tasks at this level and must be taken into consideration by the respondent.	Tasks at this level, like those at the previous levels, ask respondents to perform multiple-feature matches, cycle through documents, and integrate information; however, they require a greater degree of inference. Many of these tasks require respondents to provide numerous responses but do not designate how many responses are needed. Conditional information is also present in the document tasks at this level and must be taken into account by the respondent.
5	Higher than 376	At this level, tasks may require the respondent to search for and integrate information across multiple, dense texts; construct syntheses of similar and contrasting ideas or points of view; or evaluate evidenced based arguments. Application and evaluation of logical and conceptual models of ideas may be required to accomplish tasks. Evaluating reliability of evidentiary sources and selecting key information is frequently a key requirement. Tasks often require respondents to be aware of subtle, rhetorical cues and to make high-level inferences or use specialised background knowledge.	Some tasks at this level require the respondent to search for information in dense text which contains a number of plausible distractors. Others ask respondents to make high-level inferences or use specialised background knowledge. Some tasks ask respondents to contrast complex information.	Tasks at this level require the respondent to search through complex displays that contain multiple distractors, to make high-level text-based inferences, and to use specialised knowledge.

Table A.4. Descriptors of literacy proficiency levels

Level	Score range	Survey of Adult Skills (PIAAC) (RP67)	ALL (RP80)
1	Lower than 225	Tasks at this level require the respondent to carry out basic mathematical processes in common, concrete contexts where the mathematical content is explicit with little text and minimal distractors. Tasks usually require one-step or simple processes involving e.g. counting, sorting, performing basic arithmetic operations, understanding simple percentages such as 50%, locating and identifying elements of simple or common graphical or spatial representations.	Tasks at this level require the respondent to show an understanding of basic numerical ideas by completing simple tasks in concrete, familiar contexts where the mathematical content is explicit with little text. Tasks consist of simple, one-step operations such as counting, sorting dates, performing simple arithmetic operations or understanding common and simple percentages such as 50%.
2	226-275	Tasks at this level require the respondent to identify and act upon mathematical information and ideas embedded in a range of common contexts where the mathematical content is fairly explicit or visual with relatively few distractors. Tasks tend to require the application of two or more steps or processes involving e.g. calculation with whole numbers and common decimals, percentages and fractions; simple measurement and spatial representation; estimation; and interpretation of relatively simple data and statistics in texts, tables and graphs.	Tasks at this level are fairly simple and relate to identifying and understanding basic mathematical concepts embedded in a range of familiar contexts where the mathematical content is quite explicit and visual with few distractors. Tasks tend to include one-step or two-step processes and estimations involving whole numbers, benchmark percentages and fractions, interpreting simple graphical or spatial representations, and performing simple measurements.
3	276-325	Tasks at this level require the respondent to understand mathematical information which may be less explicit, embedded in contexts that are not always familiar and represented in more complex ways. Tasks require several steps and may involve the choice of problem-solving strategies and relevant processes. Tasks tend to require the application of e.g. number sense and spatial sense; recognising and working with mathematical relationships, patterns, and proportions expressed in verbal or numerical form; and interpretation and basic analysis of data and statistics in texts, tables and graphs.	Tasks at this level require the respondent to demonstrate understanding of mathematical information represented in a range of different forms, such as in numbers, symbols, maps, graphs, texts, and drawings. Skills required involve number and spatial sense, knowledge of mathematical patterns and relationships and the ability to interpret proportions, data and statistics embedded in relatively simple texts where there may be distractors. Tasks commonly involve undertaking a number of processes to solve problems.
4	326-375	Tasks at this level require the respondent to understand a broad range of mathematical information that may be complex, abstract or embedded in unfamiliar contexts. These tasks involve undertaking multiple steps and choosing relevant problem-solving strategies and processes. Tasks tend to require analysis and more complex reasoning about e.g. quantities and data; statistics and chance; spatial relationships; and change, proportions and formulas. Tasks at this level may also require comprehending arguments or communicating well-reasoned explanations for answers or choices.	Tasks at this level require respondents to understand a broad range of mathematical information of a more abstract nature represented in diverse ways, including in texts of increasing complexity or in unfamiliar contexts. These tasks involve undertaking multiple steps to find solutions to problems and require more complex reasoning and interpretation skills, including comprehending and working with proportions and formulas or offering explanations for answers.
5	Higher than 376	Tasks at this level require the respondent to understand complex representations and abstract and formal mathematical and statistical ideas, possibly embedded in complex texts. Respondents may have to integrate multiple types of mathematical information where considerable translation or interpretation is required; draw inferences; develop or work with mathematical arguments or models; justify, evaluate and critically reflect upon solutions or choices.	Tasks at this level require respondents to understand complex representations and abstract and formal mathematical and statistical ideas, possibly embedded in complex texts. Respondents may have to integrate multiple types of mathematical information, draw inferences, or generate mathematical justification for answers.

Source (IALS/ALL): OECD/Statistics Canada (2011).

References

OECD (2010), PISA 2009 Results: What Students Know and Can Do: Student Performance in Reading, Mathematics and Science (Volume I), OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264091450-en.

OECD/Statistics Canada (2011), Literacy for Life: Further Results from the Adult Literacy and Life Skills Survey, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264091269-en.