This chapter focuses on how the 2023 Survey of Adult Skills was designed, managed and conducted. It discusses the target population, exclusions from the survey, sample size and response rates, as well as how the overall quality of the data was assessed.
Survey of Adult Skills – Reader's Companion
2023
Abstract
The design and implementation of the 2023 Survey of Adult Skills was guided by associated technical standards and guidelines which were developed to ensure that the survey yielded high-quality and internationally comparable data. The Technical Standards and Guidelines (TSGs) for the 2023 Survey of Adult Skills articulate the standards participating countries and economies are expected to adhere to in implementing the survey and describes the steps they should follow in order to meet them1. It also makes recommendations for actions relating to the standards which were not mandatory but could help to produce high-quality data. Standards were established for 16 distinct aspects of the design and implementation of the survey (Table 5.1).
The development and implementation of the 2023 Survey of Adult Skills was overseen by the PIAAC Board of Participating Countries (BPC). The BPC consisted of representatives from each of the participating countries and economies. Croatia participated as an observer. The BPC was responsible for making major decisions on budgets, developing and implementing the survey, reporting results, and monitoring the progress of the project. The BPC was supported in its work by the OECD Secretariat, which was responsible for providing advice to the BPC and managing the project on its behalf.
The OECD contracted an international consortium to undertake a range of tasks relating to the design and development of the assessment, implementation of the survey, and analysis of the resulting data. The consortium was responsible for developing questionnaires, assessment instruments and the computer-delivery platform; supporting survey operations; quality control; scaling; and database preparation.
Participating countries and economies were responsible for the national implementation of the survey, in particular for sampling, translation and adaptation of materials, data collection, and database production. In each country, all these activities were led and co-ordinated by a national project manager.
The TSG document is one element of a comprehensive process of quality assurance and control that was put in place to reduce potential sources of error, increase comparability and maximise the quality of the data produced by the 2023 Survey of Adult Skills. Participating countries and economies received assistance in meeting the standards in a variety of ways. Where relevant, manuals, training materials, testing plans and toolkits were produced. Training was provided to countries at appropriate stages of the project. In certain areas, such as sampling, translation and adaptation, and the operation of the computer-delivery platform, passage through the various stages of implementation was subject to a review of the steps completed, and sign-off was often required before moving to the next stage. Regular consultations were held with countries at project meetings and through bilateral contact. Compliance with the technical standards was monitored throughout the development and implementation phases through direct contact, the provision of evidence that required activities were completed and ongoing collection of data from countries about key aspects of implementation.
The quality of each country and economy’s data was reviewed prior to publication. The review was based on the analysis of the psychometric characteristics of the data and evidence of compliance with the technical standards. A data quality assessment was prepared for each country, and recommendations were made regarding release and, if necessary, any restrictions and/or qualifications that should apply to the release and publication. The Technical Advisory Group (TAG), comprised of independent experts, validated the approach to the review of data and the results of the analysis; the project’s steering body, the PIAAC Board of Participating Countries (BPC), made the final decision on release.
This chapter provides the background needed to correctly interpret the results of the reviews of data quality. It describes the following aspects of the design and the methodology of the 2023 Survey of Adult Skills:
survey and assessment design
sampling
translation and adaptation of survey instruments
survey administration
survey response rates and non-response bias analysis
the doorstep interview and literacy-related non-response
the overall assessment of data quality.
The Survey of Adult Skills (PIAAC) collects data using a combination of a personal interview and a self-completed assessment. The 2023 Survey of Adult Skills has two components: a background questionnaire and a direct assessment.
The background questionnaire (BQ) is administered as a computer-aided personal interview. Trained interviewers ask the questions contained in the background questionnaire and record the answers using a tablet and a keyboard. The time needed to complete the background questionnaire varies depending on respondents’ characteristics (many questions in the background questionnaire only concern adults who are currently employed, for instance).
The second component is a direct assessment of literacy, numeracy and adaptive problem solving (APS) completed autonomously by the respondent. To complete the assessment, respondents use the same touchscreen tablet device used by the interviewer to administer the questionnaire. The tablet interface replicates the feeling of using paper-based instruments as much as possible. If they wish, respondents can also use a digital stylus to interact with the interface. The interviewers remain with the respondents to supervise them during the assessment. The interviewer is not supposed to help the respondent during the direct assessment but can assist if there are technical problems. Normally, the interviewer encourages respondents to persist through the assessment and to attempt to provide answers to the best of their ability. The direct assessment component is untimed, allowing respondents to take as much time as they need to complete it.
Figure 5.1 summarises the design of the survey and the various elements of the interview.
The interview starts with the background questionnaire. The questionnaire collects detailed information on the socio-demographic characteristics of respondents (age, gender, migration status), as well as on their educational careers and labour-market outcomes (see Chapter 4 for more information). Respondents who are unable to complete the background questionnaire because of language barriers are administered the doorstep interview, discussed below.
Following completion of the background questionnaire, respondents first go through a short tablet tutorial to make sure they understand how to interact with the device and with the interface. The ease of use of the tablet, accompanied by a digital stylus, ensures that even adults with very low familiarity with digital devices can complete the assessment on the tablet.
The tablet-only design allows adaptive algorithms to be used to optimise the delivery of assessment items. These algorithms use information from the background questionnaire (such as age and educational attainment), as well as information from answers to previous assessment items, to select and administer assessment items that are neither too easy nor too difficult for the respondent.
The first step of this adaptive process is a locator test (also referred to as Stage 1), including eight literacy and eight numeracy items. Based on their answers to these items, respondents are sorted into three different paths:
Respondents who fail the locator test are directed to Path 1 and are administered an assessment of reading and numeracy component skills.
Respondents who pass the locator test with a low score are directed to Path 2. They take the reading and numeracy component skills assessment and then proceed to the direct assessment of literacy, numeracy and adaptive problem solving.
Respondents who pass the locator test with a high score are directed to Path 3. Some of these adults (12.5%) are randomly chosen to take the component skills assessment and then proceed to the direct assessment of literacy, numeracy and adaptive problem solving. The rest proceed directly to the assessment.
Table 5.2 reports the distribution of respondents across the different paths of the survey for each country and economy.
The reading and numeracy component skills assessment focuses on basic literacy and numeracy skills that are foundational to the more advanced skills measured in the direct assessment. The inclusion of component skills allows for a more precise estimation of literacy and numeracy at the bottom end of the skills distribution.
In the direct assessment, each respondent is only assessed in two out of the three domains (literacy, numeracy or adaptive problem solving). Respondents are first randomly allocated to one domain. Then, after they complete that assessment, they are randomly allocated to one of the two remaining domains.
The literacy and numeracy assessments use a hybrid multistage design with both an adaptive and a linear component. In both the adaptive and the linear path, respondents are administered one testlet in both Stage 2 and 3 (Stage 1 being the locator test). In the adaptive path, six different testlets are available in Stage 2, three of low difficulty and three of high difficulty. Which of these six is assigned to respondents depends on his or her performance in the locator test and personal characteristics collected in the background questionnaire (such as level of education). Stage 3 also features six testlets (two of low difficulty, two of medium difficulty and two of high difficulty). One of these six is administered to respondents based on how they performed in Stage 2 (with those that performed better having a higher chance of receiving a more difficult testlet). The linear (non-adaptive) path is used to ensure that each item is attempted by a sufficient number of respondents from a wide proficiency range. After the locator test, 25% of the respondents are randomly allocated to this path, where they take one of six possible testlets in both Stage 2 and 3.
The assessment of adaptive problem solving follows a balanced incomplete block design, where assessment items are divided into five clusters. Respondents who take the APS assessment are assigned to two randomly selected clusters of items.
A final set of questions after completion of the direct assessment asks respondents about the effort they put into completing the assessment (as compared to a high-stakes situation). Respondents are also asked how they thought they performed in the assessment.
Finally, interviewers complete a post-interview questionnaire in which they record their observations about the context, the environment and the conditions under which the interview took place.
To maximise the comparability of results, countries and economies participating in the 2023 Survey of Adult Skills were expected to meet stringent standards relating to the target population, sample design, sample selection response rates and non-response bias analysis.
The target population and sampling frame
The target population for the survey consisted of the non-institutionalised population aged 16-65 years, residing in the country at the time of data collection, irrespective of nationality, citizenship or language status. The normal territorial unit covered by the survey was that of the country as a whole. However, in two countries, the sample frame covered only a portion of the national territory: in Belgium, only residents in the Flemish Region participated in the survey, and in the United Kingdom, only residents in England participated.
To draw a sample of respondents from the target population, a sampling frame is needed. Different countries and economies used different sampling frames and sampling strategies. The main distinction is whether frames are based on population registries, or whether they are based on lists of dwellings.
Population registries are based on administrative lists of residents maintained at either national or regional level, which contain useful variables for stratification, weighting and non-response bias analyses. The frames used by countries with population registry samples are shown in Table 5.3.
When population registries are not available, countries relied on a dwelling units registry or geographical clusters, which are based on lists of dwelling units or primary sampling units maintained at the national level for official surveys or a frame of geographical clusters formed by combining adjacent geographical areas, respecting their population sizes and taking into consideration travel distances for interviewers. When sampling frames are based on dwellings, interviewers need to visit the household to screen household members, randomly selecting members who are eligible to participate in the survey (because they are part of the target population). The frames used by countries and economies that relied on screeners are listed in Table 5.4.
Note: The grey shading indicates that there is no such stage in the country’s sample design. PSU stands for “primary sampling unit”.
Note: The grey shading indicates that there is no such stage in the country’s sample design. MSOA stands for “Middle layer Super Output Areas”, as defined by the UK Office for National Statistics.
* Spain is included in this table, even though it has a population registry. Spain used a population registry as the sampling frame for the first two sampling stages only and used a household screener for the third sampling stage.
Coverage of the target population
According to the TSGs, sampling frames were required to cover at least 95% of the target population. The exclusion (non-coverage) of groups in the target population had to be limited to the greatest extent possible and be based on operational or resource constraints, as in the case of populations located in remote and isolated regions. A complete list of exclusions for countries and economies using population registries is presented in Table 5.5; Table 5.6 includes a similar list for countries using screeners.
Note: The non-coverage rate accounts for excluded subpopulations, such as undocumented immigrants or non-institutionalised collective dwelling units, with the exception that the homeless are not being considered part of this rate. Other exclusions that occurred as a natural part of the survey process are not included in the expected non-coverage rate.
Note: The non-coverage rate accounts for excluded subpopulations, such as undocumented immigrants or non-institutionalised collective DUs, with the exception that the homeless are not being considered part of this rate. Other exclusions that occurred as a natural part of the survey process are not included in the expected non-coverage rate.
* Spain is included in this table, even though it has a population registry. Spain used a population registry as the sampling frame for the first two sampling stages only and used a household screener for the third sampling stage.
Sample size
The minimum sample size required for the 2023 Survey of Adult Skills depended on two factors: the sample design and the number of languages in which the assessment was administered. It ranged between 4 000 to 5 000 completed cases per reporting language, depending upon the sample design, as shown in Table 5.7. For a case to be counted as “completed”, the following conditions must be met: i) responses to key background questions, including age, gender, highest level of schooling, employment status and country of birth, have been collected; ii) the tablet tutorial Section has been attempted; and iii) the Locator has been attempted.
Note: PSU stands for “primary sampling units”; SSU stands for “secondary sampling units”; DUs stands for “dwelling units”.
Countries that planned to report on general proficiency had to achieve the appropriate minimum completed sample size shown in Table 5.7 for their main language. Eight countries (Canada, Estonia, Finland, Israel, Latvia, the Slovak Republic, Spain and Switzerland) implemented the survey in multiple languages, but only Canada decided to provide separate results for the two assessment languages (English and French). Table 5.8 provides sampling information including the minimum based on the standards, and the expected and actual sample size for each participating country and economy. More detailed information about sampling and weighting can be found in the Technical Report of the 2023 Survey of Adult Skills (OECD, forthcoming[1]).
Note: The PIAAC technical standard targets are for a self-weighting sample of the PIAAC target population. Additional samples are needed for country-specific samples outside of the target population or oversampled populations. Targets include multiple languages. Initial sample sizes and country targets for Austria, Canada, Chile, Denmark, France, Israel, Italy, the Netherlands, New Zealand, Switzerland and the United States include oversampled populations and country-specific samples outside the target population unless otherwise noted.
Participating countries and economies were responsible for adapting and translating the assessment instruments, background questionnaires and survey materials for administration in their national languages. Any national adaptations of either the assessment instruments or the questionnaire were subject to strict guidelines, review and approval by the international consortium. The recommended procedure included a double translation from the English source version performed by professional staff, followed by a reconciliation.
All national versions of the instruments were subject to full linguistic quality-control procedures, which involved verification by the consortium of target versions submitted by each participating country/economy against the source versions, with reporting of residual errors and undocumented deviations and expert advice where corrective action was needed:
for questionnaire and assessment items newly developed for Cycle 2: full verification of all national materials
for trend items (used in the first cycle): focused verification of changes requested by countries and economies
for trend literacy units: verification of the scoring rules in literacy items.
Detailed information on the various aspects of translation, adaptation and verification of test and survey materials can be found in the Technical Report of the 2023 Survey of Adult Skills (OECD, forthcoming[1]).
The Survey of Adult Skills was administered under the supervision of trained interviewers either in the respondent’s home or in a location agreed upon between the respondent and the interviewer.
The background questionnaire, which was the first component of the interview, was administered by the interviewer. Respondents were able to seek assistance from others in the household in completing the questionnaire, for example, in translating questions and answers. Proxy respondents were not permitted.
Following completion of the background questionnaire, the respondent undertook the direct assessment on a tablet as described above. Respondents were also permitted to use calculators, notepads and a pen during the assessment. Interviewers administering the survey were required to be trained according to common standards. These covered the timing and duration of training, as well as its format and content. A full set of training materials was provided to countries. The persons responsible for organising training nationally attended training sessions organised by the international consortium.
The survey (background questionnaire plus direct assessment) was normally undertaken in a single session. However, in exceptional circumstances, a respondent could take the questionnaire in one session and the direct assessment in another. The direct assessment was required to be completed in one session. Respondents who did not complete the assessment within a single session for whatever reason were not permitted to finish it at a later time.
Data collection in the 2023 Survey of Adult Skills was scheduled from 1 September 2022 to 30 April 2023 (8 months or 242 days). Six countries completed data collection within this period (Estonia, France, Hungary, Japan, Korea and Poland). The pace of data collection in many countries was slower than projected due to unanticipated challenges such as staffing shortages from the outset, interviewer attrition or difficulties in reaching certain population subgroups (e.g. immigrants, younger respondents). As a result, the OECD and the consortium allowed some countries to extend the data collection period, with 5 countries completing it by the end of May 2023 (Austria, Germany, Lithuania, Singapore and Switzerland), 12 countries completing it by the end of June 2023 (Croatia, Denmark, England [UK], Finland, the Flemish Region [Belgium], Ireland, Latvia, Norway, Spain, the Slovak Republic, Sweden and the United States), 3 countries completing it by the end of July 2023 (Italy, Israel and Canada), and 5 countries completing it by 6 August 2023 (Chile, Czechia, the Netherlands, New Zealand and Portugal). Detailed information on the data collection timeline can be found in Chapter 19 of the Technical Report of the 2023 Survey of Adult Skills (OECD, forthcoming[1]).
In all surveys, low response rates constitute a significant threat to the quality of the data, as they may introduce non-response bias. This happen when those who do not respond are systematically different from those who decide to participate in the survey. When this is the case, the results of the survey are no longer representative of the characteristics of the underlying target population. The 2023 Survey of Adult Skills TSGs require countries and economies to put in place a range of strategies to reduce the incidence and effects of non-response, to adjust for it when it occurs, and to evaluate the effectiveness of any weighting adjustments implemented to reduce non-response bias. In particular, countries and economies were expected to establish procedures during data collection to minimise non-response. These included pre-collection publicity, selecting high-quality interviewers, delivering training on methods to reduce and convert refusals, monitoring data collection closely to identify problem areas or groups, and directing resources to these particular groups. At least seven attempts were to be made to contact a selected individual or household before it could be classed as a non-contact. The overall rate of non-contact was to be kept below 3%.
Response rates were calculated for each stage of the assessment: 1) the screening questionnaire (for countries and economies that need to sample households before selecting respondents); 2) the background questionnaire; and 3) the direct assessment.
The overall response rate was calculated as the product of the response rates (complete cases/eligible cases) for the relevant stages of the assessment. For countries and economies with a screening questionnaire, the overall response rate was the product of the response rates for the screener, the background questionnaire and assessment; for those without a screener, it was the product of the response rates for the questionnaire and the assessment.
The computations at each stage are hierarchical, in that they depend on the response status from the previous data collection stage. A completed case thus involved completing the screener (if applicable), the background questionnaire and the direct assessment. In the case of the BQ, a completed case was defined as having responses to key background questions, including age, gender, highest level of schooling and employment status, or responses to age and gender for literacy-related non-respondents. For the direct assessment, a completed case was defined as having completed the locator stage 1 in Figure 5.1) and a literacy or numeracy module (stage 2 and 3 in Figure 5.1), or a case in which the locator was not completed for a literacy-related reason (for example because of a language difficulty or because the respondent was unable to read or write in any of a country’s test languages, or because of learning or mental disability.
Countries and economies using population registry-based sampling frames were able to treat some or all of the individuals in their samples who were untraceable as exclusions (i.e. as outside the target population) and exclude them from the numerator and denominator of the response-rate calculation (provided that the 5% threshold for exclusions was not exceeded).
The TSGs set a goal of a 70% response rate. However, countries faced challenges with declining response rates, which resulted in lower rates than encountered in the first cycle. Response and coverage rates are presented in Table 5.9.
Bias from non-response can arise if non-respondents are systematically different (in terms of skills proficiency, for example) from those who agree to participate in the survey. While low response rates introduce a potential source of bias, they do not necessarily mean that a bias is present. If the decision to participate in the survey is not related to skills proficiency, very low response rates will not result in any bias.
For this reason, the non-response bias analysis (NRBA) undertaken in the 2023 Survey of Adult Skills looked at a wide range of indicators, over and above response rates, to assess the extent to which results from the survey are susceptible to non-response bias. An extended NRBA considered further indicators related to the likelihood of non-response bias in the estimation of adults’ proficiency from the survey. These include:
comparison of estimates before and after weighting adjustments
comparison of weighted estimates to external totals
correlations between auxiliary variables used for weighting and proficiency estimates
comparison of estimates from alternative weighting adjustments
analysis of variables collected during data collection
level-of-effort analysis, looking at differences in proficiencies among respondents who required different number of contacts before agreeing to participate in the survey
the sensitivity of population estimates of proficiency to a range of assumptions on the proficiency of non-respondents.
Korea was the only country that achieved a response rate above 70% and was therefore not required to undertake the extended NRBA. The Slovak Republic, despite achieving a response rate of 70%, undertook the extended NRBA because it did not fully meet the sampling standards. In particular, in both Lithuania and the Slovak Republic, evidence was found that not all eligible persons in a household were given a chance of being selected to participate in the survey, which could lead to undercoverage bias. Measures were taken to reduce undercoverage bias (weight calibration). Although some additional caution should be used when analysing data from these countries, the outcomes of additional analysis, including the NRBA, suggest that the effects of this departure from the sampling standards are rather small.
When the decision to participate in the survey is linked to specific characteristics of the sampled adults, the distribution of these characteristics among participants in the Survey of Adult Skills may not match the true distribution in the population or that observed in other surveys. However, as all surveys are affected by some non-response, it is not clear that other sources are necessarily more accurate than the Survey of Adult Skills. Misalignments between the distribution of certain characteristics of adults in the sample and in the population, as well as with other sources, have been observed in both cycles of the Survey of Adult Skills. As a result, the changes in the composition of the population observed over the two cycles of the Survey of Adult Skills may also not match exactly the changes observed in other sources.
Sampling weights can be applied to bring the composition of the sample closer to the known distribution of characteristics in the population. In the Survey of Adult Skills, countries and economies agreed with the Secretariat and the OECD contractors on which variables to use for weighting. For variables that were not used in weighting, some discrepancies may remain after weighting.
Table 5.10 provides an overview of some misalignments between the (weighted) PIAAC sample and alternative data sources. Discrepancies are listed if they are statistically significant and over 1.5 percentage points. Some of these misalignments can be explained by differences in the definition or in other methodological aspects between the Survey of Adult Skills and other sources. Moreover, not all countries could verify the alignment across all characteristics due to data availability.
Alternative weighting schemes were considered to understand the extent to which the observed discrepancies might bias the estimated proficiency of the adult population in the 2023 Survey of Adult Skills. Significant differences in estimated proficiency were found in a few countries with these alternative weights (Table 5.11). However, the impact of applying alternative weighting schemes never exceeded four score points. This means that the fact that the distribution of some characteristics in the sample is not consistent with other sources does not seem to have a major impact on estimated proficiency because these alternative weighting adjustments resulted in only minor changes to the overall results.
1: The population groups are defined as follows: Jews, Not Ultra-Orthodox; Jews, Ultra-Orthodox; Arab Population.
2: Localities are classified as follow: Jewish/Arab by population density.
Note: The table reports the difference between the average proficiency estimated using the final PIAAC weights and the proficiency obtained using alternative weighting schemes. The table only reports countries and economies for which differences are statistically significant and notable (absolute value of the difference larger than the standard error of the differences). In a few other countries, statistically significant differences are found, but they are almost all smaller than 1 score point. These are not reported to save space but can be found in the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]). The number in parenthesis next to the variables used for reweighting indicates the number of categories for each variable.
Following the extended NRBA, countries and economies were classified into different categories, reflecting a holistic assessment of how susceptible their proficiency estimates were to non-response bias, and the corresponding level of caution that is advised when interpreting the results:
pass: meaning that the analysis provided no strong evidence of non-response bias
low caution: meaning that some caution should be taken in interpreting the results, as non-response bias may be present
medium caution: meaning that proficiency estimates are more susceptible to non-response bias and more caution should be exerted
high caution: meaning that the likelihood of non-response bias is higher.
Details on the indicators included in the analysis and on the criteria followed in classifying countries and economies can be found in the Technical Report of the 2023 Survey of Adult Skills (OECD, forthcoming[1]).
Results of the NRBA are presented in Table 5.12. In interpreting the outcomes of the NRBA and the resulting classification of countries, readers should note that:
The classification reflects an assessment of the likely existence of non-response bias, and not of its magnitude. In other words, one cannot conclude that countries in the “high caution” category necessarily have a larger non-response bias than countries in the “low caution” category.
The criteria and the thresholds used for the classification have been approved by consensus by the independent experts who are member of the Technical Advisory Group.
In any such classification, threshold effects will always exist, meaning that countries might be classified in different categories even though the differences in their underlying indicator are very small.
This classification reflects a judgement on the collected data, and not on the quality of the work done by national centres and data collection agencies, which all countries completed satisfactorily and in accordance with the requirements specified in the technical standards and guidelines.
Note: The extended NRBA was not required for countries with response rates above 70%. For this reason, results of the NRBA are not applicable for Korea. The extended NRBA was conducted for the Slovak Republic, despite a response rate of 70%, because the country did not fully meet the sampling standards.
In most participating countries and economies, a proportion of respondents are unable to undertake the assessment for literacy-related reasons, such as being unable to speak or read the test language(s), having difficulty reading or writing, or having a learning or mental disability. Some of these respondents may be able to complete the background questionnaire or key parts of it, presumably with the assistance of an interviewer who spoke the respondent’s language, a family member or another person. This form of non-response could introduce bias since it is systematically concentrated among those with low literacy proficiency in the survey language (presumably migrants or people with very poor reading skills). In the first cycle, the share of such non-respondents amounted to less than 2% in most countries but exceeded 4% in four countries and economies (OECD, 2019[2]).
To reduce the bias induced by such literacy-related non-response, the 2023 Survey of Adult Skills introduced a new instrument called the doorstep interview. This is a short questionnaire offered in 43 languages which collects basic background information: gender, age, years of schooling, employment status and country of origin. This questionnaire can be easily completed by individuals who do not speak the language(s) of the assessment and are, therefore, unable to answer the regular background questionnaire and the direct skills assessment. The information collected through the doorstep interview was used to estimate the literacy and numeracy proficiency of these non-respondents. This innovation allowed the survey results to cover the entire target population.
The share of doorstep interview cases in each country and economy are shown in Table 5.13.
Note: The high number of unweighted cases in Denmark is due to the decision to oversample the immigrant population.
While clearly an improvement with respect to the first cycle of the Survey of Adult Skills, the introduction of the doorstep interview poses a challenge for comparing estimates over time, as the sampled populations are no longer fully comparable: adults who completed the doorstep interview in the second cycle would have been handled as literacy-related non-respondents in the first cycle. The OECD recommends excluding cases who only completed the doorstep interview in all analysis comparing results from the second and first cycles of the Survey of Adult Skills.
The data from participating countries and economies were subject to a process of adjudication to determine whether they were of sufficient quality to be reported and released to the public. Data adjudication can be seen as the culmination of the quality assurance and quality-control arrangements put in place to ensure that the survey produces reliable and valid data. These had included establishing the Technical Standards and Guidelines (TSGs) covering all aspects of the implementation of the survey and collecting information to monitor compliance with them, as well as identifying problems as they emerged and recommending corrective action. The international consortium also provided guidance, training, assistance and tools to help countries comply with the requirements of the TSGs.
The adjudication process used a broad definition of quality: “fitness for use”. Although countries’ compliance with the TSG was an important component of the quality assessment, the goal was to go beyond compliance to assess whether the data produced were of sufficient quality in terms of their intended uses or applications.
In assessing overall data quality, the focus was on four key areas:
sampling
survey operations and interviewer training standards
instrumentation
data output and the operation of the delivery platform.
In each of these areas, countries and economies were assessed against a set of quality indicators which reflected the major requirements of the TSGs. All countries and economies either fully met the required quality standards or met them to a degree that was believed not to compromise the overall quality of the data. The data from all participating countries and economies were determined to have met the quality standards required for reporting and public release. The project’s Technical Advisory Group reviewed the assessments of the quality of all national data before submitting them to the Board of Participating Countries.
In some countries, there were some specific concerns because unusual response patterns were identified, suggesting that some respondents may not have exerted a reasonable level of effort in answering the literacy, numeracy and adaptive problem solving assessment. This may call into question whether their responses accurately reflect their proficiency. The OECD Secretariat therefore conducted additional quality checks on the data, expanding on the quality-control procedures set in the Technical Standards and Guidelines. To identify such cases, the OECD relied on the following criteria: a very short time spent on the assessment, a high share of very rapid responses, a high share of missing answers, and locator failure (i.e. failure to answer a set of easy questions) from highly educated, native-born respondents. Anomalies were mostly found in the responses to the cognitive assessment; the pattern of responses to the background questionnaire did not raise particular concerns.
Disengaged respondents will always exist in surveys, and it is difficult to establish objective criteria to assess whether a reasonable level of effort was exerted. In some countries (Israel, Lithuania, New Zealand, Poland, the Slovak Republic, and Spain), it was found that many of these respondents were clustered around a few interviewers, suggesting that the problem may stem from such interviewers not following the PIAAC protocols. In particular, interviewers were identified for which a high share of their cases met at least two of the criteria mentioned above. This cast doubts on the quality of all data collected by such interviewers.
In Lithuania, New Zealand, the Slovak Republic and Spain, all cases from the identified interviewers (406 in Lithuania, 301 in New Zealand, 356 in the Slovak Republic, and 385 in Spain) have been excluded from the data used to estimate the population model, which establishes the relationship between the variables from the background questionnaire and performance on the direct assessment to generate proficiency estimates (plausible values; see the Survey of Adult Skills 2023 Technical Report (OECD, forthcoming[1]) for more detail). This exclusion enhances the robustness of the model, by ensuring it is estimated based only on the cases considered to be of sufficient quality. In the absence of definitive evidence of data falsification or other forms of interviewer misconduct, the responses to the cognitive assessment items still contributed to the estimation of plausible values for these and all respondents.
In Israel, six interviewers were identified as having a relatively large share of cases with unusual response patterns, using the same criteria that led to the identification of cases in the other countries. All data from these interviewers (748 cases in total) have been excluded from the data used to estimate the population model, which establishes the relationship between the variables from the background questionnaire and performance on the direct assessment to generate proficiency estimates (OECD, forthcoming[1]). This exclusion enhances the robustness of the model, by ensuring it is estimated based only on the cases considered to be of sufficient quality.
Moreover, stronger evidence was collected that three of these interviewers breached data collection protocols throughout the survey or were implausibly productive (conducting a very large number of interviews in a relatively short period). As this raised additional concerns about the quality of data, the responses to the cognitive assessment items for all cases of these interviewers were excluded from the database (572 in total). Plausible values for these cases were then estimated using only their responses to the background questionnaire (for which no unusual patterns were detected) and the parameters estimated by the population model.
In Poland, nine interviewers were identified as having a relatively large share of cases with unusual response patterns of respondents, using the same criteria that led to the identification of cases in the other countries. All data from these interviewers (774 cases in total) have been excluded from the data used to estimate the population model, which establishes the relationship between the variables from the background questionnaire and performance on the direct assessment to generate proficiency estimates (OECD, forthcoming[1]). This exclusion enhances the robustness of the model, by ensuring it is estimated based only on the cases considered to be of sufficient quality.
Moreover, stronger evidence was collected that six of these interviewers in Poland breached data collection protocols throughout the survey. For instance, some of these interviewers were implausibly productive, conducting many interviews on a single day. Others did not record interviews or obtain respondents’ phone numbers, which made validation of interviews more difficult. Yet another interviewer was found to have falsified seven cases during data collection (cases which were immediately removed from the dataset as part of the quality-control process and are not included in the 774 cases under consideration in this note). Twenty-seven other cases collected from this interviewer were, however, validated and remained in the dataset. Since these factors raise concerns about the quality of all cases completed by these six interviewers, the responses to the cognitive assessment items for all cases of these six interviewers were excluded from the database (559 in total). Plausible values for these cases were then estimated using only their responses to the background questionnaire (for which no unusual patterns were detected) and the parameters estimated by the population model.
In Poland, other cases with unusual response patterns that could suggest possible disengagement or lack of a reasonable level of effort during the assessment were identified. As these cases were not clustered within any particular interviewer, they were left in the dataset and treated as all other cases, given the difficulty of establishing objective criteria to determine whether reasonable effort was exerted, and whether the results of the assessment truly reflect the proficiency of respondents. While similar cases are present in all countries, the number of such cases in Poland can potentially have a significant impact on the estimated proficiency of the overall population. This should be kept in mind when interpreting Poland’s results. For this reason, in OECD (2024[3]) results for Poland are flagged with an asterisk.
References
[3] OECD (2024), Do Adults Have the Skills They Need to Thrive in a Changing World?: Survey of Adult Skills 2023, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/b263dc5d-en.
[2] OECD (2019), The Survey of Adult Skills: Reader’s Companion, Third Edition, OECD Skills Studies, OECD Publishing, Paris, https://doi.org/10.1787/f70238c7-en.
[1] OECD (forthcoming), Survey of Adult Skills 2023 Technical Report, OECD Publishing, Paris.
← 1. The Technical Standards and Guidelines are available at https://www.oecd.org/en/about/programmes/piaac/piaac-data.html#manuals