Results from TALIS Starting Strong 2024

Annex B. Technical notes on sampling, response rates and adjudication

Sampling procedures

The objective of the Starting Strong Teaching and Learning International Survey (TALIS Starting Strong) 2024 was to obtain a representative sample in each participating country or subnational entity of staff and leaders providing early childhood education and care (ECEC) for each level of ECEC (i.e. ISCED level 02 or settings for children under age 3) in which the country or subnational entity participated. The international sampling plan used a stratified two-stage probability sampling design. ECEC settings served as the first‑stage units (primary sampling units), randomly selected from within strata. For countries and subnational entities with age-integrated ECEC systems that participated in the data collection for both pre‑primary education (ISCED level 02) and settings for children under age 3, settings serving both age groups were randomly split between the two samples so that each setting could be selected for participation in only one level of ECEC. Staff were then randomly selected from the list of eligible staff within those settings, serving as secondary sampling units. A more detailed description of the survey design and its implementation can be found in the TALIS 2024 Technical Report (OECD, forthcoming[1]).

Participating countries and subnational entities had the option to restrict the coverage of their national implementation of TALIS Starting Strong for reasons of practicality, safety or economy. However, they were encouraged to minimise such exclusions and ensure that their national survey population covered at least 95% of ECEC staff. National project managers were required to document the reasons for each exclusion.

Definition of early childhood education and care settings, leaders and staff

The international target population of TALIS Starting Strong restricts the survey to those staff and leaders who work in officially registered settings providing ECEC. Settings exclusively for children with special educational needs are deemed out of scope. For the ISCED level 02 target population, home-based settings (i.e. within the homes of the respective staff) are not considered to be part of the target population. Officially registered home-based settings are considered in-scope for the target population of settings for children under age 3.

Staff in pre-primary settings (ISCED level 02) include all persons working regularly in a pedagogical way, providing learning opportunities or care, with children of ISCED level 02. Staff for children under age 3 include all persons working regularly in a pedagogical way, providing learning opportunities or care, with children in this age group. Staff who provide learning opportunities for both target populations in their settings are included in the TALIS Starting Strong population for the age group for which their setting is sampled. There is no minimum cut-off for how much time staff need to be engaged at either level of ECEC.

The following types of staff are considered out of scope for TALIS Starting Strong 2024: temporary substitute educators (to replace staff on sick leave), staff on long-term leave (e.g. disability, sabbatical) who are not expected to be back working at the time of data collection, persons working as nannies or in informal arrangements, staff providing services exclusively within a child’s home (e.g. home visiting), volunteer persons who occasionally come in for a special activity, and auxiliary staff (e.g. cleaners, cooking staff, etc.) who do not interact regularly in a pedagogical manner with the children.

The leader at each setting (i.e. the person with the most responsibility for administrative, managerial and/or pedagogical leadership at the ECEC setting) was automatically selected for participation. At the time of sampling, if the leader was designated as also having staff duties (i.e. meeting the sampling definition of ECEC staff), s/he was assigned to participate using the main version of the combined questionnaire (CQM) in countries and subnational entities that used this version of the questionnaire (OECD, forthcoming[2]). At the time of sampling, if the leader was the only staff member listed in an ECEC setting, s/he was assigned to participate using the short version of the combined questionnaire (CQS) in countries and subnational entities that used this version of the questionnaire.

Sample size requirements

For each level of ECEC, the minimum sample size was set at 180 ECEC settings and 1 200 staff members in total. In ECEC settings with eight or more eligible staff members, at least eight staff members were to be selected; in settings with fewer eligible staff members, all of them were to be asked to participate.

Participating countries and subnational entities could choose to augment their samples by selecting more settings, or by selecting more staff within each selected setting, or by increasing both. In some cases, because the average number of staff in ECEC settings was low, the number of settings sampled was increased.

Adjudication process

Adjudication is the process of determining whether the data released were “fit-for-use” as intended. The issues examined concerned, among others, adapting the questionnaire to national context, translation and verification, quality of the sampling frame, handling of out-of-scope and refusal units (i.e. staff and/or setting), within‑setting sampling, data collection, data cleaning, the reports of quality observers, participation rates and overall compliance with the technical standards.

During the adjudication session, each individual dataset (i.e. one per level of ECEC per country or subnational entity) was submitted to the same examination. Leader/setting participation was adjudicated independently of staff data.

Once each survey process had been assessed, a recommended adjudication rating was formulated. While the rating was mostly dictated by the participation rate thresholds set in the TALIS 2024 Technical Standards (see Table A B.1 and Table A B.2), the adjudication committee at times improved the rating of a dataset if, after expert consideration, unique and favourable conditions were met (e.g. closeness to the threshold value and/or the non-response bias analysis report showing evidence for negligible bias risks).

To note, the adjudication rating is made at the ECEC system level. However, participation rates are just one potential source of non-response bias and potential non-response bias is specific to each statistical estimate from the survey data. Therefore, some estimates of systems with an adjudication rating of “good” might still have a high non-response bias; correspondingly, some estimates of systems with an adjudication rating of “poor” might still be reliable. For example, if staff with a particular characteristic systematically do not respond to a specific question in a system with a “good” rating, but that question is answered by all staff in a system with a “poor” rating, estimates based on that question may be more affected by non‑response bias in the “good” system compared to the “poor” system. For more detailed information, please refer to the TALIS 2024 Technical Report (OECD, forthcoming[1]).

At its January 2025 meeting in Paris, the TALIS 2024 Technical Advisory Group (TAG) recommended that data from all participating countries and subnational entities be weighted and displayed in tables, regardless of the adjudication ratings. The TAG further recommended clearly indicating to readers when data were rated as “insufficient,” due to the greater risk of non-response bias in these samples. TALIS Starting Strong 2024 data from New Zealand were deemed an exception to this general principle, although the TAG supported presenting these data with the caveat that they are not representative of the population targeted by the survey.

Based on these recommendations, countries and subnational entities with “insufficient” ratings are flagged (*) in tables and figures with the note: Estimates should be interpreted with caution due to a higher risk of non-response bias. Data from New Zealand are visually separated in tables and figures and are flagged (**) in tables, figures and the text with the note: Data only represent respondents included in the sample and not the population targeted by the survey. No distinctions are made in the report for data rated as “good”, “fair” or “poor”; more detailed implications of these ratings are described in the TALIS 2024 Technical Report (OECD, forthcoming[1]).

Table A B.1. Adjudication rules for setting or leader data

Setting/leader participation before replacement	Setting/leader participation after replacement	Risk of non-response bias	Rating
≥75%	≥75%		Good
≥50% but <75%	≥75%		Fair
≥50% but <75%	≥50% but <75%	Low	Fair
≥50% but <75%	≥50% but <75%	High	Poor
<50%			Insufficient

Source: TALIS 2024 Technical Standard 3.26.

Table A B.2. Adjudication rules for staff data

Setting/leader participation before replacement	Setting/leader participation after replacement	Staff participation after setting replacement	Risk of staff non-response bias	Rating
≥75%	≥75%	≥75%		Good
≥75%	≥50% but <75%			Fair
≥50% but <75%	≥75%	≥75%		Fair
≥50% but <75%	≥75%	≥50% but <75%	Low	Fair
≥50% but <75%	≥75%	≥50% but <75%	High	Poor
≥50% but <75%	≥50% but <75%			Poor
<50%	≥75%			Poor
<50%	<75%			Insufficient

Source: TALIS 2024 Technical Standard 3.27.

Participation rates and adjudication ratings

Tables A B.3, A B.4, A B.5 and A B.6 present the participation rates and the adjudication rating per participating country and subnational entity, level of ECEC, and population (i.e. settings/leaders and staff). Detailed results of unweighted and weighted participation can be found in Annex G of the TALIS 2024 Technical Report (OECD, forthcoming[1]).

Table A B.3. Settings for children under age 3: Leader participation and recommended ratings

Participating country or subnational entity	Number of participating leaders	Estimated size of leader population	Leader participation before replacement (%)	Leader participation after replacement (%)	Recommended rating
Flemish Community (Belgium)	125	5 327	34.9	53.4	Insufficient
Germany1	165	46 847	49.5	74.2	Poor
Ireland	183	2 154	39.6	59.5	Insufficient
Israel	289	5 953	77.2	85.5	Good
New Brunswick (Canada)	84	489	31.3	35.9	Insufficient
New Zealand2	34	m	21.1	24.8	Insufficient
Norway	76	4 812	28.0	40.9	Insufficient
Quebec (Canada)	85	10 317	23.4	59.8	Insufficient

¹ The rating was defined as “poor” although the participation rate before replacement did not completely reach 50%.

² Due to low participation rates, a non-response adjustment was not implemented and the estimated population size, therefore, remains unknown.

Note: A setting was deemed a participating setting if the leader returned their questionnaire with at least one question answered.

Table A B.4. Settings for children under age 3: Staff participation and recommended ratings

Participating country or subnational entity	Number of participating settings	Number of participating staff	Estimated size of staff population	Setting participation before replacement (%)	Setting participation after replacement (%)	Staff participation in participating settings (%)	Overall staff participation (%)	Recommended rating
Flemish Community (Belgium)	128	499	16 765	35.8	55.3	83.6	46.2	Insufficient
Germany	167	876	310 220	50.5	76.0	87.8	66.7	Fair
Ireland	207	899	11 865	44.0	66.3	75.2	49.8	Insufficient
Israel	297	1 505	25 597	78.7	87.9	85.4	75.1	Good
New Brunswick (Canada)	88	275	2 345	34.7	38.6	85.4	33.0	Insufficient
New Zealand1	33	164	m	17.8	23.9	73.0	17.4	Insufficient
Norway	68	367	41 262	26.2	36.8	73.4	27.0	Insufficient
Quebec (Canada)	82	311	41 882	23.3	49.7	75.0	37.3	Insufficient

¹ Due to low participation rates, a non-response adjustment was not implemented, and the estimated population size, therefore, remains unknown.

Note: A setting was deemed a participating setting if at least 33% of the selected staff returned their respective questionnaires with at least one question answered.

Table A B.5. Pre-primary education (ISCED level 02): Leader participation and recommended ratings

Participating country or subnational entity	Number of participating leaders	Estimated size of leader population	Leader participation before replacement (%)	Leader participation after replacement (%)	Recommended rating
Chile	160	8 748	78.6	88.3	Good
Colombia	157	75 191	55.2	68.3	Poor
Denmark	150	3 378	55.5	82.4	Fair
Finland	204	4 160	92.4	97.1	Good
Flemish Community (Belgium)	158	2 288	62.7	83.2	Fair
Germany	138	50 860	45.2	70.3	Insufficient
Ireland	149	3 950	36.0	59.9	Insufficient
Israel	285	12 399	86.1	92.2	Good
Japan	189	37 007	71.4	95.0	Fair
Morocco	429	21 931	93.3	98.3	Good
New Zealand1	38	m	19.7	22.9	Insufficient
Norway	84	4 929	31.5	46.4	Insufficient
Spain	481	13 714	98.2	98.2	Good
Sweden	197	12 467	91.7	92.8	Good
Türkiye	267	23 809	99.4	99.4	Good

¹ Due to low participation rates, a non-response adjustment was not implemented and the estimated population size, therefore, remains unknown.

Note: A setting was deemed a participating setting if the leader returned their questionnaire with at least one question answered.

Table A B.6. Pre-primary education (ISCED level 02): Staff participation and recommended ratings

Participating country or subnational entity	Number of participating settings	Number of participating staff	Estimated size of staff population	Setting participation before replacement (%)	Setting participation after replacement (%)	Staff participation in participating settings (%)	Overall staff participation (%)	Recommended rating
Chile	143	741	54 091	70.7	79.5	88.0	70.0	Fair
Colombia	150	553	162 032	55.3	65.2	97.7	63.6	Poor
Denmark	148	946	27 274	54.9	81.3	89.5	72.8	Fair
Finland	204	1 386	42 851	92.4	97.1	95.4	92.7	Good
Flemish Community (Belgium)	164	1 143	21 469	66.3	86.3	89.4	77.2	Fair
Germany	142	810	456 372	47.2	72.1	82.2	59.3	Insufficient
Ireland	170	690	18 557	41.6	69.5	73.9	51.4	Insufficient
Israel	291	1 153	42 272	87.7	94.2	86.1	81.1	Good
Japan	189	1 283	317 235	71.4	95.0	98.9	93.9	Fair
Morocco	428	954	48 583	91.1	98.0	99.2	97.2	Good
New Zealand1	35	163	m	17.6	20.4	72.7	14.8	Insufficient
Norway	68	377	46 985	25.5	37.6	73.8	27.8	Insufficient
Spain	489	3 576	121 268	100.0	100.0	96.7	96.7	Good
Sweden	208	1 219	93 674	97.0	98.0	86.2	84.5	Good
Türkiye	265	1 289	93 250	97.8	97.8	98.4	96.2	Good

¹ Due to low participation rates, a non-response adjustment was not implemented and the estimated population size, therefore, remains unknown.

Note: A setting was deemed a participating setting if at least 33% of the selected staff returned their respective questionnaires with at least one question answered.

Notes regarding use and interpretation of the data

This section lists issues to be noted regarding the sampling or field operations that should be considered when interpreting the data reported for the following countries and subnational entities:

• Flemish Community of Belgium

  • For settings for children under age 3, settings without subsidy were excluded after the survey administration due to very low participation rates. They are not considered in the participation rate but contribute to the exclusion rate, which remained below 5% of staff.

• Germany

  • Home-based settings were excluded from the sample for settings for children under age 3, leading to an estimated exclusion rate of 6.5% of staff at this level of ECEC.

• Ireland

  • For both levels of ECEC, replacement settings were contacted before the participation of the originally sampled setting was defined. Consequently, replacements participated although the original setting participated as well. These replacements were moved into separate strata and treated as self-representing. They do not contribute to participation rates.

  • Home-based settings were excluded from the sample for settings for children under age 3. The exclusion rate remained below 5% of staff.

• Israel

  • Ultra-Orthodox settings did not participate in the survey at ISCED level 02 and were excluded after the survey administration. They are not considered in the participation rate but contribute to the exclusion rate, leading to an exclusion rate of 26.3% of pre-primary settings; the resulting staff exclusion rate is unknown.

• Morocco

  • Due to the very small size of ECEC settings (i.e. there are just one or two staff members in many settings), the sample size requirement for staff was reduced.

• New Zealand

  • Te Kōhanga Reo (Māori language immersion environments for children and families) did not participate in the survey at ISCED level 02, leading to an estimated staff exclusion rate of 12.8% for pre-primary staff. For settings for children under age 3, in addition to Te Kōhanga Reo, free kindergartens and playcentres did not participate in the survey, leading to an overall estimated staff exclusion rate of 29.8%.

• Norway

  • Home-based settings were excluded from the sample for settings for children under age 3. The exclusion rate remained below 5% of staff.

• Quebec (Canada)

  • Due to a high turnover in home-based childcare services during the time of recruitment for TALIS Starting Strong, notably attributable to demographic factors such as retirement, it is important to note that results for leaders/settings represent settings that were open in September 2022 and remained open at the beginning of 2024: At the time of data collection, 32% of home-based settings that had been selected for the sample were ineligible, mainly because they were closed.

References

[1] OECD (forthcoming), Teaching and Learning International Survey (TALIS) 2024 Technical Report, OECD Publishing, Paris, forthcoming.

[2] OECD (forthcoming), Teaching and Learning International Survey (TALIS) 2024 User’s Guide, OECD Publishing, Paris, forthcoming.