The Study will take a single snapshot of two cohorts of primary and secondary school students, at ages 10 and 15. The Study will assess students' social and emotional skills directly but also get information from their parents, teachers and school principals. This will allow us to understand the home and school contexts in which these skills develop.

The Study will seek to understand:

• The level and spread of students' social and emotional skills at two important points of students' lives
• The individual and contextual factors that influence the development of these skills in students
• The differences in the level and spread of social and emotional skills of students aged 10 and 15
• The similarities and differences in the effects of contextual factors on students at two different age groups.

The sample size for the Main Study will be 3 000 students for each age cohort.

The Study will use a two-stage stratified sampling model:

• First stage will select a random sample of schools
• Second stage will select a random sample of individual students within the selected schools.

Skills measurements and data collection on school learning contexts will be mainly administered in schools.

Information on students’ social and emotional skills will be obtained through questionnaires, using different and mutually independent sources via the student, parent and teacher questionnaires, while also covering the different contexts in which students live. This comprehensive approach will increase the amount and quality of the data obtained and consequently result in a better understanding of students’ social and emotional skills.

The survey will be administered online, using computers or tablets. Where needed or preferred, respondents will also have an option of using a paper-and-pencil version of the questionnaires.

The Study will be collecting data from four groups of respondents:

1. Students will provide reports on their social and emotional skills and their home and school environment.
2. Parents or legal guardians of each of the selected students will provide a report on the social and emotional skills of their child and on their home environment. Only one of the two parents/guardians will be completing the survey, and the parents/guardians themselves will decide who.
3. A teacher that knows a student the best or that has spent the most time with a student will also assess the social and emotional skills for each sampled student and provide a report on the characteristics of their school environment.
4. Finally, school principals will also participate in the Study by providing information on the relevant characteristics of each sampled school.

The social and emotional skills of school-age students are assessed both directly and indirectly. The direct assessment refers to students’ reports on their own behaviours while the indirect assessment will capture parent and teacher reports on students’ behaviours. Thus, there are three sources of information about each participating student and for each of the assessed skills. Importantly, all three respondents (students, parents and teachers) will answer the same questions (in the form of statements) thus offering the possibility for direct comparability of the used measures. Respondents are asked to indicate the degree to which they agree or disagree that these statements apply to the assessed student.

The administration of the student questionnaires will be online through desktop or laptop devices in a controlled school setting. Parent, teacher and school principal questionnaires are also primarily conducted online, but respondents will be offered a paper and pencil option to choose from in case of necessity or personal preference.

The Study will be administrated to groups of students belonging to the same cohort. Parents, teachers and principals will complete their questionnaires individually.

Triangulation is a situation in which an estimate of certain measure is provided from three different sources. In the Study, students’ socio-emotional skills are assessed by students themselves (direct assessment), but also by one of their parents and by a teacher that knows these students the best (indirect assessment). Triangulation is one of the most important study design aspects of the SSES.

The indirect assessment of students through parents and teachers will improve content validity of the assessment of students’ socio-emotional skills as it will provide information on students’ behaviours across different contexts, i.e. both in school and at home with family and friends. This is a critical aspect as students may behave differently in different settings and choosing information from any one of those settings may provide a somewhat biased representation of students’ social and emotional skills. The opportunity to combine information on students’ skills from personal, school and family perspectives will allow a better representation and understanding of students’ behaviours in the most important contexts for school-age students. Likewise, the ability to obtain information from other sources that know the student well, aside from the information obtained from students’ themselves, allows us to control for various sources of measurement error presented in self-reports, such as social desirability or unrealistic self-perceptions.

The most often used response format in educational and psychological assessment is the Likert agree/disagree scale with either four or five options. While many response formats are possible (e.g. dichotomous, frequency, forced-choice, etc.), a five-point agree/disagree Likert scale is used in the Study. Using this format allows us to align our instrument with most of the existing measures designed for assessing social and emotional skills and related psychological constructs. This response scale has the largest amount of empirical evidence about its measurement properties, making it easier to predict its benefits and to account for its drawbacks. Another important aspect of this response scale is its familiarity across survey respondents of different ages, backgrounds and nationalities. Also, using a 5-point response scale increases the amount of information per each question (compared to dichotomous, or 3- and 4-point scales) thus reducing response burden and improving the efficiency of the assessment instruments.

Anchoring vignettes

Anchoring vignettes are depictions of concrete behavioural situations especially designed to account for reference bias by adjusting assessment responses at the respondent level. Reference bias denotes a situation in which people from different countries answer the same question using different reference standards. For example, when asked how talkative they are, two equally talkative respondents might give different answers if they come from cultures where the general standard/reference for this behaviour differs. Reference bias represents one of the primary sources of cross-cultural incomparability of self-reports or other-report measures.

Anchoring vignettes provide concrete behavioural situations that help respondents to establish a common reference standard before answering the actual questions of the questionnaire. The use of a common reference standard by respondents has shown to mitigate response bias and improve cross-cultural comparability. These reference standards (e.g. what it concretely means for a particular person to be talkative) are then taken into account when calculating skill estimates based on respondents’ reports on actual assessment items. In the Study, there are three specific vignettes (representations of concrete behaviours) for each Big Five domain, designed to represent high, medium and low ends of the corresponding Big Five dimensions. Results of these three vignettes are then used for correcting responses to the skills’ scales in each domain.

Behavioural indicators

Behavioural indicators refer to a wide range of concrete manifestations of assessed social and emotional skills in the daily lives of students. Examples of these indicators are “Does homework on time”, “Skips school classes”, “Does he/she have friends from different backgrounds”, “Does s/he have trouble sleeping”, etc.

In the Study, behavioural indicators that correspond to critical social skills such as school absenteeism, inattention, disruptive actions, peer relations, etc. are included in the student, parent and teacher questionnaires. The use of behavioural indicators is especially relevant in the policy context since it would allow better knowledge of what the concrete behavioural implications of different social and emotional skills (or their lack off) are, thus offering a more straightforward way of relating survey results with possible policy actions and implementation.

All instruments in the Study are newly developed for administration in the Main Study. Wherever possible, in the process of instrument development existing scales or questions have been used either in their original form or in a somewhat modified form, more fitting to the Study’s purposes. This especially concerns a number of questions in the parent, teacher and school questionnaires from the PISA survey that are found to be fitting for inclusion in the Study.

Development of assessment instruments that will be able to obtain valid, reliable and comparable estimates of students’ social and emotional skills is especially challenging. This is why this process in the SSES was designed in a comprehensive and elaborate way, involving multiple rounds of empirical testing in various formats (both qualitative and quantitative). Assessment instruments are developed specifically to be applicable with 10- and 15-year olds, thus with a special focus on their simplicity, clarity, and an appropriate reading level. In particular, the instrument development process involved:

• Cognitive interviews (36 in-depth interviews with 10-year-old students),
• Item trial (7 participating cities and countries with 300 students per cohort),
• Field test (all 11 participating cities and countries with 500 students per cohort). 

Leading experts in the field are responsible for the instrument development, while the Technical Advisory Group is monitoring and providing advice during the implementation. Prominent experts in the areas of psychometrics, social and emotional skills assessment, cross-cultural psychology, sociology and education compose the Technical Advisory Group.

The Study will assess students in the language in which they are taught. Survey questions will therefore be translated into languages students, teachers and parents will be comfortable understanding and responding in.

Study sites will translate student assessments and contextual questionnaires, and the International Contractor will provide verification of these initial translations. Given the importance of having correct translations and the adaptation of assessment materials for achieving cross-cultural comparability of skill measures, the Study will apply a rigorous approach.

The process starts with the team translation approach in each of the participating sites. Three translators are hired by local teams and trained by the International Contractor. Each translator will then translate 2/3 of the assessment material (direct and indirect scales and contextual questionnaires), so that each question/item is translated independently by two translators. They are then to convene and adjudicate on the common version of their initial translations. This version is further reviewed by a local psychometrician or skill assessment expert to ensure that the translations are in line with the intended conceptual meaning of the original items.

After revision of the translations by a local subject-matter expert, translations are sent to the International Contractor who employs a professional translation service to verify the translations in each of the languages. Finally, in the last step, an independent translation referee checks the final agreed translations and adjudicates on any remaining discrepancies between the local teams and the International Contractor.