Mapping Quality Assurance Indicators for Non-formal Adult Learning

1. Quality assurance in non-formal adult learning

Abstract

Formal adult education and training is generally subject to quality assurance activities, however, this is often not the case for non-formal adult learning. This chapter presents why quality assurance is also relevant for non-formal adult learning. In addition, it discusses the most common approaches to quality assurance of non-formal adult learning as well as common classifications of the quality criteria included in quality assurance frameworks.

Adult training plays a crucial role in helping adults stay relevant in the labour market, either by updating adults’ skills or by providing them with new ones. Among its potential benefits, adult learning can ease access to better quality jobs and improve employability and productivity, especially at a time of changing labour markets, as well as increase adults’ sense of self-fulfilment.

However, adult training must be of high quality and ensure that learning outcomes are attained for adults, and employers, to reap these potential benefits. Quality assurance (QA) intends to safeguard the quality of education or training and includes all “activities involving planning, implementation, evaluation, reporting, and quality improvement, implemented to ensure that all education and training (content of programmes, curricula, assessment and validation of learning outcomes, etc.) meet the quality requirements expected by stakeholders” (Cedefop, 2011[1]). Formal adult education and training, provided in schools and universities, is generally subject to quality assurance activities (OECD, 2021[2]), but this is not the case for non-formal adult learning.1

There are many reasons behind the lack of a standardised approach to the QA of non-formal adult learning. These include the high costs of implementing QA mechanisms, the reluctancy from non-formal training providers to be subject to more bureaucracy, the resistance from formal training providers to the formal certification of the quality of non-formal competitors, the fragmentation of responsibilities regarding adult learning policies across different government levels or the lack of monitoring data (OECD, 2021[2]).

Despite these challenges, some countries and some smaller within-country initiatives have succeeded in implementing a QA framework for non-formal adult training. Quality provision is key to create trust in the adult training system and QA frameworks can put high-quality training providers at an advantage with respect to competitors by signalling this high quality to potential trainees.

This report analyses 12 quality assurance systems that cover non-formal adult learning with the goal to identify the quality areas and criteria typically included in these systems, as well as the evidence that training providers can submit to prove compliance. The quality assurance frameworks reviewed include Ö-Cert and Cert NÖ (Austria), Qualiopi (France), the European Quality Mark (Iceland), Quality and Qualifications Ireland (Ireland), Label de Qualité (Luxembourg), the General Directorate for Employment and Industrial Relations (Portugal), the Offering Quality Education to Adults model (Slovenia), EduQua (Switzerland), the NRTO Quality Label (the Netherlands) and the ISO quality labels 21 001 and 9 001. Relevant information was gathered through desk research and online interviews with experts.

The different quality areas, criteria and indicators included in these quality frameworks are analysed and structured in the report using the four phases of the European Quality Assurance Reference Framework for Vocational Education and Training (EQAVET). These are: planning, implementation, evaluation and review. Each chapter of the report analyses the quality areas, related criteria and potential evidence linked to a given phase in the EQAVET cycle.

Approaches to quality assurance

The systems reviewed for this report showcase a variety of approaches to quality assurance and the use of different quality assurance instruments. A review of quality assurance practices in non-formal adult learning (OECD, 2021[2]) identified three main approaches to adult learning:

1. The regulatory approach, which imposes minimum requirements that providers must satisfy to be allowed to operate or to be eligible to receive public funds.

2. The advisory approach, which uses guidelines and provides examples of good practices to incentivise and guide training providers in pursuing quality improvement activities.

3. The organic approach, which allows providers to pursue their own quality improvement activities based on their quality needs.

To operationalise these QA approaches, QA frameworks rely mostly on two quality assurance tools: quality certificates or labels and self-evaluations. These can additionally be combined with other QA activities, such as quality inspections or audits.

Systems that use a regulatory approach tend to use quality certificates or labels. These quality labels are often used to determine the eligibility of the training provider to receive public funds, as the minimum requirements imposed by the quality certification guarantees a standard level of quality of training services. A study of quality assurance systems for non-formal adult education (Espinoza and Martinez-Yarza, 2023[3]) found that most OECD countries rely on quality labels and certifications to ensure a minimum level of quality in their non-formal training provision. However, the effectiveness of quality certificates or labels in ensuring a minimum level of quality depends on whether the information in which the certificate relies on is valuable, credible and accurate (OECD, 2021[2]). Otherwise, the quality certificate could be perceived as unreliable, limiting the benefits accrued by label holders and the effectiveness of the label at guaranteeing a minimum level of quality. In addition, quality certificates or labels provide only external incentives for training providers to achieve a minimum level of quality, limiting the development of a quality culture among their holders.

On the other hand, systems that use an advisory approach generally use self-assessments. These self-assessments typically include a quality improvement cycle through which training providers must continuously define quality improvement actions. The advantage of this approach is that it develops internal incentives to improve quality, pushing training providers to develop a quality culture. However, if the system is fully based on self-assessments, neither potential trainees nor public institutions or potential employers can verify whether training providers are providing good-quality services. For this reason, many QA frameworks based on self-assessments also include an initial external assessment to verify that training providers set minimum quality requirements.

The mapping of quality criteria and indicators

Quality assurance systems include quality criteria that state the outcome that should be observed, and indicators or requirements, which allow to measure whether the respective quality criterion is satisfied by the training provider. These quality criteria and indicators are typically classified by topic or quality area. The quality areas covered by each individual quality assurance framework depend on the specific context in which the framework operates. However, some areas are generally included, given their importance for the training process. This is the case, for example, of indicators linked to the training programme, the qualifications and training of the training provider’s staff, the existence and use of a quality management system within the training provider and the training provider’s leadership and management structure and strategy, which are covered by many OECD quality frameworks for non-formal adult learning (Espinoza and Martinez-Yarza, 2023[3]).2

In addition to quality areas, criteria and indicators can also be classified depending on whether they reflect inputs, processes or outputs. The inclusion of indicators covering the three stages has been recommended in the context of the analysis of formal education policies (Gouëdard, 2021[4]). Input indicators refer to production factors that must be set up before the training provider starts its activities, such as educational programmes, infrastructure and equipment or staff. Process indicators refer mostly to the implementation of the training, such as how the programme is implemented in practice, whether there are any planned skill development activities for trainers or whether the training provider offers guidance to support trainees. Finally, output indicators analyse the outcomes of trainees leaving the training programme, such as their employability results or whether they completed the training programme.

EQAVET follows a similar structure, as it classifies the indicative descriptors that VET systems and training providers should satisfy to be of high quality in four phases: planning, which would correspond to inputs, implementation, which is related to processes, evaluation, that focuses on outputs, and review. The inclusion of the first three phases ensures that all stages in the educational process are covered by the quality assurance framework, while the last phase, review, provides inputs for the continuous improvement of the VET system or provider.

The actual structure of the quality assurance framework, including how its criteria and indicators are classified, should be developed jointly with stakeholders. This ensures that the quality assurance system is pertinent, covering the most important aspects that impact the quality of non-formal adult training and also incentivises providers to adopt the framework. To develop effective indicators, the OECD has developed a review framework in the context of analysing formal education and training policies, presented in Box 1.1, which provides guidance to define effective indicators.

Box 1.1. The development of indicators: the OECD review framework

In the context of formal education and training, the OECD has outlined the characteristics that indicators should have to provide reliable information about the effectiveness of education policies and practices. These are:

• Description: The description of the indicator is precise and understandable as it is.

• Purpose/relation to strategic goals: The indicator is goal-specific, well-aligned to the objectives of the policy and helps monitor progress towards these objectives.

• Definitions and scope: The indicator is clearly defined and has a well-delimited scope.

• Calculation methodology: If there is any calculation involved, its methodology is transparent and the results of the calculations are reliable and comparable.

• Monitoring level (school/region/national): The level at which the indicator is analysed is coherent with the objectives of the associated policy.

• Data availability and breakdown: The data and their respective sources that feed into each indicator are identified, and possible breakdowns of the data are explored (e.g. by gender or socio‑economic background).

• Frequency and coverage: The indicator can be updated as often as relevant to monitor the achievement of the related objectives and the targeted population by the policy is covered by the indicator.

• Interpretations/limitations: The indicator is simple to interpret, if this is not the case, clear guidelines discuss its limitations to reduce possible misinterpretation.

Source: OECD (2021[5]), “Enhancing data informed strategic governance in education in Estonia”, OECD Education Policy Perspectives, No. 47, https://doi.org/10.1787/11495e02-en.

Notes

← 1.  Non-formal adult education or training is institutionalised, intentional and planned education or training either of short duration (less than one semester) or not recognised by relevant authorities (OECD, 2023[15]).

← 2.  These quality areas and their respective criteria and indicators will be described in more details in subsequent chapters.