Evaluations are an important tool of quality assurance in adult learning. In particular, the practice of self-evaluations has been widely adopted throughout Europe, especially in non-formal training, since it allows providers to assess their own quality and implement plans to improve it over time that fit with their needs and constraints. Evaluations undertaken by external bodies are also common and valued because of their hybrid nature: they resemble self-evaluations in their functioning, but they are similar to quality certificates in their typically mandatory approach. In both cases, the production of guidelines and other support materials by public authorities has proved very effective to facilitate providers’ own evaluations.
Improving the Quality of Non‑Formal Adult Learning
Learning from European Best Practices on Quality Assurance
Abstract
In Slovenia, self-evaluation is commonly used among education and training providers and was until recently part of an advisory approach to quality assurance. This practice is now very much rooted in the sector partly thanks to a quality framework applying to adult learning that was introduced in 2001 for explicit use for self-evaluation by entire institutions or specific programmes. Moreover, the Slovenian Institute of Adult Education launched the Offering Quality Education to Adults (OQEA) initiative in 1999 to advise educational organisations on how to self-evaluate their own quality and determine their future development. The OQEA approach to self-evaluation is based on the principles of definition, assessment, maintenance and development of quality of one’s own work. During this process, the management and the employees reflect on their own mission, vision and values they want to develop through their education of adults. The approach includes tools for planning and implementation of continuous monitoring and in-depth assessment of the quality of services and planning of measures for constant improvement to ensure the satisfaction of organisations, of the adult learner and of business partners and the environment.
The Slovenian Institute of Adult Education also started a systematic collection of areas and indicators of quality in adult education that organisations can use in self-evaluation. There are almost 100 indicators and criteria to choose from, which are grouped into 11 overarching quality areas (Slovenian Institute of Adult Education, 2013[1]). These latter quality areas are structured depending on whether they represent: (i) transverse factors of quality, (ii) input – or infrastructural – factors, (iii) process factors, or (iv) output factors (Figure 3.1). Transverse factors include activities and processes that cannot simply be placed among the input, process or output factors, but they touch all of them. This is particularly the case of management and administration, which define the quality of the management processes from organisational and contents point of view, on different levels, in different processes. Input factors are the production factors that must be ensured before the education even begins, and consist of educational programmes, promotion of adult education, stimulating adults to enter education, staff, premises and equipment. Among the process factors a key role is played by the planning of education and the implementation of education. In addition, the development work to support the education process and the support of individuals in education are also central. They in fact complement the previous two fields, given that, in adult education, support to individuals is almost as important as the education itself, and without development work it is difficult to imagine any kind of progress, particularly progress in quality development. Finally, output quality factors are those seen as the results and effects of education activities.
Source: Adapted from Slovenian Institute of Adult Education (2013[1]), Quality indicators in adult education.
OQEA has a logo that all providers of adult education can obtain if they prove that in the past three years they have carried out self-evaluation processes systematically and produced a written action plan based on the derived in-depth self-evaluation. Note that, in contrast to the quality labels described in the previous section, the purpose of the OQEA quality logo is not to certify the fulfilment of minimum binding quality requirements. Rather, the logo aims at rewarding adult training providers who care about how they do their work and are prepared to constantly learn, test new findings, systematically assess the effects of their work and implement measures to develop quality (Broek and Buiskool, 2013[2]). Currently, approximately 40 providers have obtained the OEQA logo.1
Building on this two‑decade experience of encouraging a quality culture through self-evaluation, the Slovenian Government adopted the Adult Education Act in 2018, which effectively switched Slovenia’s approach to quality assurance in adult education from advisory to regulatory while still using self-evaluation as a tool. In fact, the 2018 Adult Education Act requires all adult education providers to have an internal quality system that includes ongoing monitoring and in-depth self-evaluation. Information on how providers conduct their self-evaluations also have to be made available publicly.
The case of Hungary is similar. In fact, Hungary recently adopted two legislative piece – Act LXXVII of 2013 and Decree 11/2020 – which in practice establish a regulatory approach to quality assurance based purely on self-evaluation. According to the more recent legislation, providers of adult learning need to have a quality management system based on self-evaluation to operate. Although the government has set certain overarching elements that providers should tackle with their quality assurance systems, the responsibility of identifying the correct quality indicators and general framework is left completely to the institutions themselves. No training on how to develop a quality management system and conduct self-evaluation is provided.
Recent efforts in Portugal also attempt to encourage the creation of a self-evaluation culture among the Qualifica Centres, which provide guidance and support for recognition of prior learning. In particular, the Centres have to submit information on enrolment, referral to education and training pathways and recognition activities to the National Agency for Qualification and Vocational Education (ANQEP), which analyses the information and sends it back to the centres in an effort to encourage self-evaluation. Self-evaluations are also common in the formal adult education sector: Box 3.1 presents, for example, the case of the well-known BRUK system in Sweden.
A more subjective way to measure quality is the satisfaction of participants with the training they undertake. This is generally measured through surveys during and/or after training participation. In the Brussels capital region (Belgium), the results from user satisfaction surveys are part of the quality evaluation done by Bruxelles Formation, the organisation in charge of adult learning for the French-speaking population in Brussels. They aim to have an average satisfaction level of at least eight out of ten, on a scale from 1 to 10 where 1 is the lowest satisfaction level. In Finland, participants’ surveys are run during and right after every training programme funded by the public employment services, and this information feeds into the evaluation process.
The Bedömning, Reflektion, Utveckling, Kvalitet (Assessment, Reflection, Development, Quality) initiative (BRUK) is a general support system developed by school authorities in Sweden in 2001 for quality assurance in preschools and public schools, and adapted specifically to formal adult education in 2008. It is based on a set of self-evaluation tools and indicators, as elaborated in national steering documents. By showing strengths and weaknesses of adult training providers, the BRUK quality model aims at giving an overview of the overall quality of the institutions and areas for improvement.
Fundamentally, the system is based on a list of questions that providers of adult training ask themselves with the final goal of identifying gaps in the quality of the delivery of their services. The structure of the questions is complex, yet elaborated such that it maximise continuous improvement. Three main areas – process, goal achievement, and contextual factors – are split into a number of indicator areas, which are then divided in sub-areas and in numerous sub-indicators. For each of these indicators and sub-indicators, precise criteria are then enumerated, and providers need to assess to what extent such criteria are met by their services. The self-evaluation questionnaire also asks providers to add new indicators, which has proven critical in the past in order to motivate providers to use the tool. After providers reply to the whole questionnaire, the criteria are examined in order to obtain an overview of the current state of the quality of the institution, and a number of follow-up actions are planned to improve the situation if needed. Importantly, for each follow-up action providers need also to designate a responsible person in the team and set a deadline.
Although with some initial difficulties in ensuring buy-in by stakeholders, nowadays BRUK is used as a quality tool by both the providers themselves and the external evaluators of educational quality in Sweden. Importantly for the whole sector, the BRUK initiative helped introduce a common language and a common framework on quality in adult learning.
An alternative strategy to monitor and evaluate the performance of training providers relies on external bodies to assess quality through inspections. External evaluations of this type are similar to self-evaluations in their functioning but they resemble certifications in their approach, since they are typically made mandatory for providers in order to access public funds.
In Norway, for example, the agency for lifelong learning (SkillsNorway) is in charge of the inspections of adult learning provided in Study Associations2 and under the publicly funded training programme for basic working life skills (SkillsPlus). The aim of the SkillsPlus initiative is to give adults the opportunity to acquire the basic skills they need to keep up with the demands and changes in modern working life and civil society. Funding and participation have increased every year since the programme was established in 2006. The number of participants who have received training now exceeds 30 000. The programme concentrates on reading, writing, numeracy, and digital skills, and, since 2014, it also includes oral communication. Any enterprise in Norway, private and public, can apply for funding if they follow three criteria: (i) the learning activity should be combined with work and basic skills training should preferably be linked to other job-relevant learning; (ii) the courses should strengthen the participants’ motivation to go on learning; and (iii) the courses have to relate to the competence goals approved by the Ministry of Education and Research. The providers – both public and private, as well as study associations – are important stakeholders in the programme, and they often write the applications on behalf of the enterprises or in their own right. In order to ensure quality of the SkillsPlus programme, SkillsNorway frequently undertakes inspections of the training providers. A negative finding from an inspection can result in an order to make changes, but also in withdrawal of public funding and/or an obligation to pay back received public funding.
In England, the Office for Standards in Education, Children’s Services and Skills (OFSTED) undertakes inspections at all levels of formal education and training, including children’s services, schools and academies, further education colleges, initial teacher training, youth work, work-based learning and adult education. OFSTED grades training providers based on their overall effectiveness, with a focus on: i) the effectiveness of leadership and management; ii) the quality of teaching, learning and assessment; iii) personal development, behaviour and welfare; and iv) outcomes for learners. Inspection judgements are based primarily on first-hand evidence gathered during on-site inspections, but inspectors also consult a range of publicly available data on learners’ and apprentices’ progress and achievement, and have access to a wide range of other information (including self-assessment reports of the providers). The criteria used by inspectors are laid out in the Further Education and Skills Inspection Handbook. Independent training providers who are judged to be inadequate will generally no longer receive funding from the Education and Skills Funding Agency. For Further Education colleges a negative review will lead to the development of a notice to improve, which sets out the conditions that the college must meet in a time bound period in order to receive continued funding.
Measuring the quality of training is challenging, even for training providers themselves, as quality is multi-dimensional and often subjective. Training providers can therefore benefit from support in implementing quality measures and systems for monitoring and evaluation. This type of support is available in some countries in the form of guidelines, criteria and quality standards, or support materials for training providers, such as good practice examples and self-evaluation tools.
Guidelines, criteria and quality standards can form the basis of a framework against which to evaluate the quality of training. Providing training organisations with guidelines will help them understand what is considered high-quality training provision and how it is measured. For example, training accredited by the Department for Adult Training (Service de la Formation des Adultes) in Luxembourg has to follow quality criteria in the areas of i) equal access, ii) transparency, and iii) trained teachers. Offering training providers access to support materials can also help them develop their quality systems. In Italy, the group involved in the Action Plan for Innovation in Adult Learning (PAIDEIA) disseminates good practices in terms of quality among training providers. In Finland, on top of carrying out evaluations, the Finnish Education Evaluation Centre (FINEEC) is tasked with supporting education and training providers in issues related to evaluation and quality assurance. In this respect, the centre formulates evaluation methods and indicators that education providers can use in self-evaluation and peer reviews. FINEEC also supports the development of an evaluation culture among education and training providers and promotes the spreading of good practices (OECD, 2019[3]).
In Ireland, the independent public body Quality and Qualifications Ireland (QQI) has the mandate of promoting quality and accountability in further education and training. Its main tool is a set of different quality assurance guidelines, that includes both a list of common core guidelines that are targeted to all providers and additional sector-specific guidelines for certain areas. In additional, QQI also has the role of disseminating other organisations’ best practices, since they collaborate with a large range of adult training stakeholders.
In Slovenia, good practices, tools and recommendations are made available on an online platform (Mozaik Kakovosti) with the goal of providing support for training providers who are developing an internal quality system. Providers that take part in the Offering Quality Education to Adults (OQEA) initiative carry out planned, systematic and regular assessments and evaluations of their quality. At the beginning of each self-evaluation cycle, the organisation decides in which areas and with which indicators the self-evaluation will take place.
In Denmark, the Ministry of Children and Education does not directly impose the use of a particular quality assurance system to providers of adult training. While formal education institutions are required to have an evaluation system, providers of non-formal adult education are left to tackle quality issues as they deem satisfying. To help providers assess their overall quality, the Danish Adult Education Association (Dansk Folkeoplysnings Samvirke) – in collaboration with the Danish Evaluation Institute (Danmarks Evalueringsinstitut) – developed a tool for self-evaluation that is focused specifically on quality developments in non-formal education.3 The process revolves around identifying and establishing in each organisation six main quality parameters, concerning:
Teacher/Instructor/Activity leader
Physical environment
Communication
Content
Organisation
Participant
The tool distinguishes between three steps. First, providers need to describe their practices and the concrete workflow of their organisation. Second, providers evaluate their practices, define their strengths and weaknesses, and identify what is important to maintain and what should improve. The third step aims at developing the provider’s workflow for the future, based on the findings gathered from the previous phases. It is important that, regardless of whether the ultimate goals are minor or major, they remain realistic within the given organisational framework. Providers should also make sure to start with the most important parts first, lay out a specific plan of action, and set a specific date for evaluation. In addition to being very simple, the tool has two other advantages: providers can establish their own timeline and the consideration necessary to reflect on the issues at hand, and the model is adaptable to any organisation, regardless of structure, size or purpose.
Danish providers of adult labour market training (including non-formal ones) have also access to another self-evaluation tool (VisKvalitet) to help measure participants’ satisfaction and learning outcomes, as well as the satisfaction of employers whose employees have participated in training programmes. The use of the VisKvalitet tool is now voluntary for continuing vocational education and training providers but was compulsory before 2014. The participant questionnaire includes: 10 common questions for unemployed participants and 12 for employed participants (e.g. “How much do you agree on the statement that the teaching was well planned?” and “Has this course meant that you can better take on new tasks in your workplace if needed?”); 3 background questions (e.g. “Whose idea was it for you to enrol in this course?” and “What is your education level?”); and the possibility of elaborating answers at the end of the evaluation. Moreover, the tool gives flexibility to training providers to add personalised questions in addition to the mandatory ones. The employer questionnaire, in contrast, includes: four common questions (e.g. “To what extent does the course meet the needs of the company?” and “Would you recommend this course to others?”); and four background questions (e.g. “What was the reason for the employee to attend the course?”).
An important step for an effective quality assurance system is also to build the capacity of staff in adult training institutions to have a good understanding of what quality is and how to monitor and assess it. In Slovenia, a training programme was developed by the Slovenian Institute for Adult Education (SIAE) for individuals to become quality counsellors in adult education. Training providers who want to improve their quality management system can have one or more staff members participate in the training or hire a qualified quality counsellor.
Rather than adopting certification, quality label systems, or external evaluations, some European countries rely on awards and prizes to develop a quality culture in the adult learning sector. For example, in Finland, the Ministry of Education and Culture organises a yearly quality award competition for adult education providers with the objective of encouraging learning centres to assess and continuously improve the quality of their activities. The rationale behind the initiative is to identify best practices that providers across the country can emulate, and to promote the overall value, attractiveness and visibility of the adult education sector. The Quality Award recognises the quality of services, continuous improvement and results, and exemplary work in the development of vocational training. The award of the prize is based on performance evaluation on predefined criteria determined by an expert committee appointed by the Ministry. A maximum of four quality awards are given each year, with the possibility of honourable mentions of merit in a specific theme, which is varies each year. For instance, in 2017 the theme was knowledge management, in 2018 it was the well-being of students and staff, while the 2020 theme was the holistic nature of quality management.4 The amount of the prize is intended to be used to develop the activities of the training provider.
A similar initiative exists in Sweden, where a Quality Prize has been established by the so-called School Act of 2010. All kinds of schools can participate in the competition, including municipal adult education, as long as they document their actions towards quality improvements. The goal of the award is again motivating and inspiring schools to put in place quality-control processes.
References
[2] Broek, S. and B. Buiskool (2013), Developing the adult learning sector: Quality in the Adult Learning Sector.
[3] OECD (2019), Getting Skills Right: Future-Ready Adult Learning Systems, Getting Skills Right, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264311756-en.
[4] OECD (2018), Skills Strategy Implementation Guidance for Slovenia: Improving the Governance of Adult Learning, OECD Skills Studies, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264308459-en.
[1] Slovenian Institute of Adult Education (2013), Quality indicators in adult education.
← 1. In Slovenia, there are approximately 500 providers of non-formal adult education and training (OECD, 2018[4]).
← 2. The provision of non-formal adult education in Norway is handled by the so-called Study Associations (or Adult Education Associations). Their main objective is to provide educational opportunities that are independent of curricula and exams. Their courses cover a large number of activities, from purely leisure activities to vocational courses and academic subjects. There are currently 15 Study Associations, with group over 450 member organisations. In 2015, around 508 000 participants were registered at study association courses.