Individual Learning Accounts

5. Training quality in Individual Learning Schemes

To ensure the effective use of public and private funding, adult training systems need to ensure training quality. The structure of the training market varies across countries, being more or less concentrated and/or segmented. However, most adult learning systems are characterised by a large number of training programmes, delivered by a significant number of training providers. In France, the training market is shared between a few large public or quasi-public training providers getting a relatively high share of the total number of training participants, some large for-profit private training firms and a large number of small private training providers (including one-person companies). Competition can be harsh on some segments of the market, and much less so on others.1 In Upper Austria, the market appears to be very concentrated, with the two largest providers – owned by social partners’organisations – providing about 85% of the training courses supported by the Bildungskonto at the end of the 2000s (LRH OOE, 2010[25]). In the United States, the market counts with many providers of all sorts, public and (for-profit and not-for-profit) private. But the larger and the more scattered the market is, the more important it is to develop a monitoring and evaluation framework to ensure the quality of training provided.

This becomes even more essential in the context of individual learning schemes, as individuals are buying training services, and they have a much less strong bargaining position and less capacity to control and enforce quality than public funders or employers when they buy training. The training market is characterised by a double asymmetry between ILS participants and training providers: an asymmetry of information on the quality of an often fragmented supply of training, on the one hand, and an asymmetry in the capacity to negotiate prices, on the other (IGAS, 2017[8]).

5.1. Certifying training providers and programmes

To guarantee that training providers comply with minimum quality requirements, many countries have put in place quality procedures involving accreditation and certification mechanisms (Box 5.1). In some countries, publicly funded training programmes can only be delivered by certified providers (OECD, 2019[23]). This certification process ensures that the provider meets certain “obligations of means” (i.e. a duty of care in performing a contractual obligation as opposed to an “obligation of results/outcomes”, which is aimed at guaranteeing the attainment of a specific result). This can be achieved through norms testifying quality management oriented towards customer satisfaction (ISO or norms more specifically related to training, such as AFNOR in France), or by checking the organisation and processes put in place by training providers (staff, equipment, learning materials, assessment environment), the preparation of trainees and customer relations. Another way is through labels developed by public bodies or the organisation representing the training providers. In some cases, regulators have also put requirements on the training programmes themselves (Section 2.4), but outcome-based criteria are rarely used. The development of individual learning schemes has often accelerated this move towards quality assurance for training, as for example in Upper Austria, France and Tuscany.

Box 5.1. Accreditation and certification: what difference?

The terms accreditation and certification are often used interchangeably and occasionally together. In reality, they are two distinct but complementary quality management processes.

Certification represents a written assurance by a third party of the conformity of a product (e.g. organic farming, Pan European Forest Certification, etc.), process and services (e.g. ISO 9001, ISO 14001, ISO/IEC 27001, etc.) or persons (e.g. auditors, property diagnosticians, etc.). It indicates that the necessary steps to achieve a certain level of compliance have been completed.

Accreditation is the formal recognition of conformity assessments bodies by an authoritative third party. The accreditation process ensures that their certification practices are acceptable, that is that the certification bodies behave ethically and employ suitable quality assurance when assessing conformity.

Accreditation and certification therefore do not intervene at the same level. Certification is delivered by control bodies, usually called certification bodies, while accreditation lies within accreditation bodies in charge to assess these control/certification bodies. Most countries have a single official accreditation body, such as COFRAC in France or UKAS in the United Kingdom. In the training area, the term “accredited institution” is often (wrongly) used in place of “certified institution”.

Source: https://apmg-international.com/fr/article/difference-between-accreditation-and-certification, https://www.cofrac.fr/quest-ce-que-laccreditation/certification-et-accreditation-quelles-differences/, https://www.iaf.nu//articles/About/2

In the case of individual learning schemes, verifying training providers’ qualifications is also important to avoid direct fraud of the system. In the UK ILA, for example, the Department for Education and Skills had decided against implementing quality assurance systems, assuming that market forces would ensure that inefficient or ineffective providers would be driven out of business – it was believed that a stringent validation process for training providers might discourage new training providers from entering the marketplace. But fraud developed at a large scale under different forms, often following aggressive marketing practices by recently set-up providers. Some providers falsely pretended that their training programme led to recognised qualifications and provided poor standard materials. Others claimed ILAs for individuals and enrolled them without their consent. Others still accessed ILA funds but provided no training, etc. (National Audit Office, 2002[9]). In part, this was due to pressure to implement the scheme rapidly, combined with a lack of resources for managing and implementing the scheme. The government was also bearing too much of the financial risk compared with the private contractor delivering the service (National Audit Office, 2002[9]).. The extent of fraud led the government to close the scheme in November 2001. One lesson was that ICT systems used to register both providers and beneficiaries need to be sufficiently rigorous to minimise abuse or fraud, particularly in terms of accreditation and quality vetting of providers. Remuneration for credits should be paid with robust evidence that the funded course or activity is legitimate and has actually been undertaken (Johnson et al., 2010[26]).

In Singapore, training providers are required to undertake a certification process, which involves meeting a number of criteria for the provider as well as for the courses on offer. The stringency of criteria depends on whether the courses are delivered under the Workforce Skills Qualification scheme – which provides training support to employers – or not.2 A significant case of fraud was detected by Skills Singapore in October 2017 when it was checking claims for training grants, and all payments were immediately suspended. The case involved three recently established training providers claiming fees for courses provided to 25 000 employees for an amount of about SGD 40 million.3 Processes have been tightened since, with the creation of a fraud and enforcement division, the use of data analytics for fraud detection at the fund claiming stage, and new enforcement strategies including on-site controls.4 Less important cases of fraud have also been reported, notably of individuals trying to have Singaporeans sign up for courses using their SkillsFuture Credit by telling them that their credits will expire. This prompted Skills Singapore to issue a public communication on the subject.5

Following the bad UK experience, the Scottish ILA scheme introduced more stringent vetting of providers and quality assurance of eligible courses. Training providers had to be approved by the Scottish Qualification Authority (SQA) or be a registered national training provider, and course details had to be provided through an online tool which, according to an evaluation commissioned by the Scottish government, raised some issues around transparency and time taken to obtain approval (Gallacher et al., 2007[20]). In the current ITA system, training providers can be approved through one of four possible routes: i) be a SQA approved centre; ii) be certified by Quality Scotland, a quality management certification6; iii) have an active contract with Skills Development Scotland to deliver national training programmes; or iv) be a college or university with proven quality assurance.7 To ensure that they are financially stable, training providers must also have been trading for at least two years. Courses have to meet eligibility requirements (mostly they need to be vocational, and concern the approved sectors for ITA funding in line with the Labour Market Strategy; see Section 2.4; and not be of postgraduate level) but, once the provider is ITA-recognised, there are no additional quality requirements on programmes.

The Tuscan Carta ILA involved no quality assurance framework in the first period (2006-2008), and even informal training was eligible. However, this implied a significant risk of fraud, which did materialise with users using their card for other purposes than training and the regional government had to reclaim the funds. Combined with the global financial crisis and the austerity that followed, eligibility was restricted to certified training institutions and programmes being part of a regional training list.

In Upper Austria, quality procedures in the training sector were not very developed when the Bildungskonto was established in the early 1990s, but the training market was rather concentrated8 and the big providers were ISO certified. But quality issues gained prominence in Austrian education policy in general, and a regional quality framework (Qualitätsgütesiegel) was implemented in 1998. By making certification mandatory to be eligible for support, the Bildungskonto contributed to spreading this quality framework. Since 2012, a national quality framework (Ö-Cert) has also been in place, and training providers have to be certified by one of these two quality frameworks.

In France, quality controls for providers used to be relatively weak and an effective certification process was lacking (France Stratégie, 2015[27]). In 2014, the new law on training made public training financing bodies responsible for monitoring and controlling the quality of training providers they work with. As a result, the various training financing bodies – including the OPCA, regions, and the Public Employment Service (Pôle Emploi) – have collaborated to harmonise quality criteria applied to training providers. This was formalised in the Décret Qualité which was introduced in 2015 and came into force in January 2017. The law establishes a set of six quality criteria which training providers need to comply with in order to access financing (the objectives of training; the existence of procedural control mechanisms; the adequacy of the pedagogical tools used; the quality of teachers; the accessibility of information on training offer to the general public; and the evaluation of training programmes). Following this decree, training providers have to provide proof of quality either through a label or quality certificate managed by the Conseil national de l'emploi, de la formation et de l'orientation professionnelles (CNEFOP) or by adhering to the internal evaluation framework of the training financing bodies. A dedicated online platform – the Data Dock (https://www.data-dock.fr/) – has been developed to that purpose to allow training providers to register in the system and self-assess against the six quality dimensions (OECD, 2019[28]). However, as underlined in the impact study accompanying the 2018 Loi pour la liberté de choisir son avenir professionnel reforming the training financing system, the proliferation of labels and quality certificates makes it difficult for individuals and firms to find their way in the training market, and does not allow a complete harmonisation of practices. From January 2021 onwards, a new law will impose the certification of training providers based on a unique national repository.9

In addition, as a counterpart to the autonomy left to the individual in mobilising his/her account and with a view to promote the transferability of skills, the French CPF also imposes constraints on training programmes. Until December 2018, training programmes had to be chosen from a list of eligible programmes established at three levels (inter-professional, branch and region) as described in Section 2.4. In the reformed CPF, all (formal and non-formal) training programmes resulting in a professional certification registered at the national level either in the Répertoire National des Certifications Professionnelles (RNCP) or in the Répertoire spécifique10 have become eligible for the CPF. Requiring that a training programme delivers a “certification” is interesting in principle as this provides some official recognition of skills acquired and could allow progression of trainees both in their employment and in their training path. However, this does not always ensure the quality of the training programme. Some training providers, for example in the language sector, have registered their programmes in existing certification standards, but without modifying the content of their programme and just registering trainees to TOEFL, BULATS or TOEIC exam sessions (IGAS, 2017[8]). Others have developed their own certification process – a longer and more complex but more flexible option in terms of pedagogy – which significantly increased the number of existing certification processes (IGAS, 2017[8]).

The US ITA is the only reviewed scheme with a quality assurance framework designed specifically for the scheme. To be eligible for the ITA scheme training programmes offered by a given provider need to be on the Eligible Training Provider List. Requirements to be on that list are numerous, but the main ones are: i) provide training for in-demand occupations; ii) offer industry-recognised qualifications; and iii) comply with reporting requirements on a number of outcomes of the programme (mainly completion rates, percentage of trainees obtaining unsubsidised employment; employment rate at 6 months; average earnings of participants at placement and after 6 months; percentage of trainees obtaining a license or an academic degree). Many of these reporting requirements make sense for jobseekers only and could not be applied to individual learning schemes covering employees. There are minimum rates for each outcome measure and a threshold for an indicator taking into account performance targets for the various measures. Minimum rates appear to be low (e.g. a completion rate of 30%). If the target is not reached, a regression‑adjusted model is used to take account of some of the characteristics of participants to control for initial conditions. Providing this information for each programme is rather costly for training providers. Community college and not-for-profit providers in particular were slow to comply, which gave rise to a number of waivers.

5.2. Providing information on the quality of programmes to the general public

For individuals to be able to make informed choices about which training to invest in, they need to have access to relevant and up-to-date information on the quality of different training providers and programmes. Certification and quality labels (mentioned above) can serve as signals of quality, but training providers can also be requested to share more in-depth information on evaluations, learning outcomes and user satisfaction with the general public to help them decide which training to invest in (OECD, 2019[23]). In practice, however, although there are plans to publish some quality measures in some schemes, this has not yet been implemented in most of the schemes reviewed but the US ITA. In the latter however, the obligation imposed by law to training providers to publish outcome measures is implemented with varying success. In Michigan, for example, it is very difficult to find a training programme including the requested variables (e.g. completion rates, employment rates, hourly and annual earnings) on the website informing about the various training programmes. This obligation seems to be better enforced in the state of Washington.

The Scottish ILA has so far not had a mechanism for trainees to feed back on their training experience. However, in the future, the intention is that all ITA trainees will receive a standardised email asking them to rate the quality of their training programme and state how they intend to use the skills obtained. It is not clear yet what precise information will be collected nor how this will be communicated to the public.

In Singapore, an assessment of the quality of training courses is meant to be disseminated to the public via Skills Singapore’s Training Quality and Outcomes Measurement (TRAQOM) initiative launched in 2018. TRAQOM contains course ratings collected from a post-course survey and was initially piloted with 50 training providers. The results were not available at the time of publication of this report. Feedback can also be provided by participants on each course page on the Training Exchange.11

The new French CPF application should also allow participants to rate their training programme and provider, and this information will be published on the information system. Practical details are not available yet.

Notes

← 1.  It is a segmented market with, on the one hand, public and not-for-profit private providers generally providing publicly-financed training schemes (most often) for jobseekers and, on the other hand, private providers who tend to sell their services to employers/firms (France Stratégie, 2015[27]). Some private providers, often set up by firms themselves or employers’ associations, provide training programmes for specific professions; others intervene in business-oriented programmes not directly linked to employees’ professions, such as: languages, IT, security or industrial maintenance, where competition is much stronger. When it comes to programmes financed by Pôle Emploi (the Public Employment Service) or the regions, only a few large public providers can mutualise resources on specific professions, while less specific programmes can be provided by smaller providers in a more competitive environment.

← 2.  The WSQ framework criteria are more stringent that the criteria for courses not delivered under the WSQ.

← 3.  Source: https://www.channelnewsasia.com/news/singapore/first-member-syndicate-jailed-40m-skillsfuture-scam-10973606

← 4.  Source: https://sg.news.yahoo.com/skillsfuture-beefs-fraud-detection-capabilities-major-scam-case-010315564.html?guccounter=1, https://www.channelnewsasia.com/news/singapore/skillsfuture-task-force-fraud-scam-9841930,

← 5.  Source: Ibid.

← 6.  The provider needs to be certified as achieving Quality Scotland Committed to Excellence or Recognised for Excellence.

← 7.  That is to be recognised by the Scottish Funding Council as being in good standing regarding the Quality Assurance Agency for Higher Education and/or recognition of good standing regarding the requirements of Education Scotland.

← 8.  The two largest providers made for 85% of the courses supported by the Bildungskonto in 2010 (LRH OOE, 2010[25]).

← 9.  According to a law enforcement decree, training providers will be assessed according to 32 indicators measuring seven quality criteria, the six previous quality criteria and an additional one (insertion of the training provider in its socio-economic environment).

← 10.  The Répertoire spécifique replaces what was called the “inventaire” (see Section 2.4).

← 11.  It does not appear to be commonly done yet.