Anticipating Skill Needs and Adapting Higher Education

2. Implementing skills assessment and anticipation exercises

Abstract

Choosing the right data to feed the SAA exercise is essential to align the exercise with the intended purpose and to enable the use of different methodologies. This chapter discusses how SAA exercises are generally implemented, including the data sources they rely on and the most common methodologies.

Once the basic features of the SAA exercise are defined, the data sources and the methodology to estimate current and/or future skill needs must be developed. Most systems analysed rely on quantitative data sources and methods, sometimes supplemented with qualitative ones. This chapter discusses the data sources that are generally used in SAA exercises, as well as the most common methodologies.

2.1. Data sources

SAA exercises may rely on quantitative and/or qualitative information to assess and forecast skill needs. Quantitative data sources generally include administrative or survey information, economic growth projections and OJV information. Qualitative information can be collected through expert groups, interviews or validation exercises, which may focus on reviewing the data sources and methodology used in the quantitative phase of the SAA exercise results.

Table 2.1 shows that all exercises analysed in the context of this report rely on quantitative information and that most exercises (12 out of 15) use at least two different data sources. Within quantitative data sources, the most common leverage employment-related administrative data, such as social security information, or data from the national labour force survey (9 out of 15), followed by education information (8 out of 15), economic growth information (7 out of 15), OJV data, employer surveys and demographic information (6 out of 15) and graduate surveys (4 out of 15). Qualitative information is used in 6 out of 15 of the exercises reviewed.

Table 2.1. Data sources used by SAA exercise analysed

SAA exercise		Country	Quantitative							Qualitative
		Labour market information	Education information	Economic information	Demographic information	Job vacancies data	Employer survey	Graduate survey	Skills taxonomy
Jobs and Skills Australia	Australia	✓	✓	✓	✓
JobBarometer	Austria					✓
Future Skills Program	Canada	Not available	Not available	Not available	Not available	Not available	Not available	Not available	Not available	Not available
OSKA	Estonia	✓	✓		✓					✓
Skills Anticipation Forum	Finland	Not available	Not available	Not available	Not available	Not available	Not available	Not available	Not available	Not available
Occupations 2030	France	✓		✓	✓	✓		✓	✓
BIBB-IAB Qualification and Occupational Fields Projections	Germany	✓		✓	✓
IAB Skills Compass	Germany					✓
Graduate Career Tracking System	Hungary	✓	✓					✓
SOLAS Skills and Labour Market Research Unit	Ireland	✓	✓	✓		✓	✓			✓
Excelsior Information System	Italy						✓
Skills and Labour Platform	Italy		✓				✓	✓	✓
Education and Labour Market project (POA)	The Netherlands	✓	✓	✓				✓
Skills Demand for the Future Economy	Singapore					✓			✓	✓
Labour Market Platform	Slovenia	✓	✓	✓	✓	✓	✓		✓	✓
SEPE Occupations Observatory	Spain	✓					✓			✓
Trends and Forecasts study	Sweden	✓	✓	✓	✓		✓			✓

Note: Data sources for projects implemented under the Future Skills Program (Canada) vary depending on the specific project, likewise, sector-specific expert groups for the Finnish Skills Anticipation Forum may decide on the data sources to use on an ad hoc basis.

Source: Authors’ elaboration.

Social security information, personal tax information or labour force surveys provide the most complete information on the evolution of employment by occupation, since these data sources generally cover all sectors, in a representative way. However, since these data only provide information on actual employment, and not on positions that have not been filled, information on skills shortages can only be analysed indirectly, for example by comparing the evolution of wages across occupations. Another drawback of these data is that they generally provide information by occupation, which does not easily translate into skill mismatches or training needs.

To overcome these challenges, some exercises use OJV postings and/or employer surveys. OJV information has a number of benefits: (1) it allows to directly observe labour demand for different occupations and skills, (2) information provided is more granular than survey data, often allowing an analysis at the skills level and the identification of new occupations, and (3) the frequency at which new information is generated is very high, allowing frequent updates and the prompt identification of labour market trends. On the other hand, OJVs may not be representative of all sectors and occupations as some jobs are more likely to be advertised online than others. Additionally, since only labour market demand is observed in job vacancies, OJV information on its own does not allow to identify skill shortages and/or skill mismatches (OECD, 2023[1]). Some examples of SAA exercises that use OJV information are the German IAB Skills Compass, for which OJVs are the main source of information, and the Labour Market Platform (Slovenia), where OJVs are used to complement the analysis of other data sources.

Employer surveys, by contrast, allow the identification of skill shortages and mismatches, as employers can be asked directly about positions that they struggle to fill, skills that they cannot find, as well as training needs for their current workforce. The Excelsior Information System (Italy) relies on monthly employer surveys using a rotating panel to gather information on the evolution of labour market needs. The SEPE Occupations Observatory (Spain) complements the analysis of employer surveys with administrative as well as qualitative information. While overall employers are a good source of information on skill needs, surveys focusing on recruitment difficulties – which could depend on several factors, including working conditions – may overstate the existence of shortages. Another drawback of employer surveys is that response rates may be low resulting in a less representative sample. This could happen, for example, if some type of firms, such as Small and Medium Enterprises (SMEs) or firms in a given sector, have a lower probability of filling in the survey.

Another source of information on skills shortages, surpluses and mismatches are graduate employability surveys, which are used, for example, in the Graduate Career Tracking System (Hungary) and the Skills and Labour Platform (Italy). In these surveys, recent graduates are asked about their experiences entering the labour market and the extent to which they have the skills required to perform their jobs. One advantage of graduate employability surveys is that they offer information by qualification, which then allows universities to finetune curricula to labour market needs. However, response rates for graduate employability surveys tend to be low, particularly when they are run online, given that most graduates are no longer enrolled in the institution they graduated from. Some institutions, such as AlmaLaurea, invest additional resources to increase the response rate of graduate employability surveys to contact by phone those graduates who do not reply to the online survey. Another solution is to complement graduate employability surveys with administrative data, matching graduates’ higher education enrolment information and labour market administrative information to obtain information on the labour market transitions of recent graduates, as is done in the Graduate Career Tracking System (Hungary).

SAAs focusing on the forecast of future supply and demand, often leverage data on economic growth and demographic trends. Examples of SAAs using economic growth information include the BIBB-IAB Qualification and Occupational Fields Projections (Germany) and the POA (the Netherlands). Generally, this information is used jointly with labour market information to better forecast occupation growth and lead to more reliable estimates of labour demand, particularly for longer term horizons. While this information is generally easily accessible, using it to forecast skill needs requires a more complex methodology, as discussed in the next section, potentially increasing the costs of developing and updating the SAA exercise.

To assess and forecast skills shortages, many SAA exercises look at labour supply in addition to labour demand, using education and demographic information - such as population ageing or migration flows – to estimate the future supply of workers for a given occupation. As in the case of economic growth information, these data sources are used by exercises that use more complex methodologies such as OSKA (Estonia) and Occupations 2030 (France).

Finally, some SAA exercises also collect qualitative information, often to refine results obtained using quantitative information, such as in Skills Demand for the Future Economy (Singapore) or to use as an input to the SAA exercise, as in Spain or Ireland. In Spain, Autonomous Communities’ experts provide information that feeds into the SAA exercise. In Ireland, skills audits conducted by Regional Skills Fora are used as input to the SOLAS Skills and Labour Market Research Unit SAA exercise.

2.2. Methodology

SAA exercises may use (1) quantitative methods, (2) qualitative methods, and (3) mixed methods to assess and forecast skill needs. Quantitative methods rely on data and statistics to produce estimates of current and future skill needs, while qualitative methods use written or oral input from experts. SAA exercises using mixed methods combine the two (OECD, 2023[1]).

In the case of quantitative methods, some of the most common methods include:

• Projections generated using simulation models, such as time series models, which extrapolate historical trends to estimate future labour market needs; regression models, which forecast skill needs using the evolution of related variables; computable general equilibrium models, which look at the effects of policies or shocks on the economy; or stock-and-flow models, which estimate future labour imbalances by forecasting labour supply and demand (OECD, 2023[1]).

• Statistical analysis – e.g. of the evolution of the proportion of new hires by occupation – to obtain information about skill shortages.

• Scenario analysis, in which computable general equilibrium models are used to determine skill needs in different possible future scenarios.

• Big data analysis, in which high-frequency data, such as OJV postings, are used to assess and forecast labour market needs.

Qualitative methods generally collect information from experts and/or stakeholders regarding the evolution of labour market needs. Some commonly used qualitative methods include expert groups, stakeholder consultations, and interviews (OECD, 2023[1]). These methods can be used on their own or in combination with quantitative methods (mixed methods), to further develop or validate quantitative results.

All examples reviewed in the context of this report use at least one quantitative method (see Table 2.2), with the most common method being simulation models to project labour market needs, generally expressed in occupations (8 out of 15), followed by statistical analysis (6 out of 15), big data analysis (5 out of 15) and scenario analysis (2 out of 15). Qualitative methods are less common in the examples reviewed, with 7 out of 15 exercises using some sort of qualitative method, in many cases to validate quantitative results (5 out of 15). However, the relatively low use of qualitative methods is misleading, as results of some of the SAA exercises analysed, such as the exercise implemented by the SOLAS Skills and Labour Market Research Unit (Ireland), are later developed using qualitative methods in other exercises.

Table 2.2. Methodology by SAA exercise analysed

SAA exercise	Country	Quantitative				Qualitative
		Projections based on a simulation model	Statistical analysis	Scenario analysis	Big data analysis	Expert group	Interviews	Stakeholder consultations
Jobs and Skills Australia	Australia	✓
JobBarometer	Austria				✓
Future Skills Program	Canada	Not available	Not available	Not available	Not available	Not available	Not available	Not available
OSKA	Estonia	✓				✓		✓
Skills Anticipation Forum	Finland	Not available	Not available	Not available	Not available	Not available	Not available	Not available
Occupations 2030	France	✓		✓				✓
BIBB-IAB Qualification and Occupational Fields Projections	Germany	✓		✓
IAB Skills Compass	Germany				✓
Graduate Career Tracking System	Hungary		✓
SOLAS Skills and Labour Market Research Unit	Ireland		✓		✓	✓
Excelsior Information System	Italy		✓
Skills and Labour Platform	Italy		✓
Education and Labour Market project (POA)	The Netherlands	✓
Skills Demand for the Future Economy	Singapore	✓			✓	✓
Labour Market Platform	Slovenia	✓	✓	Planned for the future	✓	✓
SEPE Occupations Observatory	Spain		✓				✓
Trends and Forecasts study	Sweden	✓

Note: Methodology for projects implemented under the Future Skills Program (Canada) vary depending on the specific project, likewise, sector-specific expert groups for the Finnish Skills Anticipation Forum may decide on the quantitative methods to use on an ad hoc basis.

Source: Authors’ elaboration.

2.2.1. Quantitative methods

Quantitative methods have several advantages: they allow to cover a large number of occupations/qualifications and sectors, they offer a consistent and transparent analysis across occupations, sectors and years, allowing comparisons of results over time, across occupations and sectors, and they are easily repeated (OECD, 2023[1]).

Despite these benefits, most quantitative methods, and particularly projections based on simulations, scenario analysis and big data analysis, require high levels of econometric and statistical skills, complicating the development of the model as well as subsequent updates. For this reason, econometricians have contributed to the development of some of the SAA exercises analysed in the context of this report, such as POA (the Netherlands), which was developed and is implemented by the Research Centre for Education and the Labour Market (ROA), a research institute of the Maastricht University School of Business and Economics. This is also the case in Slovenia, where the stock-flow model used by the Labour Market Platform was developed based on CEDEFOP’s skills forecast methodology1 and validated by an econometrician.

A potential risk of using quantitative models is that, given the precision of their results, results from quantitative methods might create an inaccurate sense of confidence (OECD, 2023[1]). To overcome this challenge, confidence intervals could be used, providing information on the most likely range of results.

Given the data sources used, results from quantitative methods generally provide estimates in terms of occupations or, potentially, qualifications, but not in terms of skills (OECD, 2023[1]). This may limit the use of the results of SAA exercises to update education and training curricula or occupational standards. To overcome this challenge, some SAA exercises complement their main quantitative analysis with the analysis of OJV information, obtaining information on skill needs directly from the job postings. This is the case, for example, in Skills Demand for the Future Economy (Singapore) and in the exercise implemented by the SOLAS Skills and Labour Market Research Unit (Ireland). Alternatively, SAA exercises may rely on a skills taxonomy to translate occupational forecasts into skill needs, as, for example, in the Labour Market Platform (Slovenia), which uses ESCO, or the Skills and Labour Platform (Italy), which uses an Italian skills taxonomy. However, these approaches only provide information on skills demand. To obtain information on skills supply and be able to identify future skill shortages, some exercises are currently looking at learning outcomes listed in the NQF for VET qualifications or using new technologies to identify skills supply from higher education curricula descriptors or other higher education documentation, as in Slovenia and the Netherlands. Skills developed in higher education programmes are often defined at the institution level and HEIs often are not required by public authorities to specify learning outcomes of programmes in a consistent format, complicating the identification of skills that students develop in these qualifications.

Finally, since most quantitative methods often rely on past data to forecast future trends, their results tend to be more accurate for shorter time horizons. Thus, exercises that intend to provide information for a longer time horizon use scenario analysis. For example, scenario analysis was used to develop the results of Occupations 2030 (France), as the time horizon was close to 10 years, and it is used to develop the BIBB-IAB Qualification and Occupational Fields Projections (Germany), which has a horizon of up to 20 years.

2.2.2. Qualitative methods

Qualitative methods are generally easier to develop and implement than quantitative methods, as they require less specialised skills and less data infrastructure, making them suitable to be set up in cases in which there is lower access to the required specialised skills. In addition, they allow to gather information on labour market trends and skill needs that are not (yet) observable in quantitative data. They also permit the direct identification of skill needs, rather than focusing on occupation and qualification needs, and they allow the matching of skill needs to existing or new training, potentially providing solutions to identified skills shortages (OECD, 2023[1]). Given their potential focus on skills, they could help gather information on skill mismatches, in addition to shortages.

On the other hand, qualitative methods rely on experts’ and stakeholders’ observations and opinions, which could lead to subjective and inconsistent results and limit the replicability of the exercise. For practical reasons, they tend to focus on an individual sector or group of occupations, reducing the comparability of exercise results across sectors. This sector-specific focus also limits the identification of potential transitions across sectors, missing relevant information to identify future skills shortages and surpluses (OECD, 2023[1]). Given the need to gather information from multiple experts or stakeholders, implementing qualitative methods can also be resource intensive.

2.2.3. Mixed methods

Many of the SAA exercises analysed use both quantitative and qualitative methodologies. This combination is generally seen as best practice, as one approach can validate the results of the other leading to more reliable and robust results (OECD, 2023[1]).

Almost half of the SAA exercises analysed for this review use at least one qualitative method to gather more information and complement the quantitative analysis or to validate quantitative methods or results. Examples of exercises that use qualitative information to complement their quantitative analysis include the Labour Market Platform (Slovenia) and OSKA (Estonia), which use expert groups’ inputs as information to input in or complement their quantitative analysis. Similarly, the SEPE Occupations Observatory in Spain uses interviews with sector and regional experts to gather information on skill needs. On the other hand, other exercises use qualitative methods to validate the assumptions and methodology of the quantitative model, as in Occupations 2030 (France), or to review and validate the results of the quantitative analysis. This is the case in POA (the Netherlands), where the project advisory group and project partners review the results of the quantitative analysis, or in the Labour Market Platform (Slovenia), in which regional Public Employment Service (PES) offices validate the quantitative results. Finally, in Ireland, the Expert Group on Future Skills Needs uses qualitative methods to further extend the results of the quantitative exercise implemented by the SOLAS Skills and Labour Market Research Unit (Ireland).

Note

← 1.  See https://www.cedefop.europa.eu/files/skills_forecast_methodological_framework.pdf for more details.