Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes 2023

4. Review of 50 programme evaluations from 28 OECD countries

Abstract

The meta-evaluations discussed in Chapter 3 used a wide variety of data sources and analytical methods, implying, as shown in Part I, that the findings of some studies are somewhat more reliable than others. To address this diversity, this chapter contains a detailed review limited only to a selection of evaluations where the data and the analysis satisfy our requirements for reliability. A total of 50 evaluations in 28 OECD member countries are reviewed. The chapter starts by setting out the criteria used to identify and select the 50 evaluations and offers their big picture findings. It then assesses the policy issues they raise. Finally, it discusses the scope and quality of the evaluations.

This section presents the evidence from 50 reviews of published SME and entrepreneurship policy evaluations. It differs from the reviews in the previous chapter because its coverage is highly selective. It is limited to evaluations in OECD countries, published since (OECD, 2007[1]), and it imposes a “quality” criterion on the evaluations that are included. Section 4.1 provides a brief description of how the evaluations were chosen, with full information provided in Annex A. Section 4.2 then presents the big picture findings of the evaluations. Sections 4.3 and 4.4 discuss the policy issues raised and the evaluation approaches used.

4.1. Selecting the evaluations

Our major criterion for including studies in this review was that they used robust methodologies, so enabling policy makers to place reliance on their findings. Since 2007 there have been a considerable number of evaluations of the impact of SME and entrepreneurship policy that do not meet the Step V and VI requirements of (OECD, 2007[1]) and of this Framework. Our decision was to not include them in this review. Inclusion was therefore determined by passing the OECD threshold, together with a range of other factors set out in detail in Annex A. The ultimate purpose was to reach a balanced conclusion on the effectiveness of SME and entrepreneurship programmes from reliable evaluations, as well as to illustrate good evaluation practice. The criteria set out in Annex A generated 50 evaluations in 28 OECD member countries, aimed at covering the following key SME and entrepreneurship policy areas that we identified for assessment, namely:

• Finance;

• Business Advice, Coaching, Mentoring and Counselling;

• Internationalisation;

• Innovation;

• Enterprise Culture and Skills;

• Inclusive Entrepreneurship;

• Regional and Local Evaluations;

• Cluster Policies; and

• Support in Areas of Disadvantage. 

Where there were multiple high-quality evaluations of a policy area, we have favoured the inclusion of evaluations from a country for which there were no other high-quality studies.

All 50 programme evaluations are documented in full in Annex B across fifteen dimensions. It included as diverse a range of countries as possible in order to avoid the sample being dominated by large countries that have conducted many evaluations. Annex C provides the interested reader with information about a further 25 such evaluations that were considered but not included on at least one of the above grounds.

4.2. The evaluations: overview of evaluation findings and key features

The “big picture” findings of the evaluations are shown in Table 4.1. The left-hand side of the Table documents the policy results for each evaluation; the right-hand side documents the evaluation coverage and quality. Brief scoring notes and explanations are provided at the foot of the Table, with more comprehensive coverage in Annex B.

Table 4.1. Summarising the findings and key features of 50 reliable evaluations of SME and entrepreneurship policy

POLICY ISSUES								EVALUATION COVERAGE AND QUALITY
Study number/ country	Evaluation theme	Key findings	Objectives specification score	Hard/ Soft/ Both	Programme expenditure	Lifespan of programme	Policy impact of evaluation	Performance metrics	Non-survivors included?	Step level	Evaluation quality score
1 Australia	Finance	Positive effects on sales, profits and obtaining other funding (Positive impact)	1	Hard	Unknown	2005-2010	Unknown	Sales, profit, probability of obtaining other funding	No	V	2
2 Belgium	Finance	Positive effects on fixed assets, employment, sales, value added, labour productivity and TFP growth for very small firms, but no effects for larger firms (Mixed impact)	1	Hard	250 mil. EUR	2004-2009	Results were published in newspapers, but not presented to the policymakers	Employment, fixed assets, sales, value-added, labour productivity and total factor productivity	No	VI	4
3 Canada	Finance	Positive effects on salary, employment and revenues, but no significant effects on profit (Mixed impact)	2	Hard	30 mil. USD	2004	Results were not presented to the policymakers	Employment, revenues, profit and wages	No	V	3
4 Czech Republic	Finance	Positive effects on price-cost margin, value added per labour cost, growth of sales and growth of tangible assets (Positive impact)	1	Hard	86.4 mil. EUR	2007-2013	Presentation of the findings and recommendations to the policymakers	Price-cost margin, return on assets, assets turnover, value added per labour costs, long-run risk, tangible fixed assets, labour costs, sales	No	VI	4
5 Czech Republic	Finance	Positive effects only on tangible fixed assets, otherwise no effects on the outcome variables (Mixed impact)	2	Hard	164 mil. EUR	2007-2013	Presentation of the findings and recommendations to the policymakers	Total assets, tangible fixed assets, personnel costs, sales, price-cost margin, return on assets	Yes	VI	5
6 Estonia	Finance	Positive effects on sales and labour productivity (Positive impact)	1	Hard	13.9 mil. EUR	2004-2009	Unknown	Sales, labour productivity	No	V	3
7 Hungary	Finance	Positive effects on employment, value-added, sales, profits, tangible assets, but insignificant effects on labour productivity (Mixed impact)	2	Hard	11,067 bil. HUF	2007-2013	Presentation of the findings and recommendations to the policymakers	Employment, value added, sales, profit, tangible assets, labour productivity	No	VI	4
8 Italy	Finance	Positive effects on return on investment for micro and small firms and negative for medium-sized firms (Mixed impact)	2	Hard	2 bil. EUR	2000- (ongoing)	Presentation of the findings and recommendations to the policymakers	Return on investment	No	VI	4
9 Japan	Finance	Positive effects on credit availability, but no effects on profitability, investment and employment and negative effects on credit score (Mixed impact)	2	Hard	27.1 tril. yen	2008-2011	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, loans obtained from a bank, interest payments, cash ratio, credit score, tangible fixed assets, sales, return on assets	No	VI	4
10 Korea	Finance	Positive effects on sales, employment, wage levels and survival rates. Additional effects specific across schemes are reported (Positive impact)	2	Hard	12 tril. KRW	2001-2003	Results were not presented to the policymakers.	Total factor productivity, employment, sales, wage level, investment intensity, change in R&D status, firm survival	Yes	VI	5
11 Lithuania	Finance	Participation did not result in labour productivity gains (No/negative impact)	1	Hard	498.5 mill. EUR	2007-2012	Results were not presented to the policymakers	Labour productivity	No	VI	4
12 Mexico	Finance	Positive effects on value-added, exports, sales, employment and fixed assets. However, the outcomes differed across the programmes (Positive impact)	2	Both	1,911.86 mil USD	2001-2006	Presentation of the findings and recommendations to the policymakers.	Employment, value added, gross production, sales, worked hours, wages, fixed assets, foreign sales, technology transfer payments, maquila services	No	VI	3
13 Slovenia	Finance	Positive effects on employment, but not on sales (Mixed impact)	2	Hard	688 mil. EUR	2009-2015	Results were published in newspapers, but not presented to the policymakers	Employment, sales	No	VI	4
14 United Kingdom	Finance	Positive effects on employment, but no effects on sales (Mixed impact)	2	Hard	2,106.7 mil. GBP	2009-(ongoing)	Unknown	Employment, sales	No	VI	3
15 United States	Finance	Positive effects on employment (Positive impact)	2	Hard	unknown	1992-2007	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment	No	VI	4
16 Canada	Business Advice/Coaching/ Mentoring	Positive effects on sales, patents, obtaining an angel equity investment and on formation of a strategic alliance (Positive impact)	2	Soft	662,360 USD	2007-2009	Presentation of the findings and recommendations to the policymakers	Sales, obtaining an angel equity investment, patents, formation of a strategic alliance	Yes	VI	5
17 Chile	Business Advice/Coaching/ Mentoring/ Counselling	SMEs improved their sales, employment, wages and sustainability, while large firms increased their sales and export orientation (Positive impact)	2	Soft	42.3 mil. USD.	1998-ongoing	Presentation of the findings and recommendations to the policymakers	Firm sustainability (positive sales), sales, export orientation (exporting), employment and wages	No	VI	4
18 Denmark	Business Advice/Coaching/ Mentoring/ Counselling	Positive effects on firm survival and mostly positive effects on employment, turnover and growth (Positive impact)	2	Soft	1 mil. USD	2002-2006	Presentation of the findings and recommendations to the policymakers	Survival, employment, 20% firm growth in employment or sales	Yes	VI	5
19 Germany	Business Advice/Coaching/ Mentoring / Counselling	No effects on firm-survival (No/negative impact)	2	Soft	500 mil. EUR	2016-2017	Presentation of the findings and recommendations to the policymakers	Firm survival, business scale-up	Yes	VI	3
20 Mexico	Business Advice/Coaching/ Mentoring / Counselling	Positive effects on total factor productivity, return on assets, wages, employment and entrepreneurial skills (Positive impact)	2	Soft	11,856 USD per firm	2008-2009	Presentation of the findings and recommendations to the policymakers	Employment, total factor productivity, return on assets, wages, managerial and entrepreneurial skills	No	VI	4
21 United Kingdom	Business Advice/Coaching/ Mentoring / Counselling	Positive effects of the intensive support on employment and sales growth and negative impact of less intensive support on sales per employee (Mixed impact)	2	Soft	527.63 GBP	2003	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, sales, sales revenue per employee	No	VI	4
22 Ireland	Internationalisation	Positive effects on employment (Positive impact)	2	Hard	553,286 EUR	1970-ongoing	Unknown	Employment	No	VI	4
23 Chile	Innovation	Positive effects on new business formation, firm survival and sales growth (Positive impact)	2	Hard	67,000 USD	2001-ongoing	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	New business formation, firm survival, increase in sales	Yes	V	2
24 Finland	Innovation	Positive effects on sales (Positive impact)	2	Both	102.6 mil. EUR	2008-2012	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Sales	No	VI	4
25 Poland	Innovation	Improved science-industry collaboration, increased probabilities of applying for a patent and publishing, and positive effects on the commercialisation of new products/processes (including sales) (Positive impact)	2	Hard	660,000 USD per recipient	2012-2013	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Patent application, publication of a scientific article, citations, development of a new industrial design, prototype, product or process, commercialisation of a new product/process, share of sales from new products/processes, new collaboration, commercialisation index, research and innovation index, collaboration index	No	VI	4
26 Portugal	Innovation	Positive effects on investments, sales, technological progress and job creation, but negative effects on labour productivity and value creation (Mixed impact)	2	Hard	2,000 mil. EUR	2007-2013	The results were sent to the policymakers, but not presented	Employment, sales, EBITDA, gross value-added, labour productivity, total factor productivity, value creation, tangible fixed assets, patent stock	No	V	3
27 Spain	Innovation	Positive effects on employment and sales, but no effects on firm survival (Mixed impact)	2	Hard	263.5 mil. EUR	2005-2011	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, sales, survival rate	Yes	VI	5
28 Sweden	Innovation	Positive effects on employment, sales and external equity funding (Positive impact)	3	Hard	3.64 mil. EUR in total	2002-2008	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, equity, sales	Yes	VI	5
29 Switzerland	Innovation	Positive effects on sales, reduction of production costs (Positive impact)	2	Hard	120 mil. Swiss francs (CHF)	2000-2002	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Share of sales from new products, share of sales from new markets worldwide, percentage increase in sales, percentage reduction of average variable production costs due to innovation, economic importance of the innovations, technical importance of the innovations	No	VI	4
30 Turkey	Innovation	Positive effects on share of R&D personnel, R&D expenditures per employee and R&D intensity. Effects for the remaining variables were not found to be significant (Mixed impact)	2	Hard	491 mil. USD	1995- (ongoing)	Presentation of the findings and recommendations to the policymakers	R&D intensity, R&D expenditures per employee, share of R&D personnel, export intensity, import intensity	No	VI	4
31 United States	Innovation	Positive effects on sales and employment, but negative effects on firm survival (Mixed impact)	2	Both	Unknown	1990-2007	Unknown	Firm survival, sales, employment	Yes	VI	5
32 Czech Republic	Enterprise Culture and Skills	No effects on employment (No/negative impact)	2	Soft	618 mil. EUR	2007-2013	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment	No	VI	4
33 Netherlands	Enterprise Culture and Skills	No effects on self-assessed entrepreneurial skills (and traits) and negative effects on entrepreneurial intentions (No/negative impact)	2	Soft	Unknown	2005	Unknown	Entrepreneurial competences and intentions (validated scales) measured as need for achievement, need for autonomy, need for power, social orientation, self efficacy, endurance, risk taking propensity, market awareness, creativity, flexibility	No	V	4
34 Netherlands	Enterprise Culture and Skills	Positive effects on profit and some areas of tax compliant behaviour. No impact on firm survival (Mixed impact)	2	Soft	Unknown	2008-2009	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Firm survival, profit, business costs, filing tax return correctly, completely and in time, and paying the amount of taxes due in time, bookkeeping skills	Yes	VI	5
35 United Kingdom	Enterprise Culture and Skills	Positive effects on profit margin and sales revenue per employee for firms participating in at least one training activity (Positive impact)	2	Soft	Unknown	2002-2003	Results were not presented to the policymakers	Profit margin, sales revenue per employee	No	VI	4
36 United States	Enterprise Culture and Skills	Short-term positive effects on business start-up, but no effects on business performance (Mixed impact)	2	Soft	2.8 mil. USD	2003-2005	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Business start-up, household income, employment, sales	Yes	VI	5
37 Chile	Inclusive Entrepreneurship	Positive effects on employment and earnings (Positive impact)	2	Both	1.83 mil. USD	2002-(ongoing)	Presentation of the findings and recommendations to the policymakers	Employment, earnings	No	VI	4
38 France	Inclusive Entrepreneurship	Positive long-term effects on firm survival (Positive impact)	2	Hard	700 mil. EUR	1998	Results were not presented to the policymakers	Firm survival	Yes	VI	5
39 Germany	Inclusive Entrepreneurship	Positive and long-term effects on the probability of being employed or self-employed (rather than return to unemployment) (Positive impact)	2	Hard	169.66 mil. EUR	2005-ongoing	Presentation of the findings and recommendations to the policymakers	Share of formerly unemployed participants returning to unemployment	No	VI	4
40 Germany	Inclusive Entrepreneurship	Positive and long-term effects on the probability of being employed or self-employed (rather than return to unemployment) (Positive impact)	2	Hard	268 mil. EUR	2012-ongoing	Results were not presented to the policymakers	Share of formerly unemployed establishing in self- or regular employment, earnings	No	VI	4
41 Germany	Inclusive Entrepreneurship	No effects on firm-survival (No/negative impact)	2	Both	Unknown	1986-ongoing	Unknown	Firm survival	Yes	VI	5
42 Italy	Inclusive Entrepreneurship	Positive effects on firm survival and, to some extent, on employment (Positive impact)	2	Hard	Unknown	2011-2015	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, firm survival	Yes	VI	4
43 Spain	Inclusive Entrepreneurship	No effects on firm survival (No/negative impact)	2	Hard	Unknown	2013-ongoing	Presentation of the findings and recommendations to the policymakers	Firm survival	Yes	VI	5
44 Sweden	Inclusive Entrepreneurship	Positive effects on probability of leaving unemployment (Positive impact)	2	Hard	800 mil. SEK	1984-ongoing	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Probability of leaving unemployment	No	VI	4
45 Spain	Regional and Local Evaluations	Positive effects on employment growth only in the case of soft business support, but not in the case of financial support (Mixed impact)	2	Both	Unknown	2002-2005	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment	No	VI	4
46 Germany	Support in Areas of Disadvantage	Positive effects on employment and turnover, but insignificant effects on gross fixed capital, and labour productivity (Mixed impact)	2	Hard	1.377 bil. EUR	2007-2013	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, turnover, gross fixed capital, labour productivity	No	VI	4
47 Italy	Support in Areas of Disadvantage	Positive effects on sales, value-added, employment and fixed assets, but negative effects on total factor productivity (Mixed impact)	2	Hard	23 bil. EUR	1996-2007	Results were not presented to the policymakers	Employment, sales, fixed assets, value-added per labour costs, debt costs, total factor productivity	No	V	4
48 Italy	Support in Areas of Disadvantage	Positive effects on tangible assets, turnover and employment, but insignificant effects on value-added per labour costs (labour productivity) (Mixed impact)	2	Hard	23 bil. EUR	1996-2007	Unknown	Employment, sales, fixed assets, value-added per labour costs	No	VI	4
49 Italy	Support in Areas of Disadvantage	Positive effects on fixed assets, sales and employment and negative effects on total factor productivity (Mixed impact)	2	Hard	23 bil. EUR	1996-2007	Results were not presented to the policymakers	Employment, sales, fixed assets, total factor productivity	No	VI	4
50 United Kingdom	Support in Areas of Disadvantage	Positive effects on employment and investments, but no effects on total factor productivity (Mixed impact)	2	Hard	164 mil. GBP	1972-ongoing	Presentation of the findings and recommendations to the policymakers. Some changes were implemented	Employment, investments, valueadded per employee, total factor productivity	No	VI	4

Scoring notes and explanations:

Objectives Specification Score: When ranking the programme objective setting, we used a scale from 1 to 3. We ranked 1 when the programme had only general objectives or indicators, 2 when the programme had specific objectives and indicators close to its objective, and 3 when the programme had milestones and target values in addition to specific objectives and indicators.

Hard/Soft/Both: We assigned programmes the label hard when they had a strong component of financial support. Soft programmes were those focused on advice, training and mentoring without substantial financial support. Programmes with both a financial and advice/training/mentoring element were assigned the label Both.

Step Level: To assess the methodological rigour of the evaluation study, we have followed the Six Steps to Heaven Approach described in (OECD, 2007[1]) and this Framework. It ranks sophistication of the methods used from I to VI. The approach ranks studies in the following way: Step I: take-up of schemes, Step II: recipients’ opinions, Step III: recipients’ views of the difference made by the assistance, Step IV: comparison of the performance of “assisted” with “typical firms”, Step V: comparison with match firms, and Step VI: taking account of selection bias.

Evaluation Quality Score: This assesses the quality of the evaluation on its link to the objectives of the intervention, the research sample, accounting for the impact of firm survival and non-survival, the impact variables, the evaluation methods and their implementation. Evaluations are scored on these factors on a scale from 1 to 5:

We scored 1 when the evaluation was based only on a limited sample, evaluation methods were very basic and were not implemented properly, impact variables did not match programme objectives, and survival analysis was missing.

We scored 2 when the evaluation was based only on a limited sample, evaluation methods were very basic but were appropriately implemented, impact variables did not match programme objectives, and survival analysis was missing.

We scored 3 when the evaluation was based on an adequate and representative sample, evaluation methods were appropriately implemented, impact variables did not match programme objectives, and survival analysis was missing.

We scored 4 when the evaluation was based on an adequate and representative sample, evaluation methods were appropriately implemented, impact variables matched programme objectives, but survival analysis was missing.

We scored 5 when the evaluation was based on an adequate and representative sample, evaluation methods were appropriately implemented, impact variables matched programme objectives and survival analysis was included.

4.3. Policy issues

Column 1 of Table 4.1 provides the study number, enabling the interested reader to obtain full information on the study from Annex B.

The second column shows the 50 evaluations, covering eight main SME and entrepreneurship policy areas. Almost one third (15) are of different aspects of Finance programmes and there are nine studies of Innovation programmes. A third policy area with several reliable evaluations is Inclusive Entrepreneurship where there were eight.

Despite the ubiquity of policy initiatives providing Soft business support in the form of Business Advice/Coaching/Mentoring/Counselling we were unable to find many evaluations that satisfied our criteria for reliability. There are only 6 reliable evaluations of this kind of support among the 50. We see this as a matter of concern, as was the absence of any reliable Cluster policy evaluations.

A limitation of Table 4.1 is that each programme evaluation is placed in only a single policy area, whereas several cover multiple policy areas. For example, policies to enhance innovation frequently use both public funding and advice meaning they could, in principle, be placed in the policy areas of Finance or Business Advice/Mentoring/Coaching/Counselling (Soft support). Hence placing the evaluated programmes in a single policy area could, potentially, be misleading. To address this, the Framework looks closely at any stated policy objectives and categorises the programmes on what appears to be the dominant focus1. We also favour repetitive evaluation studies of the same intervention on the grounds that policy lessons can be learnt when outcomes differ.

Results

The third column of Table 4.1 provides a verbal description of the 50 evaluation results so, in order to make the findings easier to interpret, we compress them into three groups for further discussion below.

• Positive Impact. The first group are the evaluations where the findings are either exclusively positive or, where although there are multiple performance metrics, the strong balance of metrics are positive. There are 23 such evaluations and they are defined as Positive.

• No/Negative Impact. The second group are those in which there was either no evidence of impact according to any metric or where the balance of evidence pointed to a significantly negative effect. These evaluations are defined as No/Negative Impact. There are 6 evaluations in this group.

• Mixed Impact. The third group are those where impact differs depending on the chosen metric. So, for example, Study 3 finds a positive impact on sales and employment, but no impact on profitability. The 21 evaluations of this type are classified as Mixed.

The overall picture that emerges is of one that is broadly positive but, with just over half of the evaluations pointing to either Mixed or No/Negative Impacts, SME and entrepreneurship policies are some way off being given a clean bill of health.

In part this may be because evaluation outcomes are influenced either by the sophistication of the evaluation as noted in (OECD, 2007[1]), or by the policy area under consideration. We now examine both explanations.

Policy impact and EQS

(OECD, 2007[1]) stated that:

“sophisticated evaluations of SME support are, on balance, less likely to provide evidence of policy impact than the evaluations using the less sophisticated approaches”, p50

It would be a matter of real concern if this pattern continued, with the less reliable studies being more likely to point to positive – or negative – impacts. To examine such a link we show our reliability measure – the Evaluation Quality Score (EQS) – alongside outcomes in Table 4.2. The EQS data is reported in the final column of Table 4.2 and is discussed in more detail below.

Table 4.2. Comparison of Evaluation Quality Score and estimated programme impact

Absolute number of evaluations in each category

	Evaluation Quality Score (EQS)
Evaluation outcomes	EQS 2	EQS 3	EQS 4	EQS 5	Total number of evaluations
MIXED IMPACT	0	3	13	5	21
NO/NEGATIVE IMPACT	0	1	3	2	6
POSITIVE IMPACT	2	2	14	5	23
Column Total	2	6	30	12	50

Reassuringly, this shows that, amongst the 50 high-quality evaluations documented here, outcomes do not seem to be clearly influenced by the EQS. Other implications of EQS are discussed later.

It is, of course, not possible to reach a judgement about whether, amongst the numerous SME and entrepreneurship policy evaluations that did not meet the reliability requirements of this Framework, there continues to be a link between positive estimated outcomes and low evaluation quality.

Policy impact and policy type

A second dimension on which policy impact can be reviewed is whether it varies with policy type. This Framework uses a three-way grouping of policy types, distinguishing between Hard, Soft and Both. These are shown for each evaluation in Column 5 of Table 4.1. There are 33 Hard and 11 Soft programmes, with 6 combining Hard and Soft (i.e. Both).

Using this distinction, Table 4.3 assesses whether, for example, Soft programmes are less likely to be classified as having a Positive outcome. Although there are small numbers involved, out of the 6 Evaluations with No/Negative outcomes, 3 out of the 11 Soft programmes were in this category. The comparable figure for Hard policies was 2 out of 34.

Given the difficulty of finding reliable evaluations of Soft programmes to include, this suggests the impact of Soft support continues to be open to valid questioning.

Table 4.3. Comparison of type of programme – Hard, Soft and Both – and estimated programme impact

Absolute number of evaluations in each category

	Programme type
Evaluation outcomes	Both	Hard	Soft	Total number of evaluations
MIXED IMPACT	2	16	3	21
NO/NEGATIVE IMPACT	1	2	3	6
POSITIVE IMPACT	3	15	5	23
Column Total	6	33	11	50

Objectives

Column 4 of Table 4.1 presents, for each study, the extent to which Objectives and Targets were specified – ideally prior to the programme being implemented. Our scoring system was 1 is when the programme had only general Objectives, 2 when it had indicators close to its Objective, and 3 when this was combined with specific milestones and Target values.

The results, taken from Table 4.1 are very disappointing. Only 1 evaluation out of 50 scored "3" (2%), although 44 scored "2" (88%).

Scale of expenditure and lifespan of programme

Columns 6 and 7 of Table 4.1 document the scale and duration of the 50 programmes evaluated. It confirms these are generally large-scale and had a lengthy life span. This is to be expected because clearly unsuccessful programmes do not require evaluations to provide evidence of their ineffectiveness. Secondly, as noted in (OECD, 2007[1]), since evaluations have high fixed costs they tend to be focussed on large, rather than small scale policies and programmes. Finally, there may be an element of survivor bias, with only the long-term programmes surviving for long enough to merit an evaluation.

This seems to be supported by Table 4.1. Only 4 evaluations were of short-lived programmes of less than 2 years, although a further 8 were of programmes with a lifespan of 2-3 years. In contrast, there were 13 evaluations of programmes that were both currently ongoing, and which had already had a lengthy lifespan.

Given this diversity it is unsurprising that expenditure varies considerably between the programmes but, perhaps of greatest concern is that in 10 cases it was not possible to determine either from public sources or from those undertaking the evaluation, the sums involved. In the case of small, short-life programmes the sums may have been negligible but in some cases these programmes are on-going and have had a lengthy period of operation.

Ideally expenditure should be linked to impact, so as to be able to comment upon policy effectiveness in terms of a metric such as cost per job created. This would enable more reliable comments to be made on areas of high and low policy effectiveness. Unfortunately, such metrics rarely appear in the vast bulk of the individual evaluations. It is therefore not possible to comment beyond the remarks made in relation to Table 4.3.

Impact of evaluation

The final column of the left-hand side of Table 4.1 seeks to capture the impact of the evaluation in terms of the awareness of the concerned policy makers of its findings, and any changes to policy that took place following the evaluation.

This information was never provided in the published documents consulted and had to be obtained from those undertaking the evaluation. It therefore has all the well-established limitations of self-reported data. Also of concern is that this information could not be obtained in 8 cases.2

Nevertheless, a summary of the impacts documented in Column 8 found that in 17 cases there was a presentation to policymakers and some changes were implemented. In 14 cases there was a presentation made to policymakers but no awareness of changes to the programme being implemented. In 2 cases the results were published or sent to the policymakers, but not presented to them.

Perhaps the most disappointing finding was that in 7 cases the results were never presented to policymakers and the evaluators were also unaware of any policy changes that followed from the evaluation.

Overall, this suggests that in about one-third of cases the evaluation appears to have had an impact in the sense that policymakers were both aware of its findings and changes to the programme were implemented.3 A case may also be made that evaluations were successful if policymakers were aware of their findings, even if no changes were made. On those grounds almost 75% of the evaluations where an outcome has been specified could claim to be successful. However, the reasonable aim should be to achieve 100% amongst reliable studies.

4.4. Evaluation coverage and quality

Performance metrics

Column 9 of Table 4.1 shows the performance metrics reported for each of the 50 evaluations. In some evaluations, only a single metric is used to judge effectiveness whereas in others up to eight different metrics are used. What emerges is the, almost bewildering, diversity of metrics used by those conducting evaluations of SME and entrepreneurship policies.

Table 4.4 seeks to structure that diversity. It takes only the 12 metrics that are used in more than a single evaluation and shows how, in most cases, their usage varies between the eight policy areas. The two exceptions are Employment, which is used in 28 out of 50 evaluations, and Sales, which is used in 27 evaluations.

The other metrics are used much less frequently and, as Table 4.4 shows, tend to be concentrated in some policy areas, yet absent from others. For example, the crucial metric of Survival is used in only about one-third of the evaluations, most of which are in the policy areas of Innovation and Inclusive Entrepreneurship. The absence of a Survival metric in 14 out of the 15 Finance evaluations has to be a cause for real concern. A similar pattern emerges from the other rows of Table 4.4 with important metrics such as Value Added and Productivity appearing in comparatively few evaluations and, where they are used, being limited to only a few policy areas.

The policy significance of this patchy and inconsistent use of metrics is that it makes it difficult to make informed decisions – even when evaluations have been undertaken – when each evaluation uses different metrics. It will be recalled that the theoretical ideal is for all policies to have the same marginal impact – such as cost per job created – across all policy areas, implying there was no benefit in public funds being transferred from one policy area to another.

However, to make such a judgement requires the same metric – such as cost per job created – to be used across all policy areas. The evidence from Table 4.4 clearly shows that no single metric is consistently used. Even metrics such as Sales or Employment are only used in about half of the evaluations.

Also of concern is that some policy areas seem to have “favourite” metrics which are not used in other policy areas. This makes it impossible for policymakers to assess, on the basis of evaluations, the benefits of shifting funding from one policy area to another.

The evidence from Table 4.4 points to the value of having at least three “common” metrics to be used in all evaluations of SME and entrepreneurship policies and programmes. It suggests these should be Sales, Employment and Survival. These could then be supplemented by others appropriate for the policy area – such as Patents for Innovation evaluations or Wages for Enterprise Culture and Skills or Areas of Disadvantage evaluations.

Table 4.4. Evaluation metrics used in the evaluations

Metric	Number of evaluations using the metric	Comment
Employment	28	Widely used across all policy areas
Sales	27	Widely used across all policy areas
Various Accounting Metrics	13	Never used in the policy areas of Inclusive Entrepreneurship or Enterprise Culture and Skills
Productivity Metrics	13	Used primarily in evaluations of Finance and Areas of Disadvantage
Survival	11	Used primarily in the policy areas of Inclusive Entrepreneurship and Innovation. Hardly used in Finance evaluations or Areas of Disadvantage
Wages	6	Used in Finance, Innovation and Internationalisation and Inclusive Entrepreneurship evaluations
Profits	5	Use is primarily in Finance evaluations
Value Added	4	Use is limited to evaluations in Finance, Innovation and Areas of Disadvantage
Overseas Sales	4	Used in four studies of Innovation and Internationalisation
Reported Competencies	2	Used only in Enterprise Culture and Skills evaluations
Entry into Business	2	Used once in Enterprise Culture and Skills and once in Innovation evaluations
Patents Sought	2	Used only in Innovation evaluations

Survival

It was noted earlier that an important limitation of many SME and entrepreneurship policy evaluations was their failure to take full account of the Survival/Non-Survival of enterprises. This is of particular concern because of the low survival rates of SMEs, and of new firms in particular. Evaluations which report changes in the sales or employment amongst recipients only when they are trading therefore risk overestimating the impact of the policy if a large proportion of these firms cease to trade shortly afterwards.

Unfortunately, it appears from column 10 of Table 4.1 that, even amongst this selection of high-quality evaluations, only 15 out of 50 reported taking account of enterprise Survival/Non-Survival.

Step Level and Evaluation Quality Score

The final two columns of Table 4.1 present information on the Step Level for each evaluation, together with our more challenging Evaluation Quality Score (EQS).

Using the Six Steps ranking, 43 out of the 50 Evaluations (86%) are ranked at Step VI – the highest possible rank. As noted earlier, the OECD 2007 Framework was only able to identify 6 Step VI studies out of the 41 (15%) that were included. This points to the considerable improvement in the quality – and hence the reliability – of evaluations in this policy area.4

However, this overall improvement in quality has brought with it a recognition that even Step VI evaluations have potentially important limitations. For this reason, Section 4.3.2 sets out the more challenging EQS on which each evaluation is also scored. These outcomes were used earlier in Table 4.2; it showed that 30 out the 50 evaluations scored 4 and 12 scored 5. In most cases the difference between a score of 4 and a score of 5 was that, in the former case, there was either no, or imperfect coverage, of survival/non-survival.

The key lesson is that, for most countries and for most policy areas, there are no longer either technical or data-based reasons for either not conducting evaluations, or for conducting sub-optimal evaluations.

The 50 evaluations therefore constitute a substantial and reliable group upon which to derive conclusions on the effectiveness of SME and entrepreneurship policy and its constituent policy areas. It is clear there have been considerable improvements in both data and analysis since 2007. In the review of 42 evaluations carried out in the 2007 OECD Framework, only 6 would have been of sufficient reliability to merit inclusion in the current review.

References

[1] OECD (2007), OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes, OECD Publishing, Paris, https://doi.org/10.1787/9789264040090-en.

Notes

← 1.  In a small number of cases these were also not clearly specified. Here our judgement was based on the focus of the published evaluation.

← 2.  This could be a biased sample of in many respects – favouring more recent evaluations or those where memories are more favourable. The reported views on the impact of the evaluation on policy could also be influenced by a desire to seek more work.

← 3.  Of course this does not imply that it was the evaluation findings that brought about the change

← 4.  For example, in 2007 there were no Randomised Control Trial (RCT) studies to report whereas this Framework includes RCTs from Germany, Chile, Mexico, Netherlands and the United Kingdom.