Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes 2023

Reader’s Guide

Overview

Part I of this Framework provides information for readers seeking high-level guidance on the principles of conducting reliable impact evaluation. Of particular importance is the Six Steps to Heaven tool, which identifies progressively more reliable levels of evaluation based on features of the treatment-control group match.

Part II is most relevant for readers interested in evaluation findings and their implications for SME and entrepreneurship policy. It sets out the findings and policy messages from meta-evaluations and 50 individual high-quality evaluations. Table 4.1 summarises the findings of each evaluation. Section 4.3 and chapter 5 discuss the policy issues that emerge.

Part III is recommended for readers with an interest in exploring how to adjust the mix of SME and entrepreneurship policy to focus on the more effective parts of the policy portfolio. Chapter 7 explores the relative effectiveness of “Hard” and “Soft” support, the importance of targeting policy beneficiaries, and the importance of “Macro” interventions.

Part III also provides information for readers interested in how to improve SME and entrepreneurship policy evaluation. Chapter 8 highlights a number of areas for improvement in evaluation, such as better specifying policy objectives and increasing the scale of evaluation. Readers seeking more detailed insights on potential indicators, data sources and methodologies for evaluating specific SME and entrepreneurship programmes can examine the descriptions of the 50 high-quality evaluations set out in Annex B by type of policy intervention. Those interested in how to set up an evaluation programme to evaluate the impact of the government emergency support measures for SMEs and entrepreneurship introduced during the COVID-19 crisis, and in the role of evaluation for crisis responses more generally, can gather information from Chapter 9.

The remainder of this Reader’s Guide sets out in further detail the main content included within the different parts of the Framework.

Part I: Evaluation principles and state of evaluation practice

Part I covers the “what, why and how” of SME and entrepreneurship policy evaluation. It reaffirms the case made in (OECD, 2007_[1]) that it is vital to conduct reliable evaluations of SME and entrepreneurship policies and programmes. It also addresses the state of current SME and entrepreneurship policy evaluation practice, arguing that there is insufficient reliable evaluation evidence in the field of SME and entrepreneurship policy and setting out what can be done about it.

In more detail, Part I covers:

• The meaning and role of evaluation in SME and entrepreneurship policy.

• Weaknesses in current SME and entrepreneurship policy evaluation practice.

• Lessons for evaluation.

Part II: Evaluation methods and findings

Part II reviews methods and findings from an international selection of 50 impact evaluations drawn from a range of OECD countries and policy intervention areas. All meet high standards for methodological reliability (i.e. they are placed on Step V or Step VI of the Six Steps to Heaven framework). These are exemplars for evaluation methodologies, offering models for the data sources used and the analytical tools employed. Furthermore, policy makers can be confident that where conclusions are reached, they are based upon sound data and appropriate analytical techniques. The profile of each evaluation follows a standard template covering aspects of the programme assessed, the evaluation methodology used and the evaluation findings.

In more detail Part II covers:

• Evidence from international meta-evaluations.

• Methods of individual high-quality evaluations.

• Evaluation findings.

• Lessons for policy.

• Lessons for evaluation.

Part III: Learning the lessons

Part III draws out the lessons on how to improve evaluation and policy. It explores the major finding, already highlighted in the earlier Parts of the Framework, that the results of reliable policy evaluations are mixed, in terms of whether or not SME and entrepreneurship policy is judged to be effective and efficient across a range of objectives of the policy. While some evaluations estimate positive impacts, others find no impacts on key outcomes such as sales growth, employment growth or business survival, and others still find impacts on some targeted variables but not on others. The discussion considers why. For example, the probability of impact may be related to the contexts in which different programmes are delivered, the timing of the evaluation or the nature of the policy pursued.

A key distinction is made between “Hard” and “Soft” support programmes. Hard programmes involve an important element of financial support, whereas Soft programmes focus on aspects of training, advice and mentoring. A hypothesis is put forward that governments might be able to increase the overall impact of their SME and entrepreneurship policy portfolios by recognising that the current evidence base is pointing to clearer impacts from “Hard” than from “Soft” programmes. This is just a hypothesis at the current time, but points to the need for evaluation programmes that are able to make the comparisons between Hard and Soft policy intervention types and increase the evaluation evidence for Soft policies, for which current reliable evaluations raise doubts about effectiveness.

Part III also contains a section on the role of evaluation for government crisis response measures for SMEs and entrepreneurship in times of economic shock. It illustrates the issues through an exploration of recent government COVID-19 SME and entrepreneurship support interventions, showing how key issues in this Framework need to be given greater prominence, for example in terms of setting out clear objectives and expenditures and using counterfactual evaluation methods. It stresses the need to build evaluation arrangements into future government policy crisis response measures and proposes international co-operation in the impact evaluation of the COVID-19 support.

In more detail Part III covers:

• Problems with the focus and design of SME and entrepreneurship policies.

• Possible avenues for rebalancing SME and entrepreneurship policies to increase impact.

• Improvements needed in SME and entrepreneurship policy evaluation practice.

• Applying the lessons of this report to the evaluation of COVID-19 SME and entrepreneurship support and other policy responses to shocks.

• Conclusions from the Framework overall, including 13 key recommendations for SME and entrepreneurship policy makers.

Annex A: Explanation of the template for the 50 evaluation profiles

Annex A provides outlines the template used to prepare the profiles of the 50 evaluation cases presented in detail in the report, explaining the information sought and the rationale for it.

The template includes information relevant to judging the quality and reliability of each evaluation based on how far it is in line with key evaluation principles set out in this Framework, such as clearly specifying the objectives to be evaluated against, the impact measures used, whether both survivors and non-survivors are tracked and whether control groups are established at high levels of the Six Steps to Heaven tool. These measures of quality and reliability were used to select the 50 exemplar cases described in detail in Annex B.

Annex A also gives the details and rationale for other key information contained in the evaluation profiles, such as the programme area and the target populations.

Annex B: Methods and findings of the 50 individual evaluations

Annex B provides systematic information for each of the 50 reliable evaluations identified and reviewed for this Framework through a completed template for each evaluation study. Table 1 outlines the types of information provided for each evaluation.

Table 1. Information available for each reviewed evaluation study in Annex B

Basic information on the programme evaluated	Methodology of the evaluation	Findings	Utilisation of the evaluation
Dates of programme Objectives of programme Topic/policy area Target groups Whether the programme has a regional/local focus Programme expenditure	Source of evidence Impact variables used Whether survival is taken into account Data sources Step Level and Evaluation Quality Score Reliability comments	Key findings Macro impact of the programme	Policy impact of the evaluation

Table 2 classes the individual reliable evaluations by main category of policy intervention – access to finance, business advice, internationalisation etc. Further information is given for each evaluation on our categorisation of whether the intervention is largely “Hard” or “Soft”, and on our assessment of the level of reliability of the evaluation as judged by our view of its level on the Six Steps to Heaven tool, the clarity of objective setting of the policy evaluated and our Evaluation Quality Score (explained in Annex A). This is aimed at supporting readers in browsing for information by particular types of policy interventions and in assessing the evaluation methodology used by scanning across our reliability indicators.

The full information on each evaluation can be consulted in Annex B by using the evaluation reference number given in Table 2.

Table 2. The policy evaluations reported by main category of intervention

Evaluations of finance programmes
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Specification Score	Evaluation Quality Score
B1	Direct financial assistance from the Australian government, including grants, subsidies and rebates	Australia	Hard	V	1	2
B2	Flemish government Entrepreneurship Agency´s programme	Belgium	Hard	VI	1	4
B3	Canada Small Business Financing Program (CSBFP)	Canada	Hard	V	2	3
B4	Subsidies allocated within the Czech Operational Programme for Enterprises and Innovation (OPEI)	Czech Republic	Hard	VI	1	4
B5	START and ZÁRUKA programmes	Czech Republic	Hard	VI	2	5
B6	Various grants managed by the Enterprise Estonia (EAS) government agency, i.e. start-up and development grants, research and development (R&D) grant, development of knowledge and skills grants, technology investment grants, export grants	Estonia	Hard	V	1	3
B7	Economic Development Operational Programme and Regional Development Operational Programmes from the EU Structural Funds and the Cohesion Fund	Hungary	Hard	VI	2	4
B8	Credit Guarantee Scheme Fondo Centrale di Garanzia (Central Guarantee Fund)	Italy	Hard	VI	2	4
B9	Japan’s Emergency Credit Guarantee (ECG) Programme	Japan	Hard	VI	2	4
B10	Credit guarantee schemes provided by Korea Credit Guarantee Fund (KCGF) and the Korea Technology Credit Guarantee Fund (KOTEC)	Korea	Hard	VI	2	5
B11	Modernisation of agricultural holdings, rural development programme	Lithuania	Hard	VI	1	4
B12	Entrepreneurship support programmes administered by various government agencies and ministries	Mexico	Both	VI	2	3
B13	Slovenia’s anti-crisis state aid programmes	Slovenia	Hard	VI	2	4
B14	Enterprise Finance Guarantee Scheme (EFG)	United Kingdom	Hard	VI	2	3
B15	Small Business Administration (SBA) loans (lending programmes 7a and 504)	United States	Hard	VI	2	4
Evaluations of business advice, coaching, mentoring and counselling programmes
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B16	Investment Network Programme administered by the Innovation Synergy Center	Canada	Soft	VI	2	5
B17	Chile Supplier Development Programme (Programa de Desarrollo de Proveedores - PDP) administered by the economic development agency CORFO	Chile	Soft	VI	2	4
B18	North Jutland Entrepreneurial Network (NiN) Programme	Denmark	Soft	VI	2	5
B19	Business coaching programme for new technology-based firms	Germany	Soft	VI	2	3
B20	Business counselling services for SMEs in Puebla region	Mexico	Soft	VI	2	4
B21	Business Link Programme	United Kingdom	Soft	VI	2	4
Evaluations of internationalisation programmes
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B22	Grants for industrial development allocated by the Industrial Development Agency (IDA) and by Forbairt	Ireland	Hard	VI	2	4
Evaluations of innovation programmes
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B23	Chile’s CORFO Seed Capital Programme	Chile	Hard	V	2	2
B24	Finnish Governmental National Technology Agency’s (TEKES) programme (Finnish acronym for young innovative growth companies)	Finland	Both	VI	2	4
B25	Polish In-Tech programme on science-industry collaboration, research and innovation, and product commercialisation	Poland	Hard	VI	2	4
B26	Portuguese Innovation Incentive System (SI Innovation), an instrument of the National Strategic Reference Framework (NSRF), included in the Operational Programme for Competitiveness Factors (COMPETE)	Portugal	Hard	V	2	3
B27	EBT and PYME participative loans (loan contracts) programmes allocated by the governmental agency Empresa Nacional de Innovacin (ENISA)	Spain	Hard	VI	2	5
B28	VINN NU (Win Now) programme operated by the Swedish Governmental Agency for Innovation Systems (VINNOVA)	Sweden	Hard	VI	3	5
B29	Swiss innovation policy administered by the Commission of Technology and Innovation (CTI)	Switzerland	Hard	VI	2	4
B30	The TUBITAK-TEYDEB public R&D programme administered by the Scientific and Technological Research Council of Türkiye (TUBITAK)	Türkiye	Hard	VI	2	4
B31	Business incubators in the United States	United States	Both	VI	2	5
Evaluations of enterprise skills and culture programmes
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B32	The Human Resources and Employment Operational Programme (HREOP)	Czech Republic	Soft	VI	2	4
B33	Junior Achievement Young Enterprise student mini-company (SMC) programme coordinated by the Jong Ondernemen Association	Netherlands	Soft	V	2	4
B34	Netherlands' Tax and Customs Administration (NTCA) tax training programme	Netherlands	Soft	VI	2	5
B35	Business Innovation and Skills (BIS) training programme	United Kingdom	Soft	VI	2	4
B36	Project Growing America through Entrepreneurship (GATE)	United States	Soft	VI	2	5
Evaluations of inclusive entrepreneurship programmes
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B37	Micro-entrepreneurship Support Programme (MESP)	Chile	Both	VI	2	4
B38	ACCRE start up support for the unemployed (Aide aux chômeurs créant ou reprenant une entreprise)	France	Hard	VI	2	5
B39	German start-up subsidy programme Einstiegsgeld	Germany	Hard	VI	2	4
B40	German start-up subsidy (SUS) programme Gründungszuschuss	Germany	Hard	VI	2	4
B41	German start-up subsidy programme Überbrückungsgeld (Bridging Allowance)	Germany	Both	VI	2	5
B42	Start-up Programme Fare impresa (Doing Business)	Italy	Hard	VI	2	4
B43	Social security reduction programme for youth self-employment from unemployment	Spain	Hard	VI	2	5
B44	The Swedish Start-up Grants programme (SEP Programme)	Sweden	Hard	VI	2	4
Regional and local evaluations
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B45	Pre-start support via a Funded Business Development Centre (PFBDC)	Spain	Both	VI	2	4
Evaluations of support programmes in areas of disadvantage
Annex table reference	Programme name	Country	Intervention type	Step Level	Objective Setting Score	Evaluation Quality Score
B46	Improving regional economic structures (Verbesserung der regionalen Wirtschaftsstruktur - GRW)	Germany	Hard	VI	2	4
B47	Regional policy determined by the Law 488/1992 (L.488)	Italy	Hard	V	2	4
B48	Regional policy determined by the Law 488/1992 (L.488)	Italy	Hard	VI	2	4
B49	Regional policy determined by the Law 488/1992 (L.488)	Italy	Hard	VI	2	4
B50	Regional Selective Assistance (RSA) Programme	United Kingdom	Hard	VI	2	4

Annex C: Examples of other relevant evaluation studies not included in the report

Annex C provides a list of 25 other relevant evaluation studies that are not included in the report, with information on the country, topic, year of the study and source of evidence. These are high-quality evaluations that could have been selected for inclusion, but were excluded on grounds of achieving diversity in the examples provided. Further high-quality evaluations could also have been listed and are available in the literature.

Annex D: Brief description of included evaluation methods

Annex D provides a table with information on key evaluation methods used in many of the selected evaluations. It provides a high-level overview of the methodological approaches and references to further reading.