Evaluation of STI policies


Rationale and objectives

The role of evaluation is to generate information about the appropriateness and effectiveness of public policy interventions. This information can be used to assess and “enlighten” processes of learning around policy practices and performance, stimulate discussions among actors (e.g. about appropriate evaluation criteria), signal quality and reinforce reputations (e.g. in public research), and allow policy makers to account for public spending choices. Evaluation results may prompt a re-positioning of policies and programmes, shape the allocation or re-allocation of public funding (e.g. more generous block grants to top-performing universities) and inform the development of national STI strategy.

Major aspects

There is agreement in principle that all policy interventions should be evaluated, but there is no clear consensus on the right time to do so, the level of aggregation or the assessment criteria.
Evaluation takes place at different stages of the policy cycle (ex ante, mid-term, ex post). It may be implemented as part of a contract (e.g. R&D programme funding) or enforced by law (e.g. the US Government Performance and Results Act).
Individuals, projects, organisations (e.g. universities, funding agencies), programmes, policies and even the overall STI system can be evaluated. The evaluation can examine management processes (process-oriented) or outcomes vis-à-vis pre-defined objectives (impact-oriented) (Figure 5.5). It can be carried out by external experts or by those evaluated (e.g. self-evaluation of public research institutions in the Netherlands).

Figure 5.5 Primary purposes and orientation of STI policy evaluation, 2012
Based on own country rating

Note: Country rating to the question: What are the purposes and orientations of STI policy evaluation in your country? A summative evaluation measures the impact a policy programme may have upon the problems to which it was addressed. A formative evaluation monitors the way in which a programme is being administered or managed so as to improve the implementation process.

Source: Country responses to the OECD Science Technology and Industry Outlook 2012 policy questionnaire.



Evaluation of individuals, organisations or national STI systems focuses on their performance of defined missions or functions. Evaluation of policies and programmes typically sets out to demonstrate input, output or behavioural additionality of public intervention, i.e. the extent to which intervention supplements rather than substitutes for private inputs (e.g. R&D tax incentives), contributes to create more output (e.g. reform of higher education), and changes sustainably the behaviour of a target population (e.g. green subsidies).
Assessment methods and criteria vary, depending on the kind of information sought. They matter because those who are evaluated typically learn to perform better over time. This is desirable if evaluation criteria steer actors to perform beneficial activities they otherwise might not perform, such as strengthening academic linkages with industry. However, they can also have perverse effects: for instance, peer review tends to favour conservative research and well-established research groups; a focus on publication and citation counts can discourage activities other than academic publishing, often to the detriment of teaching; and too strong a focus on patent and spin-off counts and on research income from private sources can promote short-termism.

Recent policy trends

STI policy evaluation has recently gained more policy attention because governments devote significant resources to R&D and innovation during fiscal crises. Fiscal constraints have raised the need to demonstrate value for public money but limit the resources available for evaluation.
STI policy evaluation faces the same complex challenges as STI policies themselves. Policy interventions typically seek to affect complex phenomena that involve a number of actors and institutional settings. Their evaluation must also deal with this complexity. A system-level assessment (meta-evaluation) must draw on various evaluation exercises, typically distributed across the policy landscape. Addressing social challenges requires adapting evaluation methods and criteria often based on investment models to capture non-economic outcomes and the social impact of STI policy. Adjusting to globalisation means expanding the scope of STI policy evaluation and further increases its complexity.
Governments have consolidated the legal framework of evaluation, streamlined evaluation procedures, sometimes through the establishment of a single dedicated agency, or reinforced the co-ordination of evaluation units. Besides general efforts to build an evidence-based STI policy knowledge base (through the development of impact assessment studies and the systematisation of evaluation), some countries have implemented a whole-of-government approach to evaluation, many have sought to harmonise practices by defining common methodologies and consolidating indicators, and a few are building data infrastructures and expert communities (Table 5.1). The United States and Japan have been particularly active in setting up science of science and innovation policy (SciSIP) initiatives to develop, improve and expand models, analytical tools, data and metrics that can be applied in STI policy decision-making processes. Norway also has set up a SciSIP research programme over 2010-14 called “FORFI” .

Table 5.1 Major shifts in STI policy evaluation over the past five years

Consolidating framework conditions for evaluation

Promoting a culture of evaluation

Belgium (Wallonia and Capital), Brazil, Poland, Portugal, Russian Federation, Turkey

Enforcing evaluation by law

Belgium (Wallonia and Capital), Canada, Hungary (higher education institutions)

Establishing performance agreements and/or contracts with central government

Finland (higher education institutions), France, Luxembourg

Increasing budget allocated to evaluation policy

the People's Republic of China

Agencification and co-ordination

Establishing new evaluation units

Poland, South Africa

Streamlining evaluation exercises (e.g. through a single agency)

Argentina, France, Korea, Finland, Israel, Italy, Portugal, Slovenia, Turkey, Netherlands

Increasing co-ordination of evaluation units

Poland

Evaluation capacity building

Implementing a Whole-of-Government approach/ framework for policy evaluation and impact assessment (IA)

Australia, Canada, Finland, Ireland, Japan, Russian Federation, South Africa, United Kingdom

Defining standards, guidelines and methodological framework for evaluation

Argentina, Austria, China, Colombia, Estonia, Japan, Netherlands, Spain, Switzerland, United Kingdom

Developing and consolidating STI and key performance indicators (KPIs)

Australia, Belgium (Capital), Colombia, Denmark, Finland, Norway, Slovenia, Spain, Switzerland, Turkey

Building STI policy data infrastructure, e.g. science of science and innovation policy (SciSIP) initiatives

United States, Japan, Korea

Building evaluation and IA experts community

United States

Source: Country responses to the OECD Science Technology and Industry Outlook 2012 policy questionnaire.

 

Access STI Outlook Policy database (demonstration version):

Trends in evaluation practices

Major evaluation exercises

  • System evaluation
  • Policy evaluation
  • Programme and project evaluation
  • Institutions evaluation
  • Staff evaluation

Institutionalisation of evaluation

Impact of evaluation