Improving Governance with Policy Evaluation

Annex A. Data and methodology

This comparative analysis of policy evaluation builds on the data collected though the 018 OECD Survey on Policy Evaluation (hereafter, the “OECD Survey”). The OECD Survey is a direct response to the request to collect better data in the area of evidence informed policy making by the Public Governance Committee, in the context of the development of a Policy Framework on Sound Public Governance. (see https://www.oecd.org/governance/policy-framework-on-sound-public-governance). This Framework was derived from the aim to support policymakers by consolidating an integrated vision and coherent narrative of the main elements of sound public governance. The information on policy evaluation practices was gathered as part of the effort to inform the development of the Framework chapters on policy formulation, implementation and evaluation.

While the report mainly draws on the data gathered through this OECD Survey on Policy Evaluation, it also uses related data from the OECD performance budgeting survey (https://www.oecd.org/governance/budgeting/internationalbudgetpracticesandproceduresdatabase.htm) and also from the work on measuring regulatory governance, under the Regulatory Policy Committee (http://www.oecd.org/gov/regulatory-policy/measuring-regulatory-performance.htm). It also relies on comparative data on the Centres of Government (COG) from the “Centre Stage” Publications and related COG surveys (see https://www.oecd.org/gov/centre-stage-2.pdf).

After a consultation and approval by the Public Governance Committee Delegates, and a piloting with a few volunteer countries, the survey was officially launched in mid 2018, with a range of countries involved in the OECD work on public governance. The process of data collection ended finally in mid 2019, with data being available for 42 countries. Countries were allowed to update the results to show any reforms in place and implemented as of mid 2019.

The survey is structured in two parts:

• Policy evaluation across government.

• Policy evaluation at the sector/thematic level.

In the first part of the survey, respondents were asked to provide information and data from a whole of government perspective on policy evaluation. The questionnaire was sent to the main institution responsible for the promotion, coordination and implementation of policy evaluation across government and answered by civil servants at director/assistant secretary level. The current report mainly draws from this main part of the survey.

The survey had a second component at sector and thematic level where respondents from the ministries of health and public sector reform were asked to provide information and data about the governance of policy evaluation in their institution. The current report has done a preliminary use of the data for the purpose of highlighting variations between main practices and some of the sectoral practices.

Survey Responses

Data were received from 42 countries (including 35 OECD member countries with the exception of Luxembourg). More specifically, 42 countries replied to the first part of the OECD Survey, and 33 countries’ ministries of health or equivalent institution and 31 ministries of public sector reform or equivalent institution replied to the second part of the survey. In terms of the non-members at the time of running the survey, responses were received from Argentina, Brazil, Bulgaria, Colombia, Costa Rica, Kazakhstan, and Romania.

The OECD Survey questionnaire is available at the link below: www.oecd.org/gov/institutionalisation-of-policy-evaluation-questionnaire.pdf.

Throughout the report, data are presented according to countries, which have been abbreviated according to the official ISO country codes.

Table A.1. ISO codes of the countries referred to in this report

1	Argentina	ARG	22	Israel	ISR
2	Australia	AUS	23	Italy	ITA
3	Austria	AUT	24	Japan	JPN
4	Belgium	BEL	25	Kazakhstan	KAZ
5	Bulgaria	BGR	26	Korea	KOR
6	Brazil	BRA	27	Latvia	LVA
7	Canada	CAN	28	Lithuania	LTU
8	Chile	CHL	29	Mexico	MEX
9	Colombia	COL	30	Netherlands	NDL
10	Costa Rica	CRI	31	New Zealand	NZL
11	Czech Republic	CZE	32	Norway	NOR
12	Denmark	DNK	33	Poland	POL
13	Estonia	EST	34	Portugal	PRT
14	Finland	FIN	35	Romania	ROU
15	France	FRA	36	Slovakia	SVK
16	Germany	DEU	37	Slovenia	SVN
17	Great Britain	GBR	38	Spain	ESP
18	Greece	GRC	39	Sweden	SWE
19	Hungary	HUN	40	Switzerland	CHE
20	Iceland	ISL	41	Turkey	TUR
21	Ireland	IRL	42	United States	USA

Process for data collection and validation

The survey was distributed through the network of the Public Governance Committee Delegates, who were invited to designate a focal point at national level. The Secretariat carefully reviewed all the survey submissions, with additional requests for information on approaches, clarifications and potential good practices. A process of validation and mutual dialogue with each of the responding country did follow. Moreover, an experts’ meeting was organised in September 2019 where the survey respondents were invited to the OECD to discuss preliminary results and given a final opportunity to validate their country’s data. (See http://www.oecd.org/gov/agenda%20expert%20group%20meeting%20final.pdf).

The survey questions were designed to be responded to in a quantifiable manner, such as yes or no options and multiple-choice questions. Whenever possible, the OECD Survey offered space for countries to give additional information. While the survey can provide aggregate comparative information, additional boxes and qualitative analysis were added to reflex the complex processes of policy evaluation and evidence informed policy making.