OECD Development Co‑operation Peer Reviews: Italy 2019

6. Italy’s results, evaluation and learning

Abstract

This chapter considers how Italy plans and manages for results that are in line with the Sustainable Development Goals, builds evidence of what works, and uses this to learn and adapt.

Law 125/2014 calls for a results-based management system, which Italy is in the very early stages of developing. Today, monitoring Italy’s interventions and reporting results other than output indicators in a given country, sector or partnership is challenging. The evaluation system for Italian development co-operation was reorganised following the reform: responsibility still lies with MFAIC although the budget line is with AICS. Since 2014, Italy has adopted three-year rolling evaluation plans based on defined criteria; established an evaluation advisory committee; and set up an electronic register of independent evaluators. Italy uses evaluations to inform the design of future programme phases, but less explicitly to learn from successes and failures. It lacks a knowledge management system, or an intranet to connect field offices with Rome/Florence.

Management for development results

Peer review indicator:

A results-based management system is being applied

Law 125/2014 calls for a results-based management system, and the government has to report against some corporate targets on an annual basis. Italy is in the very early stages of developing a results‑based management system, first working to adapt procedures and guidance. Monitoring Italy’s interventions and reporting results other than output indicators in a given country, sector or partnership is challenging. More staff with relevant skillsets could help Italy develop an effective results-based management and information system. Using the sustainable development goals (SDGs) as an organising framework for such a system, as it has started to do in El Salvador, is one approach it can take.

The reform law commits Italy to creating a results-based management system

Law 125/2014 binds the Italian government and all its development co-operation actors to results-based management (Republic of Italy, 2014[1]). The introduction of a results-based management system is one of the key tenets of Italy’s forthcoming three-year Action Plan for effective co-operation, and its implementation will require the government’s full backing (MFAIC/DGCS-AICS, 2019[2]).

Italy is required to report progress against each of its instruments and partnerships on an annual basis. The latest progress report on Italy’s development co-operation lists the programmes and projects funded by the different ministries and local authorities, for the first time providing a status on the compliance of each project with five aid effectiveness indicators (Chapter 5) (MFAIC, 2018[3]). In addition, a 2019-21 corporate performance system (different from a results-based aid management system) is embedded in the Ministry-AICS convenzione and holds the institutions to account for technical and operational activities and targets. Performance management systems are a good foundation, but a clear purpose statement of what Italy aims to achieve, and by what means, including how it will stimulate leadership and encourage staff initiative and learning, is needed to transition to a more results-oriented management approach (Vähämäki and Verger, 2019[4]).

Italy is in the early stages of building a results-based management system

Italy acknowledges that it still has some way to go to build a results-based management system from the strategic level all the way down to the operational level, including knowledge management. This will require an overhaul of its existing system and culture, which are singularly project-focused, and largely delinked from a pathway for strategic results. Its lack of an overarching results framework raises two main challenges for Italy. First, in the partner countries, sectors or organisations it funds, Italy is not able to demonstrate how its specific interventions contribute to overall development outcomes – a point echoed by Senegalese authorities during the field visit (Annex C.3). Second, Italy is not able to aggregate or “roll‑up” project-specific results to outcomes and impacts at agency, or ministry, level. The lack of tools and skills to analyse results information limit the building of such a system (Chapter 4).

Italy mainly uses output and performance information at project and programme level, where possible using partner countries’ data to communicate and account for what has been achieved as a direct result of its interventions. This overlooks the link to the broader development outcomes to which Italy’s interventions are contributing, a finding confirmed in the field visit to Senegal (Annex C.5).

A clear articulation of the totality of Italy’s contribution to development results is largely absent. As a first step, outlined in the Action Plan for effective co-operation, Italy plans to design a system to manage for results, identifying any links between objectives, activities, results and budget. To build its results-based system, Italy will have to introduce proper incentives, establish robust ex-ante quality assurance for programme design, and ensure that data and information are collected and used at all levels. AICS is already working hard to adapt some of the procedures and manuals related to the call for proposals process for all of its partners (e.g. agreement templates, logframes) to a results-based management system. It is also developing training courses to help staff mainstream the results-based management approach throughout all programmes. The guiding principles on Managing for Sustainable Development Results could be a useful compass for Italy in organising its approach.

Using the SDGs as an organising framework offers a good way forward

More than half of Italy’s priority countries lack country strategies, which is a missed opportunity to set out the rationale for Italy’s engagement and bring together its investments and intended results (Chapter 2). Recent country strategies have called for a focus on outcomes and impact,1 even though Italy primarily uses output, or reach, indicators in its project logframes. Where they exist, country strategies do not specify what contribution Italy aims to make to sustainable development in the country.

Notwithstanding challenges in linking progress realised at the project-level with results achieved in the country or at the portfolio level, Italy has made progress in linking output indicators to the sustainable development goals (SDGs) in El Salvador. Here, AICS’s project-level indicators are mapped to SDG targets and the objectives and priorities identified in the government’s five-year national development plan (Plan Quinquenal de Desarrollo). Identifying and using the SDGs as the organising framework to harmonise with other partners and align to partner country frameworks is good practice (OECD, 2018[5]) that could be extended to the next generation of country strategies.

Italy is to a large extent (68.5%) aligned with partner countries’ own results frameworks and monitoring systems (OECD/UNDP, 2019[6]), likely due to the fact that it mostly adopts projects or programmes in line with government priorities and programmes. This alignment was evident in the agriculture and education sectors in Senegal, where Italy did not require separate data collection or parallel reporting, which was appreciated by the partner government. To start, Italy could continue to build on partner countries’ results frameworks to arrive at a few strategic results to which its country programme contributes.

Evaluation system

Peer review indicator:

The evaluation system is in line with the DAC evaluation principles

Law 125/2014 reorganised Italy’s development co-operation evaluation system; while responsibility still lies with MFAIC, the budget line is with AICS. Since 2014, Italy has adopted three-year rolling evaluation plans based on defined criteria, established an evaluation advisory committee, and set up an electronic register of independent evaluators.

Law 125/2014 defines evaluation functions and responsibilities in MFAIC and AICS

Under the new law, the responsibility for evaluation lies in the Ministry of Foreign Affairs and International Co-operation, yet evaluation expertise lies primarily in AICS. In 2017, the Evaluation Unit in the ministry was reorganised to align to Law 125/2014, which also requires it to use independent, external evaluators.2

Annual agreements, or convenzione, between the MFAIC and AICS spell out the legal arrangements regarding the evaluation function, and the new evaluation strategy guidelines set out the division of labour between the MFAIC and AICS in terms of purpose, methodology, and dissemination of learning from evaluations. The agreement includes a dedicated budget line for evaluation which is entrusted to AICS. The budget was EUR 500 000 for 2019, which includes the annual contribution to the Multilateral Organisation Performance Assessment Network (MOPAN) (Chapter 2). The operational guidelines state that the objective of evaluations is learning in order to inform future programming decisions, improve existing interventions and assess the value of investments for accountability purposes. Although this may seem ambitious for Italy, as the sixth largest DAC donor, it is important for it to use evaluations in this way and for this purpose.

The creation of the AICS resulted in most evaluation experts moving out of the ministry and into AICS, leaving only one part-time expert, one administrative official and one junior diplomat to assist the head of Office III in charge of evaluation (Annex D.2) that works across two units in MFAIC (Government of Italy, 2018[7]).

In practice, experts in AICS and MFAIC work closely together, but the restructuring has meant that the number of centralised evaluations conducted per annum slowed down starting in 2014, and have only started to pick up recently. Evaluation reports are publicly available on the MFAIC website. The evaluability of projects is determined at the project proposal stage, and the AICS Verification Team certifies a project or programme’s monitoring and evaluation plan, which is good practice.

An evaluation plan and operational guidelines ensure quality evaluations are used as management tools

The 2014 Peer Review recommended that Italy establish a medium-term evaluation plan based on clear criteria, and a management response system (OECD, 2014[8]).

The Joint Committee approved the first rolling three-year (2018-2020) evaluation plan in December 2017, which was updated one year later to the current 2019-2021 plan (MFAIC, 2018[9]). At the end of 2018, an Evaluation Advisory Committee composed of independent members representing academia, civil society and the Italian Evaluation Association was formally set up to provide quality assurance and strategic advice to the Evaluation Unit. It advises on ways to improve the quality of evaluations and on how management should respond, and promotes the dissemination of evaluation results.

The current three-year evaluation plan includes 13 bilateral project evaluations, of which: one strategic evaluation (CSO partnerships), one country programme evaluation, one sector programme evaluation, four multi-country evaluations, and six project/programme-specific evaluations. The five criteria for inclusion in the plan are:

1. Strategic priority in relation to the partner country or area of intervention

2. Continuity in a specific area of intervention in a partner country

3. Initiatives in emergency or fragile contexts

4. High volume of financial contribution

5. Innovative initiatives

These criteria serve to prioritise scarce resources and forge stronger links to the 2017-19 Policy Programming Document that advocates for “criteria and methodologies for evaluation using a results‑based approach” (MFAIC, 2017[10]). However, the plan does not aim to cover those ODA programmes overseen by the Directorate General for Italian Citizens Abroad and Migration Policies (DGIT), the Ministry of Interior, or other government ministries. This means that direct evaluation oversight and accountability for bilateral programming is not conducted through a development lens for a significant portion of Italy’s recorded ODA (Chapter 4).

Recently approved generic terms of reference for evaluations also require an assessment of cross-cutting themes (gender, environment, and human rights), which is good practice (Chapter 3).

Joint evaluations are conducted, but could be further prioritised

Italy conducts joint evaluations with other development partners whenever possible, including a recent one with the UK Department for International Development (DFID) on agriculture and livestock support for the Syrian people (CIHEAM, 2017[11]). Italy also supports partner countries’ evaluation function for many of its programmes, and encourages joint evaluations from the onset during the programme formulation stage, also participating in mutual accountability reviews (Chapter 5). The 2017 Global Monitoring Round found that fewer evaluations were conducted jointly with partner governments in 2017 compared to 2015, which Italy is seeking to improve (Chapter 5).

The Italian National Statistics Institute (ISTAT) works to strengthen the statistical capacity of partner countries. It recently sent short-term experts to two priority partner countries to support existing projects on the population census (Ethiopia) and modernising the statistical system through a twinning effort (Tunisia) (MFAIC, 2018[3]). Another example is Italy’s support to monitor SDG progress via the Central Bureau of Statistics of the Palestinian Authority database of SDG indicators. Two components of the ex‑ante aid effectiveness marker allow for a programme to score higher points: if indicators and data are partner countries’ own, or if the intervention works to strengthen statistical systems (Box 5.1).

Institutional learning

Peer review indicator:

Evaluations and appropriate knowledge management systems are used as management tools

Italy uses evaluations to inform the design of future programme phases, but less explicitly to learn from successes and failures. It does not have a knowledge management system, or an intranet that connects field offices with Rome/Florence. A regularly-updated blog is used to communicate positive stories from across programmes.

There is no knowledge management system in place to allow practitioners to exchange

Neither MFAIC nor AICS have an institutionalised approach in place that builds on results and evidence for learning and analysis, linking back to overall programme management. Recent evaluations commissioned by Rome are available to the public on the website, but dissemination of evaluation findings from decentralised evaluations commissioned directly (or jointly) in partner countries for learning purposes is ad hoc. In fact, AICS has not set up a formal structure, platform or intranet for knowledge management, despite its wealth of knowledge from development co-operation evaluations and results.

AICS has no system-wide platform in place to exchange good practices and learn from mistakes for the future, or even an intranet for staff to access information. The lack of a working platform to access corporate-level information and standards, as well as to exchange information on projects or country and sector experience is a missed opportunity for learning from both failure and success, rendered more challenging by the limited possibilities for staff rotation (Chapter 4). While a new digital management platform connecting AICS Rome, Florence, and field offices is under development, a more fundamental shift in the working culture of AICS and DGCS will be required.

AICS does maintain a list of publications by sector and country on its website. The AICS website also links directly to the Journal of Agriculture and Environment for International Development,3 which illustrates the importance Italy places on linking scientific evidence and keeping its technical support in partner countries up to date (Annex C.3). It also publishes a monthly magazine “La Cooperazione Italiana Informa” and maintains a blog, “Oltremare”,4 containing articles and notes from the field by staff, such as on the response to Cyclone Idai in Mozambique. The focus of these stories tends to be on the positive effects of Italian development co-operation, however, and less on what was learned and how that can be useful for future decision-making.

References

[11] CIHEAM (2017), Agriculture and Livestock Support for Syrian People, CIHEAM, Valenzano, http://www.iamb.ciheam.org/uploads/attachments/655/Brochure_Agriculture_Livestock_Syria_DEF_21-06-18.pdf (accessed on 28 May 2019).

[7] Government of Italy (2018), OECD-DAC Peer Review: Italy Memorandum 2019, Government of Italy, Rome.

[9] MFAIC (2018), Programma triennale delle valutazioni delle inizative di cooperazione 2019-2021 (in Italian), Ministry of Foreign Affairs and International Co-operation, Rome, https://www.esteri.it/mae/resource/doc/2019/02/programma_triennale_delle_valutazioni_delle_iniziative_di_cooperazione_2019-2021_0.pdf (accessed on 28 May 2019).

[3] MFAIC (2018), Relazione annuale sull’attuazione delle politica di cooperazione allo sviluppo (in Italian), Ministry of Foreign Affairs and International Co-operation, Rome, https://www.esteri.it/mae/resource/doc/2018/01/relazione_2016_finale.pdf (accessed on 27 May 2019).

[10] MFAIC (2017), Three-year Programming and Policy Planning Document, 2017-2019, Ministry of Foreign Affairs and International Co-operation, Rome, https://www.esteri.it/mae/resource/doc/2018/07/pro_triennale_2017-2019_en.pdf (accessed on 24 May 2019).

[2] MFAIC/DGCS-AICS (2019), Piano di azione per l’efficacia della cooprazione pubblica allo sviluppo, 2019-2021 (in Italian), Ministry of Foreign Affairs and International Co-operation, Rome.

[5] OECD (2018), “Results approaches for the SDG era: shared challenges and collective solutions Key Messages from the Workshop”, https://www.oecd.org/dac/results-development/docs/Results-Workshop-October-18-Key-Messages-FINAL.pdf (accessed on 4 July 2019).

[8] OECD (2014), OECD Development Co-operation Peer Reviews: Italy 2014, OECD Development Co-operation Peer Reviews, OECD Publishing, Paris, https://doi.org/10.1787/9789264213241-en.

[6] OECD/UNDP (2019), 2018 Monitoring of the Global Partnership for Effective Development Co-operation, OECD, Paris and UNDP, New York, https://effectivecooperation.org/wp-content/uploads/2019/06/Part-I-of-the-Global-Partnership-2019-Progress-Report.pdf.

[1] Republic of Italy (2014), Law no. 125 of 11 August 2014 - Unofficial translation, https://www.aics.gov.it/wp-content/uploads/2018/04/LEGGE_11_agosto_2014_n_125_ENGLISH.pdf (accessed on 12 April 2019).

[4] Vähämäki, J. and C. Verger (2019), “Learning from Results-Based Management evaluations and reviews”, OECD Development Co-operation Working Papers, No. 53, OECD Publishing, Paris, https://dx.doi.org/10.1787/3fda0081-en.

Notes

← 1.  For example, the Ethio-Italian Cooperation Framework 2017-2019 says it “will promote a monitoring and evaluation system…that will focus more on outcomes and impact rather than outputs (results-based management).”

← 2.  In 2018, Italy created an open online digital evaluation roster. The list of independent evaluators draws from academia and the private sector, and is in line with the DAC evaluation criteria.

← 3.  See https://www.jaeid.it/index.php/JAEID.

← 4.  See https://www.aics.gov.it/oltremare/.