Sustainable Results in Development

Annex A. Background and methodology for the project

Background

Since 2015, the OECD-DAC Results Community has explored ways to build a more co-ordinated approach to results‑based management which supports harmonisation of indicators and promotes ownership by partner countries. Analysis has focused on how a shared commitment to achievement of the Sustainable Development Goals (SDGs) can translate into increased use of the goals, targets and indicators as a common results framework for both providers of development co-operation and partner countries. This work led to a policy paper and a guidance note (Engberg-Pedersen, 2018[1]; OECD, 2018[2]) that present a menu of SDG targets and indicators that can strengthen providers’ results frameworks, facilitate data collection and use, and offer improved platforms for dialogue with partner countries.1

Participants at the April 2018 OECD-DAC Results Community workshop expressed strong interest in building on this work to further analyse how partners and providers can, in practice, use the SDG framework as a shared platform for results measurement and management. During the workshop, providers acknowledged internal pressures to report results domestically or at corporate level that can contradict commitments to harmonise their approach, and to support and align to their partners’ country results frameworks. Participants agreed on the importance of enhanced co-ordination among providers and between partners to reduce the production of duplicative and overlapping results data. They also stressed the need to ensure the interoperability of results data systems from the outset (OECD, 2018[3]).

At the same time, developing countries are working to ensure their national development plans and country results frameworks reflect the SDGs which they prioritise, and that they have the capacity and resources to monitor progress. They want providers to align with these priorities in their strategies, commitments and results systems (Zwart and Egan, 2017[4]).

Recent OECD analyses have highlighted these tensions – which are both technical and political/organisational in nature – and the potential of the SDGs to serve as a framework to address some of these challenges.2 The SDG framework (SDG targets, indicators and reporting), and the shared commitment to achieving the SDGs provide an opportunity and a basis to practically manage this tension and work towards a more co-ordinated approach among providers, facilitating increased alignment of provider results frameworks with those of partner countries.

However, more evidence and analysis are required to examine how using the SDG framework as a shared framework for results measurement and management can support enhanced co-ordination and alignment of provider results frameworks to country-led result frameworks in practice. Workshop participants advocated for further work to identify challenges and bottlenecks, and to showcase good practice of co-ordinated collection and use of SDG (or SDG-similar) indicator data for results-based management. During the discussion, participants suggested case studies based on selected SDG targets in selected partner countries, where national development plans have already to some extent been aligned to the SDGs, as a way to focus on identifying challenges and developing solutions to facilitate a more co-ordinated approach to results-based management at country level (OECD, 2018[3]).

Purpose and objectives of the project

Purpose: to generate evidence, analysis and good practice examples of how development co-operation providers and partners3 can concretely use the SDG framework as an entry point for co-ordinating around, investing in and using country-led results frameworks and data which are aligned to the SDGs from both a technical/methodological and an organisational/political perspective.

Objectives: Undertake three case studies, each based on a selected SDG target/indicator (but taking the broader goal into account) to:

• demonstrate linkages and synergies, but also challenges, around aligning providers’ results frameworks (at corporate and country levels) to the SDGs, and to country-led results frameworks that have already domesticated the SDG framework (organisational/ political)

• explore possibilities for enhanced co-ordination and harmonisation4 around collection, analysis and use of results data in partner countries, including for voluntary national reviews (organisational/political)

• identify and document good practice in relation to investing in and using country-led results frameworks and data (especially administrative data), which are aligned to the SDGs, for results-based management by and between partners and providers (technical)

• demonstrate the potential for SDG data to: be collated and used as a platform for dialogue; and monitor country-level progress towards SDG targets and development co-operation contribution towards this progress (technical).

An analytic framework and research questions guide data collection against these objectives. Figure A.1 provides an overview of the analytical framework.

Figure A.1. Analytic framework for SDG case study project

Approach

Many other projects and initiatives contribute to enhanced SDG data collection and use in developing countries. This project looks specifically at how development co-operation providers can contribute to enhanced alignment, measurement and data use in the context of the SDGs. At the same time, acknowledging and benefiting from synergies and links with other projects will be an integral part of the project.

As an OECD-DAC project, the case studies take a provider perspective. However, they are grounded on, and take as their starting point, the underlying commitment made by development partners to honour country ownership of results and “further develop, support and use country-level results frameworks; progressively adapt results frameworks to reflect the targets and indicators of the SDGs; and make data on results publicly available” (GPEDC, 2016[5]).

While the project takes a deep dive into alignment, measurement and use challenges surrounding individual SDG targets and indicators, we acknowledge the important notion that the SDG goals, targets and indicators are inter-related and will take into account the implications this may have on results measurement. Moreover, while the project focuses on results frameworks and indicators, the need to align to national frameworks from the planning and programming phase is fully recognised. In particular, the fact that a provider’s corporate policies can have a significant influence on the extent to which their results frameworks are able to align with country-led results frameworks will be considered.

Finally, the project takes an “action research” approach. Action research involves actively participating in a change situation, and promoting learning, while simultaneously conducting research.5 The goal is that the fieldwork (and subsequent reporting) for each of the case studies will be useful to providers and partners and generate dialogue in partner countries. In addition, the case study approach serves as a pilot, and parts of it can potentially be replicated in other contexts.

Case study topics

Each of the three case studies examines one specific SDG target and related indicator(s) (within the broader context of the overall goal) and how it is applied in two partner countries. In addition, one DAC donor focal point has been identified for each partner country. The three SDG targets/indicators are as follows (hereafter referred to as “case study SDGs”):

Target 4.1: By 2030, ensure that all girls and boys complete free, equitable and quality primary and secondary education leading to relevant and effective learning outcomes.

• Indicator 4.1.1: Proportion of children and young people: (a) in grades 2/3; (b) at the end of primary; and (c) at the end of lower secondary achieving at least a minimum proficiency level in (i) reading and (ii) mathematics, by sex.

Target 6.2. By 2030, achieve access to adequate and equitable sanitation and hygiene for all and end open defecation, paying special attention to the needs of women and girls and those in vulnerable situations.

• Indicator 6.2.1: Proportion of population using safely managed sanitation services, including a handwashing facility with soap and water.

Target 7.1: By 2030, ensure universal access to affordable, reliable and modern energy services.

• Indicator 7.1.1: Proportion of population with access to electricity.

Table A.1. Case study topics and structure

	Case study 1	Case study 2	Case study 3	Donor agency focal point
	SDG 4.1.1	SDG 6.2.1	SDG 7.1.1
Partner country fieldwork	Ethiopia		Ethiopia	Finnish Ministry of Foreign Affairs
		Kenya	Kenya	European Commission
	Myanmar	Myanmar		Australian Department of Foreign Affairs and Trade

Project methodology

The methodology consists of four main elements: 1) case study planning and inception; 2) desk-based review; 3) quantitative analysis of donor/partner data for case study SDGs; 4) in-country fieldwork. Each are discussed in more detail below.

1. Case study planning and inception

This element allows time for identification of case study topics, donor focal points and other partners; development of a full work plan, including an analytical framework and research questions to guide data‑collection costings and time frames; and consultations with key stakeholders and partners to confirm milestones, roles and responsibilities.

2. Desk-based review

For each case study, the OECD Results Team undertakes background research and analysis, which informs fieldwork and the overall evidence base. An analytical framework based on case study objectives guides the review. Key documents for review (with a focus on the case study SDGs) include:

• GPEDC monitoring data for each partner country – Indicators 1a and 1b6 (2016 data and 2018 data available by end 2018)

• partner country national and sectoral plans and reports

• partner country results frameworks and reporting (national, sectoral, subnational)

• partner country UN voluntary national reviews

• provider policies, plans, data strategies, results frameworks and reporting at project, sectoral, country and corporate levels

• any documentation on data collection for case study SDGs in partner country and globally

• any documentation/information on donor co-ordination, donor/partner dialogue (relating to results data collection, management and use) at country level.

3. Quantitative analysis of donor/partner data for case study SDGs

The quantitative analysis addresses some of the technical aspects of the project, and is undertaken in parallel to the qualitative elements (desk-based and field research). The Results Team completes data analysis based on existing data for each of the three case study SDG indicators, aiming to present indicators and available data for each indicator along the results chain as follows:

• global overview on progress toward indicator

• analysis of progress towards the indicator in case study partner countries (national and subnational if possible) (outcome, impact level)

• if possible, indicators used and available data on combined (and individual) DAC donor efforts in terms of results achieved against the indicator globally and in case study partner countries (output level – from provider standard indicator data that relates directly to the case study SDG)

• data on combined official development assistance effort toward case study SDG globally and in case study partner country (input level – mapped to the OECD Creditor Reporting System purpose codes).

At each level described above, assessment of methodological challenges and data gaps with regards to collating and presenting these data is documented. Overall, the analysis feeds into the wider case studies, informs the fieldwork and serves as a pilot to confirm the methodology (and its limitations), with the possibility to repeat the analyses for other targets/indicators.

4. In-country fieldwork

To explore both technical and organisational/political elements of the research objectives from a qualitative perspective, the Results Team (in partnership with the donor focal point agency) plans and undertakes in-country fieldwork. In each country, the fieldwork is planned on the basis of the analytical framework and consists of meetings with a range of key stakeholders, including partner country government officials (ministry for planning, national statistics offices and relevant line ministries), donor focal point, bilateral and multilateral donors, civil society organisations, regional agencies, think tanks/data initiatives. The fieldwork includes two styles of engagement over the week:

1. Meetings with distinct stakeholders (e.g. donors, sector co-ordination groups, officials from line ministries, UN agencies) as individuals or small groups towards the beginning of the fieldwork enables a frank exchange of views.

2. A multi-stakeholder workshop at the end of the fieldwork enables an open exchange of views on shared challenges and opportunities, and possible actions going forward (one for each case study).

Key facets to explore for the case study SDGs are set out in the analytic framework.

The donor focal point provides important background information for the case study, but also, as much as possible, helps facilitate the in-country field work (i.e. assist with identifying key contacts and providing logistical support where possible, e.g. arranging meeting rooms).

In addition, where possible, a member of the Results Team at headquarters of the donor focal point agency joins the fieldwork, as this strengthens the learning element of the work for the donor focal point agency and assists in drawing out the dynamics and challenges of meeting headquarters’ reporting requirements at partner-country level.

A fieldwork guidance note (separate document) for donor focal points provides more detail on the fieldwork methodology, roles and responsibilities.

Outputs, milestones and timeframes

The DCD Results Team will produce interim and final reporting products. An informal project steering group peer reviews all project outputs. It is envisaged that learning from the case studies will be relevant and useful to provider staff both in headquarters and in country offices, as well as to partners and other stakeholders. A detailed dissemination plan will ensure findings from the project are communicated to the relevant audiences.

Key outputs:

• preliminary discussion paper and workshop inputs for October 2018 Results Community workshop (October 2018)

• three technical case study reports (one for each case study SDG) – to be posted on the OECD‑DAC Results Community website (January 2019)

• report/spreadsheet resource for each case study SDG based on quantitative analysis (April 2019)

• final policy or working paper (and related communication products) ahead of spring 2019 Results Community workshop/High-Level Political Forum 2019 (April 2019).

The tables presented in this annex are based on detailed “indicator inventory” spreadsheets which have been compiled for each case study SDG (tracking indicators and any data against them). The spreadsheets are based on extensive web-based research and consultation with development co‑operation providers and partners, as well as verification in the field. The objective was to identify SDG-aligned or SDG-like indicators used by partners and/or providers, and any data against these. A detailed set of criteria or rules were used for identification of indicators which were considered SDG‑aligned or SDG-like. The spreadsheets are considered a working document, but there is potential to make the inventories publicly available. The Secretariat is therefore grateful for validation of and feedback on the data presented here. Links are provided to the source of the indicator in the left-hand column.

At corporate level, all DAC member and multilateral development bank providers which are known to have adopted standard indicator sets,7 and have indicators in the relevant sectors, are included. At country level, the following providers are included:

• The United Nations via United Nations Development Assistance Framework (UNDAF) indicators; UN agencies were included in aggregate rather than each individual UN agency being considered separately – except for Myanmar, where there is no current UNDAF. Instead United Nations Children’s Fund (UNICEF) indicators and results were included. UNICEF is an active provider in the education sector.

• The World Bank Group and relevant regional multilateral development finance institution (i.e. African Development Bank or Asian Development Bank as applicable).

• The case study donor focal point.

• The top three DAC providers of aggregate bilateral official development assistance (ODA) disbursements to the partner country in that sector in 2016.

• The top three DAC providers of aggregate bilateral ODA disbursements to the partner country in that sub-sector in 2016, if different from above (e.g. for Indicator 4.1.1, the top three providers of bilateral ODA in the primary and secondary education subsector in Ethiopia in 2016).

• Additional DAC bilateral providers are included for analysis even if they are not one of the top three providers of bilateral ODA to the partner country in that sector/sub-sector if the provider has prioritised that sector in their development co-operation strategy for that partner country. For example, although Norway is not one of the top three providers of bilateral education ODA in Ethiopia, it is included for analysis, because Norway has prioritised the education sector in its development co-operation strategy for Ethiopia. This approach allows for inclusion of smaller providers who are relatively active in a particular sector and partner country, despite their lower ODA outflows.

The tables presented in this annex are based on detailed “indicator inventory” spreadsheets which have been compiled for each case study SDG (tracking indicators and any data against them). The spreadsheets are based on extensive web-based research and consultation with development co‑operation providers and partners, as well as verification in the field. The objective was to identify SDG-aligned or SDG-like indicators used by partners and/or providers, and any data against these. A detailed set of criteria or rules were used for identification of indicators which were considered SDG‑aligned or SDG-like. The spreadsheets are considered a working document, but there is potential to make the inventories publicly available. The Secretariat is therefore grateful for validation of and feedback on the information presented here. Links are provided to the source of the indicator in the left‑hand column.

At corporate level, all Development Assistance Committee (DAC) member and multilateral development bank providers which are known to have adopted standard indicator sets,8 and have indicators in the relevant sectors, are included. At country level, the following providers are included:

• the United Nations via United Nations Development Assistance Framework (UNDAF) indicators; UN agencies were included in aggregate rather than each individual UN agency being considered separately

• the World Bank Group and relevant regional multilateral development finance institution (i.e. African Development Bank or Asian Development Bank as applicable)

• the case study donor focal point

• the top three DAC providers of aggregate bilateral official development assistance (ODA) disbursements to the partner country in that sector in 2016

• the top three DAC providers of aggregate bilateral ODA disbursements to the partner country in that sub-sector in 2016, if different from above (e.g. for Indicator 4.1.1, the top three providers of bilateral ODA in the primary and secondary education subsector in Ethiopia in 2016).

Additional DAC bilateral providers are included for analysis even if they are not one of the top three providers of bilateral ODA to the partner country in that sector/sub-sector if the provider has prioritised that sector in its development co-operation strategy for that partner country. For example, although Norway is not one of the top three providers of bilateral education ODA in Ethiopia, it is included for analysis, because Norway has prioritised the education sector in their development co-operation strategy for Ethiopia. This approach allows for inclusion of smaller providers who are relatively active in a particular sector and partner country, despite their lower ODA outflows.

References

[1] Engberg-Pedersen, P. (2018), The 2030 Agenda and Development Co-operation Results, OECD Development Policy Papers, No. 9, OECD Publishing, Paris, https://www.oecd-ilibrary.org/development/the-2030-agenda-and-development-co-operation-results_2f391534-en (accessed on 2 October 2018).

[5] GPEDC (2016), “Nairobi Outcome Document”, Global Partnership for Effective Development Co-operation, http://effectivecooperation.org/wp-content/uploads/2016/12/OutcomeDocumentEnglish.pdf (accessed on 8 February 2018).

[3] OECD (2018), A Results Agenda for the 2030 Agenda: New Approaches for Changing Contexts – Key Messages from the Workshop, OECD, Paris, http://www.oecd.org/dac/results-development/docs/results-key-messages-workshop-apr-2018.pdf (accessed on 2 October 2018).

[2] OECD (2018), Development Co-operation Results for the 2030 Agenda: A Guide for Applying the Sustainable Development Goals, Targets and Indicators in Results Frameworks, OECD, Paris, http://www.oecd.org/dac/results-development/docs/guide-dev-cop-2030-agenda-results.pdf (accessed on 2 October 2018).

[4] Zwart, R. and J. Egan (2017), “Making better use of results data in development co-operation”, in Development Co-operation Report 2017: Data for Development, OECD Publishing, Paris, https://doi.org/10.1787/dcr-2017-10-en (accessed on 1 February 2018).

Notes

← 1.  A menu of 42 targets was developed, based on those which were: Tier I; relevant to development co‑operation; and based on outcomes. See: https://doi.org/10.1787/24140929.

← 2.  See, for example: www.oecd.org/dac/results-development/docs/results-key-messages-workshop-apr-2018.pdf (session 3), www.oecd.org/dac/results-development/docs/results-data-results-workshop-apr-18.pdf, http://dx.doi.org/10.1787/dcr-2017-6-en and http://dx.doi.org/10.1787/544032a1-en (section 3.4).

← 3.  This project follows OECD-DAC terminology, which can differ from terminology used at country level. “Providers’’ refers to bilateral and multilateral donors. “Partners” refers to partner country governments, and “other stakeholders” includes implementing partners, civil society organisations, think tanks, beneficiaries, regional bodies and others who have a stake in the development co-operation process.

← 4.  It is important to note that this project is concerned with harmonisation of indicators and related measurement, rather than harmonisation of development co-operation more broadly. The project looks specifically at how increased co-ordination among providers and with partners might enhance harmonisation of results indicators.

← 5.  See, for example: https://www.researchgate.net/publication/282199978_Action_research.

← 6.  http://effectivecooperation.org/wp-content/uploads/2015/05/GPEDC-Monitoring-Framework-10-Indicators.pdf

← 7.  Defined as a standardised set of indicators used by development co-operation providers to monitor results. They are typically used for three tiers of results frameworks: 1) development results; 2) development co-operation results; 3) performance information. Standard indicators at Tier II typically aggregate project-level results in a way which enables communication of results achieved across multiple projects, countries and regions (Engberg-Pedersen, 2018[1]).

← 8.  Defined as a standardised set of indicators used by development co-operation providers to monitor results. They are typically used for three tiers of results frameworks: 1) development results; 2) development co-operation results; 3) performance information. Standard indicators at Tier II typically aggregate project-level results in a way which enables communication of results achieved across multiple projects, countries and regions (Engberg-Pedersen, 2018[1]).