Strategic Joint Evaluation of the Collective International Development and Humanitarian Assistance Response to the COVID‑19 Pandemic

Annex B. Methodology

Scope of the evaluation

The evaluation assesses the collective international development and humanitarian assistance response in official development assistance (ODA) eligible countries, covering the period 1 January 2020 to 31 December 2022, the main phase of the pandemic response. The term “collective response” is used throughout this report to refer to the entirety of actions undertaken by development and humanitarian actors including bilateral development agencies (hereafter referred to as “bilateral providers”) – both DAC members and others – United Nations agencies, multilateral institutions and non-governmental actors, in responding to the COVID-19 pandemic in partner countries.

Its scope includes all development and humanitarian co‑operation including ODA (i.e. grants and concessional loans) and other official flows (i.e. non-concessional loans and financing) that were provided, including bilateral and triangular co‑operation, in-kind support and other concessional finance, to create a full picture of the response. There are three broad categories of assistance considered within the scope of this evaluation: 1) all existing assistance prior to the pandemic; 2) new COVID-19-specific funding during the pandemic; and 3) all assistance during the pandemic including non-COVID-19-related.

Trends related to development assistance prior to and after the COVID-19 pandemic are analysed using official development finance (ODF) data (i.e. total funding including ODA and other official flows), from 2016-2023, along with data on private flows available in the OECD’s Creditor Reporting System (CRS), as ODF gives a more complete picture, particularly at country level. All trends in ODA from 2020-2022 are considered to be related to or influenced by the COVID-19 pandemic and are therefore relevant to this study. ODA specifically tagged as COVID-19-related in the CRS database is used in case studies and in sector flows to highlight differences between COVID-19 funds and overall assistance, thus providing a better understanding of the international response. Case studies and DAC peer reviews were used to complement the analysis and explore factors underpinning provider decisions in how they responded.

This was not an evaluation of the performance of partner countries’ own responses to the pandemic. The case studies provided by ODA partner countries have been used in this report only to gain insight into the dynamics between national governments and international development and humanitarian providers.

The term “partner countries” is used throughout the report to refer to countries and territories receiving assistance (based on those that were ODA-eligible in 2020-2022). The term “provider” is used to denote the country or entity that supplied funding or other forms of assistance.

The analysis considers all support for equitable access to COVID-19 vaccines and vaccination rollouts including contributions to the Access to COVID-19 Tools Accelerator (ACT-A), manufacturing and donations of COVID-19 vaccine doses, and support to address issues related to vaccine manufacturing and supply, delivery, health system capacities, communication, and combating mis- or disinformation.

Design of the evaluation and its approach

The evaluation was designed to engage the intended users – including humanitarian and development agencies, national governments and implementing partners – through the Evaluation Steering Committee and via the OECD’s Development Assistance Committee (DAC).

This evaluation was structured to complement existing evaluative work, drawing on the important work of the Independent Panel for Pandemic Preparedness and Response report, COVID-19: Make it the Last Pandemic, presented to the World Health Assembly in May 2021 (The Independent Panel, 2021[1]).

Other significant international evaluations of relevance include those undertaken by the:

• Inter-Agency Evaluation of the COVID-19 Humanitarian Response (IAHE, 2022[2]),

• the System-Wide Evaluation of the UN Development System Response to COVID-19 (UNSDG, 2022[3]),

• The World Bank’s Early Support to Addressing COVID-19 Health and Social Response (An Early-Stage Evaluation) (World Bank, 2022[4])

• International Monetary Fund (IMF) responses (IMF/IEO, 2023[5])

In addition, MOPAN’s Assessment of the Multilateral System and COVID-19 (MOPAN, 2022[6]) and DAC peer reviews (OECD, 2025) provided valuable inputs.

Studies focused on COVID-19 vaccines and equitable access that informed this evaluation include:

• an evaluation of COVAX by Gavi (ITAD, 2023[7])

• WHO’s ACT-Accelerator Strategic Review (WHO, 2022[8])

• the Evaluation of CEPI’s COVID-19 Vaccine Development Agreements (CEPI, 2022[9])

• the Evaluation of Gavi’s Response to COVID-19 (Euro Health Group, 2022[10])

• real-time assessments of UNICEF’s support to the COVID-19 vaccine roll out and immunisation programme strengthening (UNICEF, 2021[11]).

The analysis looks at programmatic and policy responses using the evaluation criteria of relevance, coherence, efficiency and effectiveness, along with a set of related evaluation questions. The policy and programmatic intentions of the providers are ordered under these criteria, making it easier to address the findings at a higher level, rather than looking at individual policy and programmatic intentions in isolation.

Further detail on the overall evaluation approach, design, evaluation questions and sampling of country case studies are outlined in the inception report (Smith-Kouassi, 2023[12]).

The IOD PARC evaluation team refined the evaluation’s design to ensure it was feasible, taking into account the data available and the work that had already been undertaken. This included moving away from the use of indicators alongside the criteria and questions outlined in the original evaluation framework, and minor refinements to the evaluation questions. Planned case studies on Czechia, Mozambique, Nicaragua and Saudi Arabia were dropped or scaled back due to resource constraints.

The key questions used to guide the Module 5 final evaluation report are outlined below in Table ‎B.1.

Table ‎B.1. Evaluation questions

	Evaluation questions	Sub-questions
Descriptive	Q1. How did national governments, and development and humanitarian actors respond to the COVID-19 pandemic?	Q1.1. What were the identified needs and priorities of partner countries in addressing COVID-19?
		Q1.2. Who funded the international response to the COVID-19 pandemic, what was funded, and where were efforts focused?
		Q1.3. How and where did international development and humanitarian actors support access to vaccines and vaccination rollouts?
Relevance	Q2. To what extent did COVID-19 support meet partner country needs and priorities?	Q2.1. To what extent was funding and programming responsive to partner country needs and priorities, including those of the most vulnerable?
		Q2.2. To what extent were providers flexible and adaptive in responding to changing needs and priorities as the pandemic evolved?
Coherence	Q3. To what extent did responses align to ensure coherent approaches at global and country levels?	Q3.1. To what extent, and in what ways, was the collective response coherent at global level? At country level?
		Q3.2. To what extent were efforts focused on equitable access to vaccines, and vaccinations co‑ordinated and aligned?
Effectiveness	Q4. What are the early results of the collective response to COVID-19?	Q4.1. To what extent did DAC members deliver on joint commitments regarding support for COVID-19 responses and equitable access to vaccines in partner countries?
		Q4.2. To what extent, and in what ways, did development co-operation and humanitarian assistance contribute to alleviating the immediate public health crisis stemming from the COVID-19 pandemic?
		Q4.3. To what extent, and in what ways, did development co-operation and humanitarian assistance contribute to interventions alleviating the secondary social and economic effects of the crisis?
		Q4.4. To what extent, and in what ways, did vaccine-related support result in equitable access and greater coverage?
		Q4.5 What were the unintended effects of the development and humanitarian support provided for COVID-19 response efforts?
Efficiency	Q5. To what extent were funding and programming decisions and interventions timely and informed?	Q5.1. To what extent were providers successful in mobilising timely and flexible funding to respond to COVID-19?
		Q5.2. To what extent can the different dimensions of the development co-operation and humanitarian response be considered good value for money?
Forward looking	Q6. What good practices, innovations and lessons learned emerged from the collective response to COVID-19?	Q6.1. What good practices and innovations emerged that can inform ongoing or future responses?
		Q6.2. What are the key lessons learned and how can these inform future co‑ordination and crisis preparedness?

Figure ‎B.1. Evaluation modules

The evaluation design included a modular and case-based approach, including both partner and provider case studies. The design reflects the global scope of the pandemic, the extent of evaluative work already available, and the range of actors involved. Modules 1-4 gathered evidence to answer the six overarching evaluation questions (and sixteen sub-questions) outlined in Table ‎B.1 above.

The evaluation includes case studies of assistance received during 2020-2022 in Bangladesh, Burkina Faso, Cabo Verde, Cambodia, Georgia, Kenya, Lebanon, Nicaragua and the Large Ocean States/Small Island Developing States (SIDS) group. Partner country case studies document country-specific contextual information, detail the collective response to COVID-19 in partner countries and the COVID-19 responses of national governments. They examine alignment with national strategies and priorities, and coherence across development partners and with other sustainable development efforts.

These cases were complemented by in-depth analysis of provider experiences, including the international assistance of the People’s Republic of China (hereafter ‘China’), Germany, Mexico, the Netherlands, New Zealand, Saudi Arabia, South Africa, Spain, and the United States. Provider studies look at international co-operation and humanitarian assistance of from the provider government in 2020-2022. They included an analysis of data from the OECD CRS database and from publicly available reports and websites on assistance. Experience of other countries and providers were included through the Module 1 synthesis and the global analysis.

This evaluation report (Module 5) synthesises and triangulates evidence from each of the previous modules. The consolidated report considers the global-level data and draws on the previous modules through a documentary review of approximately 130 documents, along with 12 additional key informant interviews, to generate findings and draw conclusions about the collective response.

Management and conduct

The inception phase of the evaluation was carried out by Organisation for Economic Cooperation and Development (OECD) from November 2022 to the completion of the final inception report in June 2023, with data collection underway until October 2024. Module 1 was completed by the Nordic Consulting Group in early 2023; Module 2 was completed by the OECD in 2023; and Modules 3 and 4 were led by a range of teams made up of OECD staff, COVID-19 Coalition participant evaluation units and contracted consultants.

The OECD conducted the case studies of Bangladesh, Cabo Verde, Cambodia, Georgia and Nicaragua. Burkina Faso carried out its own country-led evaluation with support from the OECD and the Global Evaluation Initiative (GEI). The African Development Bank IDEV carried out the Kenya case. Evaluation Department at Global Affairs Canada carried out the Lebanon case. The EC DG INTPA commissioned the SIDS group case, and the study was completed by Particip/IRMA. The OECD and IOB Netherlands jointly carried out the case study of the Netherlands. The ISDB supported the Saudi Arabia case. The Mexico case was conducted by AMEXCID and the OECD. The countries’ own reports were used for analysis of the international assistance provided by China, Germany, New Zealand, South Africa, Spain and the United States.

Responsibility for completing the evaluation (Module 5) was passed on to an independent evaluation team from IOD PARC in March 2024. IOD PARC is an independent consulting company specialising in international development and performance assessment. IOD PARC’s key responsibilities included support to complete provider and partner case studies, supplementary data collection and analysis, quantitative analysis of CRS data, and the development of this Module 5 evaluation report. The report was finalised by the OECD, with input and validation from the evaluation Steering Group (Annex A).

Data collection and analysis

In addition to the document reviews and interviews conducted for the individual case studies, a further twenty interviews were conducted by IOD Parc and the OECD. An opinion survey, designed and administered by DEval and the OECD was circulated to more than 600 stakeholders. There was a limited response from stakeholders outside German development co-operation, but the analysis of German responses and qualitative data in the other survey responses was used to inform the analysis and findings.

Case studies included extensive document reviews, including available evaluation literature and interviews with key national and international stakeholders working in the country in 2020-22. Country visits and interviews were used for Bangladesh, Burkina Faso, Cabo Verde and Cambodia. The Kenya case study builds on a country evaluation by the Independent Development Evaluation (IDEV) of the African Development Bank (AfDB), which included extensive document review, quantitative data and a country visit. Resource constraints in Lebanon meant that the team was only able to carry out a handful of interviews in the country, relying primarily on secondary sources.

To analyse the data across the four modules, the evaluation team developed an expansive qualitative coding framework, which was based on the six evaluation questions and 16 sub-questions. They analysed each document the data using a coding and analysis software tool (MAXQDA). The analysis was iterative, with additional codes being identified as the analysis progressed (i.e. use of budget support, PPE etc.). The analysis used the lenses of gender, equity, human rights and inclusion, where possible. Evaluator judgement then discerned trends and themes from across the coded data to develop preliminary findings and the report narrative.

Data were triangulated through the cross-refencing of data sources, triangulation through team analysis and team “sense-making” sessions and validation of findings through the commenting process with the evaluation manager, quality assurance lead, and Steering Committee. The provider and partner case studies are used in this report to provide illustrative and descriptive examples supporting the key findings emerging from the aggregated analysis to triangulate findings and to inform lessons.

The evaluation team presented the preliminary findings in November 2024 to OECD colleagues and the Steering Committee for validation before drafting the main report. It was then quality assured ahead of submission and revised and finalised in April 2025, following feedback from OECD and Steering Committee colleagues.

Limitations

Key limitations identified and the ways these were managed, were as follows.

• Resource constraints – including numerous staffing challenges in the evaluation team – meant that only a limited number of interviews and country visits could be conducted, with much of the work relying on data collected by partners and presented in their evaluation reports. It is expected that this led to some skewing of the data, either positively or negatively.

• National actions were largely out of scope, though the policies and response programmes were the main drivers of overall pandemic outcomes.

• There was a significant amount of variation across individual institutions, countries and even time periods (from early 2020 to the end of the evaluation period in 2022, by which time many actors had shifted out of crisis mode) in terms of reporting on activities and strategies.

• Evaluations were available for a fraction of the strategies, programmes, projects and institutions involved in international co-operation during the crisis response. As a result, the pool is slightly skewed towards institutions that were more active during the crisis response and looked primarily at activities that were funded and implemented, with little analysis of opportunities that were missed or actions not taken due to the crisis or institutional failings. To address this imbalance, interviews were used to explore decision making and identify barriers to action.

• Many actors were involved in producing the case studies, which meant that there were differences in approaches and focus, and the voluntary nature of participation inevitably resulted in uneven coverage of some providers and agencies.

• The volume of information available means that the report did not fully capture the full wealth of information, insights and conclusions of evaluation reports carried out on the COVID-19 response globally. To address this limitation, the final report focuses on providing useful lessons that can inform responses to future crises.

• It should also be noted that there was some level of “COVID fatigue”, especially in 2022 when the bulk of the case study work took place. Many stakeholders wanted to move on from the pandemic, which they described as a difficult or even traumatic period. However, in 2024-2025, perhaps with the benefit of some distance, many participants reported a renewed interest in drawing lessons, as well as an increase in public debate and discussions of the response as countries sought to incorporate lessons into a changing development landscape. This limitation was managed by focusing on lessons applicable to future co-operation, relevant in the current context.

References

[9] CEPI (2022), Equitable Access Review of CEPI’s Covid-19 Vaccine Development Agreements, Center for Transformational Health Law, https://static.cepi.net/downloads/2023-12/EQUITABLE-ACCESS-REVIEW-OF-CEPIS-COVID-19-VACCINE-DEVELOPMENT-AGREEMENTS_Final_April-2022.pdf.

[10] Euro Health Group (2022), Evaluation of Gavi’s Initial Response to COVID-19 Final Report Volume I, https://www.gavi.org/sites/default/files/programmes-impact/our-impact/Evaluation-Gavi-COVID-19-Final-report.pdf.

[2] IAHE (2022), Inter-Agency Evaluation of the COVID-19 Humanitarian Response, Inter-Agency Humanitarian Evaluation, https://interagencystandingcommittee.org/sites/default/files/migrated/2023-03/Inter-Agency%20Humanitarian%20Evaluation%20COVID-19.%20Main%20Report.pdf.

[5] IMF/IEO (2023), The IMF’s Emergency Response to the COVID-19 Pandemic, Independent Evaluation Office of the International Monetary Fund, https://ieo.imf.org/en/Evaluations/Completed/2023-0313-imfs-emergency-response-to-the-covid-19-pandemic (accessed on 6 August 2025).

[7] ITAD (2023), COVAX Facility and AMC Formative Review and Baseline Study, https://www.gavi.org/sites/default/files/programmes-impact/our-impact/Final-Report_COVAX-Facility-and-COVAX-AMC-Formative-Review-and-Baseline-Study.pdf.

[6] MOPAN (2022), More Than the Sum of Its Parts?: The Multilateral Response to Covid-19, Lessons in Multilateral Effectiveness, MOPAN, https://www.mopan.org/en/our-work/performance-insights/the-multilateral-response-to-covid-19.html.

[12] Smith-Kouassi, J. (2023), Strategic Joint Evaluation of the Collective International Development and Humanitarian Assistance Response to the COVID-19 Pandemic: Final Inception Report, https://www.oecd.org/content/dam/oecd/en/toolkits/derec/evaluation-reports/derec/covid19coalition/Strategic%20Joint%20Evaluation%20Inception%20Report.pdf.

[1] The Independent Panel (2021), COVID-19: Make it the Last Pandemic, https://recommendations.theindependentpanel.org/main-report/ (accessed on 6 August 2025).

[11] UNICEF (2021), Real-Time Assessment of the UNICEF Response to COVID-19: Global synthesis report, United Nations Children’s Fund, https://evaluationreports.unicef.org/GetDocument?documentID=18214&fileID=42792.

[3] UNSDG (2022), System-Wide Evaluation of the UNDS Socio-economic Response to COVID-19: Final Report, United Nations Sustainable Development Group, https://unsdg.un.org/resources/system-wide-evaluation-unds-socio-economic-response-covid-19-final-report (accessed on 8 August 2025).

[8] WHO (2022), Accelerating COVID-19 Vaccine Deployment: Removing obstacles to increase coverage levels and protect those at high risk, World Health Organization, https://www.who.int/publications/m/item/accelerating-covid-19-vaccine-deployment (accessed on 8 August 2025).

[4] World Bank (2022), The World Bank’s Early Support to Addressing COVID-19 Health and Social Response - An Early-Stage Evaluation, Independent Evaluation Group, Washington; World Bank, https://ieg.worldbankgroup.org/sites/default/files/Data/Evaluation/files/Covid-19-health-and-social-response.pdf.