Block 3 examines the importance of robust monitoring and evaluation (M&E) frameworks to inform homelessness policy and practice. It outlines how systematic M&E enables evidence‑based decision-making, improves accountability, and facilitates more efficient use of public resources. Given that systematic M&E in the area of homelessness is unfortunately rare, this block explores ways to improve M&E in the area of homelessness and highlights the benefits of early planning, baseline research, and a focus on both individual and systemic outcomes.
OECD Toolkit to Combat Homelessness
Abstract
Robust and credible evidence generated by monitoring and evaluation (M&E) processes enables evidence‑based decision making, and helps governments show that the use of public resources and the decisions they make translate into desired outcomes. Monitoring and evaluation are distinct: policy monitoring is a measure of policy implementation to ensure that planned activities are on track and that any changes in the underlying conditions are accounted for; it provides evidence to measure performance, identify implementation challenges, and bolster accountability and efficiency. Meanwhile, evaluations assess the actual or potential impacts of policy interventions, based on their relevance, coherence, efficiency, effectiveness, impact, sustainability, and process (OECD, 2019[1]; OECD, 2023[2]). By fostering knowledge, contributing to strategic decision-making, and increasing accountability (OECD, 2019[1]), M&E supports governments in designing policies with interventions that are proven to be effective, and in adapting current measures where needed. Data and timelines for monitoring and evaluations exercises are distinct. OECD (2023[2]) emphasises the need to plan for evaluation early on in the process, in terms of identifying the measures to be evaluated, and the data and evidence to be collected before implementation and during the timeframe of the intervention. Figure 3.1 summarises key evaluation criteria (OECD, 2023[2]), underscoring the multi-dimensional nature of issues to be considered.
Note: OECD elaboration.
Source: (OECD, 2023[2]), Boosting evidence‑based policy making for economic development policies in Italy, www.oecd.org/content/dam/oecd/en/topics/policy-sub-issues/economic-surveys/italy-boosting-evidence-based-policy-making-for-economic-development-report-2023.pdf
While M&E processes are often organised at the programme level, over half of OECD countries have legal frameworks that regulate and standardise public policy evaluations across projects (OECD, 2020[3]). However, systematic M&E in the area of homelessness is unfortunately rare (Baptista and Marlier, 2019[4]). Few countries have established M&E frameworks to assess homelessness policies, and earmarked funding for M&E is generally insufficient or non-existent. When M&E exercises are undertaken, baseline research is often missing, and data are not regularly collected at more than one stage of the intervention. Evaluation also tends to be focused on individual change, rather than systemic outcomes or the context behind interventions which may impact programme success (Pauly, Wallace and Perkin, 2015[5]). Evaluations of Housing First programmes are a notable exception; such evaluations have been instrumental in demonstrating the effectiveness of housing-led interventions (compared to other types of interventions) and in helping to correct course where needed (Baptista and Marlier, 2019[4]).
The recommendations relating to M&E have been informed by the OECD Recommendation on Public Policy Evaluation, which offers valuable insights on improving evaluations (Box 3.1) (OECD, 2022[7]) and work undertaken by the OECD to strengthen evidence‑based policy making for economic development (OECD, 2023[2]).
The OECD Recommendation on Public Policy Evaluation, adopted in 2022, aims to establish robust institutions and practices that promote the use of public policy evaluations. The Recommendation refers to public policy evaluation as the structured and evidence‑based assessment of the design, implementation or results of a planned, ongoing or completed public intervention.
The Recommendation calls on adherents to:
Institutionalise evaluation from a whole of government perspective, by conducting and using public policy evaluations systematically, and by fostering demand for evaluations inside and outside the executive.
Promote the quality of evaluations, by planning and managing evaluations so that they are fit-for-purpose, as well as by establishing standards and mechanisms, and developing skills and capacities.
Conduct evaluations that impact decision-making, by embedding them into decision-making processes, and by publicising and communicating their results.
Source: (OECD, 2022[7]), OECD Recommendation of the Council on Public Policy Evaluation, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0478.
There is currently no international standard – including agreed upon methodologies, data and/or indicators – for monitoring and evaluating homelessness policies and programmes. While generic M&E frameworks can be used, further efforts are required to provide international guidance on adapting M&E frameworks to the homelessness sector. For instance, standardised indicators to monitor and evaluate the effectiveness of homelessness policies across countries would help further isolate the aspects of homelessness policies that are most efficient. Future OECD work will aim to further develop this dimension.
M&E processes should guide homelessness interventions throughout their life cycle to ensure that they are working as intended and reaching their goals. From the outset, this means articulating the purpose and objective(s) of the policy intervention and developing a results framework with relevant indicators, which can be based on existing data collection methods (Block 2). This also includes determining priorities and evaluating existing interventions to prevent and address homelessness to adapt to ongoing challenges (Blocks 4, 5 and 6). Throughout the policy lifecycle, M&E processes can help consolidate learning on homelessness across countries and sectors, contributing to a stronger evidence base for the effectiveness of specific interventions in different contexts.
The following set of operational questions is intended to guide policy makers and practitioners in strengthening monitoring and evaluation of homelessness interventions:
When should M&E be implemented in the policy design stage? What are the first steps to develop a robust, results-based M&E framework?
How can M&E skills be developed in-house, and when should an external expert be used?
How to create incentives for policy evaluation?
How can M&E findings be used to better inform policy making?
When should M&E be implemented in the policy design stage? What are the first steps to develop a robust, results-based M&E framework?
Developing a comprehensive results framework at the outset of homelessness interventions
To better understand the impact of a homelessness intervention, M&E should be considered from the beginning. Ideally, a comprehensive results framework should be designed and set in place at the start of the intervention. The framework should explicitly articulate the logic of change behind the intervention of interest – that is, how and why the intervention is expected to achieve its objective(s), including causal relationships and underlying assumptions and risks (OECD, 2023[8]; OECD, 2023[2]). For each level of the results chain, a brief set of indicators should be developed, and where possible, baseline values assigned to each of them. The results framework is an explicit articulation of the different levels of results expected from a particular policy or programme, which could take the form of a graphic display, matrix, and/or summary (World Bank, 2012[9]).
Results frameworks help policy makers identify trends in the outcomes of interest. This baseline exercise can be particularly useful to monitor and assess how well a policy or programme is performing against planned targets, and therefore facilitate an evidence‑based approach to M&E.
For instance, the National Strategy to Combat Homelessness in Spain (2023‑30) integrated an M&E framework from the outset of the policy implementation process:
First, a results framework identifies overarching objectives, goals, measurable indicators and expected results. The framework makes use of secondary data sources to assign a baseline value to each indicator, providing a starting point for future comparisons.
To ensure M&E continues throughout the policy cycle, the Spanish strategy includes a monitoring system, which includes biennial progress reports, a monitoring dashboard based on the established indicators, and a biennial programming of measures to adapt the guidelines of the strategy based on current progress.
The strategy also foresees a mid-term evaluation in 2028 and a final evaluation in 2030.
Establishing an M&E group for effective oversight of homelessness policies
Monitoring and evaluating homelessness interventions requires regular and consistent oversight and adaption. Establishing working groups that bring together key stakeholders, including different levels of government, housing and service providers, NGOs, and people with lived experience, can help to ensure that interventions are being rigorously monitored and evaluated through a multidisciplinary, collaborative, unbiased approach. The inclusion of people who work directly with people experiencing homelessness, along with people with lived experience who use the service or programme being monitored or evaluation, helps ensure that evaluations remain grounded in and responsive to direct feedback from the target population.
In Portugal, the National Strategy for the Integration of Homelessness People: Prevention, Intervention and Monitoring, 2017‑23, included the creation of the Strategy Implementation, Monitoring and Evaluation Group (GIMAE), which is composed of a range of public (e.g. ministries, public institutes, local authorities) and private (e.g. NGOs, research institutes) stakeholders (Baptista, 2018[10]). The GIMAE meets bimonthly to assess the progress of the strategy and is responsible for preparing a biennial Action Plan and annual evaluation reports. Each annual evaluation reports on progress towards the strategic objectives, constraints to achieving the objectives, and suggestions for improvement. The biennial Action Plans include consultations with partner entities and people with lived experience of homelessness and provides updated goals and strategic objectives. The creation of the GIMAE signalled a process of continued M&E from the outset of the Strategy; the group continues to meet regularly.
Identifying from the start the data and evidence to be collected ex ante and ex post and the resources needed to collect and analyse the data
Identifying the data and evidence to be collected before implementation and during the timeframe of the intervention is a critical step (OECD, 2023[2]). Some data may exist already within public administration but require linking – a process that takes time and resources. Other data may not be readily available within the public administration and will need to be acquired. For instance, recent OECD work with Italy found that, based on three counterfactual evaluations using micro-level data over the past three years, around 60% of time and resources were spent in linking existing databases within the same public administration (OECD, 2023[2]). This is especially salient with respect to homelessness, given the cross-cutting nature of the issue and relevant policy areas. There are a wide range of methods that can be considered.
How can M&E skills be developed in-house, and when should an external expert be used?
Training sessions can help build M&E skills in-house
There are numerous examples of M&E training opportunities for civil servants:
In Ireland, the Irish Government Economic and Evaluation Service (IGEES) supports the development and enhancement of M&E practices within the government through capacity building. IGEES staff, who come from diverse backgrounds including economics and statistics, are integrated across various levels of government and are specifically trained in M&E (OECD, 2020[3]).
In the Latin American region, the Inter-American Development Bank (IDB) offers an online course on skills to design an M&E framework, to ensure the public programmes they manage generate greater public value (IDB, 2024[11]). The course targets professionals from Latin America who have responsibilities in the design, implementation and evaluation of policies and programmes fostering social development. This includes staff from public institutions, non-governmental organisations, academic centres, and international organisations. Similarly, the Latin American and Caribbean Institute for Economic and Social Planning (ILPES) delivers an online course that aims to teach participants how to design and develop performance indicators as a tool for monitoring and evaluating public programmes (ECLAC, 2024[12]). The course targets public officials from central and sub-national governments and more broadly, people involved in any stage of policy making.
Evaluations of policy interventions can be conducted internally, commissioned to independent external actors, or a combination of both
Similarly, examples of outsourcing M&E are common:
In Korea, the framework act on government performance evaluation recommends that all government agencies design a yearly internal evaluation plan to identify the main policies they will evaluate in a given year. Once an evaluation is conducted, this is transferred to the Government Performance Evaluation Committee (GPEC), the actor in charge of overseeing all government’s performance and evaluation systems (OECD, 2020[3]).
In Portugal, the main policy programme aiming to simplify processes for businesses and citizens (Simplex) is evaluated trough joint internal and external peer reviews. On the internal level, project managers have the mandate to systematically report on the progress of the project. Reporting is made publicly available and incorporates insights from key stakeholders and partners, enabling citizens to regularly consult the programme’s progress. Further, external evaluators such as academics and officials from the European Commission participate in the programme evaluation (OECD, 2020[3]).
How to create incentives for policy evaluation?
Provide financial resources to local initiatives to participate in national programmes to build up the evidence base
In the United Kingdom, the Test + Learn and the Systems-Wide Evaluation programme, launched in 2023 with a budget of GBP 15 million (USD 19 million), will work with local governments and people with lived experience to conduct rigorous evaluations and trials of programmes that aim to end rough sleeping and reduce homelessness. Expected to run until January 2027, the programme will evaluate eight interventions, including, among other things, the provision of employment specialists to facilitate immediate access to paid employment for people experiencing homelessness and people sleeping rough with high support needs; rough sleeping outreach with a health specialist to assess the impacts of people with more acute health needs; the provision of financial support to people with a history of rough sleeping to help them cover costs relevant to exiting homelessness; and time‑limited accommodation and immigration advice for people with limited access to mainstream housing and support services due to their immigration status. Most evaluations will be conducted through randomised control trials. The programme was commissioned by the Department of Levelling Up and will be implemented by the Centre for Homelessness Impact.
Including legal obligations to conduct policy evaluation and assigning resources to collect data through surveys
Not explicitly part of the homelessness field, Italy’s Start-up Act mandates the annual monitoring of measures supporting start-ups and innovative SMEs, with a report submitted to Parliament by 1 September each year. The report includes detailed data on the uptake of the measures, geographical distribution, and business performance, ensuring ongoing evaluation of their effectiveness (OECD, 2023[2]).
How can M&E findings be used to inform policy making?
Ensuring that policy makers directly engage and respond to evaluation results through pre‑determined adaptation processes
M&E processes provide little benefit if policy makers do not engage with the results. From the beginning of the policy cycle, M&E processes should consider how policy makers will engage with the results. This may include requiring policy makers to directly respond to the results through mid-term reports, which include pre‑designated action areas.
In Tokyo, Japan, the Implementation Plan for Homeless Self-Reliance Support, which is currently in its fourth phase, regularly analyses and evaluates interventions to support people experiencing homelessness. Based on the data collected on the number of users, their demographic information, and the success rate of programmes, the Tokyo Metropolitan Government adapts its programmes to better suit the needs of people experiencing homelessness. For example, a recent evaluation found that, while employment counselling and placement services were helping to connect people experiencing homelessness to job opportunities, many individuals did not remain employed for long. Therefore, the programme was adapted to provide support for longer periods of time to help people make a smoother and longer lasting transition into employment. To further inform evaluations, the Tokyo Metropolitan Government holds review meetings with academic researchers, NGOs, facility managers and government departments.
Outside the homelessness sector, the Prospera1 Conditional Cash Transfer programme in Mexico demonstrates how programme implementation has been adapted to respond to evaluation findings. The programme’s ultimate objective was to foster the development of human capital through three separate components: education, health, and nutrition. The programme had a strong evidence‑based design, and a wide range of rigorous impact evaluations were conducted over the years. These evaluations not only provided evidence of impact on the outcomes of interest, but revealed design and implementation challenges that had the potential to hinder the programme’s results. As a result, the programme underwent several modifications, including the redesign of the health and nutrition components to incorporate nutritional supplements for pregnant women, breastfeeding women, and children under 5 years of age, as well as a behaviour change communication component.
Monitoring and evaluation are distinct processes, enabling evidence‑based decision-making, accountability and the improvement of policies by producing evidence on what works. Despite its significance, systematic M&E in the homelessness sector remains rare and rigorous evaluations (with a control group even rarer; most evaluations lack baseline data and a focus on systemic outcomes. Exceptions, such as Housing First programmes, demonstrate the benefits of evaluations showcase the strength of having robust evidence to adapt and scale up programmes. The OECD’s Recommendation on Public Policy Evaluation and recent work by the OECD (OECD, 2023[2]) encourages governments to institutionalise and standardise M&E and provides concrete guidance for policy evaluation. Some countries have started devising integrated M&E frameworks which can improve oversight and guide long-term homelessness strategies. Skills development to carry out M&E, data collection and data linking, and funding for M&E are highlighted as critical challenges.
Building on these operational issues, the following recommendations can help policy makers and practitioners embed systematic monitoring and policy evaluation into policy making:
Monitor and rigorously evaluate programmes by identifying policy objectives, indicators, and baseline context at the beginning of the policy design stage and carry out evaluations with stakeholders at pre‑determined stages of the policy process. This can be supported by putting in place at the outset a results framework and establishing an M&E group for effective oversight.
Identify from the start the data and evidence to be collected ex ante and ex post and the resources needed to collect and analyse the data.
Build in-house monitoring and evaluation skills, including the ability collect and analyse qualitative and quantitative data, create/manage budgets and evaluate outcomes, by conducting regular training and collaborating with academia and the private sector.
Create incentives for policy evaluation and allocate dedicated resources to set up and sustain robust monitoring and evaluation of policies and programmes throughout the policy life cycle and determine whether the evaluation should be internal, external, or hybrid.
Incorporate evaluation results in policy-making processes and adapt measures with this evidence where needed to improve the impact of interventions.
References
[9] Baptista, I. (2018), “The New Portuguese Homelessness Strategy: Recent Developments”, European Journal of Homelessness, Vol. 12/1, https://www.feantsaresearch.org/download/12-1_c1_stratreview_baptista_v024691506871026298164.pdf.
[4] Baptista, I. and E. Marlier (2019), Fighting homelessness and housing exclusion in Europe: A study of national policies, https://doi.org/10.2767/624509.
[11] ECLAC (2024), Indicadores de desempeño de la gestión pública, https://www.cepal.org/es/cursos/indicadores-desempeno-la-gestion-publica-2.
[10] IDB (2024), Monitoreo y evaluación de los programas públicos, https://cursos.iadb.org/es/indes/monitoreo-y-evaluaci-n-de-los-programas-p-blicos-curso-bajo-demanda.
[2] OECD (2023), Boosting evidence-based policy making for economic development policies in Italy, OECD, Paris, https://www.oecd.org/content/dam/oecd/en/topics/policy-sub-issues/economic-surveys/italy-boosting-evidence-based-policy-making-for-economic-development-report-2023.pdf.
[7] OECD (2023), Impact by Design: Effective Results Frameworks for Sustainable Development, OECD, Paris, https://capacity4dev.europa.eu/library/oecd-toolkit-impact-design-effective-results-frameworks-sustainable-development (accessed on 10 July 2023).
[6] OECD (2022), Recommendation of the Council on Public Policy Evaluation, OECD, Paris, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0478.
[3] OECD (2020), Improving Governance with Policy Evaluation: Lessons From Country Experiences, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/89b1577d-en.
[1] OECD (2019), Open Government in Argentina, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/1988ccef-en.
[5] Pauly, B., B. Wallace and K. Perkin (2015), “Approaches to evaluation of homelessness interventions”, Housing, Care and Support, Vol. 17/4, pp. 177-187, https://doi.org/10.1108/HCS-07-2014-0017.
[8] World Bank (2012), Designing a Results Framework for Achieving Results: a How-To Guide, World Bank, Washington D.C., https://documents1.worldbank.org/curated/en/331541563854787772/pdf/Designing-a-Results-Framework-for-Achieving-Results-A-How-to-Guide.pdf (accessed on 10 July 2023).
← 1. The programme was initially created as Progresa in 1997, and renamed Oportunidades Human Development Program in 2002, and Prospera Program of Social Inclusion in 2014. The programme finalised in 2018.