Share

Evaluation of development programmes

Evaluation Capacity Development Newsletter - April 2016

 

Evaluation capacity development banner

 

 

 

 

 

 

CONTENT

 

THEME: Work Conducted by Civil Society to Enhance Evaluation Capacity

Photo of ECD task team chair 

Stephen Porter, DFID

Editor's Message

I attended a conference last month at which there were presentations on realist, systems and behavioural experimentation (‘nudge’) approaches to evaluation. Although quite different in many ways, they all sought programme improvement through changes within agreed policy and programme objectives. When challenged on the need to evaluate outside of these objectives, the presenters recognised the importance of measuring unintended outcomes and highlighted the potential of these approaches to improve policy through radical incrementalism.

The articles contained in this newsletter, however, challenge evaluation commissioners, managers and practitioners to undertake evaluation practice that works beyond pre-defined objectives and unintended outcomes of existing interventions. Evaluation of the achievement of defined objectives is important, but does not often incorporate values of the poorest and most excluded that may view an intervention’s success differently.

The introductory and concluding articles in this newsletter highlight the importance of applying human rights-based values within evaluations to reinforce critical aspects of governance, whether to understand the system around the intervention or in helping to ask questions that challenge existing objectives. The three other articles in this newsletter highlight the importance of the capability to use data, civil society evaluating their own situation and challenging government through evaluations.

 

 

 Michael Quinn Patton photo for ECD newsletter # 3 April 2016

Reflections on the Systemic Interconnections that Weave Together Democracy, Human Rights, and Evaluation Capacity

By: Michael Quinn Patton, From a cabin in the Northwoods of Minnesota, USA

I recently encountered a major international funder who said they didn't fund Evaluation Capacity Development (ECD) because they are outcomes-driven and ECD is process-oriented. This is but one manifestation of Cartesian dualism leading to the compartmentalisation and separation of phenomena, emphasising their categorical distinctiveness rather than their interconnectedness. In the current outcomes mania, attention to processes (how one gets to outcomes) has somehow become of secondary importance, if attended to at all. Black box experimental designs epitomise this inattention to and neglect of process. But process, like context, matters. Outcomes depend on and flow from processes within some particular context.  That is one way of seeing interconnections, the traditional and dominant linear framing. 

 

But the influence of dualism runs deep and hinders the seeing, framing, understanding, and acting on interconnectedness. In graduate school I participated in a sociology of development seminar in which a major focus was on whether democracy is a means or an end. Recently, I've encountered discussions (more like arguments) concerning whether rights are a means or an end, and correspondingly, whether rights-based evaluation should be thought of primarily as process-oriented or outcomes-oriented. At stake in these arguments are intervention designs, best practice models, conceptual frameworks, evaluation questions, policy formulations, and funding for all of the preceding. 

 

Enter systems and complexity thinking

Systems thinking shifts the focus from categorical separations and siloed distinctions to interconnections and interrelationships. Let's do an exercise in connecting the dots. How are human rights, democracy, evaluation, and evaluation capacity interconnected and interdependent? This takes us in quite a different direction from asking: which is process and which is outcome, and what processes lead to what outcomes?

So let's connect the dots, just a little, perhaps something like what follows. 

 

A values commitment to human rights is a foundation of inclusive democracy. Informed, inclusive decision-making is critical to democracy. Informed decision-making requires credible, meaningful, and relevant information on which to base democratic deliberations and decisions.  Generating credible, meaningful and relevant information is the function of evaluation. Without evaluation capacity, including the capacity to think evaluatively, informed decision-making is not possible, which undercuts democracy and threatens the commitment to human rights. This web of interconnections cannot be reduced to distinct processes and outcomes. These are system, and systemic, interrelationships, each element of which must function for the whole to function.

 

These are also complex, nonlinear, and dynamic interconnections. Evaluation capacity is not some fixed state, some predetermined, operational outcome. Changes in technology, demography, cultural context, politics, the global economy, public health, climate change, gender relationships, conflicts and civil unrest, to name but a few of the more obvious factors, pose new challenges to human rights, democratic structures and decision-making processes, and meaningful, trustworthy, and useful evaluation. Evaluation capacity must evolve to deal with changing conditions and emergent evaluands beyond traditional projects and programmes. Evaluation must evolve to assess the effectiveness and system impacts of new evaluands like strategies, cross-sector interventions, integrated development initiatives, interconnected sustainability development goals, and global systems change.       

 

Evaluation is more than a compliance activity, a bureaucratic accountability imperative, and a paperwork mandate. Capacity to comply, feed the bureaucratic monster, and get paperwork done -- these do not inspire.  These are the shadow forces that undermine ECD by making evaluation itself a mere administrative or managerial task.  

 

But the vision of ECD as critical to healthy, inclusive, human-rights-affirming, and democratic dialogue, engagement, and decision-making -- that is a vision of ECD as both process and outcome, as both means and ends, and as part of the integral web of our highest ideals.

 

From this perspective, evaluation capacity is not a destination. It is a journey without a fixed or even known destination. And the fellow travellers on that journey, among many others, are human rights and democracy. All aboard?


sief logo for ECD newsletter of April 2016 issue # 3

Promoting Evidence Based Policy Making through ECD with Civil Society

By: Strategic Impact Evaluation Fund, Washington

Successful evaluations not only needs rigorous data and technical knowledge, but also buy-in from stakeholders across government and civil society. 

 

The Strategic Impact Evaluation Fund (SIEF), a partnership programme run by the World Bank and supported by the British government’s Department for International Development and the Children’s Investment Fund Foundation, has been striving to build evaluation capacity and promote evidence-based policy making, including training more than 1,000 people.  

 

SIEF recognises that support for evaluations and evidence needs to cut across society. Working with policy makers alone isn’t enough. They of course don’t operate in a vacuum. They listen to and rely on civil society organisations for information, programmes and support. Non-governmental organisations, whether local or international, are often involved in designing and running programmes. Journalists can raise issues or questions that relate to what the government wants to do or should through the use of evaluations. 

 

For SIEF all of these audiences outside governments are elements in a pipeline that feeds information to key decision makers and helps citizens have a voice. Discussing issues with and disseminating information to these groups is currently an important part of SIEF’s activities. In a programme launched earlier this year, SIEF partnered with the International Center for Journalists to create webinars for reporters in sub-Saharan Africa. These webinars will help journalists improve their knowledge of human development issues and understand the use of evidence for policy making. The programme fills a need that, until now, has often been ignored. Typical training for these journalists has focused on the nuts and bolts of reporting, ignoring what reporters may or (more often) don’t know about the issues they are covering. SIEF support aims to foster more informed reporting in key areas, specifically around evidence and why it matters. SIEF will also be partnering with the UN Foundation to work together to build broader understanding of human development and measurement issues among journalists and civil society groups. 

 

Success in evaluation is as much about the support that exists within the government and across society to use evidence as the capacity of evaluators. 


 

Theme: Work Conducted by Civil Society to Enhance Evaluation Capacity

 ‌photo for ECD newsletter number 3 of April 2016

Community-led Urban Mapping – Monitoring, Assessing Risk and Building Capacity in Nairobi’s Informal Settlements

By: Paolo Cravero, Independent consultant, Human Settlement Group, International Institute for Environment and Development, London

Nancy laughs easily and smiles more. But when she talks about her work as a food vendor and a civil society activist in the settlement of Mathare (Nairobi), her figure straightens up adding pride and gravitas to her otherwise jovial persona. “We do mapping of informal settlements,” she says. “This serves us to map out the services and social activities present in the communities. It helps us to come up with solutions that are ours. Community-led solutions.” 

 

Informal settlements are crucial in city economies and are home to millions of people worldwide. In Nairobi, for example, about two-thirds of the population lives in one of 175 informal settlements. But these places, and their needs, are very often ignored by governments and their official maps. As a result, people as well as communities officially do not exist. Maps can shed light on these communities and on their challenges providing a platform for action and advocacy. 

 

To overcome the direct and indirect issues emerging from not being officially recognised (e.g. lack of documents for people and land, basic rights and services for communities, etc.) Shack/Slum Dwellers International (SDI), a network of organisations of the urban poor active in 33 countries, has championed approaches of self-organisation to gather information on informal settlements. Providing data through ‘enumeration’—a process by which organised low-income communities living in cities count themselves, document the lands and houses they occupy and identify vacant land for future housing— the urban poor have the possibility to demonstrate their existence and enter into a political debate/negotiation with local authorities. 

 

In Kenya, Nancy and the other members of Muungano wa Wananavijiji, a federation of Kenyan slum-dwellers’ associations (affiliate to SDI), has set out numerous community-led mapping projects. Recently, with support from partners in the Urban Zoo project, aerial/balloon community-led mapping has been carried out. This aimed at creating the first maps to help ensure the safety of both vendors and sold food. “We saw where the vendors are,” says Julia Washera, a food vendor and Muungano member from Mathare. Where the drainage and hazards are… through the balloon mapping we saw recent photos of the area, which are not up to date in Google maps.”

 

In addition to the monitoring and risk assessment aspects of this project, there has also been a distinctly capacity building trait to it. After the data collection and physical mapping phases completed, the results and aerial photographs were shared and discussed with the community. Through this process of analysis the main concerns of residents emerged and were classified by importance.

 

 

Photo of street vendor Mathare for article in ECd newsletter - April 2016

 Street vendor in Mathare

 

As explained in details in two working papers published with the International Institute for Environment and Development (see here and here), the Mungaano project unveiled links between food vending, urban health, food security and environmental hazards. The challenges that the community highlighted include risks of contamination from: pests, overflowing sewers, livestock and unsafe water. It also emerged that poor hygiene and food handling practices are serious challenges.

 

The discussion and analysis of data enabled community members to identify capacity-building and communal efforts as short term solutions for this situation—activities such as training to improve food hygiene and communal clean-ups were put in place. This process also helped to focus on future, long-term solutions to advocate for with local authorities. Among others: designated waste disposal sites, improved water provision, sanitation and lighting and communal storage and refrigeration facilities.

 

In relation to this process, Rashid Mutua, National Chairperson Muungano wa Wanavijiji, says, “[The balloon mapping project] has helped the community understand the issues of health, safety, and wellbeing. The collection of data, the trainings, the awareness, the mobilisation, have helped the community […] to understand their issues so that they can push the bigger agenda of the Federation and of the community.”

 

Through community-led mapping, people in informal settlements have the opportunity to monitor their environment, assess the challenges they face and come up with solutions grounded in verifiable data.

 

By mapping themselves out the communities in Mathare, and other settlements in Nairobi, embarked in a dialogue with authorities that will they hope will lead to their recognition and support. By putting themselves on the map, they have become visible.


 ‌Erwin Geuder-Jilg newsletter photo for ECD newsletter

Stefanie Krapp photo for ECD newsletter number 3

 

DEval’s ECD Approach in Costa Rica - Involving Civil Society in Evaluation

By: Erwin Geuder-Jilg and Stefanie Krapp, DEval – FOCEVAL

The German Ministry for Economic Development and Cooperation (BMZ) has given the mandate for the area of ECD explicitly to the German Institute for Development Evaluation (DEval [1]). ECD is anchored within DEval as cross-cutting theme.

 

Since mid-2014 DEval is implementing a pilot project to strengthen evaluation capacities in Costa Rica and selected countries of Latin America (“FOCEVAL”). In this project, DEval is following a systemic approach, working at individual, institutional and societal levels which are interconnected parts of a whole, with the aim to develop far-reaching synergies. Thereby FOCEVAL contributes to strengthening the training offer in evaluation and works towards institutionalising evaluation in processes of public and non-public organisations and sub-systems of society. The project is planned and implemented together with a coordination group, composed of central representatives of public administration, civil society and science. This group is linked to various civil society organisations which are monitoring and engaging for better public services at local (municipality or community) level.  

 

Involving civil society in evaluation processes is one major pillar of the project which includes two dimensions: civil society participation in the evaluation of public programmes; and participatory evaluation.

 

Civil society in Costa Rica is mainly involved in audits as a means of oversight of local public institutions. Knowledge about the role and functions of evaluations are still scarce. FOCEVAL sensitises Civil Society Organisations on evaluation and enables them to become informed on the Costa Rican National Evaluation Agenda. Through these measures they are empowered to influence terms of reference and to improve their access to the results and recommendations of the evaluations. At the same time, FOCEVAL trains civil society organisations to enhance their skills in carrying out evaluations, especially in participatory approaches. 

 

In the next months it is planned to support civil society organisations to carry out participatory evaluations either of their own or of public programmes at local level. This approach also seeks to strengthen evaluation capacities in civil society and allows organisations to look critically both at their own as well as at the government’s performance.

 

Furthermore, FOCEVAL is developing capacities of evaluators and steering groups in public institutions, creating awareness about the involvement of civil society in the evaluation process and the use of results from participatory evaluations. It also supports events to share evaluation results with civil society and other actors. 

 

Civil society organisations engaged with FOCEVAL already are monitoring specific public services such as water supply, health or education services, and will continue this monitoring more effectively with the experience of the evaluations and information on evaluation results and recommendations. The following diagram shows how civil society involvement is strengthened by FOCEVAL and how it contributes to better public services.

 

Source graphic compilation - ECD Newsletter April 2016 article on DEval’s ECD Approach in Costa Rica - Involving Civil Society in Evaluation by Erwin Geuder-Jilg and Stefanie Krapp

Source: Own compilation

 

Both civil society and the government can gain from these processes: government’s programmes become more transparent and the public administration can open a new channel of dialogue with the population and gain legitimacy by improving performance. Civil society can hold public institutions accountable for their performance, increase its influence on agenda setting and work towards more benefits from public programmes. In 2015 and 2016 the Costa Rican National Evaluation Agenda is focusing on those areas which are relevant public services for the population: employment, mother and child care, food provision for hospitals, schools etc. and security. Civil society finds a space to engage with the government between elections. It provides the angle of the beneficiaries of public services (how did the programme change my life?), while public institutions start evaluations from the angle of service providers (how much money was spent? how many persons did we reach? etc.). Participation is a chance to move from evaluating performance towards evaluations focusing on the well-being of the people.  

 

Of course, participation is a standard element in evaluations, yet the level of participation is often limited to beneficiaries answering questions asked by experts. The delineated models give civil society a more active role and an early engagement in the evaluation process.

 

The project’s objective is to strengthen the role of evaluation in political decision-making and the steering of policies also through the participation of civil society in evaluations, together with a better quality of evaluations and institutionalising M&E processes. The final impacts are improved public programmes oriented at the society’s needs and comprehensive accountability, in other words better governance. 


 

 ‌Robert Picciotto  photo for the ECD newsletter of April 2016. Issue # 3

Rejoinder - Evaluation for Democracy

By: Robert Picciotto. King's College, London

Evaluation as a tool for effective governance is the overarching theme of the Global Evaluation Agenda. The same theme is evoked by the new development compact that emerged following the largest international consultation process ever carried out in United Nations history - the Sustainable Development Goals (SDGs). 

 

The 2030 Agenda for Sustainable Development is applicable to all countries (developed and developing). It states that review of the SDGs will be “rigorous and based on evidence, informed by country-led evaluations”. Furthermore it calls for the “strengthening of national data systems and evaluation programmes”. What does this mean for evaluation in an era of unprecedented economic uncertainty and social turmoil? 

 

Leaving no one behind is the core aspiration of the new global goals. This cross-cutting priority has major implications for evaluation processes and practices. First it transcends the current obsessive concern with the merit dimension of the evaluation craft- ascertaining whether policy interventions work as intended by power holders.  Merit oriented evaluation currently dominates the world of market driven practice. The approach is goal achievement oriented and it favours experimental approaches. It certainly has its place under the evaluation sun but as currently practiced it has a relatively narrow remit: does the intervention work? 

 

Leaving no one behind - the rallying cry of the SDGs - means that evaluators should ask other questions. How relevant is the intervention to the fate of the poor? Why did the intervention work or fail to work? For whom did it work? What else could have been done? Where does one go from here? This broader mandate asks evaluators to deliberately focus on the worth of existing interventions from the perspective of the powerless and the disadvantaged in order to create a level playing field. Doing so implies giving voice to the voiceless. It encourages a deliberate focus on indirect, secondary and unintended effects. It challenges vested interests and it helps to identify more equitable and sustainable policy solutions.

 

Leaving no one behind also implies the use of value frameworks that reflect the universal equity oriented and environmentally sustainable aspirations of the new global goals. These are not necessarily those embedded in the utilisation focused evaluations currently commissioned by decision makers. In turn this means that evaluators committed to the public interest will henceforth be called upon to exercise robust ethical judgments and to implement a multi-faceted conception of evaluation that is faithful to the consensus definition of evaluation as a process that examines not only merit but also worth and value. 

 

Getting back to basics in terms of what evaluation is has major implications in terms of achieving effective evaluation governance. Genuine evaluation combines accountability with learning - two sides of the same coin.  At the level of individual evaluators' dispositions democratic evaluation highlights the imperative of independence of mind and appearance. At the level of organisations it promotes synergy between independent and self-evaluation. Finally for the society as a whole a democratic evaluation model fit for the 21st century gives pride of place to genuinely participatory evaluation that ensures fulsome involvement of all stakeholders.

 

This is where such initiatives as those pioneered by the German Institute for Development Evaluation and the International Institute for the Environment come into their own. They focus on issues of concern to those that have been left behind. They make power holders accountable for their sins of omission as well as their sins of commission. They put the core principles of a democratic society to work by mapping the geography of community needs, shining light on top down initiatives, and promoting civil society involvement in the policy decisions that affect the lives of vulnerable citizens and communities. In a nutshell they show that evaluation can be wielded as an instrument of democratic governance.


 [1] Deutsches Evaluierungsinstitut der Entwicklungszusammenarbeit 

 

This newsletter is produced by the DAC Network on Development Evaluation’s task team on capacity development.

The DAC Network on Development Evaluation contributes to better development results by using evaluation to build a strong evidence base for policy making and for learning.  

The Network is a subsidiary body of the Development Assistance Committee (DAC) of the OECD.

If you would like to sign up for this newsletter, submit content, comments or to unsubscribe, please email: dacevaluation.contact@oecd.org

Download the newsletter (.pdf) Back to top ↑