Everybody Cares About Using Education Research Sometimes

7. Knowledge mobilisation as an engine for education research to fly

Nóra Révai
OECD

Jordan Hill
OECD

Abstract

This chapter brings together the lessons learned from this report. It begins by elaborating on the state of knowledge mobilisation in education, according to the research literature and analysis of the OECD Survey of Knowledge Mobilisation in Education, including the developments and challenges reported in OECD countries. The chapter then plots several possible courses of action for policy and intermediary actors to improve their knowledge mobilisation efforts. Finally, it offers short- and long-term options for future international efforts, ranging from modest to bold.

Introduction

Using data from the 2023 OECD Survey of Knowledge Mobilisation in Education (henceforth, the intermediaries’ survey), this report has explored the experiences, perspectives and activities of a diverse range of educational organisations that foster the use of evidence in policy and practice. As a first international effort that moves beyond merely formal knowledge intermediaries, the intermediaries’ survey aimed to characterise knowledge mobilisation activities and their effectiveness and impact across different organisations. It collected data from 288 intermediary organisations in 34 countries, which included formal knowledge intermediaries, public intermediaries (embedded in public administration), research institutes, initial teacher education institutes, inspectorates and quality assurance agencies, teacher unions and consulting firms.

As emphasised throughout this report, the analysis provided has some limitations originating from the survey design and the data sources. Despite these limitations, this analysis provides the most comprehensive overview of self-reported data focused exclusively on knowledge mobilisation in education to date. To complement survey data, this report also presented eight case studies from seven countries showcasing knowledge mobilisation practices at different levels of the education system by different types of organisations. The combined quantitative and qualitative data helped to paint a comprehensive picture of the state of knowledge mobilisation today in OECD education systems, allowed for some cautious implications and, more importantly, provided food for thought for improving this field.

This final chapter summarises the key developments in the field of knowledge mobilisation and the gaps and challenges that remain to be addressed. It then provides a set of pointers for different actors on how they might contribute to addressing these. Finally, it outlines ways forward for further research efforts and for international actors’ work on knowledge mobilisation itself.

While this synthesis draws primarily on the chapters of this report, it also incorporates conclusions that have emerged from the OECD’s previous analyses on knowledge mobilisation (OECD, 2022[1]; OECD, 2023[2]). These include analyses of the 2021 OECD Strengthening the Impact of Education Research policy survey, which collected data from 37 ministries of education in 29 countries, as well as analyses of the follow-up interviews of the two surveys, and in-depth conversations with several intermediary and policy actors.

The state of knowledge mobilisation: Developments and challenges

The findings of this report help identify the major developments in making the thoughtful and systematic use of evidence a reality, as well as the major gaps that remain.

Developments in the field

There is an increasing body of education research, which includes a range of research methodologies, such as experimental designs, cross-sectional quantitative studies and qualitative research; all of which are potentially valuable sources to inform and improve policy and practice across the world (Van Damme, 2024[3]). While some actors call for a more concerted effort to further improve the quality of education research (see e.g. Van Damme’s and Schneider’s opinion pieces in (OECD, 2022[1])), there is also a need to systematically mobilise the knowledge already available.

Education research is increasingly synthesised and made accessible to policy makers and practitioners

Many organisations reported that they disseminate research evidence in user-friendly formats, such as briefs. Evidence repositories are increasing in number; the most well-known national repositories are the Education Endowment Foundation’s (EEF) Teaching and Learning Toolkit in the United Kingdom and the What Works Clearinghouse’s (WWC) repository in the United States. A well-known international example is the repository of impact evaluations of the Abdul Latif Jameel Poverty Action Lab (J-PAL). However, the intermediaries’ survey results suggest that there are many more repositories beyond these well-known examples. Among survey respondents, 99 organisations claimed to curate and maintain an evidence repository. The types of evidence that repositories gather range from a collection of individual studies of various methodologies – through repositories specialised on intervention studies (experimental and quasi-experimental designs) – to systematic reviews. Examples of recent efforts of this latter type of evidence include the reviews produced by the Norwegian Knowledge Centre (NKC) and the German DIPF | Leibniz Institute for Research and Information in Education. Repositories can serve as key access points of evidence syntheses for policy makers and practitioners, and can greatly facilitate research mobilisation, especially if they are presented in accessible formats.

Numerous educational actors engage in knowledge mobilisation activities, and some have accumulated substantive experience

The intermediaries’ survey revealed that almost all respondent organisations engage in some form of knowledge mobilisation, at least sometimes (hence the title of this report). There are more and more formal intermediaries, and they cover an increasingly wide geographical area. In the survey, 56 organisations from 23 countries identified themselves as a formal intermediary. Several of these have been established (or have extended their scope) recently. For example, Leerpunt in Flanders was set up in 2023, while the Netherlands Initiative for Education Research (NRO) is shifting its scope from a funding organisation to a full-fledged knowledge mobilisation agency (see the Netherlands case study in Chapter 3). However, it is not just formal intermediaries that play a role in knowledge mobilisation. Research institutions and consulting firms are particularly active, but initial teacher education providers, inspectorates and teacher unions also care about mobilising evidence.

A few organisations, such as the EEF in the United Kingdom, have accumulated substantial experience in how to mobilise knowledge well (or at least better) and also some hard evidence on how to do this effectively. More recent players have also applied innovative approaches to knowledge mobilisation. The knowledge mobilisation practices presented in the eight case studies of this report all demonstrated some unique or promising features that are worth attention (see Table 7.1 for a brief summary of these).

Table 7.1. Features of the case studies included in this report

Country (system)	Type of organisation(s)	Unique features of the knowledge mobilisation initiatives
Netherlands	Formal intermediaries (embedded in public administration and independent)	Two system-wide and complementary knowledge mobilisation efforts with substantive funding from government; their activities include linear, relational and systems approaches: needs-based research production and transfer, co-creation and capacity building Co-ordinating knowledge mobilisation in a small, decentralised system
United Kingdom (England)	Research institution and funding organisation	Systemic mapping of knowledge mobilisation actors and the policy context to maximise the impact of research projects
Japan	Teacher training institution	Dialogues to build shared understanding of evidence across actors (researchers, policy makers and practitioners) in a hierarchical system
United States	Formal intermediaries, research institutions, consulting firms, etc.	Government-funded groups of actors supporting knowledge mobilisation at the state and local levels in a federal, highly decentralised system Connecting national-level (top-down) and local (bottom-up) knowledge mobilisation efforts Evaluation of programme impact and dissemination of impact stories
Latvia	Public intermediary (ministry of education)	Government funding schemes that generate policy-relevant research through involving the research community Cross-sectoral funding approach and balance of foundational and applied research
Chile	Formal intermediary (NGO)	Support of the uptake of evidence-based practices in several countries and evaluation of their impact A diverse funding model with advantages and limitations
United Kingdom (England)	Formal intermediary (charity)	Rigorous impact evaluation of a knowledge mobilisation effort, with sophisticated impact measures
Germany	Research institution	Combination of a knowledge mobilisation unit with transversal responsibilities of other units in an organisation to support evaluation Both external, formal evaluation of knowledge mobilisation efforts and internal, formative evaluation to foster evaluative thinking

Overall, the substantial growth in knowledge mobilisation activity and the much-needed efforts at synthesising knowledge are reassuring. The fact that this activity has been taking place across many systems, organisations and individual actors suggests that the agenda of evidence-informed policy and practice has been increasingly embraced.

Gaps and challenges

The rapidly accumulating experience of intermediaries and the growing body of research on knowledge mobilisation can provide systems and organisations with guidance on improving intermediary functions. However, there are still important obstacles that the field is facing.

Knowledge mobilisation activities are not sufficiently co-ordinated within and across systems, nor are the increasing number of evidence repositories

Within the rich and diverse landscape of knowledge mobilisation activities illustrated in this report, a key challenge is to ensure that these are effective, but also that they meaningfully complement each other. This requires an awareness across the actors of who does what in the system and an understanding of the needs, interests and capacity of different organisations to make sure that “the right actors play the right roles”.

The OECD’s numerous interactions with a range of major formal and public intermediaries suggest that there is very little effort to co-ordinate across the different evidence repositories. The lack of co-ordination across intermediaries, and in particular, evidence repositories, has two major implications. First, policy makers and practitioners often do not have any guidance on what kind of evidence they can find and where. There is a lot of research out there, but it can be very difficult and time-consuming to find synthesised evidence on a given topic of interest. It is equally difficult to judge the quality of that research, as evidence repositories use different quality standards and some of them are not transparent about their criteria. The transferability of the evidence across contexts can also be hard to assess for users. Second, the un-co-ordinated efforts of evidence syntheses may result in duplication of work and gaps in evidence, reflecting inefficient use of resources. In addition, most evidence repositories are practice-focused (e.g. EEF, WWC, NKC, DIPF mentioned above), while much less evidence is tailored for policy purposes.

Research on knowledge mobilisation itself has not been synthesised, and evaluations are still lacking

The growing body of knowledge and experience on knowledge mobilisation itself has not been synthesised and made widely and easily accessible to intermediaries. In addition, there are still knowledge gaps, notably on the effectiveness of various knowledge mobilisation activities. This likely originates from low levels of evaluation of current knowledge mobilisation initiatives. Evaluation practices are often sporadic and limited to simple metrics, such as counting the number of citations or event participants. A lack of causal evidence of “what works” in knowledge mobilisation and a lack of synthesis of existing knowledge makes it difficult for new intermediaries to easily develop robust strategies and work plans, and for existing intermediaries to improve their strategies and activities.

Intermediary actors lack learning opportunities to build capacity and share their experiences

The intermediaries’ survey data showed that very few organisations develop the skills and capacity of intermediaries to effectively broker knowledge. An important form of building the knowledge base on knowledge mobilisation would be to generate self- and peer-reflection on knowledge mobilisation initiatives across intermediaries. However, while some of these actors have substantive experience and engage in innovative practices, knowledge exchange to facilitate deep forms of peer learning across intermediaries might still be limited, as suggested by the lack of capacity building for intermediary actors reported in the intermediaries’ survey. Some notable efforts include the Global Partnership for Education Knowledge and Innovation Exchange (GPE-KIX) initiative and Jacobs Foundation’s education policy labs (IDRC, 2024[4]; Jacobs Foundation, 2024[5]) – both supporting countries primarily in the Global South in evidence use, the Transforming Evidence Network (Transforming Evidence, 2024[6])– a cross-sectoral forum for evidence experts to share knowledge, and the more recent Evidence for Education Network (EEF, 2024[7]) – a network of EEF’s partners that supports evidence generation and the adaption of their Toolkit. But more effort is needed to provide structured capacity building and peer learning on knowledge mobilisation for intermediaries, tailored to their needs.

Lessons for different actors

To effectively build on the aforementioned developments and address remaining challenges, action is required from all parts of the system to strengthen the impact of education research. The following suggestions are targeted at policy and intermediary actors – the respondents of the two OECD surveys.

Policy actors

Policy organisations could play a role in co-ordinating knowledge mobilisation activities, building on relevant evidence

Policy organisations need to draw on evidence on knowledge mobilisation to build effective mechanisms. This involves reflecting on evidence use in policy and understanding how this can be improved in their own organisations. In addition to rethinking evidence use in their own area, policy organisations can also reflect on the actions they can take to support other actors. Notably, linear, relational and systems approaches should be applied to the field of knowledge mobilisation.

Both ministries of education and local authorities would benefit from mapping intermediary actors in their country or region, and reflecting on the roles they are playing or should/could be playing in knowledge mobilisation. While there is no one across-the-board approach in terms of which organisations should do what, some roles can be more clearly attributed to certain types of organisations (e.g. initial teacher education institutes could develop teachers’ research literacy and use skills). An optimal distribution of roles may require an assessment of intermediaries’ capacity, needs and interests.

Such a mapping, capacity assessment and reflection on roles could be the basis for co-ordinating knowledge mobilisation at different levels. This involves creating appropriate incentives for different types of intermediaries so that they can carry out their work effectively. Diverging perspectives between policy makers and intermediaries demonstrate the need for improved communication about knowledge mobilisation mechanisms and available resources. Co-ordination could be mandated to major formal intermediaries, or central agencies (ministries, local authorities) could provide grants for intermediary networks to facilitate co-ordination and collaboration. Terms of reference of grants should align with the evidence: allocating sufficient resources for long-term collaboration, capacity building of diverse actors and evaluation mechanisms.

To generate further evidence on knowledge mobilisation, central leadership in ministries and local agencies needs to support more systematic and thoughtful evaluation. While evaluating everything all the time may be not possible, regular and rigorous evaluations of major knowledge mobilisation programmes are important. This should be accompanied by encouraging a culture of evaluative thinking, where self- and peer-reflection is integrated into intermediary actors’ standard practice.

Intermediary actors

Formal intermediaries could support co-ordination and capacity building, and be role models in evaluative thinking

Just like policy organisations, formal intermediaries need to use knowledge mobilisation research and effectively combine linear, relational and systems approaches. They can support all the systemic efforts of policy organisations, such as the systematic mapping of actors and their activities, and the co-ordination of knowledge mobilisation activities (locally or more broadly). Formal intermediary organisations could play an important role in designing and implementing capacity building for their own staff and other intermediary actors to develop brokerage skills. These organisations should be role models in developing evaluative thinking around knowledge mobilisation. This can involve systematically reflecting on and, to the extent possible, evaluating and measuring the impact of their own activities, as well as supporting or conducting rigorous evaluations of other actors’ activities.

Research organisations could strengthen policy- and practice-engagement and fill research gaps on knowledge mobilisation

Currently, research institutions’ focus is primarily concentrated on evidence generation and dissemination. In an effort to build a robust knowledge base in education, research institutions can pay greater attention to generating research that responds to practice and policy needs. To do so, they need to assess and build the capacity of the research community to engage with policy makers and practitioners, to develop a greater understanding of their contexts and to build effective partnerships. When research is generated through research-practice or research-policy partnerships, research institutions can also play a role in building the research literacy and engagement skills of practitioners and policy makers.

Research institutions can also invest in conducting research on knowledge mobilisation itself to extend this body of evidence beyond Anglo-Saxon countries. Diverse, international research collaborations can ensure that knowledge mobilisation practices are explored in various contexts and that their transferability is examined. Research should also move beyond descriptive studies of interventions and contribute to rigorous evaluations of knowledge mobilisation activities. Importantly, there is a need to apply more complex indicators of impact, as well as methodologies for measuring these.

Practice-focused organisations could focus more on developing practitioners’ research engagement skills, and generate and mobilise evidence in their context

Despite the prevalence of evidence dissemination and partnership-building efforts, there is still a lack of support for building practitioners’ capacity in research engagement. Initial teacher education institutions and professional development providers should collaboratively build these skills to enable teachers and school leaders to access relevant evidence and thoughtfully engage with it to improve teaching and school practices.

Inspectorates and quality assurance agencies generate valuable data and evidence on school and teaching quality. Some of these institutions have a pure accountability mandate, while others also provide support for schools or school boards – depending on the context. The former type can support the use of their evidence in policy making not just through publishing reports (which many of them do) but also more varied knowledge mobilisation activities, such as directly supporting ministries in policy design and implementation. The latter type can do the same not just for policy organisations, but also schools.

Teacher and school leader unions in some countries are engaged in generating evidence (e.g. through teacher surveys) and supporting its mobilisation, primarily in policy. For unions that generate evidence, it is important to have sufficient research capacity to ensure that this evidence is rigorous. Unions can work with other intermediary actors to support the mobilisation of this evidence.

Funders need to provide more comprehensive funding structures and target a broader set of actors and actions

The way knowledge mobilisation is currently funded often does not align well with the needs of policy makers, practitioners or even the rhetoric of funding organisations regarding co-created solutions. Funding for both research generation and mobilisation needs to extend to a broader array of education actors, rather than being concentrated among research institutions and formal intermediaries. To optimise knowledge mobilisation, funders must take a more inclusive approach, rebalancing funding towards schools and practitioners who are at the forefront of classroom and school innovation. Researchers and practitioners alike require targeted wrap-around schemes that combine financial support with training, networking and tailored support services to provide a more comprehensive development of the capacity of local actors to mobilise knowledge effectively.

Funding structures also need to appropriately reflect the time, effort and money needed to actually use research. More flexibility in funding schemes would support planning for impact as the context of research use becomes known. Importantly, funding needs to be allocated for evaluating research mobilisation efforts.

Ways forward: Short- and long-term pathways to improve knowledge mobilisation in policy and practice

The analyses emanating from the OECD Strengthening the Impact of Education research project (OECD, 2022[1]; OECD, 2023[2]), including this report and the applied work conducted with countries through meetings and learning seminars (OECD, 2024[8]), have helped countries to think about ways to improve knowledge mobilisation in their education systems. In 2024, during the Belgian presidency of the European Union, the European Commission accepted Council Conclusions on promoting evidence-informed policy and practice in education and training (Council of the European Union, 2024[9]) that draw heavily on OECD work. This work has also contributed to many other intermediary efforts (see e.g. Hayter and Morales (2023[10]), UNESCO (2024[11]), Ward and Mouthaan in Baek and Steiner-Khamsi (2024[12])). While such impact is important, more work is needed to ensure the systematic and high-quality use of education research in policy and practice.

The developments outlined in this chapter can act as levers to drive further change. Three desired long-term outcomes can be envisaged for the future of evidence-informed policy and practice:

• Countries (policy makers) have data and evidence on effective knowledge mobilisation, and tools to improve it in their system.

• Organisations and actors with knowledge intermediary roles know and apply effective mechanisms of knowledge mobilisation.

• Actors learn from each other and collaborate constructively in the production, mobilisation and use of quality and relevant research.

Achieving these outcomes requires concerted effort; this chapter has described ways in which different actors can contribute to such effort. In addition, international organisations can play a role in generating peer learning and co-ordinating certain efforts at the international level. The OECD is itself a knowledge intermediary and, as such, it needs to reflect on how it can improve its knowledge mobilisation activities.

Short-term modest and not-so-modest pathways forward

The next step for international work on promoting knowledge mobilisation and for international actors such as the OECD could be more practical co-ordination work to support achieving the above-mentioned outcomes through the following pillars:

Supporting intermediaries in driving the generation of policy- and practice-relevant evidence in education (quite bold)

Addressing the challenge of misalignment and lack of co-ordination across evidence repositories involves co-ordinating evidence syntheses and databases across key intermediaries. This would require technical explorations, such as the alignment of the underlying data structures of repositories (individual studies behind evidence syntheses), and testing how stronger connections can be made between them. It could extend to conceiving a platform that gathers and connects existing evidence syntheses and repositories in collaboration with the organisations hosting these.

Synthesising knowledge mobilisation research and making it accessible (somewhat modest)

Supporting the effective mobilisation of knowledge mobilisation research could involve creating a resource bank of evidence-informed practical tools that support designing knowledge mobilisation strategies, mapping actors and mechanisms, assessing capacity for knowledge mobilisation, developing policy-research and research-practice partnerships, etc. Such tools could take the form of thematic research briefs, frameworks and vignettes with use cases, and in-depth case studies showcasing effective knowledge mobilisation initiatives. To support more sophisticated evaluations, a framework of the impact of knowledge mobilisation initiatives could also be developed.

Capacity building and peer-learning for intermediaries (modest)

Filling the gap of developing intermediary actors’ capacity could involve needs-based thematic peer-learning events on knowledge mobilisation to support intermediaries in developing and improving their strategies and activities. Focusing on formal intermediaries in a first stage could make these organisations multipliers who could then build the capacity of other actors in their national contexts. Public intermediaries, i.e. knowledge and analytical units within ministries of education, could also be a key target group to get support in improving knowledge mobilisation in policy and practice.

Long term modest and bold pathways forward

In the long term, the OECD could play a role in maintaining and extending the aforementioned short-term plans. However, to ensure long-term impact, diverse national and international actors need to work together.

Create a robust knowledge base on knowledge mobilisation: Extend research of intermediaries (modest) and regularly update and synthesise this (bold)

In addition to the various directions for future research on knowledge mobilisation outlined in this and the two previous reports (e.g. beyond Anglo-Saxon countries, more evaluations of interventions and initiatives) (OECD, 2022[1]; OECD, 2023[2]), the mapping of the knowledge mobilisation landscape could also be further extended. The intermediaries’ survey did not include several actors who might play important knowledge mobilisation roles. Future efforts could map professional development providers and umbrella organisations, such as professional associations, policy and research networks. Importantly, the intermediaries’ survey focused on actors within national education systems, while it did not include international organisations, such as the OECD, UNESCO and the World Bank, nor think tanks and consulting firms of international scope, many of which provide public consultancy services.

As research on knowledge mobilisation extends, including more evaluations that provide information on what works, it will be necessary to regularly synthesise this research, e.g. through systematic reviews and meta-analyses. In addition, more cross-sectoral research synthesis can strengthen learning from knowledge mobilisation in other fields. An important condition of improving the use of evidence is to support all types of intermediary actors in producing, accessing and using this body of knowledge.

Create and sustain a self-improving intermediary ecosystem (bold)

The goal should be for the intermediary field to become an organic, self-developing ecosystem maintained and supported by adequate resources and incentive structures. This ecosystem should be able to systematically identify research gaps, thereby providing input for countries, research funding organisations and research institutions to drive a more robust knowledge building in education through relevant research grants and projects. It should also synthesise evidence in a cost-effective and co-ordinated way and provide policy makers and practitioners with adequate support to mobilise this evidence in their decisions.

To achieve this goal, it will be necessary to systematically map evidence repositories and systematise their content in an accessible and easy-to-navigate way over time. In turn, this necessitates supporting continuous collaboration of a range of intermediaries, towards a collective development of evidence syntheses with structured information about their quality and transferability across contexts. Maintaining the network of intermediaries is also necessary to ensure sustained peer learning and continuous development of the knowledge base on effective knowledge mobilisation.

Both short- and long-term pathways envisaged above are obviously resource-intensive – but is it not worth investing in the engine of education research to make it fly and help build efficient and equitable education systems across the world?

References

[12] Baek, C. and G. Steiner-Khamsi (eds.) (2024), The Rise of Knowledge Brokers in Global Education Governance, E. Elgar Publishing.

[9] Council of the European Union (2024), Council conclusions on promoting evidence-informed policy and practice in education and training to achieve the European Education Area (C/2024/3642), Official Journal of the European Union, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:52024XG03642.

[7] EEF (2024), Evidence for Education Network, https://evidence.education/about-us/the-network (accessed on 25 November 2024).

[10] Hayter, E. and M. Morales H. (2023), Review: Policy Labs and Evidence Use in Education, Jacobs Foundation, https://jacobsfoundation.org/wp-content/uploads/2023/10/JF_Policy-labs-and-evidence-use-in-education_rev-2024.pdf.

[4] IDRC (2024), The Global Partnership for Education Knowledge and Innovation Exchange (KIX), https://www.gpekix.org/ (accessed on 25 November 2024).

[5] Jacobs Foundation (2024), Policy labs and evidence use in education, https://jacobsfoundation.org/publication/policy-labs-and-evidence-use-in-education/ (accessed on 25 November 2024).

[8] OECD (2024), “Yes Minister, Yes Evidence: Structures and skills for better evidence use in education policy”, OECD Education Policy Perspectives, No. 96, OECD Publishing, Paris, https://doi.org/10.1787/6f97bcda-en.

[2] OECD (2023), Who Really Cares about Using Education Research in Policy and Practice?: Developing a Culture of Research Engagement, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/bc641427-en.

[1] OECD (2022), Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/d7ff793d-en.

[6] Transforming Evidence (2024), Transforming Evidence for Policy and Practice, https://transforming-evidence.org/ (accessed on 25 November 2024).

[11] UNESCO (2024), Functional Area 1 Learning Series: Using Evidence to Transform Education Policy, https://www.unesco.org/sdg4education2030/en/evidence-and-policy-learning-series (accessed on 18 November 2024).

[3] Van Damme, D. (2024), Keynote speech: Conference: Evidence-informed education, policy and practice. Belgian presidency Council of the European Union.