Melissa Mouthaan
OECD
Jordan Hill
OECD
Everybody Cares About Using Education Research Sometimes
Perspectives of Knowledge Intermediaries
Abstract
Critical reflection is crucial for improving and learning from any initiative. This chapter assesses the extent to which knowledge mobilisation activities are monitored and evaluated by intermediaries, and how. It examines why intermediaries evaluate their work, as well as the barriers to monitoring and evaluation. It also enquires as to how intermediaries perceive the impact of their own activities. Finally, the chapter draws on two case studies in Germany and the United Kingdom (England) to explore what lessons there may be for other organisations that wish to implement or improve monitoring and evaluation methods to strengthen their efforts to mobilise research knowledge in education.
Monitoring and evaluating knowledge mobilisation initiatives helps to ensure their quality and to build the knowledge base on what is effective in different contexts. Analysis of responses to the 2023 OECD Survey of Knowledge Mobilisation in Education highlights the need for building a better understanding of the evaluation of knowledge mobilisation practices among intermediaries, and for promoting it as a more systematic and thoughtful practice.
Currently, intermediaries tend to monitor or evaluate their knowledge mobilisation initiatives only sporadically. When they do, they often favour simple indicators, such as number of citations of a study or number of participants in an event, which do not give a full picture of impact.
A substantial number of intermediaries are transparent about their lack of monitoring or evaluation practices. Lack of funding and lack of time were most often cited as the biggest barriers. Only a minority of respondents cited a lack of skills or knowledge in evaluation as a barrier.
Among the activities in which they engage, organisations perceive linear mechanisms, such as conducting research and disseminating research, as being the most effective. This view is poorly aligned with the research literature on knowledge mobilisation that posits that linear mechanisms are necessary but not effective on their own.
Supportive leadership, at the organisational level but also crucially at the system level, can help decide how much time and resources to allocate to monitoring or evaluation activities, and ensuring this is proportionate to the time and resources invested in the initiative itself.
A focus on developing evaluative thinking is needed. This is a mindset of professional collective reflection on tasks and initiatives across intermediaries and the system, in addition to conducting and publishing more and better-quality evaluations where needed.
It seems unlikely that the conceptual and empirical experimentation required to develop measurements for the complex social indicators of impact reported in the survey has been implemented at scale.
Evaluation – i.e. the structured and objective assessment of an ongoing or completed initiative to determine the relevance and fulfilment of its objectives (OECD, 2020[1]) – is crucial for ensuring high-quality implementation because it allows organisations to learn and improve through critical reflection. In education, low levels of systematic policy and programme evaluation were noted almost three decades ago (OECD, 1997[2]), and many education systems continue to lack the regulation, funding and guidance required to support a system-wide culture of evaluation (Golden, 2020[3]). Gathering monitoring indicators is an important first step, which becomes evaluation when there is critical reflection on the data collected.
Despite a core aim of intermediary initiatives often being to close the gap between research and practice, a lack of evaluations is thought to be inhibiting the development of an international knowledge base on research use, perpetuating rather than closing this gap (Kislov et al., 2018[4]). In their review of almost two thousand initiatives worldwide, Oliver et al. (2022[5]) found that only 6% of these were evaluated. Most of these evaluations were conducted by researchers involved in the delivery of the initiatives, as opposed to independent evaluations, and were reports for funders, which lacked detail of activities and broader lessons that support knowledge mobilisation and transfer.
This situation is described by Oliver et al. (2022, p. 704[5]) as an “increasing mass of rudderless activity”, which can lead to wasted resources and fails to provide useful lessons for improving evidence use.
Although increasing evaluation activity in education could help resolve tensions between research and practice, which tend to arise when trying to scale up successful local pilot innovations (Golden, 2020[3]), such evaluation activity needs to be high-quality, independent where possible, and to try and distil actionable information for policy or practice.
This chapter analyses the results of the 2023 OECD Survey of Knowledge Mobilisation in Education (henceforth, the "intermediaries’ survey”), which collected data about the knowledge mobilisation activities of 288 intermediary organisations in 34 countries to better understand how they are supporting research engagement (see Chapter 1, Box 1.1). Where appropriate, it compares these findings with data collected by the 2021 OECD Strengthening the Impact of Education Research policy survey (henceforth, the "policy survey"). This earlier survey collected data from 37 Ministries of Education in 29 countries to map knowledge mobilisation mechanisms, actors and challenges across education systems (see (OECD, 2022[6])).
This chapter seeks to answer the following three questions:
To what extent and how do intermediaries monitor or evaluate their knowledge mobilisation activities?
What are the main barriers to monitoring and evaluation among knowledge intermediaries?
Which knowledge mobilisation activities do intermediaries perceive as the most and least impactful?
In the intermediaries’ survey, respondents were asked about the extent to which they evaluate their knowledge mobilisation activities. At first glance, the results appear quite encouraging (Figure 6.1).
Monitoring and evaluation of knowledge mobilisation among intermediaries
Almost half of intermediaries report monitoring or evaluating their knowledge mobilisation activities
On average, 44% of respondents reported that they monitor or evaluate their work at least occasionally. Just under one-quarter reported that they monitor or evaluate most, or all, of their activities. At the same time, that still leaves a substantial number of intermediaries that are transparent about their lack of monitoring or evaluation practices, as shown in the expert quote below.
I would have expected the “no, not yet” responses to be a larger percentage. Having said this, the fact that around one-fifth said they don’t do anything, as opposed to “a limited extent” means we can be fairly confident that a significant minority are really doing zero monitoring and evaluation of their activities.
– Tracey Burns, Chief Research Officer, National Center on Education and the Economy (United States)
The position of knowledge intermediaries at what Kislov et al. call “the conflict-laden interface of policy making, management, science and professional practice” can politicise any evaluation attempts, as well as the outcomes (Kislov et al., 2018, p. 14[4]). A lack of motivation for evaluation can also stem from a fear of highlighting inadequacies (Golden, 2020[3]). Even if an evaluation is carried out, the findings are often not published, particularly if the results are considered negative (see Newman and Gough (2019[7])). One analysis found the number of publicly available evaluations of knowledge mobilisation interventions in education to be just 3.5% (Boaz, Oliver and Hopkins, 2022[8]). Routinely publishing evaluations is important for building an international knowledge base to inform the wider knowledge mobilisation community (see case studies in this chapter). Publishing more evaluation results for external audiences may also encourage intermediaries to undertake more robust and rigorous evaluations, and to engage with peers regarding their findings. At the same time, publishing more evaluations could magnify publication biases, which are already a significant problem in education research evidence syntheses (see, for example (Ropovik, Adamkovic and Greger, 2021[9])) or could encourage performative evaluation exercises that do not give an accurate picture (see expert quote below).
Publishing results from evaluations that are not sufficiently independent bears a risk of being mere window dressing, rather than a true accountability mechanism.
– Tia Loukkola, Head of Innovation & Measuring Progress Division, Directorate for Education and Skills, OECD
Note: Responses were given on a five-point Likert scale. The number in parentheses after the country name is the number of respondents per country. Countries are listed in descending order in terms of the total percentage reporting a “yes, for every initiative”, “yes, for most initiatives” and “yes, occasionally”. Systems included here were selected because the sample size was 10 or more respondents. The average was calculated using all 288 respondents, regardless of country.
Source: OECD Survey of Knowledge Mobilisation in Education data, 2023.
Respondents in different education systems reported different levels of monitoring and evaluation of knowledge mobilisation
When looking at the country-level data presented in Figure 6.1, there are important differences in the frequency with which respondents report that they evaluate their initiatives. For instance, only 8% of respondents in the United States reported that they do not currently monitor or evaluate their knowledge mobilisation efforts at least occasionally. This figure was over 40% for respondents in Germany. This could be due to cross-cultural response biases (although cultural differences are not always discernible (He et al., 2014[10])). Another explanation might be that these differences are linked to the long‑standing practice of public policy evaluation and the strong accountability culture in certain countries as opposed to others. For instance, in many European countries, the institutionalisation of evaluation culture came later than in Anglo-Saxon countries (OECD, 2020[1]). Differences in the data likely reflect a combination of factors around evaluation, linked to system- and organisation-level culture, capacity and resources. The small sample sizes of the intermediaries’ survey do not allow for country-specific explanations in this case, and these factors warrant further research with larger sample sizes for each system.
Low levels of quality evaluations found in the research literature (e.g. (Boaz, Oliver and Hopkins, 2022[8]; Golden, 2020[3]; Gough, Sharples and Maidment, 2022[11])), suggest that the robustness of evaluations and the meaningfulness of the measurements may be a significant issue. Understanding how – and why – intermediaries evaluate their knowledge mobilisation initiatives can help to assess the culture of evaluation and identify ways to improve it. Respondent organisations that reported monitoring or evaluating their knowledge mobilisation activities occasionally or more often were asked which indicators they use (Figure 6.2).
Note: Responses organised by total percentage of respondents reporting “systematically”, “to a large extent” and “to some extent” to the question “To what extent does your organisation use the following indicators to measure the outcomes of its knowledge mobilisation activities?” Likert scale question. N = 227. Question conditional on having responded “Yes, occasionally”, “Yes, in most initiatives” or “Yes, in every initiative” to the preceding question: “Does your organisation monitor or evaluate the reach or impact of its knowledge mobilisation activities?”.
Source: OECD Survey of Knowledge Mobilisation in Education data, 2023.
Easy-to-measure indicators, such as reach and access, are widespread, while measurement of the impact of knowledge mobilisation on relationships and competencies is less common
The most commonly used indicators were reach and access indicators, such as the number of citations of a study or number of participants in an event. This is unsurprising given the relative ease with which this kind of data can be systematically and automatically gathered. Such types of data are required to measure the engagement of stakeholders with knowledge mobilisation activities and research outputs.
However, the use of these indicators to measure the outcomes of knowledge mobilisation activities does not provide any information on the nature and extent of this engagement, nor its impact, as noted by one expert (see below).
Respondents should be asked to clarify what measures they are using for some of these indicators. The indicators by themselves don’t tell us much about what respondents understand for these categories, or why they responded in the way that they did.
– Makito Yurita, Professor, National Institute for School Teachers and Staff Development (NITS), Japan
It is notable that relatively few organisations reported using indicators to monitor or evaluate relationships, such as an enhanced number of relationships generated between actors, or the quality of relationships having improved. Among the organisations that reported being active at least to some extent in fostering relationships, most reported evaluating these activities at least to some extent (68%). Yet a significant percentage (42%) reported that they evaluated very little of these activities or not at all. There are a few possible reasons for this. First, relationship building may be seen not as a primary activity, but as a side effect of other activities. Secondly, indicators of impact such as increased quality of relationships among actors can be difficult to measure. Some methods of evaluating relationship building include gathering data from users of resources, network participants or seminar participants (see the German Institute for Adult Education case study in this chapter). It is also possible that the responses given in this category reflect the limitations of the intermediaries’ survey (such as that examples of indicators given in the survey were insufficiently clear or concrete, or did not resonate with respondents).
In addition, the data show that competency indicators (for example policy makers’ and practitioners’ research literacy, or their attitudes towards research) are among the least commonly reported. Again, this is perhaps unsurprising if we consider that few intermediaries report conducting training or capacity building among their knowledge mobilisation activities (see Chapters 2 and 3). It may also reflect methodological challenges (see following quote).
Although measuring impact on competencies could be partially done through straightforward mechanisms, such as post-training surveys that seek to measure the knowledge gained, it may well be that intermediaries still perceive a lack of robust, simple and locally relevant instruments to measure competencies like research literacy and attitudes to research use.
– Mark Rickinson, Associate Professor in the Faculty of Education at Monash University
Behavioural change and educational and social outcomes are surprisingly highly-reported measures of impact, although this may be the result of conceptual unclarity
Some of the responses suggest there might not be a common understanding of what measures can be counted under the behavioural change and educational and social outcome indicators. As the quote below highlights, increased research use was moderately reported as a behavioural change indicator, but respondents may consider increased access to research (or, accessing of research) to be the equivalent of increased research use. Intermediary initiatives often target complex social outcomes such as student learning, or behavioural outcomes such as research use. Gough, Sharples and Maidment (2022[11]) found, however, that the impact of intermediary activities is rarely measured in these areas.
The fact that increased research use is frequently reported is surprising and respondents may have been biased in their response, equating research access with research use. But we know these are not the same thing.
– Annika Wilmers, Senior Researcher, DPIF | Leibniz Institute for Research and Information in Education
Overall, it is considered highly challenging to measure impact at the level of wider educational and social outcomes (Torres and Steponavičius, 2022[12]; Lavis et al., 2003[13]). Yet, 40% of respondents indicated measuring educational and social outcomes, such as students’ academic achievement, to a large extent or systematically as measures of the impact of their knowledge mobilisation activities. It might be that respondents were reporting measuring student achievement more generally, not as a result of knowledge mobilisation activities. Overall, it is possible that the development of the field of knowledge mobilisation (and evaluating its impact) is stymied by the lack of a common language or common principles about what “should” be measured among those working in the field. A possible future avenue for research would be to use the responses to further explore why organisations answered in the way that they did for some of these indicators.
Systematic evaluation requires sufficient regulation, funding and guidance (Golden, 2020[3]). The intermediaries’ survey asked respondents which barriers they faced in evaluating their knowledge mobilisation activities. The results can be seen in Figure 6.3 below.
Note: Barriers are listed in descending order by percentage of respondents selecting “to a large extent” plus “to some extent”.
Source: OECD Survey of Knowledge Mobilisation in Education data, 2023.
Barriers and challenges
Lack of money and time are major barriers to evaluation, but not the only ones
Lack of funding and lack of time were more often seen as the biggest barriers. Only a minority of respondents saw a lack of skills or knowledge in evaluation as a barrier. This is worth remarking on, since there is relative consensus in the research literature that measuring the impact of knowledge mobilisation initiatives is methodologically very challenging (see (Davies, Powell and Nutley, 2015[14]; Ziam et al., 2024[15])). Either this is not the case for the respondents, or respondents did not have complex or long-term social impact measures in mind when considering this question. Incentives to evaluate do not seem to be a problem for around half of respondents, but for many others they were a challenge. Those who do not carry out any evaluation (n = 57) and those who evaluate a lot/systematically (n = 64) do not report markedly different barriers. The one exception relates to skills and knowledge: those who do not carry out evaluations tend to report that low levels of skills and knowledge are more of a barrier.
The open-ended responses of the intermediaries’ survey provided some further insights into barriers to evaluation. For instance, in Japan, one respondent remarked that systematic evaluation beyond what is required by the monitoring authority is not part of the organisational culture. Another respondent from Norway noted that rigorous or systematic evaluation of research use is simply not part of their mandate. For some initiatives, it was simply seen as too soon to evaluate – either the organisation itself was newly established, or the project or programme was at an early stage of its work. This suggests that the picture on evaluating knowledge mobilisation could change rapidly within a few years, as these organisations and programmes progress in their activities.
Beyond evaluation, evaluative thinking is important for knowledge mobilisation
Evaluation is not just about performing resource-intensive methodologies, which might not be needed for every single initiative. It also means encouraging an organisational culture of evaluative thinking, which means having an organisational culture of continuous reflection and evidence-gathering, to foster a mindset of experimentation and calculated risk-taking (Earl and Timperley, 2015[16]). The open-ended responses of the intermediaries’ survey pointed to challenges around evaluation culture that are commonly reflected in the research literature. For instance, a lack of organisational awareness, leadership and prioritisation around evaluation was identified as a challenge by some intermediaries. While specific evaluation methodologies are often performed by experts (see for example the Education Endowment Foundation case study in this chapter), evaluative thinking more generally is something that everyone can be involved in. The OECD Handbook for Innovative Learning Environments (2017[17]) outlines several ways that evaluative thinking may be engaged. For instance:
by explicitly considering and/or revisiting the intention, roots, philosophy and expected impact of core activities;
by involving stakeholders in the process of critical thinking about their activities;
by developing a clear vision of the benefits of different evaluation questions and processes;
by collaboratively planning evidence-gathering methods in advance to emphasise factors beyond simply the feasibility of its collection.
Considering the above points can help apply evaluative thinking across an organisation to develop a shared understanding of whether the planned monitoring data is fit-for-purpose and how it might be to be interpreted.
Lacking comprehensive and robust evaluations, the next best information about the effectiveness of knowledge mobilisation activities is actors’ perceptions of their effectiveness. The intermediaries’ survey examined respondents’ perceptions of most and least impactful or effective knowledge mobilisation activities (see Figure 6.4 and Figure 6.5).
Note: Responses to intermediaries’ survey question: “From your perspective, which of the following types of activities have been the most effective/impactful in increasing policy makers'/practitioners' research engagement? Please respond with respect to your organisation’s knowledge mobilisation activities.” Respondents were able to select three “top 3” and three “bottom 3” activities.
Source: OECD Survey of Knowledge Mobilisation in Education data, 2023.
Respondents were able to select three “top 3” and three “bottom 3” activities they deemed to be effective or not. The options proposed were the activities that the respondent reported to carry out at least “to some extent”, when reporting about its knowledge mobilisation activities (see Chapter 3 for full list). These activities pertain to generating, synthesising and communicating research products; fostering relationships between actors; and supporting other organisations in reinforcing research use. This section uses the conceptual framework for knowledge mobilisation introduced in Chapter 1, and further elaborated in Chapter 3, to discuss different types of intermediary activities.
Intermediaries’ perceptions
Evidence generation and dissemination are perceived as the most impactful forms of knowledge mobilisation
Overall, among the knowledge mobilisation activities that they engage in, respondent organisations perceive conducting research and disseminating research as being the most effective. This is true also when disaggregating the data and comparing responses between: (1) groups of intermediaries that reported evaluating their knowledge mobilisation activities systematically with those that identified as evaluating these only “to some extent”; and (2) different types of organisations (see Figure 6.5 below). This may reflect their current portfolio of knowledge mobilisation activities, which for many actors consists of predominantly linear activities. It is possible that some respondents perceive other activities as effective but do not engage in these due to cost or practicalities. The view of many respondent intermediaries that linear mechanisms are the most effective (see Figure 6.4) is poorly aligned with the research literature on knowledge mobilisation that posits that linear mechanisms are required but not effective on their own (Campbell et al., 2017[18]; Gough, Thomas and Oliver, 2019[19]). It can also be that in many organisations, relational or systems mechanisms are rarely implemented alone, but are implemented in combination with, or as a follow-up to, linear ones. For example, a relational activity such as a workshop may be organised to disseminate research findings, where respondents may consider the main knowledge mobilisation activity to be the research produced and the workshop to be a complementary activity.
Partnership building is seen as impactful, but training researchers and intermediaries is not
Building partnerships to facilitate research use was considered impactful by many respondents. This suggests that relationship building is considered important, but still comparatively less impactful than evidence generation and dissemination. The literature nonetheless places emphasis on the importance of relational knowledge mobilisation activities (Best and Holmes, 2010[20]; Langer, Tripney and Gough, 2016[21]). Overall, this perhaps similarly suggests that intermediaries acknowledge the importance of relationship building but may still often view it as a secondary rather than a primary knowledge mobilisation activity.
Partnerships are also only effective if partners have the necessary competencies to engage in high-quality interactions about research and its use. However, training researchers and intermediaries (to develop these skills) was not seen as impactful. Analysis of the intermediaries’ survey in Chapter 3 also showed that few intermediaries are engaged in offering such learning opportunities to other actors in their systems. However, in the policy survey, over half of respondent systems perceived policy makers and practitioners to lack the skills needed to use research well (Hill and Torres, 2023[22]). This suggests that there is a discrepancy between perceived needs for capacity building and effective offers to address these. Yet, there is robust research evidence about effective forms of teachers’ continuous professional learning. One-off training sessions for individuals tend to be less effective, as opposed to needs-based, long term, collective professional development (Darling-Hammond, Hyler and Gardner, 2017[23]). It is possible that this evidence has not yet been transferred to knowledge mobilisation. The policy survey also showed that teachers lack learning opportunities to develop their research literacy and research use skills; the same might be true for policy makers. Capacity building can be embedded in long term organisational support (e.g. directly supporting schools/policy organisations to use research).
To ensure that partnership building is strategic, mapping research users and mobilisers would be a natural first step. This activity, however, was not seen as impactful. It might be that such a mapping activity is seen as a component of knowledge mobilisation, but is not in itself what respondents see as making an initiative impactful (see United Kingdom case study in Chapter 3).
Intermediaries do not see the impact of supporting knowledge mobilisation in organisations and the system
Mechanisms such as evaluating and co-ordinating knowledge mobilisation at the system level were frequently selected as low-impact knowledge mobilisation activities. In some cases, respondents might consider that certain activities are poorly matched to their organisation’s mission, profile and skillset. Even among organisations that evaluate their knowledge mobilisation at least to some extent, evaluation was not frequently selected as a high-impact activity. Evaluating initiatives in terms of their effectiveness was selected as a “top 3” effective activity by 10 respondent organisations, and a “bottom 3” effective activity by 33 respondents. Overall, respondents may be indicating that evaluation of knowledge mobilisation is less effective than other knowledge mobilisation activities – its effectiveness might be indirect by increasing the effectiveness of other activities. Inevitably, many intermediaries work in contexts of finite resources and have to make decisions on how to spend resources effectively. This may lead them to spend on knowledge mobilisation activities that they perceive to have direct impact, versus evaluation, which has longer-term gains.
What matters to research institutions is evidence generation
Research institutions (RIs) were one of the types of organisations most represented among respondents. Both RIs and other types of actors considered conducting primary research and research dissemination to be the most effective knowledge mobilisation activities (Figure 6.5). Among linear mechanisms, respondents from RIs considered maintaining an evidence repository, and conducting systematic reviews (19% of respondent RIs) to be among the lowest-impact activities. One possible interpretation is that organisations perceive the activities they most often conduct as being the most effective, and areas where they are less active as being less effective. They may undertake certain activities because they have a prior expectation that these activities are particularly effective, or consider that certain activities are more (or less) well-suited to them given their organisational profile. Activities that are resource-intensive, such as synthesising evidence in a systematic review, may also lead intermediaries to more critically assess whether such activities are impactful enough given the resources invested.
Respondents from RIs may also consider impact on their professional careers. Research has shown that there are insufficient career-related incentives for academic researchers to undertake impact-related activities such as user engagement and co-design of research agendas (Upton, Vallance and Goddard, 2014[24]), instead incentivising them to prioritise primary research and traditional dissemination in academic journals. Systematic reviews are also not always considered by academic researchers to be a “valid piece of research” when compared to primary research (Smith and Stewart, 2017[25]), while some models of policy impact have been criticised for inadvertently encouraging an individualistic approach wherein researchers primarily link impact to their own underpinning research rather than to a wider body of research knowledge or research syntheses (Boswell, Smith and Davies, 2022[26]). Yet, systematic reviews published in journals are often among the most highly-cited studies (Aksnes and Aagaard, 2021[27]; Chaignon, Docampo and Egret, 2023[28]). This implies that synthesis – while resource-intensive – has potential to be a high-impact activity in several ways if done well.
Note: The data show select knowledge mobilisation activities perceived by the respondents as being high impact, with data disaggregated for two organisation types. Responses to the intermediaries’ survey question: “From your perspective, which of the following types of activities have been the most effective/impactful in increasing policy makers'/practitioners' research engagement? Please respond with respect to your organisation’s knowledge mobilisation activities.” Respondents were able to select three “top 3” activities.
Source: OECD Survey of Knowledge Mobilisation in Education data, 2023.
Impact is based on a combination of knowledge mobilisation activities
Overall, the activities intermediaries view as the most impactful are often primarily linear. The data presented also prompt the question of whether there is an ideal balance between linear, relational and systems mechanisms in a system, and whether the balance between these types of knowledge mobilisation activities needs to be perfectly equal. It may indeed be effective for the majority of intermediaries in a system to be performing linear knowledge mobilisation mechanisms, with a smaller number undertaking relational activities, and an even smaller number undertaking systems activities. For example, it may be only one intermediary (or a small number of intermediaries) co-ordinating knowledge mobilisation at the system level across different actors. In addition, it appears that effective initiatives are often not focused on single activities, but are complementary to each other (for example, relationship building is combined with capacity building). An important limitation of the survey question explored in this section is that it is not able to capture the joint effectiveness of some activities.
High-quality, in-depth evaluation activities are conceptually and methodologically challenging. This is particularly true for measuring impact at the level of wider social outcomes, both in education and other fields. Oliver et al. (2022[5]) were unable to find a single robust evaluation which convincingly demonstrated evidence use or the impacts of evidence use on social outcomes. A big reason for this is simply that measuring these outcomes is difficult. For example, in education practice, whether or not research use has increased could be a measure of success, but such “natural experiments” would provide correlational, rather than causal, data (Gough, Sharples and Maidment, 2022[11]). When it comes to policy, it is also difficult to disentangle the impacts of a specific policy from other policies which operate in parallel, or which would have occurred as a natural consequence of the contextual situation (Golden, 2020[3]). Although there are limits to what can be expected by applying current evaluation methods to monitoring data in terms of causal links, this does not mean they are not useful. Reflecting on how impact might be achieved in an intervention by developing a theory of change1 can still be an instructive exercise. Such an approach could rely on realist evaluation methods (e.g. (Pawson and Tilley, 1997[29])) to construct a theory-based approach to understanding and investigating impact. The ESRC Education Research Programme described in Chapter 3 is an example of how deep engagement with an initiative’s theory of change to plan impact supports the success of the initiative. In that sense, these long-term complex impact indicators can offer formative rather than summative insights.
Strengthening evaluation of knowledge mobilisation is about more than just increasing measurement and quantification of impacts using resource-intensive methods (although this is essential for some initiatives), it is about building an evaluation mindset within education organisations. This requires training for intermediaries and clarifying key evaluation terms and enhancing shared understanding of what is meant by evaluation, so that a culture of evaluative thinking can become better established across organisations. It also involves learning about the processes of continuous reflection and evidence-gathering, which can foster a mindset of experimentation and calculated risk-taking. Eventually, this kind of knowledge will help organisations who have to make difficult decisions about which approach to take for a given activity, for instance choosing between self-evaluation and external evaluation. Such structured and reflective approaches could be more commonplace, and the training needed for intermediaries appears to be lacking across OECD systems generally (see Chapter 3) although there are isolated examples. One such example is the European Commission’s Learning Lab on Investing in Quality Education and Training (European Commission, n.d.[30]). This lab offers capacity building on policy evaluation, primarily to public intermediaries, such as departments within ministries of education. The literature offers insights into other aspects that intermediaries might consider to promote a mindset of evaluative thinking in their organisations and work towards achieving a better shared understanding among themselves of purposes, methods and measures of evaluation. Table 6.1 outlines a few potential considerations.
Note: A Theory of Change is an evidence-based rationale that builds a causal analysis to explain how a set of activities and outputs are expected to lead to a specific change.
Source: Adapted from Gough, Sharples and Maidment (2022[11]), “Evidence on evidence-informed policy and practice”, in Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, OECD Publishing, Paris, https://doi.org/10.1787/5ee7de7c-en; Rickinson et al. (2022[31]), Rickinson, M. et al. (2022), “Using research well in educational practice”, in Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, OECD Publishing, Paris, https://doi.org/10.1787/65aac033-en; Golden (2020[3]), “Education policy evaluation: Surveying the OECD landscape”, OECD Education Working Papers, No. 236, OECD Publishing, Paris, https://doi.org/10.1787/9f127490-en.
Focusing on the incentives for conducting evaluation is crucial when trying to strengthen approaches. For some intermediaries, evaluation may simply be seen as an accountability mechanism, such as when it is required by a funder (as found by (Oliver et al., 2022[5])). Evaluation done for accountability purposes risks becoming a tick-box exercise, where monitoring data is not used for critical reflection and findings are not integrated into organisational thinking to improve knowledge mobilisation practices. This highlights that it is crucial to capture what people understand by evaluation; what they do it for; and who they do it for. In particular, understanding if and how evaluation outcomes are subsequently used by the intermediary would help to illustrate these motivations. In many cases, lack of funding may prevent organisations from undertaking evaluation of their activities, even when evaluation is otherwise recognised and acknowledged to be valuable. In which case, increasing the percentage of funding earmarked specifically for evaluation may ease some of these perceived pressures, as long as it does not draw too many resources away from the implementation of a given activity. To date, little research has been done on how evaluation data and evidence is used to inform education policy, if it is at all (see Chapter 2). Similar research focused on intermediaries is likely to be equally, if not more, rare.
A further important consideration is whether these activities can be implemented in a way that goes beyond increasing the effectiveness of intermediaries’ knowledge mobilisation activities and could also help to build a cumulative evidence base on knowledge mobilisation. This chapter now turns to two case studies of such implementation to understand how this has been done.
The German Institute for Adult Education (Deutsches Institut für Erwachsenenbildung (DIE)) is a research institution of the Leibniz Association located in Bonn, Germany. The Leibniz Association is a union of German non-university research institutes from various disciplines. The institute is funded by the Federal Government and German States to conduct research in the field of adult teaching and learning, continuing education programmes and institutions, as well as on the political and institutional contexts of lifelong learning. This case study is informed by a group interview carried out by the OECD on 20 June 2024 and analyses the role of evaluation in the knowledge mobilisation activities of the DIE and, in particular, evaluation of the EULE (Entwicklung einer webbasierten Lernumgebung für Weiterbildung, Kompetenzerwerb und Professionalisierung von Lehrenden der Erwachsenenbildung – Open web-based learning space for continuing professional development of adult educators) learning platform.
Evaluation of transfer activities at the DIE
As seen in Figure 6.1, German respondents often reported that they did not evaluate their knowledge mobilisation initiatives. The DIE, however, did report that it occasionally carries out evaluations of its work. Unlike many similar institutes in Germany, the DIE has a dedicated knowledge transfer department, which co-ordinates monitoring, reflection and conceptual work on knowledge mobilisation through a transfer strategy (Table 6.2). The strategy itself contains a logic model,2 commonly applied in monitoring and evaluation. This model allows the strategy to differentiate between outputs (performance), outcomes (completed transfer) and impact (successful transfer) indicators. In doing so, the strategy acknowledges that access to DIE materials by policy makers and practitioners does not necessarily equal use of these materials.
As outlined in the strategy, the DIE systematically collects quantitative data on performance and use of resources. When it comes to impact data, this is mainly qualitative and collected through examples from users. For instance, different departments in the DIE maintain contact with practitioners and collect data and good practices of how knowledge learned through DIE activities is applied in practitioners’ contexts. For the most part, the Knowledge Transfer department conducts internal monitoring and evaluation of DIE activities and does not publish this information. However, some departments do publish studies on knowledge mobilisation topics that have evaluative value for DIE. For example, the System & Politics department has carried out a peer-reviewed study analysing perception and use of data from continuing education statistics, including DIE statistics, by political actors (Widany and Gerhards, 2022[32]). The published studies are not evaluations of the specific impact of the DIE per se, rather they tackle knowledge mobilisation from the broader perspective, drawing on the experiences of the DIE.
Note: The DIE uses the term “knowledge transfer” to refer to its activities, although their transfer concept goes beyond linear dissemination of research and more closely resembles the concept of knowledge mobilisation as outlined in the research literature.
Source: Adapted from DIE (2024[33]), DIE (2024), Knowledge Transfer at the DIE, Deutsches Institut für Erwachsenenbildung (German Institute for Adult Education), Bonn, https://www.die-bonn.de/docs/Knowledge%20Transfer_2024_07.pdf (accessed on 25 October 2024).
Evaluation and impact through the EULE platform
The EULE is a resource of DIE. It is an open, free self-study platform funded by the German Federal Ministry of Education and Research (BMBF) to support the professionalisation of adult educators through 75 learning pathways. The learning pathways are constructed using research literature, and cover topics such as student feedback and motivation, digital education, time management and lesson design. Practitioners can use the EULE platform to gain relevant and research-based knowledge for their professional development.
The EULE learning area was developed using an iterative, design-based approach. This was done with the aim of continuously monitoring the quality of the content and technical implementation along three research strands:
Formative evaluation through regular surveys and usability studies: Emphasis was placed on achieving satisfaction among the target group (adult education teachers) with regards to the topics, presentation of content and relevance of the guidance provided.
Impact-testing: Through three intervention studies, after completion of the learning paths, the knowledge, skills and use of competencies in everyday teaching was examined.
Transfer-securing implementation studies: Surveys and user tests draw in constant feedback from continuing education providers, to monitor how they integrate the learning pathways into the portfolio of training they provide to teachers of adult education. This allows needs-based support to supplement and enrich the courses provided by the continuing education providers.
These three strands were complemented with an independent evaluation of the EULE, which was carried out in 2018-2019 by a researcher in the adult education/further education department of the Institute for Educational Science at the Eberhard Karls University of Tübingen, and published in the professional publication Adult Education (Erwachsenenbildung) (Schöb, 2020[34]). The evaluation had a small but approximately representative sample of 34 German adult educators who used the platform, and combined data with qualitative expert interviews with six individuals (Box 6.1).
The evaluation focused on one EULE learning pathway, “Enabling self-directed learning”, and took the form of a before-and-after study, which examined the effect of the EULE on the teaching practices of the participants in terms of their ability to use the knowledge contained in the platform learning pathway.
To measure change in teaching practice:
Participants were video-recorded during their teaching time.
Teaching actions were analysed using an observation sheet that helped estimate the situational appropriateness of the actions (on a four-stage scale from 1 = appropriate to 4 = not appropriate).
The evaluation concluded that the course had brought about a change in everyday teaching. Study participants succeeded in linking the training content directly with their teaching practice and integrating what they had learned into their actions. The programme was found to be suitable to the learning requirements of teachers.
Weaker areas included a lack of socially shared learning in the EULE learning area, which was focused on individual study. Furthermore, the evaluation provided only a limited understanding of how the effects on the participating teachers are related to their educational backgrounds and the length of their teaching career.
Source: Schöb (2020[34]), “Transfer effect as a quality feature of digital learning opportunities for adult educators: A study on measurability and relevance [Transferwirkung als Qualitätsmerkmal digitaler Lernangebote für ErwachsenenbildnerInnen]. Eine Untersuchung zur Messbarkeit”, Magazine erwachsenenbildung.at. The specialist medium for research, practice and discourse [Das Fachmedium für Forschung, Praxis und Diskurs] 40, https://erwachsenenbildung.at/magazin/20-40/meb20-40.pdf.
Challenges and limitations
Through their approach, the DIE has been able to identify several challenges and limitations to knowledge mobilisation and its evaluation. A key challenge involves the difficulty of acquiring new, relevant target groups of potential research users beyond those that are already well-known. For example, there are many companies providing vocational and continuing education resources and materials, but they are for the most part not involved in DIEs network. This makes it difficult to gather information on how and why these companies may be engaging with DIE outputs.
Furthermore, problems relating to the low levels of generalisability of much education research makes designing a knowledge mobilisation platform such as EULE more challenging. The learning materials and resources need to be very tailored, which makes it resource-intensive to produce content for different parts of adult education. Part of the solution may be to plan in a way that avoids an artificial distinction between research and knowledge mobilisation. The DIE calls this planning approach “application orientated basic research” (DIE, 2024[33]). Over time, this may help orientate the DIE towards carrying out more high-quality control studies that aim to understand how the materials are used by different types of groups in different settings.
At the level of the academic system, a major challenge to the work of the DIE is the emphasis placed by the academic career model on peer-reviewed articles and highly ranked academic journals, to the detriment of other forms of research impact. This academic career model produces a very high volume of research content from a large number of actors, making it difficult to know how to focus potential research users’ attention on any specific output. To overcome this challenge, the DIE has begun to use review methods such as meta-analyses, critical reviews, systematic reviews and rapid reviews, although the resources required to conduct these are significant. Additionally, legal structures in German academia place a time limit on how long an academic is permitted to conduct research without a long-term contract. Those with short-term contracts are mostly part-time and once their contracts finish it is very difficult to get a long-term contract. This not only means that accumulating and sustaining research knowledge is difficult but also that training researchers in evaluative thinking regarding their knowledge mobilisation requires significant resources, as there is often much staff turnover.
Although the DIE collects qualitative evaluation data through selected examples of impact, these are self-reported and not compared with any counterfactual. In the aforementioned evaluation of EULE, the methodology did not rely on self-reported data, but the method used was time‑consuming and resource-intensive to develop. For example, designing the evaluation protocol and watching all the videos of the practitioners who participated in the external evaluation (Schöb, 2020[34]) to see how they were implementing the content. To streamline this process, it would be useful if there was a more developed evidence base on evaluating knowledge mobilisation.
The indicators systematically collected by the DIE are mainly in line with what is reported in the chapter (i.e. they focus on reach and access). Although the positive independent evaluation captured behavioural and skills indicators, these were specific to one learning pathway in the EULE evaluation and are not systematically collected at the organisational level. Behavioural indicators are receiving increasing attention, but still require significant methodological development.
Expanding the approach: Lessons for others
A lack of time and funding for monitoring and evaluation can be overcome by ensuring that responsibility for the task is spread evenly across departments. While the DIE has a dedicated knowledge mobilisation unit, which serves as a focal point for strategic evaluation planning to maximise the use of resources, the responsibility remains shared across the organisation. Having such dedicated knowledge mobilisation units can be hugely beneficial for research institutes, but needs to be combined with a transversal approach where all knowledge-producing departments are aware of and strategically use their mobilisation functions.
When it comes to evaluating impact, this DIE case study shows that, although it may not be feasible for intermediary organisations to gather behavioural indicators on the same scale as reach and access indicators, there is still value in using more experimental and quasi-experimental control studies to understand how knowledge mobilisation materials are used from different types of groups in different settings.
The Education Endowment Foundation (EEF) is a UK charity established in 2011 that aims to improve the educational attainment of disadvantaged pupils. Its activities include reviewing and summarising the best available evidence on teaching and learning, and presenting this in accessible formats; funding independent evaluations of programmes; and supporting policy and practice communities to make use of research evidence.
In 2015, the EEF launched a national campaign of presentations; social, print and broadcast media; direct communications to schools; and meetings with policy stakeholders to scale up the use of the guidance report: “Making best use of Teaching Assistants”. This report gave seven actionable recommendations for schools (Education Endowment Foundation, 2018[35]) concerning the deployment of Teaching Assistants (TAs). TAs comprise over one-quarter of the workforce in mainstream schools in England. They are often used as an informal instructional resource for pupils most in need of additional support, but this kind of deployment is not associated with improvement in student learning and can even be harmful. Some of the EEF’s early trials showed positive impacts when TAs delivered structured interventions. This suggested that changing the teaching practices of TAs could go a long way to improving outcomes with limited additional cost. The study found that decisions made by school leaders and teachers regarding the deployment and preparation of TAs had significant influence over the impact of TA support. Ultimately, these findings led to the decision to produce the aforementioned guidance report.
In addition to the nation-wide campaign, the EEF ran two regional scale-up pilot campaigns in England (South & West Yorkshire and Lincolnshire), which aimed to support the development of evidence-informed TA practices at a regional level. The South & West Yorkshire campaign involved commissioning advocacy providers to deliver training and support for schools during the 2015/16 academic year, and was the subject of an impact evaluation. An impact evaluation is a study that provides information about the observed changes or “impacts” produced by an intervention and aims to identify the cause(s) of those changes. Overall, the cumulative cost for the South & West Yorkshire campaign evaluation was just under GBP 100 000. This included set up and planning; piloting of incentivising guidance materials take-up; implementation and reporting costs.
Impact evaluation of the South & West Yorkshire pilot programme
Although the preparatory and follow-up work took around three years, the bulk of the South & West Yorkshire pilot programme ran for around one year. The scale-up pilots took a co-ordinated approach with the EEF working as a system-level broker whose role was to identify, fund and train a diverse group of influential local education advocacy providers (e.g. universities, schools, teacher trainers). These advocacy providers acted as local intermediaries to support schools to embed the TA evidence synthesis into their teaching practices. The campaign worked with around 480 schools and recruited advocacy providers in seven local authorities.
Given that the treatment group was non-random, the effects of the intervention could not be analysed using a randomised controlled trial. Instead, the EEF used “synthetic control” methods, which were pioneered to examine the effects of interventions affecting large geographic areas. The synthetic control method was used to construct a control group by weighting other aggregate units (different local authorities) for the treated group (South & West Yorkshire) over the pre-treatment phase (all years up to 2014/15). The difference between the synthetic control and the treatment group in the post-treatment phase represents the estimated impact of the intervention.
The impact evaluation in South & West Yorkshire found that the pilot had a positive impact on educational attainment in English at primary level (ISCED 1). No impact was found on attainment in mathematics. There is some limited evidence from the implementation and process evaluations that the schools participating in both pilots changed their practices to align more closely with the EEF recommendations. The implications of these findings have now been adopted as part of the EEF’s regional strategy.
The evaluation revealed several important components for effectiveness:
As a known and trusted actor in the field, the EEF’s teaching assistant guidance and recommendations were perceived to be credible and convincing. The implication for new intermediaries entering the field is that it is necessary to spend time building trust among stakeholders, and to consistently apply rigorous methods in order to produce work that is recognised as being of a high standard.
If evidence is synthesised in an accessible and user-friendly way, it is much easier for training organisations to package it into training activities. This suggests that training institutes need to be considered as a major audience for intermediary work.
System-level brokerage activities that are perceived to be effective include: providing hands-on steering to ensure fidelity to the evidence and advice on scale-up approaches; a balance of responsive support and challenge; facilitating linkages to professionals and academics; and building a bank of resources. Intermediaries should reflect on what the most impactful division of labour would be when it comes to system-level brokers, who might be in a better co-ordinating position, and local brokers, who may better understand actors’ needs.
Regarding regional/local brokerage, a wide range of activities were perceived to be effective, including: developing and delivering training and resources that have a clear purpose and focus on the research object (in this case, the guidance and recommendations); taking account of participants' prior knowledge and changes already implemented in their schools; enabling schools to contextualise the guidance, share ideas, consider issues, reflect, plan for implementation, and evaluate practice in discussion with other schools. Intermediaries should be open to experimenting with various approaches based on local feedback.
The evidence of promise in engaging local and regional actors in supporting the contextual application of rigorous evidence led to the EEF funding a network of research schools to build more long‑standing local capacity for brokering knowledge in a contextually appropriate way.
Challenges and limitations
Although a synthetic control group is an innovative indicator (see Box 6.2), the method is still relatively under-researched and does not have the same level of robustness compared to a true randomised control trial. Any regional evaluation work risks additional confounders through other regional initiatives or approaches, which explains the variation in results over and above the campaign approach that was tested. Another limitation is spill-over effects of the campaign. The recommendations from the campaign were available nationally through EEF guidance reports, so many of the comparison schools may also have made changes to their practices. The evaluation is only able to measure the additional benefit of advocates disseminating the findings of the work on TA deployment in the region.
Finally, the limited time period of the evaluation may inhibit measuring long-term impacts, such as those on students’ educational attainment, and may underestimate some behavioural changes (e.g. in teaching practices) that may also take more time to occur.
Expanding the approach: Lessons for others
This EEF case study carries lessons for others and is noteworthy for three key reasons. First, it takes an approach of methodological rigour and transparency of results and applies this to a knowledge mobilisation intervention. In doing so, it illustrates the benefits of “practising what you preach”, which can be brought to the work of intermediaries. For instance, undertaking the pilot evaluations has shown that linear methods (i.e. producing a guidance report) are effective when coupled with relational and systematic capacity building.
Secondly, it reveals a number of important insights that can help build a cumulative knowledge base upon which intermediaries can draw for their knowledge mobilisation work. Namely, implementation in schools appears to be more effective when there is senior leader understanding; a commitment to change; change agents that are capable, enthusiastic and enabled to effect change; and a clear process for implementation with time allocated for each activity. School leadership needs to be involved at key points.
Finally, the EEF case pioneers the use of synthetic control groups to estimate impact over large geographical areas in natural experiments, explicitly targeting one of the biggest methodological difficulties of evaluating knowledge mobilisation interventions. This synthetic control group method could be applied in other knowledge mobilisation evaluation contexts.
Developing a common understanding of what constitutes evaluation, and how to measure impact in some areas, is required for new knowledge mobilisation initiatives to build on knowledge gained from existing ones. This is especially important in the context of rapid expansion of intermediaries active in systems and a proliferation of knowledge mobilisation activities in education. An OECD review of the knowledge mobilisation literature found that a lack of evaluation continues to hamper the effectiveness of knowledge intermediary practices in education (Torres and Steponavičius, 2022[12]). The gap is especially worrisome for the perceived legitimacy of the work of formal intermediaries, since these organisations are often positioned as standard setters within an education system when it comes to evidence-informed policy and practice. It is therefore necessary for their credibility that they systematically and routinely investigate if the knowledge mobilisation activities they undertake are indeed effective (Gough, Sharples and Maidment, 2022[11]).
At present, there is a tendency among intermediaries to focus on self-reported indicators and easily available data. Although this is a necessary starting point, the inherent limitations of these indicators may negatively affect the development, support and perception of their work in the longer term (Torres and Steponavičius, 2022[12]). Furthermore, gathering monitoring indicators, although an important first step, only becomes evaluation when there is critical reflection on what is measured. This is where engagement with key methodologies, such as impact evaluation and realist evaluation, is required. Further research investigating the use and respective value of these methods would be illuminating.
Developing effective evaluation of knowledge mobilisation initiatives is not just about carrying out more and better evaluations (although, this is also needed). It also involves developing the capacity for evaluative thinking (i.e. a mindset of professional collective reflection on tasks and initiatives) across intermediaries and the system. Supportive leadership and training for intermediaries, at the organisational level but also at the system level, can help build awareness about the importance of both evaluation and evaluative thinking in contributing to wider, system-level improvements. Leadership can help to ensure there are more resources for evaluation (time and funding), and that there is better awareness of principles of evaluation by allocating staff time to training and capacity building. Leadership is also important from the perspective of deciding how much time and resources to dedicate to a given evaluation activity, and ensuring this is proportionate to the time and resources invested in the initiative itself.
In terms of a future research agenda, it would be helpful to dig deeper into what intermediaries understand by “evaluation” in different categories, and what would motivate them to conduct evaluations. A mandate to conduct systematic or rigorous evaluation is a good baseline – but is still missing in many organisations and systems. Beyond this, good quality evaluation is arguably more likely to arise from motivation that goes beyond accountability mechanisms (e.g. evaluation because it is required by a funder). There is also scope for much more conceptual and empirical experimentation when it comes to developing measurements for complex social indicators that show the impact of intermediary activities. The work that has been done suggests measuring these indicators may never be systematic in the same way that measuring simpler indicators can be. However, such work is still needed to improve existing practice and build the evidence base on which new or emerging intermediaries can draw.
For intermediary organisations
How does your organisation currently define and approach the evaluation of its knowledge mobilisation activities? What are the key challenges in your existing evaluation processes?
What are the primary barriers your organisation faces in evaluating knowledge mobilisation activities (e.g. lack of funding, time, skills)? What strategies could be employed to overcome these barriers?
What is the mindset and skillset in your organisation regarding conducting diverse types of evaluations? (If unknown, how might you address potential training needs?)
How does your organisation use evaluation findings? How does it disseminate them (e.g. to external audiences)?
What role could external partnerships play in the process of building capacity and knowledge of evaluation processes in your organisation?
For policy makers
What types of evaluation are commonly undertaken to evaluate knowledge mobilisation in your system? How are these co-ordinated (who selects initiatives to evaluate, who funds them, how is evaluation incentivised)?
How is evaluation data and evidence used in policy-making processes?
What steps could be taken to cultivate a stronger culture of evaluative thinking across the system? How can leadership support the integration of systematic evaluation practices?
What measures can be implemented to support both the quantity and quality of evaluations that are undertaken by intermediaries in your system (beyond accountability)?
How does your ministry/department of education evaluate its use of research evidence?
How can your ministry/department of education allocate funding to evaluation in a way that promotes a cumulative research base for knowledge mobilisation?
References
[27] Aksnes, D. and K. Aagaard (2021), “Lone Geniuses or One among Many? An Explorative Study of Contemporary Highly Cited Researchers”, Journal of Data and Information Science, Vol. 6/2, pp. 41-66, https://doi.org/10.2478/jdis-2021-0019.
[9] Berger, V. (ed.) (2021), “Neglect of publication bias compromises meta-analyses of educational research”, PLOS ONE, Vol. 16/6, p. e0252415, https://doi.org/10.1371/journal.pone.0252415.
[20] Best, A. and B. Holmes (2010), “Systems thinking, knowledge and action: towards better models and methods”, Evidence & Policy, Vol. 6/2, pp. 145-159, https://doi.org/10.1332/174426410x502284.
[8] Boaz, A., K. Oliver and A. Hopkins (2022), “Linking research, policy and practice: Learning from other sectors”, in Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, OECD Publishing, Paris, https://doi.org/10.1787/70c657bc-en.
[26] Boswell, C., K. Smith and C. Davies (2022), Promoting Ethical and Effective Policy Engagement in the Higher Education Sector, https://rse.org.uk/.
[18] Campbell, C. et al. (2017), “Developing a knowledge network for applied education research to mobilise evidence in and for educational practice”, Educational Research, Vol. 59/2, pp. 209-227, https://doi.org/10.1080/00131881.2017.1310364.
[28] Chaignon, L., D. Docampo and D. Egret (2023), “In search of a scientific elite: highly cited researchers (HCR) in France”, Scientometrics, Vol. 128/10, pp. 5801-5827, https://doi.org/10.1007/s11192-023-04805-3.
[23] Darling-Hammond, L., M. Hyler and M. Gardner (2017), Effective Teacher Professional Development, https://eric.ed.gov/?id=ED606743.
[14] Davies, H., A. Powell and S. Nutley (2015), “Mobilising knowledge to improve UK health care: learning from other countries and other sectors – a multimethod mapping study”, Health Services and Delivery Research, Vol. 3/27, pp. 1-190, https://doi.org/10.3310/hsdr03270.
[33] DIE (2024), Knowledge Transfer at the DIE, Deutsches Institut für Erwachsenenbildung (German Institute for Adult Education), Bonn, https://www.die-bonn.de/docs/Knowledge%20Transfer_2024_07.pdf (accessed on 25 October 2024).
[16] Earl, L. and H. Timperley (2015), “Evaluative thinking for successful educational innovation”, OECD Education Working Papers, No. 122, OECD Publishing, Paris, https://doi.org/10.1787/5jrxtk1jtdwf-en.
[35] Education Endowment Foundation (2018), Making Best Use of Teaching Assistants, EEF, https://educationendowmentfoundation.org.uk/education-evidence/guidance-reports/teaching-assistants (accessed on 10 September 2024).
[30] European Commission (n.d.), Learning Lab on Investing in Quality Education and Training, https://education.ec.europa.eu/focus-topics/improving-quality/learning-lab (accessed on 4 November 2024).
[3] Golden, G. (2020), “Education policy evaluation: Surveying the OECD landscape”, OECD Education Working Papers, No. 236, OECD Publishing, Paris, https://doi.org/10.1787/9f127490-en.
[11] Gough, D., J. Sharples and C. Maidment (2022), “Evidence on evidence-informed policy and practice”, in Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, OECD Publishing, Paris, https://doi.org/10.1787/5ee7de7c-en.
[19] Gough, D., J. Thomas and S. Oliver (2019), “Clarifying differences between reviews within evidence ecosystems”, Systematic Reviews, Vol. 8/1, https://doi.org/10.1186/s13643-019-1089-2.
[10] He, J. et al. (2014), “Socially Desirable Responding”, Cross-Cultural Research, Vol. 49/3, pp. 227-249, https://doi.org/10.1177/1069397114552781.
[22] Hill, J. and J. Torres (2023), “Terms of engagement: Where learning meets culture”, in Who Really Cares about Using Education Research in Policy and Practice?: Developing a Culture of Research Engagement, OECD Publishing, Paris, https://doi.org/10.1787/bfd04a1f-en.
[4] Kislov, R. et al. (2018), “Learning from the emergence of NIHR Collaborations for Leadership in Applied Health Research and Care (CLAHRCs): a systematic review of evaluations”, Implementation Science, Vol. 13/1, https://doi.org/10.1186/s13012-018-0805-y.
[21] Langer, L., J. Tripney and D. Gough (2016), The Science of Using Science: Researching the Use of Research Evidence in Decision-Making, https://eppi.ioe.ac.uk/cms/Portals/0/PDF%20reviews%20and%20summaries/Science%202016%20Langer%20report.pdf.
[13] Lavis, J. et al. (2003), “How Can Research Organizations More Effectively Transfer Research Knowledge to Decision Makers?”, The Milbank Quarterly, Vol. 81/2, pp. 221-248, https://doi.org/10.1111/1468-0009.t01-1-00052.
[7] Newman, M. and D. Gough (2019), “Systematic Reviews in Educational Research: Methodology, Perspectives and Application”, in Systematic Reviews in Educational Research, Springer Fachmedien Wiesbaden, Wiesbaden, https://doi.org/10.1007/978-3-658-27602-7_1.
[6] OECD (2022), Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/d7ff793d-en.
[1] OECD (2020), Improving Governance with Policy Evaluation: Lessons From Country Experiences, OECD Public Governance Reviews, OECD Publishing, Paris, https://doi.org/10.1787/89b1577d-en.
[17] OECD (2017), The OECD Handbook for Innovative Learning Environments, Educational Research and Innovation, OECD Publishing, Paris, https://doi.org/10.1787/9789264277274-en.
[2] OECD (1997), Education Policy Analysis 1997, OECD Publishing, Paris, https://doi.org/10.1787/epa-1997-en.
[5] Oliver, K. et al. (2022), “What works to promote research-policy engagement?”, Evidence & Policy, Vol. 18/4, pp. 691-713, https://doi.org/10.1332/174426421x16420918447616.
[29] Pawson, R. and N. Tilley (1997), “An Introduction to Scientific Realist Evaluation”, in Evaluation for the 21st Century: A Handbook, SAGE Publications, Inc., 2455 Teller Road, Thousand Oaks California 91320 United States , https://doi.org/10.4135/9781483348896.n29.
[31] Rickinson, M. et al. (2022), “Using research well in educational practice”, in Who Cares about Using Education Research in Policy and Practice?: Strengthening Research Engagement, OECD Publishing, Paris, https://doi.org/10.1787/65aac033-en.
[34] Schöb, S. (2020), “Transfer effect as a quality feature of digital learning opportunities for adult educators: A study on measurability and relevance [Transferwirkung als Qualitätsmerkmal digitaler Lernangebote für ErwachsenenbildnerInnen. Eine Untersuchung zur Messbarkeit”, Magazine erwachsenenbildung.at. The specialist medium for research, practice and discourse [Das Fachmedium für Forschung, Praxis und Diskurs] 40, https://erwachsenenbildung.at/magazin/20-40/meb20-40.pdf.
[25] Smith, K. and E. Stewart (2017), “We Need to Talk about Impact: Why Social Policy Academics need to Engage with the UK’s Research Impact Agenda”, Journal of Social Policy, Vol. 46/1, pp. 109-127, https://doi.org/10.1017/s0047279416000283.
[12] Torres, J. and M. Steponavičius (2022), “More than just a go-between: The role of intermediaries in knowledge mobilisation”, OECD Education Working Papers, No. 285, OECD Publishing, Paris, https://doi.org/10.1787/aa29cfd3-en.
[24] Upton, S., P. Vallance and J. Goddard (2014), “From outcomes to process: evidence for a new approach to research impact assessment”, Research Evaluation, Vol. 23/4, pp. 352-365, https://doi.org/10.1093/reseval/rvu021.
[32] Widany, S. and P. Gerhards (2022), ““Wer braucht diese Äpfel und Birnen?” Wahrnehmung und Nutzung von Daten der Weiterbildungsstatistik durch weiterbildungspolitische Akteure [“Who needs these apples and pears?” Perception and use of data from continuing education statistics”, Zeitschrift für Bildungsforschung, Vol. 12/1, pp. 145-163, https://doi.org/10.1007/s35834-022-00339-5.
[15] Ziam, S. et al. (2024), “A scoping review of theories, models and frameworks used or proposed to evaluate knowledge mobilization strategies”, Health Research Policy and Systems, Vol. 22/1, https://doi.org/10.1186/s12961-023-01090-7.
← 1. A Theory of Change is an evidence-based rationale that builds a causal analysis to explain how a set of activities and outputs are expected to lead to a specific change.