Promoting Education Decision Makers' Use of Evidence in Austria

Executive Summary

Introduction

The 2017 Education Reform Act in Austria gives education governance a regional focus and moves key decisions closer to schools and students. An important aspect, therefore, is to strengthen the capacity to use evidence for decisions, including in teaching and quality assurance. In support of this, Austria worked with the OECD to conduct a self-assessment exercise on evidence use among key decision makers at the federal and provincial levels (federal ministry and education directorate executives), regional level (school quality managers) and school leaders. Decision makers completed an online survey covering five areas that promote the capability, motivation, and opportunity to use evidence in decision making:

• The skills to access and make sense of evidence.

• Making relevant evidence conveniently available to decision makers.

• Fostering the organisational processes and structures that encourage use of evidence.

• Fostering the exchange among decision makers and their exchange with evidence producers and

• Promoting use of evidence as a principle of good decision making, building a shared understanding on what constitutes fit-for-purpose evidence, as well as how and when evidence should be used.

The results and analysis in this report act as a “thermometer" gauging areas for further investigation and informing thinking about possible next practices. It is a conversation starter rather than a comparative evaluation of practices. As a first step, this report will be discussed at a workshop with representative stakeholders from each province and decision making level.

Main findings

Some provinces systematically report efforts to promote the use of evidence. In these provinces, school quality managers consistently report frequent exchanges with colleagues and evidence providers to improve evidence quality and preparation. Across schools, school leaders systematically report that they exchange with school quality managers and peers about methods and experiences working with evidence; and that these exchanges are supported organisationally, such as through requisite time or staff resources. In other provinces, such efforts are emergent.

School quality managers play a pivotal role in fostering the use of evidence. More than 90% of school quality managers report engaging in efforts to raise awareness of the merit and importance of using evidence in decision making. While such efforts focus on school leaders, six in ten school quality managers go beyond this and directly engage teaching staff. School quality managers, although largely content with the available evidence, express motivation to be more involved in preparing and providing evidence, including greater interaction with key evidence providers.

Schools are important evidence producers and have some key organisational processes in place that can encourage the use of evidence. The vast majority of school leaders report producing and using school-level evidence from internal evaluations and standardised student testing. Many school leaders highlighted concrete efforts at their schools to prepare evidence, which can provide a starting point for further dialogue and investigation. There is an emphasis on internal exchange to increase clarity around school decisions and decision-making processes. School leaders report inviting diverse perspectives, for instance from teachers, parents, and students, mainly to help develop classroom teaching.

Evidence provided to schools is not always adequately prepared for their work. Only 50% of school quality managers agree that evidence is largely or very adequate for the work of schools (regardless of the evidence provider). Compared to other evidence providers, schools report that the federal ministry and education directorates are less concerned about preparing evidence in a user-friendly way (only 41% report they are largely or very interested in doing so).

Possible next steps

Considering the changed responsibilities brought about by the 2017 Education Reform Act and building on existing efforts and available opportunities, there are a number of possible inroads for Austria to consider in its continuing efforts to promote the use of evidence.

Structuring collegial exchanges around explicit purposes, building on existing habits. Results show that interactions with peers, for both school leaders and school quality managers, are important ways of exchanging experiences and methods of evidence use. Where evidence use is not yet systematic, collegial exchanges could be structured around an explicit purpose, such as developing skills to gather, access and make sense of evidence, developing a common understanding of what makes evidence fit for a specific purpose, and developing an agreement on how evidence should be used in a specific situation. Importantly, basing such efforts on existing work habits and processes would increase their uptake and minimise administrative burden.

Increasing availability of specific training on guiding and instructing evidence use. For both school leaders and school quality managers, developing the skills to guide and instruct the use of evidence in schools is essential. Yet, reports indicate that such training is not widely available. Increasing the availability of specific training can help promote use of evidence directly and insert important new knowledge into the widely reported collegial exchanges.

Developing a common understanding around using evidence. This pertains to developing agreement around which evidence is fit-for-purpose for which tasks and how it is best used in concrete situations. This is particularly relevant in the transition to new responsibilities as specific decision-making situations and habits are still emerging.

Collaboratively reflecting on which evidence is best gathered where. While schools are important evidence producers, in some circumstances other providers, such as school quality managers, will be in a better position to gather and prepare fit-for-purpose evidence. In the same vein, schools will be in a better position to gather evidence needed at other levels. Different levels of governance – in particular, school leaders and school quality managers – should be involved in a collaborative reflection on how to optimise evidence provision.

Improving tailoring of evidence based on direct feedback from decision makers. Not all decision makers will be equally prepared to gather and prepare evidence as needed for their new responsibilities. Evidence providers need information about decision makers’ work processes and habits, so that they can tailor evidence to their needs. Responses indicate that school quality managers are motivated to be directly involved in preparing evidence. Informational exchanges may be better suited for feedback on schools’ needs.