OECD Reviews of Evaluation and Assessment in Education: Georgia

Assessment and Recommendations

Introduction

Georgia has seen tremendous recent improvement in educational outcomes. From 2009 to 2015, 15‑year‑old students in Georgia improved their learning in reading, mathematics and science by roughly a full grade level. However, Georgia’s progress has not been equitable across all population groups. Urban students outperform rural ones; socio‑economically advantaged students outperform their disadvantaged peers; and students who speak Georgian at home outperform those who do not. Worryingly, these gaps in performance have widened from 2009 to 2015.

To improve educational equity in addition to excellence, it is critical that Georgia develop educational evaluation and assessment systems that can detect areas of low and inequitable performance and address them before they become entrenched. In particular, student assessment can more accurately identify student achievement, teachers can be trained to give students better support, schools can be given more oversight to help them succeed, and the system as a whole can develop the research capacity and data tools needed to facilitate improved learning for all students in the country.

Main trends: learning outcomes are improving but are becoming less equitable

Participation in compulsory education has increased, but many students drop out between lower and upper secondary education

Georgia has achieved near universal participation in primary education, having increased its net enrolment to 98% in 2016 (UNESCO-UIS, 2018[1]). Student enrolment in secondary education also increased significantly and is now comparable to international benchmarks. However, while participation has increased overall, a large number of students drop out of school after grade 9. This is a concern for Georgia because the vocational education sector is underdeveloped (less than 2% of upper secondary students are enrolled in vocational education), so students who drop out do not have the opportunity to develop important competencies and enter the labour market without formal qualifications (Janashia, 2017[2]).

Learning outcomes have improved, but are still low overall

National assessment data are limited in Georgia, but results from international surveys can be used to analyse student outcomes in Georgia. In the Programme for International Student Assessment (PISA), Georgia increased in science performance by 38 score points (equivalent to over one year of schooling) between 2009 and 2015. Similar improvements were observed in reading (27 score points) and mathematics (25 score points). Georgia’s improvement mostly resulted from a reduction in low-performers (students performing below PISA Proficiency Level 2) of nearly 15 percentage points in science, 11 percentage points in reading and 12 percentage points in mathematics (OECD, 2016[3]).

Despite these improvements in student learning, overall learning outcomes remain low compared to neighbouring and European countries (OECD, 2016[3]). In PISA 2015, Georgian students scored over 80 score points less than the OECD average in science, equivalent to over two years of schooling. Georgia’s share of low achievers in science (51%), while decreasing, is still one of the highest among PISA participating economies and is much larger than the OECD average (21%). In addition, less than one percent of students were considered top-performing students in science, meaning they perform at Level 5 or above, compared to 8% on average across OECD countries (see Figure 1) (OECD, 2016[3]).

Figure 1. Percentage of students in different proficiency levels in science in PISA 2015

Source: OECD (2016[3]), PISA 2015 Results (Volume I): Excellence and Equity in Education, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264266490-en.

Participation and outcomes vary considerably according to student demographics

There is significant variation in both participation and outcomes across a range of dimensions. Although Georgia has increased enrolment considerably, enrolment varies across different student population groups. For example, net enrolment in secondary education was 72% for Azerbaijani students, compared to 90% for ethnic Georgians (UNICEF, 2008[4]).

With respect to outcomes, in 2015, socio-economically disadvantaged students in Georgia scored 78 points less than advantaged students, equivalent to roughly 2.5 years of schooling. This gap is larger than in Russia (58 score points difference) and Turkey (59 score points difference) (OECD, 2016[3]). Other dimensions according to which student outcomes in Georgia vary include:

• Geographic location: Students from rural areas scored 44 score points behind their peers in cities, equivalent to nearly 1.5 years of schooling (see Figure 3). This difference can be seen at a regional level, where almost two-thirds of students are above the PISA science baseline in Tbilisi, but only one-third of students are in rural regions (see Figure 2).

Figure 2. Percentage of Georgian students above PISA 2015 science baseline, by region

Source: Author’s estimations based in PISA 2015 sampling data. OECD (2016[5]), PISA 2015 Database, www.oecd.org/pisa/data/2015database/ (accessed on 6 June 2019).

• Mother tongue: Students who speak Georgian at home scored 419 in science, whereas students who do not speak Georgian at home scored 359 (see Figure 3) (OECD, 2016[3]). Although roughly half the country’s students were below baseline proficiency, almost 90% of Azerbaijani students were and no Azerbaijani students scored above Proficiency Level 3.

• Educational track: Vocational students performed nearly 90 score points lower than their peers in general education programmes, equivalent to roughly three years of schooling (Figure 3).

Figure 3. PISA 2015 science performance between different student groups

Source: OECD (2016[5]), PISA 2015 Database, www.oecd.org/pisa/data/2015database/ (accessed on 6 June 2019).

Inequity has worsened over time

Gaps in outcomes between student groups in Georgia in PISA 2015 have actually widened compared to the same gaps in PISA 2009. In 2009, students from cities scored 30 score points more than students from rural areas. By 2015, this gap increased to 44 points. The difference in science score between students who speak mainly Georgian at home and those who speak mainly another language at home has also widened from 40 score points to more than 60 points, equivalent to almost two years of schooling (Figure 4).

Figure 4. Difference in performance on PISA between student groups over time (2009 and 2015)

Sources: OECD (2016[5]), PISA 2015 Database, www.oecd.org/pisa/data/2015database/ (accessed on 6 June 2019); OECD (2010[6]), Data base PISA 2009, www.oecd.org/pisa/data/pisa2009database-downloadabledata.htm (accessed on 6 June 2019).

Worsening educational inequities can be further observed in Georgia at the school-level, where PISA data reveal that the disparity in science performance between Georgian schools has widened considerably. Figure 5 shows that, while the highest-performing schools in 2015 exhibit much higher performance than they did in 2009, the lowest-performing schools in 2009 and 2015 have nearly identical levels of performance (OECD, 2016[3]).

Figure 5. School-level science performance and ESCS in Georgia (2009 and 2015)

Note: Each marker represents one school that was sampled to participate in PISA.

Sources: OECD (2016[5]), PISA 2015 Database, www.oecd.org/pisa/data/2015database/ (accessed on 6 June 2019);

OECD (2010[6]), PISA 2009 Database, www.oecd.org/pisa/data/pisa2009database-downloadabledata.htm (accessed on 6 June 2019).

Evaluation and assessment in Georgia

This review analyses how policies for assessing student learning, appraising and supporting teachers, evaluating schools and evaluating the performance of the education system overall can be used to improve learning for all students. The review draws upon the OECD’s analysis of policies and practices for evaluation and assessment in over 30 education systems to identify how they can best support student learning (see Box 1). In undertaking this review, the OECD team identified three interrelated, systemic issues that are important to address in order for evaluation and assessment to better support learning in Georgia.

Box 1. OECD reviews on evaluation and assessment

The OECD’s reviews show how the components of evaluation and assessment – student assessment, teacher appraisal, school evaluation, school leader appraisal and system evaluation – can be developed in synergy to enhance student achievement in primary and secondary education (Figure 6).

Figure 6. Interactions within the evaluation and assessment framework

This work has highlighted three hallmarks of a strong evaluation and assessment framework:

• Setting clear standards for what is expected nationally of students, teachers, schools and the system overall. Countries that achieve high levels of quality and equity set ambitious goals for all, but are also responsive to different needs and contexts.

• Collecting data and information on current learning and education performance. This is important for accountability – so that objectives are followed through – but also for improvement, so that students, teachers, schools and policy‑makers receive the feedback they need to reflect critically on their own progress, and remain engaged and motivated to succeed.

• Achieving coherence across the evaluation and assessment system. This means, for example, that school evaluation values the types of teaching and assessment practices that effectively support student learning, and that teachers are appraised on the basis of the knowledge and skills that promote national education goals. This is critical to ensure that the whole education system is working in the same direction, and that resources are used effectively.

Source: OECD (2013[7]), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, http://dx.doi.org/10.1787/9789264190658-en.

Modernising and professionalising teaching

Developing students who are ready to compete in 21st century economies requires teachers who are knowledgeable, skilled and motivated to continue improving. Teachers in Georgia participate in professional development much less than teachers across OECD countries. Without continuous training, teachers are not introduced to the latest evidence-based instructional practices that are proven to help students learn. Georgian teachers are also the oldest among all the Teacher and Learning International Survey (TALIS) participating countries, with over one-quarter being above the age of 60. As a result, many teachers instruct students in a traditional manner that emphasises the memorisation of facts instead of the acquisition of skills and competencies. Students in Georgia then struggle to develop the competencies that they need to succeed in higher education and the labour market.

Several factors have contributed to this situation. Georgia has historically lacked career pathways for teachers, meaning that teachers had little formal incentive to improve themselves. Recently introduced pathways, though providing financial incentives, encouraged teachers to accumulate credits instead of demonstrate that they were helping students learn. Finally, previously low teacher salaries created political pressure to allow teachers to continue teaching past their retirement age, which largely explains the greater age of Georgian teachers.

This review makes several recommendations about how teaching in Georgia can be modernised and professionalised. Minimum standards for teachers should be set and enforced to ensure that all teachers have the basic skills needed to help students learn. The teacher professional development scheme should be revised to motivate teachers to develop in ways that helps students learn. Finally, older teachers should be supported to transition out of their positions, thus opening space for young, talented teachers to renew the profession.

Embedding the use of evidence into all levels of educational decision-making

A range of recently introduced reforms - such as the teacher professional development scheme, plans to expand school authorisation, the “New School Model” and the elimination of the Secondary Graduation Examination (SGE) - suggests that Georgia is eager to improve its education system. Nevertheless, it is important that policy decisions, especially large scale ones, be made based upon a thorough review of the available evidence. Such a process helps to direct limited resources to where they can be most effectively used. Perhaps more importantly, basing policy decisions upon evidence establishes a rationale for the enacted reforms, which makes the reforms more likely to be supported across any changes in the political landscape.

While intended to positively impact the system, many of Georgia’s recent reforms have not been made based upon a rigorous evaluation of long-term evidence. This not only results in potentially less effective policies, but also creates an unstable environment where policies are quickly created or eliminated. For example, the SGE was abruptly eliminated and the structure of the Unified Entrance Examination (UEE) revised based upon a short and limited review of data. The eventual decision was implemented immediately without piloting and studying the potential effects. There are also new plans to adjust the teacher salary scale, which had already been recently revised in accordance with the teacher career scheme. These plans, however, do not seem to take into consideration the evidence around why teacher salaries remain relatively low, which is not because of the salary scale itself, but because a large number of teachers are at the lowest paid level of the profession and a significant amount work part-time.

Part of the reason that data and evidence are not used more readily is that, while Georgia’s data systems are very capable of collecting and storing information, there are not many tools available to help users analyse the information. At the central level, policy-makers need to know national performance trends to help inform their decision-making. At the school-level, principals and teachers would like to use evidence to improve their school learning environments and tailor their instruction to individual student needs.

This review proposes several measures to improve the use of evidence in education in Georgia. It suggests that a dedicated unit be created that is responsible for overseeing education research and evaluation and for convening policy meetings that are centred on reviewing the available evidence. It further recommends that Georgia’s data systems be enhanced to include analytical tools that allow persons to easily process, manipulate and view data to inform high‑level policy‑making as well as classroom-level instruction.

Strengthening school oversight and support

Schools in Georgia operate with significant autonomy. They are largely responsible for hiring teachers, adapting national curricula and managing their financial resources. Giving schools autonomy can be an effective method of tailoring education to the needs of different communities. Nevertheless, without systematic oversight and accountability, schools that are struggling to provide adequate services do not receive the support they need to improve student learning.

International assessments suggest that the variance in student outcomes between Georgian schools is considerable and growing (see Figure 5). Schools in rural environments and those serving linguistic minority populations are falling behind others. Regarding financial management, there is evidence that, even among schools with the same number of students, some schools are requesting and receiving up to three times as much funding as others. These schools require support so they can help their students learn and use their resources more efficiently.

However, while schools in Georgia are in need of effective central supports, there are inadequate oversight mechanisms to systematically identify critical issues and provide support where it is most needed. This situation should be addressed immediately. Georgia should continue with its plans to authorise all schools, but should prioritise supporting schools that are in most need of improvement. Georgia also needs to develop a comprehensive school evaluation framework that will systematically monitor schools and link their results with formative consequences. Regular national assessments should also be developed to provide more data for the evaluation framework to use, which would make school oversight more robust and relevant.

Improving learning outcomes and equity through student assessment

The primary purposes of student assessment are to determine what students know and are capable of doing, to help students advance in their learning, and to assist students in making an informed decision on the next step in their education. In Georgia, the ministry has launched many initiatives to make assessment more meaningful. However, despite these efforts, student assessment is still used mainly to grade students, not to help them improve their learning.

Several factors have prevented student assessment in Georgia from being more educationally valuable. First, despite several efforts to improve assessment literacy, teachers and the public still concentrate on the importance of numeric marks, even though those marks might not accurately represent what a student knows and can do. This understanding of assessment is reflected in teachers’ classroom assessment practices. Teachers do not use a wide variety of assessment techniques and prefer to assess students mainly through multiple-choice tests that do not assess a broad range of skills.

Furthermore, Georgia’s examinations system adds to the summative pressure that teachers and students feel. Until recently, students had to take 12 subject tests over two grades at the end of upper secondary education in order to graduate. A separate test, in many of the same subjects, needs to be taken in order to enrol in higher educational institutions. The intense attention paid to these examinations led students and teachers to focus narrowly on examinations preparation, often at the expense of students’ individual learning needs.

Policy issue 2.1. Enhancing the educational value and use of teachers’ classroom assessment

Effective classroom assessment, and formative assessment in particular, can positively affect students’ attitudes towards learning and their engagement with school (Black and Wiliam, 1998[8]). Georgia has made considerable strides to embed formative assessment practices into classrooms, but the effect has been less than desired due to a lack of alignment between assessment and the curriculum and inadequate resourcing to support teachers in their reform efforts.

Additionally, assessment in Georgia exerts considerable pressure on teaching and learning processes. The attention paid to student marking (in some cases, teachers even give students one mark per each school day) can distract teachers and students from focusing on what can be done to improve individual student learning.

Lastly, most OECD countries provide teachers with guidance on how to report student results as a means to record student progress consistently. In Georgia, however, there is no national report card template, so schools document student progress in varying ways. Without a reliable record of student progress, teachers have difficulty understanding where their students are in their learning and cannot adapt their instruction accordingly. Students, especially students who change schools, are then at risk of falling behind.

Box 2. Recommended actions for enhancing teachers’ classroom assessment

Recommendation 2.1.1. Make formative assessment a central focus of teacher practice. Georgia should use the new stage-based curriculum as a policy lever to embed its use. Curriculum materials should provide explicit direction on the use of formative approaches to assess students, including what tools teachers can use, how to provide feedback and how to use the results to individualise instruction. A particularly useful assessment tool to use in Georgia would be student portfolios, which are collections of student work that are compiled over time. Creating portfolios requires teachers to provide continuous feedback and compels students to reflect upon their own strengths and weaknesses when determining what items to add to their portfolios, two elements of assessment that are currently lacking in the Georgian educational landscape.

Recommendation 2.1.2. Reduce the pressure around summative marking and make it more educationally meaningful. Daily log grading should be discontinued. It is time consuming and does not provide rich information about how students are progressing in their learning. Teachers should be supported to make their summative marking more closely aligned with the standards embedded in the new curriculum so the marks convey meaning about student learning. To help support teachers in this regard, the ministry should allocate school-time for teachers to engage in school-based moderation, which allows teachers to convene to discuss how they mark student work and determine what students have learnt. Schools and teachers should also be given resources to help them interpret curriculum standards, such as examples of student work along with explanations about how to review the work vis-à-vis marking criteria.

Recommendation 2.1.3. Systematically record assessment results in order to track student progress and inform key decisions. The ministry should produce a common report card template and establishing national procedures around its dissemination. With a standardised report card, student records could also be more consistently entered into the Education Management Information Systems (EMIS) databases, which would further help track students and their learning as they advance through the education system. This rich source of information could then be used for strategic purposes, such as to identify and support the most vulnerable students in Georgia.

National assessments should also be developed to complement the information generated through classroom assessments. Having external assessment results would help students understand their own progress with respect to national learning standards. An examination in grade 9 (or grade 10 if compulsory education is extended) could be introduced to help students decide whether to pursue a general educational or vocational programme of study in upper secondary education.

Policy issue 2.2. Building understanding that the goal of assessment is to improve student learning

Having a high level of assessment literacy (what stakeholders understand about education assessment) is an important aspect of contemporary education (Plake, 1993[9]; Fullan, 2000[10]). In Georgia, previous efforts to strengthen assessment practices did not achieve their desired outcomes in large part because stakeholders’ assessment literacy was not sufficiently developed to support the reforms. They were asked to change but were not helped to understand why change was needed in the first place.

For most teachers in Georgia, re-orienting their assessment practices to promote student learning represents a radical departure from what they are used to. They will not be able to implement these practices without consistent support and reinforcement, which research shows is one of the primary factors associated with sustaining classroom-level reforms (Harrison, 2005[11]; Wilson, 2008[12]). Part of the challenge is initial teacher preparation, which currently lacks a graduate student standard as a reference point and does not provide teacher candidates with a strong foundation in student assessment. Another part of the challenge is relatively weak in-service training and collaboration around assessment for learning. In Georgia, over one-third of teachers never, or only once a year or less, engage in discussions with other teachers about the learning development of specific students.

Beyond teachers, the Georgian public also interprets student assessment as being critical and mainly about marking. In order for Georgia’s student assessment reforms to succeed, efforts also need to be made to improve the public’s assessment literacy so everyone understands the educational value of formative assessment, rich feedback and using assessment for learning. Without this shared conviction and willingness to improve, policy reform efforts will struggle to be successful.

Box 3. Recommended actions for building understanding of assessment

Recommendation 2.2.1. Provide teachers with assessment resources to improve student learning. A graduate teacher standard should be established and used as a reference point to strengthen what teacher candidates learn about student assessment during initial teacher education. Teachers in schools should be given time to observe each other and discuss how to assess students. Technology can be helpful in accomplishing this goal, especially for teachers who work in smaller schools with less in-house capacity. Finally, more assessment resources, such as sample tests and marking rubrics, should be created and provided via an online repository so all teachers can access them.

Recommendation 2.2.2. Communicate that the goal of student assessment is to improve learning. School leaders, such as principals and lead teachers, should be given prepared responses that explain the value of the new curriculum and assessment reforms. They can then use these responses to address the concerns of other teahers and parents. In addition, parents will need to be supported in understanding the new, common report card template. Schools, with guidance from the ministry, can hold school meetings and distribute materials for this purpose. These efforts should be part of a broader, national campaign that communicates that the purpose of assessment is not just to grade students, but to help them learn.

Policy issue 2.3. Reviewing the modes of examination for graduation and tertiary selection at the end of upper secondary education

Georgia’s dual-examinations model was well regarded, but the limitations of the system also became apparent over time. The examinations did not assess higher‑order skills and the number of subjects tested created significant pressure for teachers to “teach to the test” and for students to seek out private tutoring opportunities to improve their chances of success. The former issue prevented students from learning the full breadth of the curriculum, while the latter worsened educational equity.

The current absence of an upper secondary graduation examination presents an opportunity to establish a modern examination system that is in line with the country’s curriculum and overall vision for education. As this is occurring, certain vulnerabilities of the current system need to be addressed. In particular, certification from upper secondary education is awarded based only on students’ school-level marks, which have historically not been an accurate measure of student learning. Moreover, the absence of an upper secondary examination increases the attention on the Unified Entrance Examination (UEE) as the only external signalling mechanism of student achievement in upper secondary school. However, not all the items from the UEE are aligned with the curriculum, which can distort the teaching and learning that occurs in Georgian classrooms.

Box 4. Recommended actions for reviewing the modes of examination

Recommendation 2.3.1. Prepare for a single examination model in which one test would certify completion of upper secondary education and select students for entry into higher education. The examination should have a small number of core subjects, which would also make it easier and more cost effective to administer. It should have different versions of different subjects, such as maths. Having different versions of certain subjects would allow the exam to assess basic minimum competency in some students, while selecting others for entrance into university. A flexible design would also allow the exam to be taken by all students, including those who attended an upper secondary vocational school, which would help remove the “dead ends” that are a noted concern about the vocational education sector.

Recommendation 2.3.2. Take steps in the immediate term to improve upper secondary certification and strengthen the UEE. First, planned school-level examinations need to be strengthened so that student certification from upper secondary school is trusted. The National Assessment and Examinations Centre (NAEC) is well‑positioned to support schools in this regard by providing examination content that is based with the curriculum. Second, the UEE’s alignment with the curriculum should be strengthened to reduce the backwash effects that it creates. Adding in a final review of the UEE’s items would help tighten the link between the questions on the exam and the expectations of the curriculum.

Creating a highly qualified and motivated teaching workforce

Teacher appraisal can be a strong lever for modernising and improving teaching and learning. By providing teachers with regular feedback and setting high standards for teaching quality, appraisal encourages teachers to continually adapt and improve their practice. To strengthen teacher appraisal processes, Georgia recently implemented a performance career scheme. The adoption of the scheme coincides with efforts to shift instruction towards a more student-centred approach that is focused on the development of complex competencies.

However, the current system has so far had little impact in terms of professionalising teaching or encouraging teachers to adopt newer, more effective teaching techniques. The lack of impact reflects the scheme’s design, which makes moving up the career path contingent on form-filling and acquiring credits. These requirements do not necessarily recognise or reward effective teaching and distracts teachers from their central focus on student learning.

Other factors are also impeding the development of a highly qualified teaching force. Undergoing professional development is not mandatory in Georgia and many teachers have gone years without receiving formal support to improve their teaching. Furthermore, entrance requirements into initial teacher education programmes have been low, which has affected the calibre of persons who become teachers. Finally, Georgia’s teaching population is the oldest out of any country that participated in TALIS 2018 and some are less motivated to engage in activities that would improve their teaching. This partly explains why, several years after the implementation of the teacher career scheme, most teachers are still at the lowest level.

Policy issue 3.1. Applying minimum standards for teaching and encouraging the development of higher teaching competencies

Since 2004, Georgia has attempted to shift teaching towards competency development and a more student-focused approach. Many countries have implemented a similar change in recent years and have found that a major challenge is equipping teachers with the pedagogical skills needed to implement the new approach.

To prepare teachers to adopt more student-centred instruction, Georgia reformed the teacher development and advancement scheme in 2010 to motivate teachers to improve. However, nine years after the scheme was first introduced, it has had little impact on teaching quality, with the majority of teachers remaining at the entry level. This suggests that teachers need to be better supported to develop themselves and that the scheme itself needs to promote teachers based upon how well teachers teach.

Box 5. Recommended actions for applying teaching standards

Recommendation 3.1.1. Support all teachers to meet minimum standards. In many OECD countries, a teacher examination is administered to validate that teachers have acquired basic minimum competencies before they enter the profession. Georgia has developed teacher examinations as part of its career advancement scheme, but many teachers have never taken it. This requirement should be strictly enforced and the examinations improved to ensure that teachers who pass them are fit to teach. Schools play a vital part of this process, so principals should help their teachers develop themselves and become certified, senior teachers.

Recommendation 3.1.2. Re-focus the teacher professional development advancement scheme on demonstrating higher levels of teaching competencies. A central concern regarding the teacher career scheme is the requirement to accumulate credits for promotion, which encourages teachers to undertake activities that might contribute little to improving their teaching. The teacher standards themselves are also somewhat vague about the differences between different levels of teachers, which makes it difficult for teachers to be evaluated accordingly. The OECD recommends that the standards be clarified so they can be used as the central reference in teacher appraisal. Furthermore, the process of evaluation and promotion should focus on authentic evidence of teaching practice, not the accumulation of credits.

Policy issue 3.2. Supporting teachers to develop professionally throughout their career

In most OECD countries, an important lever to embed more student-centred teaching is providing feedback and guidance to teachers through regular appraisal. While Georgia does not have regular appraisal, the country has recently announced plans for its introduction.

For regular appraisal to be an effective driver of teacher professionalism, it must focus on to what extent teachers help their students learn. The feedback that teachers receive from their appraisals can be used to direct them towards professional development opportunities. In Georgia, such opportunities need to be created, but can be modelled after past successful programmes.

Box 6. Recommended actions for supporting teachers to develop

Recommendation 3.2.1. Focus the new regular appraisal on student learning and providing feedback for teachers’ professional learning. Creating an effective, regular appraisal is difficult and takes considerable time. In Georgia, currently the primary appraisal process for teachers is for promotion. The process is high stakes and its evaluators have not developed the capacity to provide constructive, formative feedback. For regular appraisal to be effective, Georgia needs to make the process formative and focused on how well teachers help students learn. Evaluators should be qualified and independent and need to be trained on how to review teaching help teachers grow.

Recommendation 3.2.2. Give teachers access to high quality professional development. High quality, impactful professional development is one of the key factors in improving teaching and learning. Unlike most OECD countries, Georgia does not require its teachers to engage in professional development. Georgia should make this a requirement and connect its professional development offerings with teachers’ regular appraisal. Furthermore, the development opportunities should build upon the Georgia Primary Education Project (G-PriEd) and the Millennium Challenge Corporation (MCC) projects in order to expand the availability of training opportunities that are proven effective.

Policy issue 3.3. Setting high standards for entry to teaching and provide more structured support in the early years

Countries use a combination of different mechanisms to build a high quality teaching workforce. These include selecting candidates with strong academic skills, establishing high quality initial teacher education and requiring teacher candidates to pass a formal probation appraisal. Over the past decade, Georgia has introduced wide-ranging reforms to raise the bar for entry into teaching, such as developing certification examination, creating a one-year consecutive initial teacher education programme and, most recently, introducing a new master’s degree in education.

These measures, however, have not produced the desired impact. A study of the initial preparation of mathematics teachers in 17 countries found that Georgia had among the least developed quality assurance systems for new entrants to the profession out of all the participating countries (Ingvarson et al., 2013[13]). One reason that teacher preparation continues to struggle in Georgia is that policy‑makers have been hesitant to introduce additional quality controls for entry in order to avoid dis-incentivising an already small pool of candidates. This lack of quality controls risks that new teachers will continue to lack essential academic and teaching competencies. It also creates the perception that teaching is not a demanding profession for talented school graduates, thus perpetuating the cycle of low quality teaching and contributing to low learning outcomes nationally.

Box 7. Recommended actions for setting high entrance standards into teaching

Recommendation 3.3.1. Establish more rigorous standards for entry and completion of initial teacher education. Teachers need to have the necessary subject knowledge and pedagogical skills they need to build students’ competencies. Given that beginner teachers in Georgia struggle to be effective (Ingvarson et al., 2013[13]), improving both the entrance requirements into initial teacher programmes and the rigour of licensing requirements should be a priority. Georgia should set a minimum threshold for entry into initial teacher preparation and set standards (i.e. a graduate teacher standard) for what teacher candidates should be expected to know and be able to do upon graduation. The graduate teacher standard should then inform the continuous improvement of teacher preparation programmes.

Recommendation 3.3.2. Introduce an induction period and probation appraisal for new teachers. Georgia introduced a one-year teacher induction programme in 2007, but it was never implemented partly to avoid creating another hurdle to enter the teaching profession. This review recommends that Georgia should reconsider introducing an induction period, during which new teachers should be mentored to help them improve. At the end of the induction period, a probationary appraisal should determine whether the teacher becomes fully confirmed in their post.

Policy issue 3.4. Attracting new teachers and motivating them to succeed

Several factors are currently hindering the development of a professional and qualified teaching workforce. These include a large number of older teachers who are less likely to be interested in developing themselves but continue to teach, the perception of teaching as a less prestigious career and financial incentive schemes that are misaligned with the actual causes of lower teacher earnings. These concerns will need to be addressed as part of overall strategic planning to improve the state of education in Georgia.

Box 8. Recommended actions for attracting new teachers

Recommendation 3.4.1. Encourage renewal of the teaching profession. The presence of a large share of older teachers who are not motivated to engage in career development is impeding the success of the educational reforms introduced by the ministry. By remaining in their positions, these teachers also reduce opportunities for talented young graduates to enter the profession. Georgia should implement a mandatory retirement age, which would facilitate the exiting of older teachers and thus open space for new teachers. Retiring teachers will need to be supported as they leave the profession. New teachers should be purposefully allocated to teach subjects and teach in schools that are difficult to staff.

Recommendation 3.4.2. Review planned adjustments to teacher salaries to make them impactful and educationally valuable. The introduction of career pathways and salary scale revisions have helped bring Georgian teacher pay scales in line with OECD norms. The reasons that, despite these changes, Georgian teachers’ earnings are still considered low are because most teachers are at the lesser paid practitioner level, with the majority only working part-time. To use teacher salaries as a lever to improve teacher motivation, adjustments to the salary scale should only be made to the practitioner level. Moreover, the ministry should reduce the number of part-time teachers by giving effective part-time teachers leadership and mentoring responsibilities. These measures will help motivate teachers to improve and help them earn more income without distorting the overall financial structure of the profession.

Assuring quality schooling through external evaluation and school-led improvements

Compared to international benchmarks, schools in Georgia now have significant autonomy for assessment, curriculum, human resourcing and financial management (OECD, 2016[14]). The autonomy afforded to schools is supposed to be balanced by accountability and oversight from the school board and competition arising from parents exercising school choice. In practice, however, school boards lack the capacity and authority to provide robust oversight or accountability. Parental choice is also limited outside the biggest urban areas (Transparency International, n.d.[15]). Because Georgia lacks a robust school evaluation system, the country’s schools operate with very limited oversight and accountability.

To help improve the quality of schooling, Georgia is planning to extend its authorisation model to all schools. While this will help ensure that schools are meeting basic standards, it will be difficult to accomplish because there is insufficient personnel to conduct visits to all schools in the short term. In the long term, Georgia’s school authorisation can be further developed into a comprehensive school evaluation model. For this to occur successfully though, significant expertise will have to be built within the National Centre for Education Quality Enhancement (NCEQE) as it currently operates in a mainly administrative capacity and not in a school‑improvement capacity. School self-evaluation could be an effective method of assuring school quality in the absence of established external mechanisms, especially in Georgia where all schools are already required to submit self-evaluation reports to NCEQE. Nevertheless, the process will have to be made more meaningful because most schools currently view self-evaluation as a compliance exercise rather than a way to improve themselves.

Policy issue 4.1. Reaching all schools for authorisation

Georgia is currently deliberating between several methods of assuring quality schooling, including authorising all public schools and creating composite indicators. The review team recommends that Georgia focus its attentions on authorising all public schools in the short term, which will help address the significant gap in school oversight that currently exists. While developing composite indices of school quality can help monitor schooling, the use of such measures is a complement to, not replacement for, regular school evaluation processes (OECD, 2013[7]). However, as the ministry has recognised, authorising all public schools by 2021 is not feasible. A graduated approach is needed to first identify and support those schools at greatest risk of not meeting the basic conditions needed to support the provision of quality education.

Box 9. Recommended actions for reaching all schools for authorisation

Recommendation 4.1.1. Develop a risk assessment model to guide the provisional authorisation of public schools. Georgia has strong systems for collecting basic school information. Because authorisation of public schools cannot be realistically completed in a short amount of time, Georgia should use the available information from EMIS and NAEC to develop a risk assessment model. Through this model, Georgia can identify schools that are most vulnerable and prioritise providing support to them. Schools determined by the model to be not at risk can be provisionally authorised in the short term and receive a fuller evaluation later. A critical part of developing this model will be selecting the indicators that will feed into it and what thresholds must be met in order for a school to be considered not at risk. The indicators should focus on several factors, such as material resources, financial management and student outcomes.

Recommendation 4.1.2. Focus Education Resource Centres on supporting schools. Relevant and responsive school-level support is a critical element in assuring school quality. In Georgia, Education Resource Centres (ERCs) are well‑positioned to provide school-level support, but their current role is administrative and they will need guidance to become supportive. To this end, the ministry should significantly reduce ERCs’ mandate for compliance checking, provide ERCs with staff who are qualified to support schools and rationalise the ERCs network to meet each municipality’s needs. Once ERCs are equipped to support schools, the ministry will need to develop a model that governs how ERCs should support schools, such as directing resources towards the neediest schools and reinforcing the ERCs’ role in financially auditing schools.

Policy issue 4.2. Developing an external school evaluation model over the medium to long term

School authorisation is a helpful, short-term method for instilling school accountability. In the long term, however, Georgia will need to develop a full school evaluation model. The country has already developed new draft standards for the authorisation of public schools, which go beyond the existing authorisation standards by focusing on school quality. These can be built upon to create standards that underpin a full-fledged evaluation system.

To support school evaluation, several materials and structures of the education system will need to be strengthened. The draft standards, while a significant improvement over their predecessor, can still focus more on school improvement and less on compliance. Furthermore, Georgia currently lacks a cadre of qualified school evaluators. Identifying and developing these individuals will be vital to ensuring successful school evaluation.

Box 10. Recommended actions for developing an external school evaluation model

Recommendation 4.2.1. Develop a model of school evaluation that supports schools to improve teaching and learning. School evaluation is recognised in most OECD countries and many non-member states as being an essential lever to monitor school quality, encourage future improvement and provide school-level accountability (OECD, 2013[7]). In Georgia, decentralisation of management and comprehensive curriculum reform mean that introducing school evaluation will be particularly helpful in ensuring that schools meet basic minimum standards. To develop an effective model, Georgia should anchor school evaluation standards in a clear vision of a good school and then revise the standards to focus more on school quality. Georgia will then need to create accompanying materials and improve central capacity in order to support implementation of the school evaluation framework. As the evaluation model should have consequences, schools will also need to understand what the role of external evaluation is and how they can prepare for evaluation visits.

Recommendation 4.2.2. Develop capacity for external evaluations. Implementing a new evaluation model will require a significant strengthening of Georgia’s school evaluation capacity, in terms of both numbers and expertise. Of particular importance will be identifying a pool of capable external evaluators (given their current functions and relationships with schools, ERCs staff cannot be expected to fill this role). Georgia’s plan to contract external evaluators is positive and can be strengthened further by considering previous school principals. With an established group of external evaluators, the Council’s role can be gradually changed. In the short term, it can review the quality and fairness of the authorisation process. In the long-term, the Council might not be needed and its expertise can form the basis of an independent inspectorate.

Policy issue 4.3. Creating the foundations for school-led improvement

Self-evaluation can be a powerful tool for driving school improvement, especially in Georgia where the vast majority of schools complete self-evaluations annually. However, there is broad acknowledgement nationally across policy‑makers and school practitioners that self-evaluation is not yet supporting school improvement. At the heart of the issue is that schools have not yet appropriated self-evaluation as an internal tool, integrated into their management cycles, to support improvement.

This situation reflects the fact that school-level leadership and capacity has not received sufficient attention in Georgia. In contrast to international trends towards the development of principals as instructional leaders, the principal role in Georgia is not clearly defined. Teachers become principals without having the background or preparation to meet the requirements of the position. Schools also receive little support to undertake self-evaluation or to understand its purpose. In many schools, self-evaluation is frequently limited to a cut and paste exercise to meet external requirements.

Box 11. Recommendations for creating the foundations of school-led improvement

Recommendation 4.3.1. Support schools to use self-evaluation effectively. While schools in Georgia are familiar with performing self-evaluation, they receive very little support on how to undertake self-evaluation in a meaningful way. To make self-evaluation more useful, its purpose should be redefined around school improvement and schools should be supported in using their self-evaluation results to improve student learning. While school authorisation is occurring in the short term, the ministry should make self‑evaluation support authorisation by aligning self-evaluation criteria with authorisation standards. Schools then need to be supported in exploiting the data that is referenced as part of authorisation, which can be done by ERCs or school coaches from the “New School Model”. Finally, a comprehensive self-evaluation framework should be developed alongside the long-term school evaluation framework.

Recommendation 4.3.2. Build school leadership for improvement. One of the key challenges to developing school principals as instructional leaders in Georgia is the absence of incentives to keep strengthening their abilities. This absence of incentive to develop is matched by little available professional development opportunities for principals, which further discourages them from improving themselves. To develop the capacity of school principals, which would strengthen school-led improvement in general, Georgia should identify and support promising school principals through creating a process to select talented candidates from existing teachers and introducing mandatory preparation before they become principals. Principals should also be incentivised to develop themselves, either through adjustments to their salaries or by giving them more career options (e.g. becoming an external school evaluator). With more qualified principals and greater expectations, Georgia should introduce appraisal mechanisms for principals to hold them accountable for their performance. The role of the school board will also have to be revised to allow principals to assume greater leadership over teaching and learning at their schools.

Strengthening system processes to evaluate national education performance

System evaluation is central to improving education performance. Evaluating an education system holds the government and other stakeholders accountable for meeting national goals and provides the information needed to develop effective policies. In Georgia, system evaluation has seen significant development over recent years, especially in the areas of data collection and management.

Despite these advancements, however, some elements of system evaluation are still lacking. In particular, Georgia does not have a strong culture of using evidence to inform policy-making. Also, there are few tools that can help persons analyse the rich data that are centrally collected. As a result, decisions are sometimes made without being based on relevant evidence. Furthermore, though national assessments are administered, the funding that supports these activities is being phased out and, afterwards, Georgia will not have a regular, external measure of student outcomes. In a context where educational inequity is worsening, it is problematic that these processes, which would help to systematically identify and address equity gaps, are not in place.

Policy issue 5.1. Building a culture of research, evaluation and improvement of the education system

Reviews of education systems reveal common practices related to research and evaluation that contribute to successful system evaluation. These include:

• conducting an analysis of available information to produce a rich body of information about the system

• establishing procedures that position evidence review at the centre of policy‑making

• evaluating policies to determine their effect and to inform future decision-making (OECD, 2013[7]).

In Georgia, many of the foundations upon which a culture of research evaluation can be developed are weak. There is no unit responsible for guiding the national-level evaluation and research agenda and, as a result, there is limited, analytical information produced about the education system as a whole. Without consistent reporting about the system, policies are created without reviewing key evidence that could inform their development, and resources are spent in support of unsubstantiated initiatives.

Box 12. Recommended actions for building a culture of research, evaluation and improvement

Recommendation 5.1.1. Establish a formal research and evaluation unit. In Georgia, research and evaluation responsibilities are loosely divided between NAEC and EMIS. This configuration has limitations, as both bodies mostly work with their own data and neither is responsible for the evaluation of the system as a whole. Georgia should create a research and evaluation unit at the centre of the ministry that is explicitly responsible for study of the entire system. The unit would report directly to the Minister of Education and work across a research agenda, which should include topics like manpower planning for teachers and the effects of planned adjustments to the teacher salary scale.

Recommendation 5.1.2. Encourage the dissemination and usage of research and evaluation activities. A core function of research and evaluation units in most OECD countries is the production of regular reports about the state of the system and periodic analytical reports about specific themes (OECD, 2013[7]). In Georgia, the Monitoring Report most closely approximates a report about the state of the system, but, at 135 pages, it is difficult to interpret and is more descriptive than analytical in nature. There is also no clear expectation within government that such reviews of reports and evidence takes place. Georgia should release an annual analytical report about the state of the education system, along with ad‑hoc reports about thematic issues. Policy-makers should meet according to a regular schedule to share and discuss evidence. External entities, such as universities and non-profit organisations, can be engaged to lend further research capacity. In the future, a dedicated research and evaluation institute can be established to firmly embed the use of evidence in decision-making.

Recommendation 5.1.3. Use system evaluation to enhance the value of system planning. The introduction of a Unified Strategy is a positive development, but the plan is not widely known and understood. To make strategic planning more relevant and impactful, the Unified Strategy should explicitly communicate the key issues of the Georgian education system, in particular equity of outcomes. The goals in the strategy should also be balanced in that they think of the outcomes a system wants to achieve as well as the internal processes and capacity throughout that are needed to achieve those outcomes (Kaplan and Norton, 1992[16]).

Policy issue 5.2. Making information about the education system more accessible and usable

Georgia’s information systems are modern, widely used and are highly trusted. EMIS collects data from all schools throughout the country and NAEC stores assessment and examination data for students and teachers. Both organisations identify individuals using their government identification number and simple demographic information is drawn directly from government sources instead of being re-entered.

Nevertheless, while education data are collected and managed effectively, accessing the information, particularly in an analytical manner, remains a challenge. User-friendly analytical tools have not been developed and individuals have neither the time nor the capacity to retrieve and analyse the data manually. As a result, educators and Ministry of Education, Science, Culture and Sport (MoESCS) officials do not systematically use data to help guide students’ education and inform strategic planning, which risks that systematic needs are not noticed and not addressed.

Box 13. Recommended actions for making information about accessible and usable

Recommendation 5.2.1. Introduce analytical and reporting functions for EMIS tools. It is important that data tools not only store data, but also provide an interface for users to easily retrieve and analyse data. In Georgia, the lack of such tools in the main data system discourages school staff from using data to inform their instruction, prevents MoESCS staff from using evidence to inform their decision-making and makes it difficult for the public to hold the government accountable. Georgia should introduce a reporting feature into EMIS’s E-School that allows users to organise, process and display information generated using EMIS data. A separate web portal should be built that allows the public to view some data from EMIS, which would help hold the system accountable.

Recommendation 5.2.2. Create an easier-to-use monitoring system. At present, MoESCS’s primary tool for monitoring the education system is the Monitoring Report, which is difficult to interpret and only contains information from when it was published. Georgia should complement the Monitoring Report by developing a digital performance dashboard that displays up-to-date information in real time. The dashboard should be accompanied by a tutorial that will help policy-makers understand how the dashboard should be used and for what purpose.

Policy issue 5.3. Developing and implementing a national assessment strategy that supports system goals

Research shows that having externally validated measures of student performance helps monitor performance and inform system-level policy‑making (OECD, 2013[7]). In Georgia, there is no established system of monitoring of student learning outcomes before grade 12, and what instruments there are do not cover key outcomes (such as literacy) and are not administered to all students.

Importantly, MCC funding, which is largely supporting these assessments, is phasing out and there is no guarantee that such important work will continue. A recent proposal concerning a national assessment strategy suggests that diagnostic assessments be administered at the beginning of Grades 4, 6 and 10, but this strategy is not finalised.

Box 14. Recommended actions for developing and implementing a national assessment strategy

Recommendation 5.3.1. Define a concept for the national assessments. The OECD recommends that Georgia plan for the resources and capacity that will be necessary to continue the administration of its national assessments. Georgia should also take the opportunity to improve upon the assessments and determine how the assessments should be structured to best support national goals. A steering committee should be formed to lead the planning and development of the assessment. The OECD recommends that the steering committee make providing formative feedback the primary purpose of the new national assessment strategy.

Recommendation 5.3.2. Determine the design features of the national assessments. Georgia will need to decide upon several design components related to the national assessments. These include which subjects to test, whether to test all students or a sample and which grades should be tested. In general, it is recommended that Georgia make its decisions in order to support to formative purposes of the assessments and in consideration of the specific monitoring needs of the country.

Recommendation 5.3.3. Develop a reporting scheme that serves formative purposes and avoids punitive consequences. Georgia should consider carefully how to report the results of national assessments to students, teachers, schools and the public. Georgia should avoid any suspicion that the results would be used to punish school staff. Instead, results of the national assessment should be reported in a manner that informs instruction and guides decision-making. The reports should contain different benchmarks against which schools can compare themselves, such as results by municipality and from schools with similar student intakes. Different types of reports should also be generated to accommodate the needs of different stakeholders (e.g. for teachers, principals and policy-makers).

References

[8] Black, P. and D. Wiliam (1998), Inside the Black Box: Raising Standards Through Classroom Assessment by Paul Black and Dylan Wiliam Inside the Black Box: Raising Standards Through Classroom Assessment, https://www.rdc.udel.edu/wp-content/uploads/2015/04/InsideBlackBox.pdf (accessed on 3 January 2019).

[10] Fullan, M. (2000), “The three stories of education reform”, http://michaelfullan.ca/wp-content/uploads/2016/06/13396051100.pdf (accessed on 3 January 2019).

[11] Harrison, C. (2005), “Teachers developing assessment for learning: mapping teacher change”, Teacher Development, Vol. 9/2, pp. 255-263, http://dx.doi.org/10.1080/13664530500200251.

[13] Ingvarson, L. et al. (2013), TEDS An Analysis of Teacher Education Context, Structure, and Quality-Assurance Arrangements in TEDS-M Countries Findings from the IEA Teacher Education and Development Study in Mathematics (TEDS-M), International Association for the Evaluation of Educational Achievement (IEA), Amsterdam, https://www.iea.nl/fileadmin/user_upload/Publications/Electronic_versions/TEDS-M_Findings.pdf.

[2] Janashia, S. (2017), General Education in Georgia: Policy Problems and Solutions, ISET Policy Institute.

[16] Kaplan, R. and D. Norton (1992), “The Balanced Scorecard—Measures that Drive Performance”, Harvard Business Review, Vol. 70/1, pp. 1-9, https://hbr.org/1992/01/the-balanced-scorecard-measures-that-drive-performance-2 (accessed on 16 July 2019).

[5] OECD (2016), PISA 2015 Database, http://www.oecd.org/pisa/data/2015database/ (accessed on 6 June 2019).

[3] OECD (2016), PISA 2015 Results (Volume I): Excellence and Equity in Education, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264266490-en.

[14] OECD (2016), PISA 2015 Results (Volume II): Policies and Practices for Successful Schools, PISA, OECD Publishing, Paris, https://dx.doi.org/10.1787/9789264267510-en.

[7] OECD (2013), Synergies for Better Learning: An International Perspective on Evaluation and Assessment, OECD Reviews of Evaluation and Assessment in Education, OECD Publishing, Paris, http://dx.doi.org/10.1787/9789264190658-en.

[6] OECD (2010), Data base PISA 2009, http://www.oecd.org/pisa/data/pisa2009database-downloadabledata.htm (accessed on 6 June 2019).

[9] Plake, B. (1993), “Teacher assessment literacy: What do teachers know about assessment?”, Mid-Western Educational Researcher, https://www.sciencedirect.com/science/article/pii/B9780125541558500053 (accessed on 3 January 2019).

[15] Transparency International (n.d.), Setting Georgia’s Schools Free?, Transparency International, Tbilisi, https://www.transparency.ge/sites/default/files/post_attachments/School%20Reform-ENG_0.pdf (accessed on 20 June 2019).

[1] UNESCO-UIS (2018), Education : Net enrolment rate by level of education, http://data.uis.unesco.org/# (accessed on 13 June 2019).

[4] UNICEF (2008), Georgia Monitoring the situation of children and women - Multiple Indicator Cluster Survey 2005.

[12] Wilson, N. (2008), “Teachers expanding pedagogical content knowledge: learning about formative assessment together”, Journal of In-Service Education, Vol. 34/3, pp. 283-298, http://dx.doi.org/10.1080/13674580802003540.