EDUIMHE08 › Abstracts: Assessing learning and employment outcomes
The Case for Systematic Monitoring of Learning Processes: Beyond the Politics of Outcome Measurement
Author: Motisha Kaneko, Professor, University of Tokyo
In many OECD countries higher education is challenged by the political claims for accountability in the form of measuring learning outcomes. How should higher education respond?
With this question in mind, Section 1 of this paper examines the backgrounds, and the logic, of the political claims for outcomes evaluation. By comparing the cases of a few OECD countries, it argues that the demands in fact involve different perspectives and interests, and that any form of outcome evaluation, if ever feasible, will neither satisfy the social demands nor improve educational effectiveness.
That, however, does not imply that higher education institutions can be spared from its responsibility. Section 2 argues that the pedagogy employed in classrooms still has not escaped from the shadow of Humboldtian idea, which assumes a particular set of motivations among the youths and the body of scientific knowledge. To the extent that those assumptions become less tenable, the latent rift between the teachers and learners grows, thus creating the ground for popular dissatisfaction and uneasiness.
Based on these grounds, Section 3 delineates a possible alternative: a system of instruments to monitor the learning processes of students. Such instruments would include surveys on students’ aspiration for future career, interests in classes, and the time to spend on learning. Class evaluation and records of achievements can be used in combination. The created data can be benchmarked across institutions and, indeed, across nations. This will provide a reasonable basis for accountability and for substantiating the relevance of higher education.
Assessing higher education learning outcomes as a result of institutional and individual characteristics
The development towards mass higher education is inseparably connected to the modern knowledge society. There is an increasing demand of higher education qualifications in most segments of the labour market, and a rising demand that students should develop transferable or generic skills in addition to subject-specific qualifications.
As a large and important sector in the society, higher education is also facing new demands to account for its contributions. Quality may not any longer entirely be defined internally by the higher education institutions and faculty members. Institutions are to a larger extent exposed to external quality assessment. Marks are the traditional way of formal assessment of students’ achievement, but we will argue that marks at best measure students’ qualifications in a narrow way, with grave validity problems. A broader set of criteria and indicators should be developed.
Ideally, higher education learning outcomes should be measured by a method that is as objective as possible. The initiatives taken by OECD to explore these possibilities through PISA higher education are interesting. Another way of measuring learning outcome is based on students’ reports of what they have learned. In our paper we analyse learning outcomes based on questionnaire data among Norwegian students. The paper compares students’ self-reported learning outcomes across institutions and disciplines, and by students’ characteristics (gender, age, parental education, study effort etc.). We also want to explore how self-reported learning outcome (measuring mainly by transferable skills) is related to grades.
The impact of learning outcomes in business education: Assessing value, relevance and graduate ability in a multi-country study of employers and business graduates
Authors: Ana Azevedo, F.H. Joanneum, Austria, Doris Gomezlj University of Primorska, Slovenia, Jane Andrews, Helen Higson, Aston Business School, England, Antonio Caballero, Claes Fornell International, Spain.
This paper discusses critical findings from a two-year EU-funded research project involving four European countries: Austria, England, Slovenia and Romania. The project had two primary aims. The first of these was to develop a systematic procedure for assessing the balance between learning outcomes acquired in education and the specific needs of the labour market. The second aim was to develop and test a set of meta-level quality indicators aimed at evaluating the linkages between education and employment. The project was distinctive in that it combined different partners from Higher Education, Vocational Training, Industry and Quality Assurance.
One of the key emergent themes identified in exploratory interviews was that employers and recent business graduates in all four countries want a well-rounded education which delivers a broad foundation of key business knowledge across the various disciplines. Both groups also identified the need for personal development in critical skills and competencies.
Following the exploratory study, a questionnaire was designed to address five functional business areas, as well as a cluster of 8 business competencies. Within the survey, questions relating to the meta-level quality indicators assessed the impact of these learning outcomes on the workplace, in terms of the following: 1) value, 2) relevance and 3) graduate ability.
This paper provides an overview of the study findings from a sample of 900 business graduates and employers. Two theoretical models are proposed as tools for predicting satisfaction with work performance and satisfaction with business education. The implications of the study findings for education, employment and European public policy are discussed.
Employment Outcomes of Young Graduates in Europe and Japan – Empirical Evidence from CHEERS and REFLEX
Authors: Harald Schomburg, Senior Researcher, International Centre for Higher Education Research Kassel (INCHER-Kassel) and Ulrich Teichler, Professor International Centre for Higher Education Research Kassel (INCHER-Kassel)
Based on empirical data from the CHEERS and the REFLEX study - large scale surveys with about 35,000 graduates each from higher education institutions in Europe and Japan; conducted 1999 and 2005 – the paper will highlight the methodological approach and the key findings about the labour market outcomes of young higher education graduates in an international comparative perspective. Special attention will be payed on issues of the mesasurement of competences and work requirements and their relationships to study provisions and study conditions. CHEERS and REFLEX do not only provide the best genuine comparative empirical data on employment outcomes of graduates of specific cohorts (1995 and 2000) from institutions of higher education in Europe and Japan which were asked four years and five years after graduation. The data allow also to shed light on the key question to what extent higher education was relevant for the employment outcomes of graduates. Subjective and objective indicators of professional success were used in the analysis of the relevance of a broad range of educational factors (e.g. field of study, type of institution, country, reputation, study conditions, study behaviour).
Additionally, the authors could present the new “Graduate Survey Approach” for institutional development at German higher education institutions. About 50 institutions of higher education intent to cooperate in this project wich was initiated 2007 by INCHER-Kassel (team leader: Harald Schomburg). The study is sponsored for two years by the German Federal Ministry of Education and Research (BMBF). The project aims to implement regular graduate surveys at each institution of higher education in Germany. The first pilot surveys started already 2007, and in the year 2008 up to 90,000 graduates - which represent almost 50% of all graduates of one year – will be included in the survey system. In order to provide relevant information for the ongoing process of institutional development (including evaluation, accreditation, career service, student councelling, curriculum development) members of each institution of higher education will be engaged in the development of their own questionnaire. At the same time it will ensured that the survey has a large component of common questions which allows fruitful comparative analysis especially of the effects of higher education study provisions and conditions for the employment outcomes of the graduates.
Proposing common standards for implementing learning outcomes in decentralized curricular development
Author: Peter Zervakis, Head of Bologna Centre, German Rector's Office
Courses of academic study can best be compared based on learning outcomes and the skills they provide. Yet, there is a substantial challenge for comparative analysis given the frameworks, models and trends specific to academia. The paper seeks to analyse this context from a German and European perspective and to determine the methodological demands a “Higher Education Pisa” would have to meet. Accordingly, a unique project conducted by the German Rectors’ Conference (HRK) has systematically assessed teaching and curricular development in selected universities, providing a robust empirical basis for valuable insights along with positive and negative lessons learned.
Generally, study programmes tend to be de-centrally regulated. In European tradition, universities represent autonomous educational institutions that develop study programmes independently with scientific-academic integrity. Such independence has been viewed in academia as beneficial to the quality of research and teaching. In the Bologna Process the agreements have emphasised precisely this arrangement.
Universities pursue broad educational goals. These include preparing students for their careers, for which scientific knowledge and skills or artistic ability can be useful, even necessary. Consequently, both specialised and general – and thus more comparable – skills are essential. While different institutions focus on different areas, international comparative projects and international standards could be helpful for establishing common references.
The experience of the HRK-Excellence Project, the qualification framework for the European Higher Education Area and the subject-specific results of the TUNING networks provide basic sources for determining comprehensive models, demands and common standards for implementing learning outcomes in decentralised curricular development.
Value for money and efficiency in Higher Education systems of developing countries: resources management and effective management of higher education systems in Uganda
Author : Benon Basheka, Head of Higher Degrees department/lecturer, Uganda Management Institute
Increasingly, the role of efficient higher educational systems in the development of nation states is being recognized in a number of countries in the developing world. Higher educational institutions require adequate resources in the form of financial, material and human resources but their management has received widespread attention from a number of stakeholders. While many have recognized the importance of resources in the running of effective management of higher education system, limited scientific research has been in Uganda’s higher educational systems to establish the extent to which efficient and effective resources management could contribute to effective management of higher educational systems. Our aim is to show how a well managed higher educational system is a good measure of quality and relevant higher education outcome. In this paper, we provide empirical findings from 296 respondents of 6 sampled private and higher educational institutions in Uganda. The data was analyzed using principal component factor analysis, regression analysis and correlation analysis. It was generally found out that resource management has a significant effective on the management of higher educational systems in Uganda; and this has wide contributions in enhancing the quality and relevance of higher educational systems. We recommended that all higher education systems need to create a transparent mechanism of managing resources .We then develop a conceptual framework and highlight policy, practice and managerial implications relevant for international practice.