|
Generic skills strand
The generic skills strand is an essential component of the feasibility study. Indeed, competencies in critical thinking, analytic reasoning, problem-solving, or the generation of knowledge and the interaction between substantive and methodological expertise are widely viewed as critical for the success of individuals and of rising relevance in the information age. It is therefore important for an AHELO to measure these transversal higher-order competencies that are necessary for success in both academic and business contexts – not only cognitive knowledge.
A key advantage is that such competencies are largely invariant across occupational and cultural contexts and could be applied across HEIs, departments and faculties. Moreover, a focus on higher-order skills allows the coverage of a more diverse population representing the whole undergraduate student body whereas the discipline strands will only cover a subset of students enrolled in given disciplines.
International experts gathered in 2007 and reviewed the various initiatives taken in countries to assess tertiary education learning outcomes. They were impressed with the Collegiate Learning Assessment (CLA) approach taken by the Council for Aid to Education (CAE) in the United States, which measures student ability and learning in critical thinking, writing, and synthesising quantitative and qualitative data. The experts recommended, as part of an AHELO feasibility study, to implement an international pilot test of this instrument to assess the extent to which higher order skills of the type measured by the CLA can be validly measured across different cultural, linguistic and institutional contexts. The generic skills strand will therefore consist in an international pilot test of the CLA, that will look at the CLA in terms of validity, cross-cultural appropriateness and linguistic transferability. As with PISA this direct assessment of student knowledge and ability will be complemented by contextual information which will then enable policy- and practice-related conclusions to be drawn at institutional level.
The CLA measures are focused on skill sets that students will need as they graduate and enter the work force, namely critical thinking, analytical reasoning, problem-solving and written communication. These skills are intertwined, thus the CLA measures are holistic: they require students to use these skills together to respond to tasks. All CLA measures are administered online, using open-ended prompts that require constructed responses. Each task also has an accompanying library of information which students are instructed to use in preparing their answers. Tasks often require students to marshal evidence from these diverse quantitative and qualitative sources and exercise judgment on their relevance. All tasks are appropriate for students across a wide range of undergraduate academic majors and general education programmes.
The international pilot test of the CLA will be carried out in 3-4 countries and 10 HEIs in each of these countries. For more details on specific activities, please read the roadmap .
Discipline strand
A discipline strand is intended to complement the generic skills strand. Indeed, the main limitation of an approach entirely limited to generic competencies, is that it would not assess the kind of subject-matter competencies that many tertiary education departments or faculties would consider their primary objective. There would thus be a risk that what is measured becomes too far removed from what goes on in faculties and departments and does not capture the competencies that are uniquely the province of HEIs.
The discipline strand will seek to assess discipline-related competencies. The experts who convened in 2007 considered that a feasibility study could initially focus on selected discipline areas that are common among HEIs in OECD countries and less likely to be influenced by unique cultural features, and then progressively expand the range of subject areas covered over time.
The aim will be to assess competencies that are fundamental and “above content”, i.e. with the focus on the capacity of students to extrapolate from what they have learned and apply their competencies in novel contexts unfamiliar to them, an approach that is similar to PISA. The intent will be to develop forward-looking definitions of learning outcomes, which acknowledge change in the disciplines and establish instruments around expert thinking about labour-markets, the economy and social well-being over the next decade. This will be facilitated by using a representative set of disciplines that are at the cutting edge. For the purpose of the feasibility study, the experts suggested to focus on engineering and economics could be the focus to get insight into the feasibility of measuring discipline-specific skills in both scientific and social sciences domains, with the understanding that an AHELO would aim at expanding the number of disciplines covered over time.
The discipline strand will explore the feasibility of directly measuring learning outcomes in selected disciplines and across different cultural and linguistic contexts. Planning for this strand of work is now starting. International curriculum and assessment experts will shortly be convened to review existing assessment instruments that focus on discipline-specific skills in engineering and economics, and to recommend options towards an international pilot in 3-4 countries. Possible instruments for consideration include the Brazilian Provão (Exame Nacional de Cursos) or ENADE (Exame Nacional de Desempenho), the Mexican EGEL (Exàmen General para el Egreso de la Licenciatura) or the United States’ Measure of Academic Proficiency and Progress (MAPP) or Major Fields Tests (GRE). Assessment instruments developed by professional associations will also be examined.
In terms of practical implementation, the engineering and economics strands of work will follow a similar model as for the generic skills strand once the instrument to be piloted will have been selected. To be updated on developments in this area, please join the information distribution list .
Value-added measurement strand
The OECD is committed to exploring two types of outcomes measures at the level of HEIs or departments in an AHELO to meet the information needs of the various groups of stakeholders.
• The first one relates to the absolute performance or raw scores of students, since prospective students or employers would want to know the “bottom line” of the performance of HEIs, departments or faculties.
• The second measure seeks to capture the incremental learning (or “value-added”, “gain”) deriving from higher education attendance with a view to assess the quality of the teaching services provided by HEIs. It focuses on the the scores an institution would attain after accounting for the quality of prior schooling or the degree of selectivity of the programmes and HEIs.
However, it has to be recognised that assessing the value-added or marginal gain by HEIs raises a number of scientific and practical issues and the measurement of value-added would impose layers of complexity that are difficult to resolve in large scale assessments. The proposed approach for the AHELO feasibility study is to explore both types of outcomes measures sequentially, and to examine the value-added perspective only conceptually in a first stage, while the pilot measurement of value-added in an international context would be incorporated into the design at a second stage.
The value-added measurement strand will therefore scrutinise possible methods for capturing marginal learning outcomes that can be attributed to HEI’s attendance, both at a theoretical level and in terms of psychometric approaches. It will build upon similar work carried out at school level by the OECD and review options for value-added measurement in tertiary education. Researchers will be invited to study potential data sources, methodologies and psychometric evidence on the basis of datasets existing at the national level, with a view to providing guidance towards the development of a value-added measurement approach for an AHELO. To be updated on developments in this area, please join the information distribution list.
Contextual strand
While the main focus of an AHELO would be to provide HEIs with a tool for development and improvement of teaching and learning, the OECD acknowledges that other types of uses are relevant for different purposes or groups of stakeholders. Therefore the AHELO approach seeks to develop multi-dimensional measures of tertiary education quality, with different emphases to serve the different uses and users. In recognition of this multidimensional approach, the AHELO feasibility study will also include a contextual strand of work that will explore the development of instruments to capture contextual measures at institutional level as well as appropriate indirect proxies of learning outcomes.
A possible model for this strand of work could be to devise contextual information indicators and indirect measures of outcomes of the kind developed by the German Centre for Higher Education (CHE ), and apply them in an international context. The CHE has developed a number of indicators based on a mix of factual data as well as students and teachers’ opinion surveys. These indicators cover various aspects of tertiary education quality, including:
• Academic studies and teaching (contact between students, counseling, courses offered, opportunities for e-learning, study organisation and teaching evaluation);
• Equipment (IT-infrastructure, library, computer workstations, spending for books and journals, rooms);
• International orientation (support for stays abroad);
• Job market and career orientation (employment market related programmes, practice support);
• Research (number of doctorates, publications and internationally visible publications, extent of third party funding);
• Study location and TEI (amount of sport, level of accommodation rent, size of TEI); and
• Overall opinions (study situation, reputation for academic studies and teaching, research reputation).
The emphasis for this strand of work will be to develop internationally comparable indicators of a similar kind. The aim will be to capture institutional characteristics, access patterns (such as the socio-economic makeup of the student body), students’ perceptions of their tertiary education experience as well as a range of outcome measures such as the labour market outcomes of graduates. To be updated on developments in this area, please join the information distribution list .
|
