Skills beyond school

Getting the right data: the assessment instruments for the AHELO feasibility study

 

Generic skills Discipline specific skills 

Contextual informationValue-added

 

The instruments have been developed to be suitable for use in an international context. They have been the object of small-scale validation to provide a sense of cross-linguistic and cross-cultural validity. This work has been conducted through collaboration between the Council Aid to Education (CAE), an international Consortium of experts led by the Australian Council for Educational Research (ACER), the OECD and the national teams for the participating countries/economies.

The test in all three strands (generic skills, engineering and economics) consisted of a mix of multiple choice questions (MCQs) and contructed response tasks (CRTs) and were provided through an online testing platform. These tests were administered in the participating institutions during phase 2 of the project.

 

 Generic skills – an international adaptation of the
Collegiate Learning Assessment
to measure 21stcentury skills

Development and validation

The instrument used for the assessment of generic skills has been adapted from the CAE’s Collegiate Learning Assessment (CLA) which has already been widely administered in the United States. Representatives of participating countries, along with the Council for Aid to Education and the OECD, have finalised the adaptation of two performance tasks.

The translation cycle comprised of the following steps: dual translation of performance tasks by qualified translators; review and reconciliation of the dual translations; test reconciled performance task translations with cognitive labs(these steps were added for the AHELO feasibility Study); reconcile translations across all countries based on cognitive lab findings.

What are cognitive labs? In a cognitive lab a student works through an assessment and, while doing so, verbalizes his or her thinking. As part of the AHELO generic skills Strand study, cognitive labs were intended to ensure that the performance task translations from English into each country’s language:

  1. Did not alter the constructs measured
  2. Were interpreted by the students in the ways originally intended; and,
  3. Were not more difficult for the country’s students to read and understand than it would be if the tasks had been written originally in the country’s language.

Each Generic Skills Strand country conducted cognitive labs with between six and thirteen college students. Some countries offered incentives to students to participate, ranging from 15 to 50 Euros, while other countries did not need incentives to recruit students. The country teams found the cognitive labs to be useful tools in gauging the accuracy of performance task translations.

The test

Students had 120 minutes to work on one of two performance tasks proposed and answer the multiple choice questions. The performance task requires students to use an integrated set of skills - including critical thinking, analytic reasoning, problem solving and written communication - to answer several open-ended questions about a hypothetical but realistic situation. It also requires students to marshal evidence from different sources such as letters, memos, summaries of research reports, maps, diagrams, tables, …

Sample CLA task

 Discipline-Specific skills

The discipline-specific strands measure students’ competences in their field. AHELO looks beyond the simple demonstration of factual knowledge; students are asked to demonstrate "above content" application of knowledge to a new situation.

The Feasibility Study focuses on economics and engineering. In the case of a potential large scale AHELO other disciplines could be considered.


Agreeing on what students should know:
The OECD has worked closely with the Tuning Educational Structures in Europe association and, through that collaboration with a number of recognised experts in the field two reports were prepared on the expected and desired learning outcomes in those two disciplines. These reports have been published as part of the OECD Education Working Papers. They are available free of charge on the OECD iLibrary : Economics report and Engineering report and have served as a basis for the development of the draft frameworks

 

 Economics

Development and validation

Unlike engineers, economists can work for the most part without compulsory professional accreditation. Despite the range of approaches and standpoints held by economists and the intense debates which take place within the discipline, however, there is a common language which enables them to communicate with each other. For example, in their approach to problems, issues, and events modern-day economists typically start with the identification of objectives to be achieved and a recognition of the constraints or resource restrictions that must be confronted in any attempt to achieve those objectives.

The emphasis for AHELO then, is to test students’ ability to use the ‘language of economics’ to solve real-world problems. This involves students to demonstrate that they know and understand basic economics concepts, can apply these concepts, can use appropriate tools to evaluate issues, can analyse data, and can communicate results to a range of audiences.

Over the last year the Educational Testing Services (ETS) in Princeton, New Jersey has led development of the Economics Framework and Economics Assessment. The framework specifies five learning outcomes, all of which students should be able to achieve by the end of their bachelor’s degrees:

  • subject knowledge and understanding;
  • subject knowledge and its application;
  • the ability to make effective use of relevant data and quantitative methods;
  • the ability to communicate to specialists and non-specialists; and
  • the ability to acquire independent learning skills.

 Draft assessment framework in economics

The test

The 90 minute assessment contains one constructed response task and 25 multiple choice items. Both task types avoid prompting students to simply recall factual knowledge and instead focus on ‘above content’ skills including application of concepts, use of appropriate statistical and non-statistical tools, drawing conclusions and recommending policy.

 

 Engineering

Development and validation

AHELO uses a suite of online tests to measure later-year students’ capacity to solve real-world problems by using, applying and acting on their knowledge and reasoning. The assessments employ a wide-range of methods to tap capabilities that both educators and professionals recognise as important for educational success, such as collaboration and teamwork, oral and written communication, creative and analytic ability and leadership.

Engineers globally have already developed a good sense of what graduates should know and be able to do. Learning frameworks exist in many countries, and internationally. These have influenced university level engineering curriculum and pedagogy, the revision of professional accreditation requirements, and the intensification of connections to both professional practice and to school education.

AHELO’s Engineering Framework and Engineering Assessment are designed to measure students’ ability to use the ‘language of engineering’ and display the non-technical competencies that professional engineers must possess. Like the Economics Assessment, the Engineering Assessment is nearing the end of a rigorous process of instrument validation.

The Engineering Framework and Assessment are being developed by an Australian Council for Educational Research (ACER) led consortium of international organisations that includes Japan’s National Institute for Educational Policy Research and a European network of engineers managed by the University of Florence in Italy.

Draft assessment framework in engineering

The test

The 90 minute assessment contains one constructed response task and 30 multiple choice items. Both task types avoid prompting students to simply recall factual knowledge and instead focus on ‘above content’ skills including application of concepts, use of appropriate statistical and non-statistical tools, drawing conclusions and recommending policy.

The performance tasks introduce an authentic engineering scenario or design in a specific context and present students with a set of items related to that context. These modules aim to engage students with interesting, innovative, real-world situations that arise in the profession of civil engineering and have been developed to allow students to demonstrate innovative thinking.

Multiple choice items focus on basic engineering science. They have been included to provide a fast and efficient way to collect data on students’ engineering knowledge, understanding and skills.

 

 Contextual questionnaires

Many variables influence student performance including student background, initial knowledge and skill, programme design, student effort, teaching resources and teaching quality. All students, faculty and institutions participating in AHELO have been asked to complete a brief 10-minute contextual questionnaire to gather context information on these types of input and process variables. AHELO’s explicit focus on learner outcomes examines the net learning effect of all the variables operating together. This allows the attained level of achievement to be assessed and recorded (such as grades on student transcripts), and evaluates of how well the teaching and learning system is working.

A consortium led by the Australian Council for Educational Research (ACER) and incorporating the Centre for Higher Education Policy Studies (CHEPS) and the Center for Postsecondary Research (CPR) developped these questionnaires.

 

How context data would help: with the combination of context data and learning outcomes performance measures, it would be possible to analyse what is distinctive of high-performing institutions (whether in terms of absolute performance, in helping disadvantaged students to complete their studies successfully, or in supporting other specific groups of students such as working or international students) and to identify best practice for each type of goal mission. In doing so, AHELO would contribute to moving from diagnosis to treatment and improvement. Eventually it would help identify what works, for which students and in which contexts. There is huge potential for reducing drop-out rates and enhancing more equitable outcomes.

This is an essential aspect of AHELO if one recalls that on average, 3 out of 10 students entering higher education will drop out without a degree in the OECD. With an average of 53 000 $ spent per higher education student, the costs of failure are great. The social costs for those dropping out are equally high. Students from disadvantaged backgrounds and first generation students suffer more than anyone else, thereby fostering inequity. There is a growing need to ensure that higher education delivers the right set of skills which allow all students to thrive and graduate.

 

The value-added measurement: What each university brings to the learning process

Top universities that attract A+ students and turn out A+ graduate surprise no one. But what about universities that accept B+ students and produce A+ graduates? Which is doing the better job?

AHELO aims to assess both inputs and outputs: what a student brings to a degree programme is as important as what he or she graduates with. The success of a student’s education is greatly influenced by supportive teachers, available resources and an environment conducive to learning (or the lack thereof). By assessing students’ learning gain, a more accurate measure of quality can be determined.

Value-added - or learning gain - has not been measured during the Feasibility Study. The Feasibility Study rather explored the various methodological approaches, potential data sources and psychometric evidence with a view to providing guidance in developing a value-added measurement approach if AHELO does indeed become a full-fledged study. With this objective in mind a literature review on the value-added measurement was undertaken and the conclusions from the meeting of the expert group have been published in volume 3 of the feasibility study report. 

 

Find out more

 AHELO Brochure

- Frequently Asked Questions - FAQs

Latest detailed project update

- Publications, Reports and declassified documents.

 

AHELO Homepage

Contact us: ahelo@oecd.org.

Last modified October 2013

 

Related Documents

 

AHELO Homepage

AHELO Frequently Asked Questions

Reports and Documents

 

Countries list

  • Afghanistan
  • Albania
  • Algeria
  • Andorra
  • Angola
  • Anguilla
  • Antigua and Barbuda
  • Argentina
  • Armenia
  • Aruba
  • Australia
  • Austria
  • Azerbaijan
  • Bahamas
  • Bahrain
  • Bangladesh
  • Barbados
  • Belarus
  • Belgium
  • Belize
  • Benin
  • Bermuda
  • Bhutan
  • Bolivia
  • Bosnia and Herzegovina
  • Botswana
  • Brazil
  • Brunei Darussalam
  • Bulgaria
  • Burkina Faso
  • Burundi
  • Cambodia
  • Cameroon
  • Canada
  • Cape Verde
  • Cayman Islands
  • Central African Republic
  • Chad
  • Chile
  • China (People’s Republic of)
  • Chinese Taipei
  • Colombia
  • Comoros
  • Congo
  • Cook Islands
  • Costa Rica
  • Croatia
  • Cuba
  • Cyprus
  • Czech Republic
  • Côte d'Ivoire
  • Democratic People's Republic of Korea
  • Democratic Republic of the Congo
  • Denmark
  • Djibouti
  • Dominica
  • Dominican Republic
  • Ecuador
  • Egypt
  • El Salvador
  • Equatorial Guinea
  • Eritrea
  • Estonia
  • Ethiopia
  • European Union
  • Faeroe Islands
  • Fiji
  • Finland
  • Former Yugoslav Republic of Macedonia (FYROM)
  • France
  • French Guiana
  • Gabon
  • Gambia
  • Georgia
  • Germany
  • Ghana
  • Gibraltar
  • Greece
  • Greenland
  • Grenada
  • Guatemala
  • Guernsey
  • Guinea
  • Guinea-Bissau
  • Guyana
  • Haiti
  • Honduras
  • Hong Kong, China
  • Hungary
  • Iceland
  • India
  • Indonesia
  • Iraq
  • Ireland
  • Islamic Republic of Iran
  • Isle of Man
  • Israel
  • Italy
  • Jamaica
  • Japan
  • Jersey
  • Jordan
  • Kazakhstan
  • Kenya
  • Kiribati
  • Korea
  • Kuwait
  • Kyrgyzstan
  • Lao People's Democratic Republic
  • Latvia
  • Lebanon
  • Lesotho
  • Liberia
  • Libya
  • Liechtenstein
  • Lithuania
  • Luxembourg
  • Macao (China)
  • Madagascar
  • Malawi
  • Malaysia
  • Maldives
  • Mali
  • Malta
  • Marshall Islands
  • Mauritania
  • Mauritius
  • Mayotte
  • Mexico
  • Micronesia (Federated States of)
  • Moldova
  • Monaco
  • Mongolia
  • Montenegro
  • Montserrat
  • Morocco
  • Mozambique
  • Myanmar
  • Namibia
  • Nauru
  • Nepal
  • Netherlands
  • Netherlands Antilles
  • New Zealand
  • Nicaragua
  • Niger
  • Nigeria
  • Niue
  • Norway
  • Oman
  • Pakistan
  • Palau
  • Palestinian Administered Areas
  • Panama
  • Papua New Guinea
  • Paraguay
  • Peru
  • Philippines
  • Poland
  • Portugal
  • Puerto Rico
  • Qatar
  • Romania
  • Russian Federation
  • Rwanda
  • Saint Helena
  • Saint Kitts and Nevis
  • Saint Lucia
  • Saint Vincent and the Grenadines
  • Samoa
  • San Marino
  • Sao Tome and Principe
  • Saudi Arabia
  • Senegal
  • Serbia
  • Serbia and Montenegro (pre-June 2006)
  • Seychelles
  • Sierra Leone
  • Singapore
  • Slovak Republic
  • Slovenia
  • Solomon Islands
  • Somalia
  • South Africa
  • South Sudan
  • Spain
  • Sri Lanka
  • Sudan
  • Suriname
  • Swaziland
  • Sweden
  • Switzerland
  • Syrian Arab Republic
  • Tajikistan
  • Tanzania
  • Thailand
  • Timor-Leste
  • Togo
  • Tokelau
  • Tonga
  • Trinidad and Tobago
  • Tunisia
  • Turkey
  • Turkmenistan
  • Turks and Caicos Islands
  • Tuvalu
  • Uganda
  • Ukraine
  • United Arab Emirates
  • United Kingdom
  • United States
  • United States Virgin Islands
  • Uruguay
  • Uzbekistan
  • Vanuatu
  • Venezuela
  • Vietnam
  • Virgin Islands (UK)
  • Wallis and Futuna Islands
  • Western Sahara
  • Yemen
  • Zambia
  • Zimbabwe
  • Topics list