Skills beyond school

Evaluation of the AHELO Feasibility Study Outcomes


The design of the AHELO feasibility study makes plans for the evaluation of its outcomes. The assessment design and analysis plan will outline the operational criteria and thresholds to be used to define success for each of the technical challenges that have to be addressed as part of the AHELO feasibility study.

The main criteria to assess the success of the feasibility study will be to examine evidence from its processes and results, and use this evidence to draw conclusions as to the scientific and practical feasibility of a generalised assessment.

Scientific feasibility will be assessed with these key questions such as:

  • Is it possible to develop instruments to capture learning outcomes that are perceived as valid in diverse national and institutional contexts?
  • Do the test items perform as expected and do the test results meet pre-defined psychometric standards of validity and reliability?
  • Is it possible to score higher-order types of items consistently across countries?
  • Is it possible to capture information on teaching and learning contexts that contribute to explaining differences in student performance?

Practical feasibility will be assess with key questions such as:

  • How effective are strategies implemented at national/institutional level to secure institutional and student cooperation?
  • Can students be motivated to take part in such an assessment and take it seriously?
  • To what extent does the implementation of the feasibility study assessments bring benefits to participating HEIs?
  • To what extent does the implementation of the feasibility study contribute to demonstrating its value for the improvement of teaching and building support for an AHELO?

The evaluation of the AHELO feasibility study outcomes will therefore involve extensive and careful psychometric analysis of the AHELO feasibility study data, but also technical reviews by international experts as well as the international conference in late 2012 gathering technical experts, country representatives and different stakeholder groups to discuss whether, and how, to take the results from the feasibility study forward.

In case of positive results, this proof of concept would constitute a key pillar for longer-term work as it would assist OECD countries in deciding whether to launch a fully-fledged AHELO main study.