The quality of the exam questions and the evaluation of candidates is an important consideration. The quality of the exam questions matters for all formats of the exam (written exam, WBA and oral exam), while the quality of evaluation and the skills of assessors are especially relevant for exam formats where there is no single correct answer, and the evaluation is more nuanced.
Countries that have a certification framework have been taking measures to ensure the quality of the exam. In some countries, the exam questions are reviewed by third parties. For example, in Israel, the exam questions are reviewed by the central purchasing body and the legal bureau at the Ministry of Finance before they can be included in the exam. In the United Kingdom, the assessment centre has been designed by commercial subject matter experts and occupational psychologists to ensure that they evaluate candidates in a logical and fair way. In Peru, the preparation of the exam questions involves experts specialised in the preparation of tests, psychometric analysis as well as subject matter experts in public procurement.
Lithuania and Norway adopted data analytics approaches ensure that the evaluation conforms to high standards. In Lithuania, the questions with the highest error rates are identified, reviewed, and, if necessary, revised for future exams. In Norway, the provider of the certification exams analyses the quality of exam questions at least once a year. The test provider analyses the difficulty level and correlation of the answers. Based on the analysis results, the DFØ (Public Procurement Authority of Norway) adjusts the wording of questions and answers in the certification tests. Some countries, including Austria and Peru follow international quality management standards such as the ISO 17024 (standard for the Certification of persons).
Unlike with single or multiple choice questions, open questions, essays, and WBAs as well as oral examination require capability of staff who evaluate and certify the candidates. The United Kingdom, which has a 100% oral examination-based certification framework has a group of 60 to 80 trained assessors who are either senior active internal commercial managers, or external senior commercial professionals that are paid a daily rate. Each candidate is assessed by several assessors to remove bias and provide an accurate view of the candidate.
The verification of exam questions in terms of both content and format is a clear best practice. The regular monitoring and updates to the exam to ensure that they remain relevant and in line with the evolving practices of public procurement is another key consideration.