Evaluating, Updating and Monitoring Anti‑Fraud Strategies

Annex B. Catalogue of outcome and impact-level indicators

What is the purpose of the Catalogue?

• It improves the quality design of existing NAFS

• It may serve as a reference for designing future NAFS (designing objectives, assigning actions, refining measures)

• It supports the NAFS co-ordinator and implementing agencies in tracking outcomes and impact produced by the NAFS, not only deliverables (such as outputs, e.g. no. of laws, trainings, methodologies)

• It supports monitoring and evaluation functions

Who should use it?

• NAFS co-ordinator and implementing agencies

Key features/components:

• Indicators are broken down into anti-fraud cycle phases

• Impact indicators are paired with desired impact (in NAFS, the desired impact corresponds to Level 1 objectives – often referenced as Core objectives, Main Goals or Strategic Goals)

• Outcome indicators are paired with the desired outcome (in NAFS, the desired outcome corresponds to Level 2 objectives – often referenced as Specific objectives or Operational objectives)

• Inclusion of potential data sources

• Specifying whether the indicator is qualitative or quantitative

• Section pertaining to comments, assumptions or risks

• Covers typical strategic objectives in anti-fraud strategies, e.g. on capacity-building, improving co-operation and collaboration, increasing digitalisation, etc.

Tool Complementarity and Interlinkages:

• Informs the design of the NAFS Sample Questionnaire for Evaluation (Annex A)

Guidance on application and limitations:

• Indicators draw on recurring risks and challenges identified in the practices and experiences of EU Member States, PIF reports, ECA audits, EPPO findings and OECD TSI project findings. The indicators are intended to guide readers and should not be understood as prescribed interventions. Indicators were designed to maintain a clear line of sight towards the intended impact, making it easier for authorities to define corresponding measures and establish baselines where needed

• Some of the expected impacts or outcomes and their associated performance indicators are deliberately generic. With this said, they can be adapted easily to national realities

• Indicators will not always be readily available in existing practices of EU Member States. In some areas, enabling conditions must be put in place. For example, training-related indicators may require changes to training arrangements (e.g. introducing pre- and post-training assessments and a refresher check after a defined period). For whistleblower protection indicators, this may require an agreed protocol between the central NAFS co-ordinator (e.g. the AFCOS) and the agency responsible for whistleblowing to allow relevant anti-fraud tracking. While these steps require additional efforts, they can significantly improve the quality and usefulness of implementation and assessment

• Co-ordinating authorities do not need to create a separate data collection tool for every indicator. Several indicators may be supported by the same data source (e.g. a single beneficiary survey with multiple relevant questions)

• The catalogue encourages adoption of pragmatic solutions. For instance, where the NAFS co-ordinator’s (e.g. the AFCOS) reports are identified as a potential data source, authorities may adjust reporting templates and internal procedures to capture the relevant information more systematically

• The catalogue also highlights synergies with other national strategies. Where possible, data collection and reporting should be aligned to avoid duplication and to complement existing strategic arrangements

A) PREVENTION PHASE

IMPACT-LEVEL INDICATORS

Impact indicators measure long-term systemic change at the level of NAFS core objectives (Level 1 / Strategic Goals). These correspond to the ultimate goals the strategy seeks to contribute to over its entire implementation period. Impacts are typically correlated with core/strategic objectives or high-level directions of the strategy.

 

Desired Impact	Indicators to measure impact	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Anti-fraud culture is understood and supported across all staff levels and beneficiaries	% of staff in such institutions who perceive the institutional anti-fraud culture as strong % of staff in such institutions who perceive senior management as committed to establishing an anti-fraud tone % of staff/ beneficiaries who report high awareness of the whistleblower protection	NAFS Evaluation survey NAFS Evaluation reports	Quantitative	Public awareness and transparency	Culture change might be slow, therefore these indicators should be interpreted alongside other evidence Design of the evaluation survey must mitigate social desirability bias and protect anonymity At the EU level, whistleblower protection awareness remains a weak point, which might require a standalone impact indicator rather than an outcome indicator
Strengthened frameworks, awareness and transparency to prevent fraud	% of citizens who believe fraud against EU funds is effectively addressed in their country % of entities managing EU funds with anti-fraud prevention measures assessed as effective by audit authorities Return on Investment (ROI) per prevention mechanism (e.g. training, data analytical tools) in place	Eurobarometer on citizens' perceptions of fraud and the fight against fraud Audit Authorities reports	Quantitative	Fraud risk management	ROI offers a simplified metric that expresses these results as a ratio, thereby facilitating communication with decision makers. However, ROI typically captures only monetised impacts and should therefore be interpreted alongside broader evidence on non-financial outcomes. More details on this can be found on pages 31-37 of this report

OUTCOME-LEVEL INDICATORS

Outcome indicators measure short-term and medium-term institutional and behavioural change at the level of NAFS specific objectives (Level 2 / Operational Objectives). They show whether interventions (outputs) are producing intended changes in practices, knowledge, systems, and behaviours among target groups (MAs, IBs, beneficiaries, judicial authorities). Outcomes are typically correlated with operational objectives and, to some extent, with core/strategic objectives.

 

Desired Outcome	Indicators to measure outcomes	Data sources	Qualitative/ Quantitative	Strategic category (tag)	Comments/assumptions/risks
Increased capacity to prevent fraud, facilitated through training programmes	% of staff demonstrating improved knowledge on fraud prevention in post-training assessment % of anti-fraud authorities that reviewed procedures and fraud risk management frameworks based on training participants’ inputs	Staff survey (pre/post and follow-up training surveys) Meeting minutes/list of participants NAFS Monitoring reports NAFS Evaluation reports	Quantitative	Capacity-building and training	Training programmes should be designed from the outset to measure learning outcomes, including baseline knowledge (prior to delivery) and immediate post-training results. The design should also include a follow-up assessment (e.g. a refresher after 6 months) to gauge retention. Training formats should be practice-oriented (e.g. real case work, scenario analysis, etc.) Knowledge on fraud prevention should cover all relevant topics such as conflicts of interest, whistleblower protection and reporting mechanisms, fraud risk assessments, risk controls, etc.
Enhanced fraud risk management frameworks in MA/IB	% fraud risk assessments informed by contributions from multiple relevant stakeholders (e.g. AFCOS, investigative/prosecution bodies or other specialised expert agencies) % of MAs/IBs perceiving the fraud risk frameworks as effective	AFCOS's analysis and reports ECA Reports NAFS Monitoring reports NAFS Evaluation reports	Quantitative	Fraud Risk Management	Design of the evaluation survey must mitigate social desirability bias and protect anonymity
Beneficiaries are better informed about fraud risks and reporting channels	% of beneficiaries aware of their obligations regarding of fraud and irregularities % of beneficiaries who can correctly identify at least 3 common fraud risks relevant to their programme	Beneficiaries` survey Focus groups with beneficiaries	Quantitative Qualitative	Public awareness and transparency	Beneficiaries are often the entry point for fraud, and their awareness is a prerequisite for prevention This outcome requires NAFS specific measures aimed at beneficiaries, such as periodic awareness surveys and regular focus groups designed to collect qualitative evidence on their level of understanding of fraud risks and familiarity with prevention mechanisms
Strengthened conflict of interest management within all entities managing EU funds	% of staff correctly identifying cases of conflict of interest and management options % entities with Standard Operational Procedures that include disclosing, abstaining and resolving conflicts of interest in the ex ante phase % of public procurement procedures subject to automated conflict of interest screening before contract awards % of anti-fraud authorities using dedicated tools for the detection of undisclosed conflict of interest in public procurement procedures (e.g. automatic company registry screening) % of actual/apparent/perceived conflict of interest cases identified through automated screening tools compared to manual verification % of effectively resolved cases of conflicts of interest out of the total cases identified/reported	Staff survey (pre/post and follow-up training surveys) Audit Authority reports Public Procurement Agency reports Corruption Prevention / Integrity Agencies assessments	Quantitative Qualitative	Fraud Risk Management Digitalisation and Data analytics	Training programmes should be designed from the outset to measure learning outcomes, including baseline knowledge (prior to delivery) and immediate post-training results. The design should also include a follow-up assessment (e.g. a refresher after 6 months) to gauge retention. Training formats should be practice-oriented (e.g. real case work, scenario analysis, etc.) The outcome indicator related to public procurement procedures screened for conflict of interest measures the shift to automated screening on all public procurement procedures When developing/tailoring indicators for this desired outcome, drafters might want to check the coherence/synergies with anti-corruption policies/strategy
Increased awareness on whistleblower protection and reporting mechanisms	% of beneficiaries aware of fraud reporting channels % of public officials / civil servants who are aware of existing whistleblower reporting channels and protections % of anti-fraud authorities which are compliant with the national whistleblower protection law # of confidential and nominal advice requested to assist whistleblower dilemmas #of whistleblower reports on fraud issues that received a substantive response within the legally prescribed timeframe on internal and external channels # of visits to whistleblowing reporting tool (internal or external channels)	Beneficiaries’ survey Staff surveys Eurobarometer Reports of whistleblowers oversight agency	Quantitative	Whistleblower protection and reporting	Beneficiaries are often the entry point for fraud, and their awareness is a prerequisite for prevention When developing/tailoring indicators for this desired outcome, drafters might want to check the coherence/synergies with anti-corruption policies/strategy
Promote a culture of integrity, including transparency and public accountability of anti-fraud measures	% of MAs publishing statistical information on the use of EU funds, including contracts concluded and established irregularities	MA’s Websites / Social media channels review	Quantitative	Public awareness and transparency	When communicating statistical information, authorities must decide on the channel type (website, platform, unified portal), frequency of communication and minimum content

B) DETECTION PHASE

IMPACT-LEVEL INDICATORS

Impact indicators measure long-term systemic change at the level of NAFS core objectives (Level 1 / Strategic Goals). These correspond to the ultimate goals the strategy seeks to contribute to over its entire implementation period. Impacts are typically correlated with core/strategic objectives or high-level directions of the strategy.

 

Potential Expected Impact	Indicators (that measure impact)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Proactive detection of fraud through systematic use of data-driven tools and intelligence-led approaches	% of suspected fraud detected through automated data systems (analytics, risk scoring, cross-checks) to total detected fraud Average time from fraud occurrence to detection	Irregularity reporting data / AFCOS reports	Quantitative	Digitalisation and Data analytics	Trends should also be sustainable over the programming/implementation period to qualify as a long-lasting impact of NAFS
Decrease in estimated fraud and error losses affecting EU funds over the NAFS implementation period	% of estimated fraud and error loss compared to baseline	Fraud Loss Measurement (FLM) reports or analysis of known fraud cases and historical detection data (e.g. historical detection rates, average financial impact per case)	Quantitative	Fraud Risk Management	The strategy should provide at least two supporting activities with the following outputs: 1) the adopted methodology for estimating fraud and error losses (the IPSFF Fraud Loss Measurement framework referenced in the current report provides a useful five-stage methodological model) and 2) the estimated baseline using the adopted methodology. The resulting baseline should be incorporated into the NAFS/action plan and used as an impact indicator. Values obtained during the NAFS’ final evaluation should be compared against the baseline to assess the NAFS’ contribution to reducing fraud losses. Consistent application of the same methodology is essential for comparability.

OUTCOME-LEVEL INDICATORS

Outcome indicators measure short-term and medium-term institutional and behavioural change at the level of NAFS specific objectives (Level 2 / Operational Objectives). They show whether interventions (outputs) are producing intended changes in practices, knowledge, systems, and behaviours among target groups (MAs, IBs, beneficiaries, judicial authorities). Outcomes are typically correlated with operational objectives and, to some extent, with core/strategic objectives.

 

Potential Expected Outcome	Indicators (that measure outcome)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Enhanced proactive fraud detection through data integration and automation	% of risk alerts generated by Arachne+ that result in documented follow-up action (verification, investigation, or reasoned dismissal) % of anti-fraud authorities using automated data cross-checks (e.g. cross-referencing tax, social security, procurement and company registry data) % of red flags triggered that lead to detection of an irregularity or suspected case of fraud	AFCOS reports NAFS Evaluation survey Arachne system usage statistics (if available	Quantitative Qualitative	Digitalisation and Data analytics	Arachne+ will become mandatory from 2028 Interoperability (systems exchange data and interpret automatically) matters more than technical connectivity
Improved quality and timeliness of irregularity reporting	% reduction in average time elapsed between detection of an irregularity and its complete and accurate registration in the IMS % of irregularity reports assessed as complete and accurate	Irregularity reporting data NAFS Evaluation survey EC feedback	Quantitative	Co-operation	Data completeness and accuracy primarily depend on the reporting authorities
Clarifying the management of infringement reports and enhancing feedback loops between competent authorities (administrative, investigative and judicial)	% of anti-fraud authorities that self-assess improved inter-institutional co-operation under the MoUs % referrals resulting in detected fraud (investigation, audit, mitigation measures)	NAFS Evaluation survey NAFS Monitoring reports AFCOS reports	Quantitative Qualitative	Co-operation	Efforts should target increasing co-operation with investigation/prosecution entities
Strengthened effectiveness of on-the-spot checks based on standardised methodology	% increase of irregularities detected per on-the-spot check compared to baseline before standardised methodology was introduced % of irregularities detection from on-the-spot checks compared to desk-based checks	Anti-fraud authorities verification logs Audit authority reports	Quantitative	Capacity-building and training	Methodology exists (output) differs from methodology producing better detection (outcome)
Whistleblower reporting effectively contributes to fraud detection	% of whistleblower reports that resulted in opened investigations % of whistleblower reports leading to confirmed irregularities	Reports of whistleblowers oversight agency	Quantitative	Whistleblower protection and reporting	Beneficiaries are often the entry point for fraud and their reporting can contribute to better detection When developing / tailoring indicators for this desired outcome, drafters might want to check the coherence / synergies with anti-corruption policies / strategy

C) INVESTIGATIONS PHASE

IMPACT-LEVEL INDICATORS

Impact indicators measure long-term systemic change at the level of NAFS core objectives (Level 1 / Strategic Goals). These correspond to the ultimate goals the strategy seeks to contribute to over its entire implementation period. Impacts are typically correlated with core/strategic objectives or high-level directions of the strategy.

 

Potential Expected Impact	Indicators (that measure impact)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Improved quality and efficiency of fraud cases investigation	% of substantiated fraud cases out of total suspected fraud cases % of investigations to prosecutions ratio Average duration from fraud referral to concluded investigation	Prosecutor`s Office statistics EPPO statistics AFCOS reports National judicial statistics (alternatively ECA reports) EU Justice Scoreboard data on case duration (limited to anti-corruption cases)	Quantitative Qualitative	Co-operation	For a better analysis, NAFS should provide disaggregation by fraud type When developing/tailoring indicators for this desired impact and the associated indicator referring to the duration of investigation, drafters might want to check the coherence/synergies with anti-corruption and justice policies/strategy

OUTCOME-LEVEL INDICATORS

Outcome indicators measure short-term and medium-term institutional and behavioural change at the level of NAFS specific objectives (Level 2 / Operational Objectives). They show whether interventions (outputs) are producing intended changes in practices, knowledge, systems, and behaviours among target groups (MAs, IBs, beneficiaries, judicial authorities). Outcomes are typically correlated with operational objectives and, to some extent, with core/strategic objectives.

 

Potential Expected Outcome	Indicators (that measure outcome)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Strengthened investigative capability, tools and procedures	# of investigators/prosecutors demonstrating improved competence in fraud investigation techniques (pre/post competence assessment) % of fraud cases investigated using the standardised fraud investigation protocols % of complex fraud cases referred to the specialised investigation unit % resolution rate of cases referred to investigative authorities out of total cases referred	AFCOS reports Training records Quality control checklists/police case management system Case classification logs; referral decision records	Quantitative	Capacity-building and training	Specialised capacity is a prerequisite for quality investigations

D) RECOVERY and SANCTIONS PHASE

IMPACT-LEVEL INDICATORS

Impact indicators measure long-term systemic change at the level of NAFS core objectives (Level 1 / Strategic Goals). These correspond to the ultimate goals the strategy seeks to contribute to over its entire implementation period. Impacts are typically correlated with core/strategic objectives or high-level directions of the strategy.

 

Potential Expected Impact	Indicators (that measure impact)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Dissuasive, proportionate and visibly enforced sanctions for fraud	% of citizens who believe that those who commit fraud against EU funds are sanctioned % of sanctions or judicial decisions enforced within statutory deadlines Average duration of court proceedings for fraud affecting EU funds	Court records National judicial statistics (alternatively ECA reports) EU Justice Scoreboard data on case duration (limited to anti-corruption cases) Eurobarometer	Quantitative Qualitative	Sanctions	Visibility of sanctions might have a deterrent effect
Increased proportion of fraud-related financial losses successfully recovered	% of amounts recovered out of to the total amounts identified for recovery Average time from definitive decision to completion of recovery	ARO statistics/reports Court records	Quantitative	Recovery	When developing/tailoring indicators for this desired impact and associated indicators referring to duration, drafters might want to check the coherence/synergies with anti-corruption and asset recovery policies / strategy

OUTCOME-LEVEL INDICATORS

Outcome indicators measure short-term and medium-term institutional and behavioural change at the level of NAFS specific objectives (Level 2 / Operational Objectives). They show whether interventions (outputs) are producing intended changes in practices, knowledge, systems, and behaviours among target groups (MAs, IBs, beneficiaries, judicial authorities). Outcomes are typically correlated with operational objectives and, to some extent, with core/strategic objectives.

 

Potential Expected Outcome	Indicators (that measure outcome)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments/assumptions/risks
Strengthened asset recovery and freezing	Value of assets frozen or restrained through civil/confiscation orders related to unexplained wealth % of asset recovery cases initiated on the basis of unexplained wealth (civil confiscation) vs. criminal-confiscation cases	ARO reports AFCOS reports Public databases	Quantitative	Recovery	Additional legislative and operational measures may be required to support the achievement of this outcome.
Strengthened guidance and awareness to enhance the recovery of undue payments	% of MA/IB staff demonstrating improved knowledge on recovery guidelines and procedures % increase in amounts recovered from fraud-related cases compared to previous periods % of undue payments successfully recovered compared to the total value of irregularities/fraud identified annually	Staff survey (pre/post and follow-up training surveys) ARO reports AFCOS reports MA/IB irregularity registers	Quantitative	Capacity-building and training Recovery	Training programmes should be designed from the outset to measure learning outcomes, including baseline knowledge (prior to delivery) and immediate post-training results. The design should also include a follow-up assessment (e.g. a refresher after 6 months) to gauge retention. Training formats should be practice-oriented (e.g. real case work, scenario analysis, etc.)
Enhanced efficiency of legal proceedings for cases of fraud	% average duration of first-instance court proceedings in fraud cases % of fraud cases concluded without exceeding average duration of cases # of fraud cases dismissed due to statute of limitations expiration	Court service data (national judicial statistics)	Quantitative	Court proceedings	This indicator requires that national judicial statistics disaggregate case duration data for fraud cases
Increased visibility and consistent application of sanctions for fraud	# of entities excluded from EU funding and for which the exclusion decision was made public % of confirmed fraud cases resulting in administrative or criminal sanction within a defined timeframe % of anti-fraud authorities that assess the visibility, consistency and deterrent effect of sanctions as high % of non-governmental stakeholders that assess the visibility, consistency and deterrent effect of sanctions as high	Court records NAFS Evaluation Survey Beneficiaries survey	Quantitative Qualitative	Sanctions Public awareness and transparency

NAFS Monitoring and Evaluation

OUTCOME-LEVEL INDICATOR

Outcome indicators measure short-term and medium-term institutional and behavioural change at the level of NAFS specific objectives (Level 2 / Operational Objectives). They show whether interventions (outputs) are producing intended changes in practices, knowledge, systems, and behaviours among target groups (MAs, IBs, beneficiaries, judicial authorities). Outcomes are typically correlated with operational objectives and, to some extent, with core/strategic objectives.

 

Potential Expected Outcome	Indicators (that measure outcome)	Data sources	Qualitative / Quantitative	Strategic category (tag)	Comments / assumptions /risks
NAFS implementation improves through evidence-based learning and adaptation	% of evaluation recommendations from the previous NAFS implementation cycle is implemented in the next cycle % of NAFS action plan measures or indicators modified, added or removed based on documented evidence from monitoring data or evaluation findings	NAFS Evaluation Report Maturity Evaluation Report on NAFS External evaluation PIF reports	Quantitative	Monitoring, Evaluation & Learning	According to international standards and benchmarks, M&E should also be used as a learning tool