Evaluating, Updating and Monitoring Anti‑Fraud Strategies

Annex A. Sample questionnaire for evaluation

What is the purpose of the sample questionnaire?

• It supports the evaluation (mid-term/end-term) process

• It informs the update of NAFS / the drafting of new NAFS

• It improves the quality design of existing NAFS (and thus, compliance with EC standards)

• It produces qualitative reporting (gathers perceptions and evidence from anti-fraud authorities on all anti-fraud cycle phases)

• It sets the general framework when the evaluation is outsourced or mixed

Who should use it?

• NAFS co-ordinator (lead) and implementing agencies (supporting)

• Other stakeholders: other public sector entities, beneficiaries of EU funds, business sector organisations, NGOs, academia, etc.

Key features/components:

• Questions based on OECD – DAC evaluation criteria

• Fixed-choice, multiple-choice checklists and open-ended questions/text boxes

• Likert scale on agreement and extent

Tool Complementarity and Interlinkages:

• This tool complements the ‘Catalogue of outcome and impact-level indicators.’ Several questions from the survey extract perceptions and evidence to support the assessment of impact and outcome indicators from the Catalogue

Guidance on application of the tool and limitations:

• The survey is designed to serve as a generic evaluation tool, applicable to national stakeholders in conducting the evaluation of the NAFS

• The survey gathers views and evidence from key stakeholders on whether current or past NAFS and associated action plans are working/worked as intended and have managed to fulfil initial objectives

• The survey contains two parts: A – implementing agencies: AFCOS, MAs, IBs, audit authorities, tax administration law enforcement, judiciary and other entities involved in implementation of the NAFS and B – other stakeholders such as beneficiaries of EU funds, business sector organisations, NGOs, academia, etc., who are not directly responsible for implementation of the NAFS but whose experience and perceptions are essential for a complete evaluation

• Within the evaluation exercise, survey questionnaires are typically sent by the evaluator (in this case, the co-ordinating agency of the NAFS) to implementing agencies and other stakeholders each time the evaluation (mid-term, end-term) is deployed

• The survey is designed based on the assumption that the NAFS is organised around a three-level architecture comprising: Level 1 – strategic objectives; Level 2 – operational objectives; and Level 3 – concrete actions or measures. Accordingly, the terminology used throughout the survey reflects this structure

• The survey questions are presented as generic model questions and do not capture the specific institutional, legal, or policy particularities of individual countries. As such, they are not designed, in their current form, to measure country-specific outcomes or impacts. They should therefore be reviewed and tailored to the relevant national context before being used for evaluation purposes

A: Survey for evaluation – implementing agencies

Cover page:

0. Introduction to the participating stakeholders of the survey on the mid-term/end-term evaluation of the NAFS.

List the following:

• Objectives and structure of the survey

• Description of the survey (number and content of the sections)

• Instructions on how to fill in the survey

• Deadlines and coverage

• Expected time of completion

• Contact person

Introductory details

1. Responding entity: (text box)

2. Type of entity: (select one)

• AFCOS / NAFS co-ordinating body

• Managing Authority

• Intermediate Body / Implementing Body

• Audit authority

• Law enforcement body

• Prosecution service

• Revenue administration / Customs authority

• Other (please specify)

3. Contact person: name, position, department, email/phone no. (text box)

Section 1: Relevance

Relevance examines whether the NAFS objectives and design respond to the actual needs, policies, and priorities related to the protection of the national and EU's financial interests and whether they continue to do so as circumstances change.

1.1. The strategic goals of the NAFS adequately reflect the current needs and priorities related to the protection of the national and EU's financial interests:

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) If you disagree, which needs or priorities are/were not sufficiently addressed?

(Text box)

1.2. The operational objectives and actions in the NAFS action plan are clearly linked to the strategic goals

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

1.3. The NAFS has been updated or revised in response to changes in the fraud landscape (e.g. new fraud risks, new EU funding instruments, changes in the institutional setup, and adoption of new laws)

(Yes / No / Don’t know)

a) If yes, describe what triggered the update and what was changed

(Text box)

1.4. Relevant stakeholders (both internal and external to the anti-fraud system) were consulted when setting the objectives and actions of the NAFS

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

1.5. The performance assessment indicators (output level indicators) in the NAFS are useful for monitoring the level of achievement of objectives, i.e. that is, they provide meaningful, objective, and informative data

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Which indicators worked well and which have not?

(Text box)

1.6. Since the NAFS was adopted, new risks or trends have emerged in fraud patterns that are not yet reflected in the strategy

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

1.7. The NAFS was informed by a national-level fraud risk assessment

(Yes / No / Don’t know)

1.8. There was a structured and documented process for setting the objectives and actions of the NAFS and its action plan (e.g. the following items were used in the process: evaluation report of predecessor strategies, relevant survey results, needs analysis, stakeholder mapping, etc.)

(Yes / No / Partially / Don’t know)

a) Describe the process, if applicable:

(Text box)

Section 2: Coherence

Coherence examines how well the NAFS fits with other interventions: internal (such as consistency among the NAFS's own objectives and actions) and external (alignment with other national policies and EU-level frameworks).

2.1. The NAFS is well aligned with other relevant national strategies and policies (e.g. anti-corruption strategy, sectoral anti-fraud action plans, national recovery and resilience plan, etc.)

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Where do you see the main overlaps or gaps (if applicable)

(Text box)

2.2. The different objectives of the NAFS are designed in a way that reinforces one another and creates synergies (e.g. prevention objectives connect logically to recovery and detection objectives)

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Where do you see potential disconnections (if applicable)

(Text box)

2.3. The NAFS is well aligned with EU-level requirements and frameworks relevant to the protection of the EU's financial interests (e.g. the PIF Directive; the EU Whistleblowing Directive; the EU Anticorruption Directive)

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Which EU requirements or frameworks are insufficiently reflected in the NAFS?

(Text box)

2.4. The roles and responsibilities of different institutions involved in the anti-fraud cycle are clearly defined in NAFS and do not conflict with one another

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What roles and responsibilities are not clear or conflict with one another (if applicable)

(Text box)

2.5. Co-operation agreements or Memorandums of Understanding (MoUs) between institutions involved in the anti-fraud system are in place and operational

(Yes / No / Partially / Don’t know)

Section 3: Effectiveness

Effectiveness assesses the extent to which the NAFS is achieving its objectives and targets. It also examines the major factors influencing the achievement or non-achievement of those objectives.

3.1. The measures implemented under the NAFS have contributed to improved fraud prevention

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What evidence supports your assessment?

(Text box)

3.2. The measures implemented under the NAFS have contributed to improved fraud detection

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What evidence supports your assessment?

(Text box)

3.3. The measures implemented under the NAFS have contributed to improved fraud investigation

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What evidence supports your assessment?

(Text box)

3.4. The measures implemented under the NAFS have contributed to the effective application of sanctions

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What evidence supports your assessment?

(Text box)

3.5. The measures implemented under the NAFS have contributed to the effective recovery of funds

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What evidence supports your assessment?

(Text box)

3.6. Your institution has adopted and actively implements a clearly defined anti-fraud policy, including a policy to combat conflicts of interest

(Yes / No / Partially / Don’t know)

3.7. Staff in your institution regularly use risk assessment tools to identify, assess, and mitigate fraud risks

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) How frequently are fraud risk assessments conducted?

¨ Bi-annually

¨ Annually

¨ Other (please specify__________)

¨ None

3.8. Your institution considers its fraud risk management frameworks to be effective in identifying and mitigating fraud risks

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Were the fraud risk assessments conducted with additional input from other knowledgeable actors (such as AFCOS, investigative and prosecution bodies or other specialised expert agencies)?

¨ Yes (which one_______)

¨ No

¨ Partially

¨ Don’t know

3.9. Your institution regularly uses data analytics tools (e.g. ARACHNE) for the proactive detection of suspected fraud

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What specific tools were used? What worked well and what tools need improvement?

(Text box)

3.10. Your institution has a functional internal whistleblower reporting channel and staff are aware of how to use it

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

3.11. AFCOS effectively fulfils its co-ordination role across all EU-funded areas and across all phases of the anti-fraud cycle

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What could be improved in AFCOS`s co-ordination role?

(Text box)

3.12. There is a clear division of responsibilities in terms of who does what to achieve the objectives assigned to your institution under the NAFS action plan

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) How should responsibilities be clarified in the future?

(Text box)

3.13. Over the current strategy period, reported incidents of suspected fraud have:

(Increased / Decreased / Stayed the same / Don't know)

a) What were the main contributing factors?

(Text box)

3.14. In your institution's assessment, the average time from fraud occurrence to detection has decreased during the current strategy period / past strategy period:

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

3.15. What are the main challenges your institution faced in implementing the NAFS action plan? Select all that apply:

¨ Insufficient human resources

¨ Insufficient financial resources

¨ Lack of technical know-how or specialised skills

¨ Lack of time / unrealistic deadlines

¨ Unclear division of responsibilities between institutions

¨ Insufficient co-operation or information-sharing between institutions

¨ Lack of political support or prioritisation

¨ Insufficient data or monitoring tools

¨ Lack of clear guidance or methodology

¨ Other (please specify) ______________

3.16. What have been the main success factors that enabled your institution to implement the NAFS action plan effectively? Select all that apply:

¨ Strong political support and leadership commitment

¨ Adequate staffing and financial resources

¨ Clear legal framework and mandates

¨ Effective co-operation and information-sharing between institutions

¨ Specialised training

¨ Use of data analytics or IT tools

¨ Support from EU institutions or programmes (OLAF, EU-funded technical assistance, etc.)

¨ Other (please specify) __________

Section 4: Efficiency

Efficiency examines how well human, financial, technical and time resources are used to deliver results.

4.1. Your institution has/had sufficient human resources (in terms of number and skills) to carry out the anti-fraud activities assigned to it under the NAFS action plan

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What is needed to address gaps? (e.g. additional staff, specialised training, external expertise)

(Text box)

4.2. Your institution has/had sufficient financial resources to carry out the anti-fraud activities assigned to it under the NAFS action plan

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

4.3. Anti-fraud training is conducted regularly for staff involved in the management and control of EU funds

(Yes / No /Don’t know)

a) Which topics require additional training? (e.g. management of conflict of interest)

(Text box)

4.4. The deadlines set in the NAFS action plan for completing activities are/were realistic and achievable

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Which activities or actions have / had unrealistic deadlines?

(Text box)

4.5. How many activities assigned to your institution in the NAFS action plan were fully completed within the indicated deadlines during the current period?

¨ 100%

¨ Over 90%

¨ Between 75-90%

¨ Between 50-75%

¨ Between 25-50%

¨ Less than 25%

a) For activities that were not completed on time, what were the main reasons?

(Text box)

4.6. The activities assigned to your institution in the action plan are/were clearly defined (e.g. is/was clear what is required to implement them)

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

4.7. Your institution has sufficient data to monitor progress towards the objectives and activities assigned to it

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) For which objectives and/or activities and their associated indicators are data insufficient?

(Text box)

4.8. How frequently does your institution monitor progress against the NAFS action plan indicators?

¨ Monthly

¨ Quarterly

¨ Other (please specify__________)

¨ None

4.9. The measures required under the NAFS action plan have created more administrative burden than practical benefit for your institution

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Which measures are disproportionately burdensome?

(Text box)

Section 5: Impact

Impact examines the positive and negative, intended and unintended effects produced by the NAFS. It also seeks to capture the broader changes it has contributed to, including changes in systems, norms, and institutional arrangements.

5.1. To what extent is there awareness of anti-fraud policies and procedures across your institution?

(To a great extent / To a moderate extent / To a limited extent / Not at all / Don't know)

5.2. Senior management in your institution has taken visible actions (e.g. attenting trainings, raising NAFS matters in management meetings, allocating resources) to promote anti-fraud awareness

(Yes / No /Partially / Don’t know)

5.3. Since the NAFS was adopted, what major improvements in the fight against fraud have been achieved?

(Text box)

5.4. Since the NAFS was adopted, have new systems, institutional arrangements, or legal provisions been established as a direct or indirect result of the strategy (e.g. new IT systems, new units or co-ordination bodies, legislative reforms)?

(Yes / No / Don't know)

a) If yes, describe them

(Text box)

5.5. Since the NAFS was adopted, have there been any changes (intended or unintended) in how fraud cases are identified or reported?

(Text box)

5.6. Inter-institutional co-operation for the investigation of fraud cases has improved since the adoption of the NAFS

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Provide examples or evidence to support your answer

(Text box)

5.7. Since the adoption of NAFS how has the situation changed regarding:

a) Visibility of sanctions for fraud against the EU budget (the extent to which sanctions are made known to relevant parties and the public)

Improved / Stayed the same / Got worse / Don’t know

b) Consistency of sanctions for fraud against the EU budget (the extent to which sanctions are applied in a uniform manner)

Improved / Stayed the same / Got worse / Don’t know

c) Deterrent effect of sanctions for fraud against the EU budget (the extent to which sanctions are sufficient to discourage potential fraud)

(Improved / Stayed the same / Got worse / Don’t know

5.8. Are there any unintended negative effects that have resulted from the implementation of the NAFS (e.g. excessive administrative burden, duplication of controls)?

(Yes / No / Don't know)

a) If yes, describe them

(Text box)

Section 6: Sustainability

Sustainability examines whether the benefits produced by the NAFS are likely to continue over the medium and long term.

6.1 How likely is that results achieved under NAFS will be sustained beyond the end of the strategy period?

(Very likely, Likely, Unlikely / Don’t know)

6.2 To what extent does your institution have measures in place to ensure continuity of anti-fraud knowledge and capacity when staff leave or rotate?

(To a great extent / To a moderate extent / To a limited extent / Not at all / Don't know)

6.3 The co-ordination mechanisms established under the NAFS (e.g. inter-institutional working groups, MoUs) are likely to remain functional and effective after the current strategy period ends

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

6.4. What are the main factors that could undermine the sustainability of the results achieved under the NAFS?

(Text box)

Section 7: Final questions

7.1 What is the most important recommendation you would make for the next NAFS

(Text box)

7.2. Is there any good practice developed under the Strategy that you would like to share or highlight?

(Text box)

7.3. Is there any additional information, data, or observation you would like to share that has not been covered in this survey?

(Text box)

B: Survey for evaluation – external stakeholders

Cover page:

0. Introduction to the participating stakeholders of the survey on the mid-term/end-term evaluation of the NAFS.

List the following:

• Objectives and structure of the survey

• Description of the survey (number and content of the sections)

• Instructions on how to fill in the survey

• Deadlines and coverage

• Expected time of completion

• Contact person

Introductory details

1. Responding entity: (text box)

2. Type of entity: (select one)

• Beneficiary of EU funds (company, municipality, institution)

• Company/economic operator participating in public procurement processes related to EU funds

• Non-governmental organisation (NGO)

• Business association / Chamber of commerce

• Academic or research institution

• Other_______ (please specify)

3. Contact person: name, position, department, email / phone no. (text box)

Section 1: Relevance

Relevance examines whether the NAFS objectives and design respond to the actual needs, policies, and priorities related to the protection of the National and EU's financial interests and whether they continue to do so as circumstances change.

1.1. How familiar are you with the National Anti-fraud Strategy (NAFS)?

(Very familiar, Fairly familiar, A little familiar, Not at all familiar)

1.2. The NAFS addresses the issues that matter most for protecting EU funds against fraud

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) What issues are insufficiently covered or missing?

(Text box)

1.3. Your organisation was consulted/had the opportunity to provide input when the NAFS objectives and actions were being developed

(Yes / No / Don’t know)

a) If yes, how? If no, should external stakeholders be consulted, and in what form?

(Text box)

1.4. The NAFS has been updated or adapted to reflect new fraud risks, changes in EU funding instruments, or shifts in the national context

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

Section 2: Coherence

Coherence examines how well the NAFS fits with other interventions: internal (such as consistency among the NAFS's own objectives and actions) and external (alignment with other national policies and EU-level frameworks).

2.1. Anti-fraud measures are consistent with broader governance priorities (e.g. anti-corruption strategy, sectoral anti-fraud action plans, national recovery and resilience plan, etc.)

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) Where do you see the main overlaps or gaps (if applicable)

(Text box)

2.2. The roles and responsibilities of different public institutions in preventing and addressing fraud are clear to external stakeholders

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

a) In what areas are they unclear? (if applicable)

(Text box)

Section 3: Effectiveness

Effectiveness assesses the extent to which the NAFS is achieving its objectives and targets. It also examines the major factors influencing the achievement or non-achievement of those objectives.

3.1. Do you believe that the capacity of the national anti-fraud framework to prevent, detect, investigate and sanction fraud against the EU budget has improved over the period covered by NAFS?

(Yes, significantly / Yes, moderately / Situation has not changed / It got worse / It got significantly worse / Don't know)

3.2. Your organisation has received information from public authorities about anti-fraud measures in place to protect EU funds

(Yes /No /Partially / Not applicable / Don’t know)

3.3. Your organisation has been provided by public authorities with clear information on where and how to report suspicions of fraud or irregularities affecting EU funds

(Yes /No /Partially Not applicable /Don’t know)

3.4. Your organisation has been informed by public authorities about existing whistleblower reporting channels and the protections available to those who report fraud

(Yes / No / Partially / Not applicable / Don’t know)

3.5. In your experience, when a report of suspected fraud or irregularity is made, anti-fraud authorities follow up and provide feedback to the reporting party

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

3.6. What are the main challenges that hinder effective anti-fraud efforts? Select all that apply

¨ Lack of public awareness about fraud risks and reporting channels

¨ Insufficient protection for whistleblowers

¨ Lack of transparency

¨ Weak sanctions

¨ Insufficient co-operation between public institutions

¨ Insufficient engagement with external stakeholders (NGOs, business, academia)

¨ Lack of trust that whistleblower reports will be taken seriously

¨ Other _________(please specify)

3.7. Information about anti-fraud measures, reporting channels, and the use of EU funds is easy to find and accessible to the public

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

3.8. The procedures for reporting suspected fraud or irregularities are simple and not burdensome

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

Section 4: Impact

Impact examines the positive and negative, intended and unintended effects produced by the NAFS. It also seeks to capture the broader changes it has contributed to, including changes in systems, norms, and institutional arrangements.

4.1. Fraud against EU funds is effectively addressed

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

4.2. Public trust in the management of EU funds has improved in the country over the strategy period

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

4.3. Sanctions for fraud against EU funds are visible and have a deterrent effect

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

4.4. Co-operation between anti-fraud authorities and external stakeholders (EU funds beneficiaries, NGOs, business, etc.) has improved over the strategy period

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

Section 5: Sustainability

Sustainability examines whether the benefits produced by the NAFS are likely to continue over the medium and long term.

5.1. The progress made in the fight against fraud is likely to be maintained in the long term, even after the end of the NAFS

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

5.2. Awareness of anti-fraud measures and reporting channels among beneficiaries of EU funds and the general public has reached a level that is likely to last

(Strongly agree / Agree / Neither agree nor disagree / Disagree / Strongly disagree / Don't know)

5.3. What would you recommend to strengthen the long-term sustainability of anti-fraud efforts?

(Text box)

Section 6: Final questions

6.1. In which areas do anti-fraud efforts need the most improvement (select up to three)

¨ Public awareness of fraud

¨ Transparency about how anti-fraud cases are handled

¨ Protection and support for whistleblowers

¨ Deterrence and visibility of sanctions

¨ Co-operation between public institutions

¨ Engagement with external stakeholders

¨ Accessibility and simplicity of reporting channels

¨ Other ____________ (please specify)

6.2. What is the most important recommendation you would make for the next NAFS?

(Text box)

6.3. Is there any additional information, data, or observation you would like to share that has not been covered in this survey?

(Text box)