Evaluation Guidelines for Representative Deliberative Processes

Annex D. Organiser questionnaire

The questions below are listed in the order of the criteria they are tailored to measure to facilitate their use for evaluation and interpretation. It is recommended that all questions are included as a minimum evaluation. They are intentionally non-leading and open-ended. Questions can be used as a survey, or to guide an interview. The answers should then be analysed with an interpretive approach, judging how answers provided compare with and are in coherence with results from the members’ survey, document review and other measurement methods used for evaluation.

1) Process design integrity

Clear and suitable purpose

1. What were your objectives in organising this process?

2. What policy problem was the process addressing?

3. What was the mandate that the members of a deliberative process were given?

4. In your view, was organising a deliberative process in this situation a helpful way to address the policy issue? Why?

Clear and unbiased framing

5. In your view, how well did the members of the process receive and understand the question?

Suitable design

6. What are your overall impressions about the way the process went?

7. Did you achieve the objectives that were set out for this process?

Procedural design involvement

8. What was the process of setting the question, the mandate, and the design?

9. Were stakeholders relevant to the policy issue active in providing input?

10. Please list some of the stakeholder groups which were involved.

11. Who had a final say in the design questions of the deliberative process?

Transparency and governance

12. Did you use any terms of reference, rules of engagement, codes of conduct, or ethical frameworks that govern the process? (If yes, please provide them for reference.)

13. If yes, were they helpful?

Representativeness and inclusiveness

14. How did you recruit members of the process? Please describe the process.

15. Were there any groups that you found hard to reach? If yes, how did you address this challenge?

2) Deliberative experience

Neutrality and inclusivity of facilitation 

16. What were the main tasks of facilitators?

17. How were the facilitators trained?

18. Were there any situations where some members were dominating the discussions? If yes, how did you manage this?

Accessible, neutral, and transparent use of online tools 

[If online tools were used]

19. How did you choose the online tools to use?

20. Did you have to provide substantive explanations and assist with the use of online tools?

21. Would you use the same online tools for other deliberative processes in the future? If yes, why? If not, why not?

Breadth, diversity, clarity, and relevance of the evidence and stakeholders provided

22. What was the process of preparing the information base presented to members of the deliberative process? (Such as choosing relevant evidence and identifying stakeholders for presentations and panel discussions.)?

Quality of judgement

23. What deliberative techniques did you use? This could include considering underlying values, weighing of alternatives and trade-offs, exploring uncertainties, and considering different viewpoints.

24. What were some of the most conflicting viewpoints of the members? Please list them.

Perceived knowledge gains by participants

25. Did you notice any changes, over time, in the argumentation that members used to convey their point of view?

Accessibility and equality of opportunity to speak 

26. Did you receive any feedback or suggestions from members to modify or adapt parts of the process?

Respect and mutual comprehension

27. Did you have to intervene to stop any conflicts amongst members?

28. Did you have to remind members to listen when others were speaking?

29. Did you feel like the group could listen to and incorporate challenging viewpoints from more marginalised members of society?

Free decision-making and response

30. Did you perceive any inappropriate external interference? For example, participants receiving undesired attention from lobbyists or media.

31. Could you describe the process of drafting the final recommendations?

32. In your view, were the democratic decision rules used for final decision making helpful in reaching an optimal decision? If yes, why? If not, why not?

Respect for participants’ privacy

33. What was your approach to protecting members’ privacy?

3) Pathways to impact

Member aftercare

34. What is your approach to fostering the community of members and the relationships they built throughout the deliberative process?