Public Communication Scan of France

Annex A. Matrix of performance and impact indicators

This annex includes a proposed template for a matrix of performance and impact indicators for French government communicators. The template includes a framework (see Figure A A.1) adapted from the OECD analytical framework presented in Figure A A.1 and inspired by the leading evaluation frameworks reviewed for this study. This is further adapted to reflect the macro objectives for the communication function and the aggregate performance indicators tracked by the SIG.

The tables below translate this framework into indicative indicators matrices. They include respectively: the macro communication objectives and related aggregate indicators (Table A A.1); indicators for the aggregate outcomes and impact of a specific communication activity or campaign (Table A A.2); and the indicators to assess the performance of individual levers of communication deployed (Table A A.3). Each of these tables can be used at different stages of planning and assessment to help draw conclusions from an activity or to understand how specific elements of the activity are performing at any point in time.

Figure A A.1. Framework for design and evaluation of government communication adapted to French institutions

Note: This diagram accompanies the indicator matrices in this annex as a guide for their use by DICOMs.

Source: Author’s own work, based on the macro objectives for communication and the priority indicators used by the SIG.

Table A A.1. Communication performance indicators matrix, level 1: measuring the aggregate effect of communication on public information and understanding of governmental action

Level 1 of the communication performance indicators matrix captures the three primary objectives for the function across all levels of government and the main indicators that are tracked over time to assess their attainment. These objectives and indicators are used by the SIG to assess the aggregate impact of government communication and measured through longitudinal surveys.

Macro communication objectives	Aggregate performance indicators
Attribution: citizens attribute to the state the policies, benefits and outcomes resulting from government action	Awareness, visibility of a policy Method: longitudinal surveys
Understanding: citizens understand well the actions taken by the government	Perception that a policy is useful Method: longitudinal surveys
Retention: citizens recall the policy priorities and actions taken by the government	Perception that a policy is relevant to oneself or their kin Method: longitudinal surveys
	Perception that the policy is effective Method: longitudinal surveys

Source: Author’s own work, based on the macro objectives for communication and the priority indicators used by the SIG.

Table A A.2. Communication performance indicators matrix, level 2: measuring aggregate campaign performance against objectives

Level 2 of the communication performance indicators matrix brings together a comprehensive but not exhaustive range of indicators that can be measured to understand if the overall objectives of a communication activity are being met. They refer to actions taken by audiences or shifts in awareness and opinion (outcomes) based on the communication received, and how these actions and perceptions contribute to positive change towards policy goals. The most important of these indicators should be measured at the baseline, mid- and endpoint of the activity (noting that impact will in many cases be observed with a delay compared to the timing of activities).

PURPOSE General purpose of the communication	OUTCOME INDICATORS Aggregate results for all campaign activities that relate to changes in the audience’s awareness, attitudes, and actions	IMPACT INDICATORS Partial or total contribution of all campaign activities to policy or organisational objectives
Change (behaviour)	Intention to act on the message Method: audience survey*	Compliance with message – behaviour observed (e.g. voter turnout, health prevention/screening appointments, tax filings ahead of deadline, increased tourism…) Method: database records
	Reasons for intent (or lack of intent) to act on the message Method: audience survey, focus group*	Compliance with message – behaviour proxy (e.g. household energy consumption levels, funds/investment raised,… ) Method: database records
	Downloads of information packs, forms, toolkits, etc. (as proxy for intention to act) Method: website analytics
	Inquiries received about programme/service (as proxy for intention to act) Method: database records	Return on investment (ratio of campaign cost to estimate of monetary value created) Method: calculation
	Registrations, applications, requests submitted for a programme/service Method: database records	Volume of programme enrolment / service take-up (e.g. voter registration, vaccination appointments, benefits claimed,…) Method: database records
	Relevance of registrations, applications submitted (e.g. from eligible, target groups) Method: database records
Change (perception)	Public opinion change (level of concern, level of agreement/disagreement with message) Method: polling, audience survey*	Public support for / acceptance of public policy or initiative Method: audience survey, polling*
	Public opinion change (nature and depth of concern, reason and depth of agreement with message) Method: audience survey, focus group, interviews*
	Audience association of public policy with societal problems Method: audience survey*
	Tone of audience-generated content Method: sentiment analysis
Recruitment	Applications forms downloaded/created Method: website analytics	Number of people successfully recruited Method: database records
	Applications forms submitted / share of applications submitted by target audience profiles Method: database records
	Change in sentiment towards the profession Method: audience survey*	Return on investment (ratio of campaign cost to number of people successfully recruited) Method: calculation
	Expression of interest to apply Method: audience survey*
Awareness	Unprompted campaign / issue awareness Method: audience survey*	Attribution of message to government/public entity Method: audience survey*
	Prompted campaign recall Method: audience survey*	Public satisfaction with government information Method: audience survey, polling*
	Unprompted message recall Method: audience survey*	Audience relaying the message to others (word-of-mouth) Method: audience survey*
	Audience expresses having learned new information Method: audience survey*

Notes: Shaded cells mean the indicator is qualitative. All other cells refer to quantitative indicators. Cells marked by an asterisk “*” mean the indicator refers to paid communication activities. All other cells are relevant to organic, low-cost, and paid activities alike.

Source: Author’s own work, drawing on the review of the literature and frameworks cited in Chapter 2.

Table A A.3. Communication performance indicators matrix, level 3: measuring the performance of individual levers of communication

Level 3 of the communication performance indicators matrix provides a comprehensive but not exhaustive range of indicators that can be measured to understand how each element of a communication activity performs to reach target audiences and spark interest and engagement with the message. These indicators are best measured at the early stages of the activity and at set milestones to monitor whether the strategy is working or which elements need to be reviewed. Each indicator may be compared to benchmarks in comparable past activities. Implicit in the activity lists are input indicators, which comprise the activity’s budget, staff time, the formative research and evidence, the communication strategy and creative concepts and assets.

COMM. LEVER	ACTIVITY The contents produced and activities performed	OUTPUT INDICATOR Deployment of content on information channels	OUTTAKES INDICATOR Reactions and changes in the audience resulting from exposure to the content
Advertising	Advertising placement	Estimated advertisement reach per channel (offline) Method: media agency	Prompted campaign recall Method: audience survey
			Audience attitude towards campaign content (positive/negative) Method: audience survey
		Advertisement impressions per channel (online) Method: media agency	Audience attitude towards campaign content (sentiment, depth) Method: survey/focus group/interviews
			Comprehension of the campaign content Method: audience survey
			Cost per click / engagement / views Method: calculation
		Frequency of exposure Method: media agency	Click-through-rate to website Method: UTM links and website analytics
			Perceived relevance of the campaign for self/others Method: audience survey
Media relations	Number of press releases issued	Number of press articles mentioning elements of the press release Method: media monitoring	Social media amplification of media coverage by audience Method: social listening
		Number of broadcasts mentioning elements of the press release Method: media monitoring	Click-through-rate to website Method: UTM links and website analytics
	Number of media interviews and statements by spokespeople	Number of interviews published/broadcasted Method: media monitoring
		Tone of media mentions Method: sentiment analysis	Volume of search engine queries related to campaign topic Method: search engine analytics
	Number of journalists reached out to	Total estimated audience reach (as a total of the audience reported by offline, broadcast and online media) Method: media monitoring
Web	Number of pages created	Number of page views Method: website analytics	Dwell time (average time on page) or scroll depth Method: website analytics
		Average session duration Method: website analytics	Number of pages visited on site Method: website analytics
			Conversions (downloads, registration, form completion, etc.) Method: website analytics
		Bounce rate Method: website analytics	Overall usefulness of the website Method: survey on the website
			Audience feedback about the website Method: survey on the website
Social Media / Digital	Number of posts to owned social media channels	Number of impressions per post Method: social media analytics	Number of passive and active engagements (shares, like, comments, response, quote) Method: social media analytics
			Engagement rate (engagements/impressions) Method: social media analytics
			Channel-specific traffic to website Method: UTM links and website analytics
	Number of videos created	Number of video views Method: platform analytics	Tone of comments Method: sentiment analysis
			Increase in followers, subscribers Method: social media analytics
			Duration of video views Method: platform analytics
	Number of emails, newsletters created	Number of email/newsletter recipients Method: platform analytics	Open rate Method: platform analytics
			Click through rate Method: platform analytics
Partnerships	Number of partnerships secured	Estimated audience reach on partner owned/paid channels Method: partner reporting, social listening	Channel-specific audience engagement with partner contents (refer to outtake indicators for each communication levers in this table) Method: partner reporting, social listening
	Quality of partners (relevance to objective, reach of target audience)
	Volume of content produced by partners
Events	Number of events organised (physical or online)	Number of attendees (physical or online) Method: database records	Attendees’ satisfaction rates Method: attendees survey
			Active engagement/ interactions during the event (e.g. questions, participation in polls) Method: observation
		Share of first-time and repeat attendees Method: survey or database records	Average duration of participant’s attendance (online) Method: database records
			Share of attendees who declared that they would share, speak positively about or recommend the event Method: attendees survey
			Feedback received during or after the event Method: attendees survey
		Quality of attendees (relevance to the communication objective) Method: database records	Event amplification on media and social media by attendees, stakeholders Method: monitoring, social listening
			Attendees’ actions resulting from the event (adherence to programmes, investments made, commitments made, agreements signed,…) Method: database records, observation
Branding	Development and application of state brand and/or campaign brand elements	Proportion of communication assets conforming with brand Method: internal SIG approval tracking	Brand recognition by audience Method: audience survey
	Trademark registration and domain name protection		Message attribution to a public entity Method: audience survey

Notes: Shaded cells mean the indicator is qualitative. All other cells refer to quantitative indicators.

Source: Author’s own work, drawing on the review of the literature and frameworks cited in Chapter 2.