Quality Budget Institutions

Annex C. Methodology for the OECD Fiscal Advocacy Index

General background

The OECD composite index presented in this publication was developed using the steps identified in the Handbook on Constructing Composite Indicators that are necessary for the meaningful construction of composite or synthetic indexes (European Union/EC-JRC, 2008[1]). The Index is based on a theoretical framework representing an agreed concept. The dimensions and variables comprising the index are chosen based on their relevance to the concept. The index was constructed in close collaboration with the relevant OECD expert group, which advised on the variables and the weighting schemes to use for the composite.

Statistical analyses were also conducted to ensure the validity and reliability of the composite index. The survey questions used to create the index are the same across countries and institutions. In order to eliminate scale effects, all indicator dimensions and variables are normalised between “0” and “1” for comparability. To build the composite, all indicators were aggregated using a linear method. The index scores were determined by adding together the weighted scores of each indicator. Statistical tools (i.e. Cronbach’s alpha) were also employed to establish the degree of correlation among a set of variables and to check their internal reliability. This implies that all of the variables have intrinsic value but are also interlinked and capture the same underlying concept. Finally, sensitivity analysis using Monte Carlo simulations was conducted to establish the robustness of the index scores to different weighting options.

2024 OECD Fiscal Advocacy Index

The 2024 OECD Fiscal Advocacy Index is a tool for assessing the extent to which independent fiscal institutions (IFIs) fulfil the role of fiscal advocates — institutions that champion fiscal sustainability.

The Index evaluates 35 national IFIs across four dimensions:

1. Independence - the extent to which IFIs effectively provide impartial advice to policymakers and the public. Leadership and operations should be free from outside interference and funding should be multiannual in nature, preferably with direct approval of funding from the legislature. This dimension is based on an update of earlier OECD work measuring IFI independence.

2. Analysis - as fiscal advocates, IFIs should be able to focus their work on major aspects of fiscal policy. IFIs should be able to explore long-term sustainability challenges and major fiscal risks facing the public finances. Those that focus on costings should be able to promote citizens’ interests by bringing greater transparency to government or legislative proposals as well as election platforms. At the same time, they should have the capacity to generate and deliver macroeconomic and fiscal forecasts to support their analysis, with an adequate number of analytical staff.

3. Communications Apparatus - the ability of an IFI to communicate directly with the public and engage with the media separates it from other non-partisan research services. To do this effectively, IFIs should have robust tools in place to disseminate, promote and track the impact of their work. This dimension incorporates previous OECD work, the OECD IFI Communications Index (OECD, 2023[2]).

4. Communications Impact - having a robust communications function can help generate an impact on the national debate and promote the IFIs analysis. However, it does not guarantee it. The impact of an IFI’s work often relies on more than just the methods it uses to promote that work. It relies on clear and cogent analysis in key areas. This dimension gauges the final impact an IFI has by considering its prevalence in media discussions and the public’s interest in the institution.

The four dimensions have been informed by the OECD Council Recommendation on Principles for Independent Fiscal Institutions (OECD, 2014[3]). The Recommendation contains 22 principles across nine areas related to local ownership, independence and on partisanship, scope of the mandate, resources, relation with the legislature, access to information, transparency, communication and evaluation.

The index has also been shared and discussed with delegates from the OECD’s Working Party of Parliamentary Budget Officials and Independent Fiscal Institutions.

Data collection and validation

For the first three dimensions, data used for the construction of the 2024 OECD Fiscal Advocacy Index comes primarily from the OECD Independent Fiscal Institutions Database (OECD, 2021[4]). The data was collected via desk research and then verified and validated by relevant senior officials in the OECD’s Working Party of Parliamentary Budget Officials and Independent Fiscal Institutions.

In addition to that, the fourth dimension (communications impact) draws on new indicators, created using a combination of data on national media coverage plus Google Trends data for 2021 to 2023. These are intended to assess the extent to which IFIs have entered the national debate, as recorded in mentions of their work in the media.

Dimensions and weights

Each of the four dimensions is equally weighted, has a maximum value of 1, and comprises several variables. The overall index has a maximum value of 4, with institutional scores being the sum of the scores for each dimension. A higher score indicates a greater capacity for fiscal advocacy.

The variables and weights comprised in the index were selected based on their relevance to the concept by a group of experts within the OECD and in consultation with delegates to the Working Party of Parliamentary Budget Officials and Independent Fiscal Institutions. Variables within a dimension are weighted based on 1) the number of indicators making up each variable, and 2) and the relative importance of each variable.

A linear aggregation method is applied to first aggregate the indicators into variables, the variables into sub-dimensions and then the sub-dimensions into an overall dimension or composite indicator. To obtain overall index scores, the weighted scores for each indicator are totaled.

The components used in the construction of this index, and the weights given to each, are indicated in the Figure A ‎C.1.

Figure A ‎C.1. 2024 OECD Fiscal Advocacy Index: Dimensions, sub-dimensions and weights

Source: OECD Authors.

A detailed explanation on the components of the 2024 OECD Fiscal Advocacy Index will be available in a forthcoming publication (OECD, 2024[5]), including the variables, answer options, scores and weights used to construct the composite index, as well as the statistical analysis conducted.

Statistical analyses

Several statistical tests were used to show how reliable the index is in terms of measuring a coherent underlying concept – fiscal advocacy. They also assess how valid the choices of individual parameters and variables are.

Sensitivity analysis was conducted to establish the robustness of the index scores to different weighting options through Monte Carlo simulations. The results from the sensitivity analysis at dimension level for the 2024 OECD Fiscal Advocacy Index show that, for the majority of the institutions analysed, the overall scores are not sensitive to the choice of weights given to the categories. In terms of the overall Index, the Cronbach’s alpha coefficient is equal to 0.89, indicating that the dimensions are measuring the same underlying construct.

References

[1] European Union/EC-JRC (2008), Handbook on Constructing Composite Indicators: Methodology and User Guide, OECD Publishing, Paris, https://doi.org/10.1787/9789264043466-en.

[5] OECD (2024), From fiscal watchdogs to fiscal advocates: Creating champions of fiscal sustainability, https://one.oecd.org/document/GOV/SBO(2024)13/en/pdf.

[2] OECD (2023), Government at a Glance 2023, OECD Publishing, Paris, https://doi.org/10.1787/3d5c5d31-en.

[4] OECD (2021), “2021 Independent Fiscal Institutions (IFIs) Database”, https://www.oecd.org/en/data/datasets/public-finance-and-budgets-database.html (accessed on 28 April 2024).

[3] OECD (2014), “Recommendation of the Council on Principles for Independent Fiscal Institutions”, OECD Legal Instruments, OECD/LEGAL/0401, OECD, Paris, https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0401.