How to measure progress | Further resources | Print version (pdf)
Monitoring and evaluation based on clearly defined indicators is integral to the development of sound policies. They allow policy makers to assess the extent to which policy objectives have been fulfilled and provide a basis for identifying strengths and weaknesses and making necessary adjustments. Monitoring can ensure better coordination and consistency between policies. When carried out on a comparative basis with peers, it can provide further impetus for reform. Quantitative indicators have proven highly effective in drawing attention to the burdens of business regulation, identifying priorities for reform and communicating success and progress. The use of a common ‘scorecard’ also facilitates public-private consultation. In short, by measuring what is in the national interest, it is more likely that national interest will be served.
At the same time, measures and indices have inherent limitations when used in isolation which the PFI user will need to consider. Indicators sometimes require substantial capacity to develop and use effectively: unreliable indicators are sometimes worse than none at all. By their nature, they are a simplification of complex regulatory and administrative structures and often implicitly assume that less regulation is always better. They often measure subjective perceptions, usually from an investor perspective, and when they involve external assessments, the likelihood of local ‘buy in’ is reduced. And lastly, they focus mainly on policies, not how or whether they are implemented.
As with the PFI itself, quantitative assessments – if used judiciously – are part of a process rather than a set of mechanical indicators. This section, when it is completed will explain how to form an overall assessment, benchmark information, establish priorities, develop, initiate and monitor a reform Plan of Action. Through a number of case studies, the Toolkit will illustrate how benchmarking performance in policy areas and well-presented measures of progress can act as a powerful catalyst for change (e.g. the World Bank’s Doing Business indicators and the OECD Programme for International Student Assessment).
How to measure progress
Measures can be based either on facts or on subjective perceptions. PFI users will need to decide on their own which approach to follow, depending on the situation in their countries and on their specific needs. The Toolkit will then offer a menu of alternative frameworks for measuring progress in each of the policy areas covered by the PFI. The focus will be on how information accumulated in the policy guidance section of the Toolkit is compiled to form an overall assessment of a country’s investment climate weaknesses and strengths.
Indicators are only part of the story. PFI users will also want to understand the relationship between measures of progress in one area of a country’s investment climate and those in another. For this, more detailed methods are necessary. Indicators should not stand in isolation.
Many existing indicators touch on various aspects of the PFI. Some concern broader questions of governance, while others focus specifically on a narrower set of regulations, such as those applying to foreign investors. What is most interesting for PFI users are the methodologies adopted to arrive at the level of indicators. Various indicators and their methodologies are presented below. They will help to inform the development of indicators under the PFI.
The OECD Going for Growth Indicators involve regular cross-country structural surveillance of OECD member countries. Going for Growth aims at promoting growth and fostering convergence among OECD economies. The assessment is based on a set of around 50 quantitative indicators which measure the performance of countries in different policy areas affecting product market and labour market policies and draw on expertise of a number of OECD directorates. Five structural policy recommendations are issued for each country: three based on the indicator-based assessment and two on further OECD country expertise. Every two years recommendations are issued, while follow-up research on their impact is conducted every other year.
The OECD FDI Regulatory Restrictiveness Index
measures the deviation of the treatment of foreign investors from national treatment based on a country’s legal and regulatory framework, rather than the institutional environment more generally. It maps the formal ownership, screening, approval and other limitations to foreign investment to a quantitative indicator ranging from 0 (completely open) to 1 (closed). The FDI Regulatory Restrictiveness Index is used in OECD Economic Surveys and has helped to focus attention on particular policy areas requiring attention. In some cases reforms have been implemented as a result.
The South-East Europe Investment Reform Index
(IRI) provides one example of how indicators can be developed based on the PFI, although there is not a one-to-one mapping. It was developed as part of the Investment Compact for South East Europe (an initiative of the Stability Pact for South Eastern Europe and the OECD) to measure – on a comparative basis – where countries stand on policy reform. The indices are developed with the active participation of the country concerned, including not only prior consultations but also ample opportunity for feedback. The IRI divides each policy dimension into a series of sub-dimensions and indicators (or sub-indicators). The example of competition policy is provided below.
Then for each indicator or sub-indicator, an assessment is made of the level of reform. In general terms, the measurement of reform levels under the IRI typically adopts the following approach:
1. There is no law or institution in place to cover the area concerned;
2. There is a draft law or institution, and there are some signs of government activity to address the area concerned;
3. A solid law or institution is in place to cover the area;
4. Level 3 + concrete indications of effective policy implementation of the law or institution;
5. Level 3 + significant record of concrete and effective policy implementation of the law or institution. This level comes closest to good practices in OECD countries.
The World Bank Doing Business indicators provide a quantitative measure of regulations for starting a business, dealing with construction permits, employing workers, registering property, getting credit, protecting investors, paying taxes, trading across borders, enforcing contracts and closing a business – as they apply to domestic small and medium-sized enterprises. They are based both on readings of laws and regulations and on time and motion indicators that measure the efficiency in achieving a regulatory goal. At present, they include 10 indicator sets for 181 economies.
The Foreign Investment Advisory Services (World Bank) FDI Indicators project is an extension of the Cost of Doing Business Indicators applied to foreign investors. The indicators benchmark country performance concerning the ease of opening and conducting a formal business, both with regard to formal statutory restrictions and regulatory and administrative barriers in practice. The aim is to identify, stimulate and inform investment policy reforms. Most of the indicators are measured using a hypothetical investment project case applicable to all countries. Data is collected through a questionnaire which is filled out by private sector intermediaries (e.g. investment lawyers). The investor’s point of view is then validated by government agencies. The first report on this project is scheduled to be published in 2009. The hope is that the FDI Indicators will create a momentum for reform similar to what has already been achieved through the Doing Business Indicators.
The World Bank Worldwide Governance Indicators capture six key dimensions of governance: voice and accountability, political stability and absence of violence, government effectiveness, regulatory quality, rule of law and control of corruption. They compile information and perceptions from a very diverse group of respondents, collected from a large number of surveys and other cross-country assessments of governance. Some of these instruments capture the views of firms, individuals and public officials in the countries being assessed. Others reflect the carefully considered views of NGOs and aid donors with considerable experience in the countries being assessed. Still others are based are based on the assessments of commercial risk-rating agencies. The Indicators cover 212 countries and territories.
The UNCTAD Investment Compass is an interactive tool that permits comparison of the investment environment of a country with that of other countries, regions and best performers. It comprises 60 indicators based on main economic and policy determinants that affect the investment climate. Data are now available for 55 developing countries. Many of the indices are drawn from other sources, and many of those relating to the regulatory framework involve yes/no answers by experts to questions in each area.
Many private sources, including both private firms and NGOs, compile their own indicators with varying degrees of coverage. The indicators listed below are provided for illustrative purposes only. PFI users will need to assess their usefulness based on their own specific needs.
Concerning individual policy chapters of the PFI, various indicators exist covering specific topics. In the area of public governance, for example, Transparency International publishes an annual Corruption Perceptions Index.
The Business Environment Rankings of the Economist Intelligence Unit uses quantitative data, business surveys and expert assessments to measure the attractiveness of the business environment across 82 countries. They are based on 91 separate indicators grouped into ten categories: the political and macroeconomic environment, market opportunities, policy towards private enterprise and competition, policy towards foreign investment, foreign trade and exchange controls, taxes, financing, the labour market and infrastructure.