An extended and high-quality set of quantitative indicators is necessary to the design and evaluation of science and technology (S&T) policy. Indicators allow the comparison of the relative situations of countries, the assessment of their areas of strength and weakness, and the identification of domains where policy intervention is required. They also provide feedback on the effects of policies. Above all, indicators have to be able to adapt to the changing conditions and the content of S&T policy.
At its meeting at Ministerial level held in 1995, the OECD Committee for Scientific and Technological Policy agreed in its conclusions that "there is a need for Member countries to collaborate to develop a new generation of indicators which can measure innovative performance and other related output of a knowledge-based economy". They also agreed that "trends and challenges to the science system need to be studied further by the OECD" and that "special attention should be given to the data required for assessment, monitoring and policy-making purposes".
Demand from policy makers was the rationale for past statistical endeavours that resulted, for instance, in theFrascati Manual (for research and experimental development, R&D) in the early 1960s and the Oslo Manual (for innovation) in the 1990s. As a result of such efforts, there exists today a rich set of S&T indicators, many of which are collected by countries and compiled by the OECD and other international organisations. These include R&D expenditure, head-counts of researchers, technology balance of payments, production and trade of high-technology goods and patents (these indicators are published twice yearly in the OECD's Main Science and Technology Indicators ). However, as is argued in the above-quoted ministers' request, there is a clear need to extend the list of S&T indicators. The context, as well as the content of technology policy, have been changing, resulting in demand for new indicators, while at the same time new data sources and new methodologies are becoming available, providing opportunities for the supply of new indicators.
Changes in demand for indicators
The importance of research and development (R&D) for technological change and economic growth became clear after the Second World War. The Frascati Manual ("guidelines for the collection of R&D data", first published in 1963), resulted directly from demand by policy makers for measuring this newly identified factor of growth. Since then, this view has not been disputed, although it has been broadened and deepened. Part of this change in perspective is certainly due to new conditions in the real world as well as, notably, the emergence of the "knowledge-based economy". Major aspects can be summarised as follows.
Innovation: R&D is a central piece in the process of technological change, but it is not the only one. Technological innovations are new products or new processes actually commercialised or implemented by firms. There are other sources of new technology than R&D in the strict sense, such as design, new software, training. The "innovation surveys" developed through the 1990s, notably by Eurostat, aim to capture such activities. This applies especially to the expanding service industries, where little R&D is undertaken but where innovation is spreading.
Investment in knowledge: certain activities, in addition to R&D, lead to an increase in the stock of knowledge and information available to the economy, such as investment in software, in training or in education. It is therefore useful to integrate certain of these activities under the common label of "investment in knowledge", which gives a broader view of the advance towards knowledge-based economies. A better integration of R&D into the overall framework of economic statistics, the system of national accounts (SNA), goes into the same direction, as it links R&D to other productive activities. This is the purpose of ongoing efforts at OECD.
Human resources in S&T (HRST) are the major factor that enable a country to generate and master new technology. A large HRST base is a sine qua non condition for a country to continue to innovate, as it embodies the knowledge stock of the country. In addition to researchers, engineers and technicians allow an efficient application of technology. In particular, policies aimed at encouraging firms to spend more on R&D will fail if the supply of HRST is insufficient, as more spending would result only in higher costs. Counts of researchers, that have been available for a long time, reflect only one part of the human resources. It is therefore necessary to measure other components of HRST (R&D managers, production engineers, technicians), and to investigate its structure and dynamics (such as structure by skills, S&T fields, its domestic and international mobility). The " Canberra Manual " (OECD and Eurostat, 1995) was a first step in this direction. More work needs to be done, on the methodological side (improved definitions) and on the identification and exploitation of the appropriate sources of data (e.g. labour force surveys, census and special surveys performed in many countries).
Services are becoming increasingly innovative, but this trend is not well captured by current statistics which were built on the "brick and mortar" model. Much innovation in the services is not based on formal R&D, and R&D in these sectors differs in many respects from R&D in manufacturing -- its focus is not on the natural sciences but more on the social sciences and humanities (SSH). In that sense, it is clear that the issue is not one of R&D in the service sectors, but rather service-type R&D in all sectors (as even manufacturing firms perform research in these fields). Efforts have been engaged for the past decade both to capture non-R&D innovation (through innovation surveys) and to improve the coverage of R&D surveys to service firms. The definition of R&D in the Frascati Manual was broadened in the 1993 edition in order to more clearly encompass SSH. Many countries have included service industries in their survey sample. It seems, however, that the notion of R&D in services has still to be clarified, and the fifth revision of the Frascati Manual is exactly doing that, by providing specific examples of R&D related to human behaviour (customers, workers) and to organisation.
Emerging technologies, notably information and communication technology (ICT), biotechnology, and, possibly in the future, others such as nanotechnology. These new technologies have in common a large leverage effect in that they can influence entire parts of the economy. They lead to the emergence of entirely new industries, they reshape many existing sectors and they change consumer demand. This is notably the case for ICT, which is recognised as being a "general purpose technology". For these technologies, it is important not only to measure their technological evolution (increase in performance) and the resources devoted to their improvement (R&D, patents), but also their diffusion throughout the economy, which determines their impact on performance. Indicators of use, such as the number of PCs or Internet connections tell us a great deal about the impact of ICT on the economy. Definition and measurement of ICT industries and ICT-related activities such as e-commerce have progressed. Statistical activity in this field has been expanding rapidly over the past years (see OECD, 2000, for ICT, and OECD, 2001a, for biotechnology).
Circulation of knowledge: As activities devoted to the creation of new knowledge have expanded, it has become increasingly important for the entities involved to maintain connections with one another. The increasing specialisation of knowledge producers, be they companies or individual scientists, and the acceleration of change makes it even more crucial that they remain connected, as they are all sources of new knowledge for their partners. Networks are being set up (between companies, between companies and universities, between scientists, across borders), based on various types of contractual arrangements (e.g. joint-ventures, subcontracting, exchange of personnel). The mobility of people, as carriers of human capital and knowledge, between companies, universities, across borders, is becoming increasingly important. The overall performance of a national system of innovation depends critically on its ability to make knowledge available where it is needed in the economy. This is the role of a broad set of institutions, including various types of public and semi-public research organisations. Indicators should be developed that reflect the circulation of knowledge: its intensity, its patterns, the obstacles encountered. This raises a particular challenge for statisticians as it implies not looking at one entity at one point in time, but rather looking at several entities at once and at their relationships, and following them over time. When it comes to international networks or international circulation of knowledge, the challenge is more difficult as statistics are produced by national agencies, independent from each other and whose methods and concerns often differ, making it difficult to draw a consistent international picture.
Internationalisation is a major trend experienced by all technology-related activities. Multinational firms have research facilities in many different countries, R&D joint ventures are increasingly international, human resources and ideas circulate across borders (even more with the Internet), public research bodies, in the face of rising costs, co-ordinate for their research programmes and for establishing international research facilities (EC framework programmes, space station). For all countries, foreign technology is a major factor of economic growth. All of these factors, again, raise special difficulties for national statistical agencies. Greater co-ordination among them is needed, in order to put in place common definitions and procedures for the collection of data. The ongoing OECD work on a "globalisation manual" is a step in this direction, following on from the collection of data of R&D performed by affiliates of foreign multinationals which started in the mid-1990s (OECD, 2001b). Innovation surveys include questions regarding international linkages, such as sources of knowledge and alliances related to innovation.
Behaviour of firms: Innovation is more market-driven than in the past; it is influenced by the strategy and special conditions faced by individual firms. Stronger competitive pressures push firms to become more innovative and to rapidly adopt recent technology. Businesses may face obstacles to innovation such as market access (due to incumbents or to regulation) or access to funds (lack of financing for innovative SMEs). However, this new trends are not clearly visible from aggregate (country or industry level) data. Longitudinal databases must be set up and accessed by researchers. This raises statistical difficulties (tracking a same firm from year to year is not easy, especially as firms often merge or split) and legal ones (confidentiality of firm-level data). A related challenge is to capture in the statistical net new, start-up firms, which play a key role in emerging technologies, and to improve coverage of activities related to venture capital.
Output indicators: Much experience has been gained over the past decades in measuring the input to technological activities (R&D, personnel). It has long been felt, however, that output indicators are needed as well. They are needed by governments, to evaluate their programmes and researchers; they are needed by firms, who want to assess the contribution of R&D to their global achievement. Measuring output is far more difficult than measuring inputs: whereas the units in which inputs should be counted are clear (monetary units, number of people), this is not the case for outputs. What is the output of technological activities? In which unit do they show up? How can the contribution of technology to performance be isolated from other sources? What statistical sources might be used to capture outputs? For scientific activities, number of publications and citations of these are the preferred indicator, while for technology, counts of patents (and citations) are favoured. For both publications and patents, the drawbacks are well known, but much progress is being made in mitigating them. In addition, for technology, innovation surveys provide other indicators, such as the share of new products in total turnover.
Science and technology policies have changed radically over the past decade. With increasing R&D expenditure by business, the share of government, and hence its influence, has waned; the end of the Cold War has led governments to reduce their defence budget, formerly a major contributor to public R&D. In terms of innovation policy, new instruments are being experimented with, which are more market friendly than previous ones and usually involve less money but make use of more sophisticated incentive mechanisms (tax breaks, grants, soft loans, procurement). Public science is now expected to contribute more than before to social (such as the environment) and economic (transfer to industry) goals, and public research systems are being reformed accordingly -- sometimes blurring the barriers between public and private sectors. All these changes have increased the difficulty of mapping public policies: knowing the total amount of government R&D spending is no longer sufficient to obtain a clear picture of government policy. An effort in this direction was conducted at the OECD in the 1990s ("public support to industr").
The supply side
Growing and changing demand for information is a general trend in the knowledge-based economy, and the fact that this encompasses S&T indicators too is not surprising. To a large extent, progress on the supply side has permitted statisticians to adapt their products to these new conditions. On the supply side as well, statisticians are faced with the same trends as other knowledge producers.
Advances in methodology have been made in new areas, notably by the OECD and its Member countries. This has resulted in certain cases in new manuals or new statistical publications, even in new surveys. For HRST, the "Canberra Manual" (co-produced with Eurostat) was published in 1995. It gives a definition of HRST which aims to be compatible with existing classifications by occupations (ISCO) and by educational attainment (ISCED). Implementation has been partly successful to date, as internationally comparable estimates of HRST stocks and flows have been made for European countries (by Eurostat), although not yet for other countries. The basic issue is with the national classifications of occupations, which are in many cases incompatible with the international classification. Revision of the "Canberra Manual" was launched in 2001. Concerning globalisation, the "special session on globalisation" of the Statistical Working Party of the OECD Industry Committee (SWIC) has clarified conceptual aspects in the measurement of globalisation (a manual is being prepared). R&D expenditure of foreign affiliates is being collected and published on a regular basis by the OECD (e.g. OECD, 2001b). On the R&D side, the Frascati Manual is increasingly taking into account the new demands detailed above. For ICT and biotechnology, definitions have been agreed upon by OECD Member countries, allowing collection of data to start on an internationally comparable basis. Methodological work on patents has been pursued, following the "Patent Manual" (OECD, 1994).
Innovation surveys were pioneered in the early 1980s notably in a few European countries, before spreading to other OECD countries throughout the 1990s. An enormous effort has been devoted to conceptualisation and implementation in the recent years. The Oslo Manual that sets international guidelines for innovation surveys, was first published by the OECD in 1992, and revised with Eurostat in 1997. Many countries, especially European ones (Community Innovation Surveys, co-ordinated by Eurostat), but also Canada, Australia and Korea, have carried out two or three rounds. Initially designed for manufacturing industries, such surveys now cover the services as well. The purpose of innovation surveys is to collect information on the innovation behaviour of firms: the type of innovation they have carried out (or not), their innovation intensity, the cost (beyond R&D), the objectives, the obstacles, the sources of technological knowledge (e.g. competitors, universities).
The increasing power of information technology has allowed the use of very large databases with a computer power that was previously undreamed of. This is the case especially for scientific publications, for patents and for firm level data. Every year, more than one million patents are applied for worldwide, and the number continues to rise. Much valuable information is contained in each patent record. The same applies to scientific publications (the number of journals continues to grow) and to most firm-level studies (often based on panels of thousands of firms). The more computer and cheaper computers available today mean that such data can be put to statistical use.
Linking different data sources: When it comes to mapping certain areas, such as the innovating behaviour of firms, S&T networks, the mobility of human resources or government S&T policies, more than one source of data is needed in order to obtain a complete picture. Each source of data gives specific information that makes more sense when matched with other information from other sources. For instance, a good picture of innovating firms requires knowledge of their innovation activity, skill structure, product range, market shares and more; such information can be accessed only by resorting to several different data sources. Furthermore, in a globalising world, cross-country comparisons of firms are necessary, which implies matching business surveys from different countries. All of this raises few technical problems now, thanks to the advance of computer, but it does raise legal problems due to concerns about confidentiality which drastically limit access to data. Further thinking about these issues is needed in the future, by both statisticians and law makers.
Networking statisticians: The increasing diversity of data sources, methods and areas in S&T statistics means that division of labour is progressing in this field as well. It is not the same person, or the same administrative department, which is in charge of all these data in any country: specialisation is becoming the rule. Reaping the rewards from greater division of labour (in terms of more data produced and higher competence of more specialised experts), while keeping some unity (as it is important to keep some consistency between the various indicators), is not straightforward. Statisticians in charge of the various aspects of S&T will have to think of new, more open and more flexible ways of co-ordinating with each other, within as well as between countries. As their colleagues in other domains of knowledge production, they have to work increasingly in networks.
The "Blue sky activity"
The "New S&T indicators" activity was launched in 1996 by the NESTI (National Experts on S&T Indicators) and the OECD Secretariat, following the request by ministers. It was clearly part of the broader endeavour by S&T policy makers and statisticians to adapt statistics to changes in demand and to new opportunities raised by new factors on the supply side. The activity did not aim to tackle all issues mentioned above, but instead to give a clear signal that policy makers and statisticians were aware of the new conditions in S&T, and to make substantial progress in certain areas. Over the project life, active participation and regular supervision by statisticians from OECD Member countries (especially delegates to the NESTI) was key to the quality of the results. The project benefitted from financial support from the European Commission. This special issue of the STI Review reports on the major achievements of this project. They are summarised as follows.
"Constructing Internationally Comparable Indicators of the Mobility of Highly Qualified Workers: A Feasibility Study", by Mikael Akerblom, draws on in-depth work conducted in the Nordic countries. It sets a conceptual framework for measuring mobility; there are actually many different meanings to the word "mobility" (e.g. engaging in temporary training in another firm is not the same as changing employer, or changing establishment is not changing enterprise). Not distinguishing between the various types of mobility, with the proper criteria (what kind of change, for how long) might lead to misleading indicators. The article then assesses whether existing data sources are fitted for calculating such indicators. Particular emphasis is put on two "test countries", France and the United Kingdom. In addition, the article tackles certain aspects of international mobility (for further insights on this question, see OECD, 2001c).
"Innovation Surveys: Lessons from Countries' Experience", by Dominique Guellec and William Pattinson, looks at what we can discover about various features of innovation from innovation surveys of the type conducted in the Community Innovation Survey (CIS) programme, and the issues that need to be addressed to ensure that future surveys provide even better indicators and are more helpful to policy makers. Innovation surveys have substantially improved our knowledge of innovation. They have enabled investigations of phenomena that were previously not possible and have enabled the confirmation of previously unsubstantiated ideas. For instance, innovation surveys have shown that a high proportion of firms innovate; that a great deal of innovation is taking place in the services as well as in manufacturing; that innovation affects the performance of firms in terms of profitability, productivity and employment generation; and that innovation policies are concerned with large firms more than small ones. However, despite substantial progress, there are still drawbacks to innovation surveys. For instance, definitional issues (What is a technological innovation? What is an innovative firm?) have not all been solved and statistical methodologies are not identical across countries. Easier access to micro-level data to analysts will be necessary to allow the flourishing of studies that would facilitate an evaluation of the data and provide useful information to policy makers.
"To Be or Not to Be Innovative: An Exercise in Measurement", by Jacques Mairesse and Pierre Mohnen, proposes an econometric approach to innovation survey data, with a view to extracting as much information as possible from the basic data. They first check that the "micro-aggregation" procedure used for anonymising CIS data does not bias the characteristics of the population, coming to the reassuring conclusion that it does not. It is then possible for statisticians and econometricians to use micro-aggregated data with little loss of information. They propose an econometric model for explaining the innovative performance of firms; they regress the share of new products in the sales of the firm on various factors such as size, industry, country, R&D performance, membership or not of a group. They propose an "indicator of innovativeness", which is the share of new products in sales which is not explained by the factors mentioned above. This indicator is similar to total factor productivity, in the sense that it is a residual from a model estimation, which captures the own performance of the concerned country or industry when certain known factors have been accounted for.
"Investment in Knowledge", by Mosahid Khan, proposes an instrument for measuring the advance towards knowledge-based economies. Investment in knowledge is calculated as the sum of expenditure on R&D, on higher education and on software (while clearing for the overlap). Ideally, one would like to take into account other components, such as expenditure, on design, on training and on organisational change but, for the time-being, data availability imposes limits on what can be done. Certain types of intangibles are left aside in this exercise, notably expenditures related to marketing or branding, as their contribution to economic growth is not clear (although they are of course key to companies' competitiveness). The data reported in the article show that investment in knowledge is a sizeable part of GDP of OECD countries (around 5%-6% on average), and that it has been increasing more rapidly than GDP or physical investment over the 1990s.
"Counting Patents to Compare Technological Performance across Countries" by Hélène Dernis, Dominique Guellec and Bruno van Pottelsberghe examines the drawbacks associated with the traditional published patent-based indicators, and proposes solutions. Although patents have been used for calculating statistical indicators for decades, there is no broadly accepted methodology in this area. Indicators are usually weakly comparable across countries, and the years reported do not correspond to the year of invention (reflecting instead strategic or administrative delays in the submission, processing and publication of patent applications). The method proposed to solve these issues is to count only "patent families" (a set of patents taken out in various countries to protect a single invention), by "priority date" (first date of filing for protection worldwide). The use of such indicators would give a whole new picture of cross-country comparisons of technological performance.
"Improving Measures of Government Support to Industrial Technology", by Alison Young, reports a data collection exercise conducted by the OECD in 1993-98. The purpose was to collect more precise data on government programmes aimed at supporting industrial technology, and to classify it in new ways that better reflect the type of relationship with the recipient. Three broad categories are proposed: grants and subsidies (including soft loans and tax relieves), public procurement (mainly defence-related), and public infrastructure (direct and indirect contribution of public laboratories to industrial innovation). This goes far beyond the usually accessible data, which merely identify transfers of funds (from government to business), without identifying the type of underlying contract or transaction. The classification proposed in this article is clearly more appropriate to the analysis of innovation policies. Its drawbacks are that it requires large amounts of information, which is costly to collect, and which is sometimes viewed as confidential by governments.
"Measuring the Value of R&D Tax Treatment in OECD Countries", by Jacek Warda, develops the "B-index" methodology. Most OECD governments have introduced special fiscal measures for R&D, be it depreciation allowances or (increasingly often) R&D tax credits, notably for small and new firms. Their aim is to stimulate business R&D, as governments estimate that firms do not spend enough in this domain compared with other domains (such as physical investment). It is, however, difficult to compare such tax reliefs across countries as fiscal legislation is complex and multi-dimensional. The purpose of the B-index is to measure what after-tax cost of investment on R&D for a given pre-tax cost. The B-index takes into account both corporate income tax and tax reliefs related to R&D It is based on a standard approach in the fiscal literature ("effective rate of taxation"). Comparisons of OECD countries along this scale point to large heterogeneity, and increasing generosity over the recent past.
What next?
What has been the contribution of the "New S&T indicators" project? Beyond the methodological results and indicators presented in this issue of the STI Review, two types of outcomes can be seen:
For HRST, co-operation of S&T statisticians with labour and education statisticians will allow to improve the methodology ("Canberra Manual"), broaden the range of countries which collect data, and refine the detail of this data. A review of statistical sources and problems in the field of international mobility is engaged.
For a better integration of R&D data and national accounts, a task force with experts in both fields is to be set up, which will examine how R&D data could accommodate the SNA framework, and how the SNA should be adapted for taking into account the specificities of R&D.
For investment in knowledge, the methodology is still in infancy. One avenue of future work is to broaden the scope of the concept so as to better capture expenditure on training or on design. This must be related also to national accountants efforts for improving the measurement of expenditure on software, and to efforts for improving the measurement of R&D in the services so as to better capture this R&D which is not essentially on technology (which is a major component of the revision of the Frascati Manual).
For innovation, OECD is closely following progress of innovation surveys in Europe and other countries. With an expanded range of countries conducting such surveys and improving comparability of the data, further work on indicators may become possible in the near future and will be reflected in a coming revision of the Oslo Manual.
For patents, ongoing work at OECD aims to design, calculate and test various types of indicators: indicators of output (at aggregate and industry levels), indicators of linkage (between firms, between the public and business sectors, between countries, between technology lines).
Dominique Guellec
_________________
REFERENCES
OECD (1993), Proposed Standard Practice for Surveys of Research and Experimental Development -- Frascati Manual, 5th edition, OECD, Paris.
OECD (1994), "Using Patent Data as Science and Technology Indicators -- Patent Manual", OECD, Paris.
OECD and Eurostat (1995), "Manual on the Measurement of Human Resources Devoted to S&T -- Canberra Manual", OECD, Paris.
OECD and Eurostat (1997), Proposed Guidelines for Collecting and Interpreting Technological Innovation Data -- Oslo Manual, 2nd edition, OECD, Paris.
OECD (2000), "Measuring the ICT Sector", OECD, Paris.
OECD (2001a), "Compendium of Biotechnology Statistics", OECD, Paris.
OECD (2001b), Measuring Globalisation: The Role of Multinationals in OECD Economies, OECD, Paris.
OECD (2001c), International Mobility of the Highly Skilled, OECD, Paris.
OECD (2001d), Science, Technology and Industry Scoreboard of Science and Technology Indicators 2001: Towards a Knowledge-based Economy, OECD, Paris.
Documents connexes