Measuring Carbon Footprints of Agri‑Food Products

9. Ensuring the integrity and quality of the data

Abstract

The sixth building block for carbon footprints is ensuring the integrity and quality of the data. This chapter explains how assurance can be used to ensure that a product carbon footprint calculation is the result of applying the appropriate methodology to the right data. The chapter also discusses some gaps in the current landscape and how these could be addressed.

A firm shares its product carbon footprint estimate with a customer. How can the customer be confident that the estimate is reliable?

Part of the answer lies in the building blocks discussed earlier. A product carbon footprint estimate is more likely to be reliable if it was calculated following clear reporting standards and validated science-based methods embedded in farm level tools or databases with secondary data.

But this is not the full answer. After all, even if the methods used are in principle sound, and the firm provided the correct information as inputs in the calculation, how would the customer know? If the firm used information provided by its suppliers, the customer’s challenge in verifying the calculations becomes even more daunting.

Reliable and widespread carbon footprints in food systems thus require a way of ensuring the integrity and quality of the data. To be more precise, what is needed is a way to guarantee that a product carbon footprint calculation is the result of applying the appropriate methodology to the right data.

A few examples can clarify what this means in practice. First, consider a food manufacturer who combines several ingredients into a final product, with minimal processing (e.g. muesli). If the food manufacturer does not have primary data from its suppliers, the product carbon footprint can be calculated by multiplying the amounts of each ingredient (the activity data) with a product carbon footprint from a secondary database (the emission factor). Checking the product carbon footprint calculation here means checking whether the activity data are correct (did the manufacturer not omit or understate the amounts of each ingredient?), whether the emission factor comes from a relevant and reliable database, and whether the calculation did not contain any errors.

Second, consider a farmer using a farm level calculation tool. The farmer enters activity data into the tool, and receives an estimate for farm level emissions. Checking the calculation here means checking the activity data (did the farmer not omit or understate anything?), checking whether the calculation tool used is good, and checking whether the tool indeed returns the same estimate when fed the activity data.

Third, consider a farmer calculating product carbon footprints using the cradle-to-gate approach, using primary data for on-farm emissions and some data from suppliers (e.g. from the fertiliser company) and from secondary databases for emissions embedded in inputs. The farmer then uses allocation rules to arrive at product carbon footprints for the different outputs. In this scenario, there are additional items to be checked. How reliable is the data provided by the supplier? Did the farmer use the correct allocation rules? Was the overall calculation consistent with the relevant standards (e.g. in terms of system boundaries)?

These examples illustrate the questions which must be answered to ensure the quality and integrity of product carbon footprint data in food systems. Fortunately, assurance (in particular, third-party verification) is generally well developed, including for carbon footprints. But the current system still leaves some important gaps.

9.1. How assurance works

Assurance refers to the “demonstration that specified requirements relating to a product, process, system, person or entity are fulfilled.” The terms “conformity assessment”, “certification” and “verification” are also used (ISEAL, 2023[1]).

Assurance is widespread in food systems. Examples include assessing whether a firm or product meets organic standards, ISO 9000 quality management standards, HACCP food safety standards, and more.

The details of how assurance is organised depend on the context.

• In some cases “first party” assurance (i.e. a self-declared claim) is accepted. For example, a “Supplier Declaration of Conformity” is a document where a supplier declares that a product, process, or service conforms to certain requirements (von Lampe, Deconinck and Bastien, 2016[2]).

• In other cases “second party assurance” (by, for example, a supplier, customer, or contractor) could be appropriate. For example, a buyer could inspect a product to make sure it meets its own requirements.

• In yet other cases, assurance is performed through an independent third party (also known as a conformity assessment body, a validation or verification body, or a certifier).

These different types of assurance could each be appropriate depending on the circumstances. In general, first-party assurance is cheaper and easier but also provides less confidence than third-party assurance. This may create a trade-off, especially in dealing with smaller producers and producers based in developing countries. While the remainder of this chapter discusses third-party assurance, it is useful to consider whether first-party or second-party assurance might be an appropriate alternative (Chapter 10).

The ISO 17029:2019 standard lays out general principles and requirements for validation and verification bodies. ISO 14065:2020 takes these general requirements and provides further detail related to validating and verifying environmental information, while ISO 14064-3:2019 is a standard that provides guidelines for the validation and verification of GHG assertions specifically. The latter standard is part of the ISO 14064 standards, which outline the principles and requirements for the quantification, monitoring, reporting, and verification of GHG emissions and removals.

National Accreditation Bodies such as ANAB in the United States, JAB in Japan, UKAS in the United Kingdom, ONAC in Colombia or DAkkS in Germany can accredit organisations to perform third-party verification and certification services. In Germany, for example, there are 90 organisations accredited in accordance with ISO 17029 or ISO 14065 in the agriculture, food, and sustainability category.

Third-party verification of a product carbon footprint claim consists of two steps. First, the verification body checks whether the calculation followed the methodology it claims to use (for example the ISO 14067 product carbon footprint standard). Second, the verification body checks the activity data used in the calculation (e.g. whether the data can be traced back to reliable sources or records, whether there are possible errors, and what their impact would be).

9.2. A first assessment

Third party verification of product carbon footprints is widespread, but it does not evaluate the methodology itself, merely that whatever methodology chosen has been followed. Where emission factors from a secondary database are used the verification body would check whether the database is relevant and up to date, but it would typically not evaluate the quality of the data in the database. Similarly, where a farm level calculation tool is used the verification body would not evaluate the tool itself. The quality of the databases and farm level tools would be considered part of the methodology, and hence outside the scope of third-party verification of product carbon footprints.

This leaves important gaps. First, farm level tools and secondary databases should ideally follow widely used reporting standards. Second, even if tools and databases follow the relevant standards, there are many methodological questions which can influence results (such as the choice of specific emissions models). Neither of these two important questions are addressed in a third-party verification of a firm’s product carbon footprint calculations.

Other types of third-party verification can be used to address some of these gaps. For example, farm level calculation tools often state that they are compliant with reporting standards such as ISO 14067, GHG Protocol, or even sectoral guidelines such as the IDF product carbon footprint guidelines for the dairy sector (Chapter 6). It is possible for tools to seek explicit third-party verification of their alignment with these standards. In the chemicals industry, for example, BASF developed a digital tool to calculate cradle-to-gate product carbon footprints for its own product portfolio (Box 8.1 in Chapter 8). A third-party verification confirmed that the tool is aligned with ISO 14067:2018 and the GHG Protocol Product Standard. Farm-level tools could follow a similar approach. Another possibility would be for standard setters themselves to list the tools and databases which are consistent with their requirements. In the past, the GHG Protocol did so through its “Built on GHG Protocol” programme, and providers of product category rules and sectoral guidelines could similarly indicate which tools and databases are consistent.

This still leaves the question of how to decide which tools and databases are the most suitable in a given context. Two farm-level tools or databases could both follow the same standards but make different modelling choices, leading to different results. Moreover, not every tool or database is well adapted to every geographic or technological context. Different options exist. For example, where governments have provided guidance on science-based methods (Chapter 5), third-party verification could confirm that a tool is indeed following these methods. Another option is an independent scientific assessment process which validates tools and databases best suited for specific contexts (e.g. a list of farm level tools most appropriate for crop farming in Canada). This process could function similarly to proficiency tests (Box 9.1).

A third option is to define minimum quality criteria for farm level tools (e.g. in terms of governance, transparency, independent scientific oversight) and assess tools based on those criteria.

Hence, while assurance is widespread, there are some important gaps that need to be addressed in order to guarantee that a product carbon footprint calculation is the result of applying the appropriate methodology to the right data.

Box 9.1. Proficiency testing

The question of how to evaluate the reliability of farm-level tools is similar to the question of how to evaluate the reliability of different laboratories, conformity assessment bodies, or measurement devices. Proficiency testing (also sometimes referred to as “ring trials”) are frequently used in that context (Johnson and Cabuang, 2021[3]) and could be a useful model for farm-level tools. The discussion here will use laboratories as example, although similar ideas apply to conformity assessment bodies or measurement devices.

Proficiency testing refers to an ongoing periodic assessment of test performance of different laboratories, where results are compared to the results of other participants and/or reference standards. Typically, this involves laboratories performing the same test on the same samples. The process of proficiency testing is organised by an independent provider (the ISO/IEC 17043 standard sets out the requirements for such providers).

Proficiency testing is not only useful to assess the performance of different laboratories, but can also be a powerful tool to improve performance over time. Johnson and Cabuang (2021[3]) provide several examples in the context of animal disease testing.

Comparing tests results from different laboratories against a known reference has the advantage of being able to detect and reduce both systematic errors (bias) and non-systematic errors (noise). In the context of agricultural emissions, knowing the “true” emissions may be difficult. However, even when the true value is unknown, a comparison of different results could be useful in identifying and reducing noise. Counterintuitively, reducing noise can improve performance even if the true value is unknown (Kahneman, Sunstein and Sibony, 2021[4]).

References

[1] ISEAL (2023), ISEAL Code of Good Practice V1.0, https://www.isealalliance.org/get-involved/resources/iseal-code-good-practice-sustainability-systems-v10.

[3] Johnson, P. and L. Cabuang (2021), “Proficiency testing and ring trials”, Revue Scientifique et Technique de l’OIE, Vol. 40/1, https://doi.org/10.20506/rst.40.1.3217.

[4] Kahneman, D., C. Sunstein and O. Sibony (2021), Noise: A Flaw in Human Judgment, William Collins.

[2] von Lampe, M., K. Deconinck and V. Bastien (2016), “Trade-Related International Regulatory Co-operation: A Theoretical Framework”, OECD Trade Policy Papers, No. 195, OECD Publishing, Paris, https://doi.org/10.1787/3fbf60b1-en.