Mortality Risk Valuation in Policy Assessment

Annex E. Additional sensitivity analysis and robustness checks for meta-analysis

Description and test of far-outs (outliers)

As discussed in Section 3.3 of Chapter 3 and 4.3.2 of Chapter 4 a standard outlier exclusion procedure from statistics is chosen to avoid the influence of implausibly high or low VSL estimates. In the general meta-analysis literature, it is often justified to remove an observation that is defined as an outlier in the sense that the upper bound of the 95% confidence interval of the estimate is lower than the lower bound of the pooled mean 95% confidence interval (i.e., extremely small estimates) or the lower bound of the 95% confidence interval is higher than the upper bound of the pooled mean 95% confidence interval (i.e., extremely large estimates) (Viechtbauer and Cheung, 2010[1]).

Figure A ‎E.1 shows the 95% intervals of the meta-analysis estimates using (1) all pooled estimates (N=4 128), (2) the main estimates after removing far-outs (N=3 981), (3) the upper far-out estimates (N=136) based on Tukey's fences, and (4) the lower far-out estimates (N=11). These results indicate that the removal of far-out observations is very conservative and well justified by common practice (Viechtbauer and Cheung, 2010[1]). In fact, the lower bound of the upper far-out is much higher than the upper bound of the pooled estimates (“All”) and the upper bound of the lower far-out is much lower than the lower bound of the pooled estimates (“All”) (cf. Section 4.3.1 of Chapter 4).

Figure A ‎E.1. Mean and confidence bands of far-outs (outliers) compared to all data

Note: “Far-outs (Upper end)” (Lower end) are far-out estimates removed from the main analysis (“Without Far-outs”) because they exceed/are inferior to the far-out upper (lower) threshold.

Sensitivity analysis of estimated mean VSL to main datasets

Figure A ‎E.2 shows the PUVSL estimates and 95% confidence intervals from the Old SP and RP databases only. Compared to the corresponding estimates for New SP and RP data (Figure 4.2 and Table 4.2 of Chapter 4), some mean estimates are higher and some are lower. For example, All countries, OECD and United States PUVSLs increased from USD 5.5 to 7.3, 7.1 to 7.6 and 7.2 to 9 million, respectively. For EU and EEA, PUVSLs decreased from USD 7.6 to 5.4 and 7.4 to 5.6 million, respectively. For the lower- and middle-income group, the older dataset gives much higher PUVSL, (USD 2.6 million compared to USD 0.9 million).

Note that the number of estimates of each country group is lower for all groups making confidence intervals somewhat wider for the older data compared to the new.

Figure A ‎E.2. Estimated PUVSL for country groups based on “old” data

Old meta-data

Note: Estimates based on four-level random effects meta-analysis. 99 far-out estimates excluded. Old meta-data refers to studies published in the period 1970-2008.

Figure A ‎E.3 shows the estimated mean VSL and 95% confidence intervals based on the merged data (all four datasets). In this case, the mean VSL estimates are not substantially different when merging the old dataset with the new; some country group estimates are somewhat higher, and some are lower. The increase in total number of estimates narrows the confidence intervals slightly.

Figure A ‎E.3. Estimated PUVSL by country group based on “all” data

All data

Note: Estimates based on four-level random effects meta-analysis.

Sensitivity of mean VSL to econometric specifications

Figure A ‎E.4 below shows sensitivity of mean VSL for all countries (i.e. all new SP and RP data) to three alternative econometric meta-analysis specifications: Fixed effect meta-analysis, two-levels random effects meta-analysis and three-levels random effects meta-analysis specification. The four-level random effects approach is the one used to estimate mean VSL in the main model (Table 4.2 of Chapter 4). Results show that the three- and four-level random effects models yield identical mean VSL estimates of USD 5.4 and 5.5 million, while the two-level model yields a mean VSL estimate of USD 4.9 million.

Figure A ‎E.4. Sensitivity of mean VSL to alternative econometric meta-analysis specifications

New meta-data

Note: Sensitivity test (alternative specification). New meta-data refers to studies published in the period 2009-2023.

Sensitivity of mean VSL to exclusion of outliers and far-outs

The main analysis in Chapter 4 has excluded so-called far-outs, i.e. outliers or extreme values. Figure A ‎E.5 below shows the sensitivity of the estimated mean VSL for all countries of USD 5.5 million from Table 4.2 of Chapter 4. This estimate is shown by the dot and the CI by the middle line in the figure, where far-outs have been excluded. Above this line is the mean VSL estimate if all reported VSL data is included (i.e. not excluding far-outs or outliers). In this case, the mean VSL estimate increases by USD 300 000 to USD 5.8 million. If instead a stricter exclusion criterion is used, i.e. considering more of the distribution as outliers, the mean VSL estimate decreases from USD 5.5 million to USD 5.3 million instead (bottom dot and CI). Hence, overall, the mean VSL is not very sensitive to either of these criteria.

Figure A ‎E.5. Sensitivity of estimated mean VSL to exclusion of outliers and far-outs

New meta-data

Note: Sensitivity test (Far-outs). Far-outs/Outliers are defined as outside the range of $[p25-k(p75-p25), p75+k(p75-p25\left)\right]$ where $k$=3 for far-outs and $k$=1.5 for outliers. New meta-data refers to studies published in the period 2009-2023.

Sensitivity analysis of estimated mean VSL to quality and other screening factors

Studies that are nationwide

In some contexts, it may be relevant to limit the data to studies that have a nationwide coverage, rather than specific geographical areas of a country (especially if there are potentially important differences between areas that are difficult to control for). Figure A ‎E.6 provides VSL estimates by group of studies, i.e. whether the VSL estimates come from a nationwide study or not. The “pooled” estimate is the USD 5.5 million for all countries, as shown in Table 4.2 of Chapter 4. Limiting the scope to nationwide studies increases this estimate to USD 6.9 million. On the other hand, the VSL estimate for studies which are not nationwide is USD 3.7 million. It is difficult to assess the reasons for such a difference without more detailed analysis and information about each primary study.

Figure A ‎E.6. Sensitivity of mean VSL if study is nationwide

New meta-data

Note: Sensitivity test (Nationwide). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Studies of the general population

Some VSL studies target the general population, while others target specific groups. In the same way as for the nationwide approach, it may in some contexts be desirable to limit the underlying sample of VSL estimates to studies that cover the general population. Figure A ‎E.7 shows that the difference in the mean VSL estimates between the general population and specific groups is around USD 2.8 million, as opposed to a mean VSL estimate of USD 5.3 million when pooled (Table 4.2 of Chapter 4). Similar to the results comparing national versus sub-national studies, the studies that focus on the general population tend to have been performed in economies with a higher per capita income which contributes to such studies showing a higher VSL estimate.

Figure A ‎E.7. Sensitivity of mean VSL if study is of the general population

New meta-data

Note: Sensitivity test (General Population). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Representative samples

It is often hard to judge how representative the samples are that VSL estimates are based upon. In addition, few studies provide sufficient information to be able to adjust for any non-representativeness. Representativity is affected by sampling strategy, which ideally should be random and based on a sampling frame that represents the population. However, in practice, non-response bias and sample selection effects can play a role, even if the sampling was randomised. Despite such potential risks, the use of random sampling can be seen as a proxy for the quality of a study. Comparing studies that have use such a sampling strategy with those that do not yield some differences, as shown in Figure A ‎E.8. The results suggest that non-random sampling strategies yield generally higher VSL estimates. The reasons for this are not clear, but could be affected by income, which is slightly higher for the studies in the non-random sample group. Limiting the data to only representative estimates lowers the main mean VSL estimate for All countries from USD 5.4 million to USD 4.9 million.

Figure A ‎E.8. Sensitivity of mean VSL if representative sample

New meta-data

Note: Sensitivity test (Representativeness). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

No standard errors reported

It is generally considered good practice to report standard errors (SE) of any statistical metric in a scientific study, including for mean VSL estimates. Even so, many studies do not report this (cf. Table A D.1 in Annex D). This could be a rough indicator of lack of quality (though some of the studies report confidence intervals, but not standard errors). Figure A ‎E.9 below compares VSL estimates between studies that report standard errors and those that do not, suggesting that studies that include standard errors show a lower overall VSL estimate.

Figure A ‎E.9. Sensitivity of mean VSL if study does not report standard error of VSL estimate

New meta-data

Note: Sensitivity test (No reported standard errors). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Baseline risks clearly defined in SP

In theory, the baseline risk should affect VSL. If the baseline risk is not clearly defined in the primary SP survey, it could suggest a lower quality of the SP study. Figure A ‎E.10 suggests that the articulation of base risk impacts the VSL: When baseline risk is clearly specified, mean VSL is estimated to USD 5.9 million, and USD 3.9 million when it is not. The pooled estimate is USD 4.8 million (which is different from the USD 5.5 million in Table 4.2 of Chapter 4 for all countries, because results are derived from SP data only). Note that in this case income is actually higher for those studies where baseline risk is not clearly explained (not shown in Figure A ‎E.10).

Figure A ‎E.10. Sensitivity of mean VSL to the definition of baseline risk

New meta-data

Note: Sensitivity test (SP study including Baseline Risk). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Scope sensitivity test included in SP

Scope sensitivity – the phenomenon that people generally should be willing to pay more for a large mortality risk reduction than a small one – has been important for assessing validity of SP studies since the National Oceanic and Atmospheric Administration panel on the CV method delivered its report (Arrow, 1993[2]). In recent years, the emphasis on scope sensitivity has moved somewhat from consideration of validity to the degree of economic significance of such results (Dugstad et al., 2021[3]; Kipperberg et al., n.d.[4]). One reason is that studies have shown that scope sensitivity could be relatively low, but still be in line with economic theory (Amiran and Hagen, 2010[5]). The meta-data for the current study includes a variable on whether a scope test was reported in the primary SP study. Figure A ‎E.11 shows that studies with scope tests have a mean VSL estimate that is about USD 2 million higher than those do not include a scope test.

Figure A ‎E.11. Sensitivity of mean VSL if study includes a scope test

New meta-data

Note: Sensitivity test (SP study focusing on Scope Sensitivity). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Visual aid used to display risk for SP

In the old SP database, there was a variable indicating whether or not a SP study had used visual display or other proper explanation of the risk reduction (e.g. a risk ladder, grid or similar). At the time (and still) this was seen as best practice. This coding procedure was extended into the new SP database. Figure A ‎E.12 shows that studies that do not have such explanation yield lower VSL estimates. This is contrary to the results in Lindhjem et al. (2011[6]) who find that such studies yield higher VSL estimates for the old SP database. There is some degree of subjectivity of the coding of this variable, so results here should be interpreted accordingly.

Figure A ‎E.12. Sensitivity of mean VSL if SP survey explained risk change using visual aid

New meta-data

Note: Sensitivity test (SP study including Visual Aids). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Instrument variable regression for RP

The use of instrument variables is common in hedonic wage risk studies (cf. Section 4.3 of Chapter 4). Figure A ‎E.13 shows that the use of instrument variables in the estimation of the VSL in RP studies has a relatively minor effect on the mean VSL estimate (a difference of about USD 300 000).

Figure A ‎E.13. Sensitivity of mean VSL if the RP study used instrument variable regression

New meta-data

Note: Sensitivity test (RP study using IV Estimate). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Non-fatal injury risk included in RP

Another factor that is considered important in high-quality hedonic wage studies is to control for non-fatal injury risk in the regressions. When controlling for such risks, the mean VSL estimate is USD 7.6 million compared to USD 6.1 million when not controlling for this risk (Figure A ‎E.14).

Figure A ‎E.14. Sensitivity of mean VSL to control for non-fatal injury risks

New meta-data

Note: Sensitivity test (RP study controlling for Non-Fatal Injury Risks). A three-level random effects meta-analysis model is used. New meta-data refers to studies published in the period 2009-2023.

Census of fatal occupational injuries (CFOI) data in RP

Whether the highest quality risk data is used in hedonic wage risk studies is often considered important for quality. For example, in the case of the United States, the highest quality risk data is generally considered to come from the Census of Fatal Occupational Injuries (CFOI). When such data is used, it has a substantial impact on the VSL estimates, as can be seen from Figure A ‎E.15 (with the caveat that income also appears higher in that sample – not shown).

Figure A ‎E.15. Sensitivity of mean VSL to the use of CFOI data

New meta-data

Note: Sensitivity 3 Study using CFOI data (Quality controls). A three-level random effects meta-analysis is used. New meta-data refers to studies published in the period 2009-2023.

Mean VSL estimates from studies published in journals

Table A ‎E.1 shows PUVSL results where the underlying primary studies are limited to those published in peer reviewed journals (compare to Table 4.2 of Chapter 4). The results suggest that for the full sample (“all countries”) the PUVSL increases from 5.5 to 5.9 million when limiting the sample to peer reviewed journal-published studies. Estimates for OECD and High-income countries also increase slightly, while estimates for EEA decline. The largest change is observed for the United States’s estimate, which increases from USD 7.2 to 8.7 million, though the reasons for this substantial increase remain unclear.

Table A ‎E.1. Estimated mean PUVSL estimates for (groups of) countries, limited to studies published in journals

USD million, New meta-data

Countries	Mean VSL	Confidence interval (lower bound)	Confidence interval (upper bound)	Number of studies	Number of estimates
All countries	5.9	4.1	7.7	137	1720
OECD	7.5	5.6	9.4	93	1270
EU	7.1	6.0	8.2	34	342
EEA	6.8	5.7	8.0	36	363
United States	8.7	7.2	10.3	30	605
High income countries	7.6	5.9	9.3	98	1358
Low/Middle-income countries	1.0	0.3	1.7	39	362

Note: New meta-data refers to studies published in the period 2009-2023.

References

[5] Amiran, E. and D. Hagen (2010), “The scope trials: Variation in sensitivity to scope and WTP with directionally bounded utility functions”, Journal of Environmental Economics and Management, Vol. 59/3, pp. 293-301, https://doi.org/10.1016/J.JEEM.2009.06.003.

[2] Arrow, K. (1993), Report of the NOAA Panel on Contingent Valuation, National Oceanic and Atmospheric Administration.

[3] Dugstad, A. et al. (2021), “Scope Elasticity of Willingness to pay in Discrete Choice Experiments”, Environmental and Resource Economics, Vol. 80/1, pp. 21-57, https://doi.org/10.1007/S10640-021-00577-7.

[4] Kipperberg, G. et al. (n.d.), “(Forthcoming). Scope effects in stated preference research: From construct validity to economic significance. Handbook of Nonmarket Valuation.”.

[6] Lindhjem, H. et al. (2011), “Valuing Mortality Risk Reductions from Environmental, Transport, and Health Policies: A Global Meta-Analysis of Stated Preference Studies”, Risk Analysis, Vol. 31/9, pp. 1381-1407, https://doi.org/10.1111/J.1539-6924.2011.01694.X.

[1] Viechtbauer, W. and M. Cheung (2010), “Outlier and influence diagnostics for meta-analysis”, Research Synthesis Methods, Vol. 1/2, pp. 112-125, https://doi.org/10.1002/JRSM.11.