Sensitivity analysis can dissipate some of the controversy surrounding composite indicators

It is frequently argued that composite indicators are too subjective, due to all the assumptions needed to build them:

• The model chosen for estimating the measurement error in the data;
• The mechanism for including or excluding indicators in the index;
• The transformation and/or trimming of indicators;
• The type of normalisation scheme (e.g. re-scaling or standardisation);
• The amount of missing data and the choice of imputation algorithm;
• The choice of weights (e.g. equal weights or weights derived from factor analysis and expert opinion models);
• The choice of aggregation system (e.g. additive, multiplicative, or multi-criteria analysis).

All these assumptions can heavily influence the message conveyed by a composite indicator in a way that deserves analysis and corroboration. A combination of uncertainty and sensitivity analysis can help to gauge the robustness of the composite indicator, to increase its transparency and to frame policy discussions.

Sensitivity analysis is the study of how output variation in models such as a composite indicator can be apportioned, qualitatively or quantitatively, to different sources of variation in the assumptions (Saltelli et al. 2004). In addition, it measures how the given composite indicator depends upon the information that composes it.

Sensitivity analysis is closely related to uncertainty analysis, which aims to quantify the overall variation in the countries’ ranking (score) resulting from the uncertainties in the model input. In the field of building composite indicators, uncertainty analysis is more often adopted than sensitivity analysis (Jamison & Sandbu, 2001) and the two types of analysis are almost always treated separately. A synergistic use of uncertainty and sensitivity analysis is proven to be more powerful (Saisana et al., 2005; Tarantola et al., 2000).

The types of questions for which an answer is sought via the application of uncertainty and sensitivity analysis are:

• How do country ranks (or composite indicator scores) compare to the most likely ranks under all scenarios in building the composite indicator?
• What is the optimal scenario for each country?
• Which countries are the most volatile and why?
• What are the largest influences in a composite indicator?

A plurality of scenarios (all with their implications) should be initially considered, because no scenario for the development of a composite indicator is a priori better than another, provided that internal coherence is always assured, as each scenario serves different interests. In this way, the composite indicator is no longer a magic number corresponding to crisp data treatment, weighting set or aggregation method, but reflects uncertainty and ambiguity in a more transparent and defensible fashion.

The combined use of uncertainty and sensitivity analysis during the development of a composite indicator can contribute to its well-structuring, provide information on whether the countries’ ranking measures anything meaningful and could contribute significantly to building a consensus on the usefulness and the reliability of the results of these types of measures.