SOcial multi CRiteria AssessmenT of European policieS

Quantitative evidence plays an important role in many Impact Assessments (IAs), but also qualitative data such as stakeholder input, conclusions of evaluations, as well as scientific and expert advice are frequently used. This generates a multitude of criteria of varying nature, which should be consistently integrated and evaluated when comparing policy options. The most widespread multidimensional approach to ex-ante IAs is multi-criteria decision analysis (MCDA), which forms the basis for social multicriteria evaluation (SMCE), which has been explicitly designed for public policy. SMCE allows taking into account a wide range of assessment criteria, such as the impact on SMEs, the degree of protection of fundamental rights, consumer protection, etc. while all the multidimensional profiles of the problem remain in their original scales of measurement.

SOCRATES ( SOcial multi CRiteria AssessmenT of European policieS ) is a new multiple criteria software tool designed to implement SMCE. Developed by the Joint Research Centre, SOCRATES has been explicitly designed for ex-ante Impact Assessment (IA) problems. Overall, the objective of SOCRATES and the underlying SMCE methodology is not to substitute policy-makers through a mathematical model, but to improve their understanding of the main features of the problem at hand, such as key assumptions, degree of uncertainty, robustness of results and overall technical and social defensibility of options chosen.

While SMCE has already been applied in a multitude of policy problems since, its recent technical implementation SOCRATES is now applied to support EC impact assessments, starting with DG SANTE.

SOCRATES implements the methodology Social Multi-Criteria Evaluation (SMCE). In the following we therefore describe both in more detail:

SMCE

An SMCE framework is useful for dealing with the following question: how can the Commission integrate a plurality of technical aspects and social views into its ex-ante impact assessment in a coherent and transparent manner? SMCE allows taking into account a wide range of assessment criteria; for example, the impact on SMEs, the degree of protection of fundamental rights, consumer protection, etc. All the multidimensional profiles of the problem are shown in their original scales of measurement; this is the main difference with traditional cost-benefit analysis (CBA), which grounds on steps like monetizing all social, environmental, and human rights aspects. In this respect, CBA and SMCE are not conflictual but complementary: CBA can easily be a component of a SMCE framework, dealing with the economic dimension.

The implementation of a Social Multi-Criteria framework involves the following main steps:

1. Selection of the social actors relevant for the problem at hand.
2. Definition of social actors’ values, desires and preferences.
3. Generation of evaluation criteria as a process of technical translation of social actors’ needs, preferences and desires. 
4. Construction of the multi-criteria impact matrix.
5. Construction of an equity impact matrix, illuminating all the distributional consequences of each single option in terms of stakeholder types.
6. Application of a mathematical procedure. This is normally done by using a software tool.
7. Sensitivity and robustness analysis.

The impact matrix presents in a structured way, the information on the various criterion scores, i.e. each element of the matrix represents the performance of each option according to each evaluation criterion. In general, in a multi-criterion problem, there is no “ideal” solution optimizing all the criteria at the same time, and therefore “compromise solutions” have to be found.

The importance of mathematical approaches in SMCE is their ability to allow a consistent aggregation of the diverse information. Otherwise, even if everybody would agree on the multidimensional nature of an IA study, the implementation in a real-world assessment exercise would be impossible. The standard objection might be that the aggregation of apples and oranges is impossible. Multi-criteria mathematics does answer to this objection in a definitive way. When using mathematical rules, consistency between the problem structuring and the ranking of policy options is guaranteed, this makes the overall IA study much more defensible.

In summary a SMCE approach can supply a methodological framework where the hierarchical structure of the option comparison step of a typical ex-ante IA (including dimensions, objectives and evaluation criteria) is clarified as much as possible by means of well-established concepts in the decision theory literature. This might help in increasing the degree of homogeneity across IA studies.

SOCRATES

The application of SMCE is not particularly time consuming, since it formalises in a consistent and efficient way a process that often is already done in the current practice of IA (almost all IA studies present the results in the form of an impact matrix). Moreover, JRC has developed SOCRATES (SOcial multi-CRiteria AssessmenT of European policieS), to support SMCE, which makes all required computations very quick. Three main components constitute the core of SOCRATES: multi-criteria, equity and sensitivity analyses.

From a mathematical point of view, the information contained in the impact matrix useful for solving the so-called multi-criterion problem is:

• Intensity of preference (when quantitative criterion scores are present).
• Number of criteria in favour of a given alternative.
• Weight attached to each single criterion.
• Relationship of each single alternative with all the other alternatives.

Combinations of this information generate different aggregation conventions, i.e. manipulation rules of the available information to arrive at a preference structure. The aggregation of several criteria implies taking a position on the fundamental issue of compensability. Compensability is a very important concept when MCDA is applied to integrate various policy dimensions. For example, in evaluating a policy option that presents a very bad environmental impact and a very good economic impact, it is clear that allowing or not for compensability and to which degree is the key assumption.

An aggregation rule that is simple, non-compensatory and minimises the rank reversal phenomena is the kemeny rule. Its basic idea is that the maximum likelihood ranking of policy options is the ranking supported by the maximum number of criteria (or criterion weights) for each pair-wise comparison, summed over all pairs of options considered. There is agreement in the literature that the Kemeny method is “the correct method” for ranking options, and that the only drawback of this aggregation method is the difficulty in computing it when the number of options grows. A numerical algorithm solving this computational drawback in an efficient way has been developed recently at JRC and it has been implemented in SOCRATES.

Various authors have argued that the presence of qualitative information in evaluation problems concerning socio-economic issues is a rule, rather than an exception. Thus there is a clear need for methods that are able to take into account information of a "mixed" type (both qualitative and quantitative criterion scores). Moreover, ideally, this information should be precise, certain, exhaustive and unequivocal. Nevertheless, in reality, it is often necessary to use information which does not have those characteristics so that one has to face the uncertainty of a stochastic and/or fuzzy nature present in the data. Therefore, multi-criteria methods able to tackle consistently the widest types of mixed information should be considered as desirable ones in the IA framework.

From a mathematical point of view, SOCRATES deals with two main issues:

1. the problem of equivalence of the procedures used in order to standardize the mixed criterion scores;
2. the problem of comparison of fuzzy numbers typical of all fuzzy multi-criteria methods.

These two issues are dealt with a new semantic distance that is useful in the case of continuous, convex membership functions also allowing a definite integration.

Overall, the objective of SOCRATES is NOT substitution of policy-makers through a mathematical model, on the contrary, the objective is to improve their understanding of the main features of the problem at hand, such as key assumptions, degree of uncertainty, robustness of results and overall technical and social defensibility of options chosen. The philosopher Socrates said ”I cannot teach anybody anything. I can only make them think.” This is the main inspiring principle of the SOCRATES software too.

The SOCRATES software offers a measurement framework where the various criterion scores can assess impacts by using both quantitative (e.g. as result of simulation models) and qualitative (e.g. results of participatory techniques) information, and the mathematical aggregation rule guarantees consistency and transparency of results.

Three main components constitute the core of SOCRATES: multi-criteria, equity and sensitivity analyses. Multi-criteria analysis requires the definition of relevant dimensions, objectives and criteria. It uses weights as importance coefficients and clarify their role in the hierarchical structure. The impact matrix may include both quantitative (including also stochastic and/or fuzzy uncertainty) and qualitative (ordinal and/or linguistic) measurements of the performance of an alternative with respect to an evaluation criterion. It supplies a ranking of the alternatives according to the set of evaluation criteria (i.e. the technical compromise solution/s).

Equity analysis requires as input a set of social actors and their qualitative evaluation of the alternatives considered in the multi-criteria analysis. The equity analysis produces the following information:

• indications of the distance of the positions of the various social groups (i.e. possibilities of convergence of interests or coalition formations);
• ranking of the alternatives according to actors’ impacts or preferences (social compromise solution).

The objective of sensitivity analysis is to check if the rankings provided are stable and to determine which of the input parameters influence more the model output. The whole information produced by local and global sensitivity analyses is synthesised into simple graphics.

SMCE proceeds on the basis of following main concepts: dimensions, objectives, criteria, weights, criterion scores, impact matrix and compromise solution.

• Dimension is the highest hierarchical level of analysis and indicates the scope of objectives, criteria and criterion scores. In IA studies, the general categories of economic, social and environmental impacts are dimensions.
• Objectives indicate the direction of change desired, e.g. growth has to be maximized, social exclusion has to be minimized, carbon dioxide emissions have to be reduced.
• A criterion is a function that associates alternative actions with a variable indicating its desirability.
• Weights are often used to represent the relative importance attached to dimensions, objectives and criteria. The idea behind this practice is very intuitive and easy, that is, to place the greatest number in the position corresponding to the most important factor.
• A criterion score is an assessment of the impact consistent with a given criterion with reference to a policy option. Criterion scores can be either qualitative or quantitative.
• The impact matrix presents in a structured way, the information on the various criterion scores, i.e. each element of the matrix represents the performance of each option according to each criterion.

In general, in a multi-criterion problem, there is no solution (ideal or utopia solution) optimizing all the criteria at the same time, and therefore “compromise solutions” have to be found.

A typical SOCRATES input requires the definition of policy options (called alternatives) dimensions, objectives and criteria. This information leads to the construction of an impact matrix, which may include crisp, stochastic or fuzzy measurements of the performance of an alternative with respect to an evaluation criterion. Qualitative information can be introduced too (in the form of linguistic or ordinal criterion scores). Weights as importance coefficients, may also be introduced. They can be attached to dimensions or criteria. Indifference and preference thresholds can also be introduced when needed. Generally a social conflict matrix is also constructed, where the impacts of each policy option on each social group are presented in a transparent way.