Beyond Pairwise Comparisons in Social Choice:A Setwise Kemeny Aggregation Problem

Beyond Pairwise Comparisons in Social Choice:
A Setwise Kemeny Aggregation Problem

Hugo Gilbert, Tom Portoleau and Olivier Spanjaard
1. Gran Sasso Science Institute (GSSI), L’Aquila, Italy,
2. LAAS-CNRS, IRIT-CNRS, Université de Toulouse, Toulouse, France,
3. Sorbonne Université, CNRS, Laboratoire d’Informatique de Paris 6, LIP6, Paris, France,

In this paper, we advocate the use of setwise contests for aggregating a set of input rankings into an output ranking. We propose a generalization of the Kemeny rule where one minimizes the number of -wise disagreements instead of pairwise disagreements (one counts 1 disagreement each time the top choice in a subset of alternatives of cardinality at most differs between an input ranking and the output ranking). After an algorithmic study of this -wise Kemeny aggregation problem, we introduce a -wise counterpart of the majority graph. It reveals useful to divide the aggregation problem into several sub-problems. We conclude with numerical tests.

1 Introduction

Rank aggregation aims at producing a single ranking from a collection of rankings of a fixed set of alternatives. In social choice theory (e.g., moulin1991axioms moulin1991axioms), where the alternatives are candidates to an election and each ranking represents the preferences of a voter, aggregation rules are called Social Welfare Functions (SWFs). Apart from social choice, rank aggregation has proved useful in many applications, including preference learning (chengHullermeier2009, ClemenconKS18), collaborative filtering (wang2014vsrank), genetic map creation (jackson2008consensus), similarity search in databases systems (fagin2003efficient) and design of web search engines (altman2008axiomatic, dwork2001rank). In the following, we use interchangeably the terms “input rankings” and “preferences”, “output ranking” and “consensus ranking”, as well as “alternatives” and “‘candidates”.

The well-known Arrow’s impossibility theorem states that there exists no aggregation rule satisfying a small set of desirable properties (arrow1950difficulty). In the absense of an “ideal” rule, various aggregation rules have been proposed and studied. Following Fishburn’s classification (fishburn1977condorcet), we can distinguish between the SWFs for which the output ranking can be computed from the majority graph alone, those for which the output ranking can be computed from the weighted majority graph alone, and all other SWFs111 Fishburn’s classification actually applies to social choice functions, which prescribe a subset of winning alternatives from a collection of rankings, but the extension to SWFs is straightforward.. The majority graph is obtained from the input rankings by defining one vertex per alternative and by adding an edge from to if is preferred to in a strict majority of input rankings. In the weighted majority graph, each edge is weighted by the majority margin. The many SWFs that rely on these graphs alone take therefore only pairwise comparisons into account to determine an output ranking. For a compendium of these SWFs, the interested reader may refer for instance to the book by brandt2016handbook (brandt2016handbook).

The importance of this class of SWFs in social choice theory can be explained by their connection with the Condorcet consistency property. This property states that, if there is a Condorcet winner (i.e., an alternative with outgoing edges to every other alternatives in the majority graph), then it should be ranked first in the output ranking. Nevertheless, as shown for instance by baldiga2010choice (baldiga2010choice), the lack of Condorcet consistency is not necessarily a bad thing, because this property may come into contradiction with the objective of maximizing voters’ agreement with the output ranking. The following example illustrates this point.

Example 1 (Baldiga and Green, baldiga2010choice).

Consider an election with 100 voters and 3 candidates , where 49 voters have preferences , 48 have preferences and 3 have preferences . Candidate defeats every other candidate in the strict pairwise majority sense (Condorcet winner), but is the top choice of only 3 voters. In contrast, candidate is in slight minority against and , but is the top choice of 49 voters. This massive gain in agreement may justify to put instead of in first position of the output ranking.

Following Baldiga and Green (baldiga2010choice), we propose to handle this tension between the pairwise comparisons (leading to ranking first) and the plurality choice (leading to ranking first) by using SWFs that take into account not only pairwise comparisons but setwise contests. More precisely, given input rankings on a set of candidates and , the idea is to consider the plurality score of each candidate for each subset such that , where the plurality score of for is the number of voters for which is the top choice in . The results of setwise contests for the preferences of Example 1 are given in Table 1 for . Note that the three top rows obviously encode the same information as the weighted majority graph while the bottom row makes it possible to detect the tension between the pairwise comparisons and the plurality choice.

49 51
49 51
52 48
49 3 48
Table 1: Results of setwise contests in Example 1.

One can then define a new class of SWFs, those that rely on the results of setwise contests alone to determine an output ranking. The many works that have been carried out regarding voting rules based on the (weighted) majority graph can be revisited in this broader setting. This line of research has already been investigated by Lu and Boutilier lu2010unavailable and Baldiga and Green baldiga2010choice. However, note that both of these works consider a setting where candidates may become unavailable after voters express their preferences. We do not make this assumption. We indeed believe that this new class of SWFs makes sense in the standard setting where the set of candidates is known and deterministic, as it amounts to generate an output ranking by examining the choices that are made by the voters on subsets of candidates of various sizes (while usually only pairwise choices are considered).

An SWF that seem natural in this class consists in determining an output ranking that minimizes the number of disagreements with the results of setwise contests for sets of cardinality at most . This is a -wise generalization of the Kemeny rule, which is obtained as a special case for . We recall that the Kemeny rule consists in producing a ranking that minimizes the number of pairwise disagreements (kemeny1959mathematics).

Example 2.

Let us come back to Example 1 and assume that we use the 3-wise Kemeny rule. Consider the output ranking . For set , the number of disagreements with the results of setwise contests is because is the top choice in for voters (see Table 1) while it is for . Similarly, the number of disagreements induced by , and are respectively , and . The total number of disagreements is thus . This is actually the minimum number of disagreements that can be achieved for these input rankings, which makes the -wise Kemeny ranking.

The purpose of this paper is to study the -wise Kemeny aggregation problem. Section 2 formally defines the problem and reports on related work. Section 3 is devoted to some axiomatic considerations of the corresponding voting rule, and to an algorithmic study of the problem. We then investigate a -wise variant of the majority graph in Section 4. We prove that determining this graph is easy for but becomes NP-hard for , and we show how to use it in a preprocessing step to speed up the computation of the output ranking. Numerical tests are presented in Section 5.

2 Preliminaries

Adopting the terminology of social choice theory, we consider an election with a set of voters and a set of candidates. Each voter has a complete and transitive preference order over candidates (also called ranking). The collection of these rankings defines a preference profile .

2.1 Notations and Definitions

Let us introduce some notations related to rankings. We denote by the set of rankings over . Given a ranking and two candidates and , we write if is in a higher position than in . Given a ranking and a candidate , denotes the rank of in . For instance, if is the preferred candidate of voter (the candidate ranked highest in ). Given a ranking and a set , we define as the restriction of to and as the top choice (i.e., preferred candidate) in according to . Similarly, given a preference profile and a set , we define as the restriction of to . Lastly, we denote by (resp. ) the subranking compounded of the least (resp. most) preferred candidates in .

We are interested in SWFs which, given a preference profile , should return a consensus ranking which yields a suitable compromise between the preferences in . One of the most well-known SWFs is the Kemeny rule, which selects a ranking with minimal Kendall tau distance to . Denoting by the Kendall tau distance between rankings and , the distance between a ranking and a profile reads as:

Stated differently, measures the distance between two rankings by the number of pairwise disagreements between them. The distance between a ranking and a preference profile is then obtained by summation.

However, the Kendall tau distance only takes into account pairwise comparisons, which may entail counterintuitive results as illustrated by Example 1. To address this issue, the Kendall tau distance can be generalized to take into consideration disagreements on sets of cardinal greater than two. Given a set and , we denote by the set of subsets of of cardinal lower than or equal to , i.e., . When is not specified, it is assumed to be , i.e., . For , the -wise Kendall tau distance between and is defined by:

In other words, measures the distance between two rankings by the number of top-choice disagreements on sets of cardinal lower than or equal to .

Note that has all the properties of a distance: non-negativity, identity of indiscernibles, symmetry and triangle inequality (see Proposition 2 in Appendix A). Secondly, as mentioned in the introduction, we have . Thirdly and maybe most importantly, we point out that the distances induced by can be computed in by using the following formula:


where is the set of candidates that are ranked below in . Let us give some intuition for this formula (see Proposition 3 in Appendix A for a proof of the complexity). For any pair of candidates such that and , we count the number of sets in on which there is a disagreement because the top choice is for while it is for . Such sets are of the form , where , otherwise and would not be the top choices. Hence the formula.

The distance induces a new SWF, the -wise Kemeny rule, which, given a profile , returns a ranking with minimal distance to , where:

Note that this coincides with the rule we used in the introduction, by commutativity of addition:

Determining a consensus ranking for this rule defines the -KAP optimization problem (for -wise Kemeny Aggregation Problem).

-WISE KEMENY AGGREGATION PROBLEM INSTANCE: A profile with voters and candidates. SOLUTION: A ranking of the candidates. MEASURE: to minimize.

2.2 Related Work

Several other variants of the Kemeny rule have been proposed in the literature, either to obtain generalizations able to deal with partial or weak orders (dwork2001rank, zwicker2018cycles), to penalize more some pairwise disagreements than others (kumar2010generalized), or to account for candidates that may become unavailable after voters express their preferences (baldiga2010choice, lu2010unavailable).

Indeed, despite its popularity, the Kemeny rule has received several criticisms. One of them is that the Kendall tau distance counts equally the disagreements on every pair of candidates. This property is undesirable in many settings. For instance, with a web search engine, a disagreement on a pair of web pages with high positions in the considered rankings should have a higher cost than a disagreement on pairs of web pages with lower ones. This drawback motivated the introduction of weighted Kendall tau distances by Kumar and Vassilvitskii kumar2010generalized. We compare our work to theirs in Appendix B and show that the -wise Kendall tau distance is also well suited to penalize more the disagreements involving alternatives at the top of the input rankings, as illustrated by the following example.

Example 3.

Consider rankings defined by , , and . We have while because and disagree on both subsets and . Put another way, because and disagree on their top-ranked alternatives whereas and disagree on the alternatives ranked in the last places.

The two works closest to ours are related to another extension of the Kemeny rule. This extension considers a setting in which, besides the fact that voters have preferences over a set , the election will in fact occur on a subset drawn according to a probability distribution (baldiga2010choice, lu2010unavailable). The optimization problem considered is then to find a consensus ranking which minimizes, in expectation, the number of voters’ disagreements with the chosen candidate in (a voter disagrees if ). The differences between the work of Baldiga and Green baldiga2010choice and the one of Lu and Boutilier lu2010unavailable is then twofold. Firstly, while Baldiga and Green mostly focused on the axiomatic properties of this aggregation procedure, the work of Lu and Boutilier has more of an algorithmic flavor. Secondly, while Baldiga and Green mostly study a setting in which the probability of is only dependent on its cardinality (i.e., is only a function of ), Lu and Boutilier study a setting that can be viewed as a special case of the former, where each candidate is absent of independently of the others with a probability (i.e., ). The Kemeny aggregation problem can be formulated in both settings, either by defining for , or by defining a probability that is “sufficiently high” w.r.t. the size of the instance (lu2010unavailable). Lu and Boutilier conjectured that the determination of a consensus ranking is NP-hard in their setting, designed an exact method based on mathematical programming, two approximation greedy algorithms and a PTAS.

Our model can be seen as a special case of the model of Baldiga and Green where the set is drawn uniformly at random within the set of subsets of of cardinal smaller than or equal to a given constant . While it cannot be casted in the specific setting studied by Lu and Boutilier, our model is closely related and may be used to obtain new insights on their work.

3 Aggregation with the -wise Kemeny Rule

In this section, we investigate the axiomatic properties of the -wise Kemeny rule, and then we turn to the algorithmic study of -KAP.

3.1 Axiomatic Properties of the -wise Kemeny Rule

Several properties of the -wise Kemeny rule have already been studied by Baldiga and Green baldiga2010choice, because their setting includes the -wise Kemeny rule as a special case. Among other things, they showed that the rule is not Condorcet consistent. That is to say, a Condorcet winner may not be ranked first in any consensus ranking even when one exists, as illustrated by Example 2.

The authors also show that the -wise Kemeny rule is neutral, i.e., all candidates are treated equally, and that for it is different from any positional method or any method that uses only the pairwise majority margins (among which is the standard Kemeny rule). We provide here some additional properties satisfied by the -wise Kemeny rule:

  • Monotonicity: up-ranking cannot harm a winner; down-ranking cannot enable a loser to win.

  • Unanimity: if all voters rank before , then is ranked before in any consensus ranking.

  • Reinforcement: let and denote the sets of consensus rankings for preference profiles and respectively. If and is the profile obtained by concatenating and , then .

While the fact that the -wise Kemeny rule satisfies reinforcement is quite obvious from its definition, the monotonicity and unanimity conditions are proved in Appendix C (see Propositions 5 and 6).

Besides, the -wise Kemeny rule does not satisfy Independence of irrelevant alternatives, i.e., the relative positions of two candidates in a consensus ranking can depend on the presence of other candidates. Let us illustrate this point with the following example.

Example 4.

Considering the preference profile from Example 1, the only consensus ranking for is . Yet, without the only consensus ranking would be .

Lastly, note that there exists a noise model such that the -wise Kemeny rule can be interpreted as a maximum likelihood estimator (conitzer2009preference). In this view of voting, one assumes that there exists a “correct” ranking , and each vote corresponds to a noisy perception of this correct ranking. Consider the conditional probability measure on defined by . It is easy to convince oneself that the -wise Kemeny rule returns a ranking that maximizes and is thus a maximum likelihood estimate of .

3.2 Computational Complexity of -Kap

We now turn to the algorithmic study of -KAP. After providing a hardness result, we will design an efficient Fixed Parameter Tractable (FPT) algorithm for parameter .

While -KAP is obviously NP-hard for as it then corresponds to determining a consensus ranking w.r.t. the Kemeny rule, we strengthen this result by showing that it is also NP-hard for any constant value . The proof, which is deferred to Appendix C, uses a reduction from -KAP.

Theorem 1.

For any constant , -KAP is NP-hard, even if the number of voters equals 4 or if the average range of candidates equals 2 (where the range of a candidate is defined by and the average is taken over all candidates).

Despite this result, -KAP is obviously FPT w.r.t. the number of candidates, by simply trying the rankings in . We now design a dynamic programming procedure which significantly improves this time complexity.

Proposition 1.

If is an optimal ranking for -KAP, then , where, for any subset , is defined by the recursive relation:


Given and , let us define as . The set can be partitioned into and . Given a preference profile over and a ranking , the summation defining breaks down as follows:


Using the same reasoning as in Equation 1 on page 1, the second summand in Equation 3 can be rewritten as follows:

because for all . Note that and , thus . Hence is equal to:


Consider now a ranking such that . We have:

because the second summand in Equation 4 does not depend on (it only depends on , which is the argument of the min). If one denotes by , one obtains Equation 2. This concludes the proof. ∎

A candidate that realizes the minimum in Equation 2 can be ranked in first position in an optimal ranking for . Once is computed for each , a ranking achieving the optimal value can thus be determined recursively starting from . The complexity of the induced dynamic programming method is because there are subsets to consider and each value is computed in by Equation 2. The min operation is indeed performed on values and the sum can be computed incrementally in , which entails an complexity for the second summand in Equation 2 (the factor is of course due to the sum over all ). Note that the computation of all binomial coefficients for and can be performed in in a preliminary step thanks to Pascal’s formula.

4 The -Wise Majority Digraph

We now propose and investigate a -wise counterpart of the pairwise majority digraph, that will be used in a preprocessing procedure for -KAP.

As stated in the introduction, the pairwise Kemeny rule is strongly related to the pairwise majority digraph. We denote by the pairwise majority digraph associated to profile . We recall that in this digraph, there is one vertex per candidate, and there is an arc from candidate to candidate if a strict majority of voters prefers to . In the weighted pairwise majority digraph, each arc is weighted by .

Example 5.

Consider a profile with 10 voters and 6 candidates such that:
– 4 voters have preferences ;
– 4 voters have preferences ;
– 1 voter has preferences ;
– 1 voter has preferences .
The pairwise majority digraph is represented on the left of Figure 1.

\Vertex\Vertex\Vertex\Vertex\Vertex\Vertex\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge \Vertex\Vertex\Vertex\Vertex\Vertex\Vertex\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge\Edge
Figure 1: -wise majority digraph in Example 5 for (left) and (right).

From , we can define a set of consistent rankings:

Definition 1.

Consider a digraph whose vertices correspond to the candidates in . Let denote the subsets of corresponding to the Strongly Connected Components (SCCs) of , and denote the set of linear orders on such that if there exists an arc from to then . Given , we say that a ranking is consistent with if the candidates in are ranked before the ones of when .

The following result states that, for any , there exists a consensus ranking for among the rankings consistent with .

Theorem 2 (Theorem 16 in reference charon2010updated, charon2010updated).

Let be a profile over and assume that the SCCs of are numbered according to a linear order . Consider the ranking , consistent with , obtained by the concatenation of rankings where . We have:

That is, is a consensus ranking according to the Kemeny rule. Furthermore, if and for all and when , then all consensus rankings are consistent with .

This result does not hold anymore if one uses (with ) instead of , as shown by the following example.

Example 6.

Let us denote by the profile of Example 1. The pairwise majority digraph has three SCCs , and . In this example, where . The only ranking consistent with is while the only consensus ranking w.r.t. the 3-wise Kemeny rule is .

In order to adapt Theorem 2 to the -wise Kemeny rule, we now introduce the concept of -wise majority digraph. Let . If is not specified, it is assumed to be . Given a ranking , we denote by the set . Given a profile , we denote by the value and by the difference . This definition implies that . The value is the net agreement loss that would be incurred by swapping and in a feasible solution of -KAP where and . If (resp. ) then, in a consensus ranking for where and would be consecutive, it is possible (resp. necessary) that .

The -wise majority digraph associated to a profile over a set of candidates is the weighted digraph , where and iff:

The weight of this arc is then given by:

Note that, if , we may obtain edges and both with strictly positive weights (which is impossible in the pairwise majority digraph). For instance, for the profile of Example 5, and . The digraph is shown on the right of Figure 1. Besides, for any , is the pairwise majority digraph as . Theorem 2 adapts as follows for an arbitrary (the proof is deferred to Appendix D):

Theorem 3.

Let be a profile over and assume that the SCCs of are numbered according to a linear order . Among the rankings consistent with , there exists a consensus ranking w.r.t. the -wise Kemeny rule. Besides, if and 222Or, equivalently, . for all and when , then all consensus rankings are consistent with .

Example 7.

The meta-graph of SCCs of in Example 5 is represented in Figure 2. The above result implies that there exists a consensus ranking among , , and .

Figure 2: The meta-graph of strongly connected components of in Example 5.

To take advantage of Theorem 3, one could try 1) to index the SCCs of according to a linear order , and then 2) to work on each SCC separately, before concatenating the obtained rankings. However, for a consensus ranking consistent with , the relative positions of candidates in depend on the set of candidates in (but not on their order). The influence of can be captured in the dynamic programming procedure by applying a modified version of Equation 2 separately for each subset downto . Formally, if is optimal for -KAP, then:

where, for any subset , is defined by and ( stands for ):

It amounts to replacing by in the second summand of Equation 2 to take into account the fact that there exists a consensus ranking where all the candidates of are ranked after those of . Let be a ranking of such that , where is the ranking obtained by the concatenation of rankings in this order. The ranking of is a consensus ranking w.r.t. the -wise Kemeny rule. Given Theorem 3, the -wise majority digraph thus seems like a promising tool to boost the computation of a consensus ranking. Unfortunately, the following negative result holds, whose proof is deferred to Appendix D.

Theorem 4.

Given a profile and two candidates and , determining if is NP-hard for .

Consequently, computing the digraph from is NP-hard for . In contrast, the digraph can be computed in polynomial time. Indeed, given a set such that , adding an element to increases by one for each such that and . Let us denote by the set . A set maximizing is