An Elo-based rating system for TopCoder SRM

Let $RP(n,r)$ be the performance of a player ranked $r$ in a round of $n$ players. We consider the ranking as an elimination tournament, and count the number of wins.

If the tournament has $2^n$ players, the winner must win $n$ 1-1 rounds, so we have:

The player ranked $2$ has one less win than rank $1$:

Multiplying both $r$ and $n$ by any $k>0$ does not change the number of wins:

With these constraints, we find the only solution is:

We use the standard Elo formula for expectations [4]. A player with rating $R_i$ is expected to outperform a player with rating $R_j$ with probability:

As with SRM ratings, the rating of new players is initially 1200.

In programming contests like SRM, ties generally reflect a limitation of the problem set or scoring, rather than an unexpected performance of the tied players. We would like ties to not affect the ratings.

We experimented with several accounting methods. We find the most accurate is to split the ties equally in actual and expected ranks, $.5$ in each. Slightly less accurate is to compute expected ranks regardless of ties, then split the tied ranks as is expected.

We now split the ties equally.

We have the results of a round, which may include multiple divisions and ties. We consider the results of each division separately. The result of a round is a list of scores $S$, where $s_i$ is the score obtained by player $i$.

We compute rank and expected rank for each player $i$:

The relative performance of a player in the round is the difference of actual and expected rank performance:

This can be written as:

The performance $P$ equals a number of wins above or below expectations in a tournament of appropriately matched players.

Rank performance is convex (Figure 1). The sum of performances of a set of ranks is maximal if the ranks are distinct, and minimal if the ranks are uncertain or tied. Having split ties equally in actual and expected ranks, the expected ranks are at least as tied as the actual ranks. This ensures the sum of performances in a round is positive or zero. We have:

The rank is expected in logarithm (Figure 2). A player’s performance may average $0$ in several ways:

We will compute $\Delta R\propto P$, preserving these properties.

We define the prediction error for expected and actual ranks $(\hat{r},r)$:

Our primary accuracy metric is the average error for all participants in all rated rounds.

With a prior of $1:1$, a performance $P$ is outperforming expectations by a factor $1:2^P$. A rating difference $\Delta R$ is expecting a better performance by a factor $1:{10}^{\Delta R/400}$. We can convert the performances in rating units:

If we expected the same performances the next round, and had no other information, this would be a reasonable $\Delta R$. Thus we consider $\Delta R\propto P$ as Elo-based or ’Elo’.

Here we compute $\Delta R=K\cdot P$, with K minimizing the error. We find the most accurate choice is $K=65$ (Figure 3). As we add factors in $\Delta R$, we automatically adjust $K$ to the most accurate.

A player’s prior rating should weigh in $\Delta R$ according to the player’s experience. Let $W$ be the weight, such that $\Delta R\propto 1/W$, and $NR$ the round number for the player, starting with $1$. We experimented with several possibilities, and find the best results with $W=\sqrt{NR}$. Figure 4 shows choices of $W={NR}^{\alpha }$. Thus we choose $W=\sqrt{NR}$.

$K=182$, the error is .7562.

A player’s performance has variance for various reasons, not necessarily predicting future performance. We compute the derivative of a player’s expected performance per change in rating:

We write the expected rank 1 as the loss:win ratio relative to the player’s current rating:

In units of performance, we have:

Extrapolating linearly, we can solve $P=0$ with $\Delta R=K_0\cdot \frac{P}{P^{\prime }}$.

• $P^{\prime }\le 1$, so $\Delta R=K_0\cdot P$ may be conservative.

• $P^{\prime }\to 1$ when $\hat{r}\to 1$. No extrapolation is possible.

• $P^{\prime }\to \frac{1}{n}$ when $\hat{r}\to n$. Here $P\ne 0$ has bits of precision, hence more likely predicts future performance.

We compute $\Delta R$ in a direction accounting for $P^{\prime }$ and a multiplier $C$:

We find the most accurate $C\approx 4$ (Figure 5). Thus we choose $C=4$.

We define the variance factor $V=1+C\cdot P^{\prime }$. We now have:

$K=415$, the error is .7536.

An unexpected performance predicts future performance less reliably than a consistent performance. The ratings gain accuracy if we limit the magnitude of $P$ to a maximum $M$ using a sigmoid. A sigmoid preserves symmetry, and exactly linearity of performance around $0$. We define the adjusted performance $PA$. We find the best results with:

We find the most accurate $M\approx 6.75$ (Figure 6). Thus we choose $M=6.75$. The rating change for each player is now:

$K=600$, the error is .7513.

We have computed $\Delta R\propto P$ which make the ratings more accurate after a round. Each player is expected a performance $0$, thus has an expected $\Delta R=0$. The ratings are stable in expectation.

Because performances have a positive sum, more rating is won by the outperforming players than is lost by the underperforming players. The ratings have net inflation. Because the players gain experience during a round, the players on average have better performances after the round. Thus inflation better predicts future performance than deflation. Because we minimized the prediction error, the average $\Delta R$ should approximately predict the next performances of participants relative to non-participants. We define this rate of inflation as natural inflation.

For comparison with SRM ratings, we consider natural inflation as stable. We refer to this Elo-based implementation as ’Elo’ in our results.

We have $\Delta R\propto P$, predicting the players’ relative performances. Now we would like to estimate the performances over time, such that players with stable performances have stable ratings. To maintain relative accuracy, we will not adjust our current parameters.

As long as $\Delta R\propto P$ exactly, the expected rating change of any player is zero. However, the expected performance in a round is a better performance than not participating. Players having practiced already have better performances before the round. Thus $\Delta R=0$ in expectation predicts future performance less accurately than $\Delta R>0$.

We adjust the expectation to expect inflation, as if the ratings increased. We choose a parameter $B=\Delta R$, then add the difference in expected performance to each player’s performance:

We find $B\approx 6$ (Figure 7), and little accuracy can be gained from this parameter alone.

So far we have a constant rating $R_0=1200$ for new players. However, the performance of new players is not constant. As the performance of existing players improves, SRMs become more difficult. This raises the barrier to entry.

Before participating, potential players have opportunities to practice on recent rounds. Some may be experienced players coming from other platforms. Thus the performance of new players improves over time. Thus we adjust the initial rating for inflation.

• We choose a parameter $N$, the increase in $R_0$ per 100 rounds.

• After each round, adjust $R_0$ by $\frac{N}{100}$.

• Simultaneously, adjust $B$ for accuracy.

We find most accurately $N=63,B=27$ (Figure 8).

Thus, our parameters estimating a stable performance are:

’Elo2’: $N=63;B=27$