A recurring question in sport is the extent to which skill or luck determine the outcome. Back in 2012, Michael Maubisson the Director of Research at BlueMountain Capital Management, published a book exploring the relative contributions of skill and luck in the domains of sport, business and investing.
As shown in the diagram below, Maubisson found that Premier League football sits somewhere the middle of the team sports continuum. Luck plays a big role in the NFL, but a much smaller role in the NBA. Premier League football sits in-between.
Before continuing, it is worth clarifying what Maubisson’s continuum doesn’t mean. It doesn’t mean that NBA teams are more skilled than Premier League teams; it means that skill plays a bigger part in NBA results. This arises partly because basketball is a high scoring game. Results in football on the other hand are often determined by the outcome of a small sample of shots. Another factor is the respective lengths of the NBA and football seasons. In the regular NBA season each team plays 82 games, in the Premier League they play 38. Luck plays a bigger part when the season is short.
Using Classical Test Theory to Apportion Luck and Skill
Maubisson based his calculations on the equations of Classical Test Theory, (sometimes called True Score Theory) which was originally developed in psychometrics. The key insight is that an observed score on a test – an IQ test or a personality inventory – is the sum of two independent components; the individual’s true standing on the test, or his true score, and a random error term. The aim of the test developer is to minimize the random error, so that the observed score is as far as possible an accurate reflection of the true score. Random error cannot be eliminated entirely, but we are able to estimate it. Knowing how much random error a score contains allows us to understand the limitations of the inferences we can make.
To apply the theory to sporting contests we first compute the variance of an outcome (such as end-of-season standings) assuming the results are purely based on chance. For example we could assume that all the teams in a league are exactly the same strength, and the result (win or lose) is decided by a random coin toss. On average of course all teams would finish with the same number of wins at the end of the season. But what we are interested in is the between-teams variance in the number of wins that the randomness produces. This is the variance due to “luck”. Typically however, the variance we actually observe is greater than this, the difference being due to differences in team skill. The skill differentials cause some teams to accrue more wins than expected by chance and some teams to score fewer, increasing the variance in the observed number of wins between teams. By classical test theory we know we can simply subtract the variance due to luck from the overall observed variance to find the variance due to skill.
Luck and Skill in The Top Divisions
In this post I use this general approach to estimate the contributions of skill and luck in the Premier League and the Championship over the past 25 years. However, instead of counting wins, I used a slightly different method for determining my variances. I was worried about using the number of wins because I wasn’t sure what effect draws would have on the results, so I based my calculations on goals scored. I assumed all teams had the same home scoring rate and the same away scoring rate, and worked out the variance in the number of points based on 1000 simulations of each season. I compared this with the observed variance in points. (Later I found out this was probably overkill, and using the number of wins gives almost the same result.)
I was interested in two things: Has the balance between skill and luck changed over the years? And is skill more important in the Premier League than in the Championship? The charts below show the answer.
The right-hand chart shows the results for the Premier League, and suggests the skill component has increased since the foundation of the league, although the 2010 season looks like an outlier, with the skill component dropping back to 1990 levels. Over the last few years, the skill component has been steady at around 78%. In contrast, skill plays a noticeably smaller role in the Championship, the average contribution being 58%. The season-on-season results also seem rather more variable than in the Premier League, occasionally dropping below 30%.
Of course, these findings do not mean Championship teams are less skilled than Premier League teams (although they are). They indicate that team skills in the Championship contribute less to seasonal standings than they do in the Premier League. This cannot be attributed to the different numbers of games played in the two divisions; in fact the longer Championship season tends to increase the contribution of skill and reduce the contribution of luck. Perhaps what we are seeing here is the effect of the giant clubs in the Premiership outspending the rest by a considerable margin, and increasing the variance of skill within the league. Conversely, the Championship has a reputation for being a notoriously hard division to get promoted from. Perhaps that is because skill variance is compressed, and success requires a good helping of luck. And of course the play-offs don’t help.
The End of Season Play-offs
The data suggest that the end-of-season playoffs in English football do reflect performance in the preceding regular season. Numbers from Simon Gleave show that in 88 playoffs across all divisions, the 3rd placed team won promotion on 34 occasions. This is significantly more often than the 22 occasions that would be expected by chance. However, each playoff competition consists of only a few matches, so we might expect that luck plays a very large part. To estimate how much luck contributes, we score a promotion as 1 and a non-promotion as 0. We then compute the observed variance in promotions for the 3rd team, and compare this with the variance expected by chance. (As there are four teams in the playoffs, the probability of being promoted by chance alone is 25%). It turns out that the results are determined 79% by luck and only 21% by skill. The results for promotion from the Championship/Second Division are very much the same; only around 19% of the variance in promotion outcomes are due to skill.
So although the best rated team in the playoffs is the most likely one to secure promotion, each playoff consists of so few matches that essentially the results are a lottery.
The Luck -Skill Continuum For Top English Football Competitions
Finally, we can plot the results on Maubisson’s continuum. To keep things reasonably up-to-date, I use the last ten seasons of the Premiership and the Championship; but because of the limited amount of data, I include all promotion playoffs since 1989.