Removing skill-based matchmaking leads to a drop in players, Activision study suggests

Players are less likely to keep playing an online multiplayer game if there’s no skill-based matchmaking, an Activision study suggests.

According to a research paper published online by the company (and as spotted by GamesIndustry.biz), Activision held an experiment in Call of Duty: Modern Warfare 3 in early 2024 without informing players.

Usually the game’s matchmaking takes a player’s previous overall performance – such as kills and deaths – and uses it to match them with players with similar stats, in an attempt to make a more competitive match that isn’t too easy or hard.

For its experiment, Activision took the North American player base for Modern Warfare 3. For half the players it ran a ‘Deprioritise Skill Test’, which “loosened the constraints” on skill-based matchmaking, meaning less emphasis was placed on a player’s skills when looking for opponents. The other half got the normal skill-based matchmaking algorithm.

According to Activision’s findings, most of the players in the Deprioritise Skill Test were less likely to come back to the game. While the top 10% highest skilled players returned to the game in increased numbers within two weeks (because they were winning more), when it came to the other 90%, fewer returned to the game than normal.

Activision says this should be a concern for the top 10% too, because as players continue to drop out of the game and the player base gets smaller a top 10% player could become a top 20% or 30% player.

“Ultimately, this will result in a worse experience for all players, as there will be fewer and fewer players available to play with,” Activision claims.

It also points out that the experiment only deprioritised skill in matchmaking, rather than completely removing it. “If it were completely removed, we would expect to see the player population erode rapidly in the span of a few months, resulting in a negative outcome for all our players,” it says.

Activision says it also ran other experiments in which the skill based matchmaking was tightened even more than it currently is, and this had the opposite effect, with the best players more likely to drop out and the bottom 80% more likely to return.

“This change was not rolled out as a standard approach, as we continue to strive for a balance in our approach to matchmaking,” Activision says.