Under the Hood of Call of Duty’s Matchmaking System and MMR With Charlie Olson, Part 2

What I've told a lot of our customers is that low-skill players actually are the most fragile cohort. They actually want the best match quality, so don't give them matches quickly. Give them the highest quality match that you can within a reasonable time period.—Charlie Olson

We explored Call of Duty’s MMR (matchmaking rating) and SBMM (skill-based matchmaking) systems with Charlie Olson, the CEO and Co-founder of Invokation Games, last week. Olson spent over six years working on these systems for Activision in the 2010s and continues to do so at Invokation for other studios/publishers.

We’ll wrap up this week with Olson’s perspectives on how:

1. Elo, a skill-based rating system that emerged in a chess context in the 1960s, stacks up in gaming today
2. CODs matchmaking system could function a little bit better
3. CODs Ranked Play system works at the high-end of the player skill ladder.

These insights were based on a podcast that Player Driven’s founder, Greg Posner, and I had with Olson in June, and a pair of white papers that Activision published back in 2024 (links are below).

Before diving into these tasty dishes, let’s set the table by touching on the services that Invokation offers today.

“We started off doing just skill ratings,” Olson explained to us in June, “just the MMR system, and quite a few customers licensing that.”

“We're like a premium alternative to Microsoft TrueSkill or Elo, the most common MMR system that people have heard of,” he continued. “Elo doesn't actually work. If you use it for skill-based matchmaking it breaks down.”

Hold that shade-covered beer…

“But we have a very robust, complex, skill rating system,” Olson told us. “We started with that for a couple of years and now we've moved into full matchmaking.”

He hinted that Invokation’s services are now powering the full matchmaking stacks of a few recently launched titles.

“The matchmaking, like, I always thought that was a solved problem,” Olson added. “As we've been working with customers, I came to realize this really is not solved. There's [GameLift] FlexMatch from Amazon out there, and there's, like, CODs matchmaker,” plus the aforementioned TrueSkill and Elo options.

“They're doing a lot of things that I thought we could do better,” he said. “We've spent the last year and a half kind of building that out, and we'll see” what happens in terms of its reception.

“Basically, everything related to skill-based matchmaking is what Invokation does,” summarized Olson. From a customer angle, “very generally speaking, I think they should probably consider matchmaking as part of the UX.”

UX is shorthand for user experience, the overall feeling and sentiment that a game studio hopes to engender in players.

Early in last week’s part one Design Desk breakdown with Olson, there’s an exchange in which I tried to get Olson to connect CODs matchmaking systems to the game’s UX, as well the core multiplayer game loop. Olson effectively shrugged off the question.

Now he came back to this topic.

“Like Lewis was saying right at the beginning,” matchmaking can be “part of the overall [player] arc, and it's related to progression and economy. It doesn't have to be just a technical problem. This is actually a design problem,” he said.

I felt “seen”!

“Think about the kind of matchmaking experience you want for players. That ties into things like, ‘Don't release too many game modes.’ Because you don't wanna fragment the player base so that they can't find matches in the modes they wanna play. If you take away modes, they won't like that, but you can always add more modes if you have a player base that can support it.”

“I would say start small,” Olson added. “Always start with, like, one, three max game modes, just to make sure that you can matchmake that” group of players successfully.

“Optimizing match quality over time to match” has a relationship to the UX. The topic “is a little bit tricky because…the assumption is that low-skill players wanna get into matches quickly.”

In Olson experience, this isn’t true: Most low-skill players prioritize match quality over time to match (the speed of getting back into a match). This assessment resonated with a topic we explored last week about if/how CODs MMR and SBMM systems function in the bottom 5-10% of players by skill compared to the top 5-10%. Here, Olson was saying, the algorithm should be weighted differently—even if they’re not in CODs case—because the UX ramifications for new and low-skill players is significant.

Posner switched topics and asked Olson what he thought the most overrated competitive multiplayer games in the past five years were.

“I'm gonna say Halo and Destiny,” Olson answered. “I'll tell you why, too. This is a personal preference. For me, it's the aiming.”

“I hear all these things about, like, ‘All the weapons are so great in Destiny and Halo!’ And the on gamepad, the aim assist is so heavy handed. It drives me nuts.”

“I don't love the gunplay in any of the Bungie era games, or the Bungie lineage.”

Spoken like an aiming connoisseur (Olson labored on CODs dynamic aim response curve on controller)!

Posner may have been surprised Olson didn’t put NetEase Games’ Marvel Rivals on his list.

“I was reading yesterday about Marvel Rivals,” said Posner. “There have been so many complaints about the matchmaking in the last two seasons that…[it could] truly impact long-term engagement...”

The game has certainly received its fair share of matchmaking criticism in the past year. This Reddit thread, which got nearly 1,000 comments, goes down this rabbit hole, for example.

Olson formulated a diplomatic answer.

“I haven't played it, like, so this is me just talking out of uninformed nowhere,” he said to Posner. “As far as I know, their retention is pretty good.”

“The team behind their matchmaker published a talk on engagement-optimized matchmaking.”

The aforementioned Reddit thread gets into this alternative “EOMM” approach, and suffice to say it’s not exactly a love letter.

“They're trying to build a model of player engagement, and then give you matches that are most likely to increase your engagement,” Olson explained. “Not necessarily the most fair match or something, like, you would design intuitively, but something that leads towards you coming back and spending money.”

“They have since denied that's what's going on…I do know they use TrueSkill under the hood.”

“They do have a problem of MMR not adapting quickly” with the approach they’re using, he continued. “Players will get pigeonholed to a specific skill level. And then they'll just keep getting matched over and over as though they're that skill, even though they might be having a good day or a bad day.”

“That could be one of the things that's going on,” he added. “The other thing is, it's just hard when you have a role-based game, where it matters what character you choose and who your teammates choose. That really makes matchmaking complicated. It could be they just haven't figured out the right approach to matchmaking there.”

(Design Desk doesn’t believe he was directly involved in Marvel Rivals’ MMR or SBMM/EOMM systems, but we profiled the game’s director, Thad Sasser, recently; check it out here.)

Let’s eat! Don’t forget to subscribe to our newsletter and share this article w/a shooter afficionado in your sights.

Elo, and Why MMR > Skill Percentile in COD Matchmaking

If you do Elo, everybody's gonna start in the middle, at the fiftieth percentile. Half of your players are gonna lose rank over time, half are gonna gain. And the ones losing rank are obviously not gonna be thrilled about that. They're just converging on a lower and lower number.—Charlie Olson

“If every multiplayer game had to show one piece of currently hidden data tomorrow, what would you pick?” Greg Posner asked Charlie Olson in our June discussion.

“MMR for sure,” said Olson, without missing a beat. “Casual MMR. I think every game should have that accessible somewhere, whether it's in game through a menu or an API you can connect to easily. There's no reason that should be hidden.”

I pointed out that Activision hides MMR in COD and that a justification was exposing MMR opens the door to match manipulation.

“You can manipulate the matchmaking even if you don't know your MMR,” Olson retorted. “I think they're mainly doing it because players would think it's…not reflective [of their skill], and might be upset that they're not as good as they thought they were.”

“I think there's other psychological reasons,” he continued. “In early systems they might have been hiding it because they thought people would, I mean, people do two-box, and they thought they would manipulate that somehow.”

Two-boxing is a trick players use to pull down the average skill of a match to increase their chance of performing well. What happens is that a high-skill player will squad up with a second, low-skill account (on a second device) to get into a lower-skilled match, then disconnect the low-skill account before the firefight begins. This YouTube video has some good color on the two-boxing issue in Warzone.

“You can tell when you're in a low-skill lobby,” Olson added. “Whether we show you the MMR or not, you know if you're Smurfing.” (Note: We touched on Smurfing last week.)

“I think just show it,” Olson concluded. “Not everybody wants to know [their MMR]…but it should be available somewhere for players who do wanna know.”

Given that Olson had brought up Elo, I turned back to it. There’s more background on Elo in this Wikipedia post, but some key points are that Elo relies on something called the Bradley–Terry model to assess skill, and the method was designed to rank world-class chess players in the 1960s. The system has since been adapted for 1v1 video game use, such as MUT mode in EA Sports’ Madden NFL.

I asked Olson to critique Elo.

“There's a couple issues with Elo,” he said. “It's named after Arpad Elo, who, actually, I think, lived in Milwaukee, Wisconsin, back in the ‘60s, ‘70s.”

Elo, who was born in Hungary, was a chess master and a physics professor who indeed lived in Milwaukee for decades (and taught at Marquette University).

“It's pretty easy to calculate, update,” continued Olson. “You just have two ratings between two players, and then you can calculate, if one player wins, how much Elo should they gain and how much should the other player lose. In most scenarios, it's zero sum.”

“If you use it for matchmaking, the problem is it amplifies the flaws in the model,” he told Posner and me. Say that “you have two players who are 1,000 rated, and one wins and the other loses. What happens is, instead of making a nice stable distribution, the distribution just kind of expands over time.”

FIDE, the “International Chess Federation had to reset everybody's ratings according to some formula to shrink them back into an acceptable range.”

“Inherently, there's just no global frame of reference” for Elo. “There's no population model in chess…It's just unstable over the long run.”

Elo “has a lot of nice, predictable properties. When you gain, like, plus or minus 40 for a win, like, for an evenly made match. That's intuitive to players. They like that.”

A lot of people think it's only made for 1v1, but as long as you can calculate an expected outcome, and compare it to the actual outcome, you can do an Elo update with any number of players.

Capcom’s “Streetfighter [6] has a pure Elo system…Once you get the master rank, they switch you to Elo. The catch is you have to reset the ratings every season.”

“If you don't reset the ratings, they are gonna inflate, expand, over time. You gotta bring everybody back into the middle…If you do that, you could use an Elo system” for most competitive video games.

Olson then damned Elo with faint praise: “It's not the worst.”

He plugged his home-grown alternative to boot: Invokation Games’ “MMR system does allow you to start players anywhere in the distribution and know that they're gonna converge on their actual ratings. You're gonna plateau where you belong…[around a] fifty-fifty win rate.”

The Psychology Behind CODs (and Invokation Games’) SBMM Systems

I circled back the topic of how CODs MMR and SBMM systems can, and perhaps should, work differently in the bottom 5-10% of players by skill compared the top 5-10% of players. More precisely, I asked Olson if he had another “certain optimization” that he could share in this direction.

His answer surprised me. First, he noted the “moment matching” approach that he and Invokation prefer in terms of building relatively symmetrical teams (we covered this last week), but Olson then pivoted to a different issue that impacts player toward the middle of the skill distribution.

In one of its white papers, he began, Activision showed a graph that was labeled something along the lines of “normalized scoreboard position” that was “based on net kills” per match. His recollection of that graph was that “bottom decile players generally end up towards the bottom half of the match, and then, the top decile, they’re almost always in the top half of the match.”

I doublechecked the “Call of Duty Matchmaking Intel – White Paper #2” for this graph, but couldn’t find it. Perhaps it’s in one of the other nontechnical COD-related papers/articles that Activision Publishing has posted in recent years.

“What I would actually like to see is a more linear increase,” Olson said.

“They tend to have a, like, inverted S-curve, like, inverted sigmoid. There's this plateau in the middle where the twentieth percentile through fiftieth or sixtieth percentile don't have an appreciable change in their outcomes, despite having higher skill. This is gonna be a bad player experience.”

“This is something that people complain about,” he added, “and rightfully complain about, that every match feels the same. If you're in that range where, ‘Hey, you get matched with somebody of your skills. What's the point of learning the maps better or tweaking your loadouts to have a better performance?’ You want an appreciable change in your average outcome if you actually improve.”

“You don't want that plateau in the middle. You want that to be linear,” and reward the players who are gaining in skill.

It’s definitely not the graph that Olson was referencing, but there’s a figure in the white paper we’d been discussing that contains a similar curve (an S-curve, but the point about the plateau in the middle still stands). The figure below shows the score per minute (SPM, which combines players’ kills and match objectives completed) across the two samples in the deprioritized skill test that Activision ran on Call of Duty: Modern Warfare III players 1Q 2024, which we covered in some depth in last week’s Design Desk entry.

Olson’s point, as I understood it, is that if a plateau like the one shown above exists in middle deciles of a KPI distribution, it’s likely less than an ideal UX. In this scenario, players who are grinding, and learning the nuances of your game, and moving up through the 5th, 6th, 7th and 8th skill deciles aren’t necessarily landing in the upper half of a normalized match win rate distribution. The implication is that matchmaking algorithms should include a factor that recognizes players that are on a relatively positive MMR trajectory, and then rewards them with a slightly better chance of winding up on the winning side of matches. That’s going to feel like progress.

“I think a big reason that plateau happens, to answer your initial question, is because they [COD/Demonware] use percentile-based matchmaking,” Olson continued. “Part of it is matchmaking might be a little bit too loose, but I think a bigger part is because they match players based on percentile, not based on MMR, not based on the skill number itself.”

“When you're in this [middling] range, where most players are, you get matched too tightly in a way. What they actually need is deliberately looser matchmaking for the middle tier players so that they can feel this improvement over time.”

“That's like a psychological impact,” I said. The algorithm tweak he advocated, I believe, is about throwing “real grinders” a bone, and subtle UX bonus for putting in the work (and hopefully not cheating!).

It crossed my mind that if the anti-SBMM crowd—or the anti-EBMM crowd, for that matter—get their way, then these kinds of UX “babies” will all get thrown out with the “bathwater.”

My thoughts turned to PUBG, which Olson said didn’t use SBMM (see last week’s post).

It turns out that’s not quite true. In 2024, the PUBG: BATTLEGROUNDS team laid out a plan to boosting the role of skill in matchmaking. Presumably, that has since been rolled out. PUBG players have an MMR today, and PUBG has an SBMM system.

As far as I can tell, that makes it a clean sweep: There is no major competitive video game out there that matches players randomly as they come into the matchmaking queue. PUBG’s skill parameters may still be quite loose—they acknowledged in their roadmap post that “With up to 100 players per match, ensuring an adequate pool of players for matchmaking is paramount,” which implies the game’s MMR “backoffs” happen much faster than a game that has, say, a dozen players or less per match—but skill is nevertheless a key PUBG matchmaking input.

My own assessment of this is that MMR and SBMM result in a more psychologically satisfying gameplay experience. The market has spoken: It’s simply a better UX than doing nothing.

I asked Olson if he agreed that if skill were full deprioritized in a game like COD that “the player base would quickly fall to zero.”

“I don't think it would fall to zero,” he said.

“You would weed out all of the casual players, and you'd end up with…more hardcore” players. “Retention looks great, but it's because you've already filtered out all of the casual players who would stick around longer” if they weren’t getting face-stomped into the mud over and over.

Getting rid of SBMM is “an easy suggestion to make if it benefits you.”

If you're in the top 10% of players by skill, yeah, “Get rid of skill-based matchmaking.” And you think, of course, “This is great for me. It must be great for everybody.” But, in fact, it's not good for the long-term health of the game.

Before turning to Ranked Play, I circled back to the casual/unranked white paper (#2) on the role of skill in COD multiplayer and asked Olson one last question. The paper, I pointed out, included a description of a “hypothetical max skill matching title.” In that environment all player deciles has a K/D ratio of .500. MW3 didn’t get there—presumably it’s the “Current skill matching title” referenced in the figure below—and neither did another (earlier, unspecified) “Low skill matching title,” but the implication was that Activision could tweak its MMR/SBMM systems in a way that resulted in COD modes that got close to .500 K/D ratios in all skill deciles.

My question to Olson was simple: Why doesn’t Activision go ahead and do this? If the deprioritized skill test proved that total engagement drops as the weight of skill is lowered, why doesn’t Activision go to the other extreme in its algorithm and start reaping the engagement gains that should emerge, presumably, from this “max skill matching” state?

Olson pondered the question for a beat.

“Coming back to the previous question, I don't think that's a good player experience,” he said at last. “It would be fine in Ranked Play where you have visible feedback about your skill rating, and you expect to have a perfectly even match.”

“But if you're expecting to feel your improvement over time, that completely eliminates it. Just from a design perspective, I don't think that's what you want in casual. You do want, like, a softer linear increase over time” to make casual-leaning players, especially if they’re advancing up the skill deciles through many seasons, feel that positive shift, to get that UX bump.

“The other part,” he added, “is I don't think they can do it. Technically, within their acceptable time to match, the acceptable wait time, and the way the matchmaker works, they just can’t get matches tighter than what they currently are, or they would.”

I imagined a spaceship asymptotically approaching the speed of light, but never getting there.

“It ultimately comes back to, possibly, the percentile-based matchmaking. It’s kinda distorting that middle-player experience. If I could change one thing, I would switch to MMR-based matchmaking instead of skill percentile.”

The ball’s in your court, Demonware!

A Breakdown of COD Ranked Play…Plus More Anti-cheat in IRL Markets?

If you're using wallhacks, like, you have an unfair competitive advantage. It's the anti-cheat's job to prevent people from using wallhacks. Same thing in business, I think.—Charlie Olson

Our discussion with Invokation Game’s Charlie Olson concluded with a few questions about COD Ranked Play (RP). How player skill intersected RP was the focus of a different white paper Activision Research published in late 2024: Call of Duty Matchmaking Intel – White Paper #3.

A bit of up-front context on RP is probably helpful:

1. Ranked Play features 4v4 matches. Gone are the wide variety of modes associated with casual/unranked COD multiplayer and Warzone
2. RP has seven ranks, from Bronze to Iridescent, and all RP players have an explicit per a Skill Rating (SR) that ranges from zero for noobs to >10,000 for world-class Iridescent players
3. RP players aren’t necessarily COD’s most-skilled players but most popular COD streamers and influencers are in RP, the rules associated with RP are the same as those used in COD esports (CDL), and the revolving door between RP players and pro COD players keeps spinning.

I asked Olson a two-part question off the bat: How does SR in RP relate to a casual COD player’s MMR, and was I correct that that RP players are a high-skill subset of COD’s broader player base?

“Yeah,” said Olson. “It is generally the higher skill players” that are in RP.

“Outside of Ranked Play, like, Search and Destroy [another COD mode], in a Western audience, tends to attract competitive, high-skill players. In China, Search and Destroy is, like, the default mode that everybody's familiar with. It's like [Valve’s] Counter-Strike. In the US, it's flipped.”

Team Deathmatch, another popular COD mode (which we covered last week), “is the casual mode, and then Search and Destroy tends to have a…subset of the…high-end tail” by player skill.

Got it.

“There is one thing I would point out, though, is that it's not visible MMR. Your SR is actually based on a hidden MMR system.”

Activision has “something similar to the casual rating system, and I think they did a white paper on this, too, or they've talked about it before, that your hidden rating informs your SR. As you win or lose, they'll just interpolate you towards that hidden rating.”

“It's a similar system under the hood to casual,” continued Olson. “The only difference, is as you play over time, they kind of reveal that number to you…It…interpolates over time.”

“I personally am not a big fan of that approach,” he added. “If we're gonna do a rank play system, I'd like it to be more like Elo, where the number you see is what you're being matchmade on.”

“In the current COD system” in RP, “it'll matchmake you on that hidden number, but your visible number will give you this illusion of a journey.”

“Contrast that to [COD] World War Two,” from 2017. “That was the Elo system that I wrote.”

“The World War Two Elo system was a purely visible MMR,” he continued. “What you saw next to your name was the number you were matched on and, like, your opponents, that was the number that that was our best guess of what their skill was.”

“It did increase over time. We did [intentionally] underrate players a bit, so they had room to climb, but it wasn't an artificial journey. This was just, if you win, you earn more points and you climb up the ladder. So, a little bit of nuance there.”

“In general, a rank play system where you have a visible MMR is pretty rare.”

I tried to pin Olson down on the share of all COD players that entered RP. From poking around online, it seemed to me that it has probably hovered in the 10-20% range in recent years.

“I think that is probably even a little high,” Olson responded. “I would guess it's less than that for COD. It's higher than that for some games we work with that I won't mention…”

I haven't seen the COD data, but that ballpark, 10% to 20%, sounds about right,” he continued. “It could even be much lower, way back, like, [COD] Black Ops II ranked.”

That title debuted out in late 2012. It was the franchise's first major implementation of ranked matchmaking (and RP back then was called League Play).

“I think you might be 5%” of the entire player base that entered ranked play in the 2012-13 timeframe, Olson estimated. “They tried to make it more of an elite thing, like, ‘Don't even come in here if you're not hardcore.’ That really scared a lot of the casual players away.”

“The other thing with ranked is that it usually launches kinda late. They don't have it at launch, so you've already filtered out a lot of the casual players to begin with.”

Next, I pointed out that RP seasons last two months and that players are reset down to the base tier of their current rank with each new season, except for Crimson and Iridescent, who reset down to Diamond 1. As a result, players in the top two RP skill ranks generally experience a much bigger reset than players in the lowest five ranks. For example, a Bronze ranked player may only lose a few hundred SR at the start of a new season. A hardcore Iridescent grinder, though—players who have been known to top 15,000 and even 20,000 SR—resets all the way back to 5,400. Ouch!

I asked Olson why Activision resets its top 4-12% of RP players down so much further other ranked player (note: this share range for players in the combined Crimson and Iridescent tiers comes from credible estimates from here and here).

“The highest skill players are almost always the highest engagement players,” Olson began. “They're gonna play a lot more matches in a season than a low-skill player. I think [in] every game I've looked at this…your low-skill players—the Bronze, Silver, Gold—they play a fraction of what the Diamond players will play in a season. They need a bigger reset if you're gonna give them some sort of progression.”

“The other reason they do it is because…you don't want to have players lose rank over time. If anything, you wanna be on the low side.”

“The really concise answer is simply because high-skill players almost always tend to play a lot more than low-skill players.”

I’d done some back-of-the-envelope math prior to the podcast and estimated that the average SR for Iridescent players was probably around 12,500, and for the next rank down, Crimson players, the SR average was likely closer to 8,750. To me, this implied the “skill gap” between the top 1-2% of RP players (in Iridescent) and the next 3-10% (players in Crimson) was around 43%. Put another way, CODs RP suggests the top 1-2% of players in any given season aren’t 10x more “skilled” than the next 3-10% of players but, rather, <50% more “skilled.”

I was curious what Olson made of that assessment.

“You have a lot more bad players than you have good players,” he began. “That is also true of ranked. There's a thin tail where you have players who are so much better than the average player, like, an order of magnitude better.”

“I think there probably is a longer tail on the underlying MMR than what they show players. They're kind of compressing that into something that's a little more…that doesn't have that extreme discrepancy, where you have your highest-skill player who's just, like, so far away from the average player that the average player is like, ‘What's even the point of trying?’ They probably shrink that, or they have some scaling rules in there with the hidden MMR.”

“Probably what you also see is that the high-level players just have to grind a lot to keep going up in skill” rating, Olson continued. “They don't lose enough points to come back down when they lose a match.”

“At the very extreme, the more they grind, the higher they climb. And you have to keep scaling down the amount of points they earn because there's just nobody who can pull them back down, at least not unless you wanna have, you know, earn 100 points for a win but lose 2,000 points for a loss. You don't want that scenario.”

Finally, I asked Olson to step back and think about matchmaking in the context of an IRL marketplace. Given that Activision and other esports-oriented video games have set up these elaborate skill-based systems to govern how competition works, I wondered if Olson had ever thought about how those rules might be applicable to, and indeed benefit, IRL marketplaces, where the rules of competition appeared far weaker. Don’t too many U.S. IRL markets suffer from a competitive blowout problem, for example?

Olson was skeptical about making such a comparison.

“I don't think that's a matchmaking problem as much as an anti-cheat problem,” he said at last. “It is literally about making sure everybody has a level playing field.”

“That's what the government is supposed to be doing rather than, allowing companies to [go on] kill streaks or score streaks in games.”

“In COD, an example of this is where you get so many kills, now you're just gonna pile on. And now the other team has no chance of coming back.”

“It’s a little bit harder to make that connection” in matchmaking, Olson continued, playing along, “because you're not pairing up…A small startup and Google should be not matched…I don't know exactly how to make that analogy, but definitely the anti-cheat” point is applicable in both contexts.

“You wanna make sure they're not doing unfair things to stifle competition, and it's truly about leveling the playing field, rather than matching them up together for a competition” in the context of what IRL governments should be focusing to enhance competitive dynamics.

Some tasty food for thought there!