Under the Hood of Call of Duty’s Matchmaking System and MMR With Charlie Olson

COD had some systems for this but didn't have a really strong MMR system. I just kinda got that thrown on my plate and figured some things out.—Charlie Olson

In June, Player Driven Founder Greg Posner and I caught up with Charlie Olson, the CEO and Co-founder of Invokation Games. Headquartered in Tallinn, Estonia, Invokation is laser focused on creating and supporting world-class skill-based matchmaking systems. Prior to co-founding Invokation, Olson did stints at Mountaintop Studios (based in LA) and MachineGames (a ZeniMax subsidiary based in Sweden).

Olson spent >6 years at Raven Software in Wisconsin (2014-2020) before that, mostly working on Activision Blizzard’s iconic Call of Duty (COD) franchise. More specifically, Olson was a Senior Software Engineer there and focused on COD’s MMR system. Olson wrote the original algorithm used in Infinite Warfare (2016), and a version of that algorithm and its related support systems have been used in every other mainline COD release since.

Olson grew up near Chicago. He went to college in Minnesota and, in mid-2000s, received a Masters in Computer Science from the University of Minnesota. He didn’t go directly into games, however.

“I got my background in medical imaging” he told Posner and I in our late June podcast. “I started doing engineering for a medical imaging company. Then I ended up going into games, which had always been my dream job. I think everybody in computer science always wanted to go into games.”

“I worked on Wolfenstein 2009 back at Raven,” Olson added, and “Singularity also.”

“I did the player movement, weapons, general gameplay programming, some tech rendering program too,” he said. “That that's basically how I got my start.”

A little more context may be helpful before turning to our discussion. Our primary entry point into MMR and skill-based matchmaking (SBMM) was a pair of white papers that Activision Research/Publishing put online in 2024. More specifically, we dissected their “Call of Duty Matchmaking Intel” white papers #2 and #3, which are here. Given that Activision spilled half their MMR beans in these papers, at least through the Modern Warfare III (2023) timeframe, Player Driven mainly asked Olson to grab a fork and empty the can.

So, what is MMR?

It’s “the matchmaking rating, the number that you use for finding players of comparable skill,” Olson explained. “Or, if you're doing team balancing, that would be the number you try to get to line up with the other team to get symmetrical.”

The basic idea is that—and this goes for all multiplayer COD modes and for esports- and competitive match-based video games generally—the studio wants to identify two (or more) roughly comparable teams by skill, and then square them off in a match as quickly as possible, so that matches tend to be relatively close and exciting for players. As we’ll see over the next two weeks, there are many potential paths to get there, and some edge cases that may not appear important off the bat do ultimately matter.

The genesis of Olson’s MMR work was rooted in China.

“I worked on COD Online, which was only released in China” back in 2015, he explained.

“This was the PC free-to-play…the first COD free-to-play title,” he said. He’d been doing “tech programming, initially, and then branched into analytics, some weapon balance-related analytics, cosmetics, and in-game economy stuff.”

“But Tencent was really adamant about having skill-based matchmaking,” which is when Olson rolled up his sleeves, unpacked the nuances of MMR algorithms, and ultimately overhauled COD Online’s SBMM system.

“After that, it ended up…[in] Modern Warfare Remastered. I don't know if you remember the Infinite Warfare, Modern Warfare Remastered era…”

Remastered was released in a COD Infinite Warfare special edition bundle in 2016. A stand-alone version debuted in 2017.

But the “MMR system I developed for COD Online then ended up getting pulled into MWR and Infinite Warfare and, like, all the mainline CODs after that.”

“I ended up also designing the ELO system for [COD] World War Two Ranked Play.”

COD: WWII debuted in late 2017. COD Ranked Play is a topic we’ll cover in some depth next week.

“And if you're a hardcore player, you might know the dynamic aim response curve on controller.”

Go look that up on your own time—it’s pretty slick though.

“I actually did that for COD: World War Two for the PC SKU, and then it ended up finally making it into a mainline game with, I think, MW 2019 [Call of Duty: Modern Warfare (2019)].”

“Three years later, Infinity Ward adopted it.”

Infinity Ward and Sledgehammer are basically the motherships of COD development, although Sledgehammer Games has played critical role here also since the mid-2010s. Raven has been more of an escort/support cruiser since that point.

“And what is Demonware, and how does that relate to skill-based matchmaking?” Posner asked.

“Demonware is kind of a support studio” too, Olson responded. “They just handle all the backend services for every COD title.”

Activision acquired Demonware in 2007.

“They've been doing the backend, which includes matchmaking, and they might be doing player data” Olson added. And “server orchestration. All the behind-the-scenes stuff is done from Demonware, either in Dublin, Ireland, or in Vancouver, Canada.”

At one point in the exchange, Posner asked Olson his favorite game of all time.

Olson grinned and climbed into the way back machine.

“All time, all time? It's probably, like, the OG King's Quest games,” which were published by Sierra On-Line in the mid-1980s. Or “maybe Space Quest III,” another Sierra title from 1989.

“More recent, I would say, like, Bloodborne, if it was 60 frames per second, would be my favorite game, at least single player.”

FromSoftware made and Sony published Bloodborne, an ARPG, in 2015 for the PS4.

“The game I played the most has gotta be Street Fighter 6 now, but that's like a love-hate thing.”

Capcom published that game in 2023.

“There's so many things that could be better…I hate playing it, but I also love playing it. It's just a pure competitive game.”

That’s more like it!

“It's like a bad relationship,” I blurted.

Olson nodded.

Subscribe to our newsletter if you haven’t, and please share this with a competitive gamer who has their own love-hate thing going—you know who I’m talking about!

SBMM and CODs MMR

If we're talking about a zero sum, though, then you could think about the team balancer as zero sum. So, skill, yes. We're trying to put even amounts of skill on both teams to make them a fair match.—Charlie Olson

An early question I put to Olson was meant to help contextualize matchmaking. I pointed out that every video game has a moment-to-moment core loop. That’s the actual gameplay: running, jumping, and shooting in COD, or kicking and punching in Street Fighter 6. Then there’s a medium-term loop or a group of loops and systems, I continued, and this is where I posited that matchmaking sits in PvP competitive and co-op games: It’s where you take a breath in the lobby between matches, review what just happened in the core loop, and perhaps tweak settings/loadouts and get a snack before diving into another core loop. In the background, meanwhile, this is where “magic” of MMR and SBMM is happening. Players don’t control that, except indirectly through their skill and some other play-related attributes/characteristics, unless they decide to stop playing.

There’s a link, though, from the medium-term loop/system of recurring matches, to seasons, which often last a couple of months. From there, there are connections to the metagame and to aggregate player account totals (lifetime statistics, trophies/achievements collected, etc.). In this long-term metagame, a COD player may progress through the 50+ levels required to hit Prestige, and climb that same 50+ level mountain over and over again for years on end.

After that lengthy setup, my question to Olson was rather simple: What did he think the “aesthetic goal” was that the entire COD dev team was working toward across all of these loops/systems that interacted meaningfully across such very different time and player engagement scales?

That excellent question landed like a lead balloon.

“As a caveat to this: I left in mid-2020,” Olson began. “During the first OG [COD] Warzone era, as far as I know, there was not a designer…[and] this might have changed, but in my experience, there wasn't a designer-led, like, ‘aesthetic goal.’ Mostly, it was tech driven, like, technical from Activision central or, like, central tech and Demonware and a few analytics teams keeping tabs on retention metrics and engagement, things like that.”

Our focus was “basic match quality, match tightness, both in terms of skill and latency. And then, like, the enjoyment metric or…game quits, rage quits, things like that.”

There might have been some input from developers,” Olson said, trying to throw me a bone, but it was mostly “a couple of us on the tech team determining how we wanted to do matchmaking or, like, tweaking and analyzing things.”

“For mainline [COD], it was supposed to be Demonware pushing the tech envelope, and then central tech or central analytics validating those changes.”

I attempted to refloat my lead balloon. I asked him to confirm that no one was responsible for lining “all these things up well” across the core loop, matchmaking, and the metagame.

“Yeah,” Olson confirmed. “At least not at a high-level design perspective. I would say with the MMR system itself, which, like, the one that I came up with for COD Online, that was specifically designed for persistent lobbies, for rebalancing teams in between matches.”

“That was the primary goal. It was like, ‘How can we rebalance these lobbies fairly in between games, where there are other systems, like Microsoft TrueSkill, at the time, that adapt very slowly?’”

An overview of Microsoft Research’s Bayesian approach to SBMM for Xbox is available in this Wikipedia entry. If you’d prefer the longer version straight from the horse’s mouth, it’s here.

“You basically get the same teams match-to-match with a persistent lobby,” Olson continued. And when these teams are “obviously unbalanced, that's not a good experience for players.”

To the extent there was “an aesthetic goal, it was just, ‘How can we get fair matches or fair team balance in consecutive matches in a persistent lobby?’”

In the 2016 COD Infinite Warfare timeframe, Olson added, his team adopted a “centralized matchmaker” approach. That approach offered “a little more control over how matches are created.”

“Modern Warfare 2019,” he said, is the point when most players “think matchmaking got cranked up or, you know, something dramatically changed…the real change there was for the all the playlists.”

“They added a lot more game modes, and they added a lot more dedicated servers. These disbanding lobbies, that had to get introduced [at that point] in order to get more players on more servers.”

At least my lead balloon led to a bit of useful context. If there ever was “someone” who was responsible for orchestrating how the core gameplay loop connected to matchmaking and to CODs metagame, perhaps that overarching goal work was completed before Olson landed at Raven in 2014.

I switched gears. The focus of the “Call of Duty Matchmaking Intel” white paper #2 was player skill. To be more precise, that white paper detailed the results of an A/B test (a split test) run by Activision researchers on Call of Duty: Modern Warfare III (MW3) players 1Q 2024 for about for a month. The test assessed those who played a particularly popular multiplayer mode, Team Death Match (TDM). This mode features 6v6 firefights and a round generally last 5-10 minutes. What Activision did during this experiment—which likely collected data from millions of MW3 players across tens of millions of TDM matches—was to deprioritize skill in the MMR for roughly half the sample. Everything else was held equal, and then the impact of that change was assessed relative to the control group.

My question to Olson, after another lengthy windup, was if he, and those he worked with, ever felt like they were managing a marketplace in which the commodity being exchanged was player’s skill.

In the TDM “marketplace,” as I put it, “the vital and scarce resource, once you get beyond ping [latency] and time to match [lobby wait times], is skill,” this white paper states. “If you look at it as a marketplace, that is the commodity, the thing that's being traded in the market, that's being managed by Activision,” right?

Olson saw another lead balloon appear over the horizon.

“I don't know if I would say that skill was the resource or, like, the commodity being traded,” he said. The “closest you could get would be, like, dopamine or dominance…[It’s] your chance at being the top of the leaderboard.”

That’s closer to “what gets distributed by the skill-based matchmaking” system of MW3. The “transparent objective of it is to give everybody a shot at performing better in a match, instead of always getting stomped.”

Posner asked if “TDM skill is managed the same way from the bottom 5% of players to the top 5% of players? Does anything different happen in the team balancing math at the highest and lowest ends of the MMR range?”

“It depends,” Olson responded, “if you're talking about the team balancer or the matchmaker itself.”

“With Activision, with Call of Duty's matchmaker, they do the team balancing as a last step. They try to put the players together into a match, and then with whatever they have to work with, then they sort them into teams.”

This often leads to a “case where you'll get the highest skill and the lowest skill player, generally, on the same team together, and everybody else in the middle is on the other team. But that's only one way to do it.”

“When it comes to the matchmaker itself…they don't do a great job of accounting for those extremes,” Olson continued. “The bottom 5% of players, I think, perform worse than you would ideally want them to perform.”

“There is room for improvement there…There's a lot of complexity to that discussion. How do you make that bottom 5% perform better?”

“Is it tighter matches or do you start injecting bots? In China…in COD Online's case, having bots in matches is totally acceptable. For western audiences, it's generally frowned upon.”

Olson added that “bots are the way to treat the bottom 5% if you wanna improve that, or just tighter matchmaking” applied in this entry or low skill player group.

At the other extreme, at “high skills, I think you wanna be careful not to drag them down too far because they're already giving up some of their enjoyment so other people can enjoy the game more.”

Posner wondered how random the Call of Duty matchmaker actually was terms of grouping players and trying to balance the skill level between teams.

“The way their matchmaker works,” Olson explained, “and they talk about this in one of the matchmaking papers, is if they have a party, say, with big disparity in it, then they're gonna try to find people who are close to the middle of that party.”

“I'm not sure that that's entirely accurate,” he added. “I think they might skew towards the higher MMR of the party. Either way, they're not trying to match the composition of the party. They're just trying to match that average.”

“Once they've got a bunch of players who are roughly in the same ballpark of skill, then they'll sort them in the teams. That's why you end up with those, like, asymmetrical team shapes, you know? The median player or the middle skill players, and then the extreme players on the other team.”

I double-clicked on this last point to make sure I was understanding him.

Let’s say “the target match is like a 90 skill-rating” out of 100, I said to Olson. “You've got some player that's a 95. You’ve got a player that's an 85th percentile on the same team, so that they balance out to 90,” on their average skill percentile. And then, let’s say, the other team “is all around 90, something like that. In that case, that highest-skilled player [at the 95th percentile] is essentially carrying, or subsidizing the amount of skill that the lowest team player has [the 85th percentile player], in order to match it [the teams] at 90,” right?

To this, I added, “Does that go up and down the entire [matchmaking] system, from the bottom 5-10% to the top 5- 10%” by skill? If correct, then “every single best player on a team is very likely going to be carrying somebody who's relatively lower skilled.”

“It's accurate but it's not the goal,” said Olson.

“It's what happens with COD's team balancer, but it's not necessarily what they want to happen. They do wanna have even teams, but to…make it more extreme, say, like, a 90% and a 10% on a team together, then they're gonna play against two fifties so that averages out.”

“That's just a consequence of doing the team balancing last versus, ideally, what you would want, is a 90 with a matching 90 on the other team.”

“Then it's balanced in terms of the average and total skill, but it's also balanced in terms of the standard deviation.”

Ooh, Olson had to pull out a shiny statistics card!

“That's how I prefer to match teams, is with moment matching. That was one of the things, one of the reasons for building our own matchmakers, so that we could do this moment matching between teams, and not just matching the means or the sums.”

A-ha! We’d slipped a toe into one of Invokation’s secret sauce jars…and found it contained standard deviation bell curve comparisons.

“If you have that ninety-ten team and that fifty-fifty team, who's likely to win?” Posner asked.

“Probably the fifty-fifty,” answered Olson. “It depends a bit…It's hard to say exactly what how accurate the MMR model is.

“In most cases I would see, like, the fifty-fifty are either able to ignore the 10% player and overwhelm the 90%. But it depends on the game, really. Either way, it's not a match you'd really like to make.”

Fun > Blowouts

[Most COD players] just wanna go pick up a controller, get into kind of a flow state, grind few weapon camos, unlock some content, and that's it. That's what they want out of the game. They don't wanna go in there and have it be a hardcore experience where they have to train, and learn things, just to be able to compete.—Charlie Olson

I locked and loaded another lead balloon question.

“Do you know how skill or post-match MMR outcomes, and its hidden recalculations, relate to the amount of XP you get at the end of a match?” I wondered. “If you play really well, is it true that your XP would be relatively high, even if your team loses, and vice versa?”

To be clear, the white paper mentioned above states that each player’s MMR is recalculated after every match. There appear to be several subtleties in this recalculation, but what was I really fishing for is if there was any mathematical link between how well a player performed in a given core loop match and how much XP they got—which ties into the long-term metagame leveling system noted earlier, including Prestige—after a match ends.

“As far as I know it has zero connection,” Olson said.

“Your match is a little more difficult if you are a higher MMR player,” he continued. “To really dominate a high MMR match, like, is increasingly unusual, versus if you tried to Smurf. Then you would have more chance at, you know, being the best player in a lobby, as a Smurf.”

Smurfing is a kind of cheating. It’s when a high-skill player creates a new alternate account, which have clean records and low MMR/SBMM ratings, and then wipes the floor with low-skill players and noobs.

“But I think it should be connected,” Olson elaborated. “That's kind of going back to the overall design.”

Precisely. I was asking if there was mathematical correlation in COD between the skill expressed in the core loop, which is captured in the medium-term matchmaking and seasonal loops, and then, potentially, handed off, in some form, to long-term metagame stats/totals.

“It is something that makes sense to reward players for skill, especially if it's gonna be, like, a hidden number that affects their matchmaking experience. But they get no perk for having a higher skill” Olson maintained. And “that's just unfair.”

Right?!

“At least while you were there,” I said, “there was no relationship that a good player would get 10% more XP than” a bad player in the same match?

“Correct. Your XP is independent of the MMR behind it,” confirmed Olson. “The MMR is strictly for matchmaking, not for the game economy. But I think it could be.”

It’s noteworthy that, in Olson’s mind, the long-term metagame and a player’s total XP is associated with the game’s overall “economy.”

“It makes sense to incorporate skill,” he added. “You have that number. And if it's a reliable number why not use it?”

Why not, indeed?

I went back for another bite at this skill-focused apple. I defined what a “blowout” was, according this Activision Research white paper: When “a team in a lobby wins with a score delta greater than 30.” In the A/B test that deprioritized skill, they found blowouts “increased for all players [in the affected sample] and has also been established as having a negative correlation with self-reported ‘fun.’”

Basically, I described the figure below to Olson in broad strokes.

Compared to the control group by skill decile, blowout wins/losses were up at least 4% and as much as >10% across all ten deciles. Activision didn’t specify how much they loosened their MMR standards in this test but it was enough to drive up blowouts, which undercut “fun.”

There, apparently, is at least a KPI that Activision tracks, even if “fun” many not be CODs overarching “aesthetic goal.”

I asked Olson why this negative relationship exists.

“Blowouts are not that rewarding to a high skill player,” he said…although the response begged a question about why Smurfs exist.

“It's like, you know, the football analogy or soccer analogy of, like, if you have pro players and they're beating up on a group of kids…there's no honor in that, or there's nothing rewarding about it.”

“It has to be a pretty extreme blowout to get to those diminishing returns.”

“You do like to have a match where your team just dominates, and you have a great stat sheet and, you know, that kind of outlier,” he continued.

“Everything I've looked at seems to indicate more that it's individual performance that players really care about. And it's just [that] the blowouts are correlated with that.”

If your team is on the winning side of a blowout or “at least in a more even match, then you have a higher chance of being the best player in the lobby or having, like, a great personal performance.”

“Maybe it's a little cynical, but I think it's really about individual performance for most of those cases.”

I ran through a few more white paper quotes to tee up another question:

• “We have historically found that in-match quit rates have a strong negative correlation with self-reported ‘fun’ gathered through player surveys.” (My point: Blowouts are also linked to higher match quit rates.)
• “When skill is utilized in matchmaking, 80-90% of players experience better end-of-match placement, stick with the game longer and quit matches less frequently.” (My point: Long-term engagement is positively correlated to tighter “skill market regulation.”)
• “If low skill players engage with our titles less, then higher and higher skilled players become the new low skill players (relatively speaking). As a result, they then experience the negative outcomes of being the lowest skilled players in the core multiplayer population, likely resulting in those players then returning at reduced rates. This ultimately becomes a negative feedback loop.” (My point: Player churn was higher in the depri skill test sample.)

My question to Olson: “In your experience, do you think it's generally true that if skill is loose enough [MMR/SBMM parameters are weak], and the matches become unbalanced enough, that it eventually starts killing the entire player community, or is that not fair?”

“I think that's true,” said Olsen. “As far as I know, every game that's looked at it has seen the same thing.”

Respawn’s/EA’s “Apex [Legends] has gone into this in a few presentations and papers. And [Epic Games’] Fortnite obviously has skill-based matchmaking. Pretty much every game that's tried skill-based matchmaking versus, like, a deprioritized skill [approach], has seen that skill helps in all these KPI’s that they're after.”

There’s always an outlier though.

“I think maybe, [Bluehole’s/Krafton’s] PUBG is a notable exception to that because that audience is very baked in. As far as I know, they don't have skill-based matchmaking. It's really purely random. And they have a player base that expects that and wants that.”

“It depends on who you're catering to as well.”

COD, said Olson, is “actually a very casual audience, contrary to, like, the most vocal critics, influencers.”

Those who have “been playing it for decades now, they tend to be a little more, I guess, affected by skill-based matchmaking, and are more outspoken against it. But the bulk of the audience really would fall off” if the importance of relative skill were loosened in the matchmaking system.

“You would have that cascade effect with the lower skilled players” that Activision referenced in the white paper. Those are “people who just play two hours a day and don't really care about getting better. They're not there to compete in esports.”

Added Olson, it “depends on the game, but I think most mass-market, AAA games have seen exactly that sort of cascade effect, where you have the low skilled players start falling off faster, and then that pulls the next decile with it” if skill parameters are too loose in matchmaking.

Roger that.

I dipped into the quote well one last time:

• “With deprioritized skill, returning player rate was down significantly for 90% of players. The 10% of highest skilled players came back in increased numbers, but in aggregate, we see meaningfully fewer players coming back to the game.”
• “Kills Per Minute is down significantly for the bottom 20-30% of players. The next 60% of players have no significant change, and the top 10% see significantly higher KPM.”
• “We also tightened skill in Call of Duty: Modern Warfare III. This had inverse results consistent with the results of the loosening test. The quit rate was down for 90% of players and we saw other improvements in the experience of low-skill players [i.e., KPM]. However, we observed negative impacts for high-skill players.”

There’s a lot that could be unpacked from these findings, but my basic question to Olson was this: If skill is deprioritized in matchmaking, then high skill (high MMR) players tend it like the game more but those who get crushed also like the game less, and the net effect on the community is negative in terms of engagement, which is bad for the player community as a whole and bad for Activision’s designs to monetizing COD players or to steer toward any other KPI, correct?

Olson said that Activision has “limited levers” in terms of pushing toward improved monetization or some other KPI though their MMR and SBMM systems.

“They have these crude levers,” he reiterated. “When they only have a couple levers, there's only so much they can do to make everybody happy. There's quite a bit of nuance there.”

“But Activision, indeed, like, they do choose these KPIs,” and steer toward them. The MMR and SBMM stuff he worked on can and are optimized to a significant degree, but these KPIs are “a proxy for what they really want, which is long term revenue, and they want long term retention.”

Activision naturally wants players to “keep coming back and buying the game, and then also playing the game for months at a time so that they can buy more MTX [microtransactions] and optimize for revenue.”

At the same time, “Engagement, like, that does not equal fun. We know that.”

“A lot of mobile games are engaging, but are they fun? Or are they just getting you hooked?”

“Trying to squeeze engagement out of players is maybe not the best tactic.”

“Likewise, ignoring the sort of soft data, like player complaints,” can also be counter-productive.”

Skill-based matchmaking gets tons of negative sentiment every release, every COD cycle. Ignoring that is maybe also not a great strategy. [If] you let that negative sentiment just kind of build up, and ferment…that snowballs, maybe slowly, but it builds up.

“The flipside,” Olson continued, “is people will then jump in and say, ‘Well, you just gotta get rid of skill-based matchmaking altogether.’ And that's not necessarily a good solution either. That's just the obvious solution, if it negatively affects you as a player.”

Yes, Activision does “choose the KPI. They might be measuring just what they can see, and kind of ignoring these longer-term trends, at their own peril” though, Olson warned.

“I don't think it's as extreme as some of the players would argue with Black Ops 7,” which debuted in November. “They're like, ‘Well, it's sold poorly because of skill based matchmaking.’ That's not it either. Activision is not completely correct. They're not completely wrong either.”

Spoken like someone with a deep interest in achieving balance!

Next week, we’ll wrap up with more COD insights about this depri skill test, as well as how Ranked Play works for more hardcore COD fans, courtesy of Activision Research and Charlie Olson’s expert insights.