you can't solve magic with data

A short time ago, Wizards changed how Magic Online Daily Event results were posted to mtgonline.com. Previously, every completed event had all 4-0 and 3-1 decklists posted within 24 hours; now, this is restricted to results from one event per format per day.

For my non-Magic Online-inclined readers, I implore you to temporarily suppress your yawns, eyerolls, and faux-masturbatory gestures, because this change speaks to a Wizards ideology bigger than some obscure MODO-related reporting. It is, on philosophical grounds, one of the worst decisions Wizards has made in quite some time. Fortunately, it has next to no direct impact. Wizards has finished some sort of beverage in a glass bottle, seen recycling and trash containers equidistant from themselves, considered their options, and decided to smash the bottle over the head of the closest Magic Online player.

First, to address the tangible impacts this decision does have: all the work that’s been poured into mtgoacademy, puremtgo articles, and the like is devalued, because they can no longer analyze anything but a subsection of the data. When I was actively working on decks and playing in DEs, mtgoacademy was invaluable for seeing different archetypes, variants, sideboard strategies, and linking people to my deckbuilding ouvre and demanding they find five improvements I made over earlier versions of the archetype. Will these data analyses still exist? Sure, but they’ll be notably worse due to Wizards’ change to their reporting.

Before going further, let’s look at the given reason for the change, courtesy of Wizards_Sean on the official forums after someone made a thread after noticing (ie, the change wasn’t announced until after it was implemented):

“In regards to the recent reduced event coverage, this was a conscious decision by the Wizards R&D team that wasn’t made lightly. Ultimately, we feel that publishing every deck list leads to solving constructed formats far too efficiently, resulting in early stagnation that’s not fun for anybody. We still want to show new deck ideas every day and provide insight into the Magic play environment, but we don’t want metagame development to become purely a function of data analysis.”

The easiest way to respond to this is by questioning whether such a minor change reporting a small subsection of Magic tournaments can possibly accomplish their goal of slowing the pace of metagame development; for this to occur, metagame development would have to rely to some degree on Daily Event results posting after every event rather than just some of them. I’m scratching my head trying to come up with a connection here. This is the “easiest argument” because it’s also the weakest form of this line of argumentation, and we can certainly do better if we dig a bit deeper and turn our outrage level up past three.

This Daily Event data is a snapshot in time. Its only pure usage is to show how things were at a certain place and time; it is not a window into the future. People attempting to analyze this data can, with infinite time, perfectly solve the format for that day. But, as anyone who has followed a metagame closely on a week-to-week basis at a time when there are large incentives to do well in it (PTQ invites, large prizes at SCG tournaments, etc), solving last week’s format will not help much for this week’s tournament, because last week’s solution has moved beyond its origin as a solution and become the basic problem for this week’s events. People are playing their decks this week, and they have access to last week’s results as well, and regardless of how perfectly you crafted armor to stop sword thrusts and slashes, that will not help when your opponent shoots you in the head. This is what metagame evolution is, and why reducing the amount of data will not stop the steady progress of technological development: data is definitionally old, and relevant technology is definitionally new.

There’s another term that needs defining, though, and it will certainly be difficult to pin down: “solving formats,” and what a “solved format” is. The consensus^[1] is that a solved format has exactly one best deck, and that playing any other deck is just wrong; the deck’s worst matchup is itself. The canonical examples here would be Standard Affinity (post-Skullclamp banning^[2]) and the most recent true boogeyman, Cawblade. While some people advocated playing other decks during the reign of those, they were proven more and more wrong with each tournament result.

Cawblade rose to prominence during a fairly unique period of time in Standard history: the roughly year-long heyday of the StarCityGames Open Series. With an unprecedented amount of money on the line for week-to-week Standard tournaments that previously had mostly been the domain of FNM and the occasional PTQ season, there came an equally unprecedented amount of work put into playtesting and tuning Standard decks.

Did data speed up this process? Not exactly. To develop the best deck for an upcoming tournament, players have to come up with their own data. Even if it’s publicly known that Cawblade only has a 48.47% chance of winning a match against RUG, that can be safely ignored by players that, from their own results, know that with better-than-average play, deck tuning, and sideboard, it’s more like 70% in their favor. Taking it further, because Gerry Thompson and other people constantly innovating Cawblade had such an advantage deck-wise compared to other people (their Cawblade builds were often categorized by SCG’s Too Much Information column under a subcategory due to large changes to the deck), they never produced any data that could be classified as statistically significant.

At the risk of being dismissive, I’m going to dismiss one of R&D’s arguments: metagame development will never be a function of data analysis, because development is so separate from data analysis by their nature. This leads to a basic truth about Magic: Magic is hard.

Seriously, have you played this game? Have you tried coming up with new ideas, or improving existing ones? Have you tried inventing a theoretical model to explain the game?

There is no mystical way to solve Magic. There is no super-secret system to figure out the game forever. There is no way that data analysis will be the main driving force behind metagame development, because (aside from issues of reflecting the present vs predicting the future) there is too much going on in the data of Magic for anyone to make sense of it. There are no computer models that can say what the big deck will be a month from now, because to have a computer model of something, we have to understand how it works. We do not understand how Magic metagame development works nearly well enough to model it in this way.




¹  If pressed for a source on this, I will cite my butt.

²  This is something people tend to forget, but pre-Skullclamp-banning Standard was significantly healthier than post-. While all the best decks played Skullclamp, there was at least some variety; Goblins, Affinity, and Elf and Nail all had some claim on being quite good, and Goblins probably had the edge. After Skullclamp was banned, Affinity was the only remaining deck, especially since Fifth Dawn became legal at the same time, infamously replacing one headgear with another (Cranial Plating) and actually making the deck faster.