A better way to review video games

Reviewing video games is hard. In some ways, it’s an impossible mission: a reviewer has too many conflicting interests:

1. please the publishers or else be denied access to the materials they seek to review
2. please their editor or else don’t get paid; but the editor’s primary source of capital is often advertising … from the publishers
3. answer the consumer’s main question in a way that earns their trust: should they purchase this game or not?
4. stand out from the crowd of a million game players who decide to write about their hobby

Who’s your Daddy?

This has been a problem for as long as I can remember (20+ years of game playing and reading game reviews); the consumer *believes* that the reviewer is answerable to them – but it has been a very long time (10 years now?) since consumers were the paymaster of reviewers; nowadays, it’s advertisers (which usually means: game-publishers).

Of course, consumers still wield huge power. The virtuous value circle – the only circle that matters – is driven by consumers:

• A reviewer has a “readership” of consumers who are influenced in their purchasing decisions by those reviews
• Publishers therefore court the reviewer to try and curry favour with the consumers and increase sales of the publishers’ products (to those readers, and anyone they themselves influence – friends, family, colleagues, etc)
• Reviewers earn more money, and get deeper access to development teams (courtesy of the publishers), so produce more reviews

But that power is – clearly – both indirect and hard to quantify. A consumer – even many of them – threatening to “stop reading a reviewer’s reviews” is not particularly effective.

Publications like Edge helped along the indirection of consumer-power when they decided to go out of their way to obscure the identities of their individual reviewers, turning reviews into as much of a crap-shoot as buying games was in the first place. Since the web rose to prominence, it’s been eroded at the other end – there’s now so many reviewers around that, well … who has the time to remember who any individual reviewer is?

Qui custodet custodes?

But if journalists/reviewers are supposedly there as a watchdog on the publishers’ marketing depts, supposedly helping the consumer determine which are the (non-refundable) purchases they ought to be making, then who’s checking that the journalists themselves are honest?

No-one, really. And that’s where the rot begins. The storms of outraged public opinion are nothing new: examples of journalists writing reviews of games (reviews both scathing and rejoicing) they hadn’t even played go way back into the 1980’s.

A case study in lies, damn lies, and video game journalism

In case you hadn’t heard, this week a “staff writer” from Eurogamer (a games review / news site) ripped to pieces one of the most recently-released MMOs – Darkfall. At which point Aventurine, the developer of Darkfall, responded with increasing anger and dismay.

But the really interesting thing here is that Aventurine didn’t merely rant “you bastards! Our game is Teh Awesum!!!111! STFU, Beotch!” (well, they did that as well) … no, they dropped a little A-bomb in the middle of their reply:

“We checked the logs for the 2 accounts we gave Eurogamer and we found that one of them had around 3 minutes playtime, and the other had less than 2 hours spread out in 13 sessions. Most of these 2 hours were spent in the character creator”

Pwned. MMO developers *actually know whether your journalist played the game before reviewing it*. What’s more … they have proof…

The EG reviewer (whose “references and background are immaculate”, according to the editor – but from reading his only two EG reviews, I’m afraid it does rather sound like he knows little about MMOs), responded (via his editor) with the claim:

“the logs miss out two crucial days and understate others, … and he insists he played the game for at least nine hours”

It would seem that someone is lying (and it could be either party). Worse, someone is being particularly stupid. Because the journalist is claiming “your computers lie”, and the developer is claiming “your journalist is a lier”; either way, it’s not a subtle, small, mistake – whoever is wrong, if they get discovered, they’re going to create themself a good amount of long-term trouble (bad reputation).

Lots of MMO developers write shitty server code, and honestly don’t know what the hell is going-on inside their own game-world (but fondly imagine that they do – and proudly boast to the press (in the vaguest terms) that they do). But the rule of thumb is that devs who don’t know … don’t even know what it is they ought to be claiming that they know. The specificity of Aventurine’s claims suggests that they do have the stats, and those stats are mostly correct.

(I say “mostly” because there is a bit of vagueness about what – precisely – the reviewer was doing in-game. That reeks of holes in their metrics/logging. They clearly know when the player was logged-in, and what they did/said in chat, and how many characters were created – but apparently not what they were doing in the client, e.g. how long did they spend in character creation? Implicitly: unlogged; unknown)

Whereas it’s quite likely that a non-knowledgeable journalist, accustomed to buggy games, would assume that they could safely claim “your server is buggy, those figures are wrong”.

Unfortunately for any such journalist, server logs are generally either correct, or absent entirely – there’s rarely any middle-ground. If he knew a bit more about MMO tech he might know this; very few journos (any of them?) know that much about the games they review, though.

So … based on nothing but casual observation and intimate knowledge of the tech issues (and several decades of reading game reviews…), I’m leaning in favour of Adventurine and against Ed Zelton. My guess (pure *guess*) is that he’s been caught out being either incompetent or perhaps a bit lazy as a reviewer, and he’s thought he could get away with blaming it on buggy code. From reading the review, I get the impression he wishes he were Ben “Yahtzee” Croshaw (from Zero Punctuation) – although he clearly isn’t funny enough – but he seems to like saying “it’s shit; you’re shit; you’re all shit; STFU” instead of reviewing the game, and seems to think that’s good enough. As an MMO player, my feeling was that the review was, well … useless – without even playing the game, there is so much more I would want to hear in a review, and so much of his wanky whining that I couldn’t care less about. As an MMO developer, it felt downright insulting, as if he’d made no effort at all to play the game as a game. Actually, it felt like he’d hardly played MMOs in his life, and didn’t really know what they were.

(NB, from the review: his apparent ignorance of some of the most important *and best-selling* RPG + MMORPG games of all time – the Ultima series – suggests that he really isn’t much good as a game reviewer. YMMV.)

Reviewing the reviewers

Up-front I’m going to point out that I don’t believe all MMO developers are currently capable of doing this – many people would be amazed to discover the true state of metrics collection in this industry – although *all* modern MMO developers ought to, and it’s not too hard to add-on later (add it to the list of “things MMO developers ought to do as standard practice, but many of them don’t do”). But it’s a general thing that I think we should move towards.

MMO developers (well, actually, the Operators, but that’s getting pedantic) are in an excellent position to help guard journalistic honesty, in a way that traditional game developers have never been able to. I would like to start seeing the following published by *every* MMO developer each time their game is reviewed:

1. What level the account(s) started at
2. What level the account(s) peaked at
3. How many hours the reviewer spent at the lowest levels, levelling-up manually
4. How many hours the reviewer spent at the highest levels
5. What percentage of time was spent on each of the different primary character classes and factions
6. Which areas of the game / aspects the reviewer actually engaged in (hours of combat, hours of crafting, hours of chat, etc)

…but, honestly, this isn’t so much about “journalistic honesty” (I used that phrase tongue-in-cheek above) as it is about starting a virtuous cycle of developers being more cognizant of what, actually, players “do” in their games – preferably *before* gold launch. In particular, if publishers (developers) started supplementing reviews with this info (as a matter of course), I think we’d see a sea-change in industry staff appreciating three key things about metrics:

1. How little metrics they’re actually collecting compared to how much they think they’re collecting
2. What metrics actually matter, and/or are useful?
3. How players actually play the game; by extension: how fun is the game, really, and which parts suck horribly?

Does this work / matter?

At NCsoft, I got into the habit of asking prospective partners, hires/employees, and external studios which MMO’s they played (fair enough) … and how many characters they’d got to the level-cap with / what level their characters had reached. It started as an innocent question, but I quickly noticed how often it gave early warning of failures of honesty among individuals, and how much it presaged the problems they would have in the future.

The two worst problems were “complete ignorance of the MMO industry (either of pre-existing design practices, or tech practices)” and “personal self-deceit about what the person knows, and what they don’t know”. The latter tended to be a far worse problem: when someone is deceiving *themself*, it’s doubly hard to re-educate them, because first you have to get them to accept their own deception.

Of course, it turned out to lead to a lot of defensive responses and a spew of self-justification, which made us both uncomfortable. In those situations, it can easily lead to making assumptions that certain people’s opinions are “worth less” because, say, you know for a fact they’ve never really played an MMO – at least, not in the way that most of that MMO’s players would/will/do play it. I hate that tendency, since it’s part of a snobbishness that lies at the root of a lot of oyster-like, head-in-sand behaviour in our industry. On the other hand, it’s important and useful to know when someone’s ideas are random conjecture and when they’re based on fact (and very few people in a design meeting or publisher/developer meeting will honestly tell you their ideas are conjecture :)).

On the whole, though, it turned out to be a really useful line of questioning – even bearing in mind the additional (smaller) problems it created. There are obvious problems that come from the statistical supplementing of free-form prose game-reviews – but I’m confident that these will be outweighed by the advantages (and the problems that will be shrunk).

PS:

Despite the TLC of good friends, I’m still weak and sapped of all energy from my month of illness. I’m triaging like mad to deal with urgent issues, but there’s plenty of highly important stuff that’s been pending on me for a while that I still haven’t had the time + energy to deal with. So, if you’re still waiting … I’m sorry.