\n“the logs miss out two crucial days and understate others, … and he insists he played the game for at least nine hours”\n<\/p><\/blockquote>\n
It would seem that someone is lying (and it could be either party). Worse, someone is being particularly stupid. Because the journalist is claiming “your computers lie”, and the developer is claiming “your journalist is a lier”; either way, it’s not a subtle, small, mistake – whoever is wrong, if they get discovered, they’re going to create themself a good amount of long-term trouble (bad reputation).<\/p>\n
Lots of MMO developers write shitty server code, and honestly don’t know what the hell is going-on inside their own game-world (but fondly imagine that they do – and proudly boast to the press (in the vaguest terms) that they do). But the rule of thumb is that devs who don’t know … don’t even know what it is they ought to be claiming that they know. The specificity of Aventurine’s claims suggests that they do have the stats, and those stats are mostly correct.<\/p>\n
(I say “mostly” because there is a bit of vagueness about what – precisely – the reviewer was doing in-game. That reeks of holes in their metrics\/logging. They clearly know when the player was logged-in, and what they did\/said in chat, and how many characters were created – but apparently not what they were doing in the client, e.g. how long did they spend in character creation? Implicitly: unlogged; unknown)<\/p>\n
Whereas it’s quite likely that a non-knowledgeable journalist, accustomed to buggy games, would assume that they could safely claim “your server is buggy, those figures are wrong”.<\/p>\n
Unfortunately for any such journalist, server logs are generally either correct, or absent entirely – there’s rarely any middle-ground. If he knew a bit more about MMO tech he might know this; very few journos (any of them?) know that much about the games they review, though.<\/p>\n
So … based on nothing but casual observation and intimate knowledge of the tech issues (and several decades of reading game reviews…), I’m leaning in favour of Adventurine and against Ed Zelton. My guess (pure *guess*) is that he’s been caught out being either incompetent or perhaps a bit lazy as a reviewer, and he’s thought he could get away with blaming it on buggy code. From reading the review, I get the impression he wishes he were Ben “Yahtzee” Croshaw (from Zero Punctuation) – although he clearly isn’t funny enough – but he seems to like saying “it’s shit; you’re shit; you’re all shit; STFU” instead of reviewing the game, and seems to think that’s good enough. As an MMO player, my feeling was that the review was, well … useless – without even playing the game, there is so much more I would want to hear in a review, and so much of his wanky whining that I couldn’t care less about. As an MMO developer, it felt downright insulting, as if he’d made no effort at all to play the game as a game. Actually, it felt like he’d hardly played MMOs in his life, and didn’t really know what they were.<\/p>\n
(NB, from the review: his apparent ignorance of some of the most important *and best-selling* RPG + MMORPG games of all time – the Ultima series – suggests that he really isn’t much good as a game reviewer. YMMV.)<\/p>\n
Reviewing the reviewers<\/h4>\n
Up-front I’m going to point out that I don’t believe all MMO developers are currently capable of doing this – many people would be amazed to discover the true state of metrics collection in this industry – although *all* modern MMO developers ought to, and it’s not too hard to add-on later (add it to the list of “things MMO developers ought to do as standard practice, but many of them don’t do”). But it’s a general thing that I think we should move towards.<\/p>\n
MMO developers (well, actually, the Operators, but that’s getting pedantic) are in an excellent position to help guard journalistic honesty, in a way that traditional game developers have never been able to. I would like to start seeing the following published by *every* MMO developer each time their game is reviewed:<\/p>\n
\n- What level the account(s) started at\n
- What level the account(s) peaked at\n
- How many hours the reviewer spent at the lowest levels, levelling-up manually\n
- How many hours the reviewer spent at the highest levels\n
- What percentage of time was spent on each of the different primary character classes and factions\n
- Which areas of the game \/ aspects the reviewer actually engaged in (hours of combat, hours of crafting, hours of chat, etc)\n<\/ol>\n
…but, honestly, this isn’t so much about “journalistic honesty” (I used that phrase tongue-in-cheek above) as it is about starting a virtuous cycle of developers being more cognizant of what, actually, players “do” in their games – preferably *before* gold launch. In particular, if publishers (developers) started supplementing reviews with this info (as a matter of course), I think we’d see a sea-change in industry staff appreciating three key things about metrics:<\/p>\n
\n- How little metrics they’re actually collecting compared to how much they think they’re collecting\n
- What metrics actually matter, and\/or are useful?\n
- How players actually play the game; by extension: how fun is the game, really, and which parts suck horribly?\n<\/ol>\n
Does this work \/ matter?<\/h4>\n
At NCsoft, I got into the habit of asking prospective partners, hires\/employees, and external studios which MMO’s they played (fair enough) … and how many characters they’d got to the level-cap with \/ what level their characters had reached. It started as an innocent question, but I quickly noticed how often it gave early warning of failures of honesty among individuals, and how much it presaged the problems they would have in the future.<\/p>\n
The two worst problems were “complete ignorance of the MMO industry (either of pre-existing design practices, or tech practices)” and “personal self-deceit about what the person knows, and what they don’t know”. The latter tended to be a far worse problem: when someone is deceiving *themself*, it’s doubly hard to re-educate them, because first you have to get them to accept their own deception.<\/p>\n
Of course, it turned out to lead to a lot of defensive responses and a spew of self-justification, which made us both uncomfortable. In those situations, it can easily lead to making assumptions that certain people’s opinions are “worth less” because, say, you know for a fact they’ve never really played an MMO – at least, not in the way that most of that MMO’s players would\/will\/do play it. I hate that tendency, since it’s part of a snobbishness that lies at the root of a lot of oyster-like, head-in-sand behaviour in our industry. On the other hand, it’s important and useful to know when someone’s ideas are random conjecture and when they’re based on fact (and very few people in a design meeting or publisher\/developer meeting will honestly tell you their ideas are conjecture :)).<\/p>\n
On the whole, though, it turned out to be a really useful line of questioning – even bearing in mind the additional (smaller) problems it created. There are obvious problems that come from the statistical supplementing of free-form prose game-reviews – but I’m confident that these will be outweighed by the advantages (and the problems that will be shrunk).<\/p>\n
PS:<\/h4>\n
Despite the TLC of good friends, I’m still weak and sapped of all energy from my month of illness. I’m triaging like mad to deal with urgent issues, but there’s plenty of highly important stuff that’s been pending on me for a while that I still haven’t had the time + energy to deal with. So, if you’re still waiting … I’m sorry.<\/p>\n","protected":false},"excerpt":{"rendered":"
Reviewing video games is hard. In some ways, it’s an impossible mission: a reviewer has too many conflicting interests: please the publishers or else be denied access to the materials they seek to review please their editor or else don’t get paid; but the editor’s primary source of capital is often advertising … from the […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[13,9,14,53,24],"tags":[],"_links":{"self":[{"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/posts\/553"}],"collection":[{"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/comments?post=553"}],"version-history":[{"count":0,"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/posts\/553\/revisions"}],"wp:attachment":[{"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/media?parent=553"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/categories?post=553"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/new.t-machine.org\/index.php\/wp-json\/wp\/v2\/tags?post=553"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}