ION08 – Changing a Live Game: Lessons Learned and Techniques Applied

Moderator: Jason Roberts, 38 Studios
Steve Danuser, 38 Studios
Darius Kazemi, Orbus Gameworks
Troy Hewitt , Flying Lab
Osma Ahvenlampi, Sulake

Summary

This was a surprisingly good session – not only was it 9am in the morning, the night after the official conference party, but it was also a panel session (which, as several people were commenting to me yesterday, tend to be bland and sucky at games conferences. My own experience is that moderators of panels at games conferences often have silly / selfish reasons for the panel, and so they do a poor job. e.g. when they admit that they just want to meet / befriend / privately interrogate a particular person, so they create a panel session).

In particular, props to Jason Roberts for prepping this really well. I spoke to him briefly afterwards and asked how he’d decided to do it. Apart from the obvious thing of thinking up a bunch of good, reasonably specific questions (many moderators think up bland, crappy, vague questions, usually because they themselves know NOTHING about the topic (“I’d love to learn about that topic” being one of the common crap reasons that bad moderators have for deciding to do a panel!)), he also had a 30-minute pre-panel conversation with all the panellists together, so they could meet each other, find out what each other thought and did, and experiment with what they were going to say, etc.

So, a lesson there for any future panel organizers ;).

The session in general didn’t really reveal anything greatly suprising, but was information rich, a lot of good stuff there. A few gems, like Darius saying that he would routinely datamine the chat logs for the MMOs he worked on, searching for keywords (nothing new), and use the surnames of senior executives at the MMO dev company as keywords (as far as I know, that IS unusual – and a great idea ;)).

My own commentary in [ square brackets ], any mistakes/misunderstandings my own fault :).

Moderator Questions

[ADAM: I couldn’t keep up with the questions being asked by the moderator, couldn’t type them and the answers fast enough. Darius emailed me his notes from beforehand of the questions, but I couldn’t easily fit them to where they were answered :(. So, I’ve included them here – read them first, and then hopefully the flow of the speaker’s answers below will make more sense :)]

* One of the more difficult things in makng a change is identifying areas that need it first. What techiques or lessons have you learned about where to look, how to interpret what you find, and select the right choice among the possibilities?
* Changes in a game always threaten player confidence. There is always someone who likes things exactly how they are. What kind of recommendations can you make for helping to keep player confidence high throughout the implementation of needed changes?
* Exploits and cheats. The dreaded killer of fair play. What kinds of problems should we be on the watch for, how can they be most quickly detected, and what should be done with the players who take advantage of these holes? What might be the circumstances in which an exploit is actually helpful to the game?
* When a change is necessary for the short or long term health of the game and may be negatively viewed by the player base how can this process be most effectively managed?
* Do you have any stories of a change or changes made within a game that led to a result you didn’t predict? What do you believe should have been done differently and what would you predict the outcome would be with that difference?
* Is there such a thing as a “window of change” where you can make changes to a game but if that window of time is missed it is better to leave the game alone?
* Similar to the question above is there a time when it is better to make major changes to a game than others? What or when would these be?
* Invariably with even minor changes there will be a portion of the population who will take it upon themselves to crusade or protest the modifications to their beloved pastime. How do you predict and manage these players to minimize negative perception of your game?
* The players will often be a source of change themselves asking for features or modifications. How do you go about listening to the varied opinions and deciding which of these suggestions brought from the player base itself get incorporated into the game?
* Is it ever appropriate to make changes that are so large they fundamentally alter the game and the ways in which it can be played? How far is too far when making changes, where do you draw the line?

Introductions

[ADAM: I didn’t type the personal biographies and first question, because I had Technical Problems with my laptop. Apple’s got some usability problems with the UI for Keynote. Sigh]

darius

so much vitriol can come from changing inventory related stuff, so having finegrained info on what everyone has in inv, and how much, etc helps enormously with nerfing design changes.

There was a meeting for Dungeons and Dragons Online (DDO) where they were arguing over an axe that was ridiculously over-powered, a 1-hour meeting, and darius happened to walk past, and said “well, I can tell you how many people actually have one”. turned out to be 20 people in total, so they realised it was a non-issue and they could safely nerf it.

steve

things like Guardians running around naked tanking epic mobs etc, stupid stuff because the stats were way out from what they should be.

messaging this to community has to be handled very delicately.

the combat revamp arrived on the same day as a new expansion launched. So, combined the need for players to relearn the basic game stats and balance with accessing all the new content, which was a big headache – too much change at once for players.

darius

try to identify thresholds on the metrics we collect – “we want to see a report every day on all the quests that have been abandoned more than 5% of times” – etc, so that the actual volume of queries/reports is small enough for people to handle.

Identifying outliers is really important, but sometimes those are actually what’s really fun. Sometimes Emergent Gaming is an outlier.

osma

with every change we start with a business objective: improve conversion rate, increase number of buddies that players have on average, things like that.

make the measurement ahead of time to be able to do Before – After comparisons.

Things that affect the economy and inventories are the easiest to mess up

troy

nobody likes change.

even the smallest changes can create hysteria in your community

any change we introduce (mentioned the recent server-merge as particularly disruptive) – starting well before the change we make sure we communicate as much info as possible to the community.

forums are interesting places to be, full of interesting stuff to read, but not necessarily indicative of the truth of what’s going on. metrics balance that with accurate figures of “how many people are affected” etc.

don’t treat player community like children even if they’re acting like children.

gives them more confidence in your product and more loyalty.

steve

the ideal window for making game changes is beta. Doesn’t seem to work that way in practice, so I’d say that the first year is the best time to do it out of what you practically have to do.

your name is still out there, the product is still visible on store shelves, etc.

past the first year, and dealing only with existing subs, when you have little opportunity to acquire new users, it gets dangerous

[ADAM: I was disappointed by that answer – at the 12 month threshold is definitely NOT the right time to be reducing change magnitude IMHO – that’s a time to be *consistent* about how much you change things. Also … the idea that after a year you’re mostly servicing existing customers rather than trying to acquire new ones? That’s a pretty contentious thing to advocate]

darius

if you have some data that FORCES a change, would you (all of you) show that to the community to justify what you’re about to do?

steve

be careful about making sure you’re not giving the impression you’re letting the players take a vote on it – don’t want to set the precedent that you’ll do “whatever the community demand”

troy

depends on the sensitivity of the data – in general it’s essential to put it out to the playerbase, but definitely don’t let them think you’re letting them vote on it [i.e. what steve said]

osma

there’s no such thing as window of change – or if there is, it’s “the first 20 years”

players will get used to what you do – so just keep making changes until they get used to it!

forums are an interesting place, but make sure you don’t pay too much attention to them

most players wont actually tell you on forums until too late, until you’ve screwed up in a major way.

these guys are amateur designers at best, so make decisions confidently off your own back, and make sure you have the data to back up your opinions, that you’re not guessing.

troy

early in my career I sent whole threads of info in emails to the designers, quoting huge chunks of player-wording, and freaked out the designers with details of the player base.

now I find community managers are really good at just pulling out the kernel of what’s going on, providing just the key points very quickly

too much information, and it becomes potentially unhealthy for the community and the game

steve

yeah, community mgrs distil out the nuggets of gold. SOME designers will live on the forums and have thick enough skins to stay there and see it all, but most dont.

the com mgr needs to be one of the leads on the project, and their feedback needs to be really EXAMINED and LISTENED to.

when done well, the project leads investigate this stuff and get to the root cause of what’s really wrong – which may be different from what’s being complained about; the complaints are sometimes just symptomatic of the problem

darius

exploits and cheats usually show up as outliers – or else they have no real impact and aren’t a big problem anyway

I’d find the groups of people participating in these exploits. Usually it starts with a core group of 5-10 people on just one server. so, first you can identify the people who first exploited it. Then you can watch the idea expand to more people, but also hop from server to server.

now you know who found it, who seeded it and – more importantly – who COMMUNICATED it

[interesting way to find and choose early adopters ;)]

send that to CS, who snoop on those accounts and work out if its gold farmers – i which case just ban them – or genuine players – in which case you probably have a gameplay / design problem if they’re deciding to make use of this

steve

designers in UO/EQ would innocently make mistakes where they don’t realise the impact of their changes

more modern generations are a bit more closed, don’t have so much allowance / support for emergence, with the designers exerting much more direct authorial control.

but you have to bear in mind that stuff that many people think is “unfair” is to other people actually “what makes it especially fun” – especially discovery of emergent gameplay that gives big advantage

(all)

look at the impact of the advantage on the rest of the game to decide whether its exploit or emergent gameplay

steve

SW:G obviously has the most drastic re-design of any game [ADAM: the infamously badly-received New Game Enhancements], and I think people agree that was probably too far.

you have to look at the long-term health of the game to decide whether a major sweeping round of change is too much, there’s no single line of what’s too much

osma

I think you can actually make pretty HUGE changes in a game and its OK, so long as you do it in small discrete steps, and give the players the chance to accomodate to each change a little before adding the next small step.

darius

one of the things I was most happy about with LOTRO was that by the time we went live everyone was pretty much happy with the gameplay / design, because we did so much iterating over problems using the metrics and then redesigning to fix them.

I think to this day still they’ve not needed to change much at the basic gameplay level

Questions

Q: are there any tools to help with this, or do you just have to manually mine the forums etc?

troy – I’m working with people from washington University to automatically datamine some of that and try to identify thought-leaders, then we’ll target them specifically as conduits for new information

darius – i set up a bunch of keywords and checked them against in-game chat. Stuff like names of executives, the company name, or the names of particularly important/powerful items etc.

From a more community aspect, I like to have a full social graph of the game: complete map of “who’s friends with whom?”. Can easily run many different metrics against it to determine key players in different contexts, and then you go and talk to them directly etc.

[ADAM: wonder how easy it would be to create this tool in one day using just a list of names participating in chat logs?]

osma – there are a lot of trackers for this stuff in the web world – Meme-trackers etc. Works great – but only if you know where the conversations are happening, we have 30 or 40 particularly large fansite forums where subsets of our game hang out, so we have to careful to track and examine EACH and EVERY one of them

Q: [ADAM: I didn’t hear this question]

osma – we’re following a very tight launch schedule, with a global update every month – and that’s substantial changes each time, economic changes, etc.

its not really for community sake that we do the tight iteration and schedule its for us as developers becuse it makes us more productive

steve – we had planned changes weeks ahead, but then emergencies come up that you have to do immediately, or “today”. And sometimes a change you need to make is too tightly coupled with something else you were changing in the current PLANNED update, and then you can’t deliver both together because they conflict. Have hard decisions which to drop or bump to later in schedule.

Q: best practices for scheduling play time etc?

darius – from numbers side of things its important to have a large enough sample set of people in the beta. From a technical side, at some point in your beta you want to have a full shard. Double XP weekends etc to temporarily get extra people in for TEMPORARY stress/load testing during beta etc.

steve – beta doesn’t mean what it used to mean any more? nowadays its about “how long will I get to play this game for free?” rather than “this is a pre-release”.

It’s about using players who’ve played many other games already. Be careful when you open up, don’t invite too many people too early. Although it comes down to how good the game is, better management of influx of players helps a lot with damage limitation.

I can’t advocate saying “its fixed in the next build” and not actually following through with it .

Q: re: datamining chat, I’m going to start talking in code (or Swedish perhaps). But .. .the question is what phrase do you get from communities that really winds you up?

troy: “this is a slap in the face”

steve: “the red-headed step-child”; anything reactionary

darius: I just love all the stuff they say, I’ve never been a community mgr, so I’ve been safe; especially fun to see what people say *in chat* about high level execs

osma : I don’t have any, I just get really upset when people disappear without saying anything at all

Q: [ADAM: me!] how many people is enough for a beta, and how many is “too many”? [ADAM: i.e. there are big problems from having too many too soon – mostly marketing or sales problems – but metrics won’t work if you have too few; deciding how many to have is often contentious inside MMO companies]

darius – start small, and gradually invite more and more. Aim to acheive the bulk of your beta tesers arriving just 2 weeks before launch, so they don’t have much time to play it too much etc. Start with a low level-cap and gradually increase it during beta, so you have a small number of people who’ve had the chance to get through all the content. we had 10k people in beta in LoTRO

[ADAM: it can be a real problem to have all your beta players having learnt how to powerlevel in the game BEFORE launch day, so that they churn through the live content within days or weeks of launch]

jason – depends what critical mass you need for it, how much you need to show all your core gameplay. start with lowest number and gradually increase to when you hit that number

osma – for us, its the whole community [ADAM: the beauty of non-subscription games!]

Q: signups versus concurrency, what percentages do you see in beta?

darius – 3:1 signups to concurrency. As beta went along that went up a bit. It was pretty high but thats partialy because we were targetting who we allowed in to beta