Categories
computer games design games design MMOG development programming Unity3D

Writing a #unity3d mini-game … to help you design a AAA game

One of my hobby projects is a testament to the Ultima and Elder Scrolls games – a massive open-world, where everything you do impacts the world around you. This has long been promised by commercial games – and it sucks from a gameplay perspective; it’s just not fun. But a very old ASCII-graphics game, Omega, showed a couple of ways of doing it that delivered on that promise while still being “fun” – very fun! That’s what I’m working towards.

It’s a test-bed for the high-performance Entity System I’m writing for Unity (more info here). I’m aiming for “gameplay considerably richer than Skyrim” with “graphics about the level of Oblivion”. To be clear: Oblivion was released almost 10 years ago! To reproduce those graphics today is fairly easy.

Challenge: Believable Cities that evolve over time

One of the things I have yet to see in any FPS RPG is believable cities, and more: cities that actually evolve. The nearest I’ve seen is the beautiful “large towns” of the Assassin’s Creed series (they represent cities, but the distances are way too small).

That doesn’t cover the “evolve” part though – even the AC cities are 100% static and unchanging. Hand-crafted, and because they cannot change (technical design choice) the player is incapable of altering them. You can’t burn down blocks, or build a new bridge. You can’t buy two houses and knock them together.

Hey, RPG industry, you can do a lot better!

This is one of the main new gameplay elements in my design. If all I managed was to take an off-the-shelf-RPG mod and add living, growing cities … I’d probably be content. What does this mean, though?

Categories
network programming programming

Debugging odd problems when writing game servers

Cas (@PuppyGames) (of PuppyGames fame – Titan Attacks, Revenge of the Titans, etc) is working on an awesome new MMO RTS … thing.

He had a weird networking problem, and was looking for suggestions on possible causes. I used to do a lot of MMO dev, so we ran through the “typical” problems. This isn’t exhaustive, but as a quick-n-dirty checklist, I figured it could help some other indie gamedevs doing multiplayer/network code.

Scenario

  1. Java, Windows 7
  2. DataInputStream.readFully() from a socket… a local socket at that… taking 4.5 seconds to read the first bytes
  3. and I’ve already read a few bits and bobs out of it

Initial thoughts

First thought: in networking, everything has a realm of durations. The speed of light means that a network connection from New York to London takes a few milliseconds (unavoidably) – everything is measured in ms, unless you’re on a local machine, in which case: the OS internally works in microseconds (almost too small to measure). Since Cas’s problem is measured in SECONDS – but is on localhost – it’s probably something external to the OS, external to the networking itself.

Gut feel would be something like Nagle’s algorithm, although I’m sure everyone’s already configured that appropriately ;).

That, or a routing table that’s changing dynamically – e.g. first packet is triggering a “let me start the dialup connection for you”, causing everything to pause (note that “4 seconds” is in the realm of time it takes for modems to connect; it’s outside the realm of ethernet delays, as already noted)

My general advice for “bizarre server-networking bugs”: start doing the things you know are “wrong” from a performance view, and prove that each makes it worse. If one does not, that de facto that one is somehow not configured as intended.

Common causes / things to check

1. Contention: waht’s the CPU + C libraries doing in background? If system has any significant (but small) load, you could be contended on I/O, CPU mgiht not be interrupting to shunt data from hardware up to software (kernel), up to software (OS user), up to JVM

Classic example: MySQL DB running on a system can block I/O even with small CPU load

2. file-level locking in OS FS. Check using lsof which files your JVM is accessing, and what other processes in the machine may have same files open.

(NB: there’s a bunch of tools like “lsof” which let you see which files are in use by which processes in a unix/linux system. As a network programmer, you should learn them all – they can save a lot of time in development. As a network programmer, you need to be a competent SysAdmin!)

Classic problem: something unexpected has an outstanding read (or a pre-emptive write lock) which is causing silliness. I remember several times having a text editor open, or etc, that was inadvertently slowing access to local files (doh).

3. can you snoop the traffic? (i.e. its socket, not pipe)

Classic problem: some unrelated service is bombarding the loopback with crap. eg. Windows networking (samba on linux) going nuts

Try running Wireshark, filtered to show only 127. and see what’s going through at same time

Also: this is a good way to check where the delay is happening. Is it at point of send, or point of receive? … both?

There’s “net traffic” and there’s “net traffic”. Wireshark often shows traffic that OS normally filters out from monitoring apps…

4. Check your routing table? AND: Check it’s not changing before / aftter the attempted read?

5. Try enabling Nagle, see if it has any effect?

My point is: use this as a check: it ought to make things worse. If not … perhaps the disabling wasn’t working?

6. Have you done ANY traffic shaping (or firewalling) on this machine at any time in the past?

Linux: in particular, check the iptables output. Might be an old iptables rule still stuck in there – or even a firewall rule.

Linux + Windows: disable all firewalls, completely.

7. Similarly: do you have any Anti-Virus software?

Disconnect your test machines from the internet, and remove all AV.

AV software can even corrupt your files when they mistakenly think that the binary files used by the compiler/linker are “suspicious” (IIRC, that happened to us with early PlayStation3 devkits).

8. On a related note: security / IPS tools installed? They will often insert artificial delays silently.

CHECK YOUR SYSTEM LOG FILES! …whatever is causing the delay is quite possibly reporting at least a “warning” of some kind.

9. (in this case, Cas’s socket is using SSL): Perhaps something is looking up a certificate remotely over the web?

…checked your certificate chain? (if there’s an unknown / suspicious cert in the chain, your OS might be trying to check / resolve it before it allows the connection)

10. (in this case, Cas is using custom SSL code in Java to “hack” it): Get a real SSL cert from somewhere, see if it behaves any different

(I think it would be very reasonable for your OS to detect any odd SSL stuff and delay it, as part of anti-malware / virus protection!)

11. “Is it cos I is custom?” – Setup an off-the-shelf webserver on the same machine and check if YOUR READING CODE can read from that SSL localhost fast?

(it often helps to get an answer to “is it the writer, the reader, both … is it the hacked SSL, or all SSL?” etc)

12. (getting specific now, as the general ideas didn’t help): if you bind to the local IP address instead of 127 ?

13. Howabout reverse-DNS? (again, 4 seconds is in the realm of a failed DNS lookup)

Might be a reverse DNS issue … e.g. something in OS (kernel or virus protection), or in JVM SSL library, that’s trying to reverse-lookup the 127 address in order to resolve some aspect of SSL.

I’ve had to put fake entries into hosts file in the past to speed-up this stuff, some libs just need a quick answer

Result

Turns out we were able to stop here…

Cas: “Well, whaddya know. It was a hosts lookup after all – because I’m using 127.0.0.2 and 127.0.0.3 it was doing a hostname lookup. I added those to hosts and all is solved

Me: “(dance)” (yeah. It was skype. The dance smiley comes in handy)

Categories
computer games games design MMOG development network programming networking programming system architecture web 2.0

MMO scalability is finally irrelevant for Indie MMOs

Here’s a great post-mortem on Growtopia (launched 2012, developed by a team of two)

It’s slightly buried in there, but I spotted this:

Categories
amusing education iphone programming reputation systems

Over 9,000…


profile for Adam at Stack Overflow, Q&A for professional and enthusiast programmers

I never imagined I’d reach anything even close to 10k rep. Lots of thoughts and some analysis to come on this in a future post – but I’ve got two deadlines coming up, so very rushed right now.

Categories
agile games industry marketing startup advice web 2.0

How middleware (and open source) downloads ought to work – Unity3D

While upgrading Unity, I noticed the current download page is a great example of how it SHOULD be done:

Unity 4 has some … issues … with backwards compatibility – but at least they made the “need an older version?” link prominent. And how many old versions can you download?

Many!

(it goes on right back to unity 3.0)

Old versions? Who cares!

Well, that backwards compatibility thing is a *****. If you work on a project with other people, and they’re using Unity 3.5 … you SHOULD NOT (must not?) use Unity 4 (there be Dragons).

But it’s fine; Unity makes it trivial for anyone joining such a project to get exactly the version they need.

Some games middleware *cough*Hansoft*cough* companies declare that everyone must use the latest version, even if it is buggy and breaks existing projects. Or if it requires staff retraining. You must retrain EVERYONE! NOW!

(Hansoft has probably changed by now – maybe unfair to single them out. But for a long time they only allowed you to download the “latest” version, and actively deleted everything else. As soon as a new version existed, BOOM! Everything else got wiped. A happy customer I was not)

Recap

So, here we have a piece of middleware, with a download page:

  • Lives at an obvious, permanent URL: http://unity3d.com/unity/download/
  • Makes it very easy to find the download link (many open-source projects: shame on you)
  • Uncluttered webpage, and makes it easy to understand which download you want (Eclipse.org: shame on you)
  • Every version has its release notes right there, for you to click on! (Apple (every product), and Firefox: shame on you)
  • Every version has BOTH the windows AND the mac downloads (computers today are cheaper than they’ve ever been. Many people have a laptop thats Mac, and desktop that’s Windows, or vice versa. You can’t assume that the browser they’re using dictates the desktop they’ll be working from)

Designing a website to look simple is certainly a difficult and non-trivial task.

But in the case of a download page – where almost everyone has the same needs, and there are many examples to copy (plagiarise) from – it doesn’t take much. More projects (and companies) should at least try to do this.

Categories
computer games design games design games industry network programming programming

All Game Developers should read Pat Wyatt’s blog…

I noticed a few months back that Pat Wyatt has been blogging rgularly and in a lot of detail last year. This (IMHO) is big news: Pat is an awesome developer who held key positions in the teams behind many of the bestselling computer games (e.g.: Diablo 1 + 2, Starcraft, Warcraft) and went on to co-found Arena.Net (creators of Guild Wars).

I worked with him briefly in the past, and he’s friendly and full of advice and knowledge – but while he was happy to share, IIRC it was rarely in published form.

I’ve had a tough few months, but I’ve been dipping into his blog a few times, and it delivers in spades. Here’s a few hilights:

Assertions: enable them in live builds

(I’ve always felt this was the “right” way to do it for servers – where you don’t have to worry so much about frame-time, and assertions are more valuable at runtime because they help with the hardest-to-trace bugs … but it’s hard to get broad data on what the performance cost is)
http://www.codeofhonor.com/blog/whose-bug-is-this-anyway:

“The bug was easily fixed by upgrading the build server, but in the end we decided to leave assertions enabled even for live builds. The anticipated cost-savings in CPU utilization (or more correctly, the anticipated savings from being able to purchase fewer computers in the future) were lost due to the programming effort required to identify the bug, so we felt it better to avoid similar issues in future.”

…and a great rule of thumb for any Programmer:

“After my experience reporting a non-bug to the folks at Microsoft, I was notably more shy about suggesting that bugs might be caused by anything other than the code I or one of my teammates wrote.”

Some bugs are due to … user’s broken hardware

http://www.codeofhonor.com/blog/whose-bug-is-this-anyway:

“Mike O’Brien, one of the co-founders and a crack programmer, eventually came up with the idea that they were related to computer hardware failures rather than programming failures. More importantly he had the bright idea for how to test that hypothesis, which is the mark of an excellent scientist.

He wrote a module (“OsStress”) which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second.

On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate!”

AI cheats to improve game balance in RTS’s, starting with Warcraft/Starcraft

http://www.codeofhonor.com/blog/the-making-of-warcraft-part-3:

In most Warcraft missions the enemy computer players are given entire cities and armies to start with when battling human players. Moreover, Warcraft contains several asymmetric rules which make it easier for the AI player to compete, though these rules would perhaps be called outright cheating by most players.

One rule we created to help the computer AI was to reduce the amount of gold removed from gold mines to prevent them from being mined-out. When a human player’s workers emerge from a gold mine those workers remove 100 units of ore from the mine and deliver it back to the player’s town hall on each trip, and eventually the gold mine is exhausted by these mining efforts. However, when an AI-controlled worker makes the same trip, the worker only remove 8 units of ore from the mine, while still delivering 100 units into the AI treasury.

This asymmetric rule actually makes the game more fun in two respects: it prevents humans from “turtling”, which is to say building an unassailable defense and using their superior strategic skills to overcome the computer AI. Turtling is a doomed strategy against computer AIs because the human player’s gold-mines will run dry long before those of the computer.

Secondarily, when the human player eventually does destroy the computer encampment there will still be gold left for the player to harvest, which makes the game run faster and is more fun than grinding out a victory with limited resources.”

Categories
advocacy community computer games design entrepreneurship games industry games publishing marketing MMOG development

Reaction to CoH (City of Heroes) community, and NCsoft’s response

(background: after 8 years as one of the world’s mid-tier MMO games, City of Heroes (+ City of Villains) is being shut down. The community banded together to ask if they could take over running the world that meant so much to them; NCsoft (the publisher, and a company I used to work for) said: no)

“No means no”

NCsoft is basically saying: “Please. We love you, but … you just *don’t understand*. It’s more complex than you could possibly imagine!”

That’s not a dialogue; it reads like a “this conversation ends when I stop talking” monologue.

“Why on earth wouldn’t you say yes?”

Lots of people wondering that. Obviously, being a public company, no-one’s going to answer that in public. We can only guess. But hear’s a few (over the top) suggestions…

If the community succeeds … then THE FEAR IS: some Executive(s), somewhere, are going to look like bad (I’m not accusing; I’m just saying that in corporates I’ve worked at, this kind of *fear* is common). A lot of the work they do is guess-work. That’s fine, they’re paid to make the best decision they can, while never truly know if they made the right one.

But if a bunch of inexperienced, eager novices come along and offer to do it for free. And – the worst possible outcome – they succeed … that could make someone look really bad.

Another thing I’ve seen in corporate politics at this level is a lot of “horse-trading”. i.e. sacrificing one project (that someone else resents, or has been snubbed by) in return for that person helping out out with a problem on a separate project, that you’re trying to rescue.

Who (individually or collectively) made the decision, and what did they stand to gain or lose? (they are probably worried about / aiming for / trying to win … something bigger than this single game. c.f. my 2009 post on why NCsoft is so huge a company gains nothing from “profitable” games, they need “mega profitable” games)

“Software is software”

Ha!

Has anyone found out yet what format(s) the data is in? Imagine the most insane, unwieldy, incomprehensible, inconsistent, unusable format that bears no relationship *at all* to the game itself … and you’re probably half way there.

This game was written *8 years ago*.

Read the biographies of the people involved. Were they non-game developers … academics with decades of expertise in distributed systems and real-time transaction messaging? … or … were they a bunch of smart guys trying to catch up with the academic research in the space of months, just enough to build and ship a major new computer game? And … most importantly … to make it “fun” before they ran out of budget.

I’ve not yet found an MMO where the people who made it feel – with hindsight – they had any idea what they were doing at the start. When they started, of course, many of them thought they’d covered all the bases, and were “well prepared”. Everyone tries their best up-front (or fails completely); but everyone finds it much harder than expected.

What should we/they do?

Looking at it analytically and logically, I’d give the community a very high chance of failing dismally if they were given the game. But … the eagerness, the excitement, the sheer determination: I’d give them a small chance of succeeding despite everything. Simply because: when you see this much determination, it often wins out and overcomes the obstacles in its way.

So, I say: Go for it.

They know the game they’re trying to (re-)create. The difficulty is simple: whenever you try to re-create a game, the temptation is always there to “improve” it … and 99 times in 100, you find you slightly misunderstood what you were “improving”.

Categories
MMOG development

Dropbox tech scaling – some great, some not-so-great

“I was in charge of scaling Dropbox … from roughly 4,000 to 40,000,000 users. … Here are some suggestions on scaling”

The first section is a WTF – the guy advocates deliberately over-taxing your servers, without a good explanation. I’ve got some guesses at why they did it – but in the general case I’d say: never do this. Only do it when it’s obvious, because you have a specific reason to do so (and you really know what you’re doing).

The rest is a lot clearer, good advice there.

Also, IMHO worth reading for this part alone:

“Let’s say you’re trying to debug something in your webserver, and you want to know if maybe there’s been a spike of activity recently and all you have are the logs. Having graphing for that webserver would be great,

[ADAM: but … often you don’t have the right set of graphs set up, and it takes a while to do that – no use if the server is in trouble *right now*]

Apr 8 2012 14:33:59 POST …
Apr 8 2012 14:34:00 GET …
Apr 8 2012 14:34:00 GET …
Apr 8 2012 14:34:01 POST …

You could use your shell like this:

cut -d’ ’ -f1-4 log.txt | xargs -L1 -I_ date +%s -d_ | uniq -c | (echo “plot ‘-’ using 2:1 with lines”; cat) | gnuplot

Boom! Very quickly you have a nice graph of what’s going on, and you can tailor it easily (select only one URL, certain time frames, change to a histogram, etc.).”

Categories
MMOG development programming security

LinkedIn (maybe) just leaked your password, won’t tell you; change it now

I’ve posted a few times over the years the … disappointing … state of LinkedIn’s engineering. But this takes the biscuit: it appears they were storing deliberately insecure passwords, and someone leaked the list:

http://news.ycombinator.com/item?id=4073309

(that page has links + info on how to check if your own password is in the mega list)

How bad is this?

  1. Many people have checked their personal, unique, passwords, that they claim to have only ever used on LinkedIn.com – and they’ve hit matches in the file.
  2. LinkedIn hasn’t told its users about the possible leak, more than 24 hours after it happened
  3. Many users re-use their passwords on other sites; any hackers could easily have stolen many accounts on other sites by now

How unlucky is LinkedIn?

This file is unsalted. That’s about as smart as locking your front door and then leaving the key under the mat – on the outside.

  1. Every tutorial, book, “newbie guide”, etc about using databases and writing login pages tells you never ever to do what was done here
  2. For any tech team, it is easy to check if this is what you’re doing, and tell your boss “uh, we need to fix that”
  3. It only takes a few *minutes* to prevent this problem, permanently. It’s not difficult

If LinkedIn were a small site, with a few hundred thousands users, I’d accuse them of laziness. But with 165million users, and a public company, you’d be looking at stunning incompetence by the tech wing of the company (the CIO and CTO never bothered to audit their own security?), or wilful negligence (no-one knew? really?).

Here’s hoping it’s a hoax…

Categories
entity systems games design network programming networking programming

Concepts of “object identity” in game programming…

Hume just posted his Lessons Learned from the warmup for Ludum Dare 23 (48 hours to write a game from scratch – starts this weekend!) – and his positive experience using an Entity System.

In his epic comment (sparked by a different Adam – not me, honest), is this gem:

“Using the entity system for the first time was unreal to me. It’s like polymorphic code. I did really weird things on the fly. For example:

– In the health processor, if the enemy was just destroyed, set a flag in the lifecycle component.
– In the lifecycle processor, if the fresh kill flag is set, extract its loot component and put that into a new entity with a small randomized velocity component and a gravity component so that the loot drops; then, remove most of the other components from the entity and add an explosion component.

The “enemy” still has the same entity ID — any other components that are looking for that entity will still find it (e.g. missiles homing in on the wreckage, or score processors looking for slain entities) — but by swapping one set of data with another, its implementation has changed from an enemy to some kind of non-interactive effect object.”

(emphasis mine)

Identity. It’s important.

(Quick sidenote: for all the people asking questions like “but … which variables do I put in Component A as opposed to Component B? How do I manage Events in an Entity System? … etc” – Hume’s approach above is a good concrete example of the first-draft, easy-to-write way of doing things. Copy it.)

Identity in games

This is one of those things that newbie game programmers seem to underestimate, frequently.

And when I say “newbie” I include “experienced, skilled programmers with 10+ years of coding experience – but who haven’t yet shipped a game of their *own*”.

(e.g. I’ve seen a couple of studios that started as Digital Agencies, or as Animation Studios, etc – that then transitioned to writing their own games. This is the kind of thing that they often struggle with. Not for lack of skill or general programming experience, but for lack of the domain-specific experience of game coding)

Examples of Identity in games, off the top of my head – all of these are independent, and interact in complex ways with each other :

  1. Game-role: e.g. … “enemy”, “powerup”, “start location”
  2. Code ‘object’ (in OOP terms): e.g. … “the sprite you are drawing at position (4,5) is part of Object X. X is coloured THIS colour”
  3. Gameplay ‘object’: e.g. … “the sprite at (4,5) represents a Tank. If a Tank sprite ever touches a Glass sprite, we need to play the Broken Glass noise”
  4. Physics element: e.g. … “5 milliseconds ago, our Physics Engine thought this thing was THERE. Now it’s over HERE. Don’t confuse the Physics Engine! Make sure it ‘knows’ they are the same object – not two separate objects”
  5. Network “master/clone”: e.g. … in multiplayer, there are N copies of my avatar: one per computer in the game. One of those N is the original – and changes to the original are constantly used to overwrite the clones; changes to clones are LOCAL ONLY and are discarded. Which is original? What do we do with incoming “changes” – which local Code Object do we apply them to? (your Code Object will be different from my Code Object – but they’ll both be the same identical Network Object, save mine is flagged “clone”)
  6. Proper Noun object: e.g. … “The Player’s Tank” is a very specific tank out of all the Tanks in the game. Many lines of game code don’t care about anything except finding and operating on that specific tank.
  7. Game-Over representation: e.g. … after the player has killed all the enemies, and they see a Game Over (won/lost/etc) screen, and you want to list all the enemies they killed … how do you do that? The enemies – by definition – no longer exist. They got killed, removed from the screen, removed from memory. You could store just the absolute numbers – but what if you want to draw them, or replay the death animations?
  8. …etc

Identity in Entity Systems

ES’s traditionally give you a SINGLE concept of Identity: the Entity (usually implemented as a single Integer). Hmm. That sounds worryingly bad, given what I wrote above. One identity cannot – by definition – encompass multiple, independent, interrelated identities.

But we’re being a bit too literal here. ES’s give you one PRIMARY identity, but they also give you a bunch of SECONDARY identities. So, in practice…

Secondary Identities in an ES

In OOP, the Object is atomic, and the Class is atomic. You cannot “split” an Object, nor a Class, without re-defining it (usually: re-compile).

In ES, the Entity is atomic, and the Component is atomic. But the equivalent of an OOP Object – i.e. “an Entity plus zero or more Components” – is *not* atomic. It can be split.

And from there comes the secondary identities…

A Primary Identity: e.g. “The Player’s Tank” (specific)
A Primary Identity: e.g. “a Gun Component” (generic)

A Secondary Identity: e.g. “The Gun component … of the Player’s Tank Entity” (specific)

Revisiting my ad-hoc list of Game Identities above, I hope it’s clear that you can easily re-write most of those in terms of secondary identity.

And – bonus! – suddenly the relationships between them start to become (a little) clearer and cleaner. Easier for humans to reason about. Easier *for you to debug*. Easier *for you to design new features*.

Global Identity vs. Local Identity

Noticeably, the network-related Identities are still hard to deal with.

On *my* computer, I can’t reference entities on *your* computer. I cannot store: “The Gun component … of YOUR player’s tank”, because your “Player’s Tank” only exists in the memory of your computer – not mine.

There are (trivially) obvious solutions / approaches here, not least: make your Entity integers global. e.g. split the 64bit Integer into 2 32bit Integers: first Integer is the computer that an Entity lives on, the second is the local Entity Integer. Combined, they are a “global Entity ID”.

(I’m grossly over-simplifying there – if you’re interested in this, google for “globally unique identifiers” – the problems and solutions have been around for decades. Don’t re-invent the wheel)

But … at this point, they also offer you the chance to consider your game’s network architecture. Are you peer-to-peer, or client-server?

For instance, P2P architectures practically beg for unique Global entity numbers. But C/S architectures can happily live off non-global. For instance:

  • On each client, there are ONLY local Entity numbers
  • When the client receives data from the server, it generates new, local, Entities
  • …and adds a “ServerGenerated” component to each one, so it’s easy to see that they are “special” in some ways. That component could hold info like “the time in milliseconds that we last received an update on this object” – which is very useful for doing dead-reckoning, to make your remote objects appear to move smoothly on the local screen
  • The server *does* partition all entities from all machines. But none of the clients need to know that

Or, to take it further, if your network arch is any good at all for high-paced gaming:

  • The server differentiates between:
    1. The entity that the game-rules are operating on
    2. The entity that client 1 *believes* is current
    3. …ditto for client 2, client 3 … etc (each has their own one)
    4. The entity that the game-rules accept (e.g. if a hacked client has injected false info, the game-rules may override / rewrite some data in the local object)
  • The server also tags all the entities for a single in-game object as being “perspectives on the same thing”, so that it can keep them in synch with each other
  • The server does Clever Stuff, e.g.:
    • Every 2 milliseconds, it looks at the “current entity”, and compares it to the “client’s belief of that entity”. If it finds any differences, it sends a network message to the client, telling it that “you’re wrong, my friend: that entity’s components have changed their data. They are now X, Y and Z”

… or something like that. Again, I’m grossly over-simplifying – if you want to write decent network code, Google is your friend. But the fastest / lowest latency multiplayer code tends to work something like that.

How?

Ah, well.

What do you think?

(hint: you can do wonders using Reflection/Introspection on your entity / components. By their nature, they’re easy to write generic code for.

But you WILL need some extra metadata – to the extent that you may wish to ‘upgrade’ your Entity System into a SuperEntity System – something with a bit more expressive power, to handle the concept of multiple simultaneous *different* versions of the same Entity. Ouch)

Yeah, I’m bailing on you here. Too little time to write much right now – and it’s been a *long* time since I’ve implemented this level of network code for an ES. So, I’m going to have to think hard about it, refresh my memory, re-think what I think I knew. Will take some time…

Categories
entity systems games design MMOG development programming

Entity Systems: integrating Box2D with Artemis

Thanks to Mike Leahy for spotting this:

http://blog.gemserk.com/2012/02/02/how-we-use-box2d-with-artemis/

…a short blog post (with code) on how a team is integrating Box2D (a very well known open source physics lib) with Artemis (a java implementation of Entity Systems which starts from the same point as my Entity Systems posts, but fleshes it out)

Categories
computer games design games design network programming

Realm of the Mad God – a great game, interesting monetization

I’ve recently been playing the excellent Realm of the Mad God – a very fast-paced 2d co-operative shooter. My feeling is that it’s going to be one of the most important games of 2011/2012, as it continues to grow in popularity. Typical experience of this game is that within 30 seconds of being dumped into the main level, you’re surrounded by monsters, and then surrounded by other players, all on the same screen as you, blasting away in a rainbow of colours.

Sounds good. As if that’s not enough … it’s the guys who’ve been working with AmitP (Amit J Patel) (Amit maintains one of the best up-to-date collections of links and algorithms for indie game-developers). If you haven’t seen Amit’s pages, I recommend browsing through the blog – his links collection is OK, but his blog posts on algorithm design are excellent. If you get as far as the posts on “how I auto-generated a realistic 3D-world”, you may notice a striking similarity to the 2D worlds used in RotMG …

Anyway, it parallels an idea for an MMO shooter I’ve had kicking around for a long, long time. For me, it’s been a delight to see what works (and doesn’t) about the core ideas. The RotMG authors have done a great job of making a fast, simple, quick, easy-to-grasp game.

The Good

Fast. No barriers to play

This is how MMO-shooters should be: fast, furious, permadeath – but very quick to get back into the fray. You should expect to die tens of times every hour.

Permadeath – but paralled with some perma-advancement

Your progress is split evenly between items (which can be banked) and avatar stats (which are lost forever upon death).

Perma-advancement increases variety, unlocks new features

With 10 classes, there’s plenty of variety – and each class can only be unlocked by achieving a minimum level of progress with one or more other classes.

NB: this part could be improved and expanded IMHO. In particular, the classes are wonderfully varied – but merely unlocking classes isn’t enough these days. Plenty of games have shown that permanent-unlocks work best when there’s a variety of game-features in there. Also, the classes themselves would work better if there were some cross-effects (c.f. Diablo 2’s Lord of Destruction expansion, which had abilities in one class improve abilities in your older classes, re-vitalizing them for re-play)

Free game, paying is optional – payment kicks in when you’re most bought-in to the design

Free players get a tiny storage for permanent items (1/50th of what paying players get – it’s not enough! … so pay!), and are only allowed 1 class “alive” at once.

In a delicious twist, if you don’t pay for the game, the only way to take advantage of a newly-unlocked class is … to commit suicide … since you’re only allowed a single character per account (unless you pay)

You can ONLY benefit from other players, never suffer

(there’s actually a case where you can suffer, sadly – Thief’s get killed as a secondary-effect of other players teleporting to them, since the game doesn’t have a “prevent people from teleporting to me” flag)

This is the one that should have most wannabe-MMO-designers sitting up and paying attention. If you group-up with other players:

  1. You all get the same experience-points as if you’d single-handedly killed every monster
  2. You get the points just for being nearby – no need to score hits just to “tag” it for yourself
  3. Mob strength is constant, but player damage is multiplicative on number of players present

Net effect:

Every player is willing and eager (*) to collaborate with every other player, without words being exchanged, without fear of being ripped-off.

In a game that’s fast paced and frantic, you don’t have to keep pausing to negotiate. Other players can ONLY benefit you, so … run with them.

(*) – or just ignores the other players. Their presence doesn’t provide negative impact on you. It’s only their absence that is negative (in game-design terms).

Interesting design choices for lag

As a real-time game with dozens of players on screen at once, lag is guaranteed to effect gameplay. We’re always saying “try to work around lag through game-design changes”, so here’s the decisions they made:

  1. When packets are lost, everything moves in exactly the same direction it was going, at exactly the same speed, forever
  2. “Speed” used above is the “on-screen speed, including any rubber-banding effect”
    • FAIL: this means monsters and players often move MANY times faster than they are allowed to – so that when the lag stops, the side-effects are magnified
  3. Your avatar can’t be hit NOR damaged while it’s missing packets from the server
    • For the early parts of the game, this *almost* completely fixes lag problems
  4. Projectiles (bullets etc) that your client didn’t receive are queued up and sent to you all in one go once the packet-dropping stops
    • FAIL: this multiplies the damage output of enemies (NB: not players!), breaking all the designed-in balance in the game
    • In mid to late game, this ruins the gameplay – players end up running around never seeing a single enemy, because if you’re close enough to see it, a single flicker of lag will cause you to receive ZERO damage initially, followed by MORE damage than the monster is capable of – delivered instantaneously
  5. The client is authoritative on player liveness/death
    • MILD FAIL: in effect, coupled with the other features of the game, and the lack of lag compensation … this means you CAN and SHOULD (and, for some cases, effectively: MUST) cheat. You can run a bot on your machine – and if the network is less than perfect, you have to, in order to play the game properly.

Overall, apart from the massive security flaw (where anyone could write a bot to be invulnerable – and the developers are encouraging them to do so), it seems very close to a good solution for an MMO shooter.

I’m surprised by the way they approached the “save up the enemy bullets, then unleash them all at once”. My guess is that it wasn’t designed, it was just an accident: maybe they took a slightly lazy approach to compensating for lag (they don’t), and the net effect is this. It looks very much like what you get when using TCP for game-data packets (I really hope they’re not using TCP; if so, most of the lag is the developers’ own fault)

The Bad

I’ve unlocked half the classes, and looked at what classes other people play (and which classes rise high on the leaderboards). There’s good variety, and almost all the classes get used – even the beginning class, the one you get for free, works well.

Except one.

Unfortunately, at around 50% through unlocking the special classes, one of the classes is horribly unbalanced. The Assassin (which is supposedly an upgraded Thief – but is a massive downgrade) is almost impossible to play. The special ability fails completely when there’s lag (which is frequent in this game), and the class is the weakest, lowest-range of the lot. Looking around, you rarely see Assassins (I suspect: you only see them when people are desperately trying to upgrade them, to get past this dull and frustrating point in the upgrade tree).

Worse, because the *only* way to unlock the higher level classes is to reach the level cap with this class … you’re forced to play it. Over and over again. Watching the bad game-design … over and over … and over … again.

Every time the assassin dies, it’s like another twist of the knife:

We know you don’t enjoy this

We know that a mistake in our game design has you stuck here

(and our overall game design makes that “mistake” into “a disaster”)

We know that this whole process is turning “a class that wasn’t much fun” into “a class you hate”

And there’s nothing you can do about it!

So, single-handedly, it’s driven me to *not* purchase any game credit. I’d enjoyed the game enough to that point that I’d already decided to buy it – and if this had been on iTunes, I’d have paid already. But since it’s not an iPhone/iPad game, and paying for it is a bit more difficult, I hadn’t made the payment yet.

As it stands, I’m still playing occasionally, but now it’s for research rather than for enjoyment, which is a great pity.

Monetization: money thrown away

I think the developers are missing an ENORMOUSLY successful way to make money from this game. In fact, it’s so big, I suspect they could increase their revenues by a substantial multiplier.

With a permadeath game, there really is no need to actually delete the dead character. If the player isn’t paying, they are forced to kill their character sooner or later to change characters.

Taking a leaf from Flickr’s book, why not keep ONE single character in storage, with a tempting “buy now, for goodies *and to have this one returned to you, ready to play*”?

I’d set it so that when you change class (if you’re a free player), only the last character of the PREVIOUS class is retained. If you switch from Warrior to Knight, you can die many times as a Knight, and your Warrior remains on ice. But if you then grow tired of the Knight and switch to a Rogue … the Warrior is tipped out, and replaced with the last Knight you had.

i.e. you setup the exact flow of decision-making and options and “safety” that the player would have had, if only they’d purchased sooner, and allow them to benefit from it retroactively – if only they make the decision to pay.

Of course, it’s a very limited “retroactively” – it’s a sampler, to let you see the benefits of paying.

(*) – Flickr’s early promise was “upload your photos in highest resolution, you can view them for free – but only low-res versions. HOWEVER, we keep the high-res versions for you – forever, for free – until you decide to purchase a subscription. At which point, not just your new photos, but ALL your photos, become magically available at highest res”. It was a great way of simultaneously offering a high-value to paying customers, while making non-paying customers feel they weren’t committing themselves to loss. It reassured a lot of potential customers at a time when Flickr wasn’t yet famous, and most people weren’t yet “hooked” – it bridged that gap.

Analyse this

So, the interesting question is: how common is this problem?

Are the dev team correlating “players who pay” and “the point at which they pay”?

More importantly, are they correlating “players who DON’T pay” and “how their experience differed from the average”?

The last time I saw an MMO with a level-based kick in the teeth this bad … was in Tabula Rasa. We had a point where poor signposting by the quest designers meant many players were given quests that were many levels too hard for them, and effectively impossible to do. Those players died over and over and over again in a short time – and they hated it.

The dev team knew “you’re not supposed to do that quest”, but often they (randomly) gave it to new players as the first quest. I wasn’t privy to the arguments over whether this needed to be changed (and there were definitely arguments), but I did see the analytics that eventually got produced. They showed an almost perfectly smooth, averaged, graph of player behaviour – bar a big notch at this particular location. It stuck out like Rudolph’s nose on a snowy day.

I wonder if there’s a similar notch in RotMG? For a game that’s almost *designed* to drive people to rage-quit … what stats do they see on “what the last thing was before a player stopped playing forever?” … and what stats do they see on “…stopped playing for a long time, but eventually came back”?

Categories
amusing MMOG development network programming programming system architecture

Mongo DB is WebScale. MySQL is not WebScale.

There’s good reasons for adopting Mongo, I’m unconvinced (but open-minded) that performance is one of them. Here’s a ROFLMAO viewpoint on it:

“If your write fails, you’re ****ed”

Obviously, MySQL’s not perfect, but in most cases I’ve seen, it’s been lack of competence on the developer side, and the lack of basic DBA skills – not problems with MySQL itself – that’s broken scalability. In which case, I’m a little suspicious that a company that fails to scale MySQL will equally fail to write their code correctly on Mongo. In many ways, throwing away SQL makes it much easier to prevent scalability…

Categories
amusing funny reputation systems security

DRM and the consumer

The mashup

(My new “preferred explanation of piracy + DRM”)

The original

(From Cyanide&Happiness webcomic, if you don’t know it already..)

Categories
entity systems MMOG development

Using an Entity System with jMonkeyEngine (mythruna)

If you’re interested in using an ES on indie projects, and you’re craving concrete examples, you might want to look at the comments (page 1 and page 2) on Mythruna:

“Since this is a developers forum, I’ll describe a little bit about what went on behind the scenes.

During the early development for this phase, I read about an architectural pattern called an entity system.

I had (pre Entity Systems) a plan for how I was going to store the placement of objects in the world but this past weekend when I actually got to implementing it, I couldn’t make that original plan work and came to the point where I needed to solve the general problem of world state storage. This is the kind of thing that Entity Systems make relatively straight forward… and by Saturday, Mythruna had an embedded SQL database that autogenerates tables based on the components stored. (HyperSQLdb for the win.)

So at some point I swapped out the in-memory version of the entity system with the SQL version… and suddenly objects were persistent. It was so easy I had to double check a few different ways that it was actually working. :)”

…but of course, also: use the Entity Systems wiki – put your questions there, put your ideas there, and (most of all) if you have an ES project or ES source code you want to share, please add it to the wiki!

Categories
computer games design games design

Richard Bartle: 10 Games you Should Have Played

Normally, I don’t allow “guest posts”, but I’m making an exception for my “10 Games You Should Have Played” series. I’ve been asking other games-industry people to write up their own lists + explanations, and that’s not always compatible with their personal/work/etc blog. When that happens, I’m happy to post them here instead.

So, here’s Richard Bartle‘s take (“co-creator of MUD1 (the first MUD) and the author of the seminal Designing Virtual Worlds” – but if you read this blog, you should already know who he is ;).

I think it’s a great list. I asked him to define in his own way what he meant by “should” (why are we saying “should”? who’s the audience? etc) and to run with it, which he did …

Come up with your own rules for a top-10, define it clearly, and share your list.

“OK, well modulo all the usual complaints about lists of 10, here we go.

I don’t have any rules per se, but I am sort of assuming that this is for people who play games or design games or want to know more about games.

Also, I’m going to go with categories rather than individual games (except in the last case). This is because it’s not the games themselves that are necessarily important so much as what you get from playing them. I will, however, give an example of a game in each category that I myself have tried.

1) A game you have bought but haven’t played yet.

You should always have a game ready to play. I don’t care what it is, but unless there is one you’re never going to expand your gaming horizons.

For me, right now that game is “Victoria II”, which I’ve installed and read the manual for but haven’t actually started to play. The reason I haven’t started it is because I bought “Mount & Blade: with Fire and Sword” and snuck that in front of it in the queue. Once I do start it, I’ll be looking for another game to play when it’s completed – hopefully not another damned sequel…

2) An abstract game.

Games can be many things, but unless they have gameplay they’re not games. An abstract game only has gameplay. To understand games, whether to design, play, or study them, you need to understand gameplay; an abstract game shows you the game mechanics with everything else stripped away.

You need to play one. You may, if you’re keen, try think of a skin for it, but that’s not essential.

In my case, I guess the game would be “Chess”. I captained my primary school Chess Club, but my interest in the game waned when I realised that the openings were always the same and that people who were less good at the fun, thinking part could win by doing the boring, memorise-the-openings part. That came straight from an appraisal of the clear-for-all-to-see mechanics.

That said, I’d also like to give a shout-out to the altogether more obscure “Besikovitch’s Game”. Now that’s a mechanic with potential…

3) A tabletop role-playing game.

Everyone thinks they know why they want to play games, but they also need to know why everyone else plays games. They’re not going to get that unless they understand what it means to be part of the game. In a tabletop role-playing game, with the other players right there next to you, there’s no escape: you have to participate, you have to involve yourself, you have to become part of the game, part of the narrative. In short, you have to live the game. Unless you’ve lived a game, how can you ever hope to understand what’s gamingly possible?

For me, the hours I spent playing “D&D” with my friends in my late teens were some of the best gaming experiences I ever hard. I wish I’d been able to get a “Call of Cthulhu” group going, mind you, but it came out too late for me.

4) A spectator sport.

If a game is good enough that people will pay to watch it played, you need to understand what it’s like to play it. This gives you an insight into the theatrical aspects of games that you wouldn’t easily get from merely observing the performance. You don’t actually have to be any good at the game, and the game itself doesn’t have to be all that good either (in my case, “Snooker” fits both those categories); the important thing is to understand what gives a game presence. I don’t care whether it’s high
skill, clever strategy, viscerality, physicality – if you don’t play it, you won’t appreciate it.

In my own case, I played “Association Football” (yeah, soccer) at school (attacking midfielder if you must know); I was good, but we were never taught any skills or anything and most games descended into kicking matches. I nevertheless found out what made it “the beautiful game”, though.

5) A game in which you can lose actual money.

There is a dark side to games, and gambling gives people a chance to sense it. Personally, I don’t like playing games for money at all; however, a lot of people love it. Everyone has their limits, though.

For some, gambling games are at their best when the amounts involved actually hurt if you lose them; for others, it’s the amounts that can be won that make the difference. The point of playing a gambling game from the perspective of this list is to gain an appreciation of the morality of games. When something stops being “just a game” and starts to take over the player’s life, that’s potentially a bad thing. Unless
you’ve seen it (or something close to it), you’re never going to understand that fully. Gambling games let you do that. Warning: you run a big risk with this if it turns out you’re the one who gets hooked…

For me, I used to play “Poker” with my friends over lunch when I was 17 or 18. We played for Tic-Tac mints. This was before “Texas Hold ‘Em” got big, so we’d play mainly “Draw Poker”, “5-Card Stud”, “7-Card Stud” or, occasionally, “Montana Red Dog”. We stopped playing when one of my friends, who consistently lost, had to borrow money to buy more Tic-Tacs; I decided things had gone far enough, and called
the lunchtime sessions off. From that point on, no way would I design a game that deliberately tried to addict someone to it.

6) A game released in the year you were born.

Most games are built on the foundations of games that went before them, and an appreciation of their history means you appreciate the games themselves more. Games have a very long history (indeed, they go back into prehistory), but a modern game is unlikely to quote directly from ancient archetypes. They’re more probably going to quote from games from the generation before them. You therefore need to
play a bunch of old games to see where the advances were made. Unfortunately, “old” is a relative term: what you think is old might, to me, seem fairly new. What’s old enough for both of us is something from the year we were born in (or a year close to that). Play a game from back then and see how things have (or haven’t) changed. Bonus: you’re almost guaranteed to notice the gameplay more than you do in a (what currently looks) slick, modern game.

For me, the old game would be “Diplomacy”, which was released commercially in 1959 (the year before my birth, but that’s near enough). Ah, what a game! It’s trapped in its time, because it needs 7 players and could only really be played by post. Play-by-email is even more of a niche than play-by-mail was, so it’s not a game that is played a lot nowadays. Lovely mechanics, though!

7) A really bad game.

Some games are just BAD. The mechanics are all wrong, they’re unfun, or no fun, or the rules are ambiguous, or they drag on and on, or there’s a dominant strategy, or … well, the list continues. If you play such a game, you can ascertain what it is that’s bad about it; this will enable you to avoid similar games in future and to avoid
making similar mistakes in any games you design yourself (see next point). The more you understand about games, the more you’ll be able to find the games that are right for you.

For me, tempting though it is to nominate “Trivial Pursuit” as the game that laid waste to the British board games industry, I didn’t actually play that. However, my personal pick is one that I’m sure many other people will share, too: “Monopoly”…

8) A game you wrote yourself that no-one else has played.

Game design is actually quite hard to do well. You’re not going to know quite how hard unless you try it yourself. In the attempt, you’ll come to understand more about games and what makes them tick – but only if you actually play the game (if
it needs more than one player, play it against yourself). If you actually are a game designer, this is something you will have done many, many times before, of course; just make sure you keep on doing it.

For me, well, all game designers have a corpus of games in various stages of completion that they have never shared with anyone else, simply because doing the design itself was the fun part.

I’m a bit low on computer games in this list, so I’ll go for one I did called “Mombasa” about the exploration of Africa. It’s not all that good, but the point is that I wouldn’t know that if I hadn’t played it…

9) An MMORPG.

This is because I co-wrote the first virtual world, and therefore the more people who play these, the higher my kudos rises.

I’ll list the last MMO I played through up to the level cap as my example here: “Rift”. I came away not so much impressed by the game itself but by its developer, Trion Worlds, which is more understanding of its players than any other developer I’ve
come across except perhaps CCP.

10) “Mornington Crescent”.

It’s actually called “Finchley Central”, but I’ll go with the version that’s best known. This is a very simple game, the rules of which, in their entirety, are as follows: players take it in turns to name London Underground stations, and the first to say
Mornington Crescent wins. This is a game everyone should play, because it gets to the heart of what a game is: what happens when you freely and knowingly bound your behaviour according to a set of rules in the hope of gaining some benefit that you might not get. You play it for just so long as it’s fun, with people who also play for just so long as it’s fun. It’s the Magic Circle incarnate.

So those are my top 10 games that people should play. If you already played them, my apologies for having wasted your time with this list. If you haven’t played them, I envy you the treasure trove that lies ahead.”

Categories
design games design games publishing reputation systems

“by running a spy network I am griefing”

If you’re an MMO designer, and you *still* don’t grok the griefer-mindset, or you somehow hope/believe that “one day, there will be no griefers”, then maybe this RPS interview with the always-fun-to-watch Goonswarm will help you:

MT: We are griefers. If nothing is going to happen then we’re going to try to find something that screams and bleeds and poke at it.

RPS: Griefing is something goons are known for doing, but now I’m talking to you it’s not something I can imagine you personally doing.

MT: Technically speaking, by running a spy network I am griefing.

RPS: But would you go out and aggravate other players for the Hell of it if you were a lower ranking member of Goonswarm?

MT: Well, most lower ranked Goons make their money by doing that. Scamming people is a very quick way of making money in Eve. Rather than making an honest buck, you take that buck from somebody else.

and, much further down, maybe this will help you see how griefers often serve just as positive and valuable a role as all your “preferred” player-types:

RPS: For my money, Eve might be the most fascinating game in existence today. But that doesn’t stop it from being interminably boring as well.

MT: Right. I mean most Eve players are stuck in high security space mining, and a lot of the core PvE in Eve has you sitting there are watching three grey bars slowly turn red.

Goonfleet is a socialist alliance. We give people ships so that rather than being forced to rat [fight low-powered AI NPCs] they can take part in PvP, we teach them how to scam so that they don’t have to mine, we teach them how to make ISK most effectively, we give them a lot of ISK and we reimburse their losses. This way they can focus on the fun aspects of the game, like griefing and warfare, so they’re not forced to endure derp-derp-ing around high sec.

If they play your game, you should be glad; if they grief, you should be asking yourself why – and if you’re a commercial operation, you should probably be asking:

“are they fixing a problem for us?

can we afford to leave them to it, part of our unpaid workforce?

and:

is it worth our time trying to fix the problem itself, or should we accept their help and move on down our never-ending list of pending fixes?”

Categories
design entity systems MMOG development network programming programming

Entity Systems: updates to source code

I’ve just done a round of fixes for the source-examples of ES’s. Github projects updated on this page:

http://entity-systems.wikidot.com/rdbms-with-code-in-systems

Changed:

  1. Added a complete Java implementation of the most basic ES example
  2. Fixed some minor bugs in the Objective-C basic ES example; added some missing classes
  3. Added a missing class method from the documentation (System.
Categories
dev-process games industry massively multiplayer network programming networking programming recruiting server admin

The nature of a Tech Director in games … and the evils of DevOps

Spotted this (the notion “DevOps”) courtesy of Matthew Weigel, a term I’d fortunately missed-out on.

It seems to come down to: Software Developers (programmers who write apps that a company sells) and Ops people (sysadmins who manage servers) don’t talk enough and don’t respect each other; this cause problems when they need to work together. Good start.

But I was feeling a gut feel of “you’ve spotted a problem, but this is a real ugly way to solve it”, and feeling guilty for thinking that, when I got down to this line in Wikipedia’s article:

“Developers apply configuration changes manually to their workstations and do not document each necessary step”

WTF? What kind of amateur morons are you hiring as “developers”? Your problem here is *nothing* to do with “DevOps” – it’s that you have a hiring manager (maybe your CTO / Tech Director?) who’s been promoted way above their competency and is allowing people to do the kind of practices that would get them fired from many of the good programming teams.

Fix the right problem, guys :).

Incidentally – and this will be a long tangent about the nature of a TD / Tech Director – … my “gut feel” negativity about the whole thing came from my experience that any TD working in large-scale “online” games *must be* a qualified SysAdmin. If they’re not, they’re not a TD – they’re a technical developer who hasn’t (yet) enough experience to be elevated to a TD role; they are incapable (through no fault of their own – simply lack of training / experience) of fulfilling the essential needs of a TD. They cannot provide the over-arching technical caretaking, because they don’t understand one enormous chunk of the problem.

I say this from personal experience in MMO dev, where people with no sysadmin experience stuck out like a sore thumb. Many network programmers on game-teams had no sysadmin experience (which in the long term is unforgivable – any network coder should be urgently scrambling to learn + practice sysadmin as fast as they can, since it’s essential to so much of the code they write) – and it showed, every time. In the short term, of course, a network coder may be 4 months away from having practiced enough sysadmin. In the medium term, maybe they’ve done “some” but not enough to be an expert on it – normally they’re fine, but sometimes they make a stupid mistake (e.g. being unaware of just how much memcached can do for you).

And that’s where the TD-who-knows-sysadmin is needed. Just like the TD is supposed to do in all situations – be the shallow expert of many trades, able to hilight problems no-one else has noticed, or use their usually out-dated yet still useful experience to suggest old ways of solving new problems that current methods fail to fix. And at least be able to point people in the right direction.

…but, of course, I was once (long ago) trained in this at IBM, and later spent many years in hardcore sysadmin both paid and unpaid (at the most extreme, tracking and logging bugs against the linux kernel) so I’m biased. But I’ve found it enormously helpful in MMO development that I know exactly how these servers will *actually* run – and the many tricks available to shortcut weeks or months of code that you don’t have to write.

Categories
design marketing security

Identity theft, exploitation, and Gravatar

There’s a growing problem right now with Facebook Connect: it can silently log you in to websites that you *don’t want* to share your private data with. I saw a funny example last month where a porn website had integrated Facebook Connect … so when you visit the site, one miss-click and you’ll broadcast to all your work colleagues your embarassing love of HardCoreGrannies.

But there’s another example right now that may be worse, and is definitely food for thought. Facebook doesn’t broadcast your data – not to protect your privacy, but to prevent competitors getting access to data they are currently making money out of themselves. By contrast, there’s Gravatar: these guys take your private data and give it away to everyone – and they refuse to stop doing it (I’ve asked, directly, and they refused. They had no reason to refuse – they knew my identity, they knew my request was valid, and I believe under UK / Europe law it would be *illegal* for them to refuse. But … they’re American, and I guess all they care about is money).

So, for instance, I just had one of my online identities ruined by Gravatar. A website that I rarely use recently “upgraded” and implemented the gravatar system – and immediately took a private account and publically broadcast that I was the owner. They didn’t ask me, they just went ahead and did it. Like many web developers, I’m sure they had no idea what they were doing – few seem to be aware of the scam that underlies Gravatar.

Fortunately, I’m not going to lose something massively important, like my job / marriage / life (c.f. the news stories when Google Wave launched), but the website owners had no way of knowing that. They’ve just unleashed this upon their hundreds of thousands of users; what are the chances that one of them will be affected?

(incidentally, if you’re a website owner, I strongly recommend you think twice before adding Gravatar (or any of the clones) to your own site. I don’t know if anyone’s been sued for it yet, but I’m sure it’ll happen eventually)

There are two halves to the problem. Gravatar is fundamentally a violation of privacy: they take your data and give it to *everyone* without you knowing. So what? That’s the whole point of the service! Yes, the Gravatar author is a little incompetent (c.f. OpenID for how he *should* have implemented it), but otherwise there’s no problem, is there? In theory … if you voluntarily sign-up for it, it’s all OK. Isn’t it?

Well … maybe not. They won’t let you (the user / owner) control that flow of data. What happens if you change your mind – can you delete their data? Nope. Why? I’m not sure, but I would guess: If you did that, you’d undermine their ability to make $$$ out of you. You can (theoretically) set your pictures back to empty. But …

…But there’s a second half to this. I believe most people are on Gravatar because WordPress “gave” the user’s private data to Gravatar. That’s a nasty mess right there; what does WordPress’s privacy policy say? Again, when they acquired Gravatar, they apparently didn’t ask their users what they wanted, they just forced this privacy violation on them. Back then, it didn’t have much effect (Gravatar itself was relatively unknown / little used), but as Gravatar gets used more widely, the problem becomes more acute.

And here’s the rub: Gravatar’s staff refuse to adhere to privacy requests because (precising / summarising): “you have to use your wordpress.com account”. What if you don’t have one? “you must have had one in the past and we won’t help you. Go away, and stop bothering us”.

Meanwhile, WordPress refuses to send password details to anyone, ever. A wise security decision in some ways (e.g. many people use the same password on multiple sites. Doh!). Your only choice is to delete the password and recreate it.

Is that a problem? Sadly, yes. Because (due to some very short-sighted / stupid marketing decisions by the WP folks) there are lots of admin systems – e.g. anti-spam – that are run off people’s WordPress accounts. So far as I can tell, no reason exists for this *except* to harvest email addresses and try and lure people onto paid WordPress.com plans. Further, WordPress uses an archaic password-based system (instead of e.g. Yahoo’s permission-based API – which, again, is how WP should have implemented this) – so if you change your password, all those websites will break.

Summary

These services are a nice idea in theory, but when you get terrible implementations like Gravatar, combined with lazy / stupid staff, the user does pretty badly. They get screwed, they get patronised (just look at the Gravatar.com FAQ; they’ve cleaned it up in the last 12 months, it’s no longer so actively offensive as it used to be, but it’s still pretty bad), and many times they don’t even know about it until the violation is widespread.

And, ultimately, any website that uses this system is in danger of losing badly if it goes to a court-case. I’m not a lawyer, but when there are industry standards for user-controlled privacy (OpenID), and specific laws demanding that Gravatar honour the requests it currently refuses (UK Data Protection Act, for instance), I suspect a court is unlikely to look favourably on a website claiming innocence. Ignorance isn’t generally a valid legal defence.

But how much damage do these systems do to themselves? If Automattic were a little less greedy, or a little less selfish, would a lot more people embrace the idea of sharing their identity openly? Will OpenID provide a gravatar-replacement that doesn’t shaft the user, and will that take off much bigger than the original?

Personally, I look at recent events like Google Wave, and Blizzard’s “forum identity = credit-card name” – and the s***storm of angry users in both cases – and I suspect these privacy issues are much more damaging than corporates expect. Which is good news: the world appears to be slowly waking-up to the abuses inflicted upon them in the digital world, and the importance of keeping certain things (passwords, email addresses – and now, finally: identity) sacrosanct. And that is definitely a good thing…