MMOG development

Dropbox tech scaling – some great, some not-so-great

“I was in charge of scaling Dropbox … from roughly 4,000 to 40,000,000 users. … Here are some suggestions on scaling”

The first section is a WTF – the guy advocates deliberately over-taxing your servers, without a good explanation. I’ve got some guesses at why they did it – but in the general case I’d say: never do this. Only do it when it’s obvious, because you have a specific reason to do so (and you really know what you’re doing).

The rest is a lot clearer, good advice there.

Also, IMHO worth reading for this part alone:

“Let’s say you’re trying to debug something in your webserver, and you want to know if maybe there’s been a spike of activity recently and all you have are the logs. Having graphing for that webserver would be great,

[ADAM: but … often you don’t have the right set of graphs set up, and it takes a while to do that – no use if the server is in trouble *right now*]

Apr 8 2012 14:33:59 POST …
Apr 8 2012 14:34:00 GET …
Apr 8 2012 14:34:00 GET …
Apr 8 2012 14:34:01 POST …

You could use your shell like this:

cut -d’ ’ -f1-4 log.txt | xargs -L1 -I_ date +%s -d_ | uniq -c | (echo “plot ‘-’ using 2:1 with lines”; cat) | gnuplot

Boom! Very quickly you have a nice graph of what’s going on, and you can tailor it easily (select only one URL, certain time frames, change to a histogram, etc.).”