Categories
advocacy computer games Dare 2 Be Digital education games design games industry GamesThatTeach iphone programming

Teenagers learn to program, write own 3D game, in 3 weeks

Background

Last year, Pearson ran the first ever Innov8 competition, giving tech startups a chance to make their own innovative new product/project. The grand prize was £5,000 towards building the product.

Most of the teams were adults (even: real companies), but a team of students from Blatchington Mill School won, with their idea for an iPhone/iPad app: “My Science Lab”.

Team: Quantum Games

The three students named themselves “Quantum Games”: Jon, Nick, and Oli. All three of them have been studying for their GCSE’s in parallel with this project.

They’ve been supported by Mark Leighton, Assistant Head / ICT Director at the school.

For mentoring and game-development expertise, they had me – Adam Martin – previously CTO at MindCandy and NCsoft Europe, now an iPhone/Android developer

Previously

The students chose to focus on a game that would help other students revise the “Momentum” part of GCSE Physics.

In summer/autumn 2012, they learnt the basics of game design and development. We didn’t do any formal teaching – they simply had to pick up the skills they needed as we went along. YouTube videos, and “trial and error”, were our primary techniques…

In particular, they learnt 3D-modelling and texturing (using Wings3D and Photoshop) and game-programming (using Unity3D and Javascript).

By the end of 2012, they’d written their own physics engine, some basic gameplay, and a simple simulation of an exercise/problem in Momentum.

Last month

The big thing this month has been BETT. Pearson had a large stand, and asked the students along to talk about the project. They gave an excellent presentation to an audience of approx 30 people at BETT, covering the background and some of the things that went well, that didn’t, and what they’d learnt from it.

Leading up to BETT, they worked hard to squeeze in a new build of the game, with a rethink on the interactive sections and how they hang together. Unfortunately, we hit what seemed to be a major bug in Unity’s camera-handling, and none of us could fix it in time (nor could we get an answer from Unity support in time). But the students managed to invent a workaround at the last minute which worked fine for demoing at BETT.

The game isn’t finished yet – GCSE’s and schoolwork left too little time to complete it before BETT – but we’re very close now. The students are aiming to finish it off this month and next, and I’m hoping I might even be able to take a copy to the GDC conference in March (taking place in San Francisco, GDC is the commercial games industry’s main annual conference).

In the meantime … you can sign up now on the Quantum Games website (http://quantumgames.co.uk), and we’ll email you as soon as the game is ready – or sooner, with a private beta-test!

Categories
amusing programming

Modem dialup noises … visualised (and transcribed)

Visual spectral diagram showing the noises modems make, and box-outs for what they’re actually saying:

…and Transcription for txtrs

Modem A: hey babe, you dtmf?
Modem B: u know it
Modem A: what u up 4 2nite? wanna v.8?
Modem B: i wanna ack u like my daddy net2phone use 2 ack me
Modem A: um ok… v.8 then
Modem B: lol jk, u comin?
Modem A: brt just gotta turn off echo suppressors n cancellers
Modem B: ok i wait
Modem B: my pcm is so modulated
Modem A: lol rly? u think u can handle V.90/V.92?
Modem B: D/A?
Modem A: …D?
Modem B: wtf no, im not into that
Modem A: lol jk we can do V.42 LAPM if u want im down 4 nething
Modem A: up to 3429 o/c
Modem A: u know i give as good as i get, ne way u want it, loud or soft, high or low, fast or slow, i got all the time in the world 4 u babe, my clock source is internal
Modem B: of course no 3429. and same 4 me. except i might lose track of time, lol
Modem B: and honey if u with me we gon be makin sum NOISE
Modem B: 6db at LEAST u know how i like it
Modem A: lol i hear ya, 3200 all nite long, the way u get me goin maybe we even go 2 4800 lol
Modem A: set ur pre-emphasis filter params n put on that 1920 hz carrier frequency i got u
Modem A: im here baby
[SCRAMBLED]

Categories
fixing your desktop games design games publishing programming

Unity crash: “type == kMetaAssetType && pathName.find (“library/metadata”) != 0″

Unity’s QA dept needs a smack up-side the head. If you get this bug, you’re a bit screwed – the only “fix” is to go into system settings and wipe Unity’s crud.

Error message:

type == kMetaAssetType && pathName.find (“library/metadata”) != 0

Thanks, Unity, that’s very human-readable and helpful. Not.

(appears to be an Assert written by a junior programmer – or a debug-only Assert that got left in the shipping version(!))\

Problem:

  1. Unity’s tech team wrote Unity 3 so that it would open projects BEFORE checking if there was a new version available
  2. Unity 4 is NOT backwards compatible; it corrupts projects so that they will NEVER load in Unity 3.x
  3. …also: Unity 3.5 will FATAL CRASH if it tries to open a Unity 4 project
  4. Unity (generally) is badly written and auto-opens the last project, even if you don’t want to, and EVEN IF ITS CORRUPT
  5. Unity (generally) DOESN’T BOTHER to offer any way of starting “safely”

Solution (OS X):

  1. Force-quit Unity (not only does it crash, but it hangs forever) – RMB on the icon, hold down alt/option, and the Force Quit option appears
  2. run Finder, go to (your user)/Library, aka “~/Library”. If you don’t know how, then do ONE OF the following (either works):
    • EITHER: Start at your username folder (it’s the parent folder of your Desktop folder, parent of your Documents folder, etc), un-break OS X (Apple ships it as broken-by-default): google for “enable see Library folder in Finder”, follow the instructions
    • OR: In Finder’s “Go” menu, select “Go to folder”, and type “~/Library”
      • Inside Library, find “Preferences” folder
      • Find *everything* that begins “com.unity3d.” and delete it (there are approx 20 files, of 2 different file types)
      • Re-start Unity, and it will open the default project
      • …And this time, Unity will offer to upgrade to Unity 4

UPDATE

For bonus points, the Unity 4 “upgrade” isn’t an upgrade: it’s a replacement. I expected it to download “the bits that have changed”. Nuh-uh. One gigabyte download! (try getting that in a hurry, when your project has suddenly imploded in front of you, because of the non-existent backwards-compatibility :( ).

Some idle thoughts…

I’ve seen similar behaviour before when writing very simple OS X apps. There’s some massive bugs in Apple’s code that have been around for 5+ years (and Apple shows no intention of fixing), which cause “startup actions” to happen in a semi-random order, depending on the NAME and NUMBER of recently-opened projects.

It takes very little testing to discover this. If it’s the problem with the Unity software, then Unity needs to seriously improve their testing on OS X – they should have discovered this easily.

But also … why on earth are they using the NSDocument loading system? Apple never finished writing the docs for it, they seem to have never finished implementing it (major bits of the API seem to be missing) – in general, OS X authors don’t seem to use it any more. Probably because it doesn’t work?

Categories
advocacy games industry GamesThatTeach programming web 2.0

Free art can help create a generation of non-pirates

The 21st century will be dominated by “digital” culture and art. History suggests that non-digital art will flourish too (while becoming a smaller, more specialized, part of a larger pie). So it’s all good: more people will have more opportunities to create – and more access to experience – a wider array of art. Win/win!

Except … our societies are struggling to work out how we’ll pay our artists when the marginal price of a copy is less than a penny.

Last week, something interesting happened when several unrelated projects I’m in all came together at once.

Someone is ‘stealing’ from CGTextures.com

Marcel at http://www.cgtextures.com/ gives away a huge library of high-res photo textures, aimed at game-developers, entirely for free. You don’t pay for access, you don’t pay to use them. You can include them in commercial games, make a million dollars – and you owe him nothing (bar gratitude).

Last week he came to a private forum asking for advice on suspected copyright infringers, who might have been taking his free images, removing the attribute/authorship info, and selling them for themselves.

Copying the images, and charging for them, is not theft. It’s illegal, but it’s not stealing. The original source is still available – free – to anyone who wants it. And many authors in this case are inclined to let the scummers go free, so long as they stop charging innocent users for something that’s free to all.

But CGTextures isn’t free to run; if they ever need to raise funds to pay for it, some of that money – which the community would happily donate – is being taken right now by a selfish scummer. Hmm. Tricky.

3D art is hard

If you’re “not an artist” (which for most people means: “I’m crap at drawing/painting!”) then making any kind of 2D art is very difficult, and tends to look like utter crap.

Computer games are dominated by good or great art. Even in the Indie scene, where “teams” are often no more than 2 people working together, we have a blinding array of beautiful artworks. At the opposite end of the spectrum – the AAA titles with budgets counted in “tens of millions” of dollars – it only gets better.

People love playing games – and they love making them too. Many people – artists and programmers – dream of “making a game”. But … just like “I’ve a great idea for a book” … the vast majority never manage it.

Two of the most common reasons they fail:

  1. Aiming too high: games require a lot more work than people imagine, and most people get 10% in and discover they’ve bitten off way too much to chew
  2. The artwork looks crap: everyone they show it to hates it (or they dont dare show it), the author hates it, they realise that no-one will play it, let alone pay for it, and they gradually lose the will to finish the project

Anyone can make a game: even un-trained teenagers

We’re in the final few weeks of proving this (a team of three 15-year-olds are about to publish their iPhone game that they designed, built, tested, and launched from scratch).

Starting with nothing but beginner-level knowledge of Javascript (not enough to write an app), they’ve:

  1. Learnt Javascript
  2. Learnt 3D-modelling
  3. Created all their own 3D models, with textures
  4. Built, tested, and refined a working game

Sounds hard, right? Well, yes, it was. But – if you know enough tricks of the trade – most of that can be made easy enough for anyone to do themselves.

  1. Game structure – use an established game engine
  2. Programming – stick to “simple” programming concepts
  3. In-game artwork – “stylised” 3D models are trivial to create (c.f. Minecraft)
  4. Testing – use a modern IDE with a decent debugger

This is all great, but I’ve glossed-over one item there: textures. You can avoid the need for painting skills by making your game-items 3D instead of 2D, but sooner or later you’re going to need to texture them.

JFGI

With the programming, one of the skills I’ve drummed into them is JFGI (Just F’ing Google It). Everytime you get stuck: google it. If you get no hits – fine, you’ll have to work it out yourself. But often you’ll find:

  1. It’s a bug in your tools, not your fault! Here’s a workaround…
  2. It’s practically impossible; don’t waste time trying to solve it…
  3. Your software documentation / manual was missing the following info: …
  4. It’s a generic boilerplate piece of code. Don’t worry about it, but use this copy/pasteable code solution: …

Leveraging the internet as a resource is fundamental to being a great programmer. I’ll gloss over the risks / dangers for now (I’ll write another post on that later), but most of the time you cannot JFGI too often.

But … with the 2D artwork, with the textures for 3D models … Google becomes a danger.

Google Images: the devil on your shoulder

Writing a presentation, and need an image? Google Images it!

…making a game, and need a “wood texture”? Google Images it!

WHOA THERE, JOHNNY!

Doesn’t feel like stealing (that’s cos it isn’t) – but it is something illegal: copyright infringement. It’s precisely why “copyright” was invented in the first place.

And yet: this single problem can make all your effort, all your hard work on your own creative artwork (your game), invalid. You can have the most sublime game design, a control system that a toddler can master, a frame-rate as smooth as silk … but if the 2D graphics (or the textures) are crap … the whole thing falls flat on its face. And most people can’t draw.

How the pros do it

There are simple techniques for making very good textures starting from random photographs. Even a novice can create something perfectly “good enough” in a short amount of time.

Only one thing is needed: a big library of photographs, MORE THAN ONE per real-world “texture” you need to create. If you have the money, there are dozens of Stock Photography resources, each one costing hundreds (or: thousands) of dollars a year.

But if you’re students – undergraduates, high-schoolers – or simply “not rich” (“artist” isnt’ exactly a high-paid career) and working on your own, you probably don’t have “hundreds of dollars”.

Hey, I know! Let’s use Google Imag- … crap.

Enter stage left: > http://www.cgtextures.com/ – a FREE, ROYALTY-FREE, MASSIVE collection of photographs DESIGNED FOR USING IN COMPUTER GAMES. Why? I guess Marcel is just a naturally generous person.

I showed the guys CGT. No problem; texture sources a-plenty. And it’s all free. And legal…

Full circle

  1. potential pirates who are ‘creating’ are happy to respect copyright, if you educate them early enough … so long as they have viable alternatives
  2. if you take away the alternatives, they must weigh up the moral “cost” of infringement against the moral “benefit” (and personal satisfaction) of completing their own work
    • I’m not advocating this piracy; but where no theft is involved, to most people’s minds the cost is tiny and the benefit is huge. Realistically I expected few people to resist when he temptations – both moral and practical – are so big
  3. sites like CGTextures put “artistic creation with 3D” in reach of everyone
  4. pirating art from CGTextures is – AFAICS – only a criminal activity: illegally extract money from someone else’s work, with no ‘creation’ involved
  5. …but if sites like CGTextures go away (if Marcel gives up), and the next generation of artists lose their alternatives, “copyright” has no chance at all

IMNSHO, anti-software-piracy organizations tend to be idiotic, amoral, and begging to be nuked from orbit. They’re often part of the problem, not the solution. If they genuinely wanted to reduce piracy, they should be creating sites like Marcel’s: royalty-free resources of reduced cost that their industries could easily afford to give away for free.

The debate has – for way too long – characterized software pirates as “inherently evil; bad-doers; malicious”. This is undoubtedly true of some (my opionions of anyone re-selling CGT’s free art are unprintable). But we’re not born as software-pirates; we get that way because of the culture and society we grow up in. We have the opportunity to teach new generations respect for copyright – but that cuts both ways.

In the Digital Age, copyright needs to deserve our respect, not simply demand it.

Some other free texture sites

While checking some of the points in this post, I noticed a few other photo-texture sites that offer royalty-free images suitable for games dev, worth checking out:

Categories
programming startup advice

The cost of interrupting someone … analysed

“Based on a analysis of 10,000 programming sessions recorded from 86 programmers using Eclipse and Visual Studio and a survey of 414 programmers (Parnin:10), we found:

  • A programmer takes between 10-15 minutes to start editing code after resuming work from an interruption.
  • When interrupted during an edit of a method, only 10% of times did a programmer resume work in less than a minute.
  • A programmer is likely to get just one uninterrupted 2-hour session in a day

We also looked at some of the ways programmers coped with interruption:

  • Most sessions programmers navigated to several locations to rebuild context before resuming an edit.
  • Programmers insert intentional compile errors to force a “roadblock” reminder.
  • A source diff is seen as a last resort way to recover state but can be cumbersome to review

The full article – http://blog.ninlabs.com/2013/01/programmer-interrupted/ – includes graphs, analysis techniques, suggested tools / features that help to fix the problems, etc. Good reading.

Also some good comments, for instance:

The less-quoted benefit of Pair Programming

“I was surprised to discover whenI first started pair programming 13 years ago, how one person ia a pair can hold the context while the other person in the pair gets interrupted. We could get back to work in seconds.”

The less-quoted benefit of TDD

“I was also surprised by test-driven development, how it helps keep track of where you are. Here are some of the factors I’ve noticed:
* The test tell you how far you have gone.
* Your test list serves as a reminder of where you need to still get to.
* The focus on small steps meas at any instance there are fewer balls in the air.”

Categories
computer games design games design games industry network programming programming

All Game Developers should read Pat Wyatt’s blog…

I noticed a few months back that Pat Wyatt has been blogging rgularly and in a lot of detail last year. This (IMHO) is big news: Pat is an awesome developer who held key positions in the teams behind many of the bestselling computer games (e.g.: Diablo 1 + 2, Starcraft, Warcraft) and went on to co-found Arena.Net (creators of Guild Wars).

I worked with him briefly in the past, and he’s friendly and full of advice and knowledge – but while he was happy to share, IIRC it was rarely in published form.

I’ve had a tough few months, but I’ve been dipping into his blog a few times, and it delivers in spades. Here’s a few hilights:

Assertions: enable them in live builds

(I’ve always felt this was the “right” way to do it for servers – where you don’t have to worry so much about frame-time, and assertions are more valuable at runtime because they help with the hardest-to-trace bugs … but it’s hard to get broad data on what the performance cost is)
http://www.codeofhonor.com/blog/whose-bug-is-this-anyway:

“The bug was easily fixed by upgrading the build server, but in the end we decided to leave assertions enabled even for live builds. The anticipated cost-savings in CPU utilization (or more correctly, the anticipated savings from being able to purchase fewer computers in the future) were lost due to the programming effort required to identify the bug, so we felt it better to avoid similar issues in future.”

…and a great rule of thumb for any Programmer:

“After my experience reporting a non-bug to the folks at Microsoft, I was notably more shy about suggesting that bugs might be caused by anything other than the code I or one of my teammates wrote.”

Some bugs are due to … user’s broken hardware

http://www.codeofhonor.com/blog/whose-bug-is-this-anyway:

“Mike O’Brien, one of the co-founders and a crack programmer, eventually came up with the idea that they were related to computer hardware failures rather than programming failures. More importantly he had the bright idea for how to test that hypothesis, which is the mark of an excellent scientist.

He wrote a module (“OsStress”) which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second.

On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate!”

AI cheats to improve game balance in RTS’s, starting with Warcraft/Starcraft

http://www.codeofhonor.com/blog/the-making-of-warcraft-part-3:

In most Warcraft missions the enemy computer players are given entire cities and armies to start with when battling human players. Moreover, Warcraft contains several asymmetric rules which make it easier for the AI player to compete, though these rules would perhaps be called outright cheating by most players.

One rule we created to help the computer AI was to reduce the amount of gold removed from gold mines to prevent them from being mined-out. When a human player’s workers emerge from a gold mine those workers remove 100 units of ore from the mine and deliver it back to the player’s town hall on each trip, and eventually the gold mine is exhausted by these mining efforts. However, when an AI-controlled worker makes the same trip, the worker only remove 8 units of ore from the mine, while still delivering 100 units into the AI treasury.

This asymmetric rule actually makes the game more fun in two respects: it prevents humans from “turtling”, which is to say building an unassailable defense and using their superior strategic skills to overcome the computer AI. Turtling is a doomed strategy against computer AIs because the human player’s gold-mines will run dry long before those of the computer.

Secondarily, when the human player eventually does destroy the computer encampment there will still be gold left for the player to harvest, which makes the game run faster and is more fun than grinding out a victory with limited resources.”

Categories
programming

SVGKit 2013 – Recipes

SVG is an awesome image format thats widely used, works in all browsers. SVG graphics make better apps and better games – and automatically “upgrade” themselves for future devices.

This post gives some simple 1-line / few lines of code recipes for using some of the main features of SVGKit – SVG implementation for iOS/OS X.

NOTE: this post refers to the 1.0.x version of SVGKit

Basic usage / installation

Install instructions are on the main GitHub page.

Basic usage / first-time usage info is on SVGKit 2013 – Usage.

Recipes

Load an SVG file like loading a PNG file

[objc]
SVGKImage* newImage = [SVGKImage imageNamed:@"imagename"];
[/objc]

Display an SVG file on-screen using an ImageView

[objc]
[self.view addSubView: [[SVGKFastImageView alloc] initWithImage:newImage];
[/objc]

…or an ImageView that supports CoreAnimation for every element

NB: the “Fast” imageview above saves everything as a single layer. Good for rendering, but bad for interaction. The “Layered” imageview here allows you to tap on any layer, rotate/animate/fade/drop-shadow/glow any element, etc.

[objc]
[self.view addSubView: [[SVGKLayeredImageView alloc] initWithImage:newImage];
[/objc]

Use a URL to load an SVG (open an SVG file *directly* from the web)

[objc]
// Splitting URL to multiple lines to make blogpost
// easier to read…
NSString* longURL = [NSString stringWithFormat:@"%@%@%@",
@"http://upload.wikimedia.org/",
@"wikipedia/commons/f/fd/",
@"Ghostscript_Tiger.svg";
NSURL* url = [NSURL urlWithString:longURL];
SVGKImage* newImage = [SVGKImage imageWithContentsOfURL:url];
[/objc]

Open an SVG from a custom source (maybe an in-memory SVG creation method)

First, create a class that conforms to SVGKSourceReader, i.e.:

[objc]
@interface MyNewClass : NSObject <SVGKSourceReader>

[/objc]

…then subclass SVGKSource, and over-ride the method:

[objc]
-(NSObject<SVGKSourceReader>*) newReader:(NSError**) error
[/objc]

Finally, use your customized SVGKSource to load your SVG:

[objc]
SVGKSource* myCustomSource = [[[MyCustomClass alloc] init] newReader:nil];

/** Other examples, using the default SVGKSource class:
SVGKSource* urlSource = [SVGKSource sourceFromURL: [NSURL …];
SVGKSource* fileSource = [SVGKSource sourceFromFilename: @"monkey.svg"];
*/

SVGKImage* newImage = [SVGKImage imageWithSource:myCustomSource];
[/objc]

Search an SVG file for particular tags / nodes / elements

SVG is an XML file, containing XML tags/nodes such as: “<svg>”, “<g>”, “<path>”, “<linearGradient>”, etc.

[objc]
NSString* tagToFind = @"linearGradient";
NodeList* result = [svgImage.DOMDocument getElementsByTagName:tagToFind];

for( Element* domElement in result )
{
// You can use the Element object directly:
domElement.tagName;

// …or, if it was parsed by the SVG parser, you can convert it to an SVGElement:
SVGElement* svgElement = (SVGElement*) domElement;

}
[/objc]

Search for sub-tags / sub-nodes of a particular tag/node

If you already have an SVGElement reference, you can search all its descendants:

[objc]
// To search the entire document, use:

// SVGElement* startPoint = newImage.DOMTree;
SVGElement* startPoint = … // from your code

NSString* tagToFind = @"linearGradient";
NodeList* result = [startPoint getElementsByTagName:tagToFind];

for( Element* domElement in result )
{
SVGElement* svgElement = (SVGElement*) domElement; // if your tags are all supported by SVGKit, they will be converted to SVGElement instances

}
[/objc]

Get a list of ALL descendant tags (from a particular node down)

[objc]
SVGElement* startPoint = … // e.g. for whole SVG doc, use: newImage.DOMDocument;
NodeList* allElements = [startPoint getElementsByTagName:@"*"];
[/objc]

Render one small part of the SVG file on its own

E.g. if you want to display a subset of the SVG, or want to export a single element:
[objc]
NSString* idInSVGFile = … // assuming your SVG file has an "id" attribute for this node
CALayer* absoluteLayer = [newImage newCopyPositionedAbsoluteLayerWithIdentifier:isInSVGFile];

// NB: "absoluteLayer" is now positioned in absolute space;
// if you add it to your window using e.g.:
[self.view.layer addSublayer: absoluteLayer];
// …it will appear in the same place as it appeared before,
// keeping all the offsets, rotations, etc
[/objc]

NOTE: there are several “newCopy…” methods, and each has different effects and outcomes. Read the method docs for each to decide which one you want to use. In a future version of SVGKit, we’d like to move these methods into a class of their own, and make it easier to see which one does what

Customise the parsing, using your own parser extensions

Create your custom class that adheres to SVGKParserExtension:
[objc]
@interface MyCustomSVGParserExtension : NSObject <SVGKParserExtension>

[/objc]
Then create a parser, INCLUDE THE DEFAULT extensions, and add your one on the end:
[objc]
MyCustomSVGParserExtension* myCustomExtension = [[MyCustomSVGParserExtension alloc] init];

SVGKParser* parser = [[SVGKParser alloc] init];
[parser addDefaultSVGParserExtensions]; // HIGHLY RECOMMENDED
[parser addParserExtension:myCustomExtension];

SVGKParseResult* result = [parser parseSynchronously];

SVGKImage* newImage = [[SVGKImage alloc] initWithParsedSVG:result];
[/objc]

Get help on why parsing failed (and warnings and line numbers!)

[objc]
SVGKImage* newImage = … // use methods above

// EITHER: parse using default parser:
SVGKParseResult* parseResult1 = newImage.parseErrorsAndWarnings; // this is a convenience pointer to (SVGKParser*).currentParseRun

// OR: use a custom parser:
SVGKParser* parser = … // use methods above
SVGKParseResult* parseResult2 = parser.currentParseRun;
[/objc]

And then you have the following info:
[objc]
/* array of NSError objects, each one a "WARNING" from the parser */
parseResult.warnings;

/* array of NSError objects, each one a "FATAL ERROR" from the parser – if your SVG didn’t render at all, this is why! */
parseResult.errorsFatal;

/* array of NSError objects, each one a "RECOVERABLE ERROR" from the parser – if your SVG didn’t render correctly, this is why! (although you probably still got to see something) */
parseResult.errorsRecoverable;
[/objc]

UPDATE: more Recipes!

For more recipes, see SVGKit Recipes part 2

(this covers answers to some of the common questions asked in the Comments below)

Categories
programming

SVGKit 2013 – Usage

SVG is an awesome image format thats widely used, works in all browsers. SVG graphics make better apps and better games – and automatically “upgrade” themselves for future devices.

This post explains how you can use SVGKit – the open-source SVG implementation for iOS/OS X in your own apps

(NB: basic installation is covered on the GitHub page – this is about how you *use* it)

SVGKit is like Apple’s UIKit

Originally, SVGKit was a bit tricky to use. I re-architected it to be an exact clone of Apple’s UIKit – same classes, same class structure, same method names.

So, we have:

UIKit SVGKit Notes
UI* SVGK* Apple’s standard: library names start with abbreviation of the library name
UIImage SVGKImage SVGKImage.h started as a copy/paste of UIImage.h
UIImageView SVGKImageView You cannot instantiate it directly, have to use a subclass
SVGKFastImageView Renders your SVG as a single layer, fast.
SVGKLayeredImageView Renders your SVG one CALayer per SVG element; you can hilight individual elements, tap them, animatet hem, etc

(a side note: We couldn’t use a classname prefix of “SVG” because the SVG spec reserves all classnames beginning “SVG”. Apple had a similar problem when they invented GLKit – the prefix “GL” was already used in the OpenGL library they were extending, so they named their classes “GLK” prefix. Hence … “SVGK”)

Loading an image

You already know how to load a UIImage, the “easy way”:
[objc]
UIImage* newImage = [UIImage imageNamed:@"myImage.png"];
[/objc]
Well, SVGKit is … the same:
[objc]
SVGKImage* newImage = [SVGKImage imageNamed:@"myImage.svg"];
[/objc]

Displaying an image: UIImageView

And how do you create a UIImageView?
[objc]
UIImageView* imageView = [[UIImageView alloc] initWithImage:newImage];
[/objc]
…so, for SVGKit:
[objc]
SVGKImageView* imageView = [[SVGKFastImageView alloc] initWithImage:newImage];
// ….. or:
SVGKImageView* imageView = [[SVGKLayeredImageView alloc] initWithImage:newImage];
[/objc]
Wait – why is this different?

The reference in both cases is an SVGKImageView – but you cannot alloc-init that class directly. Why?

Different people use the library in different ways. Some people need performance, others need detailed aniamtion. SVG’s contain a lot of bonus info, and it’s not possible to support every use case with a single class. So, you get to choose which one fits your needs

Advanced uses

So far, so easy. What about when you want more control? And how do you load an SVG over the internet (can you load an SVG directly from the web?)

Time to delve a little deeper…

Loading an SVG: in detail

There are four steps to loading an SVG (all of which are done automatically by SVGKImage above):

  1. SVGKSource: specify a location (a file, a filename – or an HTTP URL)
  2. SVGKParser: parse the file (all SVG’s are XML files, so your file can contain any custom XML you want)
  3. SVG classes: convert to SVG’s custom classes (from the SVG Spec); includes automatic support for CSS styling, cascades, etc
  4. SVGKImage: export the SVG to something Apple can render: e.g. a flat CALayer, or a hierarchy of CALayers, etc
  5. Experimental Exporter that saves a .svg file from your SVG file, including any changes you made to the in-memory DOM

Every stage is extensible, so you can either add features, or custommize it

Extend SVGKSource: adding new sources

Easy: create a new subclass of SVGKSource, and implement the two methods.

The method signatures are a little strange, because they have to support fast file-access (a C native library from Apple) as well as modern Objective-C methods.

One example of each: the supplied SVGKSource for loading from disk uses C methods, and the supplied one for loading from a URL uses Objective-C methods.

Extend SVGKParser: custom SVG files and formats

NOTE: do NOT subclass SVGKParser; parsing is very complex, and there’s a special mechanism that makes it easy to extend the parser (read on…)

Parsing is complex, so our parser is modular (based on ideas from a modular XML parser I wrote years ago). The SVGKParser class doesn’t parse SVG directly: instead, it parses raw XML, and converts it to a higher-level version that’s much faster to work with. It also manages error handling, loading bytes from an SVGKSource, etc.

You create a new parser instance with a helper method that includes all the modules you need by default:
[objc]
SVGKParser* parser = [SVGKParser parserWithDefaultParseExtensions];
[/objc]

SVGKParserExtension: adding your own custom parsing

Our SVG implementation is already split into sub-modules. This makes the code easier to maintain – and it makes it much easier to add support for SVG features one-by-one, in parallel with other developers.

Example classes:

  • SVGKParserSVG: parses “Basic” SVG – obvious things that require no special handling. NB: this is the most complex parser extension we have. Most are much simpler
  • SVGKParserGradients: parses SVG Gradients – because these were added later by a different developer (Stich) … likewise, if you’re adding a msising SVG feature, feel free to make a new parser-extension for it, it’s easier!
  • SVGKParserDefsAndUse: the SVG <defs> and <use> tags are much harder to parse than 99% of SVG, since they require complex cross-referencing and instancing. So, we use a separate parser extension to isolate this code.

For one of my games, I wanted to store gameplay data inside the SVG – attach info to particular SVG tags (e.g. give each SVG path a “bonusPoints” attribute).

To do this in a spec-compliant way, you should:

  1. Create a new XML namespace (requires no code: you simply invent a URL on a domain that you own)
  2. Put the namespace in your SVG file, using an <xmlns> tag, and give it a “prefix”, e.g. “my-game”
  3. Everywhere you want your custom attributes, or custom tags, prefix them, e.g.: “Write an SVGKParserExtension, and in the “supportedTags” and “supportedNamespaces” methods, return ONLY your namespace, and either nil, or the set of tags you’ve invented

Once the file is parsed, use the DOM to fetch your data (by definition, SVG parsers are required to be DOM-compliant parsers too)

e.g.:
[objc]
NodeList* myCustomNodes = [svgImage.DOMDocument getElementsByTagNameNS:@"http://my.custom.namespace" localName:@"my-custom-tag"];
[/objc]
(NB: NodeList is defined by the DOM, and is just a wrapper for an NSArray. If you #import “NodeList+Mutable.h”, you can access the NSArray directly, and use ObjectiveC fast enumeration. This is cheating, it’s not in the core SVG spec, which is why it’s hidden inside a separate header file)

Extend SVG Classes: the core SVG Spec

Don’t.

(the only valid reason for doing this is if you find features in the spec that are missing or broken in SVGKit. All other usages, you should be able to do in some other way, more easily, with less risk of your changes being broken when we upgrade SVGKit)

Extend SVGKImage export: export your SVG into custom rendering or to a new file on disk

All the parsed data for the SVGKImage is available to you directly, for exporting:

  • (SVGKImage* image).DOMTree : this property contains the entire parsed SVG-DOM (all the SVG* classes)
  • (SVGKImage* image).CALayerTree : this property contains the SVG converted into Apple’s CALayer classes, renderable directly in OS X and iOS
  • Additionally: “SVGKImage+CGContext.h” has methods to render the SVG into any CGContextRef you supply (or will autocreate one for you)
  • SVGKExporterNSData : creates NSData raw bytes, efficient and fast (good for OpenGL)
  • SVGKExporterUIImage : creates a UIImage instance, which you can use anywhere in UIKit

To save memory and CPU, the “CALayerTree” property isn’t created until you request it; if you don’t want the CALayer’s, they’ll never get created.

Improving SVGKit, fixing bugs, and ultra-advanced development

But SVGKit isn’t perfect: there are still missing features, bugs, etc.\

If you find a bug but can’t understand / fix it, please create a simple-as-possible SVG file and send it to us (create an Issue on the GitHub page, and include a link to the file). If you give us permission, we’ll add it to the suite of “test” images we use to verify each version of SVGKit, and that way it will always work, even with future versions.

If you fix bugs, or have bits you want to fix, please have a look at the SVGKit Development guide.

If you want support or help with SVGKit, the very best thing to do is create an Issue on the SVGKit page. That way, any of the contributors can see your problem and help you out. If you email me directly for help, you’re less likely to get a response (I’m busy, and I work on SVGKit entirely unpaid).

Categories
programming

SVGKit 2013 – Development

SVG is an awesome image format thats widely used, works in all browsers. SVG graphics make better apps and better games – and automatically “upgrade” themselves for future devices.

This post explains the underlying code architecture of SVGKit – the open-source SVG implementation for iOS/OS X; the target audience is developers who want to help improve SVGKit (adding missing features, fixing bugs, or making it more compliant with the SVG Specification)

Goals

Primary goals of the SVGKit project:

  1. 100% compliance with the SVG Specification
  2. Seamless integration with iOS (iPad/iPhone) and OS X
  3. Performance better than PNG/JPG/bitmap graphics
  4. …a library good enough that Apple would have liked to have included it in iOS

NB: the license terms for SVGKit are, without prejudice: “you can do anything you want with this, so long as you give credit to the SVGKit authors for their work”. Many of us are using it in commercial projects.

Core structure

The SVG Specification forces us to split the library into two parts, from the very start:

  1. SVG Spec – 100% defined by the W3 Consortium
  2. Native rendering – approx 10% defined by the W3 Consortium

The SVG Spec does have *some* requirements on the native rendering, and it has a lot of “guidelines” – but on the whole, it’s undefined, so that we can provide an implementation that makes sense on our platform (iOS/OS X).

I’ve divided this up into independent sections:

  1. SVG Spec
    1. Locating an input stream (e.g. a file, or an HTTP URL)
    2. XML parsing (low-level)
    3. DOM parsing from XML
    4. SVG parsing from DOM
  2. Native rendering – approx 10% defined by the W3 Consortium
    1. Conversion from SVG + DOM to SVG data (including: cascading, as per CSS (required by SVG Spec!))
    2. Dynamic changes to render data, to support Vector Graphics (Apple’s runtime support for vectors is – ironically – weak)
    3. Export to disk, using the latest copy of your modified DOM
    4. Export from SVG data to OpenGL (via raw bytes), to Apple’s (CAlayer/UIView), and to arbitrary CGContextRef instances

“Apple’s support for vector graphics is weak”

This was the biggest surprise to me: Apple has spent a decade marketing their OS (Mac / OS X) as “vector based”, etc.

In practice … OS X libraries were usually sparsely documented by Apple, and until iPhone came along, they were messy, buggy, poorly designed, and full of “out of date” methods. With iPhone OS (now renamed “iOS”), Apple cleaned their house out, and made some very lean, clear, logical APIs (with many fewer bugs!). They also – finally – documented it all.

That’s an amazing achievement, it’s very impressive. But along the way (probably to save time) they ignored some parts. The original iPhone’s CPU and GPU were very weak (compared to today), so it’s no surprise that Apple didn’t update their vector graphics libraries. iOS (as of 2013) is still using the under-documented and flawed OS X classes.

(NB: the lack of documentation also means that very few people know how to use Core Animation/Quartz/CALayer for high performance – you have to “experiment” and deduce what Apple *might* be doing, and test extensively. [Incidentally, there’s a lot of misinformation around – rumour and theory, in the absence of official docs from Apple])

Find the link for CALayer and bookmark it. This core class is where Apple’s vector libraries and main rendering intersect. It’s powerful – but it’s ugly and bloated too.

Simple bits, see elsewhere

“Locating an input stream (e.g. a file, or an HTTP URL)”

c.f. the SVGKit Usage post. This stuff is very simple, but it’s lacking features. Would be great for you to add some new SVGKSource subclases, with better features.

“XML parsing (low-level)”

Currently uses libxml (because that is built-in to iOS, OS X, and Xcode).

This wraps libxml, and adds three features:

  1. Captures every parse-error, and provides a list + line numbers when parsing is finished (libxml doesn’t have this feature by default)
  2. Converts low-level libxml C library to high-level ObjectiveC calls
  3. Provides a “modular” parsing system, where parsing code is very simple to write

On the whole, we have NO INTENTION of changing the parser – it works, and its intended to be as simple as possible. It’s really just an upgrade to libxml.

But there’s one thing it’s missing that we’d love to add:

  • Streaming / interrupt-based parsing

This is potentially more efficient in CPU and memory usage (not much, since we HAVE to use DOM – it’s required by the SVG Spec), but requires making the SVGKParser.m class a bit cleverer.

“Export from SVG data to OpenGL/NSData/CALayer/CGContextRef/etc”

Check out the “Exporters” sub-folder. It contains simple example classes – one per exporter – showing how to efficiently use SVGKImage to help you export stuff.

Note that approximately half of all SVG files have NO SIZE! – they are “infinite” – and you want to re-use SVGKImage’s code for calculating “correct”, or “best guess” sizes.

Since UIView uses CALayer’s internally, you can take any CALayer and add it to a UIView ([UIView.layer addSublayer:(CALayer*)myCALayer]).

Complex parts

“DOM parsing from XML”

The way this works is very rigidly defined by the SVG Spec, and you absolutely must stick to the Spec.

DOM is a major web standard, and the SVG authors thought it would save everyone a lot of time to re-use it.

Unfortunately – tragically! – iOS has no DOM implementation:

  1. Apple has a private implementation available in Safari. It’s not entirely private (we had to rename one of our classes because of a careless name from Apple), but Apples policy is “if it’s not explicitly public, we can reject your app for using it”. In theory, we could get access to this via WebKit source, or via an embedded WebView. But it would probably be much slower, and use a lot more memory, than our current native implementation
  2. There are a couple of open-source implementations, most of which have sadly been abandoned by their authors. Also, most of those I looked at are incomplete, and non-compliant; we can’t afford to rely on them.

The process for adding / modifying DOM classes goes like this:

  1. Copy/paste the DOM official class name (including the capitalization)
  2. In the header, paste the HTTP link to the *paragraph* of the DOM specification that defines that DOM class
  3. …then copy/paste the DOM’s interface/class declaration (usually 5-10 lines of code beginning “interface”, and blockquoted)
  4. Copy/paste that a second time, this time as the ObjectiveC Interface
  5. Convert every “variable” to an ObjectiveC @property
    • Note: by definition, you are supposed to replace DOMString with NSString*
  6. Convert every “method” to an ObjectiveC “-(something) methodSomething:(something);” method – NB: do *not* implement as C-methods
  7. Fill the .m file with @synthesize directives
  8. Create a blank method in the .m file for each method, and put an “NSAssert( FALSE, “Not implemented yet” );” in there (or implement it yourself)
  9. Any other DOM classes that are used as variables or method parameters … do all the above again
Hiding NSArray and NSDictionary behind SVG Spec methods

The SVG Spec is designed to work in ANY programming language – so it doesn’t support some core features of ObjectiveC, such as fast enumeration (i.e. the “for( NSObject* o in array)” syntax).

A much bigger problem for you is that you can’t include “init” methods, which are necessary for good ObjectiveC code.

Our DOM and SVG classes *must* be spec compliant, so we cannot expose the raw array – and we can’t add methods to provide fast enumeration, nor custom init methods.

Instead … when you have a situation like that, and you want users to be able to (optionally) access them … go ahead and do it, but put the “bonus” methods into a separate header file.

In Xcode, this is called a “class extension”, and it’s a special feature of ObjectiveC. Select “Class Extension” when creating the new file.

e.g. look at the source for Nodelist.h and NodeList.m – and notice that some of the methods are missing from the header file, but appear in NodeList+Mutable.h

In general, you should use the following naming strategies:

  • If the bonus features are needed to modify properties that SVG Spec says are “read only”, name the extension “Mutable” to make it clear that’s what it’s for
  • If the bonus features are ONLY a convenience, e.g. to enable fast enumeration, name the extension “NotInSpec” or similar

“SVG parsing from DOM”

Again, the SVG spec rigorously defines the name of every “SVG” class, and its methods, and its variables. You must follow these exactly.

The process is identical as for the DOM Spec notes above.

NB: SVG was designed and intended to be implemented on-top-of DOM; many of the SVG Spec methods are trivial to implement if you use the DOM methods that already exist. You are not supposed to re-invent the wheel!

For instance, have a look at DOMDocument, and Node, and Element – they have some very useful methods built-in to them.

Remmember: if you call SVGElement’s init method, then every SVG tag has already been parsed into a DOM Element (which extends DOM Node). It already has all the XML attributes pre-parsed and available to you!

Gotcha 1: SVG attributes are NOT nil

SVG Spec defines that “empty” or “missing” attributes have to be returned NOT AS NULL but as an empty string (“”).

This means you must NEVER write:
[objc]
Attr* fillAttribute = [self getAttribute:@"fill"];
if( fillAttribute ) // DO NOT DO THIS!!!

[/objc]
…because according to the spec fillAttribute can be non-null even though in the SVG it’s blank. Instead, you must (according to spec) do:
[objc]
Attr* fillAttribute = [self getAttribute:@"fill"];
if( fillAttribute.length > 0 ) // This is correct, according to SVG Spec

[/objc]

Gotcha 2: XML Namespaces

You can parse a lot of SVG’s and ignore namespaces; most SVG’s use the same “convention” for naming the XML tags.

It’s a convention; it’s a default; it IS NOT GUARANTEED.

But XML-namespaes are guaranteed. All SVGKit code should be using namespaces explicitly.

As a convenience for users, the DOM spec allows us to provide methods that do NOT need an explicit namespace – but you should not be using them! They will occasionally fail when used on some input SVG files

So, for instance, you should NOT do this:
[objc]
Attr* fillAttribute = [self getAttribute:@"fill"]; // DON’T DO THIS (it’ll work 99% of the time, but … best not to)
[/objc]
instead do this:
[objc]
Attr* fillAttribute = [self getAttributeNS:svgNamespace localName:@"fill"]; // CORRECT. (svgNamespace is the HTTP URL of the official SVG Spec)
[/objc]
At the moment, we don’t have a convenience method for “get the namespace that means SVG” – this really should be part of the SVGKparserSVG extension.

NB: if you’re afraid this namespace stuff won’t work, note that SVGKparser already has full namespace support, and will automatically create the SVG namespace if needed when parsing incoming SVG files

Gotcha 3: Cascading (as in: “Cascading Style Sheets” i.e. CSS)

The SVG spec officially is based on DOM; but it’s also (officially) based on CSS.

Fortunately, we only have to support a subset of CSS – the two parts that SVG uses are:

  1. Embedding stylesheets, or referencing them with an external “link” tag
  2. Cascading

But cascading is tricky. There are approximately 50 XML attributes that – officially – must be “cascaded” when using SVG. There’s a table of them in the spec – http://www.w3.org/TR/SVG/propidx.html

Cascading is tricky, and it’s potentially quite slow – you have to look-up the property in many different places, and check each one “in correct order” until you find the first match.

So, we have a method in SVGElement that does all this for you:
[objc]
-(NSString*) cascadedValueForStylableProperty:(NSString*) stylableProperty
[/objc]
…but that is not part of the SVG Spec, and it’s possible it *is* part of the CSS spec, but located in a different class (I haven’t found it yet). We’ll leave that method there as a convenience, but you might need to import a special header to access it (since it’s not part of the SVG Spec).

To use cascading (which you MUST do), instead of this:
[objc]
// DON’T DO THIS (it ignores cascading and styles and CSS-classes)
Attr* fillAttribute = [self getAttribute:@"fill"];
[/objc]
…do this:
[objc]
// Automatically does all the CSS stuff for you
NSString* fillAttributeValue = [self cascadedValueForStylableProperty:@"fill"];
[/objc]
…eventually, we’ll add an error / NSAssert for cases where you pass in a property that is not one of the cascadeable ones – for now, just use the table as a reference.

Class names and method names

We couldn’t use a classname prefix of “SVG” because the SVG spec reserves all classnames beginning “SVG”. Inside the project, you’ll find all of these in the “SVG DOM” folder – please note: these are MANDATED by the SVG Spec, we did NOT come up with the names.

Apple had a similar problem when they invented GLKit – the prefix “GL” was already used in the OpenGL library they were extending, so they named their classes “GLK” prefix. Hence … “SVGK”)

Whenever you create a new class that is not part of the SVG Spec – for any reason – you must prefix the name with “SVGK”.

Some of our classes – for historic reasons – don’t follow this convention. Yet. Feel free to refactor any you encounter.

Categories
programming

ObjectiveC: how to make an abstract class / forbid the “init” method

Abstract classes: saving programmers from each other

For trivial apps, no-one cares. But most libraries take huge advantage of the concept of “subclassing”, and programmers using those libraries need to make intelligent choices about “which subclass do I use?”.

Thanks to auto-complete, or “because it sounded what I needed at that moment in time” – or simply “because I was tired” – your base class gets instantiated when it shouldn’t have been. And strange bugs come from it, wasting everyone’s time. You might argue “not MY time”, but I’m a strong believer in writing code that EITHER does the obvious OR protects the people using it – my code doesn’t crash, it checks for obvious mistakes (e.g. checks that a file exists before loading it!), etc.

In the long run, that frequently comes back to help you: when YOU then re-use your own code, and make a dumb mistake because it’s been a long time and you’d forgotten how to use it.

In some languages, you can create “abstract base classes” that allow other classes to share type, but cannot be used on their own. This makes it obvious to other programmers that they should look for subclasses and pick one – instead of trying to use the superclass.

Unfortunately, ObjectiveC has no support for “abstract classes”.

…or does it?

What’s an abstract class?

An abstract class is one that cannot be instantiated. To achieve that in Objective-C, all you have to do is:

@implementation DontAllowInit

- (id)init
{
    NSAssert(false, @"You cannot init this class directly. Instead, use a 
subclass e.g. AcceptableSubclass");

    // NB: I prefer to use NSAssert because this is aimed at programmers, and 
    //     ObjC programmers should generally be using assertions during dev!
 
    // You could instead use more fancy approaches, like raising an NSException
    //     - but Apple/Cocoa are very anti-exception, and don't support them well.

    return nil;
}
@end

…but this causes a problem. Because as soon as someone creates a subclass, they’ll find their code crashes:

@interface SubClass : DontAllowInit
@end

@implementation SubClass

- (id)init
{
    self = [super init]; // CRASH!
    if( self != nil )
    {
        // all the normal setup code
    }
    return self;
}
@end

You can workaround this by writing documentation that says:

/*
This class can’t be instantiated, because I wanted an Abstract Class, but Objective-C was too primitive to allow it.

So, um, please don’t call [super init]. Instead call … ah .. [super secretInit] which does the same thing, but which other people won’t realise exists!
*/

There’s an obvious problem there … the super-secret-init is easy to call anyway, and BOOM your library. It might seem obvious to you that no-one would call that method without understanding it, but this is the way of the world.

Selective Denial: what am I?

The solution is to think about what happens when you instantiate a subclass. The key thing here is that when you call:

Super* s = [[Super alloc] init];

it’s NOT the same as when you call:

Sub* s = [[Sub alloc] init];

…in the first case, the thing that gets sent “init” is an instance of “Super”, whereas in the second case it’s an instance of “Sub”.

That might not sound interesting, but when Sub executes the first (standards-compliant) line of its init method:

The key thing here is that when you call:

-(id) init
{
     self = [super init];

…then the code in Super.m is *not* being run on an object of type “Super”, but rather an object of type “Sub”.

And so we have a solution:

@implementation DontAllowInit
- (id)init
{
	if( [self class] == [DontAllowInit class])
	{
		NSAssert(false, @"You cannot init this class directly. Instead, use a subclass e.g. MyPreferredSubclass");
		
		return nil;
	}
	else
		return [super init];
}

Does it really matter?

When writing code, you have lots to think about. In my years of experience, two of the most important questions are:

  1. Does it do what it says / work as intended?
  2. Can someone else use (and modify) the code later, when you’re not there … correctly?

Documentation goes a long way to solving both those issues. However … docs take a long time to write, and more importantly:

Other people frequently don’t read the documentation

More importantly:

If you are a great programmer, other programmers SHOULD NOT NEED TO read the code documentation any more than they expected to

“Expected to” is critical here. If your codebase is 1 million lines long, then a programmer would be insane to think they could just “dive in” and start writing / modifying it – the thing is fantastically complex. But if it’s clear and simple, then often they should expect to read the “core” documentation, and be able to work the rest out as they go, from reading your class and method names.

Abstract classes enable you – with very little effort – to use complex chains of OOP subclassing without endangering the programmers who come after you.

Categories
fixing your desktop programming

Fixing Firefox: Prevent it quitting and losing all your work

Are you having this problem?

“I tell Firefox to ask before quitting, but it always quits without asking”

Especially on Macs, where cmd-w (close tab) and cmd-q (close window) are immediately next to each other…

Solution

It looks as though Firefox won’t fix this. It’s been 3 years, and it’s been reported to them plenty.

In the meantime… I found one solution that *does* work:

Plugin: “Always Ask” (tested on OS X Mountain Lion, with Firefox 16 – works fine)

Background / history

I find it interesting. The developers decide to remove features – but seem to have misunderstood how/why those were important to the userbase.

Fortunately, the Firefox team are extremely open with their process. Anyone can see the reasoning and the debate that goes into each change – and can comment on it themselves with their own feedback.

So, reading the bug reports on Firefox.com, you can build up a picture. My impression – based on reading a bunch of these reports over the past few years – is that it went something like this:

  1. “The ONLY reason someone would want to “not quit” is because they lose all open tabs”
  2. “We’ve changed Firefox so that it re-opens all tabs by default, all the time”
  3. “Therefore: we can remove the feature”

That’s pretty sound reasoning. Although (and I’m not sure why?) … it seems they forgot to remove the GUI that lets users *insist* on the original behaviour, and to remove the three about:config flags that let advanced users fine-control it. Those are all still there, but they don’t work (any more).

But … by browsing the open bug reports, we find a bunch more reasons why Firefox had the original feature (just a sampling from what I read) :

  1. Accidentally closing would lose your set of opened-tabs (OK, they understood that one!)
  2. Firefox has disabled long-term disk caching for the past 4+ years – this is a basic feature of web-browsers, but to workaround bugs in their implementation, where data could get corrupted, Firefox mostly disabled it. It’s proving a long and difficult task to fix it, partly because the rest of the browser has changed so much in that time, and partly because the original implementation was poorly designed (I’ve been watching the bug thread for 3+ years now)
    • …so, when you restart Firefox, ALL your data has to be re-loaded from the web.
  3. If you’re offline when you accidentally quit: BANG! You (often) lose EVERYTHING.
  4. Even if you’re online: Firefox reloads every page. This can take a lot of bandwidth, and a minute or more to complete on broadband … on dialup, or tethered iPhone, it can take minutes or tens of minutes
  5. Some webpages WILL NOT reload after quitting, as a security precaution (for instance: internet banking). If you were typing into a form … the data there is lost, forever, with no workaround
  6. On Macs / OS X: the “close tab” shortcut (used hundreds of times per day) is adjacent to the “quit Firefox” shortcut (used rarely)
    • …OS X has a feature where you can re-assign those shortcuts. If you use this feature, Firefox IGNORES THE OPERATING SYSTEM, and continues to use cmd-q (although it will allow you to use whatever else you chose “in addition”)
    • On Windows, Linux, etc – the shortcuts are very different, so that it’s very difficult to achieve this. In many versions, it’s impossible – only OS X has a global “kill everything” shortcut for each app

…which explains why people get so angry and frustrated at the removal of this feature from the browser.

In short:

  • ONE reason to do with another Firefox behaviour (the thing that Firefox authors “fixed”)
  • MULTIPLE reasons to do with a major Firefox bug that no-one can fix
  • ONE reason to do with online security (that’s very unlikely to change! The security problem will always be there)
  • ONE reason to do with the design of OS X

I suspect that the most common of these reasons is the Mac-specific one. So it’s very likely that a bunch of Firefox developers – who don’t use Macs – wouldn’t have been ABLE to see the problem for themselves. That underlines the importance of consulting your user-base…

Categories
iphone programming

Updated: Static library for iOS – automatically cross-platform

I’ve just updated my script that adds a missing feature to Xcode – making your libraries automatically work on Simulator and on Devices, with no manual intervention:

http://stackoverflow.com/questions/3520977/build-fat-static-library-device-simulator-using-xcode-and-sdk-4

…and I’ve also put the script itself into GitHub as a gist you can easily copy/paste:

https://gist.github.com/3705459

Categories
programming project management

Source Code: never distribute an app as “source code”

I just ran into a 2004 piece of FREE software that I wanted to use, but can’t, because of poor choices by the original author. I’m posting this because I think the ideological reasons behind those choices are now “of historical interest only” and I’m liable to forget them completely a few years from now – but the underlying issues remain.

Especially in a world where Android, on a marketing platform of “openness”, is competing with iPhone, on a platform of “all users are lazy or idiots”.

(As a user, I hate being treated like an idiot. Except when it means a computer does all the work for me. Fair? Reasonable? Nope! When it comes to users … developers can’t win :) )

How should you distribute an application?

There used to be a raging debate, for decades, about the “correct” way to distribute applications. A bunch of well-meaning (but IMHO un-wise) Open Source programmers advocated:

“The only way to distribute a program … is as raw Source Code”

This was not about “is Open Source good?” – this was *in addition to* making source available. The question was: should you send people a copy of the source – or should you compile / package binaries (one-click applications) for people to download and “just run”?

The debate seems to be dead (finally), with the world coming down on the side of practicality, rather than theory/ideology. I’m not entirely happy with that – but it always felt obvious to me that it would go that way. I think the App Store in particular has gone a long way to “proving” it once and for all: people who want apps … want apps. They don’t want source code. Even if having the source would sometimes help.

2012: a worthy project that’s dead and useless

Today I ran into a tool that concretely demonstrates the futility of the “only source code is correct” argument: SLOCcount

The project as it stands is unusable unless you happen to be running one of the two linux distros where people have built the binary – or you’re willing to waste anything from “hours” to “days” of time “configuring” the app.

(with a normal app, that “hours of time” is replaced by “0.1 seconds it takes to double-click the app icon”)

This is a simple command-line tool. To run it, you must:

  1. Download the source
  2. Read the usage instructions
  3. Ignore the usage instructions. Start again with the “installation” instructions
  4. Install “make” (takes 0.5-3 hours)
  5. Learn how to use “make” (takes 1-3 days, if you don’t already know it)
  6. Debug “make” (takes 0.5-3 hours)
  7. Re-write the config files for the project so that they will work with “make” (takes 0.5 hours)
  8. (probably) install a new compiler (takes 0.5 hours)
  9. (probably) install a new linker (takes 0.5 hours)
  10. Cross your fingers, pray to whatever Gods you believe in, sacrifice a lamb, etc
    • In case you’re unfamiliar with “make”: it typically doesn’t work on any computer except one identical to the one where it was originally tested, so you have to go through and keep tweaking and fixing until it works. Kind of. There’s no checks-and-balances – so you NEVER ACTUALLY KNOW if make has worked, you just have to hope.
    • Finally: go to Step 1 of the usage instructions, and try to use the app

No wonder people don’t use it. No wonder people don’t update it, even though it’s “Open Source”. No wonder this – otherwise useful tool – is effectively dead.

Fundamental problem?

OK, so the straw-man example above mainly comes down to:

“make” is the world’s worst configuration tool

SLOCcount was probably killed by the choice of bad tools, as much as anything else. But – how much choice did the author have, really?

The problem is – and this is the interesting point of this blog post:

(in general) Source Code does not fully describe a program; it merely describes “SOME OF the internals of a program”

To create an actual usable program you need something like (off the top of my head, I think this is correct?):

  1. Source code
  2. Programming language definition
  3. Operating System (OS)
  4. Compiler program
  5. Programming language libraries
  6. OS-specific Linker
  7. Launch wrapper

…where the final output of step 7 is collectively known as “an application” (or just “an app”).

The folks who used to argue that all code should be distributed as Source tended to use arguments about the “value” of Source Code, as if it were a valid substitute for all the above items. It was never a *substitute*, although getting the output AND the Source would have been better than the tradition of only receiving the output.

Of course, even better would have been: receive all 8 items above (all the things necessary to make the app, and the app itself).

A brighter future

And so … if you’ve ever wondered what’s inside package-management sytems … take that list of 7 items above, and go revisit your favourite system of choice.

And … bear in mind that all the above things can have complex version dependencies – e.g. “only works with library A, version greater than 2.3, but less than 2.6, or with library Aa version 7.99 exactly”. A package-manager has rather a lot to handle…

Categories
programming server admin

Unity: Git source control – a basic .gitignore

The most popular hit for “unity gitignore” is a post on the official Unity forums that was written by someone who doesn’t seem to fully understand how git works. Which was a little disappointing.

Before you commit anything to git, you MUST go into Unity’s menus and enable the “metadata” option (in settings, moves around a bit between versions). Without that, Unity’s internal data-bugs will corrupt your project if you ever merge (happens to us approx 1 time in 3 if we forget at start of project)

After a bit of trial and error, here’s a basic .gitignore for Unity that seems to work, and cut down the commit sizes by a factor of 4 immediately:

UPDATE: updated with a tried-and-tested gitignore that covers more things. Do please note the big WARNING though, or you’ll only have yourself to blame…

##################
# Unity ignores:
#
# !!! WARNING !!!
#
# … you MUST convert Unity to using Metafiles *before* you start using this .gitignore file,
# or you WILL lose data!
#
#####
#
# !!! WARNING !!!
#
# … don’t forget to [git add “*”] (quotes are required!) when adding new files, or git will silently ignore them
#

# OS X only (not needed on other platforms)
.DS_Store
*.swp
*.Trashes

# All platforms
Library
Temp
*.csproj
*.pidb
*.unityproj
*.sln
*.userprefs

Categories
fixing your desktop Google? Doh! programming system architecture web 2.0

Google Docs 2012: Google loses your documents

Beware – latest version of Google Docs has the Gmail bug whereby emails (documents) randomly disappear and become completely inaccessible (this happens a few times a year with Gmail’s IMAP client). With Gmail, you can use the web interface to get around it and see the actual email – but with Google Docs, I don’t know of an alternative route. What you get is that Google appears to have deleted your data, with no comeback.

I just saw it now with Google Docs where a doc created yesterday allegedly didn’t exist any more – it had been removed from history, from recent documents, and … most distressing of all … it resulted in zero hits if you searched for it.

Solution?

There’s none – it’s just Google’s software being stunningly bad (again).

You have to wait. And pray. And wait some more. And click random things. And pray some more.

For me, clicking around on the different tabs on the left hand side (Home, All Items, Owned by Me) after a minute or so it randomly reappeared.

SaaS In the Cloud: Screwing the User, yet again…

I’m coming back to this topic more and more, because these abuses of web and HTML services are getting more and more damaging to the users.

If the “old” version of Google Docs doesn’t have this bug, it doesn’t matter – Google has already confiscated it from us. Unlike the core concept of desktop, laptop, and home computers … Google repeatedly takes the software that you possessed, and destroys it, for no reason other than to make things a little easier for engineers who don’t like to support the bad code they shipped.

That’s what this is about: people who use Cloud/SaaS to avoid taking responsibility for the apps they created.

Categories
fixing your desktop programming

GitHub 2nd chance: infinite crashes

I’m periodically re-trying GitHub’s GUI client, just in case it becomes stable enough for anyone to use on a real project.

Today, with the latest version: every time I start the app, it immediately hard-crashes. The app is literally unusable – it won’t run at all.

At least with the previous versions it would actually start up, so … this is a pretty big step backwards. If it had decent error reporting, like most apps do, I might be able to work out what’s wrong, but of course that’s one of the original problems with GitHub’s app – it disables all error reporting.

(as an aside: it’s interesting that Apple’s Mac App Store has made it so rare to see an app crash that … I can’t even be bothered to try and fix this app or complain about it. For an app to crash on startup is so VERY far behind the current “normal” for apps that … it’s just not worth the effort. I think the Mac App Store is one of the worst things to happen to computing in the last 20 years [if the US govt allows Apple and Microsoft to continue in this way, computers will become a thing of the past], but I have to admit it is achieving some great things)

Categories
amusing programming

Firefox 3 dead? StackOverflow ends support completely

StackOverflow.com now blocks anyone running Firefox 3.x (probably not deliberate – just someone wrote some bad code and didn’t test it?). But it’s interesting that sites that decide to go anti-HTML and pro-AJAX/JS can so easily break their whole site, just from some errant JS code.

(it appears to be some new pieces of crappy javascript that cause the AJAX requests to fail on Firefox 3 – and ALL THE BUTTONS AND FORMS ON THE SITE ARE DISABLED. Since you can’t click on anything any more, that makes the site unusable)

Categories
dev-process programming

Playing with Unity3D 3.5.x … some thoughts on starting from scratch

I haven’t shipped anything with Unity; last time I used it was early in 2.x, and it was painful enough that I never used it in a live project. I’ve re-eval’d every 6 months or so.

Now I’m onto Unity 3.5, and using it in several real projects that are due to ship this year … positive experience so far. I haven’t run anything on an iPad3 yet (that’s the next thing to try), but it’s treating me a lot better than Unity 2 did.

A few surprises along the way…

There’s still no decent “official” tutorials

There’s some video (yuck), and the first tutorial on the official tutorials page says (in slightly more words) “this is not a tutorial” in the opening paragraphs. Great! WTF?

I did find some basic ones (great for getting you into using of 3D objects *and* scripting together, at the same time) here: http://catlikecoding.com/unity/tutorials/

…and friends and colleagues provided a long list of recommended video-tutorials which I’ve been working through. Some sites have some great “beginner” tutorials, but they peter out in quality as they get more complex. Typically, the videos get more and more long-winded, boring, and a huge waste of time (many sites seem to have a hard-on for 2-hr videos, often for stuff that – if it had been a real tutorial – would have taken 30 minutes to digest. I guess they sell more advertising this way?).

The editor is very clear, self-consistent, and stable

Compared to early Unity 2.x (last time I used it), and compared to most 3D editors, Unity’s editor is very logical, consistent, and simple once you’ve had a 3-minute intro (good luck finding one; best to find a human and ask them to show you).

Overall … compared to most game tools I’ve used over the years … a delight to use. Although I’ve not hit it with any huge scenes yet (that’s where the real test will come; how often will it crash? how badly?)

Also: hasn’t crashed on me at all. Although there’s some occasional glaring bugs even in simple scenes (which is worrying) – e.g. one time, the scene playback engine corrupted itself and started “leaking” settings, which is allegedly impossible – on the whole, it’s decent quality.

(transparent) PNGs aren’t supported

2012, and Unity still can’t (fully) read PNG files. Wow.

(using PNG’s saved from Photoshop – but Unity’s crappy PNG importer screws-up the alpha. Depressing. Back in 2005 I thought we’d seen the last of the world’s “apps with only partially working PNG loaders”. Apparently not)

Javascript and C# sit very happily side by side

This is important for one project I’m working on, where we’ve got a mix of coders with diferrent backgrounds. All of them know ONE OF the Unity languages (C#, Javascript primarily … some other more odd ones too), but there’s no single language that EVERYONE knows. So far, Unity’s had zero complaints about us mixing and matching the script languages.

Categories
MMOG development programming security

LinkedIn (maybe) just leaked your password, won’t tell you; change it now

I’ve posted a few times over the years the … disappointing … state of LinkedIn’s engineering. But this takes the biscuit: it appears they were storing deliberately insecure passwords, and someone leaked the list:

http://news.ycombinator.com/item?id=4073309

(that page has links + info on how to check if your own password is in the mega list)

How bad is this?

  1. Many people have checked their personal, unique, passwords, that they claim to have only ever used on LinkedIn.com – and they’ve hit matches in the file.
  2. LinkedIn hasn’t told its users about the possible leak, more than 24 hours after it happened
  3. Many users re-use their passwords on other sites; any hackers could easily have stolen many accounts on other sites by now

How unlucky is LinkedIn?

This file is unsalted. That’s about as smart as locking your front door and then leaving the key under the mat – on the outside.

  1. Every tutorial, book, “newbie guide”, etc about using databases and writing login pages tells you never ever to do what was done here
  2. For any tech team, it is easy to check if this is what you’re doing, and tell your boss “uh, we need to fix that”
  3. It only takes a few *minutes* to prevent this problem, permanently. It’s not difficult

If LinkedIn were a small site, with a few hundred thousands users, I’d accuse them of laziness. But with 165million users, and a public company, you’d be looking at stunning incompetence by the tech wing of the company (the CIO and CTO never bothered to audit their own security?), or wilful negligence (no-one knew? really?).

Here’s hoping it’s a hoax…

Categories
entity systems games design network programming networking programming

Concepts of “object identity” in game programming…

Hume just posted his Lessons Learned from the warmup for Ludum Dare 23 (48 hours to write a game from scratch – starts this weekend!) – and his positive experience using an Entity System.

In his epic comment (sparked by a different Adam – not me, honest), is this gem:

“Using the entity system for the first time was unreal to me. It’s like polymorphic code. I did really weird things on the fly. For example:

– In the health processor, if the enemy was just destroyed, set a flag in the lifecycle component.
– In the lifecycle processor, if the fresh kill flag is set, extract its loot component and put that into a new entity with a small randomized velocity component and a gravity component so that the loot drops; then, remove most of the other components from the entity and add an explosion component.

The “enemy” still has the same entity ID — any other components that are looking for that entity will still find it (e.g. missiles homing in on the wreckage, or score processors looking for slain entities) — but by swapping one set of data with another, its implementation has changed from an enemy to some kind of non-interactive effect object.”

(emphasis mine)

Identity. It’s important.

(Quick sidenote: for all the people asking questions like “but … which variables do I put in Component A as opposed to Component B? How do I manage Events in an Entity System? … etc” – Hume’s approach above is a good concrete example of the first-draft, easy-to-write way of doing things. Copy it.)

Identity in games

This is one of those things that newbie game programmers seem to underestimate, frequently.

And when I say “newbie” I include “experienced, skilled programmers with 10+ years of coding experience – but who haven’t yet shipped a game of their *own*”.

(e.g. I’ve seen a couple of studios that started as Digital Agencies, or as Animation Studios, etc – that then transitioned to writing their own games. This is the kind of thing that they often struggle with. Not for lack of skill or general programming experience, but for lack of the domain-specific experience of game coding)

Examples of Identity in games, off the top of my head – all of these are independent, and interact in complex ways with each other :

  1. Game-role: e.g. … “enemy”, “powerup”, “start location”
  2. Code ‘object’ (in OOP terms): e.g. … “the sprite you are drawing at position (4,5) is part of Object X. X is coloured THIS colour”
  3. Gameplay ‘object’: e.g. … “the sprite at (4,5) represents a Tank. If a Tank sprite ever touches a Glass sprite, we need to play the Broken Glass noise”
  4. Physics element: e.g. … “5 milliseconds ago, our Physics Engine thought this thing was THERE. Now it’s over HERE. Don’t confuse the Physics Engine! Make sure it ‘knows’ they are the same object – not two separate objects”
  5. Network “master/clone”: e.g. … in multiplayer, there are N copies of my avatar: one per computer in the game. One of those N is the original – and changes to the original are constantly used to overwrite the clones; changes to clones are LOCAL ONLY and are discarded. Which is original? What do we do with incoming “changes” – which local Code Object do we apply them to? (your Code Object will be different from my Code Object – but they’ll both be the same identical Network Object, save mine is flagged “clone”)
  6. Proper Noun object: e.g. … “The Player’s Tank” is a very specific tank out of all the Tanks in the game. Many lines of game code don’t care about anything except finding and operating on that specific tank.
  7. Game-Over representation: e.g. … after the player has killed all the enemies, and they see a Game Over (won/lost/etc) screen, and you want to list all the enemies they killed … how do you do that? The enemies – by definition – no longer exist. They got killed, removed from the screen, removed from memory. You could store just the absolute numbers – but what if you want to draw them, or replay the death animations?
  8. …etc

Identity in Entity Systems

ES’s traditionally give you a SINGLE concept of Identity: the Entity (usually implemented as a single Integer). Hmm. That sounds worryingly bad, given what I wrote above. One identity cannot – by definition – encompass multiple, independent, interrelated identities.

But we’re being a bit too literal here. ES’s give you one PRIMARY identity, but they also give you a bunch of SECONDARY identities. So, in practice…

Secondary Identities in an ES

In OOP, the Object is atomic, and the Class is atomic. You cannot “split” an Object, nor a Class, without re-defining it (usually: re-compile).

In ES, the Entity is atomic, and the Component is atomic. But the equivalent of an OOP Object – i.e. “an Entity plus zero or more Components” – is *not* atomic. It can be split.

And from there comes the secondary identities…

A Primary Identity: e.g. “The Player’s Tank” (specific)
A Primary Identity: e.g. “a Gun Component” (generic)

A Secondary Identity: e.g. “The Gun component … of the Player’s Tank Entity” (specific)

Revisiting my ad-hoc list of Game Identities above, I hope it’s clear that you can easily re-write most of those in terms of secondary identity.

And – bonus! – suddenly the relationships between them start to become (a little) clearer and cleaner. Easier for humans to reason about. Easier *for you to debug*. Easier *for you to design new features*.

Global Identity vs. Local Identity

Noticeably, the network-related Identities are still hard to deal with.

On *my* computer, I can’t reference entities on *your* computer. I cannot store: “The Gun component … of YOUR player’s tank”, because your “Player’s Tank” only exists in the memory of your computer – not mine.

There are (trivially) obvious solutions / approaches here, not least: make your Entity integers global. e.g. split the 64bit Integer into 2 32bit Integers: first Integer is the computer that an Entity lives on, the second is the local Entity Integer. Combined, they are a “global Entity ID”.

(I’m grossly over-simplifying there – if you’re interested in this, google for “globally unique identifiers” – the problems and solutions have been around for decades. Don’t re-invent the wheel)

But … at this point, they also offer you the chance to consider your game’s network architecture. Are you peer-to-peer, or client-server?

For instance, P2P architectures practically beg for unique Global entity numbers. But C/S architectures can happily live off non-global. For instance:

  • On each client, there are ONLY local Entity numbers
  • When the client receives data from the server, it generates new, local, Entities
  • …and adds a “ServerGenerated” component to each one, so it’s easy to see that they are “special” in some ways. That component could hold info like “the time in milliseconds that we last received an update on this object” – which is very useful for doing dead-reckoning, to make your remote objects appear to move smoothly on the local screen
  • The server *does* partition all entities from all machines. But none of the clients need to know that

Or, to take it further, if your network arch is any good at all for high-paced gaming:

  • The server differentiates between:
    1. The entity that the game-rules are operating on
    2. The entity that client 1 *believes* is current
    3. …ditto for client 2, client 3 … etc (each has their own one)
    4. The entity that the game-rules accept (e.g. if a hacked client has injected false info, the game-rules may override / rewrite some data in the local object)
  • The server also tags all the entities for a single in-game object as being “perspectives on the same thing”, so that it can keep them in synch with each other
  • The server does Clever Stuff, e.g.:
    • Every 2 milliseconds, it looks at the “current entity”, and compares it to the “client’s belief of that entity”. If it finds any differences, it sends a network message to the client, telling it that “you’re wrong, my friend: that entity’s components have changed their data. They are now X, Y and Z”

… or something like that. Again, I’m grossly over-simplifying – if you want to write decent network code, Google is your friend. But the fastest / lowest latency multiplayer code tends to work something like that.

How?

Ah, well.

What do you think?

(hint: you can do wonders using Reflection/Introspection on your entity / components. By their nature, they’re easy to write generic code for.

But you WILL need some extra metadata – to the extent that you may wish to ‘upgrade’ your Entity System into a SuperEntity System – something with a bit more expressive power, to handle the concept of multiple simultaneous *different* versions of the same Entity. Ouch)

Yeah, I’m bailing on you here. Too little time to write much right now – and it’s been a *long* time since I’ve implemented this level of network code for an ES. So, I’m going to have to think hard about it, refresh my memory, re-think what I think I knew. Will take some time…