I’ve started a new open-source project to make an ES designed specifically to be ported to different platforms and work the same way, with the same class/method names, on every platform:
Here’s some thoughts on “why” I’m doing this, and “how” I’m making it work…
I’ve written Entity Systems in a few different languages:
- C (2 times: both small)
- C++ (twice: one BIG, one “proof of concept”)
- Objective-C (3 times: two small, one a port)
- Java (3 times: two small, one big)
…and I’ve noticed some recurring issues.
What is “Cross Platform”?
In 2014 game-development, there are a few monolithic platforms where you can thrive while only knowing/mastering a single language:
- Unity: C# (any other options are an illusion ;))
- Unreal: C++ (with Unreal 4…)
- Mobile (Apple): ObjectiveC (unless you work at a mainstream games-studio, in which case: C++)
- Mobile (“other”): Java (only Android and iOS have big enough market share for most of us to do *single platform* games)
- Console (both TV and handheld): C++ (of course!)
Note: each of these represents tens of millions (usually: hundreds of millions) installed userbase. When you’re shipping a game, a hardware base of “200 million” is enough for you to not bother with multi-platform code!
If you want to go cross platform, you have a couple of options:
OPTION1: Use the Lingua Franca
C++ works almost everywhere, and it has good compiler support on most platforms (not perfect – e.g. Google seems to hate it, with their ADK for Android sucking particularly hard – but close).
Downside: requires you to use C++ for everything, even game projects that don’t “need” it. C++ code is somewhere between 20% and 50% more expensive to write and maintain on like-for-like problems than most other platforms (it’s super powerful, verbose, low-level, and prone to horrors of e.g. manual memory management, ..etc).
OPTION2: Use a cross-platform engine
Unity says Hai.
Downside: You have to use C#, which is popular but not super popular outside of MS/Windows – many devs don’t (yet) use it. Also, Unity is a relatively immature platform, with many outstanding bugs and missing features. I love Unity, but it’s the cheap equivalent of many of the other platforms.
OPTION3: Learn another language
This is often the easiest and cheapest approach – but the one that developers (especially Indies) find most terrifying. Learning a new spoken language is much harder than learning a new programming language; don’t be afraid!
Also … Senior programmers today are usually expert level in at least 2 distinct languages, and highly experienced in 2-4 more. There’s a high chance they already know the extra language they’ll need.
Downside: the API’s and libraries you’ve come to know and love: do they exist on the “other” platform?
A good cross-platform Entity System
Artemis is an early, simple ES design that’s very popular among small Indie game developers. Indies who’ve been publishing games for a long time usually have their own codebases already, but new ones tend to start with Artemis. I recently re-ported it to ObjectiveC, for use on iOS, and to learn how it’s put together (https://github.com/adamgit/ArtemisObjC).
I noticed a few things when porting it:
- Some features are “core” to the ES (e.g. Entity, Component), others are Artemis’s “special sauce” (e.g. Aspect, Bag), and the rest is “useful but optional extensions” (e.g. the different EntitySystem subclasses)
- Some constructs are impossible to port (e.g. Java allows two method names with the same name but different argument-types)
- Some constructs use standard libraries that need porting (e.g. Java’s BitSet class is easier to use than Apple’s CFBit / CFBitSet C-code (non OOP!))
- Some performance-tricks work almost identically on any platform (e.g. Artemis’s Bag class is very easy to port)
A few years back I wrote a couple of blog posts, and a couple of open-source GitHub projects, for an ultra simple ES that Just Works and is easy to port across platforms. The aim was to set a lowest-common-denominator, and to help those who learn best by reading source-code (I’m not one of them, but I know people who are).
They succeeded in that sense, but weren’t practical enough for use in production. Artemis came along and showed how you could make something much better, good enough for real games, with relatively little extra work.
But Artemis (as above) was written specifically for Java, without considering how this could/would work in other languages. And it’s so popular that we have ports to 6+ other languages now – each of which has customized Artemis in substantial ways, so that it’s hard to go straight from Artemis-Java to Artemis-Javascript etc.
So … my practical benchmarks:
- ABOVE ALL: when you write game-source-code using this library … is it short, sweet, and auto-completed by the IDE, or does it require lengthy typing and manual copy/paste?
- Every time I use a language feature: first check that feature can ‘easily’ be implemented on most/all other imperative/OOP languages
- When writing public interfaces: check that every method name, class name, etc is legal in most/all other languages
- When choosing data-structures: check there’s a “natural” implementation of that in other languages (e.g. “resizeable/dynamic List” exists everywhere; “ordered HashMap” does not. Sadly “struct” does not :( )
- When using an unusual standard-library (e.g. Java’s BitSet isn’t used often in normal apps): check it has direct counterparts in all languages – or indirect it via a custom class that wraps the platform-specific one.
- When making indirect pointers: think about the “efficient” implementation of this on a C-based language; is this compatible with zero-copy memory semantics? If not, it’ll destroy performance on some platforms
The Lowest Common Denominator: Objective-C
It’s not Obj-C’s fault: most of the problems come from C. C++ was invented largely to enable OOP coding in C … so it’s obvious that C will make modern coding (especially “library/API development”) difficult.
But Obj-C does add some interesting constraints of its own. It’s a superset of C, but adds its own form of “classes” that’s easy to use but not quite as robust or flexible as the ones we see in C++, Java, C#, etc. It has poor support for structs (a pity: that’s one of C’s strengths) … etc.
All in all: an excellent LCD for porting to other C-family languages.
THIS IS MAGIC: Generics in Objective-C
This is some serious voodoo. My first commit to Aliqua’s repository has a Unit test that does this:
[objc]
Position* p1 = [[[Position alloc] init] autorelease];
p1.value = 42;
ALIListComponentInstances<Position>* positions = [ALIListOfPositions listOfPositions];
[positions set:p1 forEntity:0];
Position* fetchedPosition = [positions getPositionForEntity:0];
[/objc]
The first two lines are creating a new Component (the standard example: Position).
But the next three lines … whoa! That looks like Generics! And it’s somehow magically typesafe without casting! WTF?
Some smarter people than me (and an improved fork) figured out a clever way to auto-generate Protocols that simulate one aspect of Generics: the return-type.
I looked at this, went “WTF?” a few times. I thought about it, came back to it, tried it a few times. Then eventually figured out a way to make a Generic container for Component classes that’s automatically typesafe on iOS/ObjectiveC – using the ideas from Artemis’s Bag class.
The best bit of all?
Xcode5 auto-completes class and method names that are being auto-generated by C-style macros
Oh yeah! So all that stuff like “listOfPositions”, despite existing nowhere and being created on the fly (the Position.h and Position.m files have a one-liner to say “this is a Component, please DO YOUR VOODOO MAGIC”) … still auto-completes in Xcode. Yum.
Next steps…
I’m massively massively over-committed right now (7 x client apps to launch before the end of this month, plus a whole bunch of awesome stuff I’m trying to get funded and launched) … but I’m sneaking this library into my own projects now, and I’ll gradually add to it as I go.
I recommend Star’ing on GitHub: https://github.com/adamgit/Aliqua and following along.
As it stands, I probably won’t touch the other languages until Summer 2014, when I’m going to try and unify this with a Unity3D project (!) – giving me an excuse to try this design with C#. But anyone else who wants to port to other languages as we go along is welcome…
17 replies on “Cross-platform Entity System design”
Is there somewhere where the typical ES is documented in detail? Like… showing how you would write your own, regardless of language?
@James the wiki linked to in the article is a good start.
Or if you look back in the posts here tagged EntitySystems there’s several worked examples (in Java, ObjectiveC) a few years back.
@James
>Is there somewhere where the typical ES is documented in detail?
There is no canon for detailed ES architecture design as far as I’m aware. Snippets and ideas can be found around the web including Adam’s; which are useful to consult for ideas! The best I can offer as a larger place to look is to consider DOD (data oriented design) material since modern ES architectures fit within that paradigm. It might not be bad to take a look at the _beta_ DOD book by Richard Fabian here:
http://www.dataorienteddesign.com/dodmain/
@Adam
I do like your willingness to start new projects and even work in the open… I got to say though while reading this post I kept thinking when is this going to pop up… “I’m massively massively over-committed right now”… IMHO to move beyond prototype stage for ES architectures not only does the core architecture need to be made, but a ton of production code needs to be built on top of it providing use cases informing important architecture revisions. Until then it’s not a production suitable ES effort and just another experiment. Experiments are worthy endeavors nonetheless and I view your goals as stated in the 6 practical benchmarks you listed as likely resulting in basic research / ideas on manual cross-platform ES construction and not necessarily a production level cross-platform architecture.
Another bad side that you touched upon in respect to Artemis of being in the open while continuing work occurs and 2nd or 3rd major revisions are made to the core architecture is that if some popularity appears you get something akin to what happened with Artemis in the various ports at different stages of it’s development. Having not spent any time with the ports my guess is that the “aspect” revisions to Artemis didn’t make it to all of the ports. This may happen to Aliqua too resulting in clones with no standard.
For the holy grail of a cross-platform ES I still think an LLVM based solution will win out or at least be on the winners podium in the end. The cross-platform reach occurring after a given ES architecture reaches production level quality in one particular language / domain. This is somewhat conjecture of course, but with the success of libgdx using RoboVM and several games built on Java running well on iOS it’s a direction that can’t be ignored because there is precedent. If a library like libgdx can work in this manner there is high possibility an ES can too. This direction is more of a technical solution to what I’d expect the organic or rules of thumb results will be if you follow through on your Aliqua efforts guided by the 6 practical benchmarks you listed.
@Mike – you’re over-thinking it. All that matters is “how soon can I get this in live, shipped games?”. I’ve already tried running this code in a near-finished game (that’s being demoed in a few places), and it’s a big improvement in the amount of code, and the ease of editing. It needs less effort to be “good enough” for live games than you think…
Interestingly … many people are happily using Artemis for live games even though the original author IIRC never considered it “production ready” (still doesn’t, AFAICT). It doesn’t matter that it’s a long way short of what it “could” be – if it’s good enough, that’s all that matters.
I didn’t post this until I had something worth sharing – I’m building this because I needed it, and I’m using it in my own projects already. This is already copy/paste addable to any ObjC-based ES and would be a huge step up. What I describe (cross-platform ES) is desperately needed by many people, and since no-one’s “managing” Artemis, the fastest way to get it into existence is to build it from scratch.
@Mike – Java on iOS will continue to be ignored by 99% of iOS coders no matter what: they simply don’t care about Java. The ones that can code well enough in Java to be interested … are usually coding in C++ instead anyway.
It’s interesting and useful for niche cases (such as people writing games in Java and then cross-publishing to multiple platforms), but that’s a tiny niche compared to the juggernaut of tens of thousands of expert C, C++, and Obj-C coders who have no time to spare to look at Java because iOS already works “more than well enough” for them with their language of choice.
e.g. ObjC’s lack of Generics is a huge loss; but it’s balanced out by simple, easy lambda/closures integrated with the entire standard library (blocks). Obj-C coders don’t care about Generics because they’re too busy launching apps using the time saved by Blocks…
@Adam:
>you’re over-thinking it. All that matters is “how soon can I get this in live, shipped games?
Regarding availability it depends on what the end goal is since a production level component architecture / ES is certainly worthy of a valuable product in and of itself. I am a tools developer and am interested in selling the tool. You wrote before on the difficulty some teams you have interviewed had regarding ES usage. Given deliberate focus on usability of ES architectures for large projects / teams is not solved by getting an initial prototype in use for a game as soon as possible. Games are also just one of many niches served by component architecture / ES techniques.
On the former, while anecdotal at least in my efforts very useful patterns didn’t emerge until after significant amounts of production code was running. These patterns in my case required significant architectural changes further refining core component architecture / ES techniques accompanying lots of refactoring of the implementing code base.
>Java on iOS will continue to be ignored by 99% of iOS coders no matter what: they simply don’t care about Java.
That’s an opinion. Is Unity ignored on iOS?
>It’s interesting and useful for niche cases (such as people writing games in Java and then cross-publishing to multiple platforms)…
If Android didn’t exist this comment would have more relevance. Why ignore Android? There are tons of _average_ to expert Android / Java developers who would use assistive tooling to deliver iOS apps / games. IMHO a powerful component / ES architecture enables average developers to be more productive and experts to deliver the impossible by any other means; IE OOP. Accomplish this and language is irrelevant.
My comment though is not Java specific. Replace Java with your language of choice backed by an LLVM based solution is still likely a realistic approach to a cross-platform solution. Tooling and ease of use for small to large projects & teams is what is important and trumps choice of language. There are many problems with Obj-C / iOS dev that impacts modular / reusable library & framework development. I believe you have even written about some of these problems.
>Obj-C coders don’t care about Generics because they’re too busy launching apps using the time saved by Blocks
As things go though component / ES architecture authors care about generics because it enables the creation of a usable and powerful APIs that also supports advanced tooling. In my efforts the advanced usage of generics is baked into the API and doesn’t force the end user / coder to know anything about complex generics implementation while gaining all of the benefits and then some.
I haven’t worked with blocks, but it smells like OOP to me since data and logic are not separated from what I’ve read so far. Could you perhaps enlighten me how blocks are useful for component architectures / ES implementation?
Sad because you didn’t mention Artemis C#, it is so much better than the other implementations :)
@Alisson – I haven’t used it for anything yet. But I’m looking forward to exploring it soon when trying to integrate ES with Unity. That goin to be interesting :)…
@Michael: From experience, until your tool has shipped in multiple commercial games, you’ll have a really, really hard time selling it. As long as you have a tools developer approach it’s easy to get caught up in architecture astronaut or unproductive UX mindsets. Both of these are exacerbated by shifts away from the dominant programming paradigm.
Again from experience, a “powerful component / ES architecture enables average developers to be more productive” only if those developers understand entity systems, both strategically and for practical implementation. If you haven’t worked with a wide variety of developers at a range of experience levels on multiple projects, your ES probably isn’t ready to sell as a tool.
We often clash here, and I don’t mean it to be personal, but I think the remainder of your post underscores it: for example, you say “component / ES authors care about generics”, but your reliance on generics goes much, much farther than that of any other entity system I’ve seen. I think you’ve narrowed your focus to your project and don’t understand the range of users, use cases, and alternatives; I think this is part of why you so often have trouble making yourself understood or finding agreement for your points here.
@Tom
Nothing personal / no worries of course.. :) It’s not the first time the “architecture astronaut” label has been bandied about and kind of why I never contributed much on the JGO forum over the years. Am I on a moon shot, perhaps, but the journey itself has been rewarding from a deeper understanding of coding angle. Besides I started from the angle of a moon shot with my passion for large scale spatial audio which formed the impetus for TyphonRT as a media framework to begin with… egrsoftware.com has some picts…
>From experience, until your tool has shipped in multiple commercial games, you’ll have a really, really hard time selling it.
No doubt… The game market is going to require a hit or at least a notable game to get things kickstarted and likely it’d have to be in house developed. It’s also rather cutthroat on the tools side due to so many open source and low cost options for game dev. I’m definitely banking on the long game and applicability beyond game dev tools.
Luckily my efforts serve any niche due to a generalized component architecture and ability to create custom runtimes for any purpose. I have been biding my time to find an underserved niche and I do believe I’m onto a hot one presently. I have not posted on this much yet, but I am currently making a concerted effort to launch ASAP a middleware offering using TyphonRT. It’s not a game / game engine though, but a next gen video engine / camera manipulation framework for Android that blows away anything on Android or iOS (that I’ve seen) presently. The demo app provides an intuitive way to manipulate camera output with multiple layers of shader based effects with a fluid UI w/ no hit on recording performance. Everything that Photoshop and video editors can do one can do in real time and more regarding creative effects or more standard image manipulation. It’s neat too because effects are specified in external files, so new ones can be added without recompiling the codebase. Features can be scaled back for lower powered devices without conditional code spread throughout the codebase. Not a game of course, but it heavily uses the component architecture techniques that forms the basis of the ES aspects of my efforts as after all I’m a strong advocate of component architecture aspects of ES implementations being a superset. It’d be really hard to implement the feature set I’ll deliver via traditional OOP without things becoming a tangled mess that is hard to extend / modify. Given my video engine comes in at ~2 megs vs 50+ megs ala Vine it can be embedded in any Android app that needs integrated video recording w/ effects or not. There may be some tech giants interested in such a middleware offering; we’ll see… It’d be nice after all these years to know where rent is coming from imminently.
>a “powerful component / ES architecture enables average developers to be more productive” only if those developers understand entity systems…
Technically that is the ideal / or grail based overarching reason to do any of this in the first place. Interactive tooling / editors is the way to take the rough edges off using the framework directly for devs new to it all. Ease of setting projects and availability of many modules / components to handle advanced media tasks out of the gate doesn’t hurt either. I don’t expect much uptake without tooling and no doubt significant pedagogy resources will be necessary too, but that is the case for any break with accepted paradigms. Will there be a 200+ page book in my future; yeah, probably.. ;P I already had my rude awakening in giving an all day game dev tutorial a couple of years ago with a room full of non game devs let alone anyone aware of component architecture / ES efforts.
>for example, you say “component / ES authors care about generics”, but your reliance on generics goes much, much farther than that of any other entity system I’ve seen.
Indeed, but the only thing I’m concerned about are cross-compilation aspects. Luckily in cross-compiled code the target doesn’t need to be readable per se as long as the runtime is solid. All of the tough details are not forced on users implementing any tricky generics.
>I think you’ve narrowed your focus to your project and don’t understand the range of users, use cases, and alternatives
I read and constantly search for any discussion on component architectures / entity systems and relevant subject matter regarding data oriented design. If there are available codebases to review I examine them.
>I think this is part of why you so often have trouble making yourself understood or finding agreement for your points here.
I’m here particularly because there isn’t much ES discussion going on elsewhere, but yes I find it interesting to float ideas here. There is no canon for ES terminology per se, so finding what resonates is useful. I’d say that the essential conflict more has to do with the “indie wisdom” of skipping architecture work in favor of shipping games; ala what Adam expressed. However, to move the state of the art forward it takes all types. I wouldn’t recommend a younger developer to go down the path I took, but I knew it was going to take time from the get go with my original aspirations (large scale spatial audio).
You should probably mention MonoGame (C#). I think quite a lot of people were comfortable using XNA as a platform for their own game engines (me included) and MonoGame is basically an open-source version of it. Xamarin makes it cross-platform (not too cheap for an indie though).
@SomeGuy – I’m not saying there are no other platforms … but I’m listing the ones that are strategically sensible: affordable, well-supported, easy to hire people for, huge installed base, etc.
XNA was good, but … I’ve met very few devs who consider it a viable platform for ongoing development. e.g. Marmalade is also good, and seems a lot more popular – but even Marmalade is niche compared to Unity/iOS/Android.
@Adam
I was talking about MonoGame rather than XNA. The latter is now officially dead. I understand what you mean though, I am outside of gamedev industry so it’s hard for me to tell. Popularity is a valid reason, I am “emotionally attached” to C# so am hoping MonoGame will become more popular this year.
your es-related posts were a great read, thanks for that. can’t wait to see what you will come up with. keep it up!
Hey, where’d the repository go?
No-one was using it, and I kept making changes that break all existing code.
Rather than waste time constantly writing update instructions, I took it offline until the API is more stable.
I also realised I’m already maintaining as many open source prjcts as I have time for right now, probably too many in fact. Maintainin another one wasnt a wise decision.
Ok, fair enough. :)