Posts Tagged Species


I’ve been working on applying torques to the creatures in a meaningful and well thought out manner (read: banging my head against the keyboard whilst trying to fix all the horrible signage errors). The original intention was to simply pass the torques on to the energy system and let natural selection weed out the unbalanced and physically impossible specimens. But I realised I have the perfect opportunity to overhaul a system that has bothered me since its inception: the quadruped system.

Logically, if a creature has slightly smaller front limbs than back limbs, it should lean forward in order to walk on them (unless it’s a pangolin, but they’re too awesome to be swayed by our petty logic).

Throw logic out and do the impossible, that’s the way pangolin’s roll

Without something to enforce this, every creature in Species ALRE would be a biped, because every creature has at least a *slightly* different leg size.

The quadruped system governs how this is implemented. A hideously simplified description would be “if the torso rotation is less than 45 degrees, and it has fore or back legs, the creature will fall forward onto its front legs or chest. If it’s higher than 45 degrees, or it has mid-legs, it will remain bipedal.”

That “45 degrees” is completely arbitrary. It’s just a value I picked.

This system is why ‘head-down butt-up’ creatures, whose bodies hang forward from their back legs are so common: Any creature with large back legs and a small torso rotation will end up with this body plan.

That’s the old system. The new one is much more sophisticated.

It starts by completely eliminating the quadruped system: creatures will initially be created *exactly* as their genes would have them (ie. with only one pair of their legs reaching the ground). It then calculates the skeleton and torques as normal.

Once that’s done, it passes control to the droop system. This applies “droops” to every body part attached to the torso: limbs, neck and tail. Legs are bent upwards by the reaction force on their legs, neck and tail are bent downwards, and the torso droops based on the creature’s balance, all in direct proportion to the torque applied on the joints.

If a body part droops into the ground, it stops drooping and adds a ground contact point, which will take some of the creatures weight in the next step.

Then pass 2 starts, the body-plan is completely reinitialised: bones are regenerated, torques and forces are recalculated. The results of pass 2 are what the creature will ultimately use for most of it’s energy calculations, so as to take advantage of the creature new quadrupedal body plan. (physical droops will affect energy, but proportionally to how much they rotate rather than as a function of torque. Small drops onto legs will be a negligible cost)

This also has a noticeable effect on creature appearances. I haven’t adjusted the rotation genes at all in the following screenshot: these variations are entirely the result of torso, neck and tail thickness changes causing different torques in the creature’s body plan.



One particularly noticeable change is that the creatures tail now has an effect: increasing its mass is the only way to counter the torque produced by the head and neck in the second body plan. For bipeds with centrally situated legs (raptor-like creatures), a sizable tail will be the only way to maintain balance (other than having an upright torso rotation gene and incurring a large energy cost in droop).

I’m hoping this system will be realistic enough to promote life-like body plans in place of a real-time physics system, which would be far too CPU intensive to implement across thousands of creatures. More likely, it will be exploited ruthlessly in ways I can’t yet imagine. But hey, that’s what makes it fun!


“Anime references. There goes the neigbourhood”

, , , , , ,


Stat Encapsulation

I forgot to charge my laptop yesterday so used the forced break to practice my sketching, but I’m back in it today and development continues.

The Mutation Map tool now has the ability to view and edit both the overall map and the individual stats of each head. It can also do the same for Feet, and I’m working on getting the remaining mutation maps (Limbs, Coverings, Colour Patterns, Feature Models and Feature Textures) into the tool. Hope to be done by the end of the week and moving on to other things, though that may be a bit optimistic.

Also took some time to begin taking my Stat encapsulation to the next level. Currently we have 6 types of Stat:

  • GeneticStats encapsulate a mutation rate, plus mutate, randomise and blank routines, for every gene.
    • FloatingGeneticStat: A hereditable decimal number, like “torsoWidth” or “headSize”. Encapsulates an upper limit and lower limit in addition to the mutation rate.
    • DiscreetGeneticStat: A hereditable integer number, like “numberOfFeatures”. Just encapsulates the mutation rate.
    • BooleanGeneticStat: A hereditable yes/no value, like “hasTail” or “hasNeck”. Just the mutation rate again.
    • MappedGeneticStat: A stat with a mutation map, like “headType” or “bodyCovering”. Encapsulates the entire mutation map structure.
    • BasalStat: a non-hereditable stat which is imported from an external file, linked to a MappedGeneticStat. Like how HeadType determines Diet, or how LimbTipType determines BaseLimbDamage. These stats are rarely used directly: usually, they feed into formulas to determine other stats. (diet is an exception)
    • UpdatableAttribute: a type of Stat I have encapsulated but haven’t derived from IStat yet: these are things like Health, Energy and Biomass, which change over the course of a creature’s life. Encapsulates a list of “Deltas”, making it possible to track the source of changes to these stats (you can see these deltas listed out on the energy page in creature details).

There’s only one major type of stat still missing, at least on a per-creature level:

  • DerivedStat: Any stat which calculates itself from other stats. For instance, Speed takes into account LimbSize, LimbBaseSpeed, TorsoWidth, TorsoRotation, LimbTipBaseSpeed, and so on. DerivedStat will encapsulate all these stats, plus the formula used to combine them.

The method I’m using to encapsulate the formula uses something I haven’t spent much time with in the past: delegates. I’m storing it the entire formula as a delegate inside the DerivedStat instance. This means I get to define the formula in StatData, but still use the built-in accessors of the objects, so I get the same intellisense benefits and high performance as if I was defining  the calculation over in Phenotype.

The result looks like this:

    "mass",                                                          //Name
    "Mass",                                                          //Printable Name
    "Determines how much stuff there is in the thing",               //Printable Description
    delegate(Genome genome, BodyPlan bodyPlan, Phenotype phenotype)  //boring code stuff
        return bodyplan.Volume * genome.Skin.Density;                //Formula

The relevant bit is the second-last line: that’s the only place that formula needs to be defined. All I have to do later is call Initialise, and the DerivedStat takes care of calculating itself on behalf of it’s parent creature, pulling other stats in and combining them, as well as throwing errors if stats haven’t been correctly initialised.

Yes yes, making with the boring and the code and the words and the stuff, so what? Why do we care?

Here’s why we care: every calculation, limit and mutation rate from the raw genes to attributes like speed and stamina is now stored in StatData. This one file is becoming the spinal column of a lot of the simulation’s behaviour, so tweaking mutation rates, adding limits, changing names and descriptions, and now adjusting stat formula will all be possible simply by changing the input data here.

And I intend to expose that input data to modding.

Cue the maniacal laughter,



Leave a comment

0.7.0 Planned Features

0.6.1 is finally hotfixed to the point of “reasonable-stability”, so it’s time to begin the subjugation of the foolish and pathetic human race. Begin deploying the jetpack-equipped Tasmanian Devils at once!

While that’s happening, we should look at revising the flight plan for 0.7.0. We completed some things in 0.6.1 that were originally to be part of this update, and learned a few things as a result.




We included a lot of the aesthetic work in 0.6.1, but there was one feature we didn’t get to: improving the creatures animation. The animations we have planned will be basic but functional: think more “Minecraft” than “Spore”. They’ll include behaviors for grazing, feeding and fighting, as well as improving the walk cycle.

Metric Units

We’ll be converting the game as a whole from unspecified relative units to metres, kilograms and newtons. Our plan is to use the length of an adult Primum specium as our base scale (1Ps = 1m). This puts the larger trees at around 6m tall, and makes the rover roughly the size of Spirit and Opportunity, which is a nice co-incidence.

Physical Values and Skeletal Analysis

We’ll be defining Density and Volume, which will give us Mass and Weight, which can be converted to translation forces and torques and applied via skeletal analysis. By leveraging the stuff I learned in university and the infinite power of Google, we should be able to achieve much more realistic body-plans than “Stamina = (Base * (Bicep + Tricep) * Size * 2) * Bipedal * Streamlining”


Fences, impassible barriers you can construct to block or isolate populations of creatures.

Climate Control: Heaters, Coolers, Humidifiers and Aridifiers you can place to adjust the temperature/fertility of the local area, rather than the whole map.

Targeted Rover Programming: instruct your rovers to feed or kill creatures based on their genetic distance to a target: either the species average, an exported creature, or a target you design yourself.

Stat Map Editor

Until now I’ve been tweaking the stats on body parts, textures, biomes and trees by editing the data.txt files manually. This is a slow and error prone way of doing things: I can’t see what I’m tweaking, I can’t compare it to other things in the mutation map, etc. To compensate, I’ll be creating a Stat Editor that shows me the physical objects alongside their stats and relationships and allows for tweaks and adjustments in real time. This will mostly be for me, but it might also come in handy for modding.


Most of these items are stepping stones on the way to something greater, most notably the gameplay additions: once those three are in place, I’ll be able to start work on a feature that will form the backbone of gameplay in the future.

Now if you’ll excuse me, I need to check how the subjugation is going.




“What do you mean “you lost them”? We’re in orbit and they’re loud, violent marsupials, where could they possibly have- they’re in this room with me aren’t they? Dammit, I just know they are. Any second now I’m gonna be mauled.

Aaaanny second now…

, , , ,

Leave a comment

0.6.1 Hotfix 4

Welp, we finally managed to track down the Black Ground bug, by means of Google research and an Ouija board. Turns out AMD cards can’t use a 2.0 vertex shader model in the same pass as a 3.0 pixel shader model, but rather than doing something sensible like, I dunno, telling me about it, instead it adopts the much more interesting approach of falling over randomly and not drawing anything.


Fixing this meant adjusting a whole bunch of shader files, and I lost track of which ones had been modified somewhere along the way, so to get the hotfix you’ll need to re-download the game. If you extract the files into your existing Species folder and merge/overwrite, you should be able to keep your save games and exported creatures.


Also made a few small tweaks and fixed one of the more common exceptions. Potentially-stable release ahoy!

– Fixed the Black Ground/Invisible Creatures issue people with AMD cards were experiencing.
– Fixed the ArgumentOutOfRangeException that was causing crashes after an unspecified period of time.
– Tweaked Body Pitch so creature’s don’t lean back as much as a result of growing legs
– Tweaked CalculateReach so creature’s can’t increase their reach by having a not-expressed neck length.

, , ,

Leave a comment

Species 0.6.1 Hotfixes

We’ve been busily hotfixing since the release, and 0.6.1 is now much more stable than when it was first released. We recommend downloading the entire thing again:

(Tip: if you extract the latest version directly over the top of the previous one, you won’t lose any of your saved worlds or exported creatures!).

… but if you’re an advanced user and don’t want to download the full 75MB again, you can download the patches. Make sure you apply them in order.

Hotfix 1 Patch
Hotfix 2 & 3 Patch

, ,

Leave a comment

Species 0.6.1 Released

Welp, here we are. Species 0.6.1 can now be downloaded from the download page.

Get the latest release of Species here!

This release was all about cleaning up the back-end code for future features and making visual improvements.

Be warned, this release is likely to be amusingly buggy for a week or three. Whether or not that’s a good thing depends on how much you enjoy watching me scramble about like a lemur on fire trying to keep up with the necessary hotfixes. And let’s face it, of course you do, that’s hilarious. You bastards.


Stability Improvements

- several fatal crashes have been fixed (‘System.OutOfMemoryException’, ‘CladeDiagram.CollideBranches’, ‘UIManager.Paused_Export_Update’),
- the method for addressing problems when the game is minimised or graphics card is lost has been improved,
- built-in troubleshooting for some start-up problems (running from the zip file, running from Program Files, running on computer with unsupported graphics card) has been added

Aesthetic Adjustments

- Procedural texturing: creatures now have different top, side and underbelly textures, giving them a more distinct appearance and more recognisable body covers.
- Custom textures: beaks, claws, teeth and mouths now have their own (appropriately horrifying) textures, blended into the triplanar texturing.
- Fur: Fuzzy wuzzy was a bear, fuzzy wuzzy had no hair, but the creature’s in Species do now. And they find bear meat tasty.
- Water reflections: They only reflect the skybox, but they significantly improve the appearance of the game nonetheless.
- A new Species ALRE logo. Now with infinity% more cladeogram and cycad leaf.
- Adjusted Tree Shaders make better use of dark greys and oranges, meaning trees have more colour variety and look more natural.
- An improved wasteland texture.
- Fixed that weird blue shading on trees when you pause the game (it still happens on grass).

Content Additions

- Body coverings: fur, feathers, scales and skins. Also squid skin, toad warts, echidna spines and human forearms (ewwww…).
- Leg shapes: from 6 in 0.6.0 to 20, meaning a much greater variety of knee shapes.
- Heads: a number of additions (and of course, the existing heads had to be adjusted to provide beak and mouth textures and fur polygons).
- A selection of the Body Part Voting Competition head types.


- Added an "Indirect" rover drive mode to the game options. Seriously, try it out. It's hilariously broken.

, , , , ,


Last minute stuff…

First, fixed an introduced bug that was giving canivores the ability to eat vegetation and removing herbivores ability to eat anything. A single missing “-” and herbivory stops being viable. Let’s hope that’s the last major error and the release is stable! (“Haaaaaahahahahahah!”)


Second, since I haven’t spotted any other bugs recently, I followed through on my threat suggestion to include an Indirect Rover Control Mode, where the rover uses it’s AI to attempt to drive in the direction you point the camera. The result is every bit as broken and insane as you’d expect, and it. is. GLORIOUS.


Finally, Species has adopted a new, less minimalistic logo.




The clade diagram was planned from the start: the symbolism there is an obvious match for Species. I find it amusing how the game’s name correlates with a mass extinction, but then again I’m easily amused.


The cycad leaf was more a case of Throw It In: it was originally going to just be the middle arc, but that looked too much like a planet. While that wouldn’t have been a bad representation of the games’ scope, it’s too reminiscent of space and science fiction in general.


I was about to delete it when I threw some fronds on it on a whim, and liked the result. And the additional symbolism is nice: cycads are an ancient clade*, evocative of prehistory, nature and dinosaurs. All good evolutionary stuff.


We’re still on track to release 0.6.1 on February 1. 🙂



*footnote: if anyone uses the term “living fossil” I will personally inject live cockatoo’s into their bloodstream.** I generally don’t do this because it’s a bugger to find a syringe big enough, but I’ll make an exception.


**footnote: don’t worry, it doesn’t hurt the birds. It’s actually fairly close to how they reproduce in nature (they’re Australian).




, , , , ,



So, a topic that keeps coming up, on YouTube and elsewhere, is the concept of civilisation.

Now I’ve always said a ‘Civ Stage’ isn’t in the games scope, but I’ve never really explored in detail why that is. So let’s explore that concept, shall we? What would it take, hypothetically, to create a civilisation stage in Species? And even if we can’t do civilisation, what can we do?

(the usual disclaimer applies here: nothing I say is in anyway a promise to deliver future features. I’m just hypothesizing out loud)

First of all, in order to program something like Species we need hard definitions for terms. So what is “civilisation”? What marks it as different from just a bunch of smarty pants creatures living together? Keep in mind every definition from here on out is arbitrary, and provided purely for the sake of argument.

For civilisation, I’d say you need three things, which all start with an “s” BECAUSE REASONS: settlements, social heirachies and sapience.

Settlements are the simplest and easiest to define of these: they’re semi-permanent communal nests. Plenty of creatures build these: loads of different birds, bats, all hive-based insects, some species of spider…



The trick with settlements, of course, is feeding their population. You can effectively divide settlement species into two groups based on how they achieve this. Humans and bees do this with agriculture: ‘farming’ a renewable resource to sustain themselves on a (relatively) small territory. All other settlements consist of creatures capable of travelling great distances to find food, hunter-gatherer style: creatures like bats and penguins. 

Since the latter strategy means the creature has a lot less free time to develop complex social heirachies and sapience, it seems unlikely earth will ever have sapient penguins. This makes me sad. I feel like birds deserve a chance to be the dominant life form: penguins doubly so. On the other hand, we might well end up with sapient insects, which is…



Of the three requirements, settlements would be the easiest to implement in Species. We already have a few plans for a mechanism called “nests”: this would simply be an extention of that concept, either by allowing ‘improved’ nests to hold more than 1 creature or making creature’s build their nests in close proximity to each other. Bringing food they can’t immediately eat back to a nest, to store for later or share with other members of their species, fits beautifully into the existing game mechanics and could prove to be a winning behavioral strategy.

*Social heirachies* are trickier. See, one of the ‘rules’ of the game’s development is that there are no external constructs. You can see this in the species algorithm: it is entirely passive. You could remove the “Species” category from the game entirely, and it would make absolutely no difference to the simulation. From the simulations perspective, “species” don’t exist. They’re merely a category superimposed on the game for the benefit of the player.

The same goes for social constructs and behaviors. If they do form, it must be entirely contained in each individual creatures mind, which means any one creature’s perspective of the “tribe” or “pack” could very well differ from any other creatures perspective.

This means decisions can’t be made on behalf of the pack. Every creature is an individual. Even if they have a ‘pack leader’, there’s no guarantee that all creatures in the pack will be following the same leader, and no guarantee that if the leader starts doing something like hunting that the rest of the pack will follow it.

I couldn’t find a good “disobedience” demotivator

Naturally this gets even worse in a ‘civilisation’ format, where there can be multiple groups, and group relates to each other group in different ways. Without external constructs, managing these groups and their relationships becomes a matter of every creature having their own arbitrary perception of every other creature and what group they belong to, which would be incredibly CPU intensive and extremely unintuitive.

So we can’t actually have a complex social heirachy without external constructs. Frowney face. But… that thing I said before about “pack leaders”, and creature’s deciding on an individual level whether or not to follow them? That sounds like an AWESOME gameplay mechanic. Seriously, somebody write that down.

And finally, Sapience. The hardest one of them all.

(terminology note: all animals with brains are “sentient”, defined as the capacity to experience and “feel”. “Sapient” is the correct adjective for intelligent creatures like humans (which is called “sophonts”, a term I learned from Mass Effect. Who says games aren’t educational?)).

How do you define sapience? What even is sapience? “Intelligence”, “Judgement”, “Wisdom”, “Learning”? Fuzzy definitions. None of these help!

Since a realistic first-principles neural-network simulation of intelligence is well beyond our scope, we’d need to distil sapience down to something workable within the simulation. But how do you do that? What does a sapient creature do that non-sapient creatures do not, and how do we simulate that?

The truth is, I don’t think sapience is a description for any particular thing. Plenty of creature’s we don’t consider sapient use tools, build homes, develop social hierarchies, experience emotions, think through the consequences of their actions, and even empathise and grieve. Every time we think we’ve found something unique to humans, to sophonts, the animal kingdom proves us wrong.


We are animals, and not even particularly remarkable ones. Sapience is just a collation of different traits. It’s nothing special. (see also: “the universe doesn’t care about you”, “your hopes and dreams are meaningless”, and “one day you and everyone you love will be dead”. Yaaaay!)

Of course, this just makes our achievements even more remarkable. We as a species took a bunch of mental faculties that developed to hit each other over the heads with rocks, and with those we discovered the physical laws of the universe, uncovered the history of our planet, walked on another world, put a friggin’ car on mars. WITH A ROCKET-PROPELLED SKYCRANE.


Where was I? Oh right, simulating sapience…

… we’re not going to try to just dump sapience on Species with some sort of tech tree or an all-encompassing “intelligence” stat. That goes against our development goals.

Instead, we want to simulate the things that make up sapience. Each of those things I mentioned before: Tool use, Construction, Emotions, Forethought, Empathy: each of these is something that we can, at least conceivably, simulate.

We may never manage to do all of them. Not every feature is viable in a simulation being run on ordinary PC’s, afterall. And there is such a thing as being too ambitious.

But that’s no reason not to try. 😀

So, in conclusion: there’s definitely not going to be a ‘civilisation stage’ in species. That would effectively be a second game on top of the first, and the truth is I have no interest in making an RTS.

But we are going to try to allow for the components of civilisation: buildable nests/colonies, a level of social interaction, and as much intelligence as we can cram into their tiny heads before their beady little eyes pop out (and the CPU explodes).

These are all long term features, of course, but all of them can be implemented as emergent, evolvable behaviours, which is ultimately the entire point of Species.


, , , , ,


Torso Height

I’ve been playing with the torso height system.

As I’ve mentioned before, the creature’s are drawn from the torso outwards with their limbs ‘hanging’ off them, like a jellyfish or an octopus or the incomprehensibly alien True Form of our current Prime Minister [“He knows! Get him!”]. In order to give the impression of standing on their legs, the creatures in Species need to calculate a vertical offset called “torsoHeight” before they draw.

The previous system for establishing this offset was fairly simple: every frame, check each limb against the ground, and if none of them touch, reduce the torsoheight. Inversely, if the limbs are underground, push torsoheight upwards.

This dynamic system led to torsoHeight becoming one of the most abused values in the game: I’ve used it for gravity, anti-burrowing measures, the ‘bouncing’ when they’re attacked, etc. Unfortunately, this sort of abuse leads to emergent behavior, and not the good sort: this is the reason creatures float up out of the ground when they’re born, why they can fly when continually attacked by multiple attackers, and why they drop down onto their torso when the LOD stops drawing their limbs.

And more importantly, it’s why I haven’t been able to use torsoHeight for selection pressures. When two identical creatures can have different values based on Level Of Detail, using it would introduce strange artificial pressures based on where the camera was and what it was focusing on.

So for 0.7.0, torsoHeight is being replaced by a non-dynamic version: it will be calculated once at birth based on the creature’s skeleton, doubled when they grow to adulthood, and not updated again until they die, vanish and are recycled. I’ve broken it out into 3 components: ShoulderToCoG, ShoulderToAnkle and Tipheight.

Witness my magnificent artistry!

Witness my magnificent artistry!

It never as simple as it looks, of course: needed a bit of trigonomtery for ShoulderToCOG (why didn’t anyone tell me trig would be useful in school?), a few world-space transforms for converting bone matrices back into world co-ordinates, and I needed to find a way to test each component seperately because I tend to go through a lot of trial and error with this sort of thing.

But after a few hiccups with floating/burrowing creatures it’s finally starting to work out (though there’s still a few oddball cases that cause floaters), and eliminating those emergent properties I mentioned above makes things look quite a bit neater. Especially noticable is that creature’s no longer drift up out of the ground as you approach them, which makes it more apparent that the stuff popping in was there, but just not displayed before. It’s just polish, but it gives the game a slightly more professional feel.

And better yet, since torsoHeight is determined at birth now, I have a good solid foundation to work from and I can start considering what other parameters it’s going to affect… [insert maniacal laughter]


, ,

1 Comment

Out Of Memory Exceptions

I learned something today: Out of Memory errors are not actually caused by being out of memory! What a silly I am for expecting error messages to make sense.

So, for everyone preparing to set your RAM on fire, stomp on it and throw it out the window before burying it in a deep dark hole and pouring concrete on top: don’t bother! You RAM is fine. Assuming you haven’t already set it on fire. If you have, your RAM is not fine. And you might want to put that out.

No if you’re out of RAM you’ll get performance issues caused by ‘thrashing’ (when a computer starts relying on the Hard Drive instead of RAM), but you won’t get Out Of Memory exceptions. So, what causes OoM exceptions?

32-bit Windows causes OoM exceptions.

Screw 32-bit Windows.

See, when Microsoft says “you are out of memory”, they actually mean “you are out of consecutive address space”.

You now have several options (it’s like a game!):
– For a “proper” explanation, go read the article I read that taught me all this:
– For an “improper” explanation, go ahead and use your imagination (I’m sure it’s way more kinky than anything I can come up with).
– For an “incredibly-simplified and probably-wrong” explanation, continue reading…

When an object like a Creature or a Tree is assigned, it gets put in memory and Windows assigns it an address. And by ‘put in memory’, I don’t mean RAM. Forget RAM. RAM is a glorified performance optimisation. The memory we’re talking about here is your hard drive. And as you’re probably aware, your hard drive is huge. You’re not in any danger of running out of that unless you’ve packed the drive to it’s very limit.

Okay, so you’re not in danger of running out of Hard Drive space, and lack of RAM doesn’t cause exceptions, just performance issues. So the next place to look is at that address the object was assigned by Windows when it was put in memory.

This address is where windows get’s it’s 32/64 bit designation. The absurd oversimplification I’m about to make will likely make any computer-science types reading this swallow their hats, but it’s basically a number stating that object X can be found Y bits into the memory stream.

Always be prepared!

The cause of Out Of Memory exceptions is that “Y-bits into the memory stream”. See, even though the process has the entire hard drive to work with, a 32-bit address format can only store addresses up to ~4GB (and half of that is reserved by the operating system for other things). This is why 32-bit editions of Windows can’t make use of more than 2GB ram: the 32-bit format doesn’t have enough address space to map more than 2GB.

Okay, but surely that’s basically the same as a 2GB memory limit for the purposes of Out Of Memory exceptions?

Nope. See, to assign and map a new object, there needs to be an empty “hole” in address space for it to go into. If there isn’t a hole large enough, an Out Of Memory exception will be thrown. (finally!)

I’ll explain via metaphor. Imagine putting three baskets into your car, then filling in the area in between them with smaller items. Then you realise that you need a large esky instead, so you take out the baskets and try to put the esky in. Even though the three baskets combined took up more space than the esky, and even though they’ve been removed, the esky still won’t fit because the smaller infill items are blocking it.

That’s the problem with address space: if you fragment it with smaller items in between the big ones, the next time you need to allocate something big it’ll throw an Out Of Memory exception, even if you freed up the big ones and have plenty of cumulative space to use. *That* is what is causing the memory issues in Species.

I’m sure a decent Computer Science course would have covered all this but I’m self taught, so I get to learn by the slightly more direct route of sticking my hand in the fire and finding out what happens.


So, on to the usual question: how do we fix this?

Well, the obvious solution is “compile to 64-bit.” A 64-bit address space can map about 4 billion times as much memory (no, literally), so chances are we wouldn’t be seeing fragmentation errors any time soon. Of course, it also means people using 32-bit windows wouldn’t be able to play the game.

I’d rather solve the error at the source and retain backwards compatibility, but “solve” is a relative term. Memory fragmentation on a 32 bit machine isn’t something that can be stopped. But it can be significantly reduced, and I’d prefer to do that for the sake of memory management *before* dropping the metaphorical world-killer asteroid that is a 64-bit compilation.

The “World Killing Asteroid” compile-time option is only included in the December 21 service-release of Visual Studio 2012.

So, on to the source. What, ultimately, causes memory errors in Species?

I’d done memory profiling in the past, but I hadn’t properly understood what it was telling me until now. They indicated I had two main memory draws: grass vertices, and tree instances. But these two memory hogs always appeared in different graphs: tree instances would appear in allocation, grass vertices in snapshots.

With what I know now, this makes sense. Grass vertices are created when the terrain is built, and held from then on out. This means that the grasses memory cost at any moment in time is huge: after all, it’s storing the position of every potential billboard vertex on the map. Imagine the map completely covered in the dense savanna grass: that’s what the grass memory cost is.

But relative to the hard drive, or even the RAM, it’s not that big: I think somewhere around the 200MB mark? Can’t remember exactly, but regardless, it’s also generally a static cost. The grass grabs up all that memory when the terrain is built and doesn’t release it until the terrain is destroyed. You might be able to force an OoM exception by repeatedly loading/generating worlds, and fragmenting the memory that way. Fixing it isn’t urgent, but probably worth doing: need to ensure the memory is allocated at startup, rather than on terrain-build.

The second cost is the Tree Instances array. This one confused me because it was barely noticable on the moment-to-moment memory, but was invariably a massive cost in terms of amount allocated. This was because the Tree Instances array was designed to grow organically with the vegetation: if the number of trees grew too large for the array, it would allocate a new array with an extra 500 slots for trees and copy the data to that (well okay it actually uses Array.Resize(), but that’s what Array.Resize() does internally).

Of course, frequently allocating a massive new tree array is a great method for fragmenting your address space, which is why large (>2) worlds were invariably hitting an OoM exception during the vegetation generation at startup. Since I didn’t realise this was a problem, the amount allocated in the memory profiler didn’t mean anything to me.

I’ve already applied a few simple optimisations to this method, which makes generating size-3 worlds possible again, but fixing it will be another matter entirely. Ultimately, the vegetation should probably be capped to a set level, but this is tricky because the array is fed directly into the graphics card when drawing the trees. This means that those 500 “excess” trees still incur a GPU cost, even though they’re invisible. In a size 3 world, this cost is enough to cap FPS to 12, even when there are no creatures.

To fix this one I’m going to need to adapt the InstancedModel DrawInstances() method to take the total number of trees as a parameter, and feed that to an overload of the DynamicVertexBuffer. This should allow me to submit the entire oversized array, but then lower the number of instances that are actually drawn each frame.

(Edit) Well that was surprisingly easy. A heap of address space is now set aside for trees, but only the actual trees themselves are sent to the graphics card. Assuming XNA isn’t doing something horrible with them beneath the hood, that should take care of most of the OoM exceptions in the game. Stability improvements ahoy!


, , , , , , , , ,