Posts Tagged XNA
The code’s finally in a stable enough state that I feel justified in announcing a release date. So here goes nothing…
Species 0.6.1 will be released on the 1st February.
This update includes a lot of the cleanup and polish elements originally planned for 0.7.0. A lot of the changes are background
improvements to the underlying code, but that’s not to say you won’t see some changes to the game itself…
A variety of major 0.6.0 bugs have been addressed, most notably:
- several fatal crashes have been fixed (‘System.OutOfMemoryException’, ‘CladeDiagram.CollideBranches’, ‘UIManager.Paused_Export_Update’),
- the method for addressing problems when the game is minimised or graphics card is lost has been improved,
- built-in troubleshooting for some start-up problems (running from the zip file, running from Program Files, running on computer with unsupported graphics card) has been added
(Of course, it’s entirely likely we’ve added our own set of new errors to replace them, so don’t get too comfortable)
Quite a few visual improvements:
- Procedural texturing: creatures now have different top, side and underbelly textures, giving them a more distinct appearance and more recognisable body covers.
- Custom textures: beaks, claws, teeth and mouths now have their own (appropriately horrifying) textures, blended into the triplanar texturing.
- Fur: Fuzzy wuzzy was a bear, fuzzy wuzzy had no hair, but the creature’s in Species do now. And they find bear meat tasty.
- Water reflections: They only reflect the skybox, but they significantly improve the appearance of the game nonetheless.
Plus a number of subtler ones, like adjusted tree shaders, a better wasteland texture and we got rid of that weird blue shading on trees when you pause the game.
It’s not been all about code and shaders: we’ve also overhauled the content in some of the less-developed area’s of the game.
- Body coverings: fur, feathers, scales and skins. Also octopus suckers, toad warts and Quasars forearms (ewwww…), but we’re probably better off not mentioning those.
- Leg shapes: from 6 to 20, meaning a much greater variety of knees.
- Heads: a number of additions (and of course, the existing heads had to be adjusted to provide beak and mouth textures and fur polygons).
I’ve been creating Things That Should Not Be. Alright alright, I’ve actually just been modelling limbs, but the Things That Should Not Be are a direct side effect.
Ignore the graphical errors, such is the nature of the development version. I’m much more comfortable creating things when surrounded by unfathomable hideousness and strange transparency artifacts. It’s like Linus’ security blanket, except for me it’s non-euclidian geometry and broken texturing artifacts.
Or possibly I just have a tendency to break things and put off fixing them because coding is more fun than content creation. Irrelevant.
In addition to monstrous abominations of tentacled science, I’ve also been breaking and then fixing and then breaking again (repeat as required) a few other things.
I created a dedicated HighlightTint object for managing colour coding and status flashes on a per-creature basis. Different highlight causes now have priorities, and lower priority causes can’t override higher priority ones. This should fix those display issues with creature’s not showing that they’d been spliced or attacked because of the grazing or feeding cycles. Not the highest priority bug fix, but it came free with the code cleanup and I’ve always been a sucker for bargains (that’s a total lie, I don’t buy anything unless I’ve thought about it and decided it’s useful and practic- YOU CAN BUY QUADROTERS FOR ONLY $50?! GIMME!).
Let’s see… I also glared silently at the QuadTerrain generation algorithm until it surrendered to my will and reduced world-loading times by about 30%.
And I fixed a difficult problem that had been plaguing me for weeks, ever since I modularised the SpeciesList, which is that the species count wasn’t going down when creatures died. Found out it was caused by a typo adding Creatures to the Species twice. Easy fix.
And the Body Part Voting Competition has reached Round 7. It features a choice of heads from Avialae, which is objectively and indisputably the best-est clade. And once again, the choice I voted for is doing miserably.
I’ve been playing with the torso height system.
As I’ve mentioned before, the creature’s are drawn from the torso outwards with their limbs ‘hanging’ off them, like a jellyfish or an octopus or the incomprehensibly alien True Form of our current Prime Minister [“He knows! Get him!”]. In order to give the impression of standing on their legs, the creatures in Species need to calculate a vertical offset called “torsoHeight” before they draw.
The previous system for establishing this offset was fairly simple: every frame, check each limb against the ground, and if none of them touch, reduce the torsoheight. Inversely, if the limbs are underground, push torsoheight upwards.
This dynamic system led to torsoHeight becoming one of the most abused values in the game: I’ve used it for gravity, anti-burrowing measures, the ‘bouncing’ when they’re attacked, etc. Unfortunately, this sort of abuse leads to emergent behavior, and not the good sort: this is the reason creatures float up out of the ground when they’re born, why they can fly when continually attacked by multiple attackers, and why they drop down onto their torso when the LOD stops drawing their limbs.
And more importantly, it’s why I haven’t been able to use torsoHeight for selection pressures. When two identical creatures can have different values based on Level Of Detail, using it would introduce strange artificial pressures based on where the camera was and what it was focusing on.
So for 0.7.0, torsoHeight is being replaced by a non-dynamic version: it will be calculated once at birth based on the creature’s skeleton, doubled when they grow to adulthood, and not updated again until they die, vanish and are recycled. I’ve broken it out into 3 components: ShoulderToCoG, ShoulderToAnkle and Tipheight.
It never as simple as it looks, of course: needed a bit of trigonomtery for ShoulderToCOG (why didn’t anyone tell me trig would be useful in school?), a few world-space transforms for converting bone matrices back into world co-ordinates, and I needed to find a way to test each component seperately because I tend to go through a lot of trial and error with this sort of thing.
But after a few hiccups with floating/burrowing creatures it’s finally starting to work out (though there’s still a few oddball cases that cause floaters), and eliminating those emergent properties I mentioned above makes things look quite a bit neater. Especially noticable is that creature’s no longer drift up out of the ground as you approach them, which makes it more apparent that the stuff popping in was there, but just not displayed before. It’s just polish, but it gives the game a slightly more professional feel.
And better yet, since torsoHeight is determined at birth now, I have a good solid foundation to work from and I can start considering what other parameters it’s going to affect… [insert maniacal laughter]
I learned something today: Out of Memory errors are not actually caused by being out of memory! What a silly I am for expecting error messages to make sense.
So, for everyone preparing to set your RAM on fire, stomp on it and throw it out the window before burying it in a deep dark hole and pouring concrete on top: don’t bother! You RAM is fine. Assuming you haven’t already set it on fire. If you have, your RAM is not fine. And you might want to put that out.
No if you’re out of RAM you’ll get performance issues caused by ‘thrashing’ (when a computer starts relying on the Hard Drive instead of RAM), but you won’t get Out Of Memory exceptions. So, what causes OoM exceptions?
32-bit Windows causes OoM exceptions.
Screw 32-bit Windows.
See, when Microsoft says “you are out of memory”, they actually mean “you are out of consecutive address space”.
You now have several options (it’s like a game!):
– For a “proper” explanation, go read the article I read that taught me all this: http://blogs.msdn.com/b/ericlippert/archive/2009/06/08/out-of-memory-does-not-refer-to-physical-memory.aspx
– For an “improper” explanation, go ahead and use your imagination (I’m sure it’s way more kinky than anything I can come up with).
– For an “incredibly-simplified and probably-wrong” explanation, continue reading…
When an object like a Creature or a Tree is assigned, it gets put in memory and Windows assigns it an address. And by ‘put in memory’, I don’t mean RAM. Forget RAM. RAM is a glorified performance optimisation. The memory we’re talking about here is your hard drive. And as you’re probably aware, your hard drive is huge. You’re not in any danger of running out of that unless you’ve packed the drive to it’s very limit.
Okay, so you’re not in danger of running out of Hard Drive space, and lack of RAM doesn’t cause exceptions, just performance issues. So the next place to look is at that address the object was assigned by Windows when it was put in memory.
This address is where windows get’s it’s 32/64 bit designation. The absurd oversimplification I’m about to make will likely make any computer-science types reading this swallow their hats, but it’s basically a number stating that object X can be found Y bits into the memory stream.
The cause of Out Of Memory exceptions is that “Y-bits into the memory stream”. See, even though the process has the entire hard drive to work with, a 32-bit address format can only store addresses up to ~4GB (and half of that is reserved by the operating system for other things). This is why 32-bit editions of Windows can’t make use of more than 2GB ram: the 32-bit format doesn’t have enough address space to map more than 2GB.
Okay, but surely that’s basically the same as a 2GB memory limit for the purposes of Out Of Memory exceptions?
Nope. See, to assign and map a new object, there needs to be an empty “hole” in address space for it to go into. If there isn’t a hole large enough, an Out Of Memory exception will be thrown. (finally!)
I’ll explain via metaphor. Imagine putting three baskets into your car, then filling in the area in between them with smaller items. Then you realise that you need a large esky instead, so you take out the baskets and try to put the esky in. Even though the three baskets combined took up more space than the esky, and even though they’ve been removed, the esky still won’t fit because the smaller infill items are blocking it.
That’s the problem with address space: if you fragment it with smaller items in between the big ones, the next time you need to allocate something big it’ll throw an Out Of Memory exception, even if you freed up the big ones and have plenty of cumulative space to use. *That* is what is causing the memory issues in Species.
I’m sure a decent Computer Science course would have covered all this but I’m self taught, so I get to learn by the slightly more direct route of sticking my hand in the fire and finding out what happens.
So, on to the usual question: how do we fix this?
Well, the obvious solution is “compile to 64-bit.” A 64-bit address space can map about 4 billion times as much memory (no, literally), so chances are we wouldn’t be seeing fragmentation errors any time soon. Of course, it also means people using 32-bit windows wouldn’t be able to play the game.
I’d rather solve the error at the source and retain backwards compatibility, but “solve” is a relative term. Memory fragmentation on a 32 bit machine isn’t something that can be stopped. But it can be significantly reduced, and I’d prefer to do that for the sake of memory management *before* dropping the metaphorical world-killer asteroid that is a 64-bit compilation.
So, on to the source. What, ultimately, causes memory errors in Species?
I’d done memory profiling in the past, but I hadn’t properly understood what it was telling me until now. They indicated I had two main memory draws: grass vertices, and tree instances. But these two memory hogs always appeared in different graphs: tree instances would appear in allocation, grass vertices in snapshots.
With what I know now, this makes sense. Grass vertices are created when the terrain is built, and held from then on out. This means that the grasses memory cost at any moment in time is huge: after all, it’s storing the position of every potential billboard vertex on the map. Imagine the map completely covered in the dense savanna grass: that’s what the grass memory cost is.
But relative to the hard drive, or even the RAM, it’s not that big: I think somewhere around the 200MB mark? Can’t remember exactly, but regardless, it’s also generally a static cost. The grass grabs up all that memory when the terrain is built and doesn’t release it until the terrain is destroyed. You might be able to force an OoM exception by repeatedly loading/generating worlds, and fragmenting the memory that way. Fixing it isn’t urgent, but probably worth doing: need to ensure the memory is allocated at startup, rather than on terrain-build.
The second cost is the Tree Instances array. This one confused me because it was barely noticable on the moment-to-moment memory, but was invariably a massive cost in terms of amount allocated. This was because the Tree Instances array was designed to grow organically with the vegetation: if the number of trees grew too large for the array, it would allocate a new array with an extra 500 slots for trees and copy the data to that (well okay it actually uses Array.Resize(), but that’s what Array.Resize() does internally).
Of course, frequently allocating a massive new tree array is a great method for fragmenting your address space, which is why large (>2) worlds were invariably hitting an OoM exception during the vegetation generation at startup. Since I didn’t realise this was a problem, the amount allocated in the memory profiler didn’t mean anything to me.
I’ve already applied a few simple optimisations to this method, which makes generating size-3 worlds possible again, but fixing it will be another matter entirely. Ultimately, the vegetation should probably be capped to a set level, but this is tricky because the array is fed directly into the graphics card when drawing the trees. This means that those 500 “excess” trees still incur a GPU cost, even though they’re invisible. In a size 3 world, this cost is enough to cap FPS to 12, even when there are no creatures.
To fix this one I’m going to need to adapt the InstancedModel DrawInstances() method to take the total number of trees as a parameter, and feed that to an overload of the DynamicVertexBuffer. This should allow me to submit the entire oversized array, but then lower the number of instances that are actually drawn each frame.
(Edit) Well that was surprisingly easy. A heap of address space is now set aside for trees, but only the actual trees themselves are sent to the graphics card. Assuming XNA isn’t doing something horrible with them beneath the hood, that should take care of most of the OoM exceptions in the game. Stability improvements ahoy!
There’s not too much to say about the obvious mechanisms of the 0.5.0 in-game grass: it uses the billboard system which I explained a while back. I could mention the other technical details: it only draws a few, nearby, vegetation nodes, grazing affects the length of the entire node, it fades out at a distance because there’s way too much of it to draw it across the entire map… okay, done that, now what?
Well, seeing as how talking about implementation is boring, I guess we might as well talk about the design aspects.
The moment I decided on grazing as a feature for 0.5.0, I knew I’d have to represent it somehow. The grass itself, though, wasn’t originally meant to be that representation. Indeed, I’m still not entirely convinced that grass is the best representation for it: rendering grass has limitations that make it less than ideal.
Okay, I’m in the middle of typing up this post and I’m beginning to realise that the height of the billboard grass really isn’t the best way to represent grazable material. That’s what I get for blogging about a feature I’m in the middle of coding. Oh the joys of an evolving project.
Okay, new approach: I’m going to use this post as an opportunity to get my thoughts in order before I go off and play with the code a bit more.
The plan (prior to about about 30 seconds ago) was to include a grazables container or bucket within each square of terrain. This container would contain all the energy that creatures could graze from, and how ‘full’ it was would determine how long the grass in that square was. Fertility loss due to grazing would occur when the bucket was empty and had to ‘buy’ more energy to regrow.
The big pro to this approach is that grass height shows you at a glance how much grazable energy the local area has. Unfortunately, there’s also a lot of cons:
– all grazable scatter-material grows like grass and is edible, regardless of what it actually looks like. This includes pebbles on rocky terrain, shells on the beach, salt in a salt plain, and lava rocks in lava.
– grass in an area is all the same height. Since an ‘area’ is an exact square of roughly 10m x 10m, this would be especially noticable at borders where grass on one side is short and grass on the other is long.
– fertility loss is applied on an area-scale, not on a local one. A creature eating at the very corner of an area will affect the fertility of ground 14.1421m away (+/- 10m, anyway. Oh who am I kidding, I have no idea how big the vegetation squares are), while not affecting the ground just behind it.
– Grass is invisible at a distance, so you can’t see the direct effects of grazing from far away.
Now, all of these are things that can be dealt with to eliminate or reduce their effect: scattering a squares vegetation a bit beyond it’s border would blur the straight line between squares, by applying fertility loss on a macro level makes it less apparent that it’s related to overall area and not to the actions of individual creatures.
But what if we could deal with all of these problems just by changing the way ‘grazable energy’ is stored? This is the idea I’ve just had:
Eliminate the energy buckets in terrain squares, and effectively remove all terrain-based control over grazable energy. Grass no longer has any limit: creatures can just keep grazing and grazing within their biome… until the biome changes.
With this system, a creature emits a ‘death aura’ while grazing, gradually reducing the fertility in a small area directly underneath themselves. Eventually, the biome under them degrades. This introduces a direct correlation between fertility and energy: a creature absorbs fertility from the ground and gains energy in exchange.
Since grazables are no longer dependant on area but on biome, we’ll be able to introduce a variety of biome dependant statistics (starting with a simple isGrazable boolean) as a central function of the simulation, rather than as something tacked on afterwards as was originally planned.
A conventient bonus is the fact that the ‘death aura’ code already exists, in the form of biome stabilisation from trees. The only difference is that where tree’s stabilise the habitat they’re best suited for (with the exception of some unbalanced pioneering species which make the simulation more dynamic by stabilising towards biomes they can’t survive in), grazing creatures stabilise the habitat towards arid, desert biomes, and then have to move on to find more grass.
Of course, I’ll have to rework some of the code for this: the ‘buckets’ system already exists in the development version. But that’s the nature of prototyping.
[The following day]
Welp, that’s done. This actually makes the environment feel a lot more ‘directed’, since you can now pinpoint the source of every fertility change: it’s either grazing creatures, trees or water. I might have to add a few more fertility-change sources, just to make it less predictable.
Ultimately, this change leaves the grass itself as little more than an aesthetic item. Oh well: the system’s in place now, it’s useful in defining the presence and quantity of grazable material in each biome, and when even the placeholder art looks good, you know you’re doing something right..
He doesn’t mean it about the vegans. We actually think vegans are pretty awesome: it must take a lot of willpower and strength of conviction.
Please don’t kill us with your psychic vegan powers.
One of the priorities for 0.5.0 (the environmental update) has been to overhaul the multi-texturing system.
In 0.4.1 the best the engine could do was 4 textures (desert, grass, forests and cliffs), and those textures were placed at world-start, never changing. The end result was a very static map, very boring environment: basic tree loss/growth was the entire extent of environmental dynamics.
All this changes in 0.5.0. The design plan calls for a dynamic, responsive environment, and by crikey that’s what we’re going to have. (Crikey is Australian for… something. I dunno. Honestly I don’t even know what the word means, but it’s a stipulation of citizenship that we all say it at least once a week)
The first thing to add is more biomes. Desert/grass/forest works well as a basic, semi-tropical biome set, but what about colder biomes? Or warmer ones? Or wetter ones or drier ones? My current biome map concept has more than 20 different biomes drawn on a temperature/fertility map, including some hilariously extreme ones
But that many biomes means a whole different approach to multi-texturing… or at least, I thought it did. This was when development started to go bad…
In 0.4.1 I used a multitexturing method called ‘texture splatting’. Imagine you’re painting the ground texture on a blank canvas. What texture splatting does is gives you a ‘stencil’ for each texture: so you can paint forest through the red stencil, grass through the green stencil and desert through the blue one, and by the time you’ve used all the stencils you’ve painted the entire terrain. All these stencils are nicely wrapped up into a single texture, called the blend texture.
Unfortunately, this only works for a limited number of textures per draw call: you can’t load more than a set limit of textures onto the GPU without it having a fit. So in order to render an unlimited number of textures, I’d need to change how I was approaching it.
My first attempt at a replacement system was sort of like an extremely complicated colouring book, where each grid square is numbered 1,2,3,4,etc and our hypothetical artist fills in the grid with grass/desert/forest from a colour key. Since we can include as many numbers as we like in the key, we can have as many biomes as we want.
Once that’s done, our artist is faced with a problem: (s)he has to blur these colours together smoothly without pixilation or sharp edges. And since it’s going to be animated, a simple linear interpolation just isn’t going to cut it.
As it turns out, this is HARD. My own approach was to sample the biome-legend multiple times to give each pixel *four* biomes, each with a weighting value, which I then folded into the same channel as the key
(effectively, I designated the first digit of the channel as the key, and the remaining digits as the weight).
Honestly, the most surprising thing about this ridiculously complicated strategy is that it worked… mostly.
But “works in general”, doesn’t mean “works well enough to use”. On closer inspection, the system produced artifacts all over the terrain. (not a typo: graphical glitches are “artefacts”, ancient relics are “artifacts”. I read that somewhere and internalised it, so it must be true) I actually got it working perfectly wherever 2 biomes met, but that was the limit: at any intersection with more than 2 it produced hard edges and odd colours, and the moment it was animated these artefacts started jumping about and generally making themselves easy to spot.
So I ended up scrapping the idea entirely, and starting over with a new strategy, one requiring less math and more art.
This new strategy was much simpler: we go back to using the ‘stencil painting’ system, but this time once we’re done painting with the first 4 stencils (biomes), we put a new, transparent canvas over the existing canvas and keep on painting on that with a new set of stencils. Rinse and repeat.
This method turned out to have it’s own set of pitfalls, chief among them, alpha-blending and redrawing the entire terrain multiple times, with different textures each time. For an item which takes up as much of the screen as the terrain, this is a large graphics cost, and in a GPU-bound game probably would have spelled the end of this strategy. But Species is CPU-bound: it has GPU cycles to spare. So full-steam ahead.
The artefacts of this method also turned out to be quite different to the ones I faced with the other method. The other method loved to produce singular, localised artefacts: hard edges and biome colours where they shouldn’t be. This method’s artefacts usually affected the entire terrain. I’d say two in particular are worthy of noting here, mainly because I haven’t actually managed to fix them: double rendered polygons and biome-set edges.
Biome-set edges were where one transparent layer tried to fade out into another. I never had any trouble with the inter-set blending, but proper alpha blending is a temperamental thing. In this case, because the biome colour fades out at the same time as the opacity does, the end result was a faded-but-noticeable black ‘border’ between different blend-sets.
I managed to ‘fill-in’ the majority of these borders by extending the colour to the very edge, but there is still a faint one around the first draw pass. Thankfully it’s subtle, and dealing with it had some odd side effects, so I’m going to leave that one alone. It’s not hugely noticeable.
Double-rendered polygons, on the other hand, are a problem.
This isn’t actually a problem with the rendering method: it’s a problem with the QuadTerrain itself, which I didn’t know about until the rendering method made it visible. See, when the terrain is completely opaque, rendering a polygon twice has no effect. The colour from both renderings is the same, so it’s an invisible artefact. But when you render a *transparent* object, like, say… one of the 4-biome passes of the terrain… *then* it becomes quite visible, as you can see above.
But fixing it means re-familiarising myself with the QuadTerrain class, which I haven’t touched in quite some time. I’ve already made a little progress, eliminating about half of the artefacts above with one fix. Hopefully the next fix will get the rest, but I doubt it: bugs like this are often inversely exponential. You might be able to fix the majority of them easily, but there are always one or two subtle, extremely well hidden ones that you have next-to-no chance of ever finding.
Oh well, best I can do is to make sure I get most of them.
Currently, I have 23 biomes defined as rough shapes on a temperature/fertility axis. This includes ‘extreme’ biomes, like lava and salt-plains, and a number of underwater biomes that will only be found… err, underwater. (Fertility will at least partly be determined by height: everything below the water plain will be water).
So far everything is coming together as planned. Biomes are done, 3d trees are done (I’ll do a proper post on them later), I’m midway through tying the two together, and soon the nanozombies will be unleashed on the unsuspecting… oh wait wrong blog. You didn’t hear that.
“Other stipulations of Australian citizenship include:
– Must defend vegemite no matter personal opinion of taste. (sticky salty gunk)
– Must be willing to throw foreign tourists in front of a croc to save yourself. (Or was that the other way around?)
– Must know how to defend against drop-bears (trick question: there is no way to defend against drop bears)”