Sunday, April 30, 2017

One step forward, Two steps back

It took a bit longer to make a new post. Not necessarily because I've been busy or whatsoever...  Time just flies! When getting out of bed, I'd better lay down again because the day is over already. And yes, like any programmer geek, my nights are short.

When writing a new topic, there has to be a topic to start with. Obviously a lot has been said already on this blog. Curtain shaders, game music, stair-climbing physics, Super Mario, pathfinding, you name it. If it's not somewhere in this blog archive, it has nothing to do with games.

On the other hand, the "should I post this???" bar has been raised quite a lot. Five years ago I would share my new bumpMapping shader, or devote a topic just to show a new metal-barrel 3D prop. Nowadays, games look so darn good that I can't come around with some silly 3D brick texture, or a dull dusty room - your typical Tower22 stuff. In this age, it’s just very hard to impress. My homebrew stuff isn’t impressive anymore for today’s standards.

Or well, that's how I feel about it at least. I'm not the type of guy who shares every restaurant meal or aftermath turd on twitter or Facebook. Heck, I don't even have an account on any social media. In general I only open my mouth if there is something truly informative or pretty to share. Or if I can't resist making stupid jokes. But none of that got triggered last month, so... no new topics.

Tinkering G.I.
No development then? Well yes. Made a few more (dummy) rooms for the playable demo, and got on a dead end playing with some other G.I. approaches. Every year or so, I get bored on my current "Indirect Lighting" method, because none of the existing methods is perfect... far from. Engine22 uses Lightmaps (& probes for dynamic objects) at the moment, but there are some seam-stitch artefacts, normalMapping issues, and of course dynamic objects don't play well with lightMaps. So I tried a probe-only-approach, but found the end-result to be disappointing, quality-wise. Spend a few weeks implementing them, only to throw them away with the garbage again, grabbing the dusty Lightmap-box back from the attic.

Probes can work pretty well -I used them in the Tower22 "Subway" demo clip-, but they lack accuracy and leaks are hard to prevent. And since the average Tower22 scene is a small room with a lot of crap, having probes every 0.4 m3 isn’t dense enough. At least not when you got used to lightMaps, which can be more accurate.

Tinkering Parallax effects
So, that was a waste of time basically. What else did we do? Re-Implemented "Parallax Offset Mapping" (POM). I wrote "re-", because this technique has always been present here, but not yet in the new Engine22, as I placed my bets on Tessellation instead.

POM offsets UV coordinates (per pixel) by sending out a ray towards the camera, checking for intersections with a surface heightMap. Neat trick, but still fake. Tessellation actually sub-divides the surface in many tiny triangles (if the camera is close enough), and then offsets the (new) vertices based on that same heightMap. Even though it works surprisingly fast I'd say, you can't sub-divide far enough to reach a per-pixel level, resulting in blocky/jaggy edges. And worse, since vertices actually go back- or forward now, it can punch holes in your walls, and furniture will either float or sink above/within the displaced geometry. Conclusion, it just doesn't look that good. Of course there are a billion optimisations out there, but for what... adding 5 mm depth in some floor tile gaps. Hurray.

You've seen this corridor a billion times already, but probably it looks more interesting this time. Not because brilliant new shaders, better G.I. or improved 3D props, but because a rather simple "dusty overlay" plus a blue-ish ghostly effect.

I learned that technical tricks alone can't boost graphics. Sure they can help, but in the end it's all about the composition, the scene as a whole, where every prop, detail, decal, lightSource or particle complements. Not a single Tower22 room looked good right from the start. More the opposite. Breeding a plan + 3D geometry for a new room for 9 months, only to give birth to the ugliest babies. Wondering if it would ever be all right until the last lamp was replaced for the 60th time.

I don't know if artists have the same experience with more professional tools like a Crysis or Unreal engine. Asides from having a more complete toolset, they probably gear towards getting your preview as quick as possible, so an artist can correct the steering wheel early on. But even so, all the High-Tec is still no guarantee for a good looking scene. There needs to be a good collection of 2D/3D props, as well as an art direction. So, to sum it up: 
               Technology + Idea + Artwork

Well, and that is where this demo is a bit stuck now. I think I programmed all the techniques needed, or collected the major technical ingredients / editing tools at least. Of course things can be improved, but probably that also depends on what the artist needs. There is no artwork though, asides from the "dummy" props I made so far. Those are all place-holders really, mainly to show whatever artist who decides to help, what the "Idea" is all about, roughly. Apart from a few paper bits, this Idea was mainly between my ears in the past, making it very hard to show an artist where to go. It's like telling your contractor to "build a nice home", without giving any further blueprints.

This time the demo has been made, mostly. Hell, we can even play it. Parts of it at least. I'd like to keep some open ends, because honestly, the idea is not rock solid yet. With that I mean the design of the rooms can (should) be tweaked, and moreover, the exact demo-play-sequence needs to be tested and modified to make it as horrific as possible. Horror is a complex thing. Ever tried to explain a nightmare that scared the shit out of you? It's almost impossible to transfer that feeling to another, unless you have strong visuals or master the pen like Stephen King. All I have is Delphi and this blog.

So I've been waiting. When asking artists to help, the Technology & Idea need to be in a somewhat mature stage (unlike my previous attempt, years ago). But when is that? When is the technology far enough? And how to make sure your Idea will attract others? Today’s technology is old tomorrow, and it's very hard to show your Idea without proper artwork. Chicken Egg story.

Ready? Set? Go?
So right now, I'm more or less balancing when & how to ask for help. There is not too much anymore I can improve with programming alone, but then again, once I ask, there is no way back. I can't invite skilled people, then to let them wait because the engine needs to be fixed first, or let them change their work every week because the Idea is fluid like apple juice. And not unimportant, with 2 kids and 2 jobs, it's not like I have nothing else to do. Probably anyone who started or thought about starting their own business, knowing when you're ready is difficult. It's scary. Out of the comfort zone, at your weakest. BUT: No Pain, No Gain.

Sunday, March 19, 2017

Save Load Save

1 Continue Left
Don't know how old you are, but when I was your age, we used to play games with our bare hands. No health recharge, no "switch-to-easy?" after dying 3 times, no IDDQD, and no AutoSaves. Hell, no saves at all!

Hard to imagine, but indeed some of the old PC, NES or even later SNES games I played, didn't give you the opportunity to save, switch off the system, relax, and go play outside. No sir. Finish what you started, dammit. Countless times being in the very last level, finally, nerves up to the throat, and then mom yelling "bedtime! NOW!". Jesus, now what?! Restarting was not an option, so we had to turn off the TV but leave the system on, sleep, go to school, and hope mom didn't turn off the system (and hope the Nintendo wouldn't catch fire). And of course, next day, 16 hours later, we died over some ridiculous jump or impossible end-boss anyhow. Some games showed some mercy and would let you restart the level, but mostly you just had to replay the whole fucking game once the “Continues” depleted. Developers were soulless beings. Or maybe they were just too stupid to make a SaveState back then...

Well, as a cold comfort, most games weren't that long those days. If you include all the dying, restarts, flying gamepads, swearing and berserk kids, a game could easily take days or weeks to finish. But once you knew the trick, the average game could be finished within a few hours or less. And the somewhat better games usually had a Password system though. Or at the very least, gave you some cheats to skip sections, like the Warpzones or Magic Flutes in Super Mario Bros.
 Axe, Axe, Heart, Turd, Heart to head on Dracula's castle.

The Hard-coded magical Flute
But that was then, this is now. Glad as I am that the Tower22 playable demo finally has enough content/length that saving progress would be a nice thing, I certainly don't want to play a goddamn flute to skip parts.

“Saving & Loading” is probably not the first article you will find when learning about "Game Programming". It's one of those "yeah, whatever. Some day later I'll bother." features. Usually I just hard-coded some cheats or hacks to start elsewhere in the game, magically unlock doors, or give me item-X. That may work fairly well for level based games, but Tower22 is just one big world. And to make things worse, it's sort of a puzzle game. So a lot of conditions and events depend on stuff that has been collected or triggered earlier on. Complex "chains" like these quickly get tedious to overrule with hard-code, so basically I just had to replay the whole game to test if Script-X at the end of the demo works... which didn't because of 1 typo in the LUA script. 0 ups, game-over restart. Just like the good old days.

Yes, about time for a Save/Load system. But honestly, I never wrote a Save/Load before. Well, I did for other programs, but not games specifically. Usually the games I wrote weren't really playable, or had less than one level content ;) So what I'm about to write might not be the most solid solution, then again I'm guessing there is no universal standard answer on this topic anyway.

Thankfully, I sort of anticipated on this early on, so the engine had some tricks and mechanisms prepared. Which sure wasn't a bad idea, because I think for a more complicated (puzzle)game like this, where a lot of various stuff has to be saved in order to pick-up, it's actually quite a fundamental part of the engine entity systems, filestreams, and so on.
Save – Just a little bit
Correct me if I forget something brilliant, but I'd say there are three ways to deal with saving to begin with. Option1 is pretty simple: just save some numbers like the current level, and remaining lives (called "ups" back in the Good 'ol days). It's enough information to get us started in level-X. But what if your levels are huge, or if you don't have levels at all?

In that case you probably want to save a bit more information. Option2 does the same thing, but in more detail. You would save details like the actual player position, unlocked doors, his inventory, whether collectable items were picked yes/no, and maybe the status of your enemies (if & where they got(killed)). Now you can resume without having to restart, re-collect, or re-kill all your opponents.

Option2 is still limited though, and requires a good portion of very specific code. What needs to be saved, and what doesn't? And I'm talking about killing... but maybe you were making fishing game or something. When making engines, you don't (shouldn't) know such details, meaning you'll have to offer a more abstract solution.

Option3 is pretty hard-core: you literally dump the entire state of your world/level and every entity inside. All exact positions, bullet holes, destructed environment, caught fish, and so on. Works pretty well if your levels aren't too big (or if there is just very little you can alter), and is often used for shooters and such. And talking about old shitty games that couldn’t be saved, an Emulator can! It dumps the exact memory(“RAM”) state. However, SNES RAM was about 128 kb, thus resulting in tiny files as well. But don’t try this at home with a modern game, that may eat 3+ GB of RAM!

Oh, and I forgot option4: just don't save at all. Easy Peasy. But seriously, in case you are still baffled the devs didn't give proper "Save" options back then; old systems like the Gameboy or NES gave you a bunch of (kilo)bytes on a cartridge, go figure.

What the hell is that?! That, my grandson, is a SNES game. Didn't have Steam to download games, even CD-ROMs were yet to be born. Games looked more like computers back then, having a ROM chip and their own (battery backed) SRAM to store savegames. I always wondered why some games were almost 30 dollars more expensive than others. Well, as you can see, some games actually contained more physical hardware, like extra RAM memory or a SuperFX chip to render 3D stuff. So if your game didn't let you save anything at all, it was probably because those cheap assholes saved on a SRAM chip.

Save - anything Dynamic
I think modern games use option2 in many cases. But instead of saving a bunch of special numbers, it saves the attributes (position, state, special properties, …) of anything Dynamic – stuff that can change. Static stuff – stuff you can reload from its original source, such as sounds, textures or the World Geometry (unless destructible) can be ruled out.

The cool thing is that you can do this in the same manner as your Map-Editors would save the files… but preferably without the map geometry itself. Why? Well, if your mapfile is 500 MB of "level data", a SaveGame file would be equally big. Now storage space isn't our biggest enemy anymore, but it takes quite a while to load all that, and these days you may want to store these files in a Cloud as well, which still makes them (too) big. And don’t forget, a modern game saves your progress every 30 seconds or so. Don’t want to send those poor kids more than a minute back in time, if they gang-raped by a horde of demons again!
Yellow stuff = static data and/or initial states of dynamic data. Blue stuff = computed on the go, green stuff = data that can potentially change, and therefore has to be stored in a SaveGame file.

The Cleaning Crew
Another less technical issue then. What IF everything would be stored "as it is" in a game like GTA? The whole city would turn into a gigantic wrecking yard! Bumped lampposts, burned cars, crashed helicopters, busted hoes. Sounds pretty cool, but what if a trainwreck blocks your way into the building your next mission needs you to be? Exactly, like in the Good 'ol days, you can suck it and restart the whole game. And no, unlike SNES platformers, GTA doesn't take 2 hours to complete.

Another example, just for the fun of it, what IF you would shoot all space-crabs, flying lava dragons, and other insect-scum in a Metroid-like Exploration game? The game would become pretty boring, and in case of Metroid, even more lonesome once you killed all life on the planet!

No, that magical cleaning crew that repairs the lampposts, cleans the streets, replaces the cars and clears homicide scenes are actually quite welcome. What really happens though, you walk/drive/fly far enough from a certain spot, so it becomes "unloaded". Then when returning, the original scene -as originally created in the MapEditor- gets reloaded again. Clear from mistakes, forgiving your sins.
Please clean-up the Millenium Falcon on Isle Five.
Local versus Global entities
Tower22 does the same. You can run over a porcelain vase, but the cleaning ladies are kind enough to clean up the mess and place a new one - if you are not looking, they're shy. However, some entities should stay as they were. Unlocked doors should remain unlocked typically, collected items should stay gone, solved puzzles should stay solved. And Bosses should stay dead, definitely that.

In Engine22, when making maps in the editor, every entity (furniture, walls, items, monsters, lights, particles, and so on) can be marked as "Global" or "Local". Local entities are bound to the Sector they’re placed in, which is typically the room or corridor it stands in. When (re)loading that piece of the map, it will also load all of its local entities back in their original state (and yeah, a script can still move/hide/alter them afterwards, if needed).

Global entities on the other hand aren't part of a specific Sector. Global entities can eventually move throughout the world, for example, being carried as an item by the player. Global entities can also remain stationary, but store a certain state or other variables. Under the hood, each entity can have a property-list, or "Blackboard" as we call them now in BehaviorTree terms. This Blackboard is just a bunch of keys + values, like "doorLocked := false", "fuelLevel := 45", or "puzzleCompleted := true". Local entities can also have a blackboard, but as said, its properties will reset back to default once being reloaded.

Global entities are not saved in Sector(map-piece) files, but in 1 global file... which is pretty much the "SaveGame". When starting a new game, you actually also load a game, but filled with initial states. Then when moving on, you alter that data, and save it under a different name: your SaveGame, Slot, or whatever you would like to call it.


SaveGame file content
So résumé, how does the "Save" file actually looks? In my case it’s a binary file, with all global entities stored. This is done in the exact same way as the MapEditor would do. Entity classes have their own "readFromStream" / "writeToStream" functions and may differ in detail, but in general they would store a matrix, references like which material, sound or objectData has been assigned, its current state, eventually defined by a Blackboard (property-list). Speaking of which, there is also a Global Blackboard stored, which contains all global "game variables", that can be shared amongst all entities. Also handy, entities can also refer to each other. For example, entities like the Player or a storage trunk can have an "Inventory", which is basically a collection of items, thus pointers to other entities.

Last but not least, the engine uses a callback to let you -the Game- write custom data. In the case of Tower22 that would be the clock/calendar (which day are we, how late?) and additional non-engine attributes related to the player.

It works, though I must warn about the snakes and scorpions in this method. First of all, it’s a binary file. The good part is that ordinary teenage pukes can't mess around with those files that easily, and binary files are relative small compared to text-based files such as XML. They also load a whole lot faster. You’re not parsing stuff, you just suck stuff straight into RAM objects / arrays.

Really, I always wonder what the hell games or other programs are waiting for when hitting "START". When I click the RUN arrow in Delphi, I'm playing Tower22 4 seconds later. And yes, hundreds of (texture, 3D, material, audio, world geometry, ...) files have been loaded then. I dare to say modern software has become extremely lazy when it comes to RAM usage, file-loading, and so on. My laptop has 16 times more memory and CPU than ten years ago... but Apps also have become 16 times shittier in terms of resource management. Conclusion, everything still sucks.

So yes, I'm a binary fan. But there is one BIG BUT; they break easily. Just 1 wonky bit is enough to screw up the whole loading process, and giving you a headache trying to fix the file. Now if your binary format is final, and if you can guarantee the save-process doesn't make mistakes, binaries are your friend. Otherwise, expect trouble. Here a possible scenario. You finished your Minecraft game and sold 800 billion copies. Fine and dandy, but now the community cries for a new game element: an attribute that describes if a cube stinks or not. Being binary, you can't just add another boolean in your per-cube properties. Older versions wouldn't understand this file-format and basically read too little bytes per cube, newer versions won't understand older files, and basically read too many bytes per cube. Making the program crash.

You forgot to add some sort of "version" identifier prior to reading your cube-data, with as a result hundred millions of enraged teenagers. Kids lost their 8th-world-Minecraft-wonders, jumped in front of trains. You lost all your money, your wife ran off with that big black dude, and your mother hates you too.

That's one possible scenario. Now you could have avoided this by adding a version or something, so that your program can switch between 2 reading strategies, but what if you didn't have version numbers at all in your original files? Use them. Always. In my case, every (sub)instance that contributes to a FileStream, first writes a version byte. It could be that 99,9% of the contributors didn't change, but one specific entity type did, so I only have to alter its write/read procedure based on its own subversion.

Still, more structural adjustments can be a pain in the ass, and if you have a bug in your filewriter, it’s very hard to fix corrupted files. Think about those raged teenagers.

This is a higher level loading procedure; a Sector (map part) reading all its sub-entities (walls, floors, lights, eplosive barrels, sprites, ...). Note it first reads a version byte, which tells me how to anticipate on what follows. If something would ever change, I can "if ( version < 10 ) then ...". Entities on their turn do the same thing, inside their "ent.loadFromStream" procedures.

Incompatible Save states
Speaking of versions. Another problem my system has, are possible differences between dated SaveGames and newer versions of the game. In the playable demo, you have to collect trash at some point (not kidding). But I forgot to mark that trash as "Global" entities. So if you reload the game, all the trash would be back, even if you already collected them. Which was actually a nice bug, as you could cheat and reach the "18 items-collected" target easily. So, I fixed that by marking the cola cans, banana peels and blood-vomit pools as Global in the MapEditor. But, now all trash was suddenly gone. It wasn't stored in my local sector files anymore obviously, but neither in my SaveGame. Result? Couldn't collect all trash & finish the puzzle. The only fix was to restart the whole fucking game - like in the Good 'ol days. Doh.

I don't really know how to fix that -easily, without writing crazy hard-coded patches in the load procedure. One more sophisticated method might be not to simply dump all Global Entities into a file, but to write their "Mutations". So you would always load the initial "original state" file first, then go through a list of alterations. EntityX moved from A to B, Trash entity #123 deleted, Poor Monster's health changed from 100 to -1, and so on. That would at least respawn the trash. Heck, you could actually rewind all your actions like an UNDO function!

Yet, there is still no guarantee that this works. The initial state might be different in a newer version of the game, leading to certain events that weren’t covered in your dated file. You can’t reproduce the same state if certain actors were missing in your “flight-record” data. Plus I'm guessing this mutation-list gets veeery long, veeery quick. Basically you can’t just change the puzzles, then expect your SaveGame still works well.
Slept well Samus? Metroid would save your inventory, which bosses are dead, and which passages have been revealed, I think. But you could get locked if the map or some of those passages would be changed in a game-update... Good thing there were no patches back in 1994. Devs had only one shot to make their game work well. Good 'ol days.

The thin red line
Probably a better idea is not letting your game care about every single detail, and having a fall-back plan. In Tower22, you can't save while playing really. You save at special spots - in bed, when sleeping, to be more precise. So a puzzle like this trash-collecting-nonsense is either finished or not, don't care about the details. Tower22 doesn't have levels, but storing the current "Day" may reveal enough information to estimate what has been completed so far. A game like GTA doesn't save mission-details either, it just saves whether missionX has been done or not. Based on that, the game can decide what your next assignments and phone-calls should be, rolling on.

So in other words, it's probably a good idea to have a "Life Line" stored, telling the global game progress. Like "Chapter 6". In the very worst case, you can help your players by letting them return to a fixed state based on that, not having to replay the whole game - like in the Good 'ol days.