3D survival guide for starters #3, NormalMapping

Hey, fatman! Just heard a funny fact that makes you want to buy a horrorgame like T22, if you're too heavy that is. Seems watching scary movies or playing horror games is pretty healthy: watching The Shining burned more than 180 kilocalories! Because of fear/stress, you eat less and burn more. So go figure... playing T22 on a home-trainer screen should be even more tense than Tony Little target training or Zumba dancing.


Normals
-----------------------
Enough crap, lets continue these starters guide posts, explaining the world famous "bumpMap". Or I should say "normalMap", because that is what the texture really contains: normals. But what exactly is this weird purple-blue looking texture? So far we showed several techniques to affect surface properties per pixel:

albedo(diffuse) for Diffuse lighting
specularity for specular lighting
shininess for specular spreading
emissive for making pixels self-emitting

Another crucial attribute is the surface "normal". It’s a mathematical term you may remember from school. Or not. “A line or vector is called a normal if its perpendicular to another line/vector/object”. Basically it tells in which direction a piece of surface is heading to. In 2D space we have 2-axis’s, X(horizontal) and Y(vertical). In 3D space we have -surprise- 3 axis’s: X, Y and Z. Your floor is facing upwards (I hope so at least), your ceiling normal is pointing downwards. If the Y axis would mean "vertical", the floor-normal would be {x:0, y:+1, z:0}, the ceiling-normal {x:0, y:-1, z:0}. And then your walls would point somewhere to the -X, +X, -Z or +Z direction. In a 3D model, each triangle has a certain direction. Or to be more accurate, each vertex stores a normal, eventually a bit bended towards its neighbor to get a smoother transition. That is called "smoothing" by the way.

As explained in part 1, older engines/games did all the lighting math per vertex. I also showed some shader math to calculate lighting in part 2, but let's clarify a bit. If you shine a flashlight on a wall, the backside of that wall won't be affected (unless you have a Death Star laser beam). That's because the normal of the backside isn't facing towards your flashlight. If you shine light on a cylinder shape, the front will light up the most, the cylinder sides will gradually fade away as their normals face away from the flashlight further and further. This makes sense, as the surface there would catch less light photons. Lambertian (cosine) lighting is often used in (game)shaders to simulate this behavior:

NormalMapping
-----------------------
Since we have relative little vertices, per-vertex lighting is a rapid way to compute the scene lighting, but it doesn't allow to have a lot of geometrical detail/variation on your surfaces, unless you tessellate them (means dividing them into a LOT of tiny triangles). So, a football field made of a single quad (=2 triangles, 4 corner vertices) would only calculate the lighting at 4 points and interpolate the lighting between them to get a smooth gradient. In this case, the entire soccer field would be more or less equally litten.

However, surfaces are rarely perfectly flat. A football field contains bumps, grass, sand chunks, holes, et cetera. Same thing with brick walls, or wood planks. Even a vinyl floor still may have some bumps or damaged spots. We could tessellate the 3D geometry, but we would need millions of triangles even for a small room to get sufficient detail. C'est pas possible. That's french for "screw it".

This is why old games drew shading-relief into the (diffuse)textures. However, this is not really correct of course, as shadows depend on the lightsource(s) locations. If you move a light from up to down, the shades on a wall should change as well. Nope, we needed something else... Hey! If we can vary diffuse, specularity and emissive attributes per pixel, then why not vary the normals as well?! Excellent thinking chief. "BumpMapping" was born, and being implemented in various ways. The winning solution was "Dot3 normalMapping". As usual, we make yet another image, where each pixel contains a normal. Which is why the correct name is "normalMap" (not "bumpMap"). So instead of having a normal per vertex only, we now have a normal for each pixel (at least, depends a bit on the image resolution of course). So, for a brick wall the parts that face downwards will encode a normal value into this image, causing the pixel to catch less light coming from above.

Obviously, you can't draw multiple lighting situations in a diffuseMap. With normalMapping, this problem is fixed though. Below is a part of the brick normalMap texture:

Image colors
Now let's explain the weird colors you see in a typical normalMap. We start with a little lesson about how images are build up. Common image formats such as BMP, PNG or TGA have 3 or 4 color channels: Red, Green, Blue, and sometimes "Alpha" which is often used for transparency or masking. Each color channel is made of a byte (= 8 bits = 256 different variations possible), so an RGB image would store each pixel with 8 + 8 + 8 = 24 bits, meaning you have 256*256*256 = 16.973.824 different colors.

Notice that some image formats support higher color depths. For example, if each color channel gets 2 bytes (16 bit) instead of 1, the image size would be twice as big, and the color palette would give 281.474.976.710.656 possibilities. Now having so many colors won't be useful (yet), as our current monitors only support 16 million colors anyway. Although "HDR" monitors may not be that far away anymore. Anyway, you may think images are used to store colors, but you can also use it to store vectors, heights, or basically any other numeric data. Those 24 bits could just as well represent a number. Or 3 numbers in the case of normalMapping. We "abuse" the

Red color = X axis value
Green color = Y axis value
Blue color = Z axis value

About directional vectors such as these normals: these are noted as "normalized" vectors (also called "unit vectors"). That means each axis value is somewhere between -1 and +1. And the length of the vector must be exactly 1. If the length was shorter or longer, the vector isn't normalized (a common newbie mistakes when writing light shaders).

You can forget about "normalized" vectors for now, but it’s important you understand how such a vector value is converted to a RGB color value. We need to convert each axis value (-1..+1) to a color channel (0..255) value. This is because we had 1 byte(8 bits) per channel, meaning the value can be 0 to 255. Well, that is not so difficult:

((axisValue + 1) / 2) * 255
((-1 +1) / 2) * 255 = 0 // if axis value was -1
(( 0 +1) / 2) * 255 = 128 // if axis value was 0
((+1 +1) / 2) * 255 = 255 // if axis value was +1

Do the same trick for all 3 channels (XYZ to RGB):
color.rgb = ((axisValue.xyz) + {1.1.1}) / {2,2,2} ) * {255,255,255}
Let's take some examples. Your floor, pointing upwards would have {x:0, y:+1, z:0} as a normal. When converting that to a color, it becomes

red = ((-1 + x:0) / 2) * 255 = 128
green = ((-1 + y:0) / 2) * 255 = 255
blue = ((-1 + z:0) / 2) * 255 = 128

A bright greenish value (note red and blue are halfgray values, not 0). If your surface would face in the +X direction, the value would be bright red. If it would face in the -X direction, it would be a dark greenblue value (no red at all).

Obviously, normalMaps aren't hand-drawn with MS Paint. Not that its impossible, but it would mean you'll have to calculate the normal for each pixel. That's why we have normalMap generators, software that generates (approximates) normals out of a height-image, or by comparing a super detailed (high-poly) model with a lower detailed (game) model. The high-poly model contains all the details such as scratches, skin bumps or wood reliefs in a real 3D model. Usually programs like ZBrush or Mudbox are used to create those super high polygon models. Since we can't use those models in the game, we make a low-poly variant of it. When we compare the shapes, a normalMap can be extracted from that. Pretty awesome right?

Either way, once you have a normalMap, shaders can read these normals. Of course, they have to convert the "encoded" color back to a directional vector:

pixel.rgb  = textureRead( normalMap, texcoords );
// ! note that texture reads in a shader give colors 
//   in the (0..1) range instead of (0..255)
//   So we only have to convert from (0..1) to (-1..+1)
normal.xyz = {2,2,2} * pixel.rgb - {1,1,1} 

You are not ill, you're looking at a 3D scene drawings its normals. This is often used for debugging, checking if the normals are right. If you paid attention, you should know by now that greenish pixels are facing upwards, reddish into the +X , and blueish in the +Z direction. You can also see the normals vary a lot per pixel, leading to more detailed lighting, Thanks to normalMap textures.


Why is the ocean blue, and why are normalMaps purble-blue? > Tangent Space
--------------------
All clear so far? Good job. If not, have some coffee, play a game, eat a steak, and read again. Or just skip if you don’t give a damn about further implementations of normalMapping. With the knowledge so far, a wood floor normalMap would be a greenish texture mainly, as most normals point upwards. Only the edges or big woodnerves would be reddish or blueish / darker. However, it seems all those normalMaps appear purple/blueish (see bricks above). Why is that? The problem is that we don't always know the absolute “world” normals at forehand. What if we want to apply the same floor texture on the ceiling? We would have to invert the Y value in a different texture. If we wouldn't do that, the lighting becomes incorrect, as the shader still thinks the ceiling is facing upwards instead downwards. And how about animated objects? They can rotate, tumble and animate in all kinds of ways. There is an infinite number of possible directions a polygon can take.

Instead of drawing million different normalMaps, we make a "tangentSpace normalMap". What you see in these images, is the deviation compared to the global triangle normal. Huh? How to explain this easy… All colors with a value of {128,128,255} –yes, that blue purple color- indicate a normal of {0,0,+1}. This means the normal is exactly the same as its parent triangle(vertex) normal. As soon as the color starts “bending” a bit (less blue, more or less green/red), the normal bends away from its parent triangle. If you look at this bricks, you’ll see the parts facing forward (+Z direction) along with the wall, are blue-purple. The edges of the bricks and rough parts start showing other colors.


Ok, I know there are better explanations that also explain the math. What’s important is that we can write shaders that do tangent normalMapping. In combination with these textures, we can paste our normalMaps on all surfaces, no matter what direction they face. You could rotate a barrel or put your floor upside down; the differences between the per-pixel normals and their carrier triangle-normals will remain the same.

It’s also important to understand that these normals aren’t in “World-Space”. In absolute coordinates, “downwards” would be towards the core of the earth. But “down” (-Y = less green color) in a tangentSpace normalMap doesn’t have to mean the pixel will actually look down. Just flip or rotate the texture on your wall, or put it on a ceiling. –Y would point into a different direction. A common mistake is to forget this, and try to compare the “lightVector” (see pics above or previous post) calculated in world-space with this tangentSpace normal value. To fix this problem, you either have to convert the tangentSpace normal to worldNormals first (that’s what I did in the monster scene image above), or you convert your lightVector / cameraVectors to tangentspace before comparing them with the normal to compute diffuse/specular lighting.

All in all, tangent NormalMapping requires 3 things:
• A tangentSpace normalMap texture
• A shader that converts vectors to tangentspace OR normals to World space
• Pre-calculated Tangents and eventually BiTangents in your 3D geometry, required to convert from one space to another.

Explaining how this is done is a bit out of scope of this post, there are plenty of code demo’s out there. Just remember if your lighting is wrong, you are very likely comparing apples with oranges. Or worldspace with tangentspace.

When putting it all together, things start to make sense. Here you can see the different attributes of pixels. Diffuse(albedo) colors, specularity, normals. And of course to what kind of lighting results they lead. All that data is stuffed in several textures and applied on our objects, walls, monsters, and so on.


NormalMaps conclusion
-----------------------------------
NormalMaps are widely used, and they will stick around for a while I suppose. They won't be needed anymore if we can achieve real 3D geometry accurate enough to contain all those little details. Hardware tesselation is promising, but still too expensive to use on a wide scale. I should also mention that normalMaps have limitations as well. First of all, the actual 3D shape still remains flat. So when shining a light on your brick wall, or whatever surface, you'll see the shading nicely adapt. But when looking from aside, the wall is still as flat as a breastless woman. Also, the pixels that face away from the lightsources will shade themselves, but won't cast shadows on their neighbors. So normalMapping only affects the texture partially. Internal shadow casting is possible, but requires some more techniques. See heightMaps.

So what I'm saying, normalMaps certainly aren't perfect. But with the lack of better (realtime) techniques, you'd better get yourself familiar with them for now.


I thought this would be the last post of these "series", but I still have a few more techniques in my sleeves. But this post is already big and difficult enough so let's stop here before heads will start exploding. Don't worry, I'll finish next time with a shorter, and FAR easier post ;)

3D survival guide for starters #2

* Note. If any of you tried to contact via the T22 website "Contact page", those mails never arrived due some security doodle. It should work again. Apologies!!

Let's continue this "beginner graphics tour". Again, if you made shaders or a lot of game-textures before, you can skip. But keep hanging if you want to know more about how (basic) lighting works in games, and what kind of textures can be used to improve detail.

Last time we discussed how the classic "texture" is applied on objects. Basically it tells which colors appear on an object, wall, or whatsoever. To add a bit of realism, that color can be multiplied with the color from lightsources (sun, flashlight, lamps, ...) that affect this pixel. Being affected means the lightrays can "see"/reach this pixel directly or indirectly (via a bounce on another surface). A very crude lighting formula would go:
...resultColor = texturePixelColor * (light1Color + light2Color + ... + lightNcolor)

Thus we sum up all incoming light, and multiply it with the surface color (usually fetched from a texture). If there are no lights affecting the surface, the result appears black. Or we can use an ambient color:
...resultColor = texturePixelColor * (light1Color + light2Color + .... + lightNcolor + ambientColor)

This formula is simpl, but checking if & how much a light affects a surface is a whole different story though. It depends on distance, angles, surroundings, obstacles casting shadows, and so on. That's why old games usually had all this complex stuff pre-calculated in a so called lightMap, just another texture being wrapped over the scene and multiplied with the surface textures. The big problem of having it all pre-computed is that we can't change a lighgt. So, old games had static lights (= not moving or changing colors), no shadows on moving objects, and/or only very few physically incorrect lights that would be updated realtime.

Shadows? Anyone? The little lighting around the lamps are pre-baked in low resolution lightMaps. Halflife 1 was running on an enhanced Quake2 engine btw.

When shaders introduced themselves in the early 21th century, we programmers (and artists) suddenly had to approach the lighting on a more physical correct way. Shaders by the way are small programs running on the videocard that calculate where to place a vertex, or how a certain pixel on the screen should look(color/value/transparency). With more memory available and videocards being able to combine multiple textures in a flexible programmable way, a lot of new techniques popped up. Well, not always new really. The famous "bumpMapping" for example was already invented 1978 or something. But computers simply didn't have the power to do anything with it (realtime), although special 3D rendering software may have used it already. Think about Toy Story (1995).

Anyway, it caused several new textures to be made by the artists, and also forced us to call our textures in a bit more physical-correct way. The good old "texture" on objects/walls/floors suddenly became the "diffuseTexture", or "albedoTexture". Or "diffuseMap" / "albedoMap". Why those difficult words? Because the pixels inside those textures actually represent how the diffuse light should reflect, or the "albedo property" of your material. Don't worry, I'll keep it simple. Just try to remember your physics teacher telling about light scattering. No?


The art of lighting

When lightrays (photons) fall on a surface, a couple of things happen:

- Absorption:
A part of the photons gets absorbed (black materials absorb most).
- Specular reflection:
Another portion reflects on the surface "normal". In the pic, the normal is pointing upwards. This is called "specular lighting". The "specularity property" tells how much a material reflects. Typically hard polished surfaces like metals, plastics or mirrors have a relative high specularity.
- Diffuse scattering:
The rest gets scattered in all directions over the surface hemisphere. This is called "diffuse lighting". The "albedo property" of a material tells how much gets scattered. So high absorbing(dark) surfaces have a low albedo. The reason why it scatters in all directions is because the roughness of a material (on a microscopic level).

In other words, photons bounce of a surface, and if they reach your eye, you can see it. The amount of photons you receive depends on your position, the position and intensity of the lightsource(s), and the albedo + specular properties of the material you are looking at. The cones in your eyes convert that to an image, or audio waves if you are tripping on something. So. your texture -now called albedo or diffuseMap- tells the shader how much diffuse-light(and in which color) it should bounce to the camera. Based on that information, the shader calculates the resulting pixel color.

…
diffuseLight = sum( diffuseFromAllLights ) * material.albedoProperty;
specularLight = sum( specularFromAllLights ) * material.specularProperty; 
pixelColor = diffuseLight + specularLight + material.Emissive

// Notice that indirect light (GI) is also part of the diffuse- and specular light
// However, since computing all these components correctly, GI is usually added
// afterwards. Could be a simple color, could be an approximation, could be anything…
pixelColor += areaAmbientColor * material.albedoProperty * pixelOcclusion;

This stuff already happened in old (90ies era) games, although on a less accurate level (and via fixed OpenGL/DirectX functions instead of shaders). One of the main differences was that old graphics-engines didn't calculate the diffuse/specular lighting per screen pixel, but per vertex. Why? Because its less work, that's why. Back then, the screen resolution was maybe 640 x 480 = 307.200 pixels. That means the videocard (or still CPU back then) had to calculate the diffuse/specular for at least 307.200 pixels. Likely more, in case objects would overlap each other the same pixel would be overwritten and required multiple calculations. But what if we do it per vertex? A monster model had about 700 vertices maybe, and the room around you even less. So a room filled with a few monsters and the gun in your hands had a few thousand vertices in total. That is way less than 307.200 pixels. So, the render-engine would calculate the lighting per vertex, and interpolate the results over a triangle between its 3 corner-vertices. Interpolation is a lot cheaper than doing light math.

DiffuseLighting & DiffuseMaps
--------------------------------
I already pointed it out more or less, but let's see if we get it (you can skip the code if you are not interested in the programming part btw). Pixelcolors are basically a composition of diffuse, specular, ambient and emissive light. At least, if your aim is to render "physically correct". Notice that you don't always have to calculate all 4 of them. In old games materials rarely had specular attributes (too difficult to render nicely), ambientLight was a simple fake, and very little materials are emissive (= emitting light themselves). Also in real life, most light you receive is the result of diffuse lighting, so focus on that first.

We (can) use a diffuse- or albedo image to tell the diffuse reflectance color per pixel. The actual lighting math is a bit more complicated, although most games reduced it to a relative simple formula, called "Lambertian lighting":

// Compute how much diffuse light 1 specific lamp generates
// 1. First calculate the distance and direction vector from the pixel towards
// the lamp.
float3 pixToLightVector = lamp.position - pixel.position;
float  pixToLightDistance = length( pixToLightVector ); 
       pixToLightVector = normalize( pixToLightVector ); // Make it a direction vector
// 2. Do Cosine/Lambert lighting with a dot-product.
float diffuseTerm = dot( pixel.normal, pixToLightVector );
      diffuseTerm = max( 0, diffuseTerm ); // Value below0 will get clamped to 0

// 3. Calculate attenuation
// Light fades out after a distances (because lightrays scatter)
// This is just a simple linear fall-off function. Use sqrt/pow/log to make curves
float attenuation = 1 - min( 1, pixToLightDistance / light.falloffDistance );

// 4. Shadows (not standard part of Lambert lighting formula!!)
// At this point (in this shader) you are not aware if there are obstacles 
// between the pixel and the lamp. If you want shadows, you need additional 
// info like shadowMaps to compute wether the lamp affects the pixel or not.
float shaded = myFunction_InShadow( ... ); // return 0..1 value

// 5. Compose Result
lamp1Diffuse = (diffuseTerm * attenuation * shaded) * lamp1.color;

// ... Do the same thing for other lights, sum the results, and multiply 
// with your your albedo/diffuse Texture

Wanting to learn programming this? Print this picture and hang it on the toilet so you can think about it each time when pooping. works like a charm.

You don't have to understand the code really. Just realise that your pixels get litten by a lamp if
A: The angle between the pixel and lamp is smaller than 90 degrees (dot)
B: The distance between pixel and lamp is not too big (attenuation)
C: Optional, there are no obstacles between the pixel and the lamp (casting shadows)

One last note about diffuseMaps. If you compare nowadays textures to older game textures, you may notice the modern textures lack shading and bright highlights. The texture is more opaque... because it's a diffuseTexture. In old games, they didn't have techniques/power to compute shadows and specular highlights in realtime, so instead they just drawed ("pre-baked") that into the textures to add some fake relief. Nowadays many of those effects are calculated at the fly, so we don't have to draw them anymore in the textures. To some extent, it makes drawing diffuseMaps easier these days. I’ll explain in the upcoming parts.



SpecularLighting & SpecularMaps
--------------------------------
Take a look back at one of the first pictures. Diffuse light scatters in all directions and therefore is view-independent. Specular lighting on the other hand is a more concentrated beam that reflects on the surface. If your eye/camera happens to be at the right position, you catch specular light, otherwise not. Specular lighting is typically used to make polished/hard materials “shiny”.

Again, this is one of those techniques that came alive along with per-pixel shaders. Specular lighting has been around for a long time (a basic function of OpenGL), but not used often in pre-2000 games. A common 3D scene didn’t (and still doesn’t) had that much triangles/vertices. Since older lighting methods were using interpolation between vertices, there is a big lack of accuracy. Acceptable for diffuseLighting maybe, but not for specular highlights that are only applied very locally. With vertex-lighting, specular would appear as ugly dots. So instead, games used fake reflection images with some creative UV mapping (“cube or sphere mapping”) to achieve effects like a chrome revolver. Nowadays, the light is calculated per pixel though, so that means much sharper results.

If you looke carefully, you can literally count the vertices on the left picture. Also notice the diffuse light (see bottom) isn't as smoothy, although less noticable.

That gave ideas to the render-engine nerds... So we calculate light per pixel now, and we can vary the diffuse results by using a diffuse/albedo texture... Then why not make an extra texture that contains the specularity per pixel, rather than defining a single specular value for the entire object/texture instead? It makes sense, just look at a shiny object in your house, a wood table for example. The specularity isn't the same for the entire table. Damaged or dirty parts (food / dust / fingerprints / grease) will alter the specularity. Don't believe me? Put a flashlight on the end of the table, and place your head at the other hand of the table. Now smear a thin layer of grease on the middle of the table. Not only does it reduce the specurity compared to polished wood, it also makes the specular appear more glossy/blurry, as the microstructure is more rough. You can only see this btw if the flashlight rays exactly bounce into your eyes. If you stand above the table, the grease may become nearly invisible. That's why we call specular lighting "view dependant".

Dirty materials, or objects composed of multiple materials (old fashioned wood TV with glass screen and fabric control panel) should have multiple specular values as well. One effective way is to use a texture for that, so we can vary the specularity per pixel. Same idea as varying the diffuse colors in an ordinary texture (albedo / diffuseMap). A common way to give your materials a per-pixel specularity property, is to draw a grayscaled image where bright pixels indicate a high specularity, and dark pixels a low value. You can pass both textures to to your shader to compute results:

float3 pixToLightVector   = lamp.position - pixel.position;
float  pixToLightDistance = length( pixToLightVector ); 
       pixToLightVector   = normalize( pixToLightVector );
float3 pixToEyeVector     = normalize( camera.Position - pixel.position );

// Do Cosine/Lambert lighting
float diffuseTerm = dot( pixel.normal, pixToLightVector );
      diffuseTerm = max( 0, diffuseTerm );
float3 diffuseLight = diffuseTerm * lamp.color;

// Do Blinn Specular lighting
float3 halfAngle  = normalize( pixToLightVector + pixToEyeVector );
float  blinn      = max( 0, dot( halfAngle, pixel.normal.xyz ) );
float  specularTerm = pow( blinn , shininessPowerFactor );
float3 specularLight= specularTerm * lamp.color; // eventually use a 2nd color

...apply shadow / attenuation...

// Fetch the texture data
pixel.albedo       = readTexture( diffuseMap, uvCoordinates  );
pixel.specularity  = readTexture( specularMap, uvCoordinates );

diffuseLight  *= pixel.albedo;
specularLight *= pixel.specularity;

Notice that this is not truly physical correct lighting, it's an approximation suitable for realtime rendering. Also notice that the method above is just one way to do it. There are other specular models out there, as well as alternate diffuse models. Anyway, to save some memory and to make this code a bit faster, we can skip a "readTexture" by combining the diffuse-and specularMap. Since the specularity is often stored as a grayscale value, it can be placed in the diffuseMap alpha channel (resulting in a 32-bit texture instead of 24bit). The code would then be

// Get pixel surface properties from vertices and 2 textures
float4 rgbaValue = readTexture( diffuseSpecMap, uvCoordinates ) 
   pixel.albedo   = rgbaValue.rgb;
   pixel.specularity = rgbaValue.a;

However, some engines or materials chose to use a colored specularMap, and eventually use its alpha channel to store additional "gloss" or "shininess" data. This is a coeffcient that tells how sharp the specular highlightwill look. A low value will make the specular spread wider over the surface (more glossy), while high values make the specular spot very sharp and narrow. Usually hard polished materials such as plastic or glass have a high value compared to more diffuse materials.
spec = power( reflectedSpecularFactor, shininessCoefficient )

EmissiveLighting & EmissiveMaps
--------------------------------
This one is pretty easy to explain. Some materials emit light themselves. Think about LEDs, neon letters, or computer displays. Or radioactive zombies. When using diffuse- and specular light only, pixels that aren't affected by any light will turn black. You don't want your TV appear black in a dark room though, so one easy trick is to apply an EmissiveMap. This is just another texture that contains the colors that are emitting light themselves. Just add that value to the result, and done. The shader code from above would be expanded with

resultRGBcolor = blabla diffuse blabla specular blabla ambient
resultRGBcolor = resultRGBcolor + textureRead( emissiveMap, uvCoordinates );

Notice that emissiveMapping is just another fake though. With this, your TV screen will appear even in dark rooms. And it may also get reflected and cause a "bloom"(blur) around the screen in modern engines. BUT, it does not actually emit light to other surfaces! It only affects the pixels you apply this texture on, not the surrounding environment... unless you throw some complex code against it, but most games just place a simple lightsource inside the TV, or ignore it completely.

The charming man on the TV and the lamp in the background use emissive textures. However, the TV does not really cast light into the scene (other than the blurry reflection on the ground).



I could go on a bit longer, but let’s take a break. I remember when I had to read this kind of stuff ~7 years ago… Or well, probably I didn’t really read it. My head always starts to hurt if I see more than 3 formula’s with Sigma and 10th order stuff. But don’t worry, just go out and play, you’ll learn it sooner or later, and documents like this will confirm (or correct) your findings. The next and last part will reveal the famous “bumpMaping” technique, plus describes a few more texture techniques that can be used in modern games. Ciao!

3D survival guide for starters #1

Ever been discussing about new games with friends? Are you one of those guys/girls that go like "yeah yeah yeah, that's r-r-right" (what’s the name of that green annoying cartoondog?), once a friend mentions EngineX looks even better due parallax-normal-displacement-crunch-mapping, while you have no clue what the hell he is talking about? Are you one of those aggressive game-forum guys, arguing about whether the Xbox or PS3 looks better, but never really understand your own arguments? Or, maybe you are just curious what those other nerds are talking/arguing about, when they drop fancy-pants words like “specular-map”? If you are already a 3D artist or did shader programming, you can safely skip, but otherwise this and 1 or 2 upcoming posts may make you sound even cooler on those forums. Or well, at least you know what you’re talking about then.

Not a superb shot this unfinished room, but can decipher the techniques being used here?

There's a lot of shit arguments on the internet all right. For some reason, boys want to dominate digital discussions by saying difficult tech-terms, like a dog marks territory by pissing against poles. And the funny thing is, a lot of arguments are based on "something they heard", and then misinterpreted. I remember the “Next-Gen” vibe 5 years ago. New technologies are treated like magical ingredients only possible on platform-X, but the truth is that any modern platform can do all tricks, as long as the programmers are smart enough and the platform fast enough to handle it realtime (meaning at least 25 times per second). A Wii can do Parallax-Occlusion-Mapping or realtime G.I. too, but at such a large cost it's unlikely to be implemented in a game. If you really want to compare game consoles on visual capabilities, then look at the videocards, available memory, and shader instruction sets.

Either way, good graphics start with five main components:

A: Proper Lighting
B: Proper Texturing
C: 3D models / worlds
D: Technicians & Hardware making it possible
E: Creative people making good use of A,B,C and D

How you do it, doesn't matter until you start meeting the platform limitations (which is pretty soon usually). These posts focus on the crucial component “Texturing”, and the technology behind it. Oh, for starters, with a texture we mean the images being pasted on 3D models. High-end engines can still look like shit when using bad textures. Low-end engines can still look good when using good textures. Usually when we develop a new Tower22 environment, the first versions look pretty bad. The same happens when a amateur makes a map in a UDK or whatever powerful engine. Techniques such as shadows and reflections mask the uglyness a bit, but in the end it's still like smearing expensive make-up on a pig. Then on the other hand, games such as Resident Evil4(Gamecube / PS2), God of War(PS2) or Mario Galaxy(Wii) still look good while their engines really weren't exactly cutting edge technical miracles. Neither back then, compared with PC engines such as Source(Halflife2), CryEngine(Farcry) or IdTech (Doom3/Quake4).

How your grandpa used to Render:
Ok, but how does it work? Probably you heard about bumpMaps and stuff, but maybe no idea what it really does. Let's just start with the very basics then. Remember Quake1 (1996)? That is one of the first (if not the first commercial) true 3D games, where the worlds were made of 3D polygon models, using textures to decorate the walls and objects... Or maybe It wasn’t the first 3D game actually. The Super Nintendo already had a couple of games using the Super FX chip, like Star Fox and Super Stunt FX. These games rendered flat (animated) pictures called "sprites", and simple 3D geometry shapes (triangles, cubes, floor-plane) with a certain color. The space ships for example were a bunch of gray and blue triangles, and an orange triangle as a booster. Not much detail of course, but some of those triangles even carried an image to give it some more detail. See the cockpit above.

And also Doom, Hexen, Wolfenstein and Duke Nukem were sort of 3D (“2.5D”) of course. Although the technology used back then is way different than Quake1 used, which is still the footprint for nowadays games. The way how Wolftenstein actually looks more like how a raytracer works; each screen pixel (and the screens didn’t have much pixels back then fortunately) would fly away from the camera, and see where it would intersect a wall, floor or ceiling. Because of the limited CPU power, the collision detection had to be fast of course, and that explains the very simple level design. But what Wolftenstein also already did, was Texturing its walls. Depending on where the screen-ray intersected, the renderer would pick a pixel from the image applied on that wall.

Although old 2.5D rasterizers don't really compare to current techniques, the texture-mapping (texture-coordinates, UV-mapping, whatever they call it) is founded on the same principles.



Texturing in the 21 century
Texturing basically means to "wrap" a 2D image over a (3D) model. Typical game data therefore contains a list of 3D-models and image files, paired together. Asides having triangles, the 3D models also tell how to map the image on them by giving so called “Texture” or “UV” coordinates. As for the textures, those are just images you can draw MS Paint or Photoshop really. Nothing special. Although games often use(d) compressed files to save memory. The SNES didn't have enough memory & horsepower to texture each and every 3D shape, so most objects in those 3D games just used 1 or a few colors only.

You can do the math; an average jpeg photo from your camera or phone already takes a few megabytes. In other words, a single photo is larger than the total storage capacity of a SNES cartridge! But if you make a tiny image, with only a few colors, you can save quiet a lot though. Save a 32x32 pixels image as a 16-color bitmap, and it will only take 630 bytes = 0.62 KB = 0,0006 MB. Now that's more like it. Only problem was, the SNES still only had super little RAM memory, a matter of (64?) kilobytes. So it could only keep a few things active in its work-memory at a time.

Quake1 fully utilized texturing though. Every wall, floor, monster, gun or other object used a texture. Of course, PC's back then still had very limited memory and data bandwidth, so the textures were small and only had 256(or less?) color palettes. But for that time, it looked awesome. Either way, since Quake1 also became "Mod-able" for hobbyists at home, we ordinary gamers came in touch with editors, 3D models, sprites, UV coordinates, textures, and whatsoever. So, in resume:

- You make a 3D model (with 3D software such as Max, Maya, Lightwave, or game-map builders. Those are often made by hobbyists, or provided with the game(engine) itself.
. A game model is a list of triangles. Each triangle has 3 corners, called "vertices". A vertex is a coordinate in 3D space, having a X,Y and Z value. It can carry more (custom) data, but we talk about that later.
- so, yhe 3D model you store is basically a file that lists a bunch of coordinates
- An image is made for the object. Since 3D objects can be viewed from all direcetions, we need to unfold (unwrap) the 3D model all over the canvas.
- Each vertex(corner) also has a texture-coordinate (a.k.a. UV coordinate). These coordinates tell where the triangles are mapped on the 2D image.

The selected green triangles on the right are (2D) UV-mapped in the texture canvas on the left

That's how Quake1 worked, and that's how Halflife3 will still work (although... HL3 might not appear this century, who knows what technology we have then). So, basically this means all 3D objects will get their own texture. We also make a bunch of textures we can paste on the walls, floors and ceilings. Like putting wallpaper in your own house. Obviously, the texture quality goes hand in hand with the artist skills, and the dimensions of those textures. A huge image can hold a lot more detail than a tiny one. These days, textures are typically somewhere between 512x512 to 2048x2048 pixels, using 16 million colors. The file and RAM usage size varies somewhere between 0.7 and 4 MB for such textures, if not compressed.

The results also depend on how the UV coordinates were made. You can map a texture over a small surface, so the small surface receives a lot of pixels relative (but also repeats the texture pattern all the time). Or you stretch the texture over a wide surface, making is appear blurry and less detailed. This typically happened on large outdoor terrains in older games. Even if you have a 1000x1000 pixel texture, a square meter of terrain will still receive only 2x2 pixels if it’s about 500 x 500 meters big. Ugly blurry results. Games like the first Battlefield often fixed that by applying a second “detail texture “ over the surface, which repeated a lot more times. Such detail texture could contain the patterns of grass or sand for example.

We still use detail-mapping these days. The small wood-nerve structure on the closet is too fine/detailed to draw in the closet texture. So we use a second "wood-nerve" texture that repeats itself over the surface quite a lot of times.


Next time we crank up with lighting in (modern) engines and the kind of textures being used to achieve cool effects for that. For now, just remember textures are used to give color to those 3D models. Nothing new really, but an essential part of the process. As for the next X years in rendering evolution: Textures are here to stay, so deal with them. Keep your friends close, and your enemies closer -- Sun Tzu

Son of a Gun

Boys & weapons... If you never played with self made nun-chucks, didn't fancied the air pellet rifles at the fair, hated to see Stalin’s Organ blowing shit up in a WOII documentary, or even didn't get goose bumps when seeing the Tsar explode, then there is a 90% chance you're castrated and singing in a choir. Or you suffer from a very tenacious variant of peace-loving hippie genes. One of the differences between man and beast is that we make awesome tools to exterminate each other. Most men roughly know the Age-of-Empires evolution when it comes to weapons; Clubs, spears, swords, gladiators, Chinese gunpowder, boiling pitch kettles, catapults hurling Plague-infected bodies, muskets, cowboy rifles, Gatling guns, trenches & musterdgas, tanks, Bismarck, Spitfires, V2-rockets, MIG, B2 bombers, Tall boy, M16, Bob Hope, AK47, Bell Huey, nuclear submarines, Barett, MOAB, IED, drones. Power Rangers. Men's history in a nutshell. All other discoveries such as electricity, art of printing, thermo power-plants, computers or GPS were byproducts. And why we know all that? As I said, weapons are awesome.

Well, you don't have to tell me that those weapons are less "awesome" when you see footage from Syria. Also, those who caught Stielhandgrenades during WOII probably have less romantic memories about weapons. Yes, the USS Abraham Lincoln wasn't made to be a colorful happy Amsterdam Gay parade boat. It's made to kill & destroy. Still, that doesn't take away the fascinating creative thinking behind these devices. And probably a gun has some psychological magic (for boys at least). Carrying a device that gives a powerful blast when pulling the trigger, reloads like a small mechanical oiled wonder, and has the ability to take out whoever is fucking around with you, gives a mighty feeling. Or something.

PPS-43 in progress, by Diego

Yes, I like guns. And no, I don't have them (other than some not-so-dangerous air rifles). And no, I don't think it's a very good idea to legalize them here as well. Although I don't agree with the "guns kill people" saying. Aggressive/stupid people with guns kill other people, that's for sure. Giving weapons (for hunting, hobby or self-defense) to a society is like giving democracy; people need to be ready for it and deal in a mature way with it, otherwise it's going to be a disaster.

But now the big question... should guns be legalized in Tower22? I'm not talking whether that would be a good choice or not in a moral way. Screw that, most aggressive societies I know aren't famous for playing XBox, neither did our ultra-violent ancestors have Gameboys. No, I'm wondering if it would be a good choice to ship a horror game with self-defense equipment.


If implemented properly, guns surely add a fun element to a game. It's not without a reason that many top-rated games involve bang-bang. Doom, Goldeneye, Halflife, GTA... Guns make cool sounds, allow you to do something you're not supposed to do at home (I hope), and give a challenge. It's pretty dull if we had to jump on our GTA gangster opponents or squirt melon juice to make Doom3 Imps stick to the ground right? Weapons open a whole world of tactics, with spectacular destruction effects as a bonus. But for our game, we can also choice NOT to kill opponents.... Huh? Arrh! Blasphemy!

Saying that in Game-land is like saying the Earth is more than 6000 years old in an Amish community. It's probably the reason why pretty much every game that involves "bad guys" also has weapons of some sort. Just to be safe. Even when the focus isn't really on killing enemies. Like in Silent Hill, where the combat gameplay is pretty shit actually. The game is supposed to scare you, the stupid axe-fights with dumb fleshy things are just to fill the game a bit.

Nope, this guy didn't make it through the X-Factor selection rounds. Nevertheless, I wanted to show it anyway.

However, Amnesia showed a game can also be "good" without guns. A risky choice, but it worked out well. I placed "good" between quotes because Amnesia is not a fun game. Neither is Silent Hill btw, and also the older Resident Evils may fall to that category. Fun = having a laugh, tap yourself on the shoulder after kicking the level3 boss’s head off, or get satisfied by solving puzzles. That's not much the case with these horror games, apart from some puzzling maybe. Anyway, Amnesia wasn't much fun maybe, but it was SCARY. That sounds obvious for a HORROR game, but many horror games aren't scary really. Resident Evil turned into a farmer-shotgun-fest with a cheesy over the top Hollywood storyline. Doom2 was loads of fun, but certainly not scary. Unless you illegally played it the age of 10.

Alan Wake is another example. The story sucks you in, and the whole setting has potential. Yet it isn't really scary. Like so many other games, it decided to become a shooter mainly. Alan is blinding and shooting ghostlike entities all the time. And if you get killed? Just respawn 10 meters earlier. In other words, they decided to please the mass. Not a bad decision if you want to earn money with a game, and I'm not saying Alan Wake is a bad game. The point is, true scary games like Amnesia are only for a limited audience.


So, it seems the "action element" reduces the "fear element". Put a bit too much action on the balance, and the fear element is outweighed. It happened with Alan Wake, modern Resident Evil, and so many other games. The explanation for this phenomena isn't that difficult. Didn't I mention a gun gives a mighty feeling? Even if you have a small weiner? Same thing happens in virtual reality. Gore graphics and monsters on themselves aren't that scary once you saw them a couple of times. Once you learned how to deal with them (shoot them), you have the situation under control. Fear = losing control. Being vulnerable, not knowing what will happen, having to redo a substantial part of the game when taking a wrong step, not understanding what you see. Everything that pulls you out of your “mental safe zone”.

Obviously, guns and features such as Auto-save / Quick-load reduce fear, as they give you control. Got eaten by a dragon? Just respawn. 10 Imps with fireballs? Answer with a BFG9000. It gives you Tony Montana powers, it makes you reckless, storming in rooms without thinking. At this critical point, consider the game being an action game. Not a horror game (anymore). No matter how much blood, guts, ugly monsters and weird environments you throw against it. Just look at Duke Nukem 3D. Plenty of monsters and weird darkgreen Alien worlds that could be pretty scary actually. But when controlling the Duke, not even a Muhammad cartoon can scary you.

Don't you agree Alice in Wonderland (the old cartoon, not that Johny Depp shit) was scary? And that was not because of the monsters or blood. Just because it was so fucking weird. Like a dream/nightmare. I mean, look at that door man!


So... maybe its better just to forget about guns in T22? Hmmmm... It sure has advantages. No need to make a bunch of complicated models, death animations, destruction physics, ammo crates, et cetera. It makes the development easier, that's for sure. And it will make things more tense. T22 has only a few monsters wandering around the building. It's definitely not the goal to kill them on first sight. Bye bye big monster the T22 team spend millions of triangles and pixels on. No. The T22 monsters are stronger than the player, so a gun shouldn't suddenly change that.

But... that means you are supposed to run or hide? Ehm, yes, that has always been the idea for this game. But wouldn't that get boring after a while? Now that's a difficult question. With shooters, there is plenty of comparison material. But with a run & hide type of game, there aren't much good examples other than Amnesia (from which I only played the demo btw). To make this gameplay element work, the environment really needs to lend itself for that. Lots of corners, maze like corridor structures, dark spots or objects to hide in, and also a good dosing of monsters. If you had to run each 2 minutes while trying to solve a puzzle, it becomes very annoying. Therefore in T22, you won't see much monsters in general. But then again, what else should make the game exciting/playable? T22 is about exploring the environment, and finding new area's by solving puzzles. The balance between puzzles, tense monster chases and "rest" is crucial and requires a lot of testing.

Eventually it may turn out that the game becomes too dull, especially when it's a long game. Like with food, you need to vary to keep it tasty. Eating rice every day makes you a Chinese, potatoes every day an European farmer. Erh, right. The point is, T22 needs to be scary in the first place. But we have to prevent it from becoming boring after a while. In other words, I'm not saying no to guns right away. The T22 player sure is no hero, but neither a complete defenseless moron. What would you do if a monster chases you? I would pee & run, but also throw a Ming-vase at its head if there is no other option.


The first Resident Evil showed that weapons can be combined with scary gameplay, just as long you don't make the weapons too powerful. In RE, they smartly dosed the ammunition and save-games (ink ribbons). You could shoot zombies and reload, but be careful! There's not enough ammo to kill them all, and in narrow spaces a pistol might not be enough to defend yourself to dumb but tough zombies. Also only save when you really made progress. Having to decide all the time whether a save or pumping down a zombie is worth it, actually added some extra stress to RE. You could shoot the bastard but what if you wouldn't have enough ammo for the next boss? Better just outrun the damn thing.

Speaking of RE, here the door in-game. No slow door animations when entering a new room though.

Likely, T22 will go that way too. Although you won't be locked & loaded most of the time. There might be enough ammo to kill a few of the opponents, but pick wisely. As for us, to make sure these ingredients are in good balance, play-testing is crucial. But first we need to make some more (maps) in order to test it. You can't play hide & seek in a small house, you need a "playground" for that. And a test-person. The problem with making a horror game/movie/book is that its totally not scary for the creator. How can I be scared if I know where the enemy is, how it thinks (AI), and where to run? Nope. If we finally come to a testing phase, I think this Blog would provide some trusty fans for that ;)

Erhh....

All right, a shorter story again. When I started this Blog, I intended to keep the texts short and to the point. No one has time or interest in never-ending stories right? There's a reason why Twitter is successful. "Jerking off." is a lot easier to print in the brain than an a complete epistle of when, why and how. But it turns out I'm like The Nanny once I start writing about Tower22 / game-programming. And so, the blog posts got longer and longer and ...

Probably this Blog (and the poor guys receiving my mails who help on T22) function as an overflow-valve. At work I never tell about T22, neither do I work with other programmers. My girl still uses an abacus and thinks nVidia is a body lotion, so that's not much of a computer-girlfriend either. And as for the rest of my friends; when drinking a beer, the discussions vary from pneumatic machines to 425 kg Mexicans we saw on TV, from president elections to poop. But barely about this little "secret" project. And if they ask about the latest T22 headlines? I'll just deflect the question with a "work-in-progress" or "not much special lately". Besides, I'm too stupid to technically answer the question after 8 beers anyway.

But seriously, what to tell? It's not that we build a new awesome monster each week, or just finished a complete playable floor. Week-progress is about a few textures, a drawing, or an object to fill the rooms with. And usually a bit of code to implement whatever asset was made. For example, let the player make footstep sounds after Carsten made some audio effects for that. Normally I plan a demo, that requires a (big) bunch of assets to make. Once advanced assets such as a monster, moving elevator, special effect or machinegun are being made, it forces me to upgrade the engine to realize them. Programming on demand basically. As for the last two weeks, Borja made another living-room drawing for the next demo, Diego is busy with a weapon, I worked on the reflections and made some corridor-map pieces, and Federico modeled two apartment-doors... Obviously, I can't produce a fascinating 10-minute conversation with drunk friends about a bunch of doors we just made.

But if there is not much to report, where do those big-ass stories on this blog come from then?! Well, actually there is a lot I can tell about a simple game-door. Did you know that this door was split up in 4 sub-objects; a stationary doorframe, a rotatable doorknob, transparent(breakable) glass, and the door itself? Did you know I implemented a so called "motor" that makes an object -a door in this case- move/scale/rotate from matrix A to B, using quaternions and spherical lerps? Did you know this door uses a DLL that controls its interface (open, close, slam, locked, unlock, ...) and how it interferes with the game-world (blocking the path for monsters, blocking invisible rooms behind them, occluding sound "rays"). Did you know I'm upgrading the Object-Editor right now so Federico can attach such a motor and door behavior to his object?

Yeah probably you know such things if you visit this place more often, or if you are familiar with game-development in general. The real progress usually lays in the hidden things. Things that are completely uninteresting for the outside world. But hopefully interesting enough to bore you with ;)

Here you go, doors. And yes, to make it work as a door, there is quite a lot to implement actually. Like other moving things such as an elevator, doors have an animation, and not just rotations. Think about slide-doors, garage doors, or those opening-anus-doors you see in sci-fi movies. And not just every door can be opened the good old fashioned way of course. Some are locked, some require a puzzle, some need to be kicked by a monster. When you think about it, a whole lot more scenarios pop up, and making game entities flexible for that is a challenge. This is not an in-game shot btw, not that far yet.

Reflective conspiracy theories

A small step for man, but a giant leap for mankind
Before moving on, let's zero-G for a moment for Neil Armstrong. First man on the moon (and hopefully not the last), died at the age of 82, 25 august 2012. If that was a real step on a real moon, Neil will reserve a well deserved page in the history books for a long, long time. Something we mankind as a whole should be proud of. Although we kill each other each day for various reasons, we should realize we all share this tiny globe. Zooming out puts things in perspective, and Neil literally took that perspective when having a view on our little planet while standing on another floating rock in this endless cosmos. It’s such a huge performance that it's hard to believe we really did it...

Moon-landing hoax? Who shall say. John F Kennedy, chemtrails, 9-11 inside job, Saddam & biological weapons, Area 51, New world order? Both the present and history are full of mysteries, and the more you think about it, the more questions arise. Things aren’t always what they seem. Having a Moon-landing sure came at a good timing with those crazy Russians trying to outperform USA as well. And it surprises me that modern space missions -50 years technology evolution since the sixties- seem so extremely vulnerable (control room being overexcited because the Curiosity drove a few centimeters on Mars ?!) that it puts the much more ambitious/dangerous Moon-landing in a weird contrast.

But before just following the naysayers... Being skeptic is a natural, psychological phenomenon. And not taking everything for granted the media says is healthy. But consider other huge achievements. Didn't we laugh at the brothers Wright? Would Napoleon even dare to dream about the awesome power of a nuclear bomb? Huge pyramids being built with manpower only? Even got a slight idea of how CERN works? Would you run with thousand other soldiers on Utah beach while German bunkers are mowing down everything that moves? Men can do crazy stuff when being pushed! But the bottom line is that you or me will never know what really happened, because we weren't there, nor do we have thorough, inside knowledge of the matter. All we do is picking sides based on arguments we like to believe. And for that reason, here an *easy-to-consume* series of the Mythbusters testing some infamous Moon-Landing conspiracy theories including the footprints (on dry "sand"?), impossible light & shadows (multiple projector lights?), and the waving flag (in vacuum?). So before copying others, get your facts right and check out this must-see:
Mythbusters & Moonlanding

Let me spoil one thing already. Something we graphics-programmers *should* know. Why is that astronaut climbing of the ladder not completely black due the shadow? Exactly, because the moon surface reflects light partially. A perfect example of indirect lighting, ambient-light, Global Illumination, or whatever you like to call it. Neil, rest in peace. And for future astronauts, don't forget to draw a giant middlefinger on the Moon/Mars so we have better evidence next time. Saves a lot of discussion.


Reflections
-------------------------------------
As mentioned above, shadows and light bounce off surfaces. Not just to confuse conspiracy thinkers with illuminated astronauts, also simply to make things visible. If not directly, then indirectly eventually. Reflections are an example of that, and take an important role in creating realistic computer graphics. Unfortunately, everything with the word "indirect" in it seems to be hard to accomplish, even on modern powerful GPU's. But it's not impossible. Duke Nukem 3D already had mirrors, so did Mario 64, and Farcry was one of the first games to have spectacular water (for the time) that both refracted and reflected light.

Well, a GPU doesn't really reflect/refract light-rays. Unless you are making graphics based on Raytracing, but the standard for games still is rasterization, combined with a lot of (fake) tricks to simulate realistic light physics. Reflections are one of those hard to simulate tricks. Not that the tricks so far are superhard to implement, but they all have limitations. Let's walk through the gallery of reflection effects and conclude with a relative new one: RLR (Realtime Local Reflections) I recently implemented for Tower22. If you already know the ups and downs of CubeMaps and Planar reflections, you can go there right away.


Planar reflections
One of the oldest, accurate, tricks are planar reflections. It “simply” works by rendering the scene(that needs to be reflected) again, but mirrored, The picture below has 2 “mirror planes”. The ultra realistic water effect for example renders everything above(!) the plane, flipped on the Y axis. That’s pretty much it, although its common to render this mirrored scene into a texture(render-target) first. Because with textures, we can do cool shader effects such as colorizing, distortions, Fresnel, and so on.

Planar reflections are accurate but have two major problems: performance impact & complex (curvy) surfaces. The performance hit is easy to explain; you’ll have to render the scene again for each plane. This is the reason why games usually only have a single mirror or water-plane. Ironically the increasing GPU power didn’t help either. Sure you can re-render a scene much faster these days, but don’t forget it also takes a lot more effects to do so. Redoing a deferred-rendering pipeline, SSAO, soft shadows, G.I., parallax mapping and all other effects for a secondary pass would be too much. If you look carefully at the water(pools) in the T22 Radar movie, you’ll notice the reflected scene being a bit different… uglier. This is because lot’s of effects are disabled while rendering the mirrored scene for planar reflections. Just simply diffuseMapping with a fixed set of lights.

The second problem are complex surfaces. The mirror-planes on the image above are flat. That’s good enough for a marble floor, and even for water with waves (due all the distortions and dynamics, you won’t quickly notice the error). But how to cover a reflective ball? A sphere has an infinite amount of normals (pieces of flat surface pointing in some direction). Ok, game-spheres have a limited amount of triangles, but still a 100 sided sphere would also require 100 planar planes = 100x reflecting the scene to make a correct reflection. To put it simple, it’s WAY too much work. That’s why you won’t see correct reflections on curvy surfaces.

Conspiracy people! Notice the reflected scene in the waterpool being a bit different than the actual scene?


CubeMaps
CubeMaps are the answer on the typical problems with planar reflections… sort of. The idea is to sample the environment from all directions, and store it in a texture. Compare it with snapping a panorama photo. It’s called a cubeMap because we take 6 snapshots and “fold” them into a cube. Now we can both reflect and refract light simply by calculating a vector and sample from that location in the cubeMap texture. The crappy sample below tries to show how a cubeMap is build and how it can be used. The right-bottom image represents the scene from topview, the eye is the camera, and the red line a mirror. So if the eye would look to that mirror, it creates the green vectors for sampling from the cubeMap. In this situation the house would be visible in the mirror.

• Paraboloid maps are a varation on cubeMaps that only require 2 snapshots to fold a sphere. PM’s are faster to update in realtime, but lack some quality and require the environment to be sufficient tessellated though.

Since cubeMaps sample the environment in 360 degrees, they can be used on complex objects as well. Cars, spheres, glass statues, chrome guns, and so on. Problem solved? Well, not really. First of all, cubeMaps are only accurate for 1 point in space. In this example, the environment was sampled around the red dot. Stuff located at the red dot will correctly reflect (or refract) the environment, but the further it moves away from the sample-point, the less accurate it gets. That means we should sample cubeMaps for each possible location? No, that would be overdone. The advantage of curvy surfaces is that it’s really hard to tell whether the reflection is physically correct for an average viewer.

But at the same time, you can’t use a single cubeMap for a large reflective waterplane, because you will notice the inaccuracy at some point. What games often do is letting the map-artists place cubeMap “probes” manually at key locations. At the center of each room for example, or at places you expect shiny objects. Then reflective objects would pick the most useful(nearby) cubeMap. In Halflife 2 you can see this happening. Take a good look at the scope-glass on your crossbow… you’ll see the reflection suddenly changing while walking. This is because the crossbow switches over to another cubeMap probe to sample from.

• Tower22 updates a cubeMap nearby the camera each cycle and uses it for many surfaces. This means pretty correct (& dynamic!) reflections for nearby objects. Distant surfaces will lead to visible artifacts sometimes though.

A cubeMap requires 6 snapshots, thus rendering the scene 6 times. This is quite a lot, so cubeMaps are usually pre-rendered. Since we don’t have the scene again from that point, cubeMaps provide a much faster solution than planar reflections. However, being not updated realtime, you won’t see changes in the environment either. Wondered why soldiers didn’t get reflected in some of the glass windows or waterpools in Crysis2? That’s why. All in all, cubeMaps are only useful for (smaller) local objects, and/or stuff that only vaguely reflects such as a wet brick wall or dusty wood floor.


Other methods?
I don’t know them all, but Crytek introduced an interesting side-quest on their LPV (Lighting Propagation Volume) technique. To accomplish indirect lighting, one of the things they do is creating a set of 3D textures that contain the reflected light fluxes globally. Asides from G.I., this can also be used to get glossy(blurry) reflections by ray-marching through those 3D textures. I sort of tried this technique (different approach, but also having a 3D texture with a global/blurry representation of the surroundings). And did it work? Well judge for yourself.

Personally, I found it too slow for practical usage, although I must say I only tried it on an aging computer so far. But the real problem was the maximum ray-length. Since 3D textures quickly grow to very memory consuming textures, their sizes are limited. That means they only cover a small part of the scene (surrounding the camera), and/or a very low quality representation in case the pixels cover relative large areas. In this picture above, each cell in the 3D texture covered 20^3 centimeter. Which is quite accurate (for glossy reflections), but since the texture is only 64x64x64 pixels, a ray cannot travel further than 64 x 20cm = 12,5 meters. In practice it was even less due performance issues and the camera being in the middle. Only a few meters. So the wall behind the camera would be too far away for the wall in the front to reflect. This was fixed by using a second 3D texture with larger cells. You can see the room pixels suddenly get bigger in the bottom-left buffer picture. However, raymarching through 2 textures makes it even slower, and the raylength is still limited. All in all, reflections by raymarching through a 3D texture are sort of accurate, but very expensive, and useful for very blurry stuff only. I also wonder if Crysis2 really used reflections via LPV in the end btw… guess not.


RLR (Realtime Local Reflections)
In case you expect super advanced stuff now, nah, got to disappoint you then. If you expect a magical potion that fixes all the typical Planar & CubeMap reflection problems, I got to disappoint you as well. Nevertheless, RLR is a useful technique to use additionally. It gives accurate reflections, at a surprisingly good performance, and implementing this (post)screen effect is pretty easy. And no need to re-render the scene.

How it works? Simple. Just render the scene as you always do, in HDR if you like. Also store the normal, depth or position of each pixel, but likely you already have such a buffer for other effects, certainly if you’re having a Deferred Rendering pipeline. Now it’s MC-Reflector time. Render a screen filling quad, and for each pixel, send out a ray depending on its normal and the eyeVector. Yep, we’re raymarching again, but in 2D space this time. Push the ray forwards until it intersects elsewhere in the image. This can be checked by comparing the camera-distance-to-pixel and camera-distance-to-ray. In other words, if the ray intersects or gets behind a pixel, we break the loop and sample at that point. Now we have the reflected color. Multiply it by the source pixel specularity to get a result. The code could look like this:

pixNormal = tex2D( deferredNormalTex, screenQuadUV );
pix3DPosition = tex2D( deferredPositionTex, screenQuadUV );

int  steps = 0;
float3 rayPos = pix3DPosition.xyz;  // Start position (in 3D world space)
float3 rayDir = reflect( pixNormal, eyeVector ); // Travel direction (in 3D)
bool collided = false;
float4 screenUV;

while ( steps++  <  MAX_STEPS    &&  !collided )
{
 // Move the ray
 rayPos += rayDir * STEP_SIZE;

 // Convert the 3D position to a 2D screen space position
 screenUV     = mul( glstate.matrix.mvp, float4( ray.xyz, 1.f) );
 screenUV    /= screenUV.w;
 screenUV.z  *= -1.f;
 screenUV.xy  = (screenUV  +1.f) * 0.5f;

 // Sample pixel depth at ray location
 float enviDepth = tex2D( deferredPositionTex,  screenUV.xy ).w;

 // Check if it hits
 collided = length( rayPos – cameraPos  ) > enviDepth + SMALLMARGIN;
}

// Sample at ray target
Float3 result = tex2D(  sceneHDRtex, screenUV );

The nice thing about RLR is that it works on any surface. The green spot gets reflected on the low table, but also on the closet door. Also notice the books being reflected a bit, and the floor, and the wall. No matter how complex the scene is, the load stays the same.

Perfect! But wait, there are a few catches. How many steps do we have to take, and wouldn’t all those texture-reads hurt the performance? Well, RLR does not come for free of course, but since rays take small steps and usually travel in parallel, it allows good catching on the GPU. Second, you can reduce the number of cycles quite drastically by:

A: Do this on a smaller buffer (half the screensize for example)
B: Do not send rays at all for non-reflective pixels (such as the sky or very diffuse materials)
C: Let the ray travel bigger distances after a while

Or instead of letting the ray travel x centimeters in 3D space, you could also calculate a 2D direction vector and travel 1 pixel each loop-cyclus. If your screen is 1200 x 800 pixels, the maximum distance a ray could possibly travel would be 1442 pixels. To complement, make good use of the power of love, I mean blur. A wood floor has a more glossy reflection than a glass plate. What I did is storing the original output texture, and a heavily blurred variant on it. The end result interpolates between the two textures based on the pixel “glossiness” value.

 pixSpecularity = tex2D( deferredTexSpecular, screenQuadUV );
 pixGloss = pixSpecularity.w;

float3 reflection = tex2D( reflectionTex, screenQuadUV );
float3 reflectionBlur = tex2D( reflectionTex2, screenQuadUV );
 endResult = lerp( reflection, reflectionBlur, pixGloss ) * pixSpecularity.rgb;
// use additive blending to add the end result on top of the previous rendering work

Of course, there are ways to jitter as well, use your imagination. However, the deadliest catch of them all, giving RLR a C+ score instead of A+, is the fact this screen-space effect can only gather reflections from stuff… being rendered on the screen. Imagine the wallpaper wall in the screenshot being reflective. It should reflect something behind the camera then. But since we never rendered that part, we can’t gather it either. In other words, pixels that face towards the camera, or to something else outside the screen boundaries, cannot get their reflections. That makes RLR useless for mirrors, although some women may prefer a RLR technology mirror. Also be careful with pixels around the screen edges. Your code should have a detection for this so you can (smoothly!) blend over to a black color (= no reflection).

As said, RLR is not a substitution for CubeMaps or Planar Reflection. Be a ninja and know your tools. Planar reflections for large mirrors / water. RLR for surfaces that only reflect at a steeper view angle, (pre-rendered?) cubeMaps for other cases.

T22 Testament

Time flies when... getting older. A little special moment last week when we brought our our little girl to the elementary school for the first time. Little backpack strapped on her back, shy and carefully entering a new environment, inspecting the classroom a bit. Usually the moms are the softies, but this time I felt like Forest Gump dropping of his son at the bus as well. Probably mom and dad were more emotional than daughter herself. So sweet!


But what else did we do? Not a whole lot, since I had to visit England for work. But a week ago one of our guys -concept artist Pablo-, asked if I could write down some more about "game-mechanics". You know, how the game would play. How fast does our hero run? How to eliminate your opponents? How many hearts does his health-bar have? Does he have a health-bar at all? And, maybe more important, what will you be doing in this game anyway? Unless you have access to my head or tortured one of the team guys to extract information, likely you won't really know how Tower22 will be exactly played. It's a horror game, sure. But what kind of horror game? Killing zombie hordes with a fry pan like the addictive Left-4-Dead? Slowly exploring and puzzling through an infected mansion? Doing bondage & whipping like Castlevania? Or is it more like Luigi's Mansion?

Obviously, the horror genre splits up in several directions. Like Braindead, The Shining or Twilight(shivers) aren't the same things either. If you read the "Genre / Gameplay" or the T22 website, you do have an indication though. Tower22 won't be about killing things all the time. It's more focused on exploring an environment that gets stranger and stranger, solving puzzles, and trying to stay away from boogeymen. And as it comes to the looks, it will be a gritty semi-realistic "Soviet" style, mixed with a bizarre nightmarish/dreamy style. However, that still does not explain the deeper details or core features that should make this game "fun" or "scary" (the paradox about horror games is that they're often not fun at all in order to make them scary).

Each game attaches itself to several (new) features, trying to make a flagship of those. “Babes & Guns”, "Unbeaten 3D Graphics, using the Super FX chip!". "Customize your underpants", "Defeat enemies by combining magic spells with your turbo Vortex-Spin-Moves!". Although I didn't manage it for T22 yet, try to make a single phrase catchy slogan that described the best part of your game. Yes we can! Well, powertalk or not, in the end the fun-quality of a game depends on which rules or "mechanics" were chosen, and how well it was done. Brew the right concoction of game ingredients. 45% jump-force, some shotgun, a bit of doors with keys, et cetera. Combine that with a proper implementation, meaning your controls/game-world/design style/story exploits these ingredients wisely, and you have yourself a good game.

Easier said than done. As said, artists and audio composers need to catch the style that blends perfectly with the game theme. The programmers need to code the controls, physics, puzzles and A.I. like a well oiled machine. The map builders need to design the world in such a way it lends itself for the chosen gameplay features (whether that is jumping, gunning, running, racing, puzzling or whatever). If you could do it all yourself, you would do it right of course, as it's all in your head. But we all know we'll need extra people to realise a game project. How to make sure all of them are facing the right direction? Exactly, by giving clear instructions. And making a Game Document is one those help-tools.

Just writing an A4 with a global description of the game idea isn't enough by far. When it comes to fine-tuning, all details need to be provided. And that goes deeper than you may think. How fast is your player exactly? How does the stamina system exactly work? How long does it take before an enemy returns fire after seeing you? How much items can the player carry? Should the player be able to jump? And each feature needs to be weighed with care. Do not just throw in elements because some other cool game has it too. For example, only add the ability to roll up in a Morph Ball if it fits with the story, style, and if the environment provides plenty of puzzles that requires this feature. Otherwise it would feel like a dumb gimmick, out of place.

This will result in tons of text. And to make it worse, that text is likely going to change over time as elements need to be play-tested. Being able to do a Rambo ball-twister twirl might sound like a good idea at first, but after some testing it could still suck. Which requires parts to be rewritten / adjusted. As a machine programmer who writes manuals or guides occasionally as well, I know 2 things about documentation.
-----A: Writing them takes a lot of them, maintaining them even more.
-----B: Nobody really reads them.

Which brings me to C: documents -if there are any- are outdated, getting delayed. Documenting is a lost child, certainly in smaller companies/groups where the first priority lays on making the actual product. We all know we should write our stuff down, but... not now.

What we got cooking? A stove. Yet we have to make it a bit more dirty and old for the finishing ugly touch.


Wikipedia
This brings to Wikipedia. Wiki... That word always make me think about tropical juice with a package design containing monkeys swinging over pink crocodiles in a jungle. But I don't have to explain you what Wiki(pedia) really is. What is important, is that Wiki works. It contains a HUGE amount of information, it gets expanded, updated, refreshed and corrected every minute, and moreover, people read it. Not just professors with white moustaches smoking pipes, everyone does it.

Hmmm... wouldn't it be a good idea to use some Wiki power for your (game)documents then? Well, thanks to Brian here who pointed this out, I learned this is possible, and quite easily really! You can download "Wikimedia", the toolset that allows to install your own “Wiki” on a server computer (you can download Wamp for the additional components to setup an Apache + SQL server required by Wiki). Now if we think about Wiki, we think about a worldwide shared encyclopedia. But don’t forget you can set it up in a private network too, making it suitable for companies or turds like me who like to keep their game-document secret for now.


All right. But what exactly makes this better than any other random documentation system? Wiki, PDF or goddamn Cuneiform, the contents stay the same right? Well let me explain. But instead of taking a game as example, I’ll take a harvester-machine. Yep, I went to England last week to study a machine of our friends over there, in order to document the whole thing. Why? Well writing down stuff triggers you to learn the matter, as you do research while writing. And of course, it’s supposed to bring over knowledge to other engineers/programmers/service people some day. But as said before, writing this document involves several problems:

#1 It’s huge. As I can’t finish it in one or two days, there is a good chance a higher priority project will interrupt, leaving a half-finished(=useless) document.

#2 The machine will be changed / updated in the future. Having to do a revision is a lot of work though, as you’ll need to check the entire document for changes. This often leads to outdated or even faulty documents.

#3 Making 1 big document that reads comfortable, requires writing skills.

#4 I know the programming parts, but not specific details about which hydraulic valves were used, how a wheel steering sensor exactly works, or the electric schedules of the cabin. Need help from others, but they face the same problems and writing in the same document sucks. Having a pile of separated files sucks as well, unless well categorized.

#5 Do you really think a new programmer is going through all that stuff? Probably he will suggest to rewrite the system in his own way. So for who & why did you write that document? And even people want to read it, can they still find it 4 years later between the huge pile of other documents?

Plenty of good excuses you can use to convince your boss to keep you away from boring writing work. But sorry, Wiki eliminates all these problems more or less. Which is probably also the reason why it’s such a big success. First of all, instead of writing long chapters, you should try to write your system as separate small blocks. Don’t worry about the relation between those blocks yet. For example, for this machine I could write a specific page about the Joystick, how the Cruise Control exactly works, or the Dieselengine. Or to map it to games, a block about “Healthbar”, “Enemy 3”, or “Player biography”. There is no limit to the Wiki page length, but I’ll advise you to keep the pages short and to the point. One or 2 “screens” at max for example, and just pin down the facts and numbers rather than making a flowing story with “maybes” or “possiblys” that raises questions instead of answers.

This solves the “#5 reading” problem. Instead of having to scan large documents for useable text, the end-user now does a certain query. Want to know more about how the brakes work on this machine, or how the inventory should be implemented in your game? Search “Brakes” or “Inventory”, and go directly to a compact page. No bullshit, just useable info. Which also helps less experienced writers (problem #3) btw. Summing facts is easier than creating an informative, yet readable story.

As we know, Wiki allows linking. A page about Napoleon Bonaparte could refer to another page about Waterloo, or French Cheese. This allows to zoom in further and further. When I describe the machine software, it starts with an overview of main functions such as “Driving”, “Steering”, or “Engine”. Then each function gets its own page containing more detailed info. How it works, which sensors / actuators are involved, adjustable parameters, common problems (for a troubleshoot), et cetera. Then we can dive even further. A page that described the related source-code, or specific details about hydraulic valves being used on that particular system. Manufacturer, suppliers, maximum load, installation schemes, known problem, … In a game document, the description of a certain level could refer to characters, weapons, and other entities being used there.

Tying together the blocks allows to make a rich and informative system, yet remaining clear as the individual pages remain relative short; the reader decides how far he zooms in. Asides, it also solves some more of our typical problems. You can expand your Wiki step by step. A half finished document isn’t readable, probably neither available either as it still floats somewhere on the local hard-drive of the author. But you can already make use of a Wiki that only contains information on the top levels. A dead link simply brings the reader to a “to be written” page. Your Wiki page just gets an address like any other website, and can be seen by everyone (with access to your network) with a web-browser. This makes it easy to find, even years later, and hence, it even encourages you or other authors to fix the Wiki in case of a dead link or error. Maybe I don’t know crap about cooling fans, but if an engineer reads the document and bounces on faulty info or an unfilled page, he can quickly edit it. This makes the documentation more complete and easier up-to-date. Btw, in the case of Tower22, many of the Wiki pages also generate (concept)drawing tasks for the artists



Wiki works, and the internet proved that over the last 10 years. So if you are struggling with the documentation, feel like no one reads/checks or contributes your hard work, or getting tired of piles of files, Wiki might work for you. Whether you are writing about games, harvesters or your stamp collection.