Tuesday, January 25, 2011

Mega-Structures; Geometry Shaders #1

"Some American scientist claims that using rap makes a good way to remember and understand mathematical formulas"...
Yo yo, Pythagoras in da house. F%ck you b!tch, MC^2 on the mic, spitting square roots that make you poop in your boots. What comes up, must go down, I'll bombard you and your homies with g=(m1 + m2) / r^2. I don't talk algebra operators, I shout integrators. Respect, subtract, smoke my crack, Blaise Pascal out.

Yeah, that would learn them... O pardon me, that was the first thing going through my mind when just hearing this little news on the radio. All right, let's rap something else: Geometry Shaders. have a moment for Snoop GPU, Easy E++, Dizzy Pascal, dirty old Register. and dr.Hashpipe.


Geometry Shaders? Who what where?
-----------------------------------------------------------------
"A Geometry Shader can generate, adjust or discard primitives(triangles, dots, lines, ...)"

Even when you never touched a keyboard, you probably heard of "Ssshaders". Vertex- and fragment(pixel)Shaders. What they do? They are like tiny programs running on the videocard "GPU's", telling how to compute their output: vertex coordinates & pixel colors. Vertex shaders can move the vertex / UV coordinates. Fragment shaders calculate the pixel colors. Usually based on quick & dirty lighting physics that approximate the real thing. Some practical examples (from the streets yo):
Vertex-Shaders
- Transform 3D coordinates to screen space / eye-space / world-space / cyber-space
- Animations; apply matrices from 1 or more bones to each vertex ("skinning")
- Water ripple physics
- Displacement mapping (read a value from a texture such as an heightMap)
- GPU cloth / particle physics

Fragment-shaders
- Diffuse, specular, ambient and emissive lighting computations
- ShadowMapping
- Reflection / refraction computations
- Cell-shading (cartoon look)
- Drawing specialized textures required for other techniques (depth buffer for example)

Well if you did some shaders, that's nothing new. Hey, they already exist for about 10 years now! The word "next-gen" can be replaced by "common-tech" again, time flies. The recent "Geometry Shader" program isn't really that new either anymore. It appeared somewhere in 2007 or 2008 I believe. How does it come I didn't notice it already then? Hmmm, let's just say it's usage is pretty limited in most cases.

The “GS” is an optional third program that can be executed after the vertex-shader. Unlike the Vertex Shader, the GS does not take single vertex-points, but complete primitives as input. Primitives? You know, dots, lines, triangles, the stuff you pass with a glBegin() command. So, when working with triangles you get array’s that contain 3 positions/texcoords/tangents or whatever you pass in the vertex-shader.

The GS passes this data further on to the Rasterizer(and then Pixelshader). But you want to perform some custom actions here right? Like said, you can decide here whether to "emit" a primitive or not. You could perform sort of culling here. Of course, you can adjust all the coordinates before passing them as well. But even more spectacular is the ability to generate new primitives. When processing a simple line as input, you could break it up in 10 pieces and bend it into a curved line. It's kinda like an advanced vertex-shader, with more input data and a flexible output. And just like in any other shader, you can use custom parameters including texture reads. You could for example render a terrain with rather large quads, then let the GS sub-divide the quad based on the camera distance, using a heightMap texture to apply the correct height value for each new vertex.
Some applications
- Handling sprite/particle clouds
- Tessellation / Level Of Detail
- Beziers, curves
- Fur / fins / grass
- Generating silhouettes (stencil shadows, light shafts, …)
- Render cubeMaps in one pass (this is why I got interested finally)


WIP, Work-in-Progress. Very simple 3D geometry, lot's of stuff missing, not very special so far... Except the fact that all textures are made by our own man Julio so far, instead of stealing them somewhere. Oh, did it use Geometry Shaders? No, but since lot's of objects are reflecting the environment here, a single-pass cubeMap technique can boost the performance pretty well...

Simple demo Code?!
-----------------------------------------------------------------
Let's do an example with Cg shaders. Ow, GLSL and some other shader languages support GS as well of course, as long as you have a card that supports Shader Model 4 or higher(I believe). In Cg, creating a GS program is pretty much the same as creating and using vertex- or pixel programs:

CG_GL_GEOMETRY = 10;
// Setup
gpProfile := cgGLGetLatestProfile(CG_GL_GEOMETRY);
prog := cgCreateProgram( cgContext.context, CG_SOURCE, pchar(code),
gpProfile, 'main',
pArgs );
// Use it
cgGLBindProgram( prog );
cgGLEnableProfile( gpProfile );
// --> render something

Now let's find a good & simple application for a Geometry Shader... Uh .... That's difficult. Maybe GS isn't that useful yet... Wait, how about an electro / lightning bolt? There are more ambitious demo's, but therefore I'd like to refer to the nVidia SDK's (check OpenGL SDK 10 for example). All right, here's the idea:
- Render 1 line (glBegin( GL_LINE ), 2 points)
- Let the shader break it up in 20 pieces, then applying random coordinates on each sub-point to make sort of a zig-zag.
The CPU can do that just as well, I know. But it's just to demonstrate how a GS program looks, smells, and works.

// Dammit, SyntaxHighlighter doesn't allow to use the >< characters, so I placed
// float3 between || pipes instead
LINE void main(
// Input array’s
AttribArray|float3| position : POSITION, // Coordinates
AttribArray|float2| texCoord0 : TEXCOORD0, // Custom stuff. Just pass that

uniform sampler2D noiseTexture,
uniform float boltMadnessFactor // Pass params as usual
)
{
// The bolt is just a simple line with 2 points:
// point 1 is the origin, point 2 is the target (impact point).
// Now apply "The Zig-zag Man"

const int steps = 20; // You could use distance LOD here, although that
// won't be needed for huge lightning bolts
float3 beginPos = position[0]; // Get input for simplicity
float3 targetPos= position[1];
float2 beginTX = texCoord0[0];
float2 targetTX = texCoord0[1];
// Interpolation values
float3 deltaPos = (targetPos - beginPos) / steps;
float2 deltaTX = (targetTX - beginTX) / steps;

// Output first (begin) point, don't modify it.
emitVertex( beginPos : POSITION,
beginTX : TEXCOORD0 );

// Generate 18 random points in between
for (int i=1; i < steps-1; i++)
{
// Interpolate position and texcoord custom data
float3 newPos = beginPos + deltaPos * i;
float2 newTX = beginTX + deltaTX * i;

// Just pick some random value from our helper texture
// then use it to modify the position.
// Reduce noise strength as the bolt approaches target
float3 randomFac = (2*tex2D( noiseTexture, newTX ).rgb-1) * boltMadnessFactor
* (steps-i);
newPos += randomFac;

// Generate extra line point
emitVertex( newPos : POSITION,
newTX : TEXCOORD0 );
} // for i


// Output last (target) point, don't modify it.
emitVertex( targetPos : POSITION,
targetTX : TEXCOORD0 );

// Of course this shader sucks. In fact, I didn't even try it :#
// But it roughly shows what you can do here.
} // Roger out

Still messing around with SyntaxHighlighter. It reminds why I hate HTML and such. How to shrink the gap between a line?! Why do the tabs suck so much?!

The next part of this Geometry shader adventure will follow in two weeks. More about rendering stuff in a single pass then. One of the best reasons to use that weird Geometry Shader :)

Oh, for all us forgotten Delphi souls. Can someone explain where to upload a small code-file so it can be downloaded here? The last Cg DLL headers for Pascal I could found were for Cg 1.5, written by Alexey Barkovoy in 2006. So I updated them a little bit. Warning, I haven't tested the whole though!!!


Another WIP. Same stuff, different light.

Monday, January 17, 2011

Turd in the punchbowl

Rest in peace Major Richard Dick Winters. Now I don't this guy personally of course, but the impressive "Band of Brothers" series showed the actions of Easy-Company during WW II pretty well. Yes, asides from taking part on a couple of key battles, they really entered the Eagle's nest, discovered a concentration camp, and had that little soap with Capt. Sobel (the real one, not that guy from Friends). War, isn't it romantic? Well...


Maybe you can't care less, but here another opinion from a computer geek about Julian Assange & Wikileaks. Saw a coverage on a Belgium TV channel, so the type machine in my head started ticking.

Before starting, once again you and me should be happy we are free to write on a place like this. In my particular case, I might need to thank Easy-Company for that as well(operation Market Garden wasn't that far from here). Sure, maybe the Pentagon is watching, but so far I'm not really limited when being critical, or making jokes about Jesus balls. Wait, what is that car with the blinded windows doing outside here all day?


I haven't read through all those leaked documents. Only heard a few on the radio. Was it world-shocking? No, not really. I found the American opinions about other politicians & leaders in this world rather amusing. Especially "Batman & Robin", Poetin and Medvedev, was striking :)
Come on, EVERYONE talks shit about others. And so do countries and their leaders. Was it really necessary to leak that info? I mean, you can claim you have nothing to hide. But if some friend hacks your SMS and email accounts, I'm damn sure they'll be finding little secrets or nasty things you said about friends, your boss, your girl, your penis or whatever it is. Spreading such info is... a little bit childish. We want transparency, yet privacy at the same time. That collides…

More serious are the videos like the U.S. Apache above Baghdad of course. What you see there is horrifying. A pilot who pulls the trigger, tearing apart a group of "terrorists". Even more shocking is the radio communication. The other guy on the line permits to fire, without even asking or hesitating. As if they simply had to sweep a dirty floor... Does these guys have beards? Well let the .30 mm hollow point spit already. However, such images were already available before Wikileaks. And sure they are 100% wrong. But does it surprise? It's a damn war there. Or did you think everybody played by the rules in the romantic 40's? War is ugly per definition. And when seeing documentaries like "Restrepo" on National Geographic, it doesn't surprise me at all that some flip out completely. Is that an excuse? Certainly not, but we shouldn't act as if we never expected this either. That's just naive, stupid, or lying.

Another leak. Parts of the Player(first concept). Should we be transparent and throw all the game info on the table right now? Or is it better to keep it hidden for now?

Julian... Good or wrong? Internet-hero of freedom, or a turd in the punchbowl (Southpark), trying to make havoc? First, let's forget a moment about this Assange guy. Whether he is a girl raping devil or Mickey Mouse, it doesn't change the fact of sensitive documents being leaked & the contents of them. The goal of the Wikileaks organization (and the likes) is not just to share information. It's trying to force transparent politics, “true democracy”.

Yeah, one can't deny a lot of mysterious stuff is going on. I'm not the type of guy who believes 9-11 was an inside job, or clamping to conspiracies that also just rely on a few vague "sources". If you can't trust the government, then why trust "ctrlAlrDel H@ckEr '86" then? But you shouldn't close your eyes either when Michael More has something to say. Shit happens, all over the place. And it's not just America and their war on terrorism. North Korea & deathcamps, China & censoring, Holland & billion costing projects, uniting European countries, justice going wrong because of “mistakes”, Bilderberg group, pharmaceutical companies, oil concerns, Africa & Corruption. And all the stories about Soviet/Communistic regimes we are reading for Tower22 doesn't cheer up the vision about mankind either.

Does money & power turn people into selfish beasts? Or are only the selfish beasts attracted to (political) positions that provide money & power in the first place? I dare to say many of the politicians have their own agenda's, and sure not all for the good. If the system would be forced to make their plans and ideals transparent, wouldn't that make a better world on the longer term? I think so, really. We are not talking about cheating a little bit while playing Super Mario Kart. We are talking about serious issues, having effect on millions of people.

Off-topic. With reading juicy background info about the Soviet era and such, this is what I mean. Finding reference websites is part of the job to give drawers & modelers happy inspiration. This cozy town is somewhere in Siberia. Yes, even the weather sucks there.

But as with most of these almost utopia-ideals, the world doesn't work that way. Or at least, not yet. What if the Western world suddenly sais "Sorry Julian. Here a lollypop, and from now on we share everything". Would he be happy, or is this just a personal crusade against U.S.A.? Anyway, information and knowledge is a powerful thing. Not only for us voters that believe in democracy. Also for "the enemy". And I'm not talking about a bunch of “Allahu Akbar!!” yelling, running bombs. Do you really think other important players like Russia, China, Iran or North Korea would sign this "be cool, be transparent" contract? Or how about more local criminals? Tribes, gangs, mafia, extremists, drug cartels?

Great goods like the freedom of speech, voting, social support, or human rights are also our fragile weaknesses, as they are easy to abuse. Why not kick the balls if you don't have to play by the rules anyway? Other regimes, criminals, or extremists know that very well. You can try to be the best kid from class all the time, but you'll have to play hard if you want to win. If everyone uses steroids in Tour the France, there is not much else you can do except becoming one of them.

Politicians master this game, and know when to cheat way better than we normal people, not caring too much about money & power. Then who is right in the end? I'm afraid no one is. A social worker with the heart on the right place can't run a profit-making business between all the competition. Sure he/she wants to do it right, but a naïve attitude at the wrong place brings the whole company in danger.

Sharing all our information is like disarming the whole world from nuclear weapons, except North Korea; it only works if EVERYONE plays according rules. Which is clearly not going to happen anywhere soon.



So, is leaking all wrong then? No, I guess not. Look, this whole thing started the discussion about our ethics, politics and global roles on this planet. And for you and me, yet another showcase not to follow everything blindly. Always think for yourself. You have to start somewhere if you want to make a revolution. Maybe a more subtle way would have been wise, although... People don't tend to change old rusty habits, unless they are confronted with drastic news. Tell a smoker that he will die if he doesn't stop, and he will laugh. Tell that we screw up the environment, and we will be laconic. Show some soldiers who shoot an injured men, and we will say "Oh well, war is hell". It takes some dead relatives to set the smoker in motion. It takes a big (overdone) movie from Al Gore to start those brains thinking about the environment. And it might just as well take a massive leak to start questioning our governments and "democracy". Small doses don't help, as we like to face away from problems. NIMBY, Not in my backyard.

NIMBY? What strikes me in many "Soviet inspiration" pictures is that EVERYTHING is left behind as if… Godzilla suddenly appeared on the scene. For example, why not having a sub-marine in your backyard?


Good or wrong...? I can't chose, it's not black and white. Personally I think Wikileaks and the likes should think better about the possible consequences. What if Batman & Robin decide to break contact with the Western World. Does that makes us all better? What if Afghan helpers get found and slaughtered by Taliban? What if a crazy bored fool gets the lay-out of a nuclear power plant in your country? What if ...? One shouldn't live in fear, but thinking that people won't hurt each other is naive.

But ok, the leaking already happened. Whether you agree or not, let's focus on the future. It would be nice if America wouldn't respond that harsh on this whole WikiLeak thing. Dangerous or not, learn something from it. And not how-to hide information better the next time, but to stop lying and to prevent "incidents" like Abu Ghraib or pilots playing Call of Duty. Don't shoot the messenger (literally, in the case of Julian who probably has some sleepless nights). The same lesson goes for all the other leaders. Take responsibility for your deeds.

But to be honest, I'm afraid that's just wishful thinking. Business has to go on, with all the related unethical aspects. Therefore I tend not to care too much about. Just make the best of your own lives and help the ones who come along your path.

Sunday, January 9, 2011

Take me to Screen-Space land

- As promised previous week, a techno-post here. Next week it will be less technical again. -

While making a new real-time GI technique, I ran into SSAO & SSGI again. Sounds like a sexual disease and a new crime series, but actually they are screen effects to enhance(fake) ambient lighting / occlusion:
- SSAO = Screen-Space-Ambient-Occlusion
- SSGI = Screen-Space-Global-Illumination

SSAO
Programmers here probably know SSAO already. Crytek showed this technique in their first Crysis game, a couple of years ago. Hell, time flies. In short, SSAO is a (post)effect that approximates the occlusion per pixel by using other pixeldata from the rendered screen. The idea is as follow: the more nearby/surrounding objects, the less (ambient)light a pixel catches. This is somewhat true. Look in your room and you may notice the corners or space below cabins are darker. Because less light reaches here of course.

Well, it all depends from where the light is falling in. But as graphics programmers probably know, it's near to impossible to compute where indirect light comes from, in real-time. SSAO is another cheap hack to approximate this effect without actually knowing shit about light. Although cheap... even today it requires quite a lot to compute decent looking SSAO, while the effect isn't that big. Which is why I had to have a look in my existing SSAO shader again. And as some notified on Youtube, the SSAO effect caused an ugly black halo around objects.

Nice thins about Screen-Space techniques is that the complexity of the scene doesn't matter at all. 1 box, 600 planets, pixelcount stays the same.

How does it work? Pretty simple, although the shader requires a few error-sensitive tricks: For each pixel on your screen:

- Calculate where it is (reconstruct position via depth, or store positions in another deferred buffer)
- Take n samples in a circle around the source pixel. Check if these neighbor pixels are occluding
the source pixel by comparing depth/positions, and eventually normals.
- Take the average of all samples
- You can use a (Gaussian) blur pass in the end to smooth the results
- In the end, multiply the grayscale SSAO texture with the (ambient) scene.

Since the amount of samples is limited, 16 or something, you only have a relative low amount of references. Don't make the sample circle too large (which is why SSAO only works locally, in corners and such), and use a "dither" or noise texture to vary the sample coordinates for each pixel. Some pixels sample nearby, others \ use a somewhat bigger range. This leads to varying pixel-results, but a blur can smooth that away.

Besides taking the proper sample coordinates, the difficult part is to decide which neighbor pixels occlude. I've seen several implementations, but in my case they always lead to weird results. Half-grayish walls when rotating the camera, or an either darkened or highlighted edge everywhere. With the wrong comparisons, SSAO quickly looks like an ordinary edge-detector effect, while it shouldn't be. So instead of lazy copying shaders from others, I took a try for myself this time:

for (int i=0; i <16; i++) // a few less samples is possible
{
// Create neighbour sample texcoords
// w and h depend on a variable sampleRadius, screenSize and distance from camera
half2 tx = half2( iTex.x + sampleDir[i].x * w , iTex.y + sampleDir[i].y * h );
// Get neighbour data
half4 nbPix = tex2D( posTex, tx ).xyzw; // get WORLD position & depth(w)
half3 nbNrm = tex2D( nrmTex, tx ).xyz; // get WORLD normal

// Occlude if:
// - Neighbour pixel is not too far away
// - Direction between the 2 pixels can affect on the sourcePixel normal
half3 dir = nbPix.xyz - srcPix.xyz;
half dist = length( dir );
dir = normalize( dir );
half shineFac= saturate( dot( dir, srcNrm ) ); // Prevents self occlusion, compare with source normal

half ao = shineFac * saturate( (nbPix.w - srcPix.w + maxPixDist )*10000 ); // Discard pixels that are too far away
aoSum += ao;
} // for i
ao = 1 - (aoSum / 16 );

It uses the deferred render buffers as input instead of depth reconstructions. Simple, just like my brains are. Probably not the fastest way around, but it works pretty well. Also on background buildings. It prevents self-occlusion or foreground objects to mix with background stuff. Surfaces can still self-occlude with their normalMaps though, although the effect is barely visible in the end-result. But if you get it for free, why not. Bullet hole decals that affect the normalMap will create darkening for example, pretty neat.

Tip
Not implemented here yet, but I always have fights with the skybox as that area doesn’t have a position, normal, or depth by default. By rendering an extreme high depth in the position or depth buffer (glClearColor( much ) ), it will be skipped here now though. You can abort the shader right away when the sourcePixel depth is also high, as you don’t have to process the skybox. In outdoor area’s that can save up to 50% of the calculations!


SSGI
Anyway, what I really wanted to share was that other technique: Screen-Space-Global-Illumination. Used to spread light to create "color bleeding". No, that won't be my top-secret next take on real-time G.I. but it *might* be useful to complete it. Just like SSAO. Due limitations the few "realtime G.I." solutions available so far, including the Crytek LPV one, are still computing the indirect light distribution on a rough, inaccurate scale. To deal with the small details, SSAO and SSGI can be used. Crytek for example uses SSGI to approximate G.I. for background scenery that falls outside the LPV workspace (3D volume textures around the camera).

So... what is SSGI then? If you can compute occlusion by looking at neighbor pixels, then why not using it to reflect (direct) light? Hey, another nice usage of the Inferred Rendering pipeline approach, where we produce a diffuse & specular light screentexture. Just copy the SSAO shader, and in addition read the diffuse value from the neighbor pixels. I also read the reflectance & emissive value from a second texture. Those colors roughly represent the outgoing light from a neighbor pixel. Now we only have to test if it reaches the source pixel... yep, same stuff as the formula I did above, but with a small addition:

half3 giCol = tex2D( diffuseTex , tx ).rgb * 0.5f + tex2D( additiveTex, tx ).rgb;
giCol *= shineFac * saturate( dot(nbNrm, -dir) );
giSum += giCol * saturate( (nbPix.w - srcPix.w + maxPixDist )*10000 );

You can simply add these lines in the loop so you can calculate AO and SSGI at the same time. Ow, the brighter the G.I., the less Ambient Occlusion should occur of course. You can simply lerp between the two, based on the G.I. result luminance.

Direct light falls on the ground here, then the surrounding walls / objects pick it up again. Without any G.I., the backsides of the boxes would be pitch black. Also, the emissive monitor creates a blur.

Life can be so simple. But does it really work? Hmmm... well... Three problems. First of all, it makes the already expensive SSAO shader even nastier. Second, the SSGI effect is, just like SSAO, only very local. Again, you barely see it unless applied on really bright colored objects such as a computer monitor or bright green plastic wall.

The third problem is the product of problem one and two. To make the effect more noticeable (worth the additional cost), SSAO and SSGI should use different sample radiuses. The bigger the circle, the wider the light spreads (or actually gathered) of course. But that doesn't work too well with SSAO, unless you like blurry crap. So, the only proper solution I could think about, was to put the SSGI in a separate loop that uses a wider sampling range around the source pixel. And thus requires even more horsepower, for just a small effect. Is it worth it? Mehh, if you target for somewhat older hardware, NO.

In a scenario like this SSGI helps (though a cubeMap could do as well). But when the hell do you see things like this?

Just when I pushed the speed to 70 FPS (30 on my older card), SSGI is making havoc again. Currently SSAO & SSGI are done in the same pass, on a buffer half the size of the screen. What I could do is moving SSGI to a separate pass on a 1/4 sized buffer. Less quality, but then again SSGI allows more blurring & smearing than SSAO does I think. Didn't try it yet though.


And that boys & girls, was probably the most technical piece of text I ever wrote.

Tuesday, January 4, 2011

In the year 2011

And a healthy 2011! I already made a good start, producing magical shader/particle effects above the toilet on 1 January morning. Nah, I didn't make any promises. Nothing is going to change when it comes to bad habits, as I actually love my bad habits too much to give them up already. Although taking up jogging may be a good one. I used to run ~6 kilometers about three times in a week to a place where my friends gathered... to end up with a beer and a cigarette, very healthy. Nevertheless, after two years I finally had that Mike & Jim Ab-Pro belly. Leading to a girlfriend, leading to get well fed by her, leading to getting lazy as you don't have to score a girl anymore, leading to gain 25 kilograms again :) Oh, when she complains about that, I have some words to defend myself:
"At least you don't have to be jealous & nervous for the competition of other girls, as they don't look at me anymore"
1-0 for the Fatman.



Well, 2010 was a bumpy year for many. Quite a few big disasters here and there, a record when it comes to the death toll actually. Leslie Nielsen died, North Korea barks again, half of the Polish government died in a plane crash on an already terrible place. The Economical crisis saga continues, eating jobs along with it. Cancer took the lives of a couple of beloved ones, and Holland took a relative drastic turn in the political spectrum. Even worse, it seems the iPhone alarm didn’t work on 1 January. Thank God I’m still using an old Prince of Bell air Buzzer. My phone isn’t even capable of calling, let alone running a real-time clock!

For me personally, 2010 was an easy ride. Nothing really changed, except that our little girl learned how to talk my ears of. When it comes to game-development, a few milestones were set though. This blog is exactly one year old now, but the desire to make a game was already there after playing Doom2 when I was 11 years old. I've been trying to do something for years and years, but so far I never really shared it with another person. Don't like to enter the spotlights, so starting a blog to announce something was quite a big step.

Sooner than I expected, this blog would get the attention of a few readers. Thank you for that! Getting noticed and receiving some feedback is what it makes all worth it. Sure I know sites like these won’t reach Perez Hilton statistics, but really, it boosts the enthusiasm and devotion for this project. In fact, it triggered to make a very first movie of a real project: "Tower 22". Placing that movie on Youtube and getting all the positive feedback was something I could only dream of when 2010 started. Who the hell would be interested in the programming attempts of yet another fool on the internet?

As a cherry on the pie, and what I expected least, was to form a small team in 2010 already as well. Initially I intended to delay such a request for help until I really got something awesome to show. But hey, what the heck. I've been way too passive on this whole game-programming thing for too long, just go for it! Wait too long and… as Acda & de Munnik sing:
“van al zijn jongensdromen was alleen het oud worden behaald”
Which means something like “from all his youth(boys) dreams, only “getting old” has been realized.

And so the prayers were answered. Not with my 14 year old nephew who would like to help after learning Java and MS Paint for 2 months. But with real creative people, some of them with actual experience in the (commercial) game-biz.

Quickie by Julio. Making ideas for the environment in a second demo...

All in all, 2010 was a fabulous year for this project. And the nice thing about a blog is that you write your own history/diary as you go, this whole course has been documented so far. Nice for the grandchildren around the hearth over 30 years. But we have to look forward. There is no game yet. In fact, the whole thing has just begun! Besides implementing new techniques into the Engine, 2011 will be the year where I hopefully learn how to instruct a small team. Hey, it's not easy to make fun & challenging assignments for five hungry men! Let's hope we as a team can lift this project to a next level;

2011
Game & Team
- Player character (not that rusty box-robot)
- Making a more definite game-plan, including environment sketches
- Documenting those ideas in a private(sorry!) Wiki
- Second (and maybe third) demo movie with more gore, more atmosphere, and more advanced techniques
- Creating a real sound library instead of "borrowing" it from other sources
- Maybe looking for another modeler/mapper when Demo #2 releases

Programming
- Volumetric light-shafts & fog
- More lights & better shadows
- Geometry shaders
- Upgraded real-time G.I. (ambient lighting)
- FMOD sound library
- Upgraded AI module
- Upgraded Physics module, possibly with the help of a second programmer

As for this Blog, a few things may change as well. As expected, the poll turned out that most of the visitors would like to see some more technical specs. Not a surprise, although I won't turn this blog into Nehe, Humus3D or the likes. I still like to maintain the non-technical aspect as well. So, after some puzzling I thought about:

- 1 week a technical post, the other week a non or less technical post. So, you know when to skip a post ;)
- Can't guarantee a post every Sunday / Monday anymore. Man, I'm just too busy!
- But, hopefully some of the other team members can occasionally write about their sound / modeling / drawing or writing experiences. Meet the Creative side!
- Trying to make a few more short game-stories. Not revealing clues though.
- To make the technical info somewhat more accessible, a new “indexing” page that refers to other history Blog posts. Well, just have a look the
“blog index” page to see what I mean.

Next week I'll tell you something about either Geometry shaders or Light shafts. Well, a happy 2011 to all of you!

Monday, December 27, 2010

Need for Speed

Right, how many kilograms of beef, potatoes, ice-cream, chocolate saus and bread did you eat last weekend? Enough to promise losing some weight in 2011? Ah Christmas... All those magical cozy lights, Wham! music, and not to forget: Home Alone, Critters and Gremlins 1..18. But the best memories are probably those of unwrapping Command & Conquer, Goldeneye (n64), Zelda OOT / Majora's Mask. Each year my little brother and I would nervously wait for out next game. Inspecting all the packages beneath the tree and knowing the exact dimensions / weight of a N64 game, we already knew which box to keep an eye on weeks before 24 December.

The time of getting has transformed into giving presents. Hence, I wouldn't even know what to ask anymore. The downside of getting older is also getting more spoiled. At least where I live. What do I need anyway? A working computer, a chair, a bike to go to work. Clothes maybe... There is more joy in buying Shrek for our daughter, or giving a Blu-Ray player to grandpa.

However... I realized my videocard was pretty old again. Bought it end 2007, so that
is ~25 in dogyears, and 2.435 BC in hardware-years. In other words, extremely old in
hardware-land, where videocards older as fast as they render pixels. So, after donating Santa some money, a shiny box with a EVGA GeForce 4700 GTS came in. And damn, it even worked right after replacing it with the older card. Our family has a long history of fooling around with computer parts. Dad never bought a complete (working) system. 4 MB RAM here, a 60 Hz processor there. A 0Kb modem elsewhere, etcetera. And of course, it NEVER worked. Had to travel the entire country with dad to get computer parts in the summer of 1994, waiting weeks and weeks before I could finally play Doom2 (with PC-speaker).

Was it worth the money? Hmmm, I can imagine there are more useful things in life, but:

- Tower22 on GeForce 8800 GTS (640 MB) : ~30 FPS
- Tower22 on EVGA GeForce 4700 GTS : ~56 FPS

Almost doubled, pulled the T22 Engine out of the mud. But don't worry, we'll bring that card down to its knees again in no-time, begging for mercy. More light, realtime volumetric lightshafts/fog and updated Ambient lighting are on the menu.

Work in progress: improved volumetric light. Not blurring a bright spot, but raytracing through space to see "how much particles" were lit. The lower-left corner shows the lightshaft-buffer.

FBO Sandwich
Talking about speed. As a programmer, I'm sure you wondered several times
"How the hell can Crysis/Halflife/... run that fast on my machine, while my game runs like a crippled grandma?"
Ifso, here a last programmers advise for 2010.

After the transformation into the Inferred rendering pipeline we discussed earlier was completed, the speed dropped from ~30 to ~22 FPS (on the old card). Inferred Rendering has slight more overhead, but that drop was ridiculous. Where did we go wrong?! Bad shaders? Maybe the new shadowMapping storage technique (I'll discuss that another time)?

Then an old fiend flashed by; Captain Framebuffer. In OpenGL terms, a FBO is a collection of targetbuffers you can render on. Well, with all those (background) techniques, we change that FBO plenty of times. But as I discovered years ago, when playing around with shadowMaps for the first time, mistakes are easily made. A wrong switch or MRT setting, and your engine neck snaps like a lucifer stick. Here a few important advises:

Try to prevent switching resolutions
Switching targets always takes time, but especially when hopping from one resolution to another. For example, a pipeline might do this:

- render to 4 1024 x 768 textures for Deferred input
- render to a 256 x 256 texture for a light shadowMap
- render to a 512 x 512 SSAO buffer
- render to another 1024 x 768 texture for depth
- render to a 512 x 512 DOF input buffer

Five switches. With all those techniques, switching is enevitable. But at least you can order things better:

- render to 4 1024 x 768 textures for Deferred input
- render to another 1024 x 768 texture for depth
- render to a 512 x 512 SSAO buffer
- render to a 512 x 512 DOF input buffer
- render to a 256 x 256 texture for a light shadowMap

See that? Only 3 switches instead of 5.


Make a FBO for each resolution
Not 100% sure about this, but people say it's best to make a FBO for each possible resolution, instead of only changing the rendertarget for a single FBO. In the example above, we would need 3 FBO's; 1024 x 768 , 512 x 512 and 256 x 256. Each one can have it's own depth buffer.

Atlas renderTargets
Use bigger atlas textures to perform multiple passes in a single buffer texture. When having to blur or downscale, you quickly end-up with a large number of different resolutions. For example, the HDR technique requires to downsample the average luminance of the screen contents. At first, my engine would do this:
1.- Render luminance values to a 128 x 128 texture
2.- Downscale step 1 texture to 64 x 64
3.- Downscale step 2 texture to 16 x 16
4.- Downscale step 3 texture to 3 x 3

4 switches. But you could also perform everything in a single, larger buffer.

Only 1 switch, ow hurray! I also render all shadowMaps in a single large atlas texture by the way, but I'll give details about that another time.


After simply re-ordering the passes, to reduce the amount of FBO switches, the framerate was restored.

Don't worry. New content will come. In 2011!

Sunday, December 19, 2010

From Deferred to Inferred, part drie

The final chapter in this dramatic trilogy. We saw Deferred rendering having relation problems with his transparent cousins. Inferred Lighting suddenly made its appearance, trying to steal the hearts. But Ridge Forrester pointed out that the charming dr. Inferred has a dark side as well, and accused him of being a fake... Still, translucent Barbara has feelings for the Diffuse and Specular skills of this mysterious Inferred Renderer. How will this end?

Again, a shot with input textures for the Deferred / Inferred pipeline.


Right. As said I'm not really amused by the stippling and the detail loss that come with Inferred Rendering. But please don’t take my word on that. Judge for yourself, as it all depends on the scenery you had in mind. Yet I'm pleased with the separate pass for doing Diffuse and Specular lighting. For the 63th time, Lighting is one of the (technical) key ingredients for “good” graphics. Asides from having proper art resources of course. However, engines have so many effects these days, that it's hard to trace problems once the result isn’t quite what you expected. What you see on the screen is not just “albedo x sum(lights)”. We have:

- Specular lighting, the shiny gloss on metal, pottery, plastic, wet bricks or polished floors
- Reflections (cubeMaps, mirrors)
- Ambient light
- SSAO, DoF, Fog, noise
- Emissive textures
- And worse, HDR & Tonemapping messing around with the colors to bring it in a certain range
- And so on...

If the graphics suck, then what went wrong? Bad shaders? That's an easy answer, but in fact it's often a combination of overdone HDR, wrong reflection quantities, bad color contrast/saturation in the texture maps, or not-so-good light colors. The only problem is, it's hard to find the cause.

Obviously, with a separated Diffuse and Specular texture, it's easy to test if at least the basic lighting went properly.

See? Really useful. But maybe not a reason to change your rendering pipeline (again) though... Ok, then maybe I have a few other reasons that may convince you:

- Improved HDR Bloom (blur on bright spots)
- Diffuse blurring for special surfaces (human skin for example)
- Easy to enhance the light contrast / maximum intensity or limit the overall light result
- Can be used as input for other specific shaders

The HDR Bloom in my case is often too bright, and the overall screen is to dark because the Tone Mapper adjusted the eye on bright walls. Let me explain how it works (in my case):
1- Scene is rendered completely (including transparent crap and everything)
2- Average screen luminance is measured
3- Everything brighter than X, depending on the current eye adaption, will result
in a blur
4- Tone mapper scales the color range from the input texture (step 1) in 8-bit
screen range. Again depending on the current eye adaption level.

Say what? If you are looking outside, you can be blinded for a while since the sky and sun are much more intense than that dull light in your stinky office. Computers can fake this effect with HDR rendering. Back in the old days, a 8bit RGB color of 255;255;255 ("1.0" in shader terms), would mean "White". But what is white? Paper is white, but yet far less intense than the sun I guess. One trick to make a sun look brighter than a piece of paper, is to render a “bloom” / “blur” around the sun. But again, what exactly is bright? Depends how many lights are shining on a piece of surface, how bright these lights are, how much the material reflects, and eventually how much light the material produces itself (neon lights, TV screens, …).

The only problem with 8 bit buffers is that your value will stay clamped to: 1 + 1 = 1. ? Yes, because 8 bit can’t hold higher values than that. With High-Range buffers (16 bit or more), you won’t be limited anymore. High-Dynamic-Range lighting is just a way to make use of that advantage. You can sum many lights, or use bigger variances in the intensity values. Yet in the end the the results still have to be rendered on a 8-bit target; your monitor.

Scale the full-range colored scene into a lower-range target texture. The same happens with your eyes in reality. As you can’t see the full color spectrum, your eyes are adjusted to a certain level. In games that would be the average luminance of the scene you are seeing.


The problem with my HDR approach is that bright surfaces (such as paper or white wallpaper) quickly result in bright blurs and darkened scenes if there are a few lights shining on them. This is because that “average luminance” was simply an average of all the result-colors in the scene. So a bunch of white papers would quickly be considered as “very bright” when laying on a darker wooden table. But… did you ever get blinded by a paper? I was, but that's a whole different story.

The "Blur" / "Bloom" should only occur at highly emissive sources (lights), the sky, reflective surfaces (car metal, water), or extremely litten surfaces. Now that we have a diffuse and specular buffer as well, we can focus more on the specular (light reflected straight to your eye) quantities, instead of the color as a whole. The diffuse portion is ignored more by giving it a lower weight. That prevents blurs on white chalk walls or tax-papers. It also stabilizes the tone-mapper. When measuring the average luminance, I’ll ignore the specular light more. As specular lighting is very dependent on the eye position/direction, it can change every step you take, while diffuse light remains the same.

Abstract art by an upside down Chinese master? No, the input texture for the Bloom, using a brightpass filter.

Sure, it’s still as fake as Hulk Hogan defeating The Undertaker, but at least the overdone blooming and weird luminance peaks are reduced. I can’t show you proper final results yet, as I’m still struggling with the new pipeline. Not only the inferred approach was applied, had to fix bugs, clean up code. Also added a new method for storing shadowMaps, and a new framebuffer switching mechanisms (had a big performance drop suddenly, more about that another time). But ok, in the shot below the left wall and those pretty heads would be blurred in the previous pipeline. Now the blur is only applied at the tiled floor.

Other tricks you can do is a processing the light buffers before using them. Blur, contrast, saturate, maxOf, minOf, you name it. Do you really need that then? Uhm, maybe not. Though blurring diffuse can be interesting for special type of surfaces such as human skin. I'm not really into that yet, but techniques like sub-surface scattering / skin rendering seems to blur the diffuse light in some cases for a soft appearance. Makes sense. Take a magnifierglass and have a look at a girl. Plenty of bumps to do normalMapping, even through all that make-up. But still soft as silk, so you can't use standard normalMapping. It would make the head look like plastic or 300 year old stone. Well, I’m sure we’ll be looking into skin-rendering sooner or later.


All in all, there are probably other workarounds, but these 3 features convinced me. After all, without the stippling and DSF filtering stuff, this is only a small change in the rendering pipeline and the additional overhead isn’t that scary. And it becomes more attractive to have a try with the transparent filtering techniques from the Inferred Lighting paper… Nothing ventured, nothing gained.


Ow, I heard Santa might bring me a new videocard (after telling him my creditcard number).
Merry Christmas bastards!

Monday, December 13, 2010

I love it when a plan comes together

Where is part three of the Deferred/Inferred story? Well, last weekend I had beer-drinking duty on a little vacation with friends :p So, next week hopefully. Adjusting the rendering pipeline correctly (including some other aspects) is quite a struggle.

As for the game & next demo-movie, there are some interesting developments going on that I'd like to share. The chaos in the mailbox with people offering their help was over, but last week there were about 10 replies again. Varying from students who like to build their skills with this project, artists, web-designers, and also interesting; Pascal community members who asked if this project could become "Open Source"... With limited time and hands full on programming and keeping the three other team members busy I have to pick carefully though. Don't carry more than you can hold!


As said before, replying to all these mails is difficult for me. Not that I don't like answering mails, but I just don't want to sound like a jerk when refusing someone’s help. As the whole thing is based on charity basically… "Kijk een gegeven paard nooit in de bek"

In the ideal situation, one or two experts reply right at once for either sound, modeling, mapping or concept drawing. Two days later a team was formed… Right. In reality a mixture of qualities, styles and experience levels drop a mail once in a while, and you have to pick really carefully. Don't rush (difficult!!!), and don't put four men on the same task. If I would have let everyone in so far, we would already had about 5 character artists/modelers for example. While one or two is more than enough. Hey, we’re not making World of Warcraft here! I'm not an expert, but I guess mixing different styles and ideas is not a good idea anyway. Steering a ship with 8 captains at the same time...

Work from Jesse. Nothing to do with Tower22, but nice to show neverthless. Uhm... I still don't have new game-pictures anyway

Nevertheless, I was pleased with the offer from a concept-artist who mainly showed environment-art. Exactly one of the missing keys in our team setting so far. Hey, that whole skyscraper has to get filled right? From macro level (global overviews like the Zelda or Metroid maps) from micro (corridor atmosphere, bizarre environment ideas). Say hello to Jesse Maccabe as our Environment artist:
http://www.jessemaccabe.com/

Now that I'm thinking about it, I guess it's fun to let the modelers/artists/writers and sound composers write "how they do stuff" here on this Blog once a while. A little view in the Tower22 kitchen, asides from spicey programming with Gordon Fuck! Ramsey.

And… we’re getting help from another person on the main character of the game. Worked on a couple of games, including F.E.A.R. 3. Currently teaches at the Full Sail University. Hands up for Robert Brown:
http://www.robertakbrown.com/
Right now we are brainstorming how the player character should look like. No, I’m not telling anything yet, but that rusty robot has to be replaced obviously. Started to malfunction anyway. Well, with six people in total now, it’s time to shut the cry for help, as we are pretty much complete. For now. In the future we probably need a map & asset modeler, but… let’s first make a second demo movie ok? All in all, I can’t complain! Seriously, I really didn’t expect to get so many reactions, and certainly not from people with this kind of experience!

Robert's golden handshake

About Open Source… This game is made with ancient alien techniques so far. Delphi 7, OpenGL1.X and Windows XP Paint of course. Anyway, the Pascal(programming language) got interested because of that of course (thank you!). All those C++ boys keep telling that Delphi sucks, here, eat that sucker  But seriously, some were interested in the code. How to reply on that? Since I'm using quite a few free tools, and learned most of my skills from free tutorials/demo's/projects, it would be a little bit selfish to keep the cake for myself right? Yet I refused for now. Why?
- No time (to do it properly)
- This Blog gives some learnful information, hopefully. Using that to repay my debts :)
- I worked hard on it for many years. Sorry, but giving it all away right away...
- What if... this project would actually get a chance to become something more serious?

Basically I have no problems with Open Source. In fact, if T22 would be released tomorrow, you are free to look in the code in the next week. Kinda sympathized id Software for opening their Quake2 code (years after though) for example. But the main focus is to have fun and to create a game/movies for now. I simply don't have time for side projects, teaching students (I would like to though) or very detailed tutorials on this Blog. Sorry!

And wouldn't it be stupid to give it all away if this project might get a "commercial chance" in the future (after making more movies)? I'm not much of a materialistic guy (give me a chair and a computer, that's enough), but if making a living with doing what you like most is something everyone likes. I can't look in the future, but watching your steps is always wise.


Last but not least, we have a second movie idea. Quite a lot more complicated than the first one. Not in size, but it requires more interaction, showing some actual action-puzzling elements the game should have. And a far more bizarre creature than the Meathook guy… Since there are artists now, I'm hoping to post a couple of teasers in the next few months!

Sunday, December 5, 2010

From Deferred to Inferred, part deux

Rest in peace Frank Drebin!

Bleh, still no new pictures to post. Believe it or not, but our little girl tought it was an excellent idea to throw black ink all over the keyboard. So, my dev-machine is out of order until I have a working keyboard again, and I was too lazy to copy the whole project to my laptop. Pics or no pics, let’s continue our Deferred / Inferred rendering story ok?

Stop smiling you evil dwarf!

One week ago I explained Deferred Rendering / Lighting. To end with a couple of issues. There is a solution for everything in this world, but translucent rendering in a Deferred pipeline… With translucent stuff I mean glass, fences, Doom2 monster sprites, tree leaves, grass, and table cloths with a complex pattern, stitched by grandma.

Allow me to explain the problem. In a Deferred Rendering solution, each pixel from the (screenview) buffers contains information about one, ONE, pixel you can see. Info such as position, color, and its facing direction(normal). But what happens if you have 2 layers behind each other? Let’s say you are looking through a gelatin pudding… The gelatin has to be litten, but also the object behind it. But due the very limited storage space per pixel (16 scalars in 4 texture buffers), we can only store info for one piece of surface (pixel). And no, blending is not an option. Colors can be blended, but not positions or normals.

Dude, then simply create two sets of buffers! One of the opaque surfaces, another set for the transparent portion! Hmm, not that simple. Would work if there are exactly two layers behind each other, but in complex scenes such as your typical Crysis jungle, it could just as well be 10 layers. Think about grass billboards. So… can we throw away Deferred Rendering already?

The common solution is to render all the opaque stuff first, then to switch over to traditional “Forward Rendering” to render the remaining transparent geometry. The goods news is that the transparent part isn’t that much usually, as grass, foliage or fences are merely simple quad shapes. The bad news is that you have to implement two different types of rendering, making a messy whole. Plus you either have to sort out geometry again, and/or suffer lighting quality on the transparent parts. Multi-pass rendering on transparent surfaces can be tricky as well, as the type of blending can differ. Some use additive blending, others multiply or perform a simple alpha-test only. My engine “fixes” the problem by activating the most important lights per sector, and then render the transparent geometry in a single pass with all lights applied at once.

Inferred Rendering to the resque!? … The transparency issue was one of the motivations to make an adjusted variant called Inferred Rendering. But does it fix the problem? In my opinion; not really unfortunately. But I still have to try it out further (and I need working keyboard  ). Because it probably depends on what you are trying to do. Anyhow, it has some other interesting features though. But first, let's compare the pipelines:

For extended info about DSF and such, see the links at the bottom
The main differences are the separate lighting pass, and rendering the transparent surfaces into the “info buffers” by stippling them between the opaque pixels. That means there is actually less info available for each (resulting in a somewhat lower resolution unless you render on up-scaled buffers). The DSF edge filter technique smoothes the edges and fills the gaps again though. But yet, all forms of interpolation means quality loss in the end.

The good thing is that transparent geometry can be done in exactly the same way. No different shaders, no light sorting crap, and potentially a lot faster when having many lights + many transparent surfaces. Another small bonus for somewhat limited or older hardware is that we can possibly do with one less info buffer in the first pass, as the albedo color can be rendered later on. Don’t be fooled though, the lighting and DSF passes still require additional energy and extra buffers. Last but not least, the edge correction gives you some sort of Anti-Aliasing, which means less pixilated edges. By nature Deferred Rendering doesn’t have AA, another nasty little issue.

But it still doesn't really work when having, let's say, 10 grass billboards behind each other. As you can guess, that buffer still has a limited set of pixels. Depending on your stipple pattern, you could make 2 or 4 layers for the transparent geometry. Then sort out all transparent entities and tell which layer (stipple pattern) to look at when rendering them. YES, you need to perform Z-sorting to do this, but in case you have many transparent surfaces, you should be doing that anyway. But having 2, 4, or 6 layers for that matter, is still not much. Either you have to skip surfaces (which ones?), or accept the rendering bugs. Plus as mentioned before, you will miss small details (problematic for detail normalMapping) as pixels got offered when sharing the same buffer for multiple layers.

Why bothering then? Well…

- Unless you are rendering jungles or glass villa’s, how big is the chance you have more than 4 transparent pixels behind each other? Particles BTW can still use a simplified lighting method in a pass afterwards, if they need lighting at all.
- Having a separated light-pass got my attention.

Inferred Rendering produces one or two textures (depending if you want colors or intensity only for specular); Diffuse Light & Specular Light. The good thing is that these buffers do NOT contain dozens of other tricks such as emissive light, reflections, ambient or the surface material colors (albedo texture). That allows a couple of useful tricks, including improving HDR Bloom and debugging your lights. But boys and girls, that is for next week. Either you plan to use Inferred Rendering or not, this not-too-difficult and not-too-long paper is a comfortable read:
Inferred Lighting paper by Kircher & Lawrance
And some more details + DEMO/SOURCE by Matt Pettineo:
Dangerzone


And if you wondered why there was a pot of black ink on the computer desk? Well, I had to draw the new Dutch prime minister, Mark Rutte. As a birthday present for my little brother. Not that he is a Markie-fan in particular, but since I gave him a poster of our prime minster Balkenende 3 years ago as well... ;)