Normalmaps for the Technical Game Modeler
By Ariel Chai, 15th of June 2010, all rights reserved.
*Article is constant work in progress*

This article is aimed towards technical modeling (such as weapons vehicles, pros, etc), and is made to give an overview of common normalmap issues, walkarounds and solutions.
This article assumes knowledge of UV's, and the core idea of what normalmaps are.

Theory : Implementation :
Intro : Smoothed and Unsmoothed Tangents

Normalmaps are baked differently depending on object's smoothing.
The renderer goal is to compensate for the smoothing, so the lowpoly with the applied result will be as close as possible to the highpoly version.

Each method has its issues and therefore we want to mix them to produce good maps.
Smoothed Tangents - Smoothing Errors Problem and Walkaround

Smoothing on faces with hard angles (such as cubes, and sharp slopes), leads to bad shading - commonly referred as smoothing errors.
This varies between baking engines, with more advanced ones producing better results.

The theoretical walkaround to this problem, is to add more geometry to the edges - which makes them smoother, and lead to more accurate smoothing results by the realtime renderer.

Although this walkaround is useful for objects with a few problematic areas - on objects with a lot of sharp edges the polycount overhead is tremendous and overcomplicates things.
It's not an issue of speed (as breaking objects to smoothgroups technically introduces new vertices as well) - but rather more polys to work with, unwrap, etc ...
Unsmoothed Tangents - Edges Errors Problem and Walkaround

Unsmoothed Tangents don't use smoothing, and therefore wouldn't have the above smoothing errors.
However let's render the simple box with Unsmoothed Tangents and see the problem that occurs :

The problems occurs because each face normalmap points to a different direction as illustrated below.
Even if the uv edge is exactly between the pixels, the way those pixels would be pulled by the realtime renderer will mix them up resulting in a sharp normal transition which stands out from the rest.

So, where there are seams, this problem won't occur - therefore using unique uv's would completly eliminate the problem.

As you can see, this UV both sacrifices precious texture real-estate, and also impractical to texture first hand.
Obtaining Clean Results Using Hybrid maps

Baking - hybrid smooth/unsmoothed tangents using vertex normals (xNormal, maya) / smoothing groups. (max)
  1. UV borders should generally be converted to hard edges, this dramatically reduces smoothing errors. (with no seam problems)
  2. To reduce smoothing errors in an area, you can either split the uv map and introduce more hard edges - or manually round the geometry.
  3. Experiment with your baking package to see how far you can push it - xNormal bakes for example have less smoothing errors than 3ds max bakes.

Exporting final model - cross-engine compatibility using split edges
Not all engines support vertex normals or smoothing groups.
After baking, the safest way to export the asset ingame is to seperate the hardedges manually in the geometry and apply uniform smoothing to the entire model.
This way you will get the closest shading to your bake model, and the applied normalmap will be at its highest quality regardless of engine.
Overlapping UV's - Cloning

Will explain how to take advantage of ovelapping uv's, and mirroring capabilities with normalmaps.

This documented is free to be distributed as long as it remains intact and unedited.