Normal Mapping (part 1)

I started a project recently in which I’m experimenting with some random stuff. I think it can be called a “handmade” project, currently I use only the C runtime library, Windows API, Direct3D12 and the AVX instruction set. (I don’t use COM objects or the d3dx12.h helper header.)

Anyway, I made a simple Perlin noise generator, and used it for creating height maps, and some images with “infinite detail” (that is, they can be zoomed into arbitrary many times). Let me call the latter ones simply just fractals in the following. I also mixed the two types, that is, I made a height map which is a fractal as well. Here is the link to the github repo:

https://github.com/SandorDeak/Experiments

I guess the code is in a very messy state, and it will definitely stay in this state for a while, since I don’t have any reason to make a proper interface and code clean-up, and anyway, it’s just the code. Below, there is a demonstrating video. Unfortunately the recording caused some frame rate issues and popping.

Currently the images are generated by the CPU, and for the fractals, they are constantly recomputed asynchronously and uploaded to the GPU when they are ready. The non-fractal, static images (height maps) and the meshes are generated at startup.

About the height mapping: after generating the height maps with Perlin noise, I computed the normal maps from them. I used the height map for offsetting the vertices in the vertex shader (yep, no nice enough tessellation yet, perhaps in the future), and used the normal map in the pixel shader to compute the proper normal. As you can see in the video, the height offsetting is relatively large, so it’s important for the normal mapping to be precise enough, that is, the tangent, bitangent vectors, and the scaling should work well. In the remaining of this post I describe briefly what normal mapping is, how it is usually done (more precisely, I describe what I’ve seen in other tutorials about how it is usually done). In the next post, I will derive the precise formula for how it should be done, at least in theory.

Preliminaries

To make the geometry of a 3D object look more detailed, than the geometry defined by the vertices, we usually use normal maps. A normal map is just a texture whose value at a point is the more detailed version of the normal vector, expressed in the basis defined by the tangent, bitangent and the less detailed normal vector. The tangent vector encodes the direction and rate in which the first texture coordinate changes along the surface. The bitangent vector is the same but for the second texture coordinate. More precisely, if we have an $r: A\rightarrow \mathbb{R}^3$ function, which is the parametrization of our surface, where $A$ is some 2-dimensional subset of $\mathbb{R}^2$ , and also, we have a $t: A\rightarrow [0,1]^2$ function, which gives the texture coordinates (uv coordinates) of the surface; then the tangent and bitangent vectors are the first and second colums of the matrix $D(r\circ t^{-1})$ , respectively. (For a function $f$ , let’s denote its Jacobi matrix by $Df$ .) Note, that $D(r\circ t^{-1}) = (Dr)(Dt^{-1})=(Dr)(Dt)^{-1}$ .

Before going further, let’s take a well-known example. Let’s say that our surface is just a triangle, with vertices $p_0, p_1, p_2$ , and with correspondig texture coordinates $t_0, t_1, t_2$ . Then both $r$ and $t$ are just parametrizations of triangles, namely $A$ is the triangle whose vertices are $(0,0)$ , $(0,1)$ and $(1,0)$ ,

$r(a,b) = (1-a-b)p_0+ap_1+bp_2$ , and

$t(a,b) = (1-a-b)t_0+at_1+bt_2$ .

Hence $Dr=(p_1-p_0, p_2-p_0)$ and $Dt=(t_1-t_0,t_2-t_0)$ , so the tangent and bitangent vectors are the columns of the matrix $(p_1-p_0, p_2-p_0) (t_1-t_0,t_2-t_0) ^{-1}$ . This expression is usually used to calculate the tangent and bitangent vectors for a triangulated mesh at the vertices $p_0,p_1,p_2$ . Of course, a vertex can be a vertex of more than one triangles (and usually is). In that case, for a vertex we can simply take the average of the tangent and bitangent vectors computed for the triangles containing that vertex. It’s important to note that the normal vector is always perpendicular to both tangent vector (after all, this is how the normal vector is defined), so we have to make the averaged tangent and bitangent vectors perpendicular to the normal vector.

It’s important to notice that there is no reason for the tangent and bitangent vectors to be orthonormal (have unit length and perpendicular to each other). But we should usually aim for that, since that would mean that the texture is mapped to the surface without any distortion. Since the mesh is usually fixed, this means that we should define the texture coordinate function $t$ in a way that the columns of the matrix $(Dr)(Dt)^{-1}$ are orthonormal. Intuitively, this just means that we have to unfold the surface, flatten it without any stretching. If that can be done, it’s great, but it’s usually not the case. Without going into details, I mention that for a smooth surface, such $t$ exists only if the surface has zero Gaussian curvature, and that’s just simply not the case for most surfaces.

Despite that, (at least what I’ve seen in every tutorial about normal mapping) it is assumed that the TBN matrix (the matrix consisting of the tangent, bitangent, and normal vectors) is orthogonal. I guess the reason for this is that it’s usually a good enough approximation, but as we seen above, it can be good enough only if the texture coordinate function $t$ is chosen well enough. In this project, this was not the case, and I got horrible results when I orthogonalized the TBN matrix.

Another problem occurs when we want to scale the height map. For example, how should we change the normal vector, if we offset the vertices by the value of the height map multiplied by two? What I’ve seen so far, people usually use an empirical intensity value for the normal map, which is used to scale the coordinates of the normal map corresponding to the tangent and bitangent vectors (the first two coordinates). This, of course, makes sense, but only if there is no actual height map being used for the normal map, otherwise the scaling of the height map and the normal map has to be synchronized properly. Actually, synchronizing them is pretty easy, but I keep the explanation for the next post.

One thing before wrapping up: notice that we can assume that the parametrization of a given surface can be selected in a way that its domain is the texture coordinates, that is, we have $r:[0,1]^2\rightarrow\mathbb{R}^3$ , and the texture coordinates of the point $r(u,v)$ is just $(u,v)$ . This can be achieved by replacing $r$ with $r\circ t^{-1}$ . This simplifies things a little bit, since then, the tangent vector is $\partial _u r = \frac{\partial r}{\partial u}$ , and the bitangent vector is $\partial _v r = \frac{\partial r}{\partial v}$ . In the next post, I will assume that this holds for $r$ .