Real-Time Rendering of Dynamic Displacement Maps Tu The Hien ...

ABSTRACT
Real-Time Rendering of Dynamic Displacement Maps
Tu The Hien ∗ and Low Kok Lim †
Department of Computer Science
School of Computing,
National University of Singapore
Displacement mapping is a well-known approach in computer graphics to add realistic visual
details to simple 3D geometric models. Until recently, displacement mapping had been
practical only for offline rendering. However, with the advent of modern inexpensive and
powerful graphics processors (GPU), it has become feasible to render displacement maps
in real time. In this project, we reviewed latest development in displacement mapping
algorithms as well as newest advancement in GPU programming. Based on the experiences
gained on re-implemented some of these novel methods, advantages and disadvantages in
terms of performance and quality of each methods are compared and presented. We also
look into some interesting possible application of displacement mapping.
1. INTRODUCTION
1.1. The Problem
In general, displacement mapping technique render the 3d surface detail given the image
surface and its height map . There’s a trade off between quality and performance of each
techniques we’re going to discuss in later chapter . Most of the technique use an offline
preprocessing step to have the distance map information before rendering the displacement
surface. The largest barrier is the preprocessing time is too long, it’s very hard to dynamic
update the texture map and render it beautifully in real time.
1.2. Background
In this section, we briefly discuss the history and background of the problem. A detail
literature survey is presented in section 2
Object representation is usually defined in 3 levels: macrostructure, mesostructure and
microstructure. Macrostructure is the geometric model of the object. Mesostructure refers
to small geometric detail but still visible to human’s eye. Microstructure is the microfacet
surface detail of the object which is indistinguishable to human’s eye. Displacement mapping
is the technique that add mesostructure detail to macrostructure of object, creating an effect
that the actual geometric position of points over the textured surface are displaced thus
enhancing the visual effects of object representation. (Szirmay-Kalos & Umenhoffer, 2008)
Displacement mapping is done by first taking a sample point on the surface and then
displace it in the normal direction with the distance obtained from the height map. The
displacement can be done by shifting the position of the vertex ( per-verctex displacement
mapping) or a points inside the surface (per-pixel displacement mapping). The new generations
of graphics card allow programmers to program on the vertex shader and pixel shader
of the GPU. In this report, we will review both implementation of vertex shader and pixel
shader approach and analyse their disadvantages and advantages.
∗Student
†Supervisor

1.2.1. Per-vertex displacement mapping
Displacement mapping can be implemented on vertex shader by modifying the vertex to the
normal direction of the mesh. For older graphic cards ( support up to shader model 3.0),
it’s impossible to change the topology of the mesh ( add new vertex), thus only the original
vertex is perturbed.
The vertex of is displaced in the normal direction by a distance h read from the height
map. The formula is discussed as in the previous section.
// get h by looking up the heightmap
h = texture2D( displacementMap, glMultiTexCoord0.xy );
// calculate the new vertex
newVertexPos = vec4(glNormal * h * scale, 0.0) + glVertex;
// return the new vertex position
glPosition = glModelViewProjectionMatrix * newVertexPos;
The advantages of per-vertex displacement mapping is that it actually changes the geometry
thus can automatically resolve silhouette. However, it has some serious drawbacks.
• Vertex shader has less processing power than pixel shader and has limited access to
texture.
• Vertex shader execute each vertex once whether it’s visible or not thus wasting GPU
resources.
1.2.2. Per-pixel displacement mapping
The vertex shader transforms only the macrostructure geometry, then the height map is used
to adjust the texture color. Since we do not change the geometry, the visibility problem
needs to be solved in the fragment shader program by a raytracing algorithm. Started at the
top of the height field, we casting rays into the height field to obtain the texture coordinates
of the visible point.
The vertex shader processes the macrostructure geometry, and pass to the fragment
shader input with texture coordinates [u, v]. This processed point has (u, v,0) coordinates
in tangent space. The fragment shader program find that point of the height field which is
really seen by the ray connecting the pixel center and processed point (u, v,0). The direction
of this ray is defined by tangent space view vector V ⃗ . The visible point is on the height field
and thus has tangent space coordinates (u 0 , v 0 , h(u 0 , v 0 )) for some unknown (u 0 , v 0 ). This
visible point is also on the ray of equation
(u 0 , v 0 , h 0 ) = (u, v, 0) + V ⃗ (t)
for some ray parameter t, thus we need to solve the following equation: (u 0 , v 0 , h(u 0 , v 0 )) =
(u, v, 0) + V ⃗ (t) for unknown u 0 , v 0 , t parameters.
Per-pixel displacement mapping approaches can be further categorized to safe and unsafe
method depends on whether it finds the correct first intersection with the height field.
Since unsafe methods are much faster than safe methods it makes sense to combine
the two approaches. In combined methods first an unsafe iterative method aggressively
leap through the empty space , then a safe method computes the accurate intersection .
(Szirmay-Kalos & Umenhoffer, 2008)

2. RELATED WORK
In this section, we present some previous research results that is related to our work.
2.1. Simulation of wrinkled surfaces
Introduced by Blinn in 1978, this paper placed the very first stone for all subsequent displacement
mapping techniques later on. It’s commonly known as bump mapping and can
be seen as a simplified version of displacement mapping. Although computer generated
images has get some degrees of realism, they are not so realistic. The surfaces still looks
artificial because of its unrealistic smoothness. The paper presents a new method of using
a texturing function to perform perturbation in the direction of the surface normal before
using it in the intensity calculations. (Blinn, 1978)
The key idea is by changing the normal of the surfaces, usually by looking up in the
normal texture, it tricks the viewer that the surfaces geometry itself actually changes. The
result is much better than normal texture mapping. However, the technique is unable to
simulate self-occlusion, self-shadowing and viewing parallax. Although it has many flaws
and cannot compete with other realistic displacement map later, this paper had an enormous
influence to all subsequent research in displacement mapping, how to add realistic visual
detail to the surface without actually adding any geometry to the surface
2.2. Detailed sharp representation with parallax mapping
More than 20 years after the first paper of Blinn , a group of graphics researcher in Tokyo,
Japan invented a technique which greatly improve the visual effects of bump mapping, called
parallax mapping. The method allows adding more apparent depth as well as enhances the
visual appearance for bump mapping. It takes into account not only the normal of the
texture for shading but also displaces the texture coordinate based on the viewing angle
and the height of the current texel. Given the original texture coordinate (u,v), the new
texture coordinate is replaced by (u’,v’) which calculated from tangent space view vector
and height value h(u,v) read from normal texture. We simplify the ray equation:
And the solution:
(u ′ , v ′ , h(u, v)) = (u, v, 0) + −→ V (t)
(u ′ , v ′ ) = (u, v) + h(u, v)( V x
V z
, V y
V z
)
(Tomomichi Kaneko & Tachi, 2001)
The greater between the view angle and the displaced texture mapped surfaces, the
more distortion occur. It creates the illusion of depth due to parallax effect when the view
changes, thus create more realistic bump with almost the same speed as bump mapping.
Although parallax mapping is a great improvement from bump mapping, it still inherits
the same drawbacks from bump mapping such as unable to simulate self-occlusion, self
shadowing. Especially, when the viewing angle is more grazing (Vz approaches 0), the offset
value becomes infinity and creates many random artifacts. Subsequent algorithm has been
made to improve the algorithm such as parallax with offset limiting, iterative parallax, steep
parallax, relief parallax. (Szirmay-Kalos & Umenhoffer, 2008)

2.3. Relief Texture Mapping
Relief texture mapping is an alternative approach of parallax mapping with much higher
accuracy. The key idea is simple, a relief texture is an extended texture with orthogonal
displacement texel. When the viewing direction changes, based on the viewing direction and
height map value the texture is warped using a two simple 1D transform before mapped onto
the surface to create the viewing parallax. The warping function can be simple described
as :
What coordinates ( ut, vt) should the source pixels (us , vs ) have so that a
view of such a flat distorted image on the source image plane from the target
COP would be identical to a 3-D image warp of the source image onto the target
image plane? (Oliveira, 2000)
2.4. Per-pixel displacement mapping with distance function
Published by William Donnelly in the famous graphics series of Nvidia, GPU Gems 2 , the
paper presented a new approach for adding geometry detail to surfaces yet still remaining its
performance in real time. The key idea is he treats displacement mapping as a ray tracing
problem. In conventional displacement mapping, given a piece of geometry, we find which
pixel in the image it maps to. In this algorithm, the problem is inversed, given a pixel in the
image; we find its corresponding geometry. Imagine the surface as an axis aligned box, we
start at the top of the surface and calculate which coordinates the viewing ray intersects with
the displaced surface. We define the distance map of the surface, which give the shortest
distance from any points in the surface texture space to the surface and store it into a 3D
texture. Then the intersection point is advanced in the viewing direction by a distance d
by looking up in the distance map. Each step brings the point closer and finally converges
to the real intersection point given enough advanced steps. The introduced algorithm has
many advantages over other previous work:
• Taking advantages of the processing power of pixel shader over vertex shader of new
generation graphics cards. - Can resolve fine detailed surfaces without aliasing or
leaving any gaps between the geometry but still keep the same number of tracing
steps.
• Can resolve self occlusion and self-shadowing easily. Through experiments, the algorithm
performs very well on detail surfaces and still keep its performance rendering in
real time. However, it still has some issues:
• Although the algorithm guarantees to find the exact intersection point, the number
of tracing steps in pixel shader is usually fixed for performance optimization. Given a
high frequency surfaces, the algorithm may need to calculate more depths in the 3D
texture map and take more tracing steps to converge, thus decreasing the performance
heavily.
• Calculating the distance map takes a lot of time and thus not desirable for dynamic
real time rendering. Storing 3D textures is memory consuming, especially when the
depth is increased.
(William, 2005)

2.5. Cone step mapping: An iterative ray-height field intersection algorithm
Inspired by sphere tracing displacement mapping algorithm, the paper introduced a new
approach to ray-height field intersection problem with many improvements compare with
sphere tracing approach. It also uses preprocessed data to advance the viewing ray to find
the real intersection with the surface. For a given texel, we calculate its conservative cone
bounding volume. The tip of the cone is on the surface and the side of the cone touches the
surface (Jonathan, 2006). The key idea is similar the ”sphere leaping”, in this case, the point
is advanced to the intersection of the viewing ray and the cone. Cone step mapping can
be seen as an improvement version of ”sphere step” mapping as in per-pixel displacement
mapping with distance mapping. It inherits it advantages as well as drawbacks.
• Can resolve highly detail surfaces and self occlusion very well.
• Can be rendered in real time with pre-processed data. The pre-processed data is stored
in a 2D relief map textures, hence requires less memory than ”sphere step” mapping
but takes longer time to compute.
• Suffered from high frequency detailed surfaces. Using the fixed number of cone steps
leads to inaccuracy, but calculating the exact intersection point affects the rendering
time.
2.6. Relaxed cone step mapping for relief mapping
This paper is a continue work of Cone Step Mapping, introduced in the newest series of
Nvidia GPU Gems . Using conservative cone approach, the ray tends to stop before the
actual intersection which may introduces to different kinds of artifact. The new algorithm
combines both strength of cone step mapping and binary search using a ”relaxed cone”.
Unlike the conservative cone, the relaxed cone allows the ray to pierce the surface at most
once.(Policarbo & Oliveira, 2007)
That produces much wider cone and faster convergence. The key idea is first using an
aggressive space leaping approach with relaxed cone. Once the cone is inside the surface, we
can safely apply the binary search to find the actual intersection point. The performance is
increased noticeably, using the same number of cone steps, it’s usually provide better result
than other algorithm using similar approach such as cone step mapping, parallax mapping.
However, the preprocessing texture data takes much longer time than other algorithms.
Although the relaxed cone step provide a very realistic image, preprocessing step make it
almost impossible to dynamically update relief textures and render it in real time.
2.7. Accurate per-pixel displacement mapping using a pyramid structure
This paper presents a new per-pixel displacement mapping method called pyramidal displacement
mapping that provides high accuracy as well as real-time performance. The idea
arises from the observation that we can safely skip empty regions by moving a ray to the
maximum height value. Similar to similar cone step mapping and sphere tracing, it also
uses the preprocess texture to advance the viewing ray and find the intersection with the
displaced surfaces. In this algorithm, the author uses mipmap pyramid structure to store
the bounding height field information and the ray along the view direction advances hierarchically
via this pyramid structure. A pyramidal displacement map is a quadtree image
pyramid, each leaf at the lowest level of mipmap contains the height difference between the

maximum height surface and the current height at each texel (Oh, Ki, & Lee, 2006). The
highest level of the mipmap is the global minimum difference between the height of the
surface and a given texel.
In the experiments, the algorithm performs very fast and accurate. There’s no comparison
between relaxed cone mapping and pyramidal displacement mapping but the quality
of the image and rendering time are more or less the same. The new method also performs
very well in very spatially-varying height field textures which relief mapping and parallax
occlusion mapping are unable to render correctly. However, the biggest achievement in this
method is the preprocessing step. Other per-pixel displacement mapping, although can
achieve a very good frame rate in rendering time, relies a lot on its preprocessing step which
usually takes a lot of time to compute makes them very hard for dynamically displacement
mapping rendering. It takes relaxed cone step mapping O(n 4 ) time, sphere tracing
O(depth ∗ n 2 ) to preprocess data. Pyramidal mapping only requires a simple 3 for loops
which is levelxNxN in total to compute preprocess the pyramidal map and the algorithm is
possible to implement in both cpu and gpu. In the experiment, this step spends approximately
1.8ms in CPU for a 256x256 texture. Another advantage of pyramidal displacement
mapping is it doesn’t need a 3D texture to store the preprocessing data but can instead use
a NxNx2 texture. Although the texture size is doubled compared to other methods, many
other advantages is enough to compensate for this. The new technique makes displacement
mapping now is possible to render dynamically in real time.(Oh et al., 2006)
3. IMPLEMENTATION
3.1. Per-pixel displacement mapping with distance map
The algorithm is implemented with Visual C++ 2008 and GLSL. I followed the algorithm
described in chapter 8 of GPU Gems 3.
3.1.1. Preprocessing 3D texture map
Computing a distance transform is a well-studied problem. In our implementation, we used
Danielsson’s (Danielsson, 1980) algorithm that runs in O(k.n) time with n is the number of
pixels in the map and k is the depth level. The idea is to create a 3D map where each texel
stores a 3D displacement vector to the nearest point on the surface. The algorithm then
performs a small number of sequential sweeps over the 3D domain, updating each pixels
displacement vector based on neighboring displacement vectors. Once the displacements
have been calculated, the distance for each pixel is computed as the magnitude of the
displacement.
When the distance transform algorithm is finished, we have computed the distance from
each pixel in the distance map to the closest point on the surface, measured in pixels. To
make these distances lie in the range [0, 1], we divide each distance by the depth of the 3D
texture in pixels.
3.1.2. Vertex shader
In vertex shader, we just simply transform light vector and eye vector to the tangent space.
The binormal vector and tangent vector is calculated before hand before passing to vertex
shader. The calculation of tangent space matrix has been discussed in the previous section.

3.1.3. Fragment shader
In fragment shader, we started from the top of the height map. Using the distance from the
distance map we recursively traced in a fixed number of steps until we reach the surface.
3.2. Relaxed cone step mapping
3.2.1. Preprocess relief texture map
Using an O(n2) to calculate the relaxed cone maps offline. The idea is, for each source texel
ti, trace a ray through each destination texel tj, such that this ray starts at (ti.texCoord.s,
ti.texCoord.t, 0.0) and points to (tj.texCoord.s, tj.texCoord.t, tj.depth). For each ray,
compute the intersection with the height field and use this intersection point to compute
the cone ratio cone r atio(i, j). The coneratio = w/h is illustrated in figure 3.1 . After
process all the source texels, we obtain the relaxed cone map.
3.2.2. Vertex shader
Similar to per-pixel displacement mapping with distance map. We just simply transform
view vector and light vector to tangent space.
3.2.3. Fragment shader
The key idea is first using an aggressive space leaping approach with relaxed cone. Once
the cone is inside the surface, we can safely apply the binary search to find the actual
intersection point.
m = d × rayRatio
We also have,
m = g × coneRatio = ((currentT exelDepth − rayCurrentDepth) − d) × coneRatio
Solve the 2 equations, we have:
d =
Finally, the intersection point is:
4. CONCLUSION
(currentT exelDepth − rayCurrentDepth)coneRatio
rayRatio + coneRatio
I = rayCurrentP osition + d × rayDirection
Given the limited time for the a 1-semester UROP project. We don’t have enough time to
both understand and implement all of the displacement techniques. What more important
is, through out this project, I’ve learned and fully understood the problems and various
approaches to solve it. In addition to this, given having zero GPU programming background
when I first started the project, after implemented 2 state-of-the-art algorithms in
the famous series GPU Gems books, I’ve learned and be able to use either GLSL or CG
programming languages as well as shader developing software such as ATI’s Rendermonkey,
Nvidia’s FXComposer. Since GPU’s programming is a very low-level programming it’s
unable to print out and test the output as we normally do with high level programming
language like Java or C++, it’s also important that I experienced and learned to use shader
debugger and shader optimizer tool such as GDebugger, GLSL Devil Shader debugger and
GPU’s Shader Analyzer.

4.1. Future work
Realtime displacement mapping is a very interesting area which attracted many graphics
researchers all over the world. There’re always new techniques or improvement over the
existing algorithm published every year. Together with the recently achievements in graphics
hardware, it’s become more and more feasible to render displacement mapping in realtime.
Although this area is becoming crowded, there’s still rooms for improvement since nobody
has been able to achieve an algorithm that can render displacement mapping fast and
accuracy.
Displacement mapping also open many possible interesting applications in image based
rendering such as terrain rendering. For example, we could take a terrain image and terrain
height in Google earth then using displacement mapping to render a 3D view of the
earth. May be, in the future, given the graphics hardware is strong enough, and an efficient
algorithm is achieved, we could have a totally 3D view of the earth terrain.
5. REFERENCES
[1] Blinn, J. F. (1978). Simulation of wrinkled surfaces. 12 (3), August, 1978, 286–292.
[2] Danielsson, P.-E. (1980). Euclidean distance mapping. Computer Graphics and Image
Processing, 14 , 1980, 227–248.
[3] Jonathan, D. (2006). Cone step mapping: An iterative ray-heightfield intersection.
algorithm.
[4] Oh, K., Ki, H., & Lee, C.-H. (2006). Pyramidal displacement mapping: a gpu based
artifacts-free ray tracing through an image pyramid. VRST ’06: Proceedings of the
ACM symposium on Virtual reality software and technology (pp. 75–82), 2006.
[5] Oliveira, M. M. (2000). Relief texture mapping. PhD thesis, University of North Carolina.
[6] Policarbo, F., & Oliveira, M. M. (2007). Gpu gems, Vol. 3, Chap. 18, pp. 409–428.
Addison Wesley.
[7] Szirmay-Kalos, L., & Umenhoffer, T. (2008). Displacement mapping on the GPU -
State of the Art. Computer Graphics Forum, 27 (1), 2008.
[8] Tomomichi Kaneko, Toshiyuki Takahei, M. I. N. K. Y. Y. T. M., & Tachi, S. (2001).
Detailed shape representation with parallax mapping. Proceedings of ICAT (pp. 205–
208), 2001.
[9] William, D. (2005). Gpu gems, Vol. 2, Chap. 8, pp. 123–136. Addison Wesley.