From digital image series to 3D models - Arctron

Martin Schaich
From digital image series to 3D models
Possibilities for software application
From digital image series to 3D models
Three-dimensional computer models have a lot of advantages for use in as-built
documentations and damage mapping. This software generates precise 3D models
based on a series of digital images and offers an alternative solution to expensive 3D
scanning.
3D models in a restorational context
3D models present us with new opportunities that can help us in the complex everyday work
of restoration and cultural heritage preservation. On one hand, they assist in digital
documentation. Findings and damage areas can be reliably mapped. This is especially
interesting for complex 3D objects. On the other hand, this technology is capable of virtual
reproduction of works of art. The dissemination of these models using internet and
multimedia solutions (including publication and presentation, animations and elaborate
computer films) offers exciting and interesting possibilities.
There are different ways to create a 3D model. The most common one is the 3D scan, during
which the object is scanned without contact. This can be done using laser scanners,
triangulation scanners or structured-light scanners (for high resolution). If necessary, the
scanning data can be combined with total station or GPS data for referencing.
In practice, a digital 3D scan (requiring special equipment and knowledge) is way too
laborious and expensive for restorers. To these professionals, the software aSPECT 3D offers
an alternative. This software generates 3D models from digital image sequences, which can
be recorded using commercially available digital cameras.
Photographic recording and image quality
At the beginning, a suitable image series of the object has to be recorded. For that, basic
photographic knowledge about the correlation between exposure time and aperture, depth of
field and the use of different lenses is – of course – required. Distortions, for example
produced by using so-called “fish-eye”-lenses and extreme wide angle lenses, cannot be
processed by the algorithm.
In general, no calibrated cameras are needed. Even small, low-cost compact cameras or
system cameras deliver usable results. Nonetheless, for high quality survey results, semiprofessional
or professional SLR systems are recommended.
The software requires well-focussed images with a sufficient depth of field. Moving shadows
or light points affect the computation process as well as reflections (e.g. from direct flash
light). While homogenous lighting is easy to realize in a studio setting, it is not always
possible during outdoor projects. However, experience has shown that usable results can
even be derived from data with difficult content like hard shadows. During the photographic
recording, it is important to circle the object as “hemispherically” as possible with different

From digital image series to 3D models
vertical angles (one per circle) (Fig. 1). A single panorama recording, during which the
photographer stays in one position, does not produce 3D results.
Fig. 1: aSPECT 3D restoration project “Old Fritz”. Left: Pictures taken with a commercially available camera. Result after the
bundling with the cameras positioned and oriented in space. The coloured prisms around the images visualize the neighbouring
qualities of the images and the lenses’ focal lengths. Right: First “basic 3D result” as a dense, coloured point cloud. © ArcTron 3D
GmbH, 2012.
Preselection of images
Before the software aSPECT 3D can start computing the model, the images have to be
checked and sorted with relation to the criteria described above. Large projects often
produce images in various stages with very different perspectives and lighting conditions. In
these cases, it is useful to sort the images: e.g. all aerial images of the building, all pictures
from the ground, by time of day or room location etc. A similar approach applies to
archaeological or restoration projects which document stratigraphic layers. For that, the
software offers an image database, in which the images can be tagged. Furthermore, all
internal camera information (EXIF-data) concerning sensor, lenses, focal length, shutter
speed, ISO-value etc. are stored and later on used for the computation.
From digital image series to 3D point cloud
After checking and preselecting the images, the software computes a 3D point cloud from the
selected pictures. It utilizes a technology, which has established itself in computer based
image processing: SFM (“Structure From Motion” 1 ).
Different algorithms (e.g. Sift, ASift, Hesian-affine, MSER etc. 2 ) are used for computing 3D
object information. They identify distinctive points (“features”) in several pictures from
1 Different providers allow working with such solution in the „cloud“. For that, the images have to be uploaded
to the respective providers, which is hard to realize and not really sensible for large image series, sensitive data
or when operating in remote regions. Generally compare: Remondino, F., El-Hakim, S.F., Gruen, A. & Zhang, L.
(2008): Turningimagesinto 3D-models. IEEE Signal Processing Magazine 25 (4), S. 55-65; Kersten, T.,
Lindstaedt, M., Mechelke, K., & Zobel, K. (2012): Automatische 3D-Objektrekonstruktion aus unstrukturierten
digitalen Bilddaten für Anwendungen in Architektur, Denkmalpflege und Archäologie. 32. Wiss.-techn.
Jahrestagung der DGPF (März 2012), Potsdam, Germany.

From digital image series to 3D models
different perspectives. The algorithm also includes the EXIF-data of each picture, e.g. which
camera was used with what kind of sensor and lenses. Photogrammetric procedures are
used in order to reconstruct the camera position and orientation. All is done by correlating the
images to each other.
In the first step, the images are put into spatial correlation using the so-called bundle block
adjustment 3 . The result is a 3D point cloud of all identified features. In a subsequent
computational step, this basic data can be transformed into an already very detailed, socalled
“dense”, coloured point cloud 4 . (Fig. 1)
SFM-technology is suitable for objects with amorphous geometries and / or objects with
structured / multi edged surfaces. Artifacts, which display an abundance of corresponding
image points and shades lend themselves to a successful result. Among the objects
rather unsuited for SFM are unstructured, monochrome, translucent, reflective, evenly
patterned or so-called self-similar surfaces, in which structures and characteristics are
repeated. Such objects produce rather bad or incomplete results.
Fully automated processes
The computational process for generating 3D point clouds from images is fully automated.
That means that the user does not have to carry out any more steps after selecting the
images. Processing time is dependant on the number and size of the images. Data banks
containing several hundred or thousand pictures, require high quality and very powerful
computers. Most importantly, these computers need a multiple processor architecture with
64bit processors, a lot of main memory and they need to support parallel computational
processes on the graphic board (GPU). Usually, current “workstations” meet these
requirements.
Scaling and georeferencing
Initially, the resulting point cloud is neither scaled nor georeferenced. This means, that the
ratio of the point cloud to reality remains unknown. The point cloud needs to be allocated to
definite coordinates in a superordinate coordinate system. Afterwards, the 3D object is
scaled, meaning the size of the object is within a defined, tolerable deviation.
aSPECT 3D offers different approaches for this procedure. The simplest way is scaling the
object via a scale that was photographed along with the object. The GPS and total station
2 Lowe, D. (2004), Distinctive image features from scale-invariant keypoints. International Journal of Computer
Vision 60,2, S. 91-110.
3 th
Wu, C., Agarwal, S., Curless, B. & Seitz, S. (2011): Multicore Bundle Adjustment. 24 IEEE Conference on
Computer Vision and Pattern Recognition. Colorado Springs, USA.
4
Furukawa, Y. & Ponce, J. (2007): Accurate, dense and robust multiview stereopsis. In: Proc. International
Conference on Computer Vision and Pattern Recognition. S. 1-8.

From digital image series to 3D models
interface allows correlating reference points within the point cloud to independently
measured control points. By deploying a so-called multi-point transformation, the point clouds
are transformed into the target coordinate system, whereupon the errors are checked and
can be optimized in general (Fig. 2).
Fig. 2: Scaling and georeferencing in aSPECT 3D . Left: Photographs of a house façade as a SFM image series. Middle: Single
reference points are recorded with a total station and processed in aSPECT 3D . Right:: Using a multi-point transformation, the
scaling and georeferencing is carried out with the computation of the errors. © ArcTron 3D GmbH, 2012.
Larger objects (e.g. roof landscapes, façades) sometimes consist of several point clouds
merged together, because they were recorded from very different points of view – e.g. via
airborne photography or from the ground. For this purpose, aSPECT 3D includes an ICPalgorithm,
which can fuse overlapping point clouds together. This is especially suited for
image series with very little overlap. Once this step is done, the 3D point model is finished.
How precise are 3D models?
3D models have to depict the original object as precisely as possible for restoration,
preservative or archaeological purposes. Size and shape need to have the smallest possible
deviation. Depending on the deployed survey procedure (total station, laser scanner etc.),
the accuracies can vary.
The creators of the software, which generates models based on SFM-images, asked
themselves if aSPECT 3D could compete with other 3D methods. In order to answer this
question, ArcTron 3D carried out accuracy analyses during different projects 5 . The results
(below) show, that for both small and large objects, significantly precise 3D models can be
generated.
5
Publications of the author concerning the topic are currently in print or preparation. Compare also: Kersten, T.
&Mechelke, K., Fort Al Zubarah in Katar – 3D-Modell aus Scanner- und Bilddaten im Vergleich. In:
Luhmann/Müller (Ed.), Photogrammetrie, Laserscanning, Optische 3D-Messtechnik. Beitr. Oldenburger 3D-
Tage 2012, S. 89-98.

From digital image series to 3D models
Fig. 3: Project “wall documentation”. Veitsberg, Bad Neustadt a.d. Saale (in cooperation with the University of Jena,
Professorship for Prehistoric and Protohistoric Archaeology). Left: Photographic documentation airborne and from the ground.
Middle: Reference surveys with total station and laser scanner. Right: Deviation result after a “best fit” registration of the SFM
point cloud onto the terrestrial laser scan with standard deviation of below 1 cm. © ArcTron 3D GmbH, 2012.
Example 1: In cooperation with the University of Jena, ArcTron 3D surveyed a small
representative area of wall fragments in Veitsberg showing early medieval Carolingian
remains (Fig. 3). For the accuracy analysis, the model generated by aSPECT 3D was
compared to models generated in a different way. In one test various control points placed in
the area were recorded via total station. In a second test the reference area was scanned
with a terrestrial 3D laser scanner which produces accuracies of about 3 mm.
The aSPECT 3D model was computed from airborne pictures taken by a so-called ‘Octocopter’
drone and pictures taken from the ground. The comparison of the reference data from SFM
point cloud and laser scan showed a very small standard deviation of about 1 cm. If only a
total station is employed, the standard deviation still remains smaller than 3 cm. When a
reference difference (e.g. a scale photographed with the object) is the only device for scaling
the point cloud, excavation findings of the respective size provide accuracies of around 5 cm.
Fig. 4: Project “Enigma 3D” (in cooperation with the “celtic + roman museum” Manching). Left: Angular “bread loaf idol” in
photorealistic and shaded depiction. Right: The highly precise structured light scanner PT-M 1280 serves as reference
measuring device. For the automated, photographic SFM-documentation, two Nikon cameras with macro-lenses and a turn
table were set up in a homogenous white light room. The deviation analysis proved accuracies better than 0.2 mm. © ArcTron 3D
GmbH, 2012.
Example 2: Another accuracy test showed that the software is also suited for small objects. It
was carried out in cooperation with the “celtic + roman museum” Manching 6 and focussed on
6 M. Schaich, „Aenigma 3D“. Zur Generierung hochaufgelöster 3D-Modelle mit Fotos einer handelsüblichen
Spiegelreflexkamera – eine Fallstudie mit ArcTron’s Software aSPECT 3D am Beispiel der „Brotlaibidole“. In W.
David (Ed.), Aenigma. Katalog zur Ausstellung (2012) – in preparation.

From digital image series to 3D models
more than a hundred of so-called “bread loaf idols” and other small archaeological objects.
Professional camera systems with macro lenses produced accuracies of 0.2 mm for the
resulting SFM-point cloud. A high-resolution structured light scan with a resolution of 0.05
mm served as a reference.
Processing photorealistic 3D models
Using one additional step, point clouds can be processed into polygonally meshed 3D
models. First of all, the coloured point clouds should be filtered and rid of interfering objects
(e.g. scaffold elements). aSPECT 3D offers different kinds of tools for reducing noise in 3D
data or deleting dispersive points. Afterwards, a closed 3D mesh is laid over the point cloud
using a so-called “Poisson triangulation” – a probability distribution. This 3D mesh consists of
numerous polygons, which connect at least three different points each. Simultaneously, the
software generates a so-called “vertex texture”. For this, a colour gradient is computed from
the respective RGB colours of each triangle. That is why the coloured surface of an object
with a very dense point cloud is already realistically depicted.
However, it takes another texturing for generating the photorealistic 3D model. During this
step, the optimized photographs of the SFM-process are mapped onto the geometries of the
model surface. An Apsis-fresco may serve as an illustrating example (Fig. 5).
Fig. 5: Project “Ninfa 3D” (in cooperation with the German Historical Institute, Rome). Left: SFM-documentation of a fresco.
Middle: 3D model with vertex texture. Right: Texture with high-resolution photographs on the 3D model. © ArcTron 3D GmbH,
2012.
Combined airborne and terrestrial surveys
During the past few years at ArcTron 3D , a combination of airborne and terrestrial
documentation technologies has proven very suitable for comprehensive 3D surveys. In
targeted survey flights anultra-light paraglider trike is deployed by the company.
Up to 90 % of all required information can be gathered by aerial surveys. That is why since
2010 ArcTron has also been using a remote controlled “camera drone” (Fig. 6). This aircraft
robot can be completely autonomously controlled via a ground station and is especially
programmed for taking software-specific, photogrammetrical survey images. The system is
able to carry cameras of over 2 kg and is equipped with a stabilized and separately
controllable camera platform. This technology is suited for replacing bulky lifting platforms
and can even reach remote objects very precisely (e.g. the gothic embellishment of dome
spires).

From digital image series to 3D models
Fig. 6: Project “Ninfa 3D” (in cooperation with the German Historical Institute, Rome). Left: Octocopter, camera drone with a fully
stabilized camera platform. Right: Technology in practice. Deployment of the system for the 3D documentation of medieval
remains of the city Ninfa (Italy). © ArcTron 3D GmbH, 2012.
Data management and exploitation
The finished photorealistic model can now be further edited and processed using the
software’s inherent database.
• Data management
All object geometries can be individually ordered and described in the database. Different
drafts of database structures for archaeological, architectural or restoration purposes are
available. They can be adjusted to any complex project. The database provides a
hierarchical object system (“part of relation system”) as well as an additional classification
tree structure. If necessary, a completely new global structure can be created.
• Further processing
The software allows to allocate additional information – e.g. for damage mapping – to the 3D
model. Damage mapping can be applied directly onto photorealistic models (Fig. 7). Mapping
can also be carried out on location just beside the original using a notebook or a tablet PC
with the respective 3D software. This capability facilitates precise and reliable documentation
in the field (essential to complex 3D projects). Furthermore, the 3D models can be linked with
additional meta data such as texts, CAD-files, images, films, PDFs etc. Using the included
tools complex documentation is a logical, step by step process.
Fig. 7: Damage mapping of the Bronze Statue of general Yorck of Wartenberg (1759-1830), Berlin. The complex 3D mapping of
all damage areas with the integrated database was carried out by the restoration office Helmich (Berlin) with the 3D GIS system
aSPECT 3D . It enables database enquiries of deliberate 3D damage mappings and restoration planning. Graphics: © Landmark
Preservation Authority Berlin & restoration office Helmich, based on a laser scan by the company Laserscan, Berlin. Software
Screenshots: ArcTron 3D GmbH, 2009.
• Print versions
Printed documentations on paper, CAD-maps and high-resolution, scaled and rectified
screenshots of many mosaic images (orthophotos) can be derived from different
perspectives.

From digital image series to 3D models
• Presentations
The 3D models are able to rotate on the screen for presentational purposes. Interior rooms
can be entered and walked through in full screen mode – similar to video games. 3D stereo
technologies are also supported if a respective 3D monitor is available. This allows viewing
the objects in “real 3D” using 3D glasses.
Future prospects
The company ArcTron 3D has been developing the presented software continually for the last
10 years. By now, it provides tools and applications for restorers, heritage preservers and
archaeologists. aSPECT 3D continues on a developmental path. Currently, ArcTron 3D is
working on a process manager, which will lead the user systematically through the complete
workflow process, from point cloud to textured 3D model. This brand new development will
be presented on the trade fair “denkmal” in November.