Accurate, Dense, and Robust Multi-View Stereopsis (PMVS) [1,2,3]

Accurate, Dense, and Robust 

Multi-View Stereopsis 

(PMVS) [1,2,3] 

[1] Yasutaka Furukawa and Jean Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis”, in CVPR, 2007 

[2] Yasutaka Furukawa , “HIGH-FIDELITY IMAGE-BASED MODELING”, PhD thesis, 2008 

[3] Yasutaka Furukawa and Jean Ponce, “Accurate, Dense, and Robust Multi-View Stereopsis”, in PAMI, 2009 

1

Abstract 

This article proposes a novel algorithm for multi-view 

stereopsis that outputs a dense set of small rectangular 

patches covering the surfaces visible in the images. 

Stereopsis is implemented as a match, expand, and 

filter procedure, starting from a sparse set of matched 

key-points, and repeatedly expanding these before using 

visibility constraints to filter away false matches. 

2

Abstract 

Simple but effective methods are also proposed to turn 

the resulting patch model into a mesh which can be further 

refined by an algorithm that enforces both photometric 

consistency and regularization constraints. 

A quantitative evaluation on the Middlebury benchmark 

shows that the proposed method outperforms all others 

submitted so far for four out of six datasets. 

3

1. Feature Detection & Matching 

2. Expansion 

Key Elements Of PMVS 

Detection: DOG & Harris 

Matching : Epipolar Constrain 

Identifying Cells for Expansion 

Expansion Procedure 

3. Filtering : Enforces visibility consistency 

4. Polygonal surface reconstruction 

Iterate 

3 times 

4


5

PMVS 

6

1. Patch 

Definitions 

: : ,, ,, =−(,,) 

=−(,,) 

() () = 

∑ ] ] 

 

∑ ] 

∑ ] 

 

= 

∑ = 

 

∑ 

∑ ] 

∑ ] 

 

= 

∑ = + 

 

∑ + 

7

2. Photometric Discrepancy Function 

= 

1 

|()\R()| 

ℎ(,, ) 

()\() 

ℎ , , = 1−(,, ) 

: visible image which ℎ 

() ∶ 

Definitions 

8

() ∶ {| ∙ 

 

>cos()} 

== 60°, ℎ = {1, 2, 3} 

= 0.7 + 0.4 = 1.1 

∗ : ∈,ℎ, , ≤ } 

== 0.5 ℎ ∗ = {1,3} 

∗ = 0.4 

9

3. Image Models 

Definitions 

 

10


2. Expansion 








11

1. Detect blob and corner features using the DoG 

and Harris operators 

(32*32 grids & 4 local maxima with strongest response) 

Harris: 

Feature Detection 

 

 

 

 

 

 

12


Harris: M matrix derivation(local auto-correlation function) 

13


DOG : 

14

Feature Matching 

15


16


P matrix is known for each image 

= ] 

= − ∗ 

17


All features have been detected 

by last step “Feature Detection” 

18


19


(): { ’} 

: /| | 

: 

20


:{| ∙ 

 

>cos()} 

∗ :∈,ℎ, , ≤ } 

21


Patch Optimization 

22

Optimization: The geometric parameters and 

are optimized by simply minimizing the 

photometric discrepancy score 

∗ = 


1 

|∗ ()\R()| 

ℎ(,, ) 

∗ ()\() 

1. () and () determine a plane which includes the patch 

2. Sampled pixels of patch change with the modification of 

() and () which result in the change of discrepancy 

score 

23


3 freedoms : 

1. Constrain c(p) lie on a ray such that its image 

projection in R(p) does not change, reducing 

its number of degrees of freedom to one and 

solving only for a depth 

() 

 

P’ 

p 

p’ 

24


2. n(p) : parameterized by Euler angles 

(pitch(x axis) and yaw(y axis) ) 

25


() () ℎ 

:{| ∙ 

 

>cos()} 

∗ :∈,ℎ, , ≤ } 

26

true 


The number of V*(p) should 

bigger than threshold r 

27

false 


The number of V*(p) should 

bigger than threshold r 

28


Store p 

Remove cell(p) in Image I 

Add p to P 

Return to the second For-loop 

29


Step1 produces sparse set 

of patches P 

30

Special case solution 

We have so far assumed that the surface of an object or a 

scene is lambertian, and the photometric discrepancy function 

g(p) defined above may not work well in the presence of 

specular highlights or obstacles. In the proposed approach , we 

handle non-lambertian effects by simple ignoring images with 

bad photometric discrepancy scores. 

If the matching procedure starts with a feature in an image 

containing artifacts, the image becomes a reference and the 

patch optimization fails. However, this does not prevent the 

procedure starting from another image without artifacts, 

which will succeed. 

31


2. Expansion 








32

Expansion 

The goal of the expansion step is to reconstruct 

at least one patch in every image cell , and 

we repeat taking existing patches and generating 

new ones in nearby empty spaces. 

More concretely, given a patch p, we first 

identify a set of neighboring image cells C(p) 

satisfying certain criteria, then perform a patch 

expansion procedure for each one of these. 

33


34


Expanding each image cell (, ) 

containing p (seed) 

35


Identifying Cells for Expansion satisfying 

certain criteria 

36


Given a patch p, we first identify a set of neighboring 

image cells C(p) 

= , ∈ , ,|−′|+|−′|=1} 

The expansion is not performed for an image cell if 

1. Neighbor: An image cell ( ,′)contains a patch p’, which is 

a neighbor of p, is removed from the set of 

− ∙ + − ∙ ′ < 2 

2. Depth discontinuity: ( ,′)already contains a patch whose 

photometric discrepancy score associated with is better than 

the threshold . 

37

Neighbor of p 

− ∙ + − ∙ ′ < 2 

38

Neighbor of p 

− ∙ + − ∙ ′ < 2 

− ∙ 

− ∙ ′ 

 

Normal vector () or (’) 

p 

l 

m 

l’ 

p’ 

cell 

N 

39

Neighbor of p 

− ∙ + − ∙ ′ < 2 

− ∙ 

− ∙ ′ 

 


p 

l 

m 

l’ 

p’ 

cell 

N 

is determined automatically as the 

distance at the depth of the mid-point 

of () and (’) corresponding to an 

image displacement of pixels in () 

40

Neighbor of p 

− ∙ + − ∙ ′ < 2 

− ∙ 

− ∙ ′ 

 


p 

l 

m 

l’ 

p’ 

cell 

N 

is determined automatically as the 

distance at the depth of the mid-point 

of () and (’) corresponding to an 

image displacement of pixels in () 

p 

l 

m 

l’ 

p’ 

cell 

N 

41

Depth Discontinuity 

In practice, difficult to judge the presence of depth discontinuities 

before actually reconstructing a surface. 

We simply judge that the expansion is unnecessary due to a depth 

discontinuity if ∗ (’, ’) is not empty : If (’, ’) already contains 

a patch whose photometric discrepancy score associated with is 

better than the threshold . 

42


1. Initialize patch p’, copying R(p’), V(p’), 

n(p’) from p 

2. c(p’): Intersection of optical ray 

through center of C(x’, y’) with plane 

of p 

3. c(p’) is not same as c(p), so update 

V*(p’) 

p’ 

p 

R(p) 

cell 

43


Details at 

slide 19 & slide 20 

44


1. After the c(p’) n(p’) are refined , 

add new visible images to V(p’) 

according to a depth-map test 

(calculated by projection) 

2. by V(p’) has been modified, so 

V*(p’) should be updated 

45


If ∗ ’ ≥ γ, we accept the patch as a 

success and update , & ∗ , 

46


Step 2 

Sparse P ---> Denser P 

47


2. Expansion 








48

Filtering 

Filtering step should be taken because occlusion doesn’t be considered 

in the expansion step. The first two filters enforce visibility consistency , 

and the last filter enforces a weak form of regularization . 

49

Filtering 

p is filtered out as outlier if the following inequality holds: 

∗ ∗ 

∈() 

When p is an outlier , both ∗ and 1− are expected to 

be small, and p is likely to be removed. 

Let U(p) denote the set of patches p’ 

that are inconsistent with current 

visibility information --- that is, p and 

p’ are not neighbors but are stored in 

the same cell of one of the images 

where p is visible 

50

2. Second filter 

Filtering 

For each patch p, we compute the number of images in V*(p) 

where p is visible according to depth-map test. If ∗ ’ < γ , 

remove p from P. Note that the recomputed values of ∗ ’ may be 

different from those obtained in the expansion step since more 

patches have been computed after the reconstruction of p. 

51

3. Enforce a weak form of regularization as follows: 

For each patch p, we collect the patches lying in its own and 

adjacent cells in all images of V(p). If the proportion of patches that 

are neighbors of p − ∙ + − ∙ ′ < 2 

in this set is lower than 0.25, p is removed as an outlier. 

p’ 

p 

Filtering 

52

Conclusion 

The algorithm starts by detecting features in 

each image, matches them across multiple images 

to form an initial set of patches, and uses an 

expansion procedure to obtain a denser set of 

patches, before using visibility constraints to filter 

away false matches 


2. Expansion 

3. Filtering 

Iterate 3 times 

53

Results 

54

Results 

55

Thank you! 

56

Accurate, Dense, and Robust Multi-View Stereopsis (PMVS) [1,2,3]

Create successful ePaper yourself

Delete template?

Save as template?