View Synthesis of Symmetric Objects using Examples - International ...
View Synthesis of Symmetric Objects using Examples - International ...
View Synthesis of Symmetric Objects using Examples - International ...
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>View</strong> <strong>Synthesis</strong> <strong>of</strong> <strong>Symmetric</strong> <strong>Objects</strong> <strong>using</strong> <strong>Examples</strong>Visesh Chari P. J. Narayanan C. V. JawaharCentre for Visual Information Technology<strong>International</strong> Institute <strong>of</strong> Information Technology{visesh@research.,pjn@,jawahar@}iiit.ac.inAbstractTraditional image based rendering methods synthesizenew views from a few input views <strong>of</strong> an object frompositions defined by the input views. In this paper, weextend this approach <strong>using</strong> information from views <strong>of</strong>other “similar” objects. We pose the view synthesisproblem as an image completion problem <strong>using</strong> a suitablerepresentation <strong>of</strong> the scene. We specifically showhow a database <strong>of</strong> images <strong>of</strong> symmetric objects can beused to generate new views <strong>of</strong> the input object that areimpossible with other image based rendering methods.This is done by cleverly sampling the Plenoptic function<strong>of</strong> objects in a mosaic so that information about its symmetryin 3D is preserved. Algorithms based on texturetransfer are further used to refine the solution. Resultsshow the usefulness <strong>of</strong> our approach.1. IntroductionImage Based Rendering (IBR) has been a widely researchedarea in recent years [13]. Since typical applications<strong>of</strong> IBR include virtual reality, animations, etc.,generating high quality images is <strong>of</strong> prime importance.Accomplishing this task is difficult because it requirescomplete knowledge <strong>of</strong> the geometry and photometry<strong>of</strong> the object. Additionally, producing high quality resultsseem to require elaborate camera setups [8] or producesvisible holes in the novel view because <strong>of</strong> occlusion[13]. In this paper, we present a method to computenovel views <strong>of</strong> nearly symmetric objects from positionsimpossible to current methods: views that require informationnot available in the input sequence. This is doneby transferring semantic information from a database <strong>of</strong>“similar” objects, for which it is affordable to capturemultiple views with precision.IBR methods pertaining to 3 basic categories havebeen proposed earlier. Techniques based on stereo reconstruction[13] and volumetric techniques [7] rely onFigure 1: Traditional IBR vs Our Approach. Whiletraditional IBR use information only from input sequences,we transfer Semantic information from similarscenes too.accurate and complete knowledge <strong>of</strong> the geometry <strong>of</strong>the scene for rendering purposes. In contrast to thesegeometry based approaches, methods based on samplingthe Plenoptic function [8] aim at capturing all raysrequired for rendering views within a specified 3D volume.A hybrid approach where geometry is implicityused to generate images at novel view points is viewdependent geometry [5].These techniques, however, only use infomationavailable in the input image/depth map sequence,thereby limiting the gamut <strong>of</strong> possible reconstructions.By augmenting the current scene <strong>using</strong> “similar” scenesthat might be available to us, this bottleneck may beovercome. Advantages <strong>of</strong> such an approach are threefolds.The availability <strong>of</strong> more information increasesthe gamut <strong>of</strong> possible novel views. Secondly, only thedatabase <strong>of</strong> “similar” scenes needs to be carefully cap-
tured. Thirdly, the presence <strong>of</strong> a database provides severalalternative solutions in the absence <strong>of</strong> knowledgeabout the scene’s actual content (Figure 7) Effectively,where other IBR approaches describe algorithms forview interpolation, we describe an algorithm for viewextrapolation: the generation <strong>of</strong> plausible views givenan input image sequence.We demonstrate our approach <strong>using</strong> a representationthat is a hybrid <strong>of</strong> Plenoptic modeling and stereo reconstructionbased methods [11]. Sections 2 discuss theproblem and our approach, and we show how the representation<strong>of</strong> [11] allows us to transfer information likesymmetry and texture across objects in Section 3. Details<strong>of</strong> implementation are presented in Sections (3,3.1)and results in Section 4. Finally we enumerate futuredirections <strong>of</strong> research and conclude with Section 5.Figure 2: Symmetry extraction: Top row: Original image,difference between original and flipped images, opticalflow information between the original and flippedversion computed <strong>using</strong> a variation <strong>of</strong> [9]. Bottom row:A frontal image formed by <strong>using</strong> symmetry informationfrom different faces.2. Problem DefinitionGiven a collection <strong>of</strong> n 2D images I 1...n ={I 1 , . . . , I n } and their corresponding depth mapsD 1...n = {D 1 , . . . , D n } <strong>of</strong> an object, we wish to infera new view V, all <strong>of</strong> whose information cannot besupplied by the input images and depth maps. Additionalinformation in the form <strong>of</strong> knowledge from similarscenes is available to us. Currently, let us denote thisinformation by S 1...m = {S 1 , . . . , S m }, where m is thenumber <strong>of</strong> scenes available.Since much <strong>of</strong> the information available in the inputsequence is redundant, it would be advantageousto have a representative mosaic <strong>of</strong> the scene. We usea Multiple Centre <strong>of</strong> Projection (MCOP) image [11]to represent the information about the geometry <strong>of</strong>the symmetric object, since this representation easilycaptures semantic information like the symmetry <strong>of</strong> ascene (Section 3.2), while allowing texture transfer approachesto augment or replace information in any part<strong>of</strong> the scene [4]. A limited number <strong>of</strong> images <strong>of</strong> a sceneproduces an incomplete MCOP image, and producingnovel views <strong>of</strong> a scene now amounts to completing thisimage. We use a nearest neighbhor approach to findMCOP images similar to the input sequence from thedatabase, and then transfer semantic information fromthem to complete the input MCOP image.3. RepresentationsWe use methods from the Plenoptic modeling literatureto find a representation <strong>of</strong> the entire scene <strong>using</strong>a single image. Such images have been given manynames, one <strong>of</strong> them being Multiple Centre <strong>of</strong> Projection(MCOP) Images [11], and are the collection <strong>of</strong> lightrays and range maps computed <strong>using</strong> a slit camera anda laser range finder, moving along a pre-defined paththrough the scene. The 3D model to be rendered is nowgiven by the equation [11]( xyz)= δ i,j[Uix V ix O ixU iy V iy O iyU iz V iz O iz] ( ij1)+(Cix)C iyC izwhere for each pixel (i, j), C i represents the centre<strong>of</strong> projection, O i represents a vector from C i to the imageplane origin, U i represents the horizontal axis <strong>of</strong>the projection plane, and V i represents the vertical axis.Each pixel’s depth is represented by δ i,j .3.1. Semantic MatchingIn order to transfer semantic information, we firstpick an MCOP image from the database, that is similarin a suitable feature space to the incomplete inputMCOP image. First, a feature vector containing grayvalue and gradient information for each MCOP image(texture and depth) is constructed. To ensure that colorsimilarity is not given importance, a weighted cost function<strong>of</strong> the feature distance between input and databaseimages is minimized.mini∑ ∑sΩW ∗ (F sinput − F si )2 (2)where s denotes the scale, Ω denotes the region <strong>of</strong>interest, W denotes the weight matrix and F denotesthe feature vector for each pixel <strong>of</strong> the image.(1)
3.3. Texture Transfer(a) Input(b) Database image(c) Symmetry transferFigure 3: Symmetry Transfer: (a) The input to symmetrybased hole filling. (b) The database image used totransfer symmetry. (c) The generated result.The MCOP images generated <strong>using</strong> few images willcontain holes (Figure 3a). Symmetry based methodscan be used to fill some <strong>of</strong> these holes. In some caseshowever, enough information is not available for symmetrybased methods to produce novel views withoutholes (Figure 3c). Thus, methods based on texturetransfer from other images have to be employed.Given the closest MCOP image(s) to the input as perequation 2, we transfer appropriate texture from thedatabase image and use to method <strong>of</strong> [10] to blendthe textures from different images into the currentone [6](Figure 3b). Unlike [6], pixels belonging to theMCOP image are left unmodified by setting them ashard constraints. To ensure consistency, all the texturetransferred to fill a hole is taken from one image andblended <strong>using</strong> a poisson solver [2].3.2. SymmetryAny symmetry in a 3D object is captured by its correspondingPlenoptic function. Since MCOP imagescapture a sample <strong>of</strong> the Plenoptic function, by cleverlydesigning the path <strong>of</strong> the camera employed to capturethe desired MCOP image, we may capture the symmetryproperty in an image.Real world objects are however never perfectly symmetric(Figure 2). This asymmetry is sometimes differentfor objects <strong>of</strong> the same class (eg. faces). By registeringthe different parts <strong>of</strong> the object with each other,this slight deviation from symmetry may be capturedaccurately. We register an MCOP image with its mirrorimage by <strong>using</strong> a variation <strong>of</strong> the optical flow algorithmin [9] presented in [12]. The only modification we addis to weight each <strong>of</strong> the different channels according toour need. In essence we try to find the flow that minimizesthe following functional∫F low(u, v) = (W ∗ Ψ(I [k] (x + w) − I [k] (x))) 2 (3)Ω+Ψ(|∆u| 2 + |∆v| 2 )Once this registration information is given, images<strong>of</strong> the whole object may be generated from its partialviews. In the absence <strong>of</strong> this information, it may beborrowed from other objects <strong>of</strong> the same class, as shownin Figure 2.Figure 4: Texture Transfer: Result <strong>of</strong> grafting a noseonto the result in Figure 3c4. ResultsResults for symmetry and texture synthesis areshown in Figure 6. Here, the nose <strong>of</strong> the subjectwas removed from the depth map before capturingits various views into an MCOP image. Figure 3ashows the MCOP image after symmetry informationwas transferred from the database. The remainingtexture gap was then filled <strong>using</strong> the subject in Figure4. Novel views <strong>of</strong> the subject are then rendered(Figures 6a, 6b, 6c). Additional results may be foundhere [1].An advantage <strong>of</strong> such augmentation <strong>of</strong> informationis the variety <strong>of</strong> modifications that are also available tothe user. Figure 5 shows one such result. We call this“symmetrizing” an object, in the sense that the inherentasymmetry present in the object is removed. This isdone by computing the optical flow between an MCOPimage and its mirror image, and warping the MCOPimage with the flow. Then, the warped MCOP image is
5. Conclusions(a)Figure 5: (a): Original image. (b) Symmetrized result.Notice the change in the right half <strong>of</strong> the face.(b)We presented a novel framework to extend view synthesisto situations where enough information for renderingthe entire scene is not given as input. Semanticinformation is taken from images <strong>of</strong> similar objects orscenes that are available in the database to synthesizeviews that are impossible for traditional methods. Webelieve this paper is an important first step towards theuse <strong>of</strong> semantic information from a known database <strong>of</strong>images into view synthesis.References(a) (b) (c)Figure 6: Symmetry and Texture transfer: Novel views<strong>of</strong> the scene generated in Figure 4.merged with its intact mirror image to produce a “symmetric”version <strong>of</strong> the person.Finally, any missing information has several alternatesources to be borrowed from. Figure 7 shows thevariations <strong>of</strong> the same object produced with informationtaken from the first 5 nearest neighbors. As can be seen,different sources not only produce change in texture information,but also produce changes in the geometry <strong>of</strong>the object. All the results in this paper are on the USFHuman ID 3D Face Database [3].Figure 7: Various variations <strong>of</strong> the nose, mouth andears <strong>of</strong> a person by transferring information from thedatabase.[1] http://research.iiit.ac.in/∼visesh/myweb/Publications/ibr.pdf.[2] A. Agrawal, R. Raskar, and R. Chellappa. What is therange <strong>of</strong> surface reconstructions from a gradient field. EuropeanConference on Computer Vision (ECCV), 2006.[3] V. Blanz and T. Vetter. A morphable model for the synthesis<strong>of</strong> 3d faces. In SIGGRAPH ’99: Proceedings <strong>of</strong> the26th annual conference on Computer graphics and interactivetechniques, pages 187–194, New York, NY, USA,1999. ACM Press/Addison-Wesley Publishing Co.[4] A. Efros and W. T. Freeman. Image quilting for texturesynthesis. ACM SIGGRAPH, pages 341–346, 2001.[5] A. W. Fitzgibbon, Y. Wexler, and A. Zisserman. Imagebased rendering <strong>using</strong> image based priors. <strong>International</strong>Conference on Computer Vision (ICCV), 2003.[6] J. Hays and A. A. Efros. Scene completion <strong>using</strong> millions<strong>of</strong> photographs. In SIGGRAPH ’07: ACM SIGGRAPH2007 papers, page 4, New York, NY, USA, 2007. ACM.[7] K. N. Kutulakos and S. M. Seitz. A theory <strong>of</strong> shapeby space carving. Int. J. Comput. Vision, 38(3):199–218,2000.[8] M. Levoy and P. Hanrahan. Light field rendering. In SIG-GRAPH ’96: Proceedings <strong>of</strong> the 23rd annual conferenceon Computer graphics and interactive techniques, pages31–42, New York, NY, USA, 1996. ACM.[9] N. Papenberg, A. Bruhn, T. Brox, S. Didas, and J. Weickert.Highly accurate optic flow computation with theoreticallyjustified warping. <strong>International</strong> Journal <strong>of</strong> ComputerVision, 67(2):141–158, 2006.[10] P. Pérez, M. Gangnet, and A. Blake. Poisson image editing.In SIGGRAPH ’03: ACM SIGGRAPH 2003 Papers,pages 313–318, New York, NY, USA, 2003. ACM.[11] P. Rademacher and G. Bishop. Multiple-center-<strong>of</strong>projectionimages. In SIGGRAPH ’98: Proceedings <strong>of</strong> the25th annual conference on Computer graphics and interactivetechniques, pages 199–206, New York, NY, USA,1998. ACM.[12] P. sand and S. teller. Particle video. IEEE Conference onComputer Vision and Pattern Recognition (CVPR), 2006.[13] D. Scharstein. <strong>View</strong> synthesis <strong>using</strong> stereo vision. volume1583 <strong>of</strong> Lecture Notes on Computer Science (LNCS),Springer-Verlag New York, Inc.