Deliverable 5.2 - the School of Engineering and Design - Brunel ...

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsProject Number:Project title:Deliverable Type:CEC Deliverable Number:2484203D VIVANTPublicIST-248420/Brunel/WP05/PU/R/Del-5-2Contractual Delivery Date: 31 th January 2013Actual Delivery Date: 4 th March 2013Title of the Deliverable:Workpackage:Nature of the Deliverable:Organisations:Search and Retrieval Mechanisms and ToolsWP5Report1 Brunel University2 Centre for Research and Technology Hellas – Informatics andTelematics Institute3 Institut für Rundfunktechnik GmbH4 Holografika5 RAI researchcentre6 Rundfunk Berlin-Brandenburg7 Instituto de Telecomunicações8 European Broadcast Union9 Arnold & Richter Cine TechnikAuthors:Circulation List:Theodoros Semertzidis, Vasilis Lovatsis, Michael. G. Strintzis,Petros DarasPartners and Public on InternetKeywords:search, content-based retrieval, holoscopic imaging4/03/2013 1

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsVersion ControlChange Log Version Date1 st draft(Iordanis Biperis - CERTH) 0.1 12/10/20102 nd draft (Thodoris Semertzidis - CERTH) 0.2 10/8/20123 rd draft (Vasilis Lovatsis – CERTH) 0.3 16/8/20124 nd draft (Thodoris Semertzidis – CERTH) 0.4 17/12/20125 rd version (Vasilis Lovatsis – CERTH) 0.5 04/3/20134/03/2013 2

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsTABLE OF CONTENTS1 INTRODUCTION ................................................................................................................................... 61.1 EXECUTIVE SUMMARY .......................................................................................................................... 61.2 DESIGN OBJECTIVES AND DOCUMENT STRUCTURE ....................................................................................... 72 FRAMEWORK ARCHITECTURE AND GRAPHICAL USER INTERFACE ......................................................... 83 DATABASE PREPARATION ................................................................................................................... 123.1 DATABASE STRUCTURE .............................................................................................................................. 123.2 DESCRIPTORS EXTRACTION ......................................................................................................................... 133.3 CODEBOOK GENERATION AND BAG-OF-WORDS .............................................................................................. 163.4 MANIFOLD LEARNING AND UNIFIED SEARCH SPACE .......................................................................................... 163.5 INDEX CREATION ...................................................................................................................................... 174 SEARCH FOR SIMILAR CONTENT ......................................................................................................... 184.1 SINGLE FRAME SEARCH .............................................................................................................................. 184.2 BATCH SEARCH TARGETING HYPERLINKER ENVIRONMENT .................................................................................. 215 IMPLEMENTATION DETAILS AND TOOLS ............................................................................................. 236 CONCLUSIONS .................................................................................................................................... 257 BIBLIOGRAPHY ................................................................................................................................... 264/03/2013 3

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsLIST OF FIGURESFigure 1: Search and Retrieval framework architecture ......................................................................... 8Figure 2: 3DVIVANT S&R framework starting GUI ............................................................................ 9Figure 3: The menu of Search & Retrieval software ............................................................................ 10Figure 4: Dynamic behaviour of the search and retrieval framework. ................................................. 12Figure 5: Database filesystem structure................................................................................................ 13Figure 6: Descriptors Extraction User Interface ................................................................................... 14Figure 7: descriptors extraction is a lengthy process. The console prints logs on the progress of theprocess .......................................................................................................................................... 15Figure 8: file format for the local features file ..................................................................................... 15Figure 9: the manifold learning process produces a single multimodal descriptor per object ............. 16Figure 10: kd-tree indexing structure ................................................................................................... 17Figure 11: Query formulator window ................................................................................................... 18Figure 12: segmentation mask is loaded and the user may click the object to use as query ................ 19Figure 13: results window with the retrieved similar objects. .............................................................. 20Figure 14: examine the 3D model from the relevant results ................................................................ 20Figure 15: XML schema for the output data, targeting the hyperlinker environment .......................... 224/03/2013 4

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsGLOSSARY2DTwo-dimensional3DThree-dimensionalGUIGraphical user interfaceBowBag of wordsS&RSearch and Retrieval4/03/2013 5

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools1 INTRODUCTION1.1 EXECUTIVE SUMMARYThe past decade has witnessed an exponential growth in digital multimedia production andcommunication. If nowadays a huge number of images and thousands hours of video are created eachday by professionals or home users, the establishment of the holoscopic imaging technology,advanced by 3D VIVANT, will lead literally to an explosion in the digital content production. Withsuch an increasing rate of content production, the development of effective and efficient contentbasedretrieval tools becomes of utmost importance.3D VIVANT aims to provide tools that will allow the retrieval of similar objects from holoscopiccontent databases. Given the inherent problems of current text-based search engines relying onsubjective manual or automatic annotation, 3D VIVANT opted for a search-by-example approach.The search and retrieval framework can be used from both the professionals and home users.However, in the hyperlinker scenario only the professional users will have access to the integral videoediting and thus also to the search and retrieval framework. Due to this fact, the emphasis in theGraphical User Interfaces was given with the professional user in mind. For a home user, thedifference in the GUI design would be to have less configuration options and emphasize the ease ofuse and not focus on the advanced control of the framework.The usage of the search and retrieval framework will enable efficient editing of hyperlinked integralvideo and content reuse in different scenarios. Moreover, the tools and methodologies can be used bythe home / amateur user to search inside large multimedia databases.To bridge the gap between conventional 2D and 3D technology with holoscopic imaging, 3DVIVANT develops content-based retrieval mechanisms that support multimodal queries, i.e. the useris able to pose holoscopic content queries and the framework will answer these queries with similarmultimedia content from various modalities, such as: 2D images, full 3D models or range scans withor without texture information.In order to reduce the integration constraints and enable a modular usage of the software systemsproduced in 3D VIVANT, allow easy adaptation of the framework to the Content Editing Terminaland enable the usage of different PCs in editing and pre-processing of the content, the software wasdeveloped as an autonomous system. In this approach the S&R framework takes the original integralvideo sequence and metadata and extracts an XML file to be used in the hyperlinker framework. Theselected approach enables integral video pre-processing in advance and distribution of thecomputational workload on different PCs. The hyperlinker environment is presented in deliverableD5.4[2], however, a short presentation of the usage of S&R framework with hyperlinker is discussedlater in this document.Finally, it is worth noting that the software is based on open source libraries that can be compiled onvarious operating systems and it can be easily customized.4/03/2013 6

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools1.2 DESIGN OBJECTIVES AND DOCUMENT STRUCTUREThe document provides a detailed description of the search and retrieval framework that wasdeveloped in the context of Task 5.1 under WP5. It is emphasized that this document presents theframework, i.e. the environment, mechanisms, tools and User Interfaces, while the actual search andretrieval algorithms are described in deliverable D5.1[1]. The search and retrieval framework is asoftware platform that provides the tools and user interfaces to: a) prepare the integral videosequences to be searchable b) to search inside a multimedia database for similar content c) to extractXML files to be used in the hyperlinker environment.This deliverable initially presents the overall architecture of the search and retrieval framework andthen separates the presentation of the framework in two fundamental sections: a) the databasepreparation and b) the actual search inside the database. The graphical User Interfaces (GUI) for eachfunctionality of the system are presented in-between the discussion of the components.The core of the system is the low-level feature extraction process, during whichmultimediadescriptors are extracted for each multimedia object in the database. Then a manifold learningalgorithm combines descriptors from different modalities to unify the search space and providemultimodal search capabilities.The descriptor extraction procedure is followed by a feature matching step, whose aim is to establisha similarity measure between any two objects. This step is highly dependent on the low-level featureextraction module and together they form the basis of the search and retrieval framework.Finally, the search engine’s graphical user interface is extensively presented, enhanced with examplesthat demonstrate each step of the search and retrieval procedures.The rest of the document is structured as follows: Section 2describes the framework’s architecture,basic modules and the menu items. Section 3 discusses the structure of the database and the actionsfor preparing the database for search queries. Section 4 presents the procedures for searching insidethe database using two different approaches, while in section 5 the implementation details andprogramming tools are presented. Finally, in section 6 the conclusions of the document are drawn.4/03/2013 7

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools2 FRAMEWORK ARCHITECTURE AND GRAPHICAL USERINTERFACEThe search and retrieval framework was developed to have a modular and easily expandable sourcecode for further future improvements based on the upcoming requirements. With this concept inmind, most of the components of the framework are thread based classes that work independently ofthe other classes and interconnect and exchange data through the application’s root process. Thisapproach enabled us to perform and test several configurations until the final decision. A roughschematic diagram of the framework’s architecture is presented in Figure 1.Graphical User InterfaceDescriptorextractor threadsDatabasethreadSearch process and resultspresentation threadLoggerthreadFigure 1: Search and Retrieval framework architectureThe descriptor extractor threads are triggered to extract one of the supported descriptoralgorithms in the selected database. After of the descriptor extraction process is finished thethreads terminate. A new thread is started each time a new database should be processed.The database thread keeps track of the access to the filesystem database and mainly keeps inmemory the index of the database to be ready for search queries. The database thread lifespan is for the entire applications uptime and thus it is considered permanent thread.The search process thread is started each time a new search query is posed on the system.After processing the query, asking the database for similar content, and building the rankedlist of the results the thread terminates.Finally, the logger thread keeps track of all the actions that occur from the start to the end ofthe application’s lifecycle and displays them in a window of the UI.The Search and Retrieval Framework was developed as a multi-document interface (MDI)environment in order to support multiple functionalities in parallel. The startup UI is presented inFigure 2. At the top of the UI there is a toolbar menu with all the available functionalities as well assome icon buttons for the most common features. The toolbar from left to right has a search menu, adatabase preparation menu, an options menu, a“window” tiling and navigation menu, and a “help”that navigates the user to the “about” window for information about the software.4/03/2013 8

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsFigure 2: 3DVIVANT S&R framework starting GUIIn Figure 3 presents the expanded menu with all the available functionalities. Figure 3(a) presents the“Search” menu from where the user can start a new search process, open and examine the results of asearch process and save the results in a file. Figure 3(b) presents the two functionalities in preparingthe database. The first is the descriptor extraction process which is discussed in details in section3.2and the creation or update of the index to enable fast search inside the database. In Figure 3(c) theoptions menu presents the two available options. The first is to open the settings UI and change theconfiguration of the software and the second option to open the terminal window where log messagesappear during each process. The “window” menu in Figure 3(d) enables the positioning andnavigation through the available open windows. Finally, the“help” menu provides an “about” windowwith information on the software and acknowledgement to 3DVIVANT project.(a)(b)4/03/2013 9

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools(c)(d)(e)Figure 3: The menu of Search & Retrieval softwareFor the configuration of the software, a configuration file is available. The configuration file keepsfundamental information for the operation of the application and should be changed with care. Theconfiguration file can be accessed both from the filesystem and the application interface. ThevivantEngine.ini file is in the application folder.Figure 4 presents the settings window and thecorresponding vivantEngine.ini file.Figure 1(a)(b)Figure 4: (a) settings window (b) vivantEngine.ini file containing the same info with settings window4/03/2013 10

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsThe configuration file contains the following information:Latest window positionWindow sizeDatabase pathJava pathBinary files pathSelected default descriptorBatchMode flagFrom the above settings the database path is the most critical one since it points to the database towhich all operations will be applied. Java path parameter is used if CEDD descriptors are selectedsince the implementation is provided as a jar file. The binary files path contain other helperapplications for the processing of the data. Finally the batch mode flag is used to process batch dataand extract the XML files for the hyperlinker.4/03/2013 11

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools3 DATABASE PREPARATIONA fundamental step in the preparation of a multimedia database in order to be searchable is to extractdescriptor vectors that represent each multimedia object. In the case of 3DVIVANT S&R frameworkthis also stands true. This section describes the database structure, the descriptors extraction processand the final indexing structure creation to enable multimedia similarity search. This process is alsoknown as offline process as it is depicted in Figure 5.OfflineContentRepositoryPre-ProcessingLow-level FeatureExtractionDescriptorsBrowsing / Selectingexisting objectFeatureRetrievalFeatureMatchingResultsSubmitting newobjectPre-ProcessingLow-level FeatureExtractionOnlineFigure 5: Dynamic behaviour of the search and retrieval framework.The steps of the offline process are discussed in the following subsections. Section 3.1 refers to thedatabase structure, section 3.2 describes the descriptors extraction process and the user’s options,section 3.3 discusses the codebook generation process and the extracted histograms (bag of words)and finally section 3.5 describes the creation of indexes for fast similarity search inside the database.3.1 DATABASE STRUCTUREIn 3DVIVANT the database is actually a filesystem structure that contains all the necessary data andmetadata. The structure aims to organize the data in a) the raw multimedia files b) the features files c)the bag-of-words files d) and finally the index files. The filesystem structure is presented in Figure 6.In the “data” directory of the filesystem, the raw data are separated based on the modality in order todrive the appropriate descriptor extraction algorithms properly for each modality. Moreover, theproposed structure enables easier reference of the multimedia objects in the retrieval process.The decision to select the S&R database to be a filesystem structure is supported by variousarguments. First, a filesystem database is very easy to create even for a novice user. The possibility4/03/2013 12

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Toolsfor a user to have direct access to the actual content provides many advantages in a process of a videoediting scenario such as the “hyperlinker scenario”. Moreover, the S&R framework aims to be usedalso by novice/home users that need to search inside their multimedia content, that most of the timeisstored inside a folder on their hard disk drive.Figure 6: Database filesystem structure3.2 DESCRIPTORS EXTRACTIONThe selected descriptor extraction algorithms run through the “data” directory in the appropriatemodality (viewpoint images, depth maps, curvature images, 2D snapshots etc.) and extract onefeatures’ file for each multimedia file. For each supported descriptor there is a directory with the same4/03/2013 13

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Toolsname under the “descr” folder. Moreover, under each descriptor folder there are two subfolders (seeFigure 6): one for the extracted features (“features” folder) and one for the bag-of-words vector(“words” folder) that will be generated for each feature file, using a generated codebook (see section3.3).The Search and Retrieval Framework supports the following descriptor extraction algorithms for theextraction of local features of the multimedia objects:1. CSIFT: Colour SIFT2. OSIFT: Opponent SIFT. A Colour SIFT variant where a different colour model is used.3. GB: Geometric Blur4. SSIM: Self Similarity5. PI: Projection Images. Applicable only in depth dataThe User Interface for extracting descriptors (see Figure 7) enables the user to select whether thesystem will generate descriptors for one descriptor algorithm or for all of them. Based on the selectionof the user, the system triggers one or multiple threads that extract descriptors for the same raw data(under “data” directory) in parallel. From this step until the finish of the final descriptors the systemworks without the need for user intervention. The extraction of the local descriptors is a lengthyprocess and thus a progress bar informs the user for the progress of the computations (Figure 8).Moreover, the console window prints log messages to give more information on the runningprocesses.Figure 7: Descriptors Extraction User Interface4/03/2013 14

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsFigure 8: Descriptor extraction is a lengthy process. The console prints logs on the progress of the processAs it was stated above, each descriptor extraction algorithm generates its own local feature vectorsand stores them under the “descr” directory in the corresponding name. To unify the process of datastorage and ease the Input /Output processes we chose to have a unique format for all the availablelocal features files. The format include the algorithm name, the dimensions of the feature vector, thenumber of the local features in the file and then one vector per line along with the positioninginformation for each local feature vector. The file format is presented in Figure 9.Figure 9: file format for the local features fileWhen the descriptor extraction processes finish, the bag-of-words histogram should be computed. Tocalculate this final histogram a codebook is needed. Section 3.3 discusses this process.4/03/2013 15

ICTProjectContract no.:2484203D VIVANT–Deliverable5.2Search& Retrieval Mechanismss &Tools3.3CODEBOOK GENERATION AND BAG-OF-WORDSSince most of the well performing descriptor extraction algorithms extract local features, there aremany feature vectors that correspondtoa single multimedia object. The local features for eachmultimedia object may be hundreds or thousands andthus it is very difficult to use in an efficient wayfor the matchingstep. A common approach to this problem is thebag-of-words (BoW) method. Withthe BoW, a codebook with the most dominant code words (quantized descriptor vectors) is generated.The codebook isgenerated using a k-means algorithmwith a typical k of 5000 to 1000 dimensions andby sampling the local features available in the database.Theneach set of local features is checked against the codebook to generate a histogram ofoccurrences of code words for each features’ file (which correspond to one multimedia object). Thefinalhistogram of the BoWis actually a new descriptor that represents themultimediaobject withonlyone descriptor vector. These final BoW descriptor vectors are stored inthe database under the“words” directories.3.4MANIFOLD LEARNING AND UNIFIED SEARCH SPACEThe next step after the generation of the bag-of-words is the creation of the unified search space as itwas described in details inDeliverable D5.1. Forthis the manifold learning process starts andcombines the different modalities (viewpoints, depth, curvature) to build a unified vector space.Thethatfinal outcome of the process is a single descriptor vector (see Figure 10) per multimedia objectcorrespondss to the unified multimodal search space.Figure 10: the manifold learning process produces a single multimodal descriptor per objectThisfinal singledescriptor vector per multimedia object is thenindexed to prepare the database forsimilarity searchqueries.4/03/201316

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools3.5 INDEX CREATIONIt is very common in a search environment to use indexing structures to enable faster similaritysearch. For the 3DVIVANT search and retrieval framework the kd-tree indexing structure (see Figure11) was selected. The kd-tree builds a tree data structure (or a number of trees – kd-tree forest) wherethe most similar descriptor vectors are close together. By using such a structure, the exhaustive linearsearch is avoided and thus the framework answers faster in similarity search queries.List ofSpace partitiona: [v a,1v a,2...v a,n]b: [v b,1v b,2...v b,n]⁞k-D12 34 5 6a b c d e f gFigure 11: kd-tree indexing structureIn the cases that the database is small or there are no external-to-the-database queries the kd-treeindexer may not be used. In that case the engine can pre-compute the search results for each record inthe database and store these results for future queries.4/03/2013 17

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools4 SEARCH FOR SIMILAR CONTENTIn theonline phase of the framework, the database should be already prepared and waiting for queries.The framework supports two kinds of querying methods. The first is the typical method of a userposing a query multimedia object (a single query – see section 4.1) and the system replies with theresults list. The second method is a “batch search” method where a user may need to pre-compute theresults for a set of queries to use them in the future or outside the framework (section 4.2). Thissecond approach is used in the hyperlinker demo process. This will be presented in detail inDeliverable D5.44.1 SINGLE FRAME SEARCHIn the single frame search functionality the user selects to perform a new query from the menu or thetoolbar icon. A new search thread is generated and a new window is presented in the UI (Figure 12).The window provides an “open” button that opens the platform’s native “file open” dialog. The userselects the integral frame file that he wants to use as query and loads it into the window. The “loadmask” button loads the corresponding segmentation mask from the database. The segmentation maskloading opens the mask image file and reads from the corresponding XML file or computes on the flythe bounding boxes and clickable areas for each object in the frame. After this process the user is ableto click the exact object (see Figure 13) in the loaded integral image and use it as a query. In this casethe query is only the selected object and not the entire query image. If the user selects to not load themask and just click the search button, the whole integral image is used as a query image.Figure 12: Query formulator window4/03/2013 18

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsFigure 13: segmentation mask is loaded and the user may click the object to use as queryAfter the query is submitted with either of the two methods, the search process is started. Thedescriptor extractor process is started and the descriptors are generated. Then the manifold learningalgorithm is called to project the set of selected descriptors to the multimodal space. This finaldescriptor vector is used to query the index for similar content.The final results of the process appear in a new window (see Figure 14) that displays a list of objectsfrom the most to the least similar one, using both a central viewpoint image and the original modalityobject (e.g. integral image or 3D model etc.).4/03/2013 19

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsFigure 14: Results window with the retrieved similar objects.In the case that one of the retrieved results is a 3D model, the user can click and select the object for adetailed examination. In this case a new window pops up and the 3D model is loaded in a 3Drendering environment. The environment provides easy 3D model manipulation using only mousegestures. Figure 15 depicts this 3D rendering window with the 3D manipulator UI appear in red lineon top of the 3D model.Figure 15: Examining the 3D model from the relevant results4/03/2013 20

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools4.2 BATCH SEARCH TARGETING HYPERLINKER ENVIRONMENTThe batch search method is targeting the hyperlinker environment. The goal of this feature is to preprocessan integral video sequence (the raw data database) and generate the whole set of ranked listsfor the available clickable objects in each frame. The output of the process is one XML file for eachintegral video frame. The XML file along with the corresponding video frame, are used as input to theHyperlinking environment tool, to enable the tool to present search results for the clickable objects.These results are presented to the user / editor to select possible hyperlinks of the object to anothermultimedia object or just a web URI.In the batch search approach the user should feed the engine with all the necessary raw data (integralimages, central viewpoints etc.) and segmentation masks and the appropriate XML files providedwith the segmented masks for each frame of a video sequence.The batch process is triggered by setting batchMode=yes in the configuration file and then clickingbuild index button. By selecting to process the sequence in batch mode the engine generates an all-toallsearch matrix. The engine uses all the database records, one at a time, as a query object to findsimilar objects inside the database. For each one of them a ranked list is generated that references thesimilar objects of the database.The difference in this mode is that the user does not need to select and perform search for all theintegral frames by hand. The automated process enables faster and less error prone results.Since the batch mode is used in the hyperlinker demo workflow the appropriate XML file for eachranked list is generated to be used as input to the hyperlinker environment. The XML files containinformation for the bounding boxes of the segmented objects for each frame and the search results ofeach bounding box if available. The schema serves as a general metadata exchange schema betweenthe hyperlinker environment and peripheral tools so the search results are optional for each boundingbox. Figure 16 presents the XML schema of the generated files.4/03/2013 21

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsFigure 16: XML schema for the output data, targeting the hyperlinker environment4/03/2013 22

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools5 IMPLEMENTATION DETAILS AND TOOLSThe search and retrieval framework integrates various tools and technologies in order to fulfil therequired functionalities.The Software is implemented in C++ using the Qt 4.7.3 framework [3], the opencv (Open ComputerVision) library[4] and GLC library for 3D rendering [5]. The software was compiled with MS VisualStudio 2008 compiler and tested on Windows XP and Windows 7 platforms.Qt frameworkQt is a cross platform application and UI framework that provides a broad set of tools for fast andefficient coding. The most important libraries are:Qt Core, which contains all the necessary classes for a core application. It is the foundationof all Qt-based applications. The key functions are: File I/O, objects handling, multithreadingand concurrency, plugins and settings management and finally signals and slots interobjectcommunications.Qt GUI module contains the functionality needed to develop advanced UI applications. Animportant feature of the module is that it uses the native graphics API of each platform itsupports, taking full advantage of the system resources.Except from these basic library modules, Qt includes also the following libraries:1. 2D graphics Canvas2. OpenGL, which was used with GLC library3. WebKit, which was used for the rendering of the hyperlinked frames from the integralsequences4. Scripting5. Multimedia6. Networking7. XML8. Database9. Unit Testing10. Declarative UIs for touch enables and embedded devicesOpenCV libraryOpenCV is a well-known, open source, Computer Vision library with a large community ofdevelopers and users. OpenCV provides a set of image I/O, image processing, computer vision andmachine intelligence algorithms that are well documented and thoroughly tested from the users’community.GLC libraryGLC library provides input/output functionality for most of the well-known 3D data formats such as:4/03/2013 23

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &ToolsCollada V1.4, 3DXML ASCII V3 and V4, OBJ, 3DS, STL (ASCII and binary), OFF and COFF.Moreover, GLC provides easy mechanisms for OpenGL based 3D rendering inside a Qt windowenvironment. A very useful feature of the library is the build-in 3D view manipulator that gives anintuitive UI for the user.4/03/2013 24

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools6 CONCLUSIONSThis document presents the 3D VIVANT Search and Retrieval framework. It presents the actualsoftware and the User Interfaces that incorporate the search and retrieval algorithms described indeliverable D5.1.The deliverable begins with a report on the overall User Interfaces and the menufunctionalities. Then the following sections describe the offline and online processes of themultimedia similarity search concept and present in detail parts of the UI for each function. Thedeliverable describes in depth the database structure that the S&R engine uses and the file formats forall the types of I/O files used. Finally, section 5 discusses the software tools and implementationdetails.The 3DVIVANT search and retrieval framework provides the tools and functionality for efficientlycreate a multimodal multimedia database and perform multimodal queries. The multimodal approachenables the search for similar content using any available modality e.g. search for 3Dmodels but useintegral images as queries.4/03/2013 25

ICT Project 3D VIVANT– Deliverable 5.2Contract no.:248420Search & Retrieval Mechanisms &Tools7 BIBLIOGRAPHY[1]. 3DVivant deliverable D5.1[2]. 3DVivant deliverable D5.4[3]. http://qt.nokia.com/[4]. http://opencv.willowgarage.com/wiki/[5]. http://www.glc-lib.net/4/03/2013 26