Deliverable 5.2 - the School of Engineering and Design - Brunel ...

ICTProjectContract no.:2484203D VIVANT–Deliverable5.2Search& Retrieval Mechanismss &Tools3.3CODEBOOK GENERATION AND BAG-OF-WORDSSince most of the well performing descriptor extraction algorithms extract local features, there aremany feature vectors that correspondtoa single multimedia object. The local features for eachmultimedia object may be hundreds or thousands andthus it is very difficult to use in an efficient wayfor the matchingstep. A common approach to this problem is thebag-of-words (BoW) method. Withthe BoW, a codebook with the most dominant code words (quantized descriptor vectors) is generated.The codebook isgenerated using a k-means algorithmwith a typical k of 5000 to 1000 dimensions andby sampling the local features available in the database.Theneach set of local features is checked against the codebook to generate a histogram ofoccurrences of code words for each features’ file (which correspond to one multimedia object). Thefinalhistogram of the BoWis actually a new descriptor that represents themultimediaobject withonlyone descriptor vector. These final BoW descriptor vectors are stored inthe database under the“words” directories.3.4MANIFOLD LEARNING AND UNIFIED SEARCH SPACEThe next step after the generation of the bag-of-words is the creation of the unified search space as itwas described in details inDeliverable D5.1. Forthis the manifold learning process starts andcombines the different modalities (viewpoints, depth, curvature) to build a unified vector space.Thethatfinal outcome of the process is a single descriptor vector (see Figure 10) per multimedia objectcorrespondss to the unified multimodal search space.Figure 10: the manifold learning process produces a single multimodal descriptor per objectThisfinal singledescriptor vector per multimedia object is thenindexed to prepare the database forsimilarity searchqueries.4/03/201316