3.4.4 Content-Based Image Retrieval
To address their challenges, multimedia signal-processing methods must allow efficient access to processing and retrieval of content in general, and visual content in particular. This is required across a large range of applications, in medicine, entertainment, consumer industry, broadcasting, journalism, art and e-commerce. Therefore, methods originating from numerous research areas, that is, signal processing, pattern recognition, computer vision, database organization, human-computer interaction and psychology, must contribute to achieving the image-retrieval goal. An example of image retrieval is Given: A query
Retrieve: All images that have similar content to that of the query.
Image-retrieval methods face several challenges when addressing this goal [3.68]. These challenges, which are summarized in Table 3.1, cannot be addressed by text-based image retrieval systems, which have had an unsatisfactory performance so far. In these systems, the query keywords are matched with keywords that have been associated to each image. Because of difficult automatic selection of the relevant keywords, time consuming and subjective manual annotation is required. Moreover, the vocabulary is limited and must be expanded as new applications emerge.
To improve performance and address these problems, content-based image retrieval methods have been proposed. These methods have generally focused on using low-level features such as color, texture and shape layout, for image retrieval, mainly because such features can be extracted automatically or semiautomatically.
Statistical and syntactic texture description methods have been proposed. Methods based on spatial frequencies, co-occurrence matrixes and multiresolution methods have been frequently employed for texture description because of their efficiency [3.69]. Methods based on spatial frequencies evaluate the coefficients of the autocorrelation function of the texture. Co-occurrence matrixes identify repeated occurrences of gray level pixel configurations within the texture.
Table 3.1 Image retrieval challenges [3.68].
Query types Color based/shape based/color and shape based
Quantitative, for example, find all images with 30% amount of red
Query by example, for example, image region/image/sketch/other examples
Various content For example, natural scenes/head-and-shoulder images/MRIs
Matching types Object to object/image to image/object to image
Exact versus similarity-based match
Presentation of results Application specific
Multiresolution methods describe the texture characteristics at coarse-to-fine resolutions. A major problem that is associated with most texture description methods is their sensitivity to scale, that is, the texture characteristics may disappear at low resolutions or may contain a significant amount of noise at high resolutions [3.70, 3.71, 3.72].
Describing quantitatively the shape of an object is a difficult task. Several contour-based and region-based shape description methods have been proposed. Chain codes, geometric border representations, Fourier transforms of the boundaries, polygonal representations and deformable (active) models are some of the boundary-based shape methods that have been employed for shape description. Simple scalar region descriptors, moments, region decompositions and region neighborhood graphs are region-based methods that have been proposed for the same task [3.73, 3.74]. Contour-based and region-based methods are developed in either the spatial or transform domains, yielding different properties of the resulting shape descriptors. The main problems that are associated with shape description methods are high sensitivity to scale, difficult shape description of objects and high subjectivity of the retrieved shape results.
Color description methods are generally color histogram based, dominant color based and color moment based [3.75, 3.76]. Description methods that employ color histograms use a quantitative representation of the distribution of color intensities. Description methods that employ dominant colors use a small number of color ranges to construct an approximate representation of color distribution. Description methods that use color moments employ statistical measures of the image characteristics in terms of color.
The performance of these methods typically depends on the color space, quantization, and distance measures employed for evaluation of the retrieved results. The main problem that is associated with histogram-based and dominant-color-based methods is their inability to allow the localization of an object with the image. A solution to address this problem is to apply color segmentation, which allows both image-to-image matching and object localization. The main problem of color-moment-based methods is their complexity, which makes their application to browsing or other image-retrieval functionalities difficult.
Examples of content-based image and video-retrieval systems are included in Table 3.2. Some or all of the limitations of these systems are the following [3.68]:
~ Few query types are supported
~ Limited set of low-level features
~ Difficult access to visual objects
~ Results partially match user”s expectations
~ Limited interactivity with the user
~ Limited system interoperability
~ Scalability problems
Table 3.2 Examples of content-based image and video-retrieval systems [3.68].
Features System Image/Video Provider
WebSeek I, V Columbia University
Picasso I University of Florence
Color and text
Chabot I University of California, Berkeley
* I University of Toronto
QBIC I IBM
PhotoBook I MIT
Color, texture and shape
BlobWorld I University of California, Berkeley
VIR I, V Virage
Color, shape and scale Nefertiti I National Research Council of Canada
NeTra I University of California, Santa Barbara
Color, texture, shape and
spatial location Digital I Kodak
WebClip V Columbia University
Color, texture and
Jacob I, V University of Palermo
* V IMAX
N/A * V NASA
* No name has been adopted for the corresponding system.
3.4.4 Content-Based Image Retrieval