Figure 1: Fractal Mosaic created with graffiti image set (best viewed large)
Figure 1: Fractal Mosaic created with graffiti image set (best viewed large)
For more Fractal Mosaics, check out my Flickr page.
-Figure 2 shows three traditional photomosaics created with AndreaMosaic using the same image set and target image.
+For more Fractal Mosaics, check out my Flickr page.
+Figure 2 shows three traditional photomosaics created with AndreaMosaic using the same image set and target image.
Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles (middle), and small tiles (right)
As can be seen, using large tiles (left), finer details are lost. Using small tiles (right), we capture target image details, but can’t see individual image detail. Using medium size tiles (middle), we get a compromise between the two. Fractal Mosaics aims to create mosaics that capture both target image details and individual image details, in an aesthetically interesting way.
-Many people have, of course, created photomosaic variants with similar aims. Too many to list here. Some of my favorite photomosaic pioneers that I’m aware of: Tsevis, Village9991, and StellaMe. If I’m unaware of your work, contact me!
+Many people have, of course, created photomosaic variants with similar aims. Too many to list here. Some of my favorite photomosaic pioneers that I’m aware of: Tsevis, Village9991, and StellaMe. If I’m unaware of your work, contact me!
By allowing for arbitrary placement, rotation, and scaling of the “library images” (images making up the mosaic), there are an many more possible matches to evaluate. This allows for larger, often more aesthetically interesting matches than possible with traditional photomosaics. However, searching such a large solution space on a personal computer presents a challenging problem. This “paper” presents my solution to this problem (two years worth of unpaid work...). It explains the algorithm in depth for those wishing to hack the Matlab code. It should also aid those using the code to create mosaics, as understanding how the algorithm works will help tweak parameters that shape mosaic output.
@@ -89,10 +89,10 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tilesAfter an initial survey of the image registration literature, a frequency domain image registration algorithm was evaluated; specifically, Fourier-Mellin-based image registration, as implemented in this Matlab code. This would have been much more computationally efficient than the spatial domain algorithm eventually chosen, as rotation, scale, and translation are efficiently recovered in one step. However, this technique proved not robust enough for the photomosaic application. This is likely because it relies more heavily on the assumption made by all image registration algorithms: that the two images being matched are of the same object/scene. In photomosaics however, we are matching parts of the target image to random library images. The assumption that the library images are rotated, scaled, and translated versions of the target image is weak.
+After an initial survey of the image registration literature, a frequency domain image registration algorithm was evaluated; specifically, Fourier-Mellin-based image registration, as implemented in this Matlab code. This would have been much more computationally efficient than the spatial domain algorithm eventually chosen, as rotation, scale, and translation are efficiently recovered in one step. However, this technique proved not robust enough for the photomosaic application. This is likely because it relies more heavily on the assumption made by all image registration algorithms: that the two images being matched are of the same object/scene. In photomosaics however, we are matching parts of the target image to random library images. The assumption that the library images are rotated, scaled, and translated versions of the target image is weak.
An image registration algorithm that operates in the spatial domain was tried; specifically, the “Log-polar registration” algorithm found in the paper “Robust Image Registration Using Log-Polar Transform” (pdf). This algorithm proved very robust. And most importantly, it met the main aesthetic criteria: for any two images presented to the algorithm, no matter how dissimilar, it aligned them the way a human would. To test this claim out for yourself, check out the video below showing the algorithm at work matching library images to a target image.
-
The downside to Log-polar registration is increased computation. Because Log-polar registration only recovers rotation and scale, to recover translation, it must be applied to every location (pixel) in the target image. The location resulting in the lowest rms error is then chosen as the estimate. This creates a lot of extra computation. To mitigate this, the authors of the algorithm use a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. The following section details the implementation of Log-polar registration in Fractal Mosaics, which deviates somewhat from the original implementation.
@@ -102,7 +102,7 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tilesTo understand Log-polar registration, we briefly review polar coordinates.
+To understand Log-polar registration, we briefly review polar coordinates.
When transforming from Cartesian to polar coordinates, each pixel location (x,y) is represented as a radius and angle (r,a), where r is the distance from the center of the image (xc,yc), and a is the angle in degrees from the x-axis.
@@ -158,10 +158,12 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tilesAs mentioned previously, Log-polar registration does not recover translation. It assumes the two images presented to it line up, except for rotation and scale differences. To recover translation, the algorithm must be applied to every location (pixel). The translation resulting in the best match (i.e. lowest rms error) is then chosen as the translation estimate. This makes log-polar registration fairly computationally expensive compared to other registration techniques.
To reduce computation, the authors of log-polar registration implement a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. This framework was tweaked in several places to apply it to Fractal Mosaics, as outlined below in the ‘Image Registration Algorithm Steps’ section.
+
An image search (i.e. retrieval) algorithm is used to filter out statistically unlikely matches before the computationally intensive image registration step (i.e., don’t waste cycles trying to match a white part of the target image with an image of the night sky). Specifically, Fractal Mosaics performs Content-Based Image Retrieval (CBIR, similar to Google’s search by image feature. The “search image” (the part of the target image we’re looking to match), is compared to every library image using a similarity metric. Then only the most similar library images are evaluated using image registration.
+An image search (i.e. retrieval) algorithm is used to filter out statistically unlikely matches before the computationally intensive image registration step (i.e., don’t waste cycles trying to match a white part of the target image with an image of the night sky). Specifically, Fractal Mosaics performs Content-Based Image Retrieval (CBIR, similar to Google’s search by image feature. The “search image” (the part of the target image we’re looking to match), is compared to every library image using a similarity metric. Then only the most similar library images are evaluated using image registration.
Numerous similarity metrics were evaluated, including histograms, standard deviation, Hu moments, General Fourier Descriptors (GFD), and Fourier Mellin (FM) shape descriptors. The metrics which performed best were histograms, GFDs, and standard deviation. These metrics were combined to create a custom similarity metric that slightly outperformed any individual similarity metric. This metric is currently only used in black and white mode, as I ran out of time to extend it to color. For color, a relatively simple yet effective similarity metric (histogram distance in the HSV color space) is used. Implementation details for black and white and color retrieval are given in the sections below.
@@ -179,7 +181,7 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tilesWhen creating a color mosaic, the similarity metric is the distance between 3D histograms in the HSV color space. The code used to implement this (and implementation details) can be found here. Thanks to Theodoros Giannakopoulos for sharing!
+When creating a color mosaic, the similarity metric is the distance between 3D histograms in the HSV color space. The code used to implement this (and implementation details) can be found here. Thanks to Theodoros Giannakopoulos for sharing!
