The Fractal Mosaic algorithm is a rotation, scale, and translation invariant photomosaic algorithm. While traditional photomosaics create a larger image out of smaller square images on a rectangular grid, Fractal Mosaics creates a larger image out of rectangular images that can be placed at arbitrary locations, can be scaled to any size, and can be rotated by any angle.
+ + +Abstract
+ +The Fractal Mosaic algorithm is a rotation, scale, and translation invariant photomosaic algorithm. While traditional photomosaics create a larger image out of smaller square images on a rectangular grid, Fractal Mosaics creates a larger image out of rectangular images that can be placed at arbitrary locations, can be scaled to any size, and can be rotated by any angle.
This is accomplished with a combination of image search (i.e. image retrieval) algorithms and an image registration algorithm developed with NASA funding by George Wolberg Siavash Zokai at the City College of New York (pdf).
-->Figure 1: Fractal Mosaic created with graffiti image set (best viewed large<-
Figure 2 shows three traditional photomosaics created using the same image set and target image.
+ +
+Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles (middle), and small tiles (right)
As can be seen, using large tiles (left), finer details are lost. Using small tiles (right), we capture target image details, but can’t see individual image detail. Using medium size tiles (middle), we get a compromise between the two. Fractal Mosaics aims to create mosaics that capture both target image details and individual image details, in an aesthetically interesting way.
+ +By allowing for arbitrary placement, rotation, and scaling of the “library images” (images making up the mosaic), there are an many more possible matches to evaluate. This allows for larger, often more aesthetically interesting matches than possible with traditional photomosaics. However, searching such a large solution space on a personal computer presents a challenging problem. This “paper” presents my solution to this problem (two years worth of unpaid work...). It explains the algorithm in depth for those wishing to hack the Matlab code. It should also aid those using the code to create mosaics, as understanding how the algorithm works will help tweak parameters that shape mosaic output.
+ +Introduction / Problem Statement
+ +In a traditional photomosaic, the number of possible matches is large but still easily manageable on a modern PC. The number of possible matches is simply the number of “tiles” (square areas in the target image) times the number of library images. For each tile, a similarity metric is calculated (root mean squared (rms) error, typically) for every library image. The library image with the smallest rms error is placed in the mosaic.
+ +In a Fractal Mosaic, the number of possible possible matches is effectively infinite. For example, let’s say we use 5,000 library images, a 3072 x 1024 pixel target image, and do the following fairly rough quantization of the possible translations, rotations, and scalings:
+ +-
+
- 2 degree rotations (180 possible rotations) +
- Every fifth pixel is a possible placement location (629,145 possible translations) +
- 50 possible scaling values +
This gives us (180 rotations) x (629,145 translations) x (50 scalings) x (5,000 library images) = 28,311,525,000,000 possible matches to search. Or ~28 trillion matches. Obviously, a brute force search is not feasible (I tried).
+ +Fractal Mosaics tackles this complexity with three main strategies. One strategy is inherent in the “Log-polar” image registration algorithm at the core of Fractal Mosaics. This algorithm uses a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. An in-depth explanation of how log polar registration is implemented in Fractal Mosaics is given in the “log-polar registration” section.
+ +The second main strategy is to use an image search algorithm to filter out library images that are statistically unlikely to produce a match. Details on this in the ‘Image Search (Retrieval) Filtering’ section.
+ +The third main strategy for reducing the number of matches evaluated is to look for large matches first, then not look for smaller matches “under” those matches (i.e., don’t look for matches in areas where we’ve already placed an image). This saves considerable computation. However, it will cause the algorithm to miss aesthetically better matches under the placed matches. For this reason, Fractal Mosaics can be configured to also look for alternate matches under placed images. All matches can optionally be output to PNG files that can be loaded into Photoshop for later compositing.
+ +The following sections will walk you through the main pieces of the Fractal Mosaic algorithm.
+ +-
+
- Image Registration +
- Image Search (Retrieval) +
- Mosaic Rendering +
- Conclusions / Future Directions +
Image Registration
+ +Image registration is generally defined as the lining up of images misaligned due to rotation, scale, and position (translation). The field of image registration is mature, constituting of several decades of research. The key insight of Fractal Mosaics is that image registration algorithms can be applied to efficiently search the photomosaic solution space. Instead of evaluating every possible rotation, scaling, and translation for every library image (computationally impossible), we can use image registration to efficiently estimate these parameters. Once these parameters are estimated, they are used to rotate, scale, and translate the library image in question to best match the target image. A similarity metric (rms error in this case) is applied. If the rotated, scaled, and translated library image scores high enough on the similarity metric, it is placed in the mosaic.
+ +Registration Algorithm Choice
+ +After an initial survey of the image registration literature, a frequency domain image registration algorithm was evaluated; specifically, Fourier-Mellin-based image registration, as implemented in this Matlab code. This would have been much more computationally efficient than the spatial domain algorithm eventually chosen, as rotation, scale, and translation are efficiently recovered in one step. However, this technique proved not robust enough for the photomosaic application. This is likely because it relies more heavily on the assumption made by all image registration algorithms: that the two images being matched are of the same object/scene. In photomosaics however, we are matching parts of the target image to random library images. The assumption that the library images are rotated, scaled, and translated versions of the target image is weak.
+ +An image registration algorithm that operates in the spatial domain was tried; specifically, the “Log-polar registration” algorithm found in the paper “Robust Image Registration Using Log-Polar Transform” (pdf. This algorithm proved very robust. And most importantly, it met the main aesthetic criteria: for any two images presented to the algorithm, no matter how dissimilar, it aligned them the way a human would. To test this claim out for yourself, below is a video showing the algorithm at work matching library images to a target image.
+ +Welcome to GitHub Pages.
@@ -62,8 +112,11 @@ $ git checkout gh-pagesHaving trouble with Pages? Check out the documentation at http://help.github.com/pages or contact support@github.com and we’ll help you sort it out.