<p>The Fractal Mosaic algorithm is a rotation, scale, and translation invariant photomosaic algorithm. While traditional photomosaics create a larger image out of smaller square images on a rectangular grid, Fractal Mosaics creates a larger image out of rectangular images that can be placed at arbitrary locations, can be scaled to any size, and can be rotated by any angle. </p>
<p>This is accomplished with a combination of image search (i.e. image retrieval) algorithms and an image registration algorithm developed with NASA funding by George Wolberg Siavash Zokai at the City College of New York (<ahref="http://www.researchgate.net/publication/2830220_Robust_Image_Registration_Using_Log-Polar_Transform/file/d912f51116dedc8a30.pdf">pdf</a>).</p>
Figure 1: Fractal Mosaic created with graffiti image set (best viewed <ahref="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/eye_mosaic_flickr_large.jpg?login=s-ben&token=87b09134a5f66d314547e419299a65b7">large</a></p>
<p>Figure 2 shows three traditional photomosaics created using the same image set and target image. </p>
Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles (middle), and small tiles (right)</p>
<p>As can be seen, using large tiles (left), finer details are lost. Using small tiles (right), we capture target image details, but can’t see individual image detail. Using medium size tiles (middle), we get a compromise between the two. Fractal Mosaics aims to create mosaics that capture both target image details and individual image details, in an aesthetically interesting way.</p>
<p>By allowing for arbitrary placement, rotation, and scaling of the “library images” (images making up the mosaic), there are an many more possible matches to evaluate. This allows for larger, often more aesthetically interesting matches than possible with traditional photomosaics. However, searching such a large solution space on a personal computer presents a challenging problem. This “paper” presents my solution to this problem (two years worth of unpaid work...). It explains the algorithm in depth for those wishing to hack the Matlab code. It should also aid those using the code to create mosaics, as understanding how the algorithm works will help tweak parameters that shape mosaic output.</p>
<h1>Introduction / Problem Statement</h1>
<p>In a traditional photomosaic, the number of possible matches is large but still easily manageable on a modern PC. The number of possible matches is simply the number of “tiles” (square areas in the target image) times the number of library images. For each tile, a similarity metric is calculated (root mean squared (rms) error, typically) for every library image. The library image with the smallest rms error is placed in the mosaic.</p>
<p>In a Fractal Mosaic, the number of possible possible matches is effectively infinite. For example, let’s say we use 5,000 library images, a 3072 x 1024 pixel target image, and do the following fairly rough quantization of the possible translations, rotations, and scalings: </p>
<ul>
<li>2 degree rotations (180 possible rotations)</li>
<li>Every fifth pixel is a possible placement location (629,145 possible translations)</li>
<li>50 possible scaling values</li>
</ul><p>This gives us (180 rotations) x (629,145 translations) x (50 scalings) x (5,000 library images) = 28,311,525,000,000 possible matches to search. Or ~28 trillion matches. Obviously, a brute force search is not feasible (I tried).</p>
<p>Fractal Mosaics tackles this complexity with three main strategies. One strategy is inherent in the “Log-polar” image registration algorithm at the core of Fractal Mosaics. This algorithm uses a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. An in-depth explanation of how log polar registration is implemented in Fractal Mosaics is given in the “log-polar registration” section. </p>
<p>The second main strategy is to use an image search algorithm to filter out library images that are statistically unlikely to produce a match. Details on this in the ‘Image Search (Retrieval) Filtering’ section. </p>
<p>The third main strategy for reducing the number of matches evaluated is to look for large matches first, then not look for smaller matches “under” those matches (i.e., don’t look for matches in areas where we’ve already placed an image). This saves considerable computation. However, it will cause the algorithm to miss aesthetically better matches under the placed matches. For this reason, Fractal Mosaics can be configured to also look for alternate matches under placed images. All matches can optionally be output to PNG files that can be loaded into Photoshop for later compositing. </p>
<p>The following sections will walk you through the main pieces of the Fractal Mosaic algorithm. </p>
<ul>
<li>Image Registration</li>
<li>Image Search (Retrieval)</li>
<li>Mosaic Rendering</li>
<li>Conclusions / Future Directions</li>
</ul><h1>Image Registration</h1>
<p>Image registration is generally defined as the lining up of images misaligned due to rotation, scale, and position (translation). The field of image registration is mature, constituting of several decades of research. The key insight of Fractal Mosaics is that image registration algorithms can be applied to efficiently search the photomosaic solution space. Instead of evaluating every possible rotation, scaling, and translation for every library image (computationally impossible), we can use image registration to efficiently estimate these parameters. Once these parameters are estimated, they are used to rotate, scale, and translate the library image in question to best match the target image. A similarity metric (rms error in this case) is applied. If the rotated, scaled, and translated library image scores high enough on the similarity metric, it is placed in the mosaic. </p>
<h2>Registration Algorithm Choice</h2>
<p>After an initial survey of the image registration literature, a frequency domain image registration algorithm was evaluated; specifically, Fourier-Mellin-based image registration, as implemented in this <ahref="http://www.mathworks.com/matlabcentral/fileexchange/3000-fourier-mellin-based-image-registration-with-gui">Matlab code</a>. This would have been much more computationally efficient than the spatial domain algorithm eventually chosen, as rotation, scale, and translation are efficiently recovered in one step. However, this technique proved not robust enough for the photomosaic application. This is likely because it relies more heavily on the assumption made by all image registration algorithms: that the two images being matched are of the same object/scene. In photomosaics however, we are matching parts of the target image to random library images. The assumption that the library images are rotated, scaled, and translated versions of the target image is weak. </p>
<p>An image registration algorithm that operates in the spatial domain was tried; specifically, the “Log-polar registration” algorithm found in the paper “Robust Image Registration Using Log-Polar Transform” (<ahref="http://www.researchgate.net/publication/2830220_Robust_Image_Registration_Using_Log-Polar_Transform/file/d912f51116dedc8a30.pdf">pdf</a>. This algorithm proved very robust. And most importantly, it met the main aesthetic criteria: for any two images presented to the algorithm, no matter how dissimilar, it aligned them the way a human would. To test this claim out for yourself, check out this <ahref="http://www.youtube.com/watch?v=fw7deJSLVE4">video showing the algorithm at work</a> matching library images to a target image. </p>
<p>The downside to Log-polar registration is increased computation. Because Log-polar registration only recovers rotation and scale, to recover translation, it must be applied to every location (pixel) in the target image. The location resulting in the lowest rms error is then chosen as the estimate. This creates a lot of extra computation. To mitigate this, the authors of the algorithm use a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. The following section details the implementation of Log-polar registration in Fractal Mosaics, which deviates somewhat from the implementation presented in the paper. </p>
<p>In “Robust Image Registration Using Log-Polar Transform”, the authors lay out a “two module” approach. First, a “Log-polar registration” module estimates rotation, scale, and translation. Then, a second module uses a nonlinear least squares optimization method to refine this estimation and obtain sub-pixel accuracy. Since Fractal Mosacis does not require sub-pixel accuracy, and the log polar registration module was found to be accurate enough for this application, just the log-polar registration module is implemented.</p>
<p>When transforming from Cartesian to polar coordinates, each pixel location (x,y) is represented as a radius and angle (r,a), where r is the distance from the center of the image (xc,yc), and a is the angle in degrees from the x-axis.</p>
<p>Figure 3 illustrates the polar transform using an image of Abraham Lincoln. Here we see a crop of the Lincoln image (a), the Lincoln image rotated by 90 deg (c), and the log-polar transforms of both ((b) and (d)).</p>
<p>As can be seen, in the polar space, the 90 degree rotation manifests as a circular shift along the x-axis of the polar image (i.e. the angle axis). If we take the cross-correlation of the polar transformed images ((b) and (d)), the maximum of the cross-correlation gives us the rotation. However, scale cannot be recovered from the cross-correlation.</p>
<h3>Log-polar Transform</h3>
<p>Here’s where we take advantage of some math magic. According to the product rule of logarithms, </p>
<p>This automatic page generator is the easiest way to create beautiful pages for all of your projects. Author your page content here using GitHub Flavored Markdown, select a template crafted by a designer, and publish. After your page is generated, you can check out the new branch:</p>
<pre><code>$ cd your_repo_root/repo_name
$ git fetch origin
$ git checkout gh-pages
</code></pre>
<p>If you're using the GitHub for Mac, simply sync your repository and you'll see the new branch.</p>
<h3>Designer Templates</h3>
<p>We've crafted some handsome templates for you to use. Go ahead and continue to layouts to browse through them. You can easily go back to edit your page before publishing. After publishing your page, you can revisit the page generator and switch to another theme. Your Page content will be preserved if it remained markdown format.</p>
<h3>Rather Drive Stick?</h3>
<p>If you prefer to not use the automatic generator, push a branch named <code>gh-pages</code> to your repository to create a page manually. In addition to supporting regular HTML content, GitHub Pages support Jekyll, a simple, blog aware static site generator written by our own Tom Preston-Werner. Jekyll makes it easy to create site-wide headers and footers without having to copy them across every page. It also offers intelligent blog support and other advanced templating features.</p>
<h3>Authors and Contributors</h3>
<p>You can <ahref="https://github.com/blog/821"class="user-mention">@mention</a> a GitHub username to generate a link to their profile. The resulting <code><a></code> element will link to the contributor's GitHub Profile. For example: In 2007, Chris Wanstrath (<ahref="https://github.com/defunkt"class="user-mention">@defunkt</a>), PJ Hyett (<ahref="https://github.com/pjhyett"class="user-mention">@pjhyett</a>), and Tom Preston-Werner (<ahref="https://github.com/mojombo"class="user-mention">@mojombo</a>) founded GitHub.</p>
<h3>Support or Contact</h3>
<p>Having trouble with Pages? Check out the documentation at <ahref="http://help.github.com/pages">http://help.github.com/pages</a> or contact <ahref="mailto:support@github.com">support@github.com</a> and we’ll help you sort it out.</p>
