[UNDER CONSTRUCTION]

Abstract

The Fractal Mosaic algorithm is a rotation, scale, and translation invariant photomosaic algorithm. While traditional photomosaics create a larger image out of smaller square images on a rectangular grid, Fractal Mosaics creates a larger image out of rectangular images that can be placed at arbitrary locations, can be scaled to any size, and can be rotated by any angle.

This is accomplished with a combination of image search (i.e. image retrieval) algorithms and an image registration algorithm developed with NASA funding by George Wolberg Siavash Zokai at the City College of New York (pdf).

Figure 1: Fractal Mosaic created with graffiti image set (best viewed large

Figure 2 shows three traditional photomosaics created using the same image set and target image.

Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles (middle), and small tiles (right)

As can be seen, using large tiles (left), finer details are lost. Using small tiles (right), we capture target image details, but can’t see individual image detail. Using medium size tiles (middle), we get a compromise between the two. Fractal Mosaics aims to create mosaics that capture both target image details and individual image details, in an aesthetically interesting way.

By allowing for arbitrary placement, rotation, and scaling of the “library images” (images making up the mosaic), there are an many more possible matches to evaluate. This allows for larger, often more aesthetically interesting matches than possible with traditional photomosaics. However, searching such a large solution space on a personal computer presents a challenging problem. This “paper” presents my solution to this problem (two years worth of unpaid work...). It explains the algorithm in depth for those wishing to hack the Matlab code. It should also aid those using the code to create mosaics, as understanding how the algorithm works will help tweak parameters that shape mosaic output.

Introduction / Problem Statement

In a traditional photomosaic, the number of possible matches is large but still easily manageable on a modern PC. The number of possible matches is simply the number of “tiles” (square areas in the target image) times the number of library images. For each tile, a similarity metric is calculated (root mean squared (rms) error, typically) for every library image. The library image with the smallest rms error is placed in the mosaic.

In a Fractal Mosaic, the number of possible possible matches is effectively infinite. For example, let’s say we use 5,000 library images, a 3072 x 1024 pixel target image, and do the following fairly rough quantization of the possible translations, rotations, and scalings:

• 2 degree rotations (180 possible rotations)
• Every fifth pixel is a possible placement location (629,145 possible translations)
• 50 possible scaling values

This gives us (180 rotations) x (629,145 translations) x (50 scalings) x (5,000 library images) = 28,311,525,000,000 possible matches to search. Or ~28 trillion matches. Obviously, a brute force search is not feasible (I tried).

Fractal Mosaics tackles this complexity with three main strategies. One strategy is inherent in the “Log-polar” image registration algorithm at the core of Fractal Mosaics. This algorithm uses a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. An in-depth explanation of how log polar registration is implemented in Fractal Mosaics is given in the “log-polar registration” section.

The second main strategy is to use an image search algorithm to filter out library images that are statistically unlikely to produce a match. Details on this in the ‘Image Search (Retrieval) Filtering’ section.

The third main strategy for reducing the number of matches evaluated is to look for large matches first, then not look for smaller matches “under” those matches (i.e., don’t look for matches in areas where we’ve already placed an image). This saves considerable computation. However, it will cause the algorithm to miss aesthetically better matches under the placed matches. For this reason, Fractal Mosaics can be configured to also look for alternate matches under placed images. All matches can optionally be output to PNG files that can be loaded into Photoshop for later compositing.

The following sections will walk you through the main pieces of the Fractal Mosaic algorithm.

• Image Registration
• Image Search (Retrieval)
• Mosaic Rendering
• Conclusions / Future Directions

Image Registration

Image registration is generally defined as the lining up of images misaligned due to rotation, scale, and position (translation). The field of image registration is mature, constituting of several decades of research. The key insight of Fractal Mosaics is that image registration algorithms can be applied to efficiently search the photomosaic solution space. Instead of evaluating every possible rotation, scaling, and translation for every library image (computationally impossible), we can use image registration to efficiently estimate these parameters. Once these parameters are estimated, they are used to rotate, scale, and translate the library image in question to best match the target image. A similarity metric (rms error in this case) is applied. If the rotated, scaled, and translated library image scores high enough on the similarity metric, it is placed in the mosaic.

Registration Algorithm Choice

After an initial survey of the image registration literature, a frequency domain image registration algorithm was evaluated; specifically, Fourier-Mellin-based image registration, as implemented in this Matlab code. This would have been much more computationally efficient than the spatial domain algorithm eventually chosen, as rotation, scale, and translation are efficiently recovered in one step. However, this technique proved not robust enough for the photomosaic application. This is likely because it relies more heavily on the assumption made by all image registration algorithms: that the two images being matched are of the same object/scene. In photomosaics however, we are matching parts of the target image to random library images. The assumption that the library images are rotated, scaled, and translated versions of the target image is weak.

An image registration algorithm that operates in the spatial domain was tried; specifically, the “Log-polar registration” algorithm found in the paper “Robust Image Registration Using Log-Polar Transform” (pdf. This algorithm proved very robust. And most importantly, it met the main aesthetic criteria: for any two images presented to the algorithm, no matter how dissimilar, it aligned them the way a human would. To test this claim out for yourself, check out this video showing the algorithm at work matching library images to a target image.

The downside to Log-polar registration is increased computation. Because Log-polar registration only recovers rotation and scale, to recover translation, it must be applied to every location (pixel) in the target image. The location resulting in the lowest rms error is then chosen as the estimate. This creates a lot of extra computation. To mitigate this, the authors of the algorithm use a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. The following section details the implementation of Log-polar registration in Fractal Mosaics, which deviates somewhat from the implementation presented in the paper.

Log Polar Image Registration

In “Robust Image Registration Using Log-Polar Transform”, the authors lay out a “two module” approach. First, a “Log-polar registration” module estimates rotation, scale, and translation. Then, a second module uses a nonlinear least squares optimization method to refine this estimation and obtain sub-pixel accuracy. Since Fractal Mosacis does not require sub-pixel accuracy, and the log polar registration module was found to be accurate enough for this application, just the log-polar registration module is implemented.

Polar Coordinates

To understand Log-polar registration, we briefly review polar coordinates.

When transforming from Cartesian to polar coordinates, each pixel location (x,y) is represented as a radius and angle (r,a), where r is the distance from the center of the image (xc,yc), and a is the angle in degrees from the x-axis.

Figure 3 illustrates the polar transform using an image of Abraham Lincoln. Here we see a crop of the Lincoln image (a), the Lincoln image rotated by 90 deg (c), and the log-polar transforms of both ((b) and (d)).

As can be seen, in the polar space, the 90 degree rotation manifests as a circular shift along the x-axis of the polar image (i.e. the angle axis). If we take the cross-correlation of the polar transformed images ((b) and (d)), the maximum of the cross-correlation gives us the rotation. However, scale cannot be recovered from the cross-correlation.

Log-polar Transform

Here’s where we take advantage of some math magic. According to the product rule of logarithms,

Assuming we scale an image up by a factor of 3, in the polar space, this is equivalent to multiplying the radius values by 3.

If we apply the log function to the radius values (i.e. the log-polar transform),

Scale now manifests as a scalar phase shift (log(3)) in the radius axis (i.e. x-axis). Rotation still manifests as a phase shift in the angle axis (i.e. y-axis), as it did in the straight polar transform Section. We can now recover both scale and rotation from the cross-correlation of the log-polar transformed images! Magic!

Figure 4 shows a crop of the Lincoln image (a), the Lincoln image rotated by 45 deg and scaled up by a factor of 2 (c), and the log-polar transforms of both ((b) and (d)).

As can be seen, the 45 deg rotation shows up as a circular shift along the x-axis (i.e. angle axis) and the 2X scaling shows up as a phase shift along the y-axis (i.e. angle axis).

Rotation and Scale Estimation

To estimate rotation and scale, we take the cross-correlation of the log-polar transformed images ((b) and (d) in Figure 4). The maximum of the cross-correlation (dark red peak) corresponds to rotation and scale (equations for calculating rotation and scale from peak can be found in the code).

For computational efficiency, the cross-correlation is computed in the Fourier domain using 2D FFTs,

where L1 and L2 are the log-polar transformed library and target images, respectively.

Recovering Translation

As mentioned previously, Log-polar registration does not recover translation. It assumes the two images presented to it line up, except for rotation and scale differences. To recover translation, the algorithm must be applied to every location (pixel). The translation resulting in the best match (i.e. lowest rms error) is then chosen as the translation estimate. This makes log-polar registration fairly computationally expensive compared to other registration techniques.

To reduce computation, the authors of log-polar registration implement a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. This framework was tweaked in several places to apply it to Fractal Mosaics, as outlined below in the ‘Image Registration Algorithm Steps’ section.

Image Registration Algorithm Steps

1. Create low resolution versions of target image

For each “level” (levels are explained shortly), we require five different resolutions of the target image.

First, we create the lowest resolution version of the target image. Because 8x8 images were found (in this author’s experiments), to be the lowest resolution images capable of producing meaningful rotation and scale estimates, the lowest resolution target image is composed of 8x8 images, or “tiles”. For example, Figure 6 shows a target image of Abraham Lincoln at full resolution (3072 x 2606 pixels), and an 8x8 resolution version (24x20 pixels).

Note that the low resolution version is three 8x8 “tiles” high. In other words, the largest possible images in the final mosaic will be approximately 1/3 of the mosaic height. This is a reasonable assumption, as we’re very unlikely to find a random library image that exactly matches more than a third of the target image.

Next, we create the intervening resolutions. These resolutions are powers of two derived from the 8x8 image, up to 128x128. I.e., we have an 8x8 resolution image, as shown in Figure 6, a 16x16 resolution image which is four times the size of the 8x8 resolution image, a 32x32 resolution image which is four times the size of the 8x8 image, and so on, down to 128x128. 128x128 is the highest resolution we perform image registration on because higher resolutions do not produce meaningful refinements of scale and rotation and are therefore a waste of computation. The original library images (used to render the final mosaic) are 256x256 pixels.

Note that while black and white images are used in this example, the same math applies to color. The same steps are simply applied to each color channel (R,G,B) in parallel.

1. Perform Log-polar registration on each pixel

Since Fractal Mosaics doesn’t perform “wrapping” of pixels on the boundary of the target image, we only evaluate pixels in the “valid pixel area” (red box in Figure 7).

For each location (x,y) in the valid pixel area, we perform the following steps:

Welcome to GitHub Pages.

This automatic page generator is the easiest way to create beautiful pages for all of your projects. Author your page content here using GitHub Flavored Markdown, select a template crafted by a designer, and publish. After your page is generated, you can check out the new branch:

$ cd your_repo_root/repo_name
$ git fetch origin
$ git checkout gh-pages

If you're using the GitHub for Mac, simply sync your repository and you'll see the new branch.

Designer Templates

We've crafted some handsome templates for you to use. Go ahead and continue to layouts to browse through them. You can easily go back to edit your page before publishing. After publishing your page, you can revisit the page generator and switch to another theme. Your Page content will be preserved if it remained markdown format.

Rather Drive Stick?

If you prefer to not use the automatic generator, push a branch named gh-pages to your repository to create a page manually. In addition to supporting regular HTML content, GitHub Pages support Jekyll, a simple, blog aware static site generator written by our own Tom Preston-Werner. Jekyll makes it easy to create site-wide headers and footers without having to copy them across every page. It also offers intelligent blog support and other advanced templating features.

Authors and Contributors

You can @mention a GitHub username to generate a link to their profile. The resulting <a> element will link to the contributor's GitHub Profile. For example: In 2007, Chris Wanstrath (@defunkt), PJ Hyett (@pjhyett), and Tom Preston-Werner (@mojombo) founded GitHub.

Support or Contact

Having trouble with Pages? Check out the documentation at http://help.github.com/pages or contact support@github.com and we’ll help you sort it out.