From 29e83c7f1d741d69fb59415fffa233f621f5164c Mon Sep 17 00:00:00 2001 From: Macster Date: Sat, 27 Apr 2013 15:50:27 -0700 Subject: [PATCH] uploading manually formatted index page --- index.html | 62 +++++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 29 deletions(-) diff --git a/index.html b/index.html index 6ad73b1..ef37bf0 100644 --- a/index.html +++ b/index.html @@ -37,26 +37,24 @@

[UNDER CONSTRUCTION]

- -

framework link

- -

Abstract

+

Abstract

The Fractal Mosaic algorithm is a rotation, scale, and translation invariant photomosaic algorithm. While traditional photomosaics create a larger image out of smaller square images on a rectangular grid, Fractal Mosaics creates a larger image out of rectangular images that can be placed at arbitrary locations, can be scaled to any size, and can be rotated by any angle.

This is accomplished with a combination of image search (i.e. image retrieval) algorithms and an image registration algorithm developed with NASA funding by George Wolberg and Siavash Zokai at the City College of New York (pdf).

-

Alt text +

Alt text Figure 1: Fractal Mosaic created with graffiti image set (best viewed large)

-

Figure 2 shows three traditional photomosaics created with AndreaMosaicusing the same image set and target image.

+

For more Fractal Mosaics, check out my Flickr page.

+

Figure 2 shows three traditional photomosaics created with AndreaMosaic using the same image set and target image.

-

Alt text +

Alt text Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles (middle), and small tiles (right)

As can be seen, using large tiles (left), finer details are lost. Using small tiles (right), we capture target image details, but can’t see individual image detail. Using medium size tiles (middle), we get a compromise between the two. Fractal Mosaics aims to create mosaics that capture both target image details and individual image details, in an aesthetically interesting way.

-

Many people have, of course, created photomosaic variants that accomplish this general goal in different ways. Too many to list here. Some of my favorite photomosaic pioneers that I’m aware of: Tsevis, Village9991, and StellaMe. If I’m unaware of your work, contact me!

+

Many people have, of course, created photomosaic variants with similar aims. Too many to list here. Some of my favorite photomosaic pioneers that I’m aware of: Tsevis, Village9991, and StellaMe. If I’m unaware of your work, contact me!

By allowing for arbitrary placement, rotation, and scaling of the “library images” (images making up the mosaic), there are an many more possible matches to evaluate. This allows for larger, often more aesthetically interesting matches than possible with traditional photomosaics. However, searching such a large solution space on a personal computer presents a challenging problem. This “paper” presents my solution to this problem (two years worth of unpaid work...). It explains the algorithm in depth for those wishing to hack the Matlab code. It should also aid those using the code to create mosaics, as understanding how the algorithm works will help tweak parameters that shape mosaic output.

@@ -81,11 +79,11 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

The following sections will walk you through the main pieces of the Fractal Mosaic algorithm.

Image Registration

+
  • Image Registration
    • +
  • Image Search (Retrieval)
    • +
  • Mosaic Rendering
    • +
  • Conclusions / Future Directions
    • +

    Image Registration

    Image registration is generally defined as the lining up of images misaligned due to rotation, scale, and position (translation). The field of image registration is mature, constituting of several decades of research. The key insight of Fractal Mosaics is that image registration algorithms can be employed to efficiently search the photomosaic solution space. Instead of evaluating every possible rotation, scaling, and translation for every library image (computationally impossible), we can use image registration to efficiently estimate these parameters. Once these parameters are estimated, they are used to rotate, scale, and translate the library image in question to best match the target image. A similarity metric (rms error in this case) is applied. If the rotated, scaled, and translated library image scores high enough on the similarity metric, it is placed in the mosaic.

    @@ -93,7 +91,8 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

    After an initial survey of the image registration literature, a frequency domain image registration algorithm was evaluated; specifically, Fourier-Mellin-based image registration, as implemented in this Matlab code. This would have been much more computationally efficient than the spatial domain algorithm eventually chosen, as rotation, scale, and translation are efficiently recovered in one step. However, this technique proved not robust enough for the photomosaic application. This is likely because it relies more heavily on the assumption made by all image registration algorithms: that the two images being matched are of the same object/scene. In photomosaics however, we are matching parts of the target image to random library images. The assumption that the library images are rotated, scaled, and translated versions of the target image is weak.

    -

    An image registration algorithm that operates in the spatial domain was tried; specifically, the “Log-polar registration” algorithm found in the paper “Robust Image Registration Using Log-Polar Transform” (pdf). This algorithm proved very robust. And most importantly, it met the main aesthetic criteria: for any two images presented to the algorithm, no matter how dissimilar, it aligned them the way a human would. To test this claim out for yourself, check out this video showing the algorithm at work matching library images to a target image.

    +

    An image registration algorithm that operates in the spatial domain was tried; specifically, the “Log-polar registration” algorithm found in the paper “Robust Image Registration Using Log-Polar Transform” (pdf). This algorithm proved very robust. And most importantly, it met the main aesthetic criteria: for any two images presented to the algorithm, no matter how dissimilar, it aligned them the way a human would. To test this claim out for yourself, check out the video below showing the algorithm at work matching library images to a target image.

    +

    The downside to Log-polar registration is increased computation. Because Log-polar registration only recovers rotation and scale, to recover translation, it must be applied to every location (pixel) in the target image. The location resulting in the lowest rms error is then chosen as the estimate. This creates a lot of extra computation. To mitigate this, the authors of the algorithm use a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. The following section details the implementation of Log-polar registration in Fractal Mosaics, which deviates somewhat from the original implementation.

    @@ -107,47 +106,50 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

    When transforming from Cartesian to polar coordinates, each pixel location (x,y) is represented as a radius and angle (r,a), where r is the distance from the center of the image (xc,yc), and a is the angle in degrees from the x-axis.

    -

    equation

    +

    equation

    -

    equation

    +

    equation

    Figure 3 illustrates the polar transform using an image of Abraham Lincoln. Here we see a crop of the Lincoln image (a), the Lincoln image rotated by 90 deg (c), and the log-polar transforms of both ((b) and (d)).

    -

    Alt text

    +

    Alt text

    +

    Figure 3: Polar transform of Lincoln image rotated 90°

    As can be seen, in the polar space, the 90 degree rotation manifests as a circular shift along the x-axis of the polar image (i.e. the angle axis). If we take the cross-correlation of the polar transformed images ((b) and (d)), the maximum of the cross-correlation gives us the rotation. However, scale cannot be recovered from the cross-correlation.

    Log-polar Transform

    Here’s where we take advantage of some math magic. According to the product rule of logarithms,

    -

    equation

    +

    equation

    Assuming we scale an image up by a factor of 3, in the polar space, this is equivalent to multiplying the radius values by 3.

    -

    equation

    +

    equation

    If we apply the log function to the radius values (i.e. the Log-polar transform),

    -

    equation

    +

    equation

    Scale now manifests as a scalar phase shift (log(3)) in the radius axis (i.e. x-axis). Rotation still manifests as a phase shift in the angle axis (i.e. y-axis), as it did in the straight polar transform. We can now recover both scale and rotation from the cross-correlation of the log-polar transformed images. Magic!

    Figure 4 shows a crop of the Lincoln image (a), the Lincoln image rotated by 45 deg and scaled up by a factor of 2 (c), and the log-polar transforms of both ((b) and (d)).

    -

    Alt text

    +

    Alt text

    +

    Figure 4: Log-polar transform of Lincoln image rotated 45° and scaled by 2X

    As can be seen, the 45 deg rotation shows up as a circular shift along the x-axis (i.e. angle axis) and the 2X scaling shows up as a phase shift along the y-axis (i.e. angle axis).

    Rotation and Scale Estimation

    To estimate rotation and scale, we take the cross-correlation of the log-polar transformed images ((b) and (d) in Figure 4). The maximum of the cross-correlation (dark red peak) corresponds to rotation and scale (equations for calculating rotation and scale from peak can be found in the code).

    -

    Alt text

    +

    Alt text

    +

    Figure 5: Cross-correlation of the log-polar transformed images ((b) and (d) in Figure 4)

    For computational efficiency, the cross-correlation is computed in the Fourier domain using 2D FFTs,

    -

    equation

    +

    equation

    ,where L1 and L2 are the log-polar transformed library and target images, respectively.

    @@ -157,7 +159,7 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

    To reduce computation, the authors of log-polar registration implement a coarse-to-fine multi-resolution framework, whereby estimates computed in lower resolution images are used as initial guesses in higher resolution images. This framework was tweaked in several places to apply it to Fractal Mosaics, as outlined below in the ‘Image Registration Algorithm Steps’ section.

    -

    Image Search (Retrieval) Filtering

    +

    Image Search (Retrieval) Filtering

    An image search (i.e. retrieval) algorithm is used to filter out statistically unlikely matches before the computationally intensive image registration step (i.e., don’t waste cycles trying to match a white part of the target image with an image of the night sky). Specifically, Fractal Mosaics performs Content-Based Image Retrieval (CBIR, similar to Google’s search by image feature. The “search image” (the part of the target image we’re looking to match), is compared to every library image using a similarity metric. Then only the most similar library images are evaluated using image registration.

    @@ -171,7 +173,7 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

    The histogram distance, GFD distance, and standard deviation difference are then input into respective Probability Density Functions (PDFs), which were calculated empirically using 430,500 pseudo-random samples of the solution space. The sum of the output of these PDFs is the “Match PDF”.

    -

    equation

    +

    equation

    All library images are then sorted by their Match PDF scores, and only the images with the highest probability of being a match are sent to the image registration step.

    @@ -179,7 +181,7 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

    When creating a color mosaic, the similarity metric is the distance between 3D histograms in the HSV color space. The code used to implement this (and implementation details) can be found here. Thanks to Theodoros Giannakopoulos for sharing!

    -

    Mosaic Rendering

    +

    Mosaic Rendering

    The main Fractal Mosaic script (mosaic_registration....m) renders a version of the mosaic as it goes. This mosaic image is stored in the ‘Render_target’ variable, which is written to ‘fractal_mosaic_eye.jpg’ when the script completes. If the script is interrupted (Ctrll+C), the mosaic “thus far”, can be viewed from the Matlab command line:

    @@ -190,14 +192,16 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles

    Alt text

    +

    Figure 6: Default blending window (25% of image edge blended)

    Alt text

    +

    Figure 7: Example of using color images when rendering a black and white mosaic

    -

    Conclusion / Next Steps

    +

    Conclusion / Next Steps

    Considerable effort has been put into automating the entire mosaic creation process, and making beautiful mosaics. However, Fractal Mosaics is not programmed to make artistic or conceptual choices on individual images. Therefore, its most promising use in some use cases will be generating input into a another process (e.g. a human using Photoshop or other software).