copilot/fractal-mosaics
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Create gh-pages branch via GitHub

Browse source
This commit is contained in:
s-ben 2013-04-27 11:01:28 -07:00
parent 7f49432bfb
commit af653a9f3f
2 changed files with 76 additions and 17 deletions
Download patch file Download diff file Expand all files Collapse all files

91
index.html
View file

@ -171,32 +171,91 @@ Figure 2: Traditional Photomosaics using large tiles (left), medium sized tiles
<p>For each location (x,y) in the valid pixel area, we perform the following steps:</p>
<h3>Welcome to GitHub Pages.</h3>
<ol>
<li>Crop 8x8 pixel portion of target image around pixel x,y. This is the “target tile” IT. Figure 8 shows an example crop, with the red pixel at x,y.</li>
</ol><p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/Lincoln_8x8_crop_illustrate.png?login=s-ben&amp;token=f402b26d3651f713d3009bd7159b4722" alt="Alt text"></p>
<p>This automatic page generator is the easiest way to create beautiful pages for all of your projects. Author your page content here using GitHub Flavored Markdown, select a template crafted by a designer, and publish. After your page is generated, you can check out the new branch:</p>
<ol>
<li><p>Input target tile IT into image search module. The image search module returns a list of library images most statistically similar to the target tile. Details on this found in the Image Search section. The number of images returned is set via the ‘num_lib_test_pix_v’ parameter, as explained in the Variables page.</p></li>
<li><p>For each library image returned by Step 2:</p></li>
<li><p>Resize library image IL being evaluated to 8x8 pixels</p></li>
<li><p>Take Log-polar transform of target tile IT and library image IL</p></li>
<li><p>Compute cross-correlation of Log-polar transformed IT and IL</p></li>
<li><p>Find maximum of cross-correlation and compute rotation and scale estimate from position of maximum.</p></li>
<li><p>Rotate and scale library image by estimated scale and rotation.</p></li>
<li><p>Compute rms error between scaled and rotated library image and target tile</p></li>
<li><p>If rms error is below a threshold (i.e. good enough match), proceed to next resolution.</p></li>
</ol><p>If the rotated and scaled 8x8 library image produces a low enough rms error, the above steps are repeated at the next resolution up, 16x16. Simply replace ‘8x8’ with ‘16x16’ in step 1 and the steps are identical. </p>
<pre><code>$ cd your_repo_root/repo_name
$ git fetch origin
$ git checkout gh-pages
<p>There are now four 16x16 resolution pixels (locations) “under” our initial 8x8 pixel (x,y) that need to be evaluated. Each 16x16 location is evaluated the same way we evaluated the 8x8 location. Pixel locations that result in rms errors lower than a threshold are saved as “candidate pixels” to be evaluated at the next higher resolution. In this way, we repeat the same steps for each resolution, refining our position estimate at each stage. Figure 9 illustrates this process. The pixel locations in red are the candidate pixels at each resolution As can be seen, in this case, we eventually converge to a single 128x128 resolution pixel location. </p>
<p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/Single_pixel_example_red-cand.png?login=s-ben&amp;token=f0e2b3e020af656ecdcf39836c39142b" alt="Alt text"></p>
<p>Note that for the first three resolutions (8x8, 16x16, 32x32), we let through up to four candidate pixels, if they’re below the rms threshold. At the 64x64 resolution, only one candidate pixel is chosen (the one with the lowest rms error). This is because at lower resolutions (especially 8x8 and 16x16), sometimes the location with the lowest rms error doesn’t correspond to final location estimate. After the 32x32 resolution however, the lowest rms error almost always converges to the correct location.</p>
<h3>Levels</h3>
<p>The steps described in the first two sections will produce a mosaic. However, the scale range of Log-polar registration restricts the size of images in the mosaic. In the paper outlining Log-polar registration (pdf), the authors claim up to 10X scale recovery. In this author’s experiments, scale was only recovered reliably up to 2X. This is due to the fact that we’re not registering images of the same objects/scene, but rather random library images. If we were doing traditional image registration, this would be equivalent to matching images with extreme warping/distortion. Therefore, using the image resolutions presented thus far, the smallest image in the mosaic would be the size of a target tile (1/3 of the mosaic size), scaled down by 1/2, or 1/6th the size of the target image. This means the “resolution” of the overall mosaic is low. We can only have relatively large matches (“tiles”), some of which may not match the target image very well. To create a “higher resolution” mosaic, one where smaller library images fill in target image details, we introduce higher “levels”. </p>
<p>Essentially, at each level we repeat the same algorithm steps, but start with a higher resolution “lowest resolution” target. At level one (L1), we use the 8x8 “lowest resolution” target image, as illustrated in Figure 8. For level 2 (L2), we start with a 16x16 lowest resolution target image, as illustrated in Figure 10. This does not change the resolutions used in Log-polar registration. The 16x16 lowest resolution image is still treated as if it were created from 8x8 tiles. There are just more 8x8 tiles to evaluate now, and those tiles are smaller relative to the actual size of the overall mosaic. Again, we create higher resolution versions of the target image from the lowest resolution image (8x8, 16x16....down to 128x128). At higher levels, the reference target images are simply larger.</p>
<p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/Lincoln_16x16_crop_illustrate.png?login=s-ben&amp;token=112c03e7447796e1060a8f8a121ac105" alt="Alt text"></p>
<p>Figure 11 shows another way to visualize levels. At level 1 (L1), the size of the 8x8 target “tiles” registered are the size of the entire image. At L2, the 8x8 images registered are 1/4th the size of the entire image. At L3, they are 1/16th. At L4, 1/32, etc., as illustrated by the nested squares. In all, there are eight levels. This allows Fractal Mosaics to place very small images, if desired (for more on how to control image size, see the Variables page).</p>
<p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/Lincoln_grid_Levels.png?login=s-ben&amp;token=c2a7912c8f53bc233adbf467f1b7dace" alt="Alt text"></p>
<h3>Error Thresholds</h3>
<p>The error thresholds that determine if a library image gets passed to the next resolution are a function of level and resolution. These were determined empirically. At lower resolutions, the threshold is higher. This is because the rms error can fluctuate significantly between 8x8 and 128x128, and we don’t want to prematurely filter out good matches. All thresholds are calculated relative to a baseline value, stored in the ‘bias_floor’ variable. As explained in the Variables page, this is the main way to control the “resolution” of the overall mosaic. Raise the ‘bias_floor’, and Fractal Mosaics will permit larger matches, which may not match the target image as well. Lowering ‘bias_floor’ will create a mosaic with smaller images that more closely match the target image.</p>
<h1>Image Search (Retrieval) Filtering</h1>
<p>An image search (i.e. retrieval) algorithm is used to filter out statistically unlikely matches before the computationally intensive image registration step (i.e., don’t waste cycles trying to match a white part of the target image with an image of the night sky). Specifically, Fractal Mosaics performs Content-Based Image Retrieval (<a href="http://en.wikipedia.org/wiki/Content-based_image_retrieval">CBIR</a>, similar to Google’s <a href="http://support.google.com/images/answer/1325808?hl=en">search by image</a> feature. The “search image” (the part of the target image we’re looking to match), is compared to every library image using a similarity metric. Then only the most similar library images are evaluated using image registration.</p>
<p>Numerous similarity metrics were evaluated, including histograms, standard deviation, Hu moments, General Fourier Descriptors (GFD), and Fourier Mellin (FM) shape descriptors. The metrics which performed best were histograms, GFDs, and standard deviation. These metrics were combined to create a custom similarity metric that slightly outperformed any individual similarity metric. This metric is currently only used in black and white mode, as I ran out of time to extend it to color. For color, a relatively simple yet effective similarity metric (histogram distance in the HSV color space) is used. Implementation details for black and white and color retrieval are given in the sections below.</p>
<p>Once the similarity metric is evaluated for all library images, the library images are sorted by similarity score. Only the images with the highest similarity scores are sent to the image registration step. The number of library images sent to the image registration step is controlled by the ‘num_lib_test_pix_v’ variable, as explained in the Variables page. </p>
<h2>Black and White</h2>
<p>For black and white mosaics, a custom similarity metric was developed. This metric combines three commonly used similarity metrics: histogram distance, General Fourier Descriptor (GFD) distance, and difference in standard deviation. For a given pixel location, the histogram, GFD, and standard deviation of the “search image”, or target tile (piece of the target image we’re matching) are calculated. These are then compared to the histogram, GFD, and standard deviation of every library image, which have been precomputed. For the histogram and GFD, euclidean distance is computed. For standard deviation, the absolute difference is used. </p>
<p>The histogram distance, GFD distance, and standard deviation difference are then input into respective Probability Density Functions (PDFs), which were calculated empirically using 430,500 pseudo-random samples of the solution space. The sum of the output of these PDFs is the “Match PDF”. </p>
<p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/Match_PDF.gif?login=s-ben&amp;token=43c084e1129c2e6a5fd517bd3410cfa3" alt="equation"></p>
<p>All library images are then sorted by their Match PDF scores, and only the images with the highest probability of being a match are sent to the image registration step.</p>
<h2>Color</h2>
<p>When creating a color mosaic, the similarity metric is the distance between 3D histograms in the HSV color space. The code used to implement this (and implementation details) can be found <a href="http://www.mathworks.com/matlabcentral/fileexchange/22030-image-retrieval-query-by-example-demo">here</a>. Thanks to <a href="http://www.mathworks.com/matlabcentral/fileexchange/authors/30223">Theodoros Giannakopoulos</a> for sharing!</p>
<h1>Mosaic Rendering</h1>
<p>The main Fractal Mosaic script (mosaic_registration....m) renders a version of the mosaic as it goes. This mosaic image is stored in the ‘Render_target’ variable, which is written to ‘fractal_mosaic_eye.jpg’ when the script completes. If the script is interrupted (Ctrll+C), the mosaic “thus far”, can be viewed from the Matlab command line:</p>
<pre><code>imshow(Render_target)
</code></pre>
<p>If you're using the GitHub for Mac, simply sync your repository and you'll see the new branch.</p>
<p>Alternately, the mosaic can be rendered after the fact using the ‘fractal_mosaic_render.m’ script. This gives the user the ability to play with different rendering parameters. Notably, you can set:</p>
<h3>Designer Templates</h3>
<ul>
<li>Whether the images will be placed on top of the target image or on a black background.</li>
<li>How much blending to do between images. Blending is currently done with a simple linear gradient. The user sets a variable, ‘perc_sloped’, which sets the percentage of the image that will be blended. Figure 12 shows a blending window that uses the default, 25%.</li>
</ul><p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/blending_window.png?login=s-ben&amp;token=a0227807701da25bdbebd0b297376baf" alt="Alt text"></p>
<p>We've crafted some handsome templates for you to use. Go ahead and continue to layouts to browse through them. You can easily go back to edit your page before publishing. After publishing your page, you can revisit the page generator and switch to another theme. Your Page content will be preserved if it remained markdown format.</p>
<ul>
<li>Whether to use color images or not. For color mosaics, this is typically a no brainer. However, you can render a black and white mosaic using the color versions of the black and white images used for matching. This can often lead to interesting, accidental results, as illustrated in Figure X.</li>
</ul><p><img src="https://raw.github.com/s-ben/Fractal-Mosaics/gh-pages/images/graffiti_SF_color_render_ex.png?login=s-ben&amp;token=7dd819d3aeae99b287ad3ae610342b0e" alt="Alt text"></p>
<h3>Rather Drive Stick?</h3>
<h1>Conclusion / Next Steps</h1>
<p>If you prefer to not use the automatic generator, push a branch named <code>gh-pages</code> to your repository to create a page manually. In addition to supporting regular HTML content, GitHub Pages support Jekyll, a simple, blog aware static site generator written by our own Tom Preston-Werner. Jekyll makes it easy to create site-wide headers and footers without having to copy them across every page. It also offers intelligent blog support and other advanced templating features.</p>
<p>Considerable effort has been put into automating the entire mosaic creation process, and making beautiful mosaics. However, Fractal Mosaics is not programmed to make artistic or conceptual choices on individual images. Therefore, its most promising use in some use cases will be generating input into a another process (e.g. a human using Photoshop or other software). </p>
<h3>Authors and Contributors</h3>
<p>Considerable effort has been put into optimizing the code, both on the algorithmic and code level. However, it still takes a considerable amount of time to run. Promising optimization strategies include translation from Matlab to another language such as C/C++, accelerating key functions on the GPU (graphics card), and cloud computing (i.e. parallelizing processing across many computers in the cloud). </p>
<p>You can <a href="https://github.com/blog/821" class="user-mention">@mention</a> a GitHub username to generate a link to their profile. The resulting <code>&lt;a&gt;</code> element will link to the contributor's GitHub Profile. For example: In 2007, Chris Wanstrath (<a href="https://github.com/defunkt" class="user-mention">@defunkt</a>), PJ Hyett (<a href="https://github.com/pjhyett" class="user-mention">@pjhyett</a>), and Tom Preston-Werner (<a href="https://github.com/mojombo" class="user-mention">@mojombo</a>) founded GitHub.</p>
<h3>Support or Contact</h3>
<p>Having trouble with Pages? Check out the documentation at <a href="http://help.github.com/pages">http://help.github.com/pages</a> or contact <a href="mailto:support@github.com">support@github.com</a> and we’ll help you sort it out.</p>
<p>Considerable effort has been put into creating art with the code. However, Fractal Mosaics constitutes a new tool. One that exposes many new possibilities. These possibilities, as with any new art form, will take time to flesh out. It is my hope that by putting this code out there, artists will take up this challenge and create amazing art the likes of which the world has never seen before. </p>
</section>
<footer>
<p><small>Hosted on <a href="https://pages.github.com">GitHub Pages</a> using the Dinky theme</small></p>

2
params.json
View file

File diff suppressed because one or more lines are too long