Mozilla

content aware image resizing

Note: The author of the demo, Stéphane Roucheray, is a member of the PIMS team. The demo was first posted on the Pims World Labs weblog.

View the demo in Firefox 3.5.

Content Aware Image Resizing is a way to re-target an image size without modifying its content ratio, in other words : non-linear image resizing. The algorithm was first explained by Shai Avidan and Ariel Shamir and published in 2007 (“Seam Carving for Content-Aware Image Resizing.”)

Since then several excellent Open Source implementations have been released. For example, there is a plugin for The Gimp and CAIR a standalone application in C++.

But thanks to Canvas and JavaScript it’s now possible to do this in the browser without a plugin.

Since version 1.5, Firefox offers bitmap manipulation through the Canvas API. Version 3.5 introduced not only the fastest Firefox JavaScript engine ever, but also a new Canvas method – createImageData – providing a much more powerful environment.

For this demo, a sub part of the Content Aware Image Resizing algorithm has been implemented. The width of the image can be reduced interactively without modifying its height. This implementation uses seam carving to re-size the image, subtracting the less visible vertical lines. It is a four step iterative algorithm. One iteration is one pixel width re-size. First an image is loaded into the Canvas context and then the iteration starts :

  1. A grayscale version of the image has to be calculated
  2. The edges of the image (Sobel convolution is used in our case) and its energy matrix is computed
  3. The seam of least energy (1 pixel vertical line from the bottom to the top of the energy matrix) is detected
  4. Then the pixel of the detected seam is removed from the original image and the result is re-injected as a source image to step 1

Each of the previous steps stores a whole matrix of data at the source image size. While these matrices are not all images but actually artifacts of the algorithm, storing them in an ImageData object is more convenient than using simple Arrays. This is why the createImageData method of the Canvas context is used. One of the benefits of this process is to allow showing the intermediate computations made under the hood.

This demo shows that’s possible to do more intelligent image resizing than just flattening an image’s pixels with CSS. Having computational and image manipulation capabilities directly in the browser opens up a new range of possibilities of how image data can be displayed to users. This is only one small demonstration of that.

36 comments

Comments are now closed.

  1. Markus wrote on June 8th, 2009 at 09:15:

    Quite impressive. But is it really that slow or did you just make the demo slower to visualize the algorithm?

  2. Paul wrote on June 8th, 2009 at 09:24:

    @Markus The first pass is slow. But after that, it’s really smooth (try to resize the image via the handle).

  3. sroucheray wrote on June 8th, 2009 at 09:40:

    @markus : I’ve noticed different rendering speeds on different platforms. However, the algorithm makes a lot of computation and furthermore is iterative. It means the image “width – 2px” cannot be computed before the image “width – 1px” has been computed. That’s some of the reason it seems slow.
    To improve performance it is worse trying to uncheck “Refresh images during process”. Displaying intermediates images results in lost of time for computing. The algorithm tries to compute as fast as it can and stores the intermediate results. So you can play with it while process.

  4. John Drinkwater wrote on June 8th, 2009 at 09:57:

    Probably something another demo could use to show off web workers ;)

  5. Fred wrote on June 8th, 2009 at 13:43:

    Not sure how much web workers would help there, except for not blocking JS execution while they are running. For multiple workers to lead to a big speedup, I would expect the algorithm to be required to be parallelizable.

  6. Boris wrote on June 8th, 2009 at 13:48:

    Is there a reason to use keydown, not keypress, for the arrow key thing? The latter should work better cross-browser for holding the key down…

  7. Pingback from 颠覆网络35天 ─ 内容感知的图片大小调整 < MJiA on June 9th, 2009 at 02:45:

    [...] 原文地址:http://hacks.mozilla.org/2009/06/content-aware-image-resizing/ [...]

  8. Pingback from Le Meilleur du Peer » content aware image resizing in javascript on June 9th, 2009 at 02:50:

    [...] A Javascript implementation of the content aware image resizing algorithm » (d’autres détails ici) Classé dans (X)HTML, CSS & Cie, Liens, Mini-Posts [...]

  9. Pingback from Content Aware Image Resizing @ Mozilla Hacks | Lambda on June 9th, 2009 at 05:21:

    [...] The post is here http://hacks.mozilla.org/2009/06/content-aware-image-resizing/ [...]

  10. Pingback from Ajaxian » Animating SVG with Canvas and Burst on June 11th, 2009 at 05:57:

    [...] content aware image resizing [...]

  11. Pjdkrunkt wrote on June 11th, 2009 at 17:48:

    I don’t get it. How is driving my hard-drive into a fit for 10 minutes to produce strange distortions in parts of the image an improvement in the web? The initial codebase of searching the image for “inactive” regions should just be used to CROP the image. Most images fall into one of three catagories… left or right heavy, center heavy, top or bottom heavy. It seems like it would be rediculously faster to scan an image, figure out which it is, and simply crop to effect. Maybe their could even be a tag included in the image description to specify which to use, which would not only cut down on all this crazy computation, but allow the designer to make sure the wrong thing doesn’t get cropped accidentally.

    I’m not saying this to just be cross, as a member of the photographic community you would be surprised how upset photographers will be when they find out their web-browser is destroying the integrity of their images. Typical image resizing by squishing may be crude and even ugly… but it is at least honest… and someone can always “resquish” an image to see how it basically originally looked. This “feature” goes so far down the path of corrupting the integrity of the original image, it’s just about criminal. Not to say that the programmers haven’t done an excellent job, it’s a really sophisticated piece of programming… but there’s a moral issue to consider here. Just because you CAN do advanced level Photoshop editing right in the browser doesn’t mean you should.

    Consider the multiple cases last year alone of journalists caught in the midst of scandals for Photoshopping images. And especially for artists and professional art photographers… the “dead space” being stolen out of these images isn’t just dead space. It’s called negative space and it’s often as and sometimes MORE crucial than the apparent subject in conveying the meaning of the image. The groups of people sitting on the steps is a great example… in the initial photo, there is plenty of space between the people and the photo is friendly and inviting. After being “resized” it’s a cramped claustrophobic atmosphere. The supposedly same image now has a completely different feeling. Cropping can be just as dangerous to image integrity, but at least it’s considered a normal publishing practice.

  12. Nickolay wrote on June 12th, 2009 at 04:27:

    Pjdkrunkt: Firefox doesn’t do this transformation automatically. It’s just an example of what you can program in JavaScript thanks to the improved JS speed and canvas.

  13. Alex wrote on June 12th, 2009 at 05:05:

    Having a CSS property to define method to use when resizing image could be useful :
    .img {
    resize-policy: liquid;
    resize-policy: linear;
    resize-policy: crop;
    }

  14. Jep wrote on June 17th, 2009 at 06:00:

    TOO BAD ;)
    There’s a new STATE-OF-THE-ART algoritm now!
    ( http://www.vimeo.com/5024379 )

  15. Pingback from Monthly Interesting Links Roundup (June 2009) on June 26th, 2009 at 00:23:

    [...] things that can be done by Firefox 3.5 (all native, without plugins) Content Aware Image Resizing http://hacks.mozilla.org/2009/06/content-aware-image-resizing/ Blink detection http://ajaxian.com/archives/finally-a-useful-blink-tag-detecting-your-user-blinking [...]

  16. Pingback from Techknology’s Blog » Firefox 3.5 Will Arrive June 30 on June 26th, 2009 at 17:23:

    [...] 3.5’s new features and even has some examples of how developers are taking advantage of the new tools to push the limits of the web. Share and [...]

  17. Pingback from Design Industries Blog » Blog Archive » Firefox 3.5 Will Arrive June 30 on June 27th, 2009 at 21:27:

    [...] 3.5’s new features and even has some examples of how developers are taking advantage of the new tools to push the limits of the web. Posted in IT / Tech, News, Facts & Inspiration | Leave a [...]

  18. Pingback from Firefox 3.5 Will Arrive June 30 | Tech Invaders on June 28th, 2009 at 11:56:

    [...] If you’d like more information about Firefox 3.5 and what’s new, check out our links to previous coverage listed below, or head over to Mozilla’s Chris Blizzard’s blog. Blizzard is almost through with his 35 days project in which he highlights Firefox 3.5’s new features and even has some examples of how developers are taking advantage of the new tools to push the limits of the web. [...]

  19. Pingback from Firefox 3.5 to be released June 30 | BlogoFlux - Information Technology Blog on June 28th, 2009 at 12:08:

    [...] has been showing different 3.5 features within a 35 day period, and has also shown the new tools develops can take advantage of in the new [...]

  20. Pingback from ParsanIT » امروز، مرورگر فایرفاکس ۳/۵ منتشر می شود |،مقالات مفید کامپیوتر و این on June 29th, 2009 at 21:40:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده [...]

  21. Pingback from Firefox 3.5 on June 30th, 2009 at 09:33:

    [...] perioada de 35 de zile diferite functii si caracteristici ale versiunii 3.5 si pe langa aceastea si noile instrumente de care pot beneficia developerii in noua [...]

  22. Pingback from Mozilla Firefox 3.5 Released | BlogoFlux - Information Technology Blog on June 30th, 2009 at 10:46:

    [...] has been showing different 3.5 features within a 35 day period, and has also shown the new tools developers can take advantage of in the new [...]

  23. Pingback from !!! فایر فاکس 3.5 هم آمد | تشکل سیاسی - دانشجویی هرمان on June 30th, 2009 at 23:56:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده [...]

  24. Pingback from Shayan Amiri WebSite » Blog Archive » دانلود نسخه 3.5 مرورگر فایرفاکس رسید on July 1st, 2009 at 01:22:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده [...]

  25. Trackback from LaBoX on July 1st, 2009 at 07:01:

    The Untold What’s New in Firefox 3.5…

    Firefox 3.5 is now out and Mozilla Advances the Web again, this time being the first to release a new final browser with support for more web standards, upcoming standards, and some very useful HTML5 support!

    So What’s New in Firefox 3.5…

  26. Pingback from مرورگر فایرفاکس ۳/۵ (نسخه نهایی ) on July 1st, 2009 at 12:42:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده است. همچنین با وجود آنکه نسخه [...]

  27. Pingback from نسخه نهایی مرورگر فایرفاکس ۳/۵ | پاکباز دات آی آر |*_*| PAKBAZ.IR on July 2nd, 2009 at 01:25:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده [...]

  28. Pingback from Holnap érkezik a Firefox 3.5 | wordpress on July 3rd, 2009 at 00:25:

    [...] Tavaly novemberben még 250 felett volt a megjelenést akadályozónak minősített hibák száma, ezt az utóbbi hónapokban sikerült lefaragni, ám többek között az első Release Candidate változat is a megmaradó, makacs bugok miatt késett több hetet. A pre-RC1, majd az RC1 felbukkanása után azonban sebességet váltottak, rövid időben belül pedig még további három RC-változat jelent meg a világhálón. Az utóbbi 35 napban a Mozilla fokozatosan mutatta be az új opciókat, ezzel párhuzamosan pedig a fejlesztőknek szánt eszközöket is ismertették. [...]

  29. Pingback from نسخه نهایی مرورگر فایرفاکس ۳/۵ جهت دانلود منتشر شد on July 3rd, 2009 at 09:42:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده [...]

  30. Pingback from نسخه نهایی مرورگر فایرفاکس ۳٫۵ جهت دانلود منتشر شد | شبكه اطلاع رساني هاديران on July 5th, 2009 at 23:08:

    [...] این نسخه جدید را در وب سایت خود معرفی کرده بود. همچنین ابزارهای جدیدی برای توسعه دهندگان و برنامه نویسان معرفی شده [...]

  31. Pingback from 谋智社区 » Blog Archives » 颠覆网络35天 ─ 内容感知的图片大小调整 on July 6th, 2009 at 05:49:

    [...] 原文地址:http://hacks.mozilla.org/2009/06/content-aware-image-resizing/ [...]

  32. Pingback from Zmiana rozmiaru zdjęcia z uwzględnieniem zawartości – stream of bytes on July 20th, 2009 at 13:56:

    [...] tekst jest tłumaczeniem artykułu “Content aware image resizing“, zamieszczonego na stronie hacks.mozilla.org na licencji Creative Commons Attribution 3.0 [...]

  33. design wrote on September 24th, 2009 at 06:31:

    Urgh, no, liquid resize (which is what this is) is terrible for day-to-day resizing. It has value from a content production point of view, but it introduces so many artifacts and weirdnesses to images that Id never want my browser to do it automatically to any image I view.

  34. kourge wrote on September 24th, 2009 at 13:25:

    @Alex: image-rendering is the CSS property you’re looking for. I believe the IE equivalent is -ms-interpolation-mode.

  35. ivo wrote on February 24th, 2010 at 06:53:

    Nice tool, but as a webdeveloper you don’t know anymore what your user is seeing.

  36. Max wrote on February 15th, 2012 at 14:54:

    There is another easy-to-use content aware resizing software iResizer:
    http://www.iresizer.com

Comments are closed for this article.