Mozilla

open video codecs and quality

This is a re-post (with permission) of a post that Greg Maxwell wrote in response to a comment by Chris DiBona from Google on a whatwg mailing list. The codecs being discussed are the same ones we’ll be including in Firefox 3.5 and are also the same codecs that Mozilla, Wikipedia and others have been investing in.

Recent developer nightlies of Google Chrome support these codecs and a future version of Opera will also support them. Theora and Vorbis also work in Safari if you install the Xiph Qt component. We’re quickly reaching the point where all modern browsers support these open codecs with full support for the video tag.

You’ll note that Greg’s post doesn’t have the tone of a marketing document – it’s not meant to. Nor is this a comparison against HD-sized, high-bitrate video. Instead it’s an attempt to give an honest comparison of how the open codecs fare against commonly-used formats and sizes used on the world’s largest video site. I think you’ll agree with Greg’s conclusions at the bottom of the document, especially with audio where Vorbis really shines.

Greg’s post follows.

Purpose

On Jun 13th 2009 Chris DiBona of Google made a remarkable claim on the WhatWG mailing list:

“If [youtube] were to switch to theora and maintain even a semblance of the current youtube quality it would take up most available bandwidth across the Internet.”

Unfortunately, open video formats have been subjected to FUD so frequently that people are willing to believe bold claims like these without demanding substantiation.

In this comparison I will demonstrate that this claim was unfair and unreasonable. Using a simple test case I show that Theora is competitive and even superior to some of the files that Google is distributing today on YouTube.

Theora isn’t the most efficient video codec available right now. But it is by no means bad, and it is substantially better than many other widely used options. By conventional criteria Theora is competitive. It also has the substantial advantage of being unencumbered, reasonable in computational complexity, and entirely open source. People are often confused by the correct observation that Theora doesn’t provide the state of the art in bitrate vs quality, and take that to mean that Theora does poorly when in reality it does quite well. Also, the Theora encoder has improved a lot lately so some older problems no longer apply.

While different files may produce different results, the allegation made on WhatWG was so expansive that I believe a simple comparison can reliably demonstrate its falsehood.

I do not believe Chris intended to deceive anyone, only that he is a victim of the same outdated and/or simply inaccurate information that has fooled many others. Automotive enthusiasts may make a big deal about a 5 horsepower difference between two cars, but these kinds of raw performance differences are not relevant to most car buyers nor are they even the most important criteria to people who race. Likewise, videophiles nitpick the quality of compression formats and this nitpicking is important for the advancement of the art. But I believe that people are mistaking these kinds of small differences for something which is relevant to their own codec selection.

Results

A 499kbit/sec H.264+AAC output and a 327kbit/sec H.263(Sorensen Spark)+MP3 output were available via the download service. The YouTube-encoded files are available on the YouTube site. Because the files on YouTube may change and the web player does not disclose the underlying bitrate, I have made the two encoded files available.

~499kbit/sec comparison

YouTube

Download (H.264+AAC; 17MiB)

Ogg/Theora+Vorbis

Download / Watch (Ogg/Theora+Vorbis; 17MiB)

~327kbit/sec comparison

YouTube

Download (H.263+MP3; 12MiB)

Ogg/Theora+Vorbis

Download / Watch (Ogg/Theora+Vorbis; 12MiB)

A slightly lower bitrate was used for the Theora+Vorbis test cases to avoid any question of quality improvement resulting from larger outputs.
For a fair comparison you must compare the audio as well. Even without audio differences, still image comparisons are a poor proxy for video quality.
I provided this random frame still image comparison only because I expect that people will not bother watching the examples without evidence that the results are interesting.

Methodology

In order to avoid any possible bias in the selection of H.264 encoders and encoding options, and to maximize the relevance for this particular issue, I’ve used YouTube itself as the H.264 encoder. This is less than ideal because YouTube does not accept lossless input, but it does accept arbitrarily high bitrate inputs.

I utilized the Blender Foundation’s Big Buck Bunny as my test case because of its clear licensing status, because it’s a real world test case, and because I have it available in a lossless format. I am not aware of any reason why this particular clip would favor either Theora or H.264.

I chose to use a test case with a soundtrack because most real usage has sound. No one implements HTML5 video without audio, and no one is implementing either of Theora or Vorbis without the other. Vorbis’s state-of-the-art performance is a contributor to the overall Ogg/Theora+Vorbis solution.

  • Obtain the lossless 640×360 Big Buck Bunny source PNGs and FLACs from media.xiph.org.
  • Resample the images to 480×270 using ImageMagick’s convert utility.
  • Use gstreamer’s jpegenc, produce a quality=100 mjpeg + PCM audio stream. The result is around 1.5Gbytes with a bitrate of around 20Mbit/sec.
  • Truncate the file to fit under the YouTube 1Gbyte limit, resulting in input_mjpeg.avi (706MiB).
  • Upload the file to YouTube and wait for it to transcode.
  • Download the FLV and H.264 files produced by YouTube using one of the many web downloading services. (I used keepvid)
  • Using libtheora 1.1a2 and Vorbis aoTuv 5.7 produce a file of comparable bitrate to the youtube 499kbit/sec from the same file uploaded to YouTube (input_mjpeg.avi).
  • Resample the file uploaded to YouTube to 400×226.
  • Using libtheora 1.1a2 and Vorbis aoTuv 5.7 produce a file of comparable bitrate to the youtube 327kbit/sec from the 400×226 downsampled copy of input_mjpeg.avi.

I later discovered that YouTube sometimes offers additional sizes. I tried the youtube-dl utility and it indicated that these other sizes were not available for my file. Otherwise I would have also included them in this comparison.

A keyframe interval of 250 frames was used for the Theora encoding. The theora 1.1a2 encoder software used is available from theora.org. The Vorbis encoder used is available from the aoTuV website. No software modifications were performed.

My conclusions

It can be difficult to compare video at low bitrates, and even YouTube’s higher bitrate option is not high enough to achieve good quality. The primary challenge is that all files at these rates will have problems, so the reviewer is often forced to decide which of two entirely distinct flaws is worse. Sometimes people come to different conclusions.

That said, I believe that the Theora+Vorbis results are substantially better than the YouTube 327kbit/sec. Several other people have expressed the same view to me, and I expect you’ll also reach the same conclusion. This is unsurprising since we’ve been telling people that Theora is better than H.263, especially at lower bitrates, for some time now and YouTube only uses a subset of H.263.

The low bitrate case is also helped by Vorbis’ considerable superiority over MP3. For example, the crickets at the beginning are inaudible in the low rate YouTube clip but sound fine in the Ogg/Theora+Vorbis version.

In the case of the 499kbit/sec H.264 I believe that under careful comparison many people would prefer the H.264 video. However, the difference is not especially great. I expect that most casual users would be unlikely to express a preference or complain about quality if one was substituted for another and I’ve had several people perform a casual comparison of the files and express indifference. Since Theora+Vorbis is providing such comparable results, I think I can confidently state that reports of the internet’s impending demise are greatly exaggerated.

Of course, YouTube may be using an inferior processing chain, or encoding options which trade off quality for some other desirable characteristic (like better seeking granularity, encoding speed, or a specific rate control pattern). But even if they are, we can conclude that adopting an an open unencumbered format in addition to or instead of their current offerings would not cause problems on the basis of quality or bitrate.

But please— see and hear for yourself.

Greg Maxwell

27 comments

Comments are now closed.

  1. Pingback from candrews's status on Monday, 15-Jun-09 14:40:37 UTC - Identi.ca on June 15th, 2009 at 07:40:

    […] comparison of the Ogg family to H.263 used by Youtube now. http://ur1.ca/5pre I wish Google would support Open […]

  2. Question wrote on June 15th, 2009 at 08:43:

    I don;t know whether Youtube is using 1-pass or 2-pass. If the video encoded in 2-pass @ High Profile Level5.1, I think the result may be slightly (?) different.

  3. Christopher Blizzard wrote on June 15th, 2009 at 08:54:

    @Question – It’s not a question of how to squeeze every little bit out of the encoders. Totally not the point. It’s a comparison of how youtube happens to encode today and how our encoders compare to what people expect to see for a specific amount of bandwidth. Speculating on how youtube might improve their encodings is fine, but it’s not what we care about right now. They encode how they encode.

  4. CHL wrote on June 15th, 2009 at 09:16:

    I think what Google is claiming is that, since they already have all their videos stored in the H.263/H.264 (or whatever-that-isn’t-Theora) format, reencoding will introduce even more compression artifacts, and the only effective way to reduce these artifacts is to reencode into Theora with an unrealistically high bitrate.

  5. Christopher Blizzard wrote on June 15th, 2009 at 11:08:

    @CHL That’s not what Google is claiming at all. They are saying that to use and deliver Theora would require encoding videos at a much higher bitrate than the equivalent H.263 and H.264. And this demo shows that’s not true – it’s about the same.

  6. Maik Merten wrote on June 15th, 2009 at 12:22:

    Some people like to point out that YouTube most likely doesn’t use “best encoding options” (extra-high-quality settings or 2-pass encoding). However, there are good reasons for this: YouTube has to encode massive amounts of content every minute, so they need fast encoders and can’t afford to tweak output to maximum quality (and e.g. doing 2- or 3-pass encoding ~doubles/~triples encoding time – H.264 encoding *is* very processing-intensive). It’s all about the economics of delivering “good enough” quality at affordable expenses. It appears Theora can deliver a nice quality/computation ratio, too, given that Theora has less computational complexity (so you don’t have to drop encoding quality to get affordable speed).

    This doesn’t mean H.264 as a compression format has been questioned as leader in achievable compression performance, it simply points out that in the economics of online streaming other factors play a significant role, too.

  7. Michael Kozakewich wrote on June 15th, 2009 at 12:34:

    It looks amazing.
    I’ve recently seen some post somewhere showing how far Theora has come in the last little while. There were two theora screenshots, and the newer one was tremendously better.
    I can see great things in its future.

  8. Pingback from Comparación de cálidad entre Ogg/Theora+Vorbis y Youtube h264 : Blogografia on June 15th, 2009 at 16:00:

    […] en si este nuevo estandar ofrecerá la misma cálidad que los formatos existentes. De ahí que via hacks.mozilla.org descubra una comparativa [Traducción parcial] de el nuevo formato Ogg Theora+Vorbis(MP3), un estandar abierto y disponible […]

  9. Olly wrote on June 15th, 2009 at 16:40:

    This is all wonderful stuff.

    Let’s say I’ve just bought myself a shiny new MacBook. I record something on my camcorder, import it to iMovie, do some editing and now I want to export it in Theora+Vorbis format to put on my website. How would I go about doing this? Remember, I don’t even know what the Terminal is :)

    Secondly, my mates all have Windows PCs and iPhones. Is there a way I can embed these videos on the page such that they can see them? Is it a good idea to export it to H263/4 aswell so Flash Player / Quicktime / iPhones can view it too?

    (Is all this coming in a future hacks.mozilla article?)

  10. Christopher Blizzard wrote on June 15th, 2009 at 19:22:

    @Olly – those are all good questions. For exporting to ogg/theora, you can use the xiph quicktime components:

    http://www.xiph.org/quicktime/

    Version 0.19 was just released which fixed some problems with iMovie 08. Other operating systems sometimes need video in other formats (especially mobile) but there’s a good overview here:

    http://reports.graymattergravy.com/2009/06/02/video-tag-embed-with-fallbacks/

  11. Pingback from 颠覆网络35天 ─ 开放视频的编码和质量 < MJiA on June 15th, 2009 at 23:09:

    […] 原文地址:open video codecs and quality 系列地址:颠覆网络35天 […]

  12. dave wrote on June 16th, 2009 at 03:30:

    I’m glad to see the quality issue being addressed so publicly, but I’m wary of allowing the debate to be framed in this manner. Theora and Vorbis are more than “good enough”, with Vorbis being class leading as far as I’m aware (being able to challenge both AAC and AAC+, at their respective bests) but the reason for choosing them is because they enable the open web.

    According to Adobe I can’t even watch their Flash-delivered H.264 videos unless it is for my “own non-commercial” purposes.

    http://labs.adobe.com/wiki/index.php/Flash_Player:9:Update:H.264#Q:_What_parts_of_the_H.264_license_are_included_when_I_buy_Adobe_Products.3F

    Ben Waggoner pointed out on Slashdot that Youtube’s 12 minute limit on video length is a license related issue. Why should we accept such arbitrary limits on web video. If Google can’t afford to pay it how is Wikipedia or your local University going to?

    http://tech.slashdot.org/comments.pl?sid=1268327&cid=28329643

    There’s an ongoing debate about codec choice for HTML5 yet i can’t find a single readable explanation of what I can and can’t do with H.264/AAC, as a consumer or a producer, and how much it would cost me. Everything surrounding H.264 licensing assumes you are either a large commercial TV station or Video-on-Demand producer or you’re a passive consumer of TV and movies. That alone is enough to put it directly at odds with the ethos of the open web.

    What if I want to put videos of cats with humorous subtitles on the web? Or videos of lectures? At the moment it appears that I may have to pay up to $10,000 dollars annually after next year but as I say the issue is so clouded it’s not actually clear if that’s true.

  13. Asa Dotzler wrote on June 16th, 2009 at 12:19:

    Olly, I’ve written up a simple tutorial that should help you out a lot. See https://air.mozilla.com/exporting-to-open-video/

  14. Jacob wrote on June 16th, 2009 at 16:07:

    I don’t exactly see why youtube videos absolutely have to be encoded on their server. It could significantly reduce their required processing power and provide a better quality result if they had some sort of applet that encoded the videos client-side on operating systems that supported it, and just do the old-fashioned server side encoding on operating systems that didnt. Heck, if they did that, it would also be a smart decision for them to provide an option to let us encode it with multiple passes and even Theora if we wanted. Am I missing something or would it be a reasonable solution to implement client-side encoding?

  15. Pingback from an update on open video codecs and quality at hacks.mozilla.org on June 16th, 2009 at 21:25:

    […] days ago we posted a comparison by Greg Maxwell of low and medium resolution YouTube videos vs. Theora counterparts at the same bit rates. The […]

  16. Natanael L wrote on June 17th, 2009 at 00:57:

    Dave: “Ben Waggoner pointed out on Slashdot that Youtube’s 12 minute limit on video length is a license related issue. Why should we accept such arbitrary limits on web video. If Google can’t afford to pay it how is Wikipedia or your local University going to?”
    LOL! So that’s why they’ve got *that* limit? Well, too bad for them then. But I’ll keep watching the videos using Gnash and stuff anyway.

    Jacob: Well, that depends on the video. Not everybody has Java installed, but for the people who di it could detect it and ask if they want to re-encode it locally or let YouTube do it.
    I assume that most people won’t do it themselves unless there’s a “HIGHLY RECOMMENDED!”-option as default option, with a text underneath that “too much quality” would be lost otherwise if everything had to be encoded by the server. And that option must be a reasonable option.
    Such as OGG Theora/Vorbis because of the license and the blog article I’m commenting on.

  17. Christopher Blizzard wrote on June 17th, 2009 at 05:46:

    @Jacob – if you go look at firefogg.org you can see an extension for Firefox that you can use to transcode videos on the client. Additionally web sites can ask your computer to do the transcoding and then uploading the resulting .ogv file. It’s actually pretty neat.

  18. Pingback from Ajaxian » Open Codecs, Open Video on June 17th, 2009 at 06:24:

    […] Maxwell posting about his experience with the quality of Ogg Theora with real world examples, and Chris Blizzard linked it up with 35 days offering the following lead in: The codecs being discussed are the same ones we’ll be including […]

  19. Pingback from Andres Chavez Ulfe » Comparación de cálidad entre Ogg/Theora+Vorbis y Youtube h264 on June 17th, 2009 at 15:11:

    […] en si este nuevo estandar ofrecerá la misma cálidad que los formatos existentes. De ahí que via hacks.mozilla.org descubra una comparativa [Traducción parcial] de el nuevo formatoOgg Theora+Vorbis(MP3), un estandar abierto y disponible […]

  20. Pingback from Open Codecs, Open Video | rapid-DEV.net on June 18th, 2009 at 00:05:

    […] Maxwell posting about his experience with the quality of Ogg Theora with real world examples, and Chris Blizzard linked it up with 35 days offering the following lead in: The codecs being discussed are the same ones we’ll be including […]

  21. Pingback from Open Codecs, Open Video | Guilda Blog on June 18th, 2009 at 01:26:

    […] Maxwell posting about his experience with the quality of Ogg Theora with real world examples, and Chris Blizzard linked it up with 35 days offering the following lead in: The codecs being discussed are the same ones we’ll be including […]

  22. Pingback from Open Codecs, Open Video on June 18th, 2009 at 15:34:

    […] Maxwell posting about his experience with the quality of Ogg Theora with real world examples, and Chris Blizzard linked it up with 35 days offering the following lead in: The codecs being discussed are the same ones we’ll be including […]

  23. moe wrote on June 24th, 2009 at 07:28:

    I’m watching the videos, side by side at normal zoom.

    There are some very subtle but noticeable differences in these test cases that I think place Theora/Vorbis better than H.263/MP3. It seems to have better color at times or a much sharper contrast. Also, some really high detailed areas tend to be sharper where they were blurry on the H.263. Another key difference i picked up on was the gradients in the sky: H.263 had some serious “color banding/ macro blocking” where theora was a nice smooth color transformation. The 1.1a2 really cleans up theora’s “Short falls”

    Basically, if one day all YouTube videos were suddenly in the Theora/Vorbis format, I think the general consensus among viewers would be “not different or slightly better” in all or most cases. Theora is great candidate for open web video.

  24. jon wrote on June 27th, 2009 at 12:01:

    of course you realize that Theora is based on On2 VP3, correct?

    http://en.wikipedia.org/wiki/Theora

    do you realize that VP8 is soon to be debuted?
    just asking, since ON2 has not been mentioned here at all, and I find that strange to say the least.

  25. Pingback from 谋智社区 » Blog Archives » 颠覆网络35天 ─ 开放视频的编码和质量 on July 21st, 2009 at 23:57:

    […] 原文地址:open video codecs and quality 系列地址:颠覆网络35天 […]

  26. Trackback from Digital Blog India.in on January 10th, 2010 at 17:43:

    Firefox 3.5 supports open video formats…

    The new Firefox 3.5 now supports open video formats. What does this mean ? Firefox now supports Theora, which is a truly free and open video format for the internet. This means that we would have to do away with proprietary plugins or restricted codecs…

  27. albinoz wrote on February 5th, 2010 at 01:56:

    Really ?

    H263, is a old codec, vs the Ogg, it can not be compared.
    Test the same same with h264 or x264, you will see like the bigger size, the color respect + quality will be better…

    The Ogg is not HD ready…
    The Vorbis is almost same quality like mp4, but really better than mp3.

    So, using Ogg today, is too late.
    Now the avc made better and this is not comparated to H263 what will change my opinion, but make me laught.

    It’s a good codec, but we will not go back…

Comments are closed for this article.