hacks.mozilla.org

Less than a year after the release of Theora 1.0, the wonderful people at Xiph have released Theora 1.1. The 1.1 release is a software-only release of the Theora encoder and decoder. It does not include any changes to the Theora format. Existing Theora videos should continue to play with the new decoder and the new Theora encoder generates bitstreams that will work in existing players that can play Theora content.

The 1.1 release is largely an improvement to the Theora encoder. This post will attempt to give people a high-level overview of the changes and what they mean to web developers and people who are thinking of deploying Theora to support HTML5 video. Theora is an important technology to web developers – it’s the only competitive codec that currently complies with the W3C patent policy.

Here’s a quick list of important things that have changed in this release. We’ll go into more detail on each of these items.

• Video quality between Theora 1.0 and Theora 1.1 has been improved.
• Rate control for live streaming now works well.
• A two-pass mode has been added to the encoder that can create rate controlled videos with very predictable bandwidth requirements.
• CPU usage during encoding is much more consistent.
• Decoder performance has been improved.

Video quality between Theora 1.0 and Theora 1.1 has been improved.

One issue that people had with the Theora 1.0 encoder was that it produced video that appeared fuzzy. The 1.1 improvements are clear in these two images provided by Monty, one of the Xiph Developers. Open each of these images in new tabs and flip between them. You can really see the difference.

This was also very visible at the edges of text. Here’s an example taken from one of our Firefox 3.5 promotional videos. The first is with the 1.0 encoder (9.0MB) and the second is with the 1.1 encoder (8.2MB). You will notice that not only are the edges more defined but there’s a lot less noise in the area around the edges of the text. Once again, if you open them in tabs and flip between them you can see the difference.

Note that the original video is nearly 17MB. That was done largely to get the text crisp. With these changes we can likely use a much lower-bandwidth version of the video, probably as small as 9.9MB. That’s a pretty big difference.

Note that we’re talking about an improvement of quality at the same video bitrate. This means that we’re either able to produce higher quality videos at the same file size or we’re able to reduce the file size and keep the same quality – either way it’s a big win.

Rate control for live streaming now works well.

Before describing this change, something important must be described. This is the difference between videos encoded with a variable bitrate (VBR) and a constant bitrate (CBR).

In variable bitrate encoding the amount of data that’s required to represent the difference between two frames in a video is allowed to grow. This happens most often when shifting from a scene where there isn’t much movement to a scene where there’s a lot of motion. You could easily go from requiring 40Kb/sec to 400Kb/sec because the entire background moves.

In constant bitrate encoding the amount of data that you’re allowed to use to represent a change from one frame to the next is pinned at some maximum value. If you’ve got a low maximum value and there’s a set of frames that requires a lot of bits to represent the changes from one to the next you will need to sacrifice something in order to stay inside of that maximum value. Very often it’s some amount of video quality or the encoder will start dropping frames in order to keep under the watermark.

This leads to a pretty simple rule: If you want the highest quality video possible, you should be using variable rate encoding. This means that when you’re encoding a video you should be using quality settings (0-99, low/medium/high, 1-10) instead of picking bitrates (60Kb/sec, 200Kb/sec.) For most use cases on the web VBR-encoded videos actually work very well because users are allowed to buffer quite a bit of video out ahead of their current position so these bursts of data don’t affect the user’s experience.

But there are some use cases where having a constant bitrate is very important. These include:

1. Live, low delay streaming over HTTP with a lot of clients.
2. Streaming large files where a large read-ahead buffer is not desired.
3. Situations where large bursts of data result in large bursts in CPU to handle them.

For live, low-delay streaming over HTTP it’s important to realize what happens when there’s a sudden burst of data to handle. HTTP runs over TCP. In TCP it takes a while for a connection to increase its bandwidth. (And by “a while” I mean “not that long” but it’s long enough to affect the low latency connection that we want for this use case. This is why many low-latency applications don’t use TCP. But we’re talking about delivering video over HTTP.) If you’ve got a big burst of data and the TCP window takes a long time to open up you start building up a big send buffer on the server. (And remember in this use case you’ve got a lot of clients connected!) That requires a lot of memory to hold the send buffers for each client. What happens then is that servers will start closing connections en masse because it needs to save memory or because it thinks that the client has become somehow unreachable. This is made worse by the fact that even if the connection scales up and then scales back down it re-settles at the low rate and the process has to be repeated. The user’s experience is that the video stream stops and restarts or just stops working altogether when the server hangs up. The solution? Using a constant rate that doesn’t require the TCP window to open up suddenly and doesn’t require large send buffers for each client.

For the use case where you’re streaming large files it might not be reasonable for the client to cache a lot of data. You also might be serving up a lot of data to a lot of clients and you might want to avoid the large send buffer problem as well, just for different reasons.

And for the last use case where you’re in a CPU-constrained environment the bursting nature of variable bitrate videos means it often takes a large bursts of CPU to handle those bursts. While CPUs do burst up faster than TCP does, you might be talking to constrained processors (think mobile) or you might be serving up files near HD-sized content, which CPUs often struggle to decode.

In any case there are a number of use cases for constant bitrate encoding. Back to the question of what’s improved in Theora 1.1.

In Theora 1.0 the rate controlled encoding mode was very very broken. This resulted in two things:

1. People trying to do live streaming ran into problems.
2. People who used rate controlled settings to compare overall Theora quality to the quality of other encoders saw worse results than the format actually represented.

The first issue is clear – it was broken, it should be fixed. And it has been. The new encoder does a pretty good job of maintaining bitrates, changes quality on the fly, drops frames and even includes a “soft-target” mode so that bitrates can fluctuate a little bit to maintain quality while occasionally breaking the bandwidth rules.

The encoder also has a wonderful new piece of functionality that people will find very useful. It’s now possible to specify a maximum rate ceiling for video encoding while also specifying a minimum quality floor. What this means is that the encoder will try and maintain very crisp video frames within rate constraints. This means that it will aggressively drop frames instead of creating frame deltas that are fuzzy or low-quality. While this might sound like a poor trade-off it’s actually very useful. If you’re showing a live video of a presentation you usually want a crisp video of the slides and having a lower frame update rate is very acceptable.

The second issue that was caused by the bad rate control in Theora 1.0 is an issue of marketing. People would often use the encoder with the fixed bitrate mode instead of the quality mode and dismiss the results as a reflection of the format instead of problems with the encoder. We hope that people find better results with the new encoder.

A two-pass mode has been added to the encoder that can create rate controlled videos with very predictable bandwidth requirements.

In addition to fixing the single pass rate controlled encoder in 1.1 a two-pass encoding option has been added. This means that if you are transcoding a file (as opposed to doing a live stream) you can create a very consistent bitrate in a file if you want. This is because the encoder can look ahead in the stream to allocate bits efficiently. Monty from Xiph made a graph that shows one example of the bitrate in a file with one pass and two pass.

Above: graph of quantizer choice (which roughly corresponds to the encoding quality) when using default two-pass bitrate management as opposed to one-pass (with –soft-target) when encoding [the Matrix movie clip] at 300kbps. Both encodes resulted in the same bitrate. The quality of a one-pass encode varies considerably as the encoder has no way to plan ahead.

CPU usage during encoding is much more consistent.

People who were doing live streaming often saw huge spikes in CPU usage during high-motion events. This has been fixed and now CPU usage is much more consistent during single pass rate constrained encoding making it much easier to live stream video.

Decoder performance has been improved.

And last but not least the decoder has been made faster during the 1.1 release. How much faster depends quite a bit on the clip, but people are reporting that the new encoder is anywhere from 1.5-2x faster than the 1.0 of release of libtheora.

Coming soon to a product near you.

This release is a library release. It’s not a product in itself, but is instead something that other products include. So over the next days and weeks we’ll see other products pick up and start using this as part of their releases.

Enjoy!

The BBC has a post up describing an experiment that they have put together that uses HTML5 video that works in Firefox 3.5 and Safari.

The demo uses jQuery and drives a simple carousel that shows the current chapter as the video plays. It also shows subtitles as the video plays.

One really great, and somewhat subtle thing about this experiment? You can select the subtitles. They are just text.

On Tuesday, July 28th, 2009, Mozilla hosted a brownbag and lunch discussion with Davis Freeberg and Dan Miller on the subject of open video codecs. Dan Miller is one of the founders of On2 and is largely responsible for the free release of VP3, which is the basis for the Theora codec. Davis Freeberg hosts a popular blog in which he comments extensively on the state of video on the Internet.

Recently Arun Ranganathan, one of the members of the Mozilla Evangelism team, created a video for MozCamp Mumbai. It’s about 20 minutes long and he covers a huge number of topics: the new @font-face CSS property and how it affects the ability for people to receive properly localized content, the differences between the various standards efforts (there’s more than just HTML5) and gives some demos that show what’s possible when you combine video with the web.

This article is written by Paul Rouget, Mozilla contributor and purveyor of extraordinary Open Web demos.

Starting with Firefox 3.5, you can embed a video in a web page like an image. This means video is now a part of the document, and finally, a first class citizen of the Open Web. Like all other elements, you can use it with CSS and JavaScript. Let’s see what this all means …

The Basics

First, you need a video to play. Firefox supports the Theora codec (see here to know all media formats supported by the audio and video elements).

Add the video to your document:

<video id="myVideo" src="myFile.ogv"/>

You might need to add some “fallback” code if the browser doesn’t support the video tag. Just include some HTML (which could be a warning, or even some Flash) inside the video tag.

<video id="myVideo" src="myFile.ogv">
<strong>Your browser is not awesome enough!</strong>
</video>

Here’s some more information about the fallback mechanism.

HTML Attributes

You can find all the available attributes here.

Some important attributes:

• autoplay: The video will be played just after the page loads.
• autobuffer: By default (without this attribute), the video file is not downloaded unless you click on the play button. Adding this attribute starts downloading the video just after the page loads.
• controls: by default (without this attribute), the video doesn’t include any controls (play/pause button, volume, etc.). Use this attribute if you want the default controls.
• height/width: The size of the video

Example:

<video id="myVideo" src="myFile.ogv" 
   autobuffer="true" controls="true"/>

You don’t have to add the “true” value to some of these attributes in HTML5, but it’s neater to do so. If you’re not in an XML document, you can simply write:

<video id="myVideo" src="myFile.ogv" autobuffer controls/>

JavaScript API

Like any other HTML element, you have access to the video element via the Document Object Model (DOM):

var myVideo = document.getElementById("myVideo");

Once you obtain a handle to the video element, you can use the JavaScript API for video.

Here is a short list of some useful methods and properties (and see here for more of the DOM API for audio and video elements):

• play() / pause(): Play and pause your video.
• currentTime: The current playback time, in seconds. You can change this to seek.
• duration: The duration of the video.
• muted: Is the sound muted?
• ended: Has the video ended?
• paused: Is the video paused?
• volume: To determine the volume, and to change it.

Example:

<button onclick="myVideo.play()">Play</button>
<button onclick="myVideo.volume = 0.5">Set Volume</button>
<button onclick="alert(myVideo.volume)">Volume?</button>

Events

You know how to control a video (play/pause, seek, change the volume, etc.). You have almost everything you need to create your own controls. But you need some feedback from the video, and for that, let’s see the different events you can listen to:

• canplay: The video is ready to play
• canplaythrough: The video is ready to play without interruption (if the download rate doesn’t change)
• load: The video is ready to play without interruption (the video has been downloaded entirely)
• ended: The video just ended
• play: The video just started playing
• pause: The video has been paused
• seeking: The video is seeking (it can take some seconds)
• seeked: The seeking process just finished
• timeupdate: While the video is playing, the currentTime is updated. Every time the currentTime is updated, timeupdate is fired.

Here’s a full list of events.

For example, you can follow the percentage of the video that has just been played:

function init() 
{
  var video = document.getElementById("myVideo");
  var textbox = document.getElementById("sometext");
  video.addEventListener("timeupdate", function() {
  textbox.value = Math.round(100 * (video.currentTime / video.duration)) + "%"; }
 
}

<video id="myVideo" src="myFile.ogv" 
            autoplay="true" onplay="init()"/>
<input id="sometext"/>

Showing all this in action, here’s a nice open video player using the Video API.

Now that you’re familiar with some of the broad concepts behind the Video API, let’s really delve into the video as a part of the Open Web, introducing video to CSS, SVG, and Canvas.

CSS and SVG

A video element is an HTML element. That means you can use CSS to style it.

A simple example: using the CSS Image Border rule (a new CSS 3 feature introduced in Firefox 3.5). You can view how it works on the Mozilla Developer Wiki.

And obviously, you can use it with the video tag:

 
<video id="myVideo" src="myFile.ogv" 
style="-moz-border-image: 
           url(tv-border.jpg) 25 31 37 31 stretch stretch; 
           border-width: 20px;"/>

One of my demos uses this very trick.

Since Firefox 3.5 provides some new snazzy new CSS features, you can do some really fantastic things. Take a look at the infamous washing machine demo, in which I subject an esteemed colleague to some rotation.

It uses some CSS rules:

• filter, clip-path.
• css transform.

And some SVG:

• SVG & foreignobject.

Because the video element is like any other HTML element, you can add some HTML content over the video itself, like I do in this demo. As you can see, there is a <div> element on top of the video (position: absolute;).

Time for a Break

Well, we’ve just seen how far we can go with the video element, both how to control it and how to style it. That’s great, and it’s powerful. I strongly encourage you to read about the new web features available in Firefox 3.5, and to think about what you can do with such features and the video element.

You can do so much with the power of the Open Web. You can compute the pixels of the video. You can, for example, try to find some shapes in the video, follow the shapes, and draw something as an attachment to these shapes. That’s what I do here! Let’s see how it actually works.

Canvas & Video

Another HTML 5 element is canvas. With this element, you can draw bitmap data (see the canvas reference, and I strongly suggest this canvas overview). But something you might not know is that you can copy the content of an <img/> element, a <canvas/> element and a <video/> element.

That’s a really important point for the video element. It gives you a way to play with the values of the pixels of the video frames.

You can do a “screenshot” of the current frame of the video in a canvas.

function screenshot() {
 var video = document.getElementById("myVideo");
 var canvas = document.getElementById("myCanvas");
 var ctx = canvas.getContext("2d");
 
 ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
}

<video id="myVideo" src="myFile.ogv" autoplay="true" with="600" height="400"/>
<canvas id="myCanvas" with="600" height="400"/>
<button onclick="screenshot()">Copy current frame to canvas</button>

You can first apply a transformation to your canvas (see the documentation). You can also copy a thumbnail of the video.

If you draw every frame in a canvas, your canvas will look like a video element. And you can draw what you want in this canvas, after drawing the frame. That’s what I do in this demo.

Once you have a video frame in your canvas, you can compute the values of the pixels.

Some things you should know if you want to compute the pixels values of a frame:

• you can’t use this mechanism with a video from another domain.
• you can’t use this mechanism with a video from a file:/// URL (which would be useful during the development of your web application). But you can change this behavior for testing: in about:config, change the value of “security.fileuri.strict_origin_policy” to “false”. But be very careful! editing about:config — that’s an expert feature!
• There are two ways to display the result of your application on the top of the video:
  • use your canvas as a video (if you draw the frame every time), and then draw directly into the canvas
  • use a transparent canvas on the top of the video
• the canvas element can be “display: none”
• the video element can be “display: none”

About JavaScript

For the image processing, you will need to do a lot of computation. Here are some tricks:

• copy your frame in a small canvas. If the canvas is three times smaller than the video, it means nine times fewer pixels to compute.
• avoid recursion. In a recursion, the script engine doesn’t use the JIT optimization.
• if you want to do a distance between colors, use the L.A.B colorspace.
• if you want to find the center of an object, compute its centroid. See the “computeFrame” function that I use in this JavaScript snippet for my demo.
• if the algorithm is really heavy, you can use a Worker thread, but take into account that you will need to send the content of the canvas to the thread. It’s a big array, and objects are automatically JSONified before being sent. It can take a while.

Conclusion

As you can see, you can do powerful things with the video element, the canvas element, CSS3, SVG and the new JavaScript engine. You have everything in your hands to create a completely new way to use Video on the web. It’s up to you now — upgrade the web!

References

• LAB Color Space – for color space distances.
• How to use threads in your JavaScript code.
• The centroid definition – good way to compute the center of an object from its pixels.
• TraceMonkey – the new Firefox JavaScript engine
• Pixel manipulation with canvas.
• The “Dynamic Content Injection” demo.
• The canvas.drawImage API – useful to inject the frame of a video into a canvas element.
• How to do transformation in your canvas.
• A good overview of the canvas API.
• Everything you need to know about canvas.
• All the new OpenWeb features in Firefox 3.5.
• Layering a div over a video.
• Transformation with CSS 3.
• Including HTML inside SVG.
• Applying SVG effects to HTML content.
• A 100% OpenVideo player.
• The reference for the video and the audio HTML5 tags.
• The Javascript API for the video and the audio tags.
• The video HTML attributes.
• The video fallback mechanism.
• Media formats supported by the audio and video elements.
• My demos.