hacks.mozilla.org

This article is written by Paul Rouget, Mozilla contributor and purveyor of extraordinary Open Web demos.

Starting with Firefox 3.5, you can embed a video in a web page like an image. This means video is now a part of the document, and finally, a first class citizen of the Open Web. Like all other elements, you can use it with CSS and JavaScript. Let’s see what this all means …

The Basics

First, you need a video to play. Firefox supports the Theora codec (see here to know all media formats supported by the audio and video elements).

Add the video to your document:

<video id="myVideo" src="myFile.ogv"/>

You might need to add some “fallback” code if the browser doesn’t support the video tag. Just include some HTML (which could be a warning, or even some Flash) inside the video tag.

<video id="myVideo" src="myFile.ogv">
<strong>Your browser is not awesome enough!</strong>
</video>

Here’s some more information about the fallback mechanism.

HTML Attributes

You can find all the available attributes here.

Some important attributes:

• autoplay: The video will be played just after the page loads.
• autobuffer: By default (without this attribute), the video file is not downloaded unless you click on the play button. Adding this attribute starts downloading the video just after the page loads.
• controls: by default (without this attribute), the video doesn’t include any controls (play/pause button, volume, etc.). Use this attribute if you want the default controls.
• height/width: The size of the video

Example:

<video id="myVideo" src="myFile.ogv" 
   autobuffer="true" controls="true"/>

You don’t have to add the “true” value to some of these attributes in HTML5, but it’s neater to do so. If you’re not in an XML document, you can simply write:

<video id="myVideo" src="myFile.ogv" autobuffer controls/>

JavaScript API

Like any other HTML element, you have access to the video element via the Document Object Model (DOM):

var myVideo = document.getElementById("myVideo");

Once you obtain a handle to the video element, you can use the JavaScript API for video.

Here is a short list of some useful methods and properties (and see here for more of the DOM API for audio and video elements):

• play() / pause(): Play and pause your video.
• currentTime: The current playback time, in seconds. You can change this to seek.
• duration: The duration of the video.
• muted: Is the sound muted?
• ended: Has the video ended?
• paused: Is the video paused?
• volume: To determine the volume, and to change it.

Example:

<button onclick="myVideo.play()">Play</button>
<button onclick="myVideo.volume = 0.5">Set Volume</button>
<button onclick="alert(myVideo.volume)">Volume?</button>

Events

You know how to control a video (play/pause, seek, change the volume, etc.). You have almost everything you need to create your own controls. But you need some feedback from the video, and for that, let’s see the different events you can listen to:

• canplay: The video is ready to play
• canplaythrough: The video is ready to play without interruption (if the download rate doesn’t change)
• load: The video is ready to play without interruption (the video has been downloaded entirely)
• ended: The video just ended
• play: The video just started playing
• pause: The video has been paused
• seeking: The video is seeking (it can take some seconds)
• seeked: The seeking process just finished
• timeupdate: While the video is playing, the currentTime is updated. Every time the currentTime is updated, timeupdate is fired.

Here’s a full list of events.

For example, you can follow the percentage of the video that has just been played:

function init() 
{
  var video = document.getElementById("myVideo");
  var textbox = document.getElementById("sometext");
  video.addEventListener("timeupdate", function() {
  textbox.value = Math.round(100 * (video.currentTime / video.duration)) + "%"; }
 
}

<video id="myVideo" src="myFile.ogv" 
            autoplay="true" onplay="init()"/>
<input id="sometext"/>

Showing all this in action, here’s a nice open video player using the Video API.

Now that you’re familiar with some of the broad concepts behind the Video API, let’s really delve into the video as a part of the Open Web, introducing video to CSS, SVG, and Canvas.

CSS and SVG

A video element is an HTML element. That means you can use CSS to style it.

A simple example: using the CSS Image Border rule (a new CSS 3 feature introduced in Firefox 3.5). You can view how it works on the Mozilla Developer Wiki.

And obviously, you can use it with the video tag:

 
<video id="myVideo" src="myFile.ogv" 
style="-moz-border-image: 
           url(tv-border.jpg) 25 31 37 31 stretch stretch; 
           border-width: 20px;"/>

One of my demos uses this very trick.

Since Firefox 3.5 provides some new snazzy new CSS features, you can do some really fantastic things. Take a look at the infamous washing machine demo, in which I subject an esteemed colleague to some rotation.

It uses some CSS rules:

• filter, clip-path.
• css transform.

And some SVG:

• SVG & foreignobject.

Because the video element is like any other HTML element, you can add some HTML content over the video itself, like I do in this demo. As you can see, there is a <div> element on top of the video (position: absolute;).

Time for a Break

Well, we’ve just seen how far we can go with the video element, both how to control it and how to style it. That’s great, and it’s powerful. I strongly encourage you to read about the new web features available in Firefox 3.5, and to think about what you can do with such features and the video element.

You can do so much with the power of the Open Web. You can compute the pixels of the video. You can, for example, try to find some shapes in the video, follow the shapes, and draw something as an attachment to these shapes. That’s what I do here! Let’s see how it actually works.

Canvas & Video

Another HTML 5 element is canvas. With this element, you can draw bitmap data (see the canvas reference, and I strongly suggest this canvas overview). But something you might not know is that you can copy the content of an <img/> element, a <canvas/> element and a <video/> element.

That’s a really important point for the video element. It gives you a way to play with the values of the pixels of the video frames.

You can do a “screenshot” of the current frame of the video in a canvas.

function screenshot() {
 var video = document.getElementById("myVideo");
 var canvas = document.getElementById("myCanvas");
 var ctx = canvas.getContext("2d");
 
 ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
}

<video id="myVideo" src="myFile.ogv" autoplay="true" with="600" height="400"/>
<canvas id="myCanvas" with="600" height="400"/>
<button onclick="screenshot()">Copy current frame to canvas</button>

You can first apply a transformation to your canvas (see the documentation). You can also copy a thumbnail of the video.

If you draw every frame in a canvas, your canvas will look like a video element. And you can draw what you want in this canvas, after drawing the frame. That’s what I do in this demo.

Once you have a video frame in your canvas, you can compute the values of the pixels.

Some things you should know if you want to compute the pixels values of a frame:

• you can’t use this mechanism with a video from another domain.
• you can’t use this mechanism with a video from a file:/// URL (which would be useful during the development of your web application). But you can change this behavior for testing: in about:config, change the value of “security.fileuri.strict_origin_policy” to “false”. But be very careful! editing about:config — that’s an expert feature!
• There are two ways to display the result of your application on the top of the video:
  • use your canvas as a video (if you draw the frame every time), and then draw directly into the canvas
  • use a transparent canvas on the top of the video
• the canvas element can be “display: none”
• the video element can be “display: none”

About JavaScript

For the image processing, you will need to do a lot of computation. Here are some tricks:

• copy your frame in a small canvas. If the canvas is three times smaller than the video, it means nine times fewer pixels to compute.
• avoid recursion. In a recursion, the script engine doesn’t use the JIT optimization.
• if you want to do a distance between colors, use the L.A.B colorspace.
• if you want to find the center of an object, compute its centroid. See the “computeFrame” function that I use in this JavaScript snippet for my demo.
• if the algorithm is really heavy, you can use a Worker thread, but take into account that you will need to send the content of the canvas to the thread. It’s a big array, and objects are automatically JSONified before being sent. It can take a while.

Conclusion

As you can see, you can do powerful things with the video element, the canvas element, CSS3, SVG and the new JavaScript engine. You have everything in your hands to create a completely new way to use Video on the web. It’s up to you now — upgrade the web!

References

• LAB Color Space – for color space distances.
• How to use threads in your JavaScript code.
• The centroid definition – good way to compute the center of an object from its pixels.
• TraceMonkey – the new Firefox JavaScript engine
• Pixel manipulation with canvas.
• The “Dynamic Content Injection” demo.
• The canvas.drawImage API – useful to inject the frame of a video into a canvas element.
• How to do transformation in your canvas.
• A good overview of the canvas API.
• Everything you need to know about canvas.
• All the new OpenWeb features in Firefox 3.5.
• Layering a div over a video.
• Transformation with CSS 3.
• Including HTML inside SVG.
• Applying SVG effects to HTML content.
• A 100% OpenVideo player.
• The reference for the video and the audio HTML5 tags.
• The Javascript API for the video and the audio tags.
• The video HTML attributes.
• The video fallback mechanism.
• Media formats supported by the audio and video elements.
• My demos.

XMLHttpRequest is used within many Ajax libraries, but till the release of browsers such as Firefox 3.5 and Safari 4 has only been usable within the framework of the same-origin policy for JavaScript. This meant that a web application using XMLHttpRequest could only make HTTP requests to the domain it was loaded from, and not to other domains. Developers expressed the desire to safely evolve capabilities such as XMLHttpRequest to make cross-site requests, for better, safer mash-ups within web applications. The Cross-Origin Resource Sharing (CORS) specification consists of a simple header exchange between client-and-server, and is used by IE8’s proprietary XDomainRequest object as well as by XMLHttpRequest in browsers such as Firefox 3.5 and Safari 4 to make cross-site requests. These browsers make it possible to make asynchronous HTTP calls within script to other domains, provided the resources being retrieved are returned with the appropriate CORS headers.

A Quick Overview of CORS

Firefox 3.5 and Safari 4 implement the CORS specification, using XMLHttpRequest as an “API container” that sends and receives the appropriate headers on behalf of the web developer, thus allowing cross-site requests. IE8 implements part of the CORS specification, using XDomainRequest as a similar “API container” for CORS, enabling simple cross-site GET and POST requests. Notably, these browsers send the ORIGIN header, which provides the scheme (http:// or https://) and the domain of the page that is making the cross-site request. Server developers have to ensure that they send the right headers back, notably the Access-Control-Allow-Origin header for the ORIGIN in question (or ” * ” for all domains, if the resource is public) .

The CORS standard works by adding new HTTP headers that allow servers to serve resources to permitted origin domains. Browsers support these headers and enforce the restrictions they establish. Additionally, for HTTP request methods that can cause side-effects on user data (in particular, for HTTP methods other than GET, or for POST usage with certain MIME types), the specification mandates that browsers “preflight” the request, soliciting supported methods from the server with an HTTP OPTIONS request header, and then, upon “approval” from the server, sending the actual request with the actual HTTP request method. Servers can also notify clients whether “credentials” (including Cookies and HTTP Authentication data) should be sent with requests.

Capability Detection

XMLHttpRequest can make cross-site requests in Firefox 3.5 and in Safari 4; cross-site requests in previous versions of these browsers will fail. It is always possible to try to initiate the cross-site request first, and if it fails, to conclude that the browser in question cannot handle cross-site requests from XMLHttpRequest (based on handling failure conditions or exceptions, e.g. not getting a 200 status code back). In Firefox 3.5 and Safari 4, a cross-site XMLHttpRequest will not successfully obtain the resource if the server doesn’t provide the appropriate CORS headers (notably the Access-Control-Allow-Origin header) back with the resource, although the request will go through. And in older browsers, an attempt to make a cross-site XMLHttpRequest will simply fail (a request won’t be sent at all).

Both Safari 4 and Firefox 3.5 provide the withCredentials property on XMLHttpRequest in keeping with the emerging XMLHttpRequest Level 2 specification, and this can be used to detect an XMLHttpRequest object that implements CORS (and thus allows cross-site requests). This allows for a convenient “object detection” mechanism:

if (XMLHttpRequest)
{
    var request = new XMLHttpRequest();
    if (request.withCredentials !== undefined)
    {
      // make cross-site requests
    }
}

Alternatively, you can also use the “in” operator:

if("withCredentials" in request)
{
  // make cross-site requests
}

Thus, the withCredentials property can be used in the context of capability detection. We’ll discuss the use of “withCredentials” as a means to send Cookies and HTTP-Auth data to sites later on in this article.

“Simple” Requests using GET or POST

IE8, Safari 4, and Firefox 3.5 allow simple GET and POST cross-site requests. “Simple” requests don’t set custom headers, and the request body only uses plain text (namely, the text/plain Content-Type).

Let us assume the following code snippet is served from a page on site http://foo.example and is making a call to http://bar.other:

 
var url = "http://bar.other/publicNotaries/"
if(XMLHttpRequest)
{
  var request = new XMLHttpRequest();
  if("withCredentials" in request)
  {
   // Firefox 3.5 and Safari 4
   request.open('GET', url, true);
   request.onreadystatechange = handler;
   request.send();
  }
  else if (XDomainRequest)
  {
   // IE8
   var xdr = new XDomainRequest();
   xdr.open("get", url);
   xdr.send();
 
   // handle XDR responses -- not shown here :-)
  }
 
 // This version of XHR does not support CORS  
 // Handle accordingly
}

Firefox 3.5, IE8, and Safari 4 take care of sending and receiving the right headers. Here is the Simple Request example. It is also instructive to look at the headers sent back by the server. Notably, amongst the other request headers, the browser would send the following in order to enable the simple request above:

GET /publicNotaries/ HTTP/1.1
Referer: http://foo.example/notary-mashup/
Origin: http://foo.example

Note the use of the “Origin” HTTP header that is part of the CORS specification.

And, amongst the other response headers, the server at http://bar.other would include:

Access-Control-Allow-Origin: http://foo.example
Content-Type: application/xml
......

A more complete treatment of CORS and XMLHttpRequest can be found here, on the Mozilla Developer Wiki.

“Preflighted” Request

The CORS specification mandates that requests that use methods other than POST or GET, or that use custom headers, or request bodies other than text/plain, are preflighted. A preflighted request first sends the OPTIONS header to the resource on the other domain, to check and see if the actual request is safe to send. This capability is currently not supported by IE8’s XDomainRequest object, but is supported by Firefox 3.5 and Safari 4 with XMLHttpRequest. The web developer does not need to worry about the mechanics of preflighting, since the implementation handles that.

The code snippet below shows code from a web page on http://foo.example calling a resource on http://bar.other. For simplicity, we leave out the section on object and capability detection, since we’ve covered that already:

var invocation = new XMLHttpRequest();
var url = 'http://bar.other/resources/post-here/';
var body = '
Arun';
function callOtherDomain(){
if(invocation)
{
    invocation.open('POST', url, true);
    invocation.setRequestHeader('X-PINGOTHER', 'pingpong');
    invocation.setRequestHeader('Content-Type', 'application/xml');
    invocation.onreadystatechange = handler;
    invocation.send(body);
}

You can see this example in action here. Looking at the header exchange between client and server is really instructive. A more detailed treatment of this can be found on the Mozilla Developer Wiki.

In this case, before Firefox 3.5 sends the request, it first uses the OPTIONS header:

OPTIONS /resources/post-here/ HTTP/1.1
Origin: http://foo.example
Access-Control-Request-Method: POST
Access-Control-Request-Headers: X-PINGOTHER

Then, amongst the other response headers, the server responds with:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: http://arunranga.com
Access-Control-Allow-Methods: POST, GET, OPTIONS
Access-Control-Allow-Headers: X-PINGOTHER
Access-Control-Max-Age: 1728000

At which point, the actual response is sent:

POST /resources/post-here/ HTTP/1.1
...
Content-Type: application/xml; charset=UTF-8
X-PINGOTHER: pingpong
...

Credentialed Requests

By default, “credentials” such as Cookies and HTTP Auth information are not sent in cross-site requests using XMLHttpRequest. In order to send them, you have to set the withCredentials property of the XMLHttpRequest object. This is a new property introduced in Firefox 3.5 and Safari 4. IE8’s XDomainRequest object does not have this capability.

Again, let us assume some JavaScript on a page on http://foo.example wishes to call a resource on http://bar.other and send Cookies with the request, such that the response is cognizant of Cookies the user may have acquired.

var request = new XMLHttpRequest();
var url = 'http://bar.other/resources/credentialed-content/';
function callOtherDomain(){
  if(request)
  {
   request.open('GET', url, true);
   request.withCredentials = "true";
   request.onreadystatechange = handler;
   request.send();
  }

Note that withCredentials is false (and NOT set) by default. The header exchange is similar to the case of of a simple GET request, with the exception that now an HTTP Cookie header is sent with the request header. You can see this sample in action here.

A Note on Security

In general, data requested from a remote site should be treated as untrusted. Executing JavaScript code retrieved from a third-party site without first determining its validity is NOT recommended. Server administrators should be careful about leaking private data, and should judiciously determine that resources can be called in a cross-site manner.

References

• Mozilla Developer Wiki documentation on CORS (formerly called Access Control)
• Mozilla Developer Wiki documentation for server administrators
• Examples of Cross-Site XMLHttpRequest (XS-XHR)
• CORS in the context of Web Fonts, and how to use .htaccess on an Apache server to ensure the right CORS headers get sent back

We’ve just launched Firefox 3.5, and we’re incredibly proud. Naturally, we have engaged in plentiful Mozilla advocacy — this site is, amongst other things, a vehicle for showcasing the latest browser’s new capabilities. We like to think about this release as an upgrade for the whole World Wide Web, because of the new developer-facing features that have just been introduced into the web platform. When talking about some of the next generation standards, the appearance of the number “5″ is almost uncanny — consider HTML5 and ECMAScript 5 (PDF). The recent (and very welcome) hype around HTML5 in the press is what motivates this article. Let’s take a step back, and consider some of Mozilla’s web advocacy in the context of events leading up to the release of Firefox 3.5.

Standardization of many of these features often came after much spirited discussion, and we’re pleased to see the prominent placement of HTML5 as a key strategic initiative by major web development companies. Indeed, exciting new web applications hold a great deal of promise, and really showcase what the future of the web platform holds in store for aspiring developers. Many herald the triumphant arrival of the browser as the computer, an old theme that gets bolstered with the arrival of attractive HTML5 platform features that are implemented across Safari, Chrome, Opera, and of course, Firefox (with IE8 getting an honorable mention for having both some HTML5 features and some ECMAScript, 5th Edition features).

Call it what you will — Web 5.0, Open Web 5th Generation (wince!), or, (R)evolution # 5, the future is now. But lest anyone forget, HTML5 is not a completed standard yet, as the W3C was quick to point out. The editor doesn’t anticipate completion till 2010. The path taken from the start of what is now called HTML5 to the present-day era of (very welcome) hype has been a long one, and Mozilla has been part of the journey from the very beginning.

For one thing, we were there to point out, in no uncertain terms, that the W3C had perhaps lost its way. Exactly 5 summers ago (again, with that magic number!), it became evident that the W3C was no longer able to serve as sole custodian of the standards governing the open web of browser-based applications, so Mozilla, along with Opera, started the WHATWG. Of course, back then, we didn’t call it HTML5, and while Firefox itself made a splash in 2004, the steps taken towards standardization were definitive but tentative. Soon, other browser vendors joined us, and by the time the reconciliation with W3C occurred two years later, the innovations introduced into the web platform via the movement initiated by Mozilla had gained substantial momentum.

The net result is a specification that is not yet complete called “HTML5″ which is implemented piecemeal by most modern browsers. The features we choose to implement as an industry are in response to developers, and our modus operandi is (for the most part) in the open. Mozilla funds the HTML5 Validator, producing the first real HTML5 parser, which now drives W3C’s markup validation for HTML5. That parser has made its way back into Firefox. It’s important to note that capabilities that are of greatest interest (many of which are showcased on this blog) are not only developed within the HTML5 specification, but also as part of the W3C Geolocation WG, the Web Apps WG, and the CSS WG.

The release of Firefox 3.5, along with updates to other modern browsers, seems to declare that HTML5 has arrived. But with the foresight that comes with having been around this for a while, we also know that we have a lot of work ahead of us. For one thing, we’ve got to finish HTML5, or at least publish a subset of it that we all agree is ready for implementation, soon. We’ve also got to ensure that accessibility serves as an important design principle in the emerging web platform, and resolve sticky differences here. Also, an open standard does not an open platform make, as debates about web fonts and audio/video codecs show. We’ve got a lot of work ahead of us, but for now, 5 years after the summer we started the ball rolling, we’re enjoying the hype around (R)evolution Number 5.

The JavaScript Object Notation (JSON) mechanism for representing data has rapidly become an indispensable part of the web developer’s toolkit, allowing JavaScript applications to obtain and parse data intuitively, within scripts, with lightweight data encapsulation. Firefox 3.5 includes support for JSON natively by exposing a new primitive — window.JSON — to the top level object.

Native “out of the box” support for JSON comes to the web courtesy of the ECMAScript 5th Edition (PDF link to specification), other aspects of which will also be supported by Firefox 3.5. Presently, native JSON is supported by Firefox 3.5 and IE8, with a strong likelihood of other browsers supporting it soon as well.

Native JSON support has two advantages:

1. Safety. Simply using eval to evaluate expressions returned as strings raises security issues. Also, the native JSON primitive can only work with data. It can’t be used to parse objects with method calls; attempting to do so returns an error.
2. Performance. Parsing JSON safely, using third-party scripts and libraries, is likely to be slower than native JSON parsing within the browser.

Let’s look at some examples.

A JSON API for search results might look like this:

/* 
Assume that you obtained var data
as a string from a server
For convenience we display this search
result on separate lines
*/
 
var data = ' { "responseData":
{"results": [
    {
        "SafeSearch":"true",
        "url":"http://www.arunranga.com/i.jpg",
    },
    {
        "SafeSearch":"false",
	 "url":"http://www.badarunranga.com/evil.jpg",
    }
]}}';

Such a resource could be returned by a simple HTTP GET request using a RESTful API.

Using native JSON, you could do something like this:

/* 
 Obtain a handle to the above JSON resource
 This is best and most conveniently done
 via third-party libraries that support native JSON
*/
 
if (window.JSON) {
    var searchObj = JSON.parse(data);
    for (var i=0; i++; i < searchObj.responseData.results.length) {
        if (searchObj.responseData.results[i].SafeSearch) {
            var img = new Image();
            img.src = searchObj.responseData.results[i].url;
            // ... Insert image into DOM ...
    }
}

You can also stringify the object back to the string:

// Back to where we started from
 
var data = JSON.stringify(searchObj);
 
// data now holds the string we started with

Of course, to really enable the power of JSON, you’ll want to retrieve JSON resources from different domains, via callback mechanisms like JSONP. Many web developers are unlikely to use the JSON primitive directly. Rather, they’ll use them within libraries such as Dojo and jQuery. These libraries allow for the retrieval and direct parsing of JSON resources from different domains, and add a great deal of syntactic sugar to the process of callbacks and DOM insertions.

The native JSON primitive works with the popular json2.js library (which correctly detects if native JSON is enabled), so developers can seamlessly use JSON parsing on browsers that don’t support native JSON. As of this writing, Dojo and jQuery have committed to supporting native JSON:

• Detection and use of native JSON in jQuery by version 1.3.3
• Detection and use of native JSON in Dojo by version 1.4