See the demo online

You can download the animation demo for yourself and try it out locally – all you need is a browser that supports HTML5 video. I know – the music is not quite the same as in the movies, but at least this one is not copyright infringing and it came from the heart (5 minutes in a meeting room in the Mozilla office).

So how was this done and what are problems that needed solving? Here’s how and what.

Step 1: Find the movie and get it to the right format

That was the easy part. Archive.org has a lot of awesome public domain movies available for you and they are already in the formats needed to use in an HTML5 video element. In this case, I took the short movie of Charles Lindbergh taking off for his record breaking flight from New York to fly to Paris in 1927.

Step 2: Displaying the video

Using the video is pretty simple:

<div id="stage">
  <video>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff_512kb.mp4" 
type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff.ogv"
 type='video/ogg; codecs="theora, vorbis"'> 
  </video>
</div>

The MP4 format will be used by Webkit based browsers and the Ogg version by Firefox and others. As we want to control the video we omit the controls attribute on the video element – instead we create a button to play the video with JavaScript:

window.addEventListener('load', 
  function() {
    var stage = document.getElementById('stage');
    var v = document.getElementsByTagName('video')[0];
    but = document.createElement('button');
    but.innerHTML = 'Click to see Lindbergh\'s flight';
    stage.appendChild(but);
    but.addEventListener('click',function(e) {
      v.play();
      e.preventDefault();
    },false);
  }, 
false);

As the video is markup we can do whatever we please with it – the power of open technologies. For example as we will do here we can set its opacity in CSS and put in on top of a map.

Step 3: Create the map path animation

Talking of which, let’s get that moving path done. Google Earth has an API to do that, but it needs a special plugin. Google Maps allows you to paint paths on maps (which actually are SVG, another open standard). Put that in a recursive function and you get the desired effect:

In essence, what I did was take the latitude and longitude of the start and end points and calculate as many points in between the two as I need for the duration of the animation. I store the points in an array called pos and then paint a path from the start to the current point and move the map centre to this point on every iteration.

spirit.draw = function(){
  var path = new google.maps.Polyline({
        path: [startpos,pos[now]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
  });
  path.setMap(map);
  map.panTo(pos[now])
  now = now + 1;
  if(now < animationend-1){
    setTimeout(spirit.draw,200);
  }
}

Check the highly commented source of the map example for the details. Now, we could use this kind of animation and play the video over it – the problem though is that they may get out of sync. When the movie stalls (as it frequently does on this hotel wireless connection) we don’t want the animation to keep moving, right?

Step 4: Syncing video and the map movement

Instead of having two sources of timing information we have to limit ourselves to one source of truth. This is the time stamp of the currently playing movie.

By the way – you might have noticed that I wrapped the map code in a tilesloaded event handler. This is another safeguard for keeping things in sync. I found that on slow connections the tile loading can delay the whole interface immensely (because of all the subdomain lookups), so I make the whole interface dependent on the loading of the map and only proceed when the tiles have finished loading. As the tilesloaded event also fires when the map pans we need to use a boolean to stop it from starting the effect several times:

google.maps.event.addListener(map,'tilesloaded',function(){
  if(played === false){
    // [...other code...]
    played = true;
  }
});

You can read the current timestamp of a video with video.currentTime and whilst the movie is playing it constantly fires an event called timeupdate. As the event fires a lot we need to throttle it somehow. The trick here is to only take the full seconds and increase a counter when a new second is reached. You can see the timestamp and the second interval firing in the video syncing demo:

var now = 0;
v.addEventListener('timeupdate',function(o){
  log.innerHTML = v.currentTime; /* logging the real timestamp */
  var full = parseInt(v.currentTime);
  if(full >= now) {
    seqlog.innerHTML = now;  /* logging the seconds firing */
    now = now + 1;
  }
},false);

That way the movie can lag in between and the sequence still stays in sync. Check the source of this demo on Github.

Putting it all together

And that was about it – all I had to do is to set the movie’s opacity at a certain time stamp, start the sound at another and show and hide the copyright complaint at another. As we rely on the timestamp for the other effects we needed a boolean switch to avoid repeated firing:

v.addEventListener('timeupdate',function(o){
  full = parseInt(v.currentTime);
  if(full === now-1){
    mapelm.style.opacity = .8;
    v.style.opacity = .4;
  }
  if(full === animationstart+1 && audioplay === false){
    a.play();
    audioplay = true;
  }
  if(full === animationstart+2 && hidden === true){
    drmbedamned.style.display = 'block';
    hidden = false;
  }
  if(full === animationstart+3 && hidden === false){
    drmbedamned.style.display = 'none';
    hidden = true;
  }
  if(full >= now) {
    path = new google.maps.Polyline({
        path: [startpos,pos[full]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
    });
    path.setMap(map);
    map.panTo(pos[full])
    now = now + 1;
  }
},false);

Another event we needed to subscribe to was the movie ending so we can stop the music and start to roll the credits:

v.addEventListener('ended',function(o){
  a.pause();
  spirit.credslist.parentNode.style.display = 'block';
  spirit.creds();
},false)

As the theme is too short for the whole animation we need to loop it. This can be done by testing for the ended event and rolling back the time to 0:

a.addEventListener('ended', function(o) {
  a.currentTime = 0;
},false);

Summary

And there you have it – Indiana Jones style maps using open services and open technologies. A workaround for the copyrighted audio (recorded, edited and converted with the free Audacity sound editor) and using Google’s Web Fonts as graphics.

You can now take this and change it for even more awesome:

• Replace Google Maps with Openstreetmap to avoid going over the limit
• Add a slight curve to the path from NYC to Paris to make it more accurate (but then again the time is not accurate either – it took charles a tad longer)
• Use a static map and paint the path with Canvas to speed up and smoothen the animation

Why not have a go – it is free and fun to play.

Today (6/12/11) two calendar blogs delivered an article of mine talking about the benefits of using local storage in browsers and how to implement it.

You can find the English version on the 24ways.org calendar and a German translation at the Webkrauts calendar.

In essence, the article explains why it is a good idea to use local storage, explains how to use it, how to work around the issue that local storage can only store strings and gives example code on how to speed up web service use by caching information client-side much like you would do it on a server.

Have a read and go speed up your own solutions by using what browsers provide you these days.

As of this minute, Firebug 1.5 is sitting comfortably in its third beta and available for download. This version is shaping up to be our best release yet and initial reports have been very positive regarding its stability, UI improvements and new features. So let’s take a look at some of the new features.

Improved Net Panel accuracy

One of the problems with Firebug’s Net panel in the past has been inaccurate timings. Because Firebug is entirely written in JavaScript some network and UI activity could block Firebug during long operations and cause the timings displayed there to be less than accurate. This has finally been corrected with the landing of a new service called the http-activity-distributor. For more details on the implementation and use of the newly-improved Net panel, see Honza’s blog post on the topic.

New Break Functionality

In Firebug 1.4, we introduced the concept of “break-on-next” to the Script panel. This was a “pause” button sitting between the inspect icon and the Console tab. In 1.5, we’ve extended this concept to the Console, HTML and Net panels to allow more exciting types of breaks.

In the Console, we’ve replaced the mini-menu Break-on-Errors option with the pause button. The reasons for this possibly contentious change was it made for a more consistent use of menus and the break button. Now, to enable Break on Errors, select the Console panel and hit the pause button. You’ll see that familiar glow to indicate that it’s waiting for an error. Now whenever an error occurs on the page, you’ll be dropped into the script panel at the line where the error occurred.

The HTML panel’s break button is a little different. This is the Break-on-Mutation feature. When this is enabled, whenever a bit of JavaScript modifies an HTML element, you’ll be taken to the Script panel and the modifying code while be highlighted. Related to this, you should be able to see modified HTML occurring in real-time in the HTML panel with affected elements and attributes being highlighted as they change in the page.

Finally in the Net panel, the break button acts as a Break-on-XHR button. This is intended to help debug AJAX apps by allowing you to halt the debugger during an XmlHttpRequest send. As in the other break types, you’ll be transported to the script panel when an XHR object fires off its request and you’ll be given the option to copy the message.

John Barton and Honza have written a great interactive demo page describing these new features on getfirebug.com.

Mixed Development

• We made a few tweaks to the UI in this version. We replaced the “Off” label with a single “power” button (or window close button on Mac) as promised during the last release.
• Kevin Decker added the search panel originally intended for version 1.4 with some nice options.
• Persist option on Console and Net Panel. Save your data!
• Improvements to the Inspector.
• Still more to come. Between now and final release we plan on hunting down a few more bugs to make this even more stable. Feel free to download and give it a try.

The demo uses jQuery and drives a simple carousel that shows the current chapter as the video plays. It also shows subtitles as the video plays.

One really great, and somewhat subtle thing about this experiment? You can select the subtitles. They are just text.

Firefox 3.5 has a new JavaScript engine, TraceMonkey, that runs many JavaScript programs 3-4x faster than Firefox 3, speeding up existing web apps and enabling new ones. This article gives a peek under the hood at the major parts of TraceMonkey and how they speed up JS. This will also explain what kinds of programs get the best speedup from TraceMonkey and what kinds of things you can do to get your program to run faster.

Why it’s hard to run JS fast: dynamic typing

High-level dynamic languages such as JavaScript and Python make programming more productive, but they have always been slow compared to statically typed languages like Java or C. A typical rule of thumb was that a JS program might be 10x slower than an equivalent Java program.

There are two main reasons JS and other dynamic scripting languages usually run slower than Java or C. The first reason is that in dynamic languages it is generally not possible to determine the types of values ahead of time. Because of this, the language must store all values in a generic format and process values using generic operations.

In Java, by contrast, the programmer declares types for variables and methods, so the compiler can determine the types of values ahead of time. The compiler can then generate code that uses specialized formats and operations that run much faster than generic operations. I will call these type-specialized operations.

The second main reason that dynamic languages run slower is that scripting languages are usually implemented with interpreters, while statically typed languages are compiled to native code. Interpreters are easier to create, but they incur extra runtime overhead for tracking their internal state. Languages like Java compile to machine language, which requires almost no state tracking overhead.

Let’s make this concrete with a picture. Here are the slowdowns in picture form for a simple numeric add operation: a + b, where a and b are integers. For now, ignore the rightmost bar and focus on the comparison of the Firefox 3 JavaScript interpreter vs. a Java JIT. Each column shows the steps that have to be done to complete the add operation in each programming language. Time goes downward, and the height of each box is proportional to the time it takes to finish the steps in the box.

In the middle, Java simply runs one machine language add instruction, which runs in time T (one processor cycle). Because the Java compiler knows that the operands are standard machine integers, it can use a standard integer add machine language instruction. That’s it.

On the left, SpiderMonkey (the JS interpreter in FF3) takes about 40 times as long. The brown boxes are interpreter overhead: the interpreter must read the add operation and jump to the interpreter’s code for a generic add. The orange boxes represent extra work that has to be done because the interpreter doesn’t know the operand types. The interpreter has to unpack the generic representations of a and i, figure out their types, select the specific addition operation, convert the values to the right types, and at the end, convert the result back to a generic format.

The diagram shows that using an interpreter instead of a compiler is slowing things down a little bit, but not having type information is slowing things down a lot. If we want JS to run more than a little faster than in FF3, by Amdahl’s law, we need to do something about types.

Getting types by tracing

Our goal in TraceMonkey is to compile type-specialized code. To do that, TraceMonkey needs to know the types of variables. But JavaScript doesn’t have type declarations, and we also said that it’s practically impossible for a JS engine to figure out the types ahead of time. So if we want to just compile everything ahead of time, we’re stuck.

So let’s turn the problem around. If we let the program run for a bit in an interpreter, the engine can directly observe the types of values. Then, the engine can use those types to compile fast type-specialized code. Finally, the engine can start running the type-specialized code, and it will run much faster.

There are a few key details about this idea. First, when the program runs, even if there are many if statements and other branches, the program always goes only one way. So the engine doesn’t get to observe types for a whole method — the engine observes types through the paths, which we call traces, that the program actually takes. Thus, while standard compilers compile methods, TraceMonkey compiles traces. One side benefit of trace-at-a-time compilation is that function calls that happen on a trace are inlined, making traced function calls very fast.

Second, compiling type-specialized code takes time. If a piece of code is going to run only one or a few times, which is common with web code, it can easily take more time to compile and run the code than it would take to simply run the code in an interpreter. So it only pays to compile hot code (code that is executed many times). In TraceMonkey, we arrange this by tracing only loops. TraceMonkey initially runs everything in the interpreter, and starts recording traces through a loop once it gets hot (runs more than a few times).

Tracing only hot loops has an important consequence: code that runs only a few times won’t speed up in TraceMonkey. Note that this usually doesn’t matter in practice, because code that runs only a few times usually runs too fast to be noticeable. Another consequence is that paths through a loop that are not taken at all never need to be compiled, saving compile time.

Finally, above we said that TraceMonkey figures out the types of values by observing execution, but as we all know, past performance does not guarantee future results: the types might be different the next time the code is run, or the 500th next time. And if we try to run code that was compiled for numbers when the values are actually strings, very bad things will happen. So TraceMonkey must insert type checks into the compiled code. If a check doesn’t pass, TraceMonkey must leave the current trace and compile a new trace for the new types. This means that code with many branches or type changes tends to run a little slower in TraceMonkey, because it takes time to compile the extra traces and jump from one to another.

TraceMonkey in action

Now, we’ll show tracing in action by example on this sample program, which adds the first N whole numbers to a starting value:

 function addTo(a, n) {
   for (var i = 0; i < n; ++i)
     a = a + i;
   return a;
 }
 
 var t0 = new Date();
 var n = addTo(0, 10000000);
 print(n);
 print(new Date() - t0);

TraceMonkey always starts running the program in the interpreter. Every time the program starts a loop iteration, TraceMonkey briefly enters monitoring mode to increment a counter for that loop. In FF3.5, when the counter reaches 2, the loop is considered hot and it’s time to trace.

Now TraceMonkey continues running in the interpreter but starts recording a trace as the code runs. The trace is simply the code that runs up to the end of the loop, along with the types used. The types are determined by looking at the actual values. In our example, the loop executes this sequence of JavaScript statements, which becomes our trace:

    a = a + i;    // a is an integer number (0 before, 1 after)
    ++i;          // i is an integer number (1 before, 2 after)
    if (!(i < n)) // n is an integer number (10000000)
      break;

That’s what the trace looks like in a JavaScript-like notation. But TraceMonkey needs more information in order to compile the trace. The real trace looks more like this:

  trace_1_start:
    ++i;            // i is an integer number (0 before, 1 after)
    temp = a + i;   // a is an integer number (1 before, 2 after)
    if (lastOperationOverflowed())
      exit_trace(OVERFLOWED);
    a = temp;
    if (!(i < n))   // n is an integer number (10000000)
      exit_trace(BRANCHED);
    goto trace_1_start;

This trace represents a loop, and it should be compiled as a loop, so we express that directly using a goto. Also, integer addition can overflow, which requires special handling (for example, redoing with floating-point addition), which in turn requires exiting the trace. So the trace must include an overflow check. Finally, the trace exits in the same way if the loop condition is false. The exit codes tell TraceMonkey why the trace was exited, so that TraceMonkey can decide what to do next (such as whether to redo the add or exit the loop). Note that traces are recorded in a special internal format that is never exposed to the programmer — the notation used above is just for expository purposes.

After recording, the trace is ready to be compiled to type-specialized machine code. This compilation is performed by a tiny JIT compiler (named, appropriately enough, nanojit) and the results are stored in memory, ready to be executed by the CPU.

The next time the interpreter passes the loop header, TraceMonkey will start executing the compiled trace. The program now runs very fast.

On iteration 65537, the integer addition will overflow. (2147450880 + 65537 = 2147516417, which is greater than 2^31-1 = 2147483647, the largest signed 32-bit integer.) At this point, the trace exits with an OVERFLOWED code. Seeing this, TraceMonkey will return to interpreter mode and redo the addition. Because the interpreter does everything generically, the addition overflow is handled and everything works as normal. TraceMonkey will also start monitoring this exit point, and if the overflow exit point ever becomes hot, a new trace will be started from that point.

But in this particular program, what happens instead is that the program passes the loop header again. TraceMonkey knows it has a trace for this point, but TraceMonkey also knows it can’t use that trace because that trace was for integer values, but a is now in a floating-point format. So TraceMonkey records a new trace:

  trace_2_start:
    ++i;            // i is an integer number
    a = a + i;      // a is a floating-point number
    if (!(i < n))   // n is an integer number (10000000)
      exit_trace(BRANCHED);
    goto trace_2_start;

TraceMonkey then compiles the new trace, and on the next loop iteration, starts executing it. In this way, TraceMonkey keeps the JavaScript running as machine code, even when types change. Eventually the trace will exit with a BRANCHED code. At this point, TraceMonkey will return to the interpreter, which takes over and finishes running the program.

Here are the run times for this program on my laptop (2.2GHz MacBook Pro):

This program gets a huge 22x speedup from tracing and runs about as fast as Java! Functions that do simple arithmetic inside loops usually get big speedups from tracing. Many of the bit operation and math SunSpider benchmarks, such bitops-3bit-bits-in-byte, ctrypto-sha1, and math-spectral-norm get 6x-22x speedups.

Functions that use more complex operations, such as object manipulation, get a smaller speedup, usually 2-5x. This follows mathematically from Amdahl’s law and the fact that complex operations take longer. Looking back at the time diagram, consider a more complex operation that takes time 30T for the green part. The orange and brown parts will still be about 30T together, so eliminating them gives a 2x speedup. The SunSpider benchmark string-fasta is an example of this kind of program: most of the time is taken by string operations that have a relatively long time for the green box.

Here is a version of our example program you can try in the browser:

Understanding and fixing performance problems

Our goal is to make TraceMonkey reliably fast enough that you can write your code in the way that best expresses your ideas, without worrying about performance. If TraceMonkey isn’t speeding up your program, we hope you’ll report it as a bug so we can improve the engine. That said, of course, you may need your program to run faster in today’s FF3.5. In this section, we’ll explain some tools and techniques for fixing performance of a program that doesn’t get a good (2x or more) speedup with the tracing JIT enabled. (You can disable the jit by going to about:config and setting the pref javascript.options.jit.content to false.)

The first step is understanding the cause of the problem. The most common cause is a trace abort, which simply means that TraceMonkey was unable to finish recording a trace, and gave up. The usual result is that the loop containing the abort will run in the interpreter, so you won’t get a speedup on that loop. Sometimes, one path through the loop is traced, but there is an abort on another path, which will cause TraceMonkey to switch back and forth between interpreting and running native code. This can leave you with a reduced speedup, no speedup, or even a slowdown: switching modes takes time, so rapid switching can lead to poor performance.

With a debug build of the browser or a JS shell (which I build myself – we don’t publish these builds) you can tell TraceMonkey to print information about aborts by setting the TMFLAGS environment variable. I usually do it like this:

TMFLAGS=minimal,abort
dist/MinefieldDebug.app/Contents/MacOS/firefox

The minimal option prints out all the points where recording starts and where recording successfully finishes. This gives a basic idea of what the tracer is trying to do. The abort option prints out all the points where recording was aborted due to an unsupported construct. (Setting TMFLAGS=help will print the list of other TMFLAGS options and then exit.)

(Note also that TMFLAGS is the new way to print the debug information. If you are using a debug build of the FF3.5 release, the environment variable setting is TRACEMONKEY=abort.)

Here’s an example program that doesn’t get much of a speedup in TraceMonkey.

function runExample2() {
  var t0 = new Date;
 
  var sum = 0;
  for (var i = 0; i < 100000; ++i) {
    sum += i;
  }
 
  var prod = 1;
  for (var i = 1; i < 100000; ++i) {
    eval("prod *= i");
  }
  var dt = new Date - t0;
  document.getElementById(example2_time').innerHTML = dt + ' ms';
}

If we set TMFLAGS=minimal,abort, we’ll get this:

Recording starting from ab.js:5@23
recording completed at  ab.js:5@23 via closeLoop
 
Recording starting from ab.js:5@23
recording completed at  ab.js:5@23 via closeLoop
 
Recording starting from ab.js:10@63
Abort recording of tree ab.js:10@63 at ab.js:11@70: eval.
 
Recording starting from ab.js:10@63
Abort recording of tree ab.js:10@63 at ab.js:11@70: eval.
 
Recording starting from ab.js:10@63
Abort recording of tree ab.js:10@63 at ab.js:11@70: eval.

The first two pairs of lines show that the first loop, starting at line 5, traced fine. The following lines showed that TraceMonkey started tracing the loop on line 10, but failed each time because of an eval.

An important note about this debug output is that you will typically see some messages referring to inner trees growing, stabilizing, and so on. These really aren’t problems: they usually just indicate a delay in finishing tracing a loop because of the way TraceMonkey links inner and outer loops. And in fact, if you look further down the output after such aborts, you will usually see that the loops eventually do trace.

Otherwise, aborts are mainly caused by JavaScript constructs that are not yet supported by tracing. The trace recording process is easier to implement for a basic operation like + than it is for an advanced feature like arguments. We didn’t have time to do robust, secure tracing of every last JavaScript feature in time for the FF3.5 release, so some of the more advanced ones, like arguments, aren’t traced in FF3.5.0. Other advanced features that are not traced include getters and setters, with, and eval. There is partial support for closures, depending on exactly how they are used. Refactoring to avoid these constructs can help performance.

Two particularly important JavaScript features that are not traced are:

• Recursion. TraceMonkey doesn’t see repetition that occurs through recursion as a loop, so it doesn’t try to trace it. Refactoring to use explicit for or while loops will generally give better performance.
• Getting or setting a DOM property. (DOM method calls are fine.) Avoiding these constructs is generally impossible, but refactoring the code to move DOM property access out of hot loops and performance-critical segments should help.

We are actively working on tracing all the features named above. For example, support for tracing arguments is already available in nightly builds.

Here is the slow example program refactored to avoid eval. Of course, I could have simply done the multiplication inline. Instead, I used a function created by eval because that’s a more general way of refactoring an eval. Note that the eval still can’t be traced, but it only runs once so it doesn’t matter.

function runExample3() {
  var t0 = new Date;
 
  var sum = 0;
  for (var i = 0; i < 100000; ++i) {
    sum += i;
  }
 
  var prod = 1;
  var mul = eval("(function(i) { return prod * i; })");
 
  for (var i = 1; i < 100000; ++i) {
    prod = mul(i);
  }
  var dt = new Date - t0;
  document.getElementById('example3_time').innerHTML = dt + ' ms';
}

There are a few more esoteric situations that can also hurt tracing performance. One of them is trace explosion, which happens when a loop has many paths through it. Consider a loop with 10 if statements in a row: the loop has 1024 paths, potentially causing 1024 traces to be recorded. That would use up too much memory, so TraceMonkey caps each loop at 32 traces. If the loop has fewer than 32 hot traces, it will perform well. But if each path occurs with equal frequency, then only 3% of the paths are traced, and performance will suffer.

This kind of problem is best analyzed with TraceVis, which creates visualizations of TraceMonkey performance. Currently, the build system only supports enabling TraceVis for shell builds, but the basic system can also run in the browser, and there is ongoing work to enable TraceVis in a convenient form in the browser.

The blog post on TraceVis is currently the most detailed explanation of what the diagrams mean and how to use them to diagnose performance problems. The post also contains a detailed analysis of a diagram that is helpful in understanding how TraceMonkey works in general.

Comparative JITerature

Here I will give a few comparisons to other JavaScript JIT designs. I’ll focus more on hypothetical designs than competing engines, because I don’t know details about them — I’ve read the release information and skimmed a few bits of code. Another big caveat is that real-world performance depends at least as much on engineering details as it does on engine architecture.

One design option could be a called a per-method non-specializing JIT. By this, I mean a JIT compiler that compiles a method at a time and generates generic code, just like what the interpreter does. Thus, the brown boxes from our diagrams are cut out. This kind of JIT doesn’t need to take time to record and compile traces, but it also does not type-specialize, so the orange boxes remain. Such an engine can still be made pretty fast by carefully designing and optimizing the orange box code. But the orange box can’t be completely eliminated in this design, so the maximum performance on numeric programs won’t be as good as a type-specializing engine.

As far as I can tell, as of this writing Nitro and V8 are both lightweight non-specializing JITs. (I’m told that V8 does try to guess a few types by looking at the source code (such as guessing that a is an integer in a >> 2) in order to do a bit of type specialization.) It seem that TraceMonkey is generally faster on numeric benchmarks, as predicted above. But TraceMonkey suffers a bit on benchmarks that use more objects, because our object operations and memory management haven’t been optimized as heavily.

A further development of the basic JIT is the per-method type-specializing JIT. This kind of a JIT tries to type-specialize a method based on the argument types the method is called with. Like TraceMonkey, this requires some runtime observation: the basic design checks the argument types each time a method is called, and if those types have not been seen before, compiles a new version of the method. Also like TraceMonkey, this design can heavily specialize code and remove both the brown and orange boxes.

I’m not aware that anyone has deployed a per-method type-specializing JIT for JavaScript, but I wouldn’t be surprised if people are working on it.

The main disadvantage of a per-method type-specializing JIT compared to a tracing JIT is that the basic per-method JIT only directly observes the input types to a method. It must try to infer types for variables inside the method algorithmically, which is difficult for JavaScript, especially if the method reads object properties. Thus, I would expect that a per-method type-specializing JIT would have to use generic operations for some parts of the method. The main advantage of the per-method design is that the method needs to be compiled exactly once per set of input types, so it’s not vulnerable to trace explosion. In turn, I think a per-method JIT would tend to be faster on methods that have many paths, and a tracing JIT would tend to be faster on highly type-specializable methods, especially if the method also reads a lot of values from properties.

Conclusion

By now, hopefully you have a good idea of what makes JavaScript engines fast, how TraceMonkey works, and how to analyze and fix some performance issues that may occur running JavaScript under TraceMonkey. Please report bugs if you run into any significant performance problems. Bug reports are also a good place for us to give additional tuning advice. Finally, we’re trying to improve constantly, so check out nightly TraceMonkey builds if you’re into the bleeding edge.

Introduction

Drag and drop is one of the most fundamental interactions afforded by graphical user interfaces. In one gesture, it allows users to pair the selection of an object with the execution of an action, often including a second object in the operation. It’s a simple yet powerful UI concept used to support copying, list reordering, deletion (ala the Trash / Recycle Bin), and even the creation of link relationships.

Since it’s so fundamental, offering drag and drop in web applications has been a no-brainer ever since browsers first offered mouse events in DHTML. But, although mousedown, mousemove, and mouseup made it possible, the implementation has been limited to the bounds of the browser window. Additionally, since these events refer only to the object being dragged, there’s a challenge to find the subject of the drop when the interaction is completed.

Of course, that doesn’t prevent most modern JavaScript frameworks from abstracting away most of the problems and throwing in some flourishes while they’re at it. But, wouldn’t it be nice if browsers offered first-class support for drag and drop, and maybe even extended it beyond the window sandbox?

As it turns out, this very wish is answered by the HTML 5 specification section on new drag-and-drop events, and Firefox 3.5 includes an implementation of those events.

If you want to jump straight to the code, I’ve put together some simple demos of the new events.

I’ve even scratched an itch of my own and built the beginnings of an outline editor, where every draggable element is also a drop target—of which there could be dozens to hundreds in a complex document, something that gave me some minor hair-tearing moments in the past while trying to make do with plain old mouse events.

And, all the above can be downloaded or cloned from a GitHub repository I’ve created expecially for this article.

The New Drag and Drop Events

So, with no further ado, here are the new drag and drop events, in roughly the order you might expect to see them fired:

dragstart

A drag has been initiated, with the dragged element as the event target.

drag

The mouse has moved, with the dragged element as the event target.

dragenter

The dragged element has been moved into a drop listener, with the drop listener element as the event target.

dragover

The dragged element has been moved over a drop listener, with the drop listener element as the event target. Since the default behavior is to cancel drops, returning false or calling preventDefault() in the event handler indicates that a drop is allowed here.

dragleave

The dragged element has been moved out of a drop listener, with the drop listener element as the event target.

drop

The dragged element has been successfully dropped on a drop listener, with the drop listener element as the event target.

dragend

A drag has been ended, successfully or not, with the dragged element as the event target.

Like the mouse events of yore, listeners can be attached to elements using addEventListener() directly or by way of your favorite JS library.

Consider the following example using jQuery, also available as a live demo:

    <div id="newschool">
        <div class="dragme">Drag me!</div>
        <div class="drophere">Drop here!</div>
    </div>
 
    <script type="text/javascript">
        $(document).ready(function() {
            $('#newschool .dragme')
                .attr('draggable', 'true')
                .bind('dragstart', function(ev) {
                    var dt = ev.originalEvent.dataTransfer;
                    dt.setData("Text", "Dropped in zone!");
                    return true;
                })
                .bind('dragend', function(ev) {
                    return false;
                });
            $('#newschool .drophere')
                .bind('dragenter', function(ev) {
                    $(ev.target).addClass('dragover');
                    return false;
                })
                .bind('dragleave', function(ev) {
                    $(ev.target).removeClass('dragover');
                    return false;
                })
                .bind('dragover', function(ev) {
                    return false;
                })
                .bind('drop', function(ev) {
                    var dt = ev.originalEvent.dataTransfer;
                    alert(dt.getData('Text'));
                    return false;
                });
        });
    </script>

Thanks to the new events and jQuery, this example is both short and simple—but it packs in a lot of functionality, as the rest of this article will explain.

Before moving on, there are at least three things about the above code that are worth mentioning:

• Drop targets are enabled by virtue of having listeners for drop events. But, per the HTML 5 spec, draggable elements need an attribute of draggable="true", set either in markup or in JavaScript.

  Thus, $('#newschool .dragme').attr('draggable', 'true').

• The original DOM event (as opposed to jQuery’s event wrapper) offers a property called dataTransfer. Beyond just manipulating elements, the new drag and drop events accomodate the transmission of user-definable data during the course of the interaction.
• Since these are first-class events, you can apply the technique of Event Delegation.

  What’s that? Well, imagine you have a list of 1000 list items—as part of a deeply-nested outline document, for instance. Rather than needing to attach listeners or otherwise fiddle with all 1000 items, simply attach a listener to the parent node (eg. the <ul> element) and all events from the children will propagate up to the single parent listener. As a bonus, all new child elements added after page load will enjoy the same benefits.

  Check out this demo, and the associated JS code to see more about these events and Event Delegation.

Using dataTransfer

As mentioned in the last section, the new drag and drop events let you send data along with a dragged element. But, it’s even better than that: Your drop targets can receive data transferred by content objects dragged into the window from other browser windows, and even other applications.

Since the example is a bit longer, check out the live demo and associated code to get an idea of what’s possible with dataTransfer.

In a nutshell, the stars of this show are the setData() and getData() methods of the dataTransfer property exposed by the Event object.

The setData() method is typically called in the dragstart listener, loading dataTransfer up with one or more strings of content with associated recommended content types.

For illustration, here’s a quick snippet from the example code:

    var dt = ev.originalEvent.dataTransfer;    
    dt.setData('text/plain', $('#logo').parent().text());
    dt.setData('text/html', $('#logo').parent().html());
    dt.setData('text/uri-list', $('#logo')[0].src);

On the other end, getData() allows you to query for content by type (eg. text/html followed by text/plain). This, in turn, allows you to decide on acceptable content types at the time of the drop event or even during dragover to offer feedback for unacceptable types during the drag.

Here’s another example from the receiving end of the example code:

    var dt = ev.originalEvent.dataTransfer;    
    $('.content_url .content').text(dt.getData('text/uri-list'));
    $('.content_text .content').text(dt.getData('text/plain'));
    $('.content_html .content').html(dt.getData('text/html'));

Where dataTransfer really shines, though, is that it allows your drop targets to receive content from sources outside your defined draggable elements and even from outside the browser altogether. Firefox accepts such drags, and attempts to populate dataTransfer with appropriate content types extracted from the external object.

Thus, you could select some text in a word processor window and drop it into one of your elements, and at least expect to find it available as text/plain content.

You can also select content in another browser window, and expect to see text/html appear in your events. Check out the outline editing demo and see what happens when you try dragging various elements (eg. images, tables, and lists) and highlighted content from other windows onto the items there.

Using Drag Feedback Images

An important aspect of the drag and drop interaction is a representation of the thing being dragged. By default in Firefox, this is a “ghost” image of the dragged element itself. But, the dataTransfer property of the original Event object exposes the method setDragImage() for use in customizing this representation.

There’s a live demo of this feature, as well as associated JS code available. The gist, however, is sketched out in these code snippets:

    var dt = ev.originalEvent.dataTransfer;    
 
    dt.setDragImage( $('#feedback_image h2')[0], 0, 0);
 
    dt.setDragImage( $('#logo')[0], 32, 32); 
 
    var canvas = document.createElement("canvas");
    canvas.width = canvas.height = 50;
 
    var ctx = canvas.getContext("2d");
    ctx.lineWidth = 8;
    ctx.moveTo(25,0);
    ctx.lineTo(50, 50);
    ctx.lineTo(0, 50);
    ctx.lineTo(25, 0);
    ctx.stroke();
 
    dt.setDragImage(canvas, 25, 25);

You can supply a DOM node as the first parameter to setDragImage(), which includes everything from text to images to <canvas> elements. The second two parameters indicate at what left and top offset the mouse should appear in the image while dragging.

For example, since the #logo image is 64×64, the parameters in the second setDragImage() method places the mouse right in the center of the image. On the other hand, the first call positions the feedback image such that the mouse rests in the upper left corner.

Using Drop Effects

As mentioned at the start of this article, the drag and drop interaction has been used to support actions such as copying, moving, and linking. Accordingly, the HTML 5 specification accomodates these operations in the form of the effectAllowed and dropEffect properties exposed by the Event object.

For a quick fix, check out the a live demo of this feature, as well as the associated JS code.

The basic idea is that the dragstart event listener can set a value for effectAllowed like so:

    var dt = ev.originalEvent.dataTransfer;
    switch (ev.target.id) {
        case 'effectdrag0': dt.effectAllowed = 'copy'; break;
        case 'effectdrag1': dt.effectAllowed = 'move'; break;
        case 'effectdrag2': dt.effectAllowed = 'link'; break;
        case 'effectdrag3': dt.effectAllowed = 'all'; break;
        case 'effectdrag4': dt.effectAllowed = 'none'; break;
    }

The choices available for this property include the following:

none

no operation is permitted

copy

copy only

move

move only

link

link only

copyMove

copy or move only

copyLink

copy or link only

linkMove

link or move only

all

copy, move, or link

On the other end, the dragover event listener can set the value of the dropEffect property to indicate the expected effect invoked on a successful drop. If the value does not match up with effectAllowed, the drop will be considered cancelled on completion.

In the a live demo, you should be able to see that only elements with matching effects can be dropped into the appropriate drop zones. This is accomplished with code like the follwoing:

    var dt = ev.originalEvent.dataTransfer;
    switch (ev.target.id) {
        case 'effectdrop0': dt.dropEffect = 'copy'; break;
        case 'effectdrop1': dt.dropEffect = 'move'; break;
        case 'effectdrop2': dt.dropEffect = 'link'; break;
        case 'effectdrop3': dt.dropEffect = 'all'; break;
        case 'effectdrop4': dt.dropEffect = 'none'; break;
    }

Although the OS itself can provide some feedback, you can also use these properties to update your own visible feedback, both on the dragged element and on the drop zone itself.

Conclusion

The new first-class drag and drop events in HTML5 and Firefox make supporting this form of UI interaction simple, concise, and powerful in the browser. But beyond the new simplicity of these events, the ability to transfer content between applications opens brand new avenues for web-based applications and collaboration with desktop software in general.

The border-radius property is probably one of the most interesting parts of the CSS3 specification, where it allows you to create rounded corners on elements. For example:

div {
  border-radius: 10px;
  -moz-border-radius: 10px;
  -webkit-border-radius: 10px;
}

In Firefox 3.5, -moz-border-radius has been updated to match the latest revision to the CSS3 specification. Thus now, we can create elliptical borders.

So, what does that mean? According to the specification, the syntax is:

-moz-border-radius: <border-radius>{1,4} [ / <border-radius>{1,4}]?

It shows the set of values accepted for this property, plus an optional slash with another set of values. This is where the magic comes from. If two sets of values are given, values before the slash set the horizontal radius and values after it set the vertical radius.

This opens a few interesting possibilities. This demo shows some experiments on certain types of shapes that you can play with.

View the Demo in Firefox 3.5

In the demo, it’s not only possible to modify the border width, style and radius, but also the style as well, such as ridge, double and groove. As of now, dotted and dashed doesn’t work yet and being rendered as solid. For more information, see Mozilla bug 431176 .

Starting with Firefox 3.5, you can embed a video in a web page like an image. This means video is now a part of the document, and finally, a first class citizen of the Open Web. Like all other elements, you can use it with CSS and JavaScript. Let’s see what this all means …

The Basics

First, you need a video to play. Firefox supports the Theora codec (see here to know all media formats supported by the audio and video elements).

Add the video to your document:

<video id="myVideo" src="myFile.ogv"/>

You might need to add some “fallback” code if the browser doesn’t support the video tag. Just include some HTML (which could be a warning, or even some Flash) inside the video tag.

<video id="myVideo" src="myFile.ogv">
<strong>Your browser is not awesome enough!</strong>
</video>

Here’s some more information about the fallback mechanism.

HTML Attributes

You can find all the available attributes here.

Some important attributes:

• autoplay: The video will be played just after the page loads.
• autobuffer: By default (without this attribute), the video file is not downloaded unless you click on the play button. Adding this attribute starts downloading the video just after the page loads.
• controls: by default (without this attribute), the video doesn’t include any controls (play/pause button, volume, etc.). Use this attribute if you want the default controls.
• height/width: The size of the video

Example:

<video id="myVideo" src="myFile.ogv" 
   autobuffer="true" controls="true"/>

You don’t have to add the “true” value to some of these attributes in HTML5, but it’s neater to do so. If you’re not in an XML document, you can simply write:

<video id="myVideo" src="myFile.ogv" autobuffer controls/>

JavaScript API

Like any other HTML element, you have access to the video element via the Document Object Model (DOM):

var myVideo = document.getElementById("myVideo");

Once you obtain a handle to the video element, you can use the JavaScript API for video.

Here is a short list of some useful methods and properties (and see here for more of the DOM API for audio and video elements):

• play() / pause(): Play and pause your video.
• currentTime: The current playback time, in seconds. You can change this to seek.
• duration: The duration of the video.
• muted: Is the sound muted?
• ended: Has the video ended?
• paused: Is the video paused?
• volume: To determine the volume, and to change it.

Example:

<button onclick="myVideo.play()">Play</button>
<button onclick="myVideo.volume = 0.5">Set Volume</button>
<button onclick="alert(myVideo.volume)">Volume?</button>

Events

You know how to control a video (play/pause, seek, change the volume, etc.). You have almost everything you need to create your own controls. But you need some feedback from the video, and for that, let’s see the different events you can listen to:

• canplay: The video is ready to play
• canplaythrough: The video is ready to play without interruption (if the download rate doesn’t change)
• load: The video is ready to play without interruption (the video has been downloaded entirely)
• ended: The video just ended
• play: The video just started playing
• pause: The video has been paused
• seeking: The video is seeking (it can take some seconds)
• seeked: The seeking process just finished
• timeupdate: While the video is playing, the currentTime is updated. Every time the currentTime is updated, timeupdate is fired.

Here’s a full list of events.

For example, you can follow the percentage of the video that has just been played:

function init() 
{
  var video = document.getElementById("myVideo");
  var textbox = document.getElementById("sometext");
  video.addEventListener("timeupdate", function() {
  textbox.value = Math.round(100 * (video.currentTime / video.duration)) + "%"; }
 
}

<video id="myVideo" src="myFile.ogv" 
            autoplay="true" onplay="init()"/>
<input id="sometext"/>

Showing all this in action, here’s a nice open video player using the Video API.

Now that you’re familiar with some of the broad concepts behind the Video API, let’s really delve into the video as a part of the Open Web, introducing video to CSS, SVG, and Canvas.

CSS and SVG

A video element is an HTML element. That means you can use CSS to style it.

A simple example: using the CSS Image Border rule (a new CSS 3 feature introduced in Firefox 3.5). You can view how it works on the Mozilla Developer Wiki.

And obviously, you can use it with the video tag:

 
<video id="myVideo" src="myFile.ogv" 
style="-moz-border-image: 
           url(tv-border.jpg) 25 31 37 31 stretch stretch; 
           border-width: 20px;"/>

One of my demos uses this very trick.

Since Firefox 3.5 provides some new snazzy new CSS features, you can do some really fantastic things. Take a look at the infamous washing machine demo, in which I subject an esteemed colleague to some rotation.

It uses some CSS rules:

• filter, clip-path.
• css transform.

And some SVG:

• SVG & foreignobject.

Because the video element is like any other HTML element, you can add some HTML content over the video itself, like I do in this demo. As you can see, there is a <div> element on top of the video (position: absolute;).

Time for a Break

Well, we’ve just seen how far we can go with the video element, both how to control it and how to style it. That’s great, and it’s powerful. I strongly encourage you to read about the new web features available in Firefox 3.5, and to think about what you can do with such features and the video element.

You can do so much with the power of the Open Web. You can compute the pixels of the video. You can, for example, try to find some shapes in the video, follow the shapes, and draw something as an attachment to these shapes. That’s what I do here! Let’s see how it actually works.

Canvas & Video

Another HTML 5 element is canvas. With this element, you can draw bitmap data (see the canvas reference, and I strongly suggest this canvas overview). But something you might not know is that you can copy the content of an <img/> element, a <canvas/> element and a <video/> element.

That’s a really important point for the video element. It gives you a way to play with the values of the pixels of the video frames.

You can do a “screenshot” of the current frame of the video in a canvas.

function screenshot() {
 var video = document.getElementById("myVideo");
 var canvas = document.getElementById("myCanvas");
 var ctx = canvas.getContext("2d");
 
 ctx.drawImage(video, 0, 0, canvas.width, canvas.height);
}

<video id="myVideo" src="myFile.ogv" autoplay="true" with="600" height="400"/>
<canvas id="myCanvas" with="600" height="400"/>
<button onclick="screenshot()">Copy current frame to canvas</button>

You can first apply a transformation to your canvas (see the documentation). You can also copy a thumbnail of the video.

If you draw every frame in a canvas, your canvas will look like a video element. And you can draw what you want in this canvas, after drawing the frame. That’s what I do in this demo.

Once you have a video frame in your canvas, you can compute the values of the pixels.

Some things you should know if you want to compute the pixels values of a frame:

• you can’t use this mechanism with a video from another domain.
• you can’t use this mechanism with a video from a file:/// URL (which would be useful during the development of your web application). But you can change this behavior for testing: in about:config, change the value of “security.fileuri.strict_origin_policy” to “false”. But be very careful! editing about:config — that’s an expert feature!
• There are two ways to display the result of your application on the top of the video:
  • use your canvas as a video (if you draw the frame every time), and then draw directly into the canvas
  • use a transparent canvas on the top of the video
• the canvas element can be “display: none”
• the video element can be “display: none”

About JavaScript

For the image processing, you will need to do a lot of computation. Here are some tricks:

• copy your frame in a small canvas. If the canvas is three times smaller than the video, it means nine times fewer pixels to compute.
• avoid recursion. In a recursion, the script engine doesn’t use the JIT optimization.
• if you want to do a distance between colors, use the L.A.B colorspace.
• if you want to find the center of an object, compute its centroid. See the “computeFrame” function that I use in this JavaScript snippet for my demo.
• if the algorithm is really heavy, you can use a Worker thread, but take into account that you will need to send the content of the canvas to the thread. It’s a big array, and objects are automatically JSONified before being sent. It can take a while.

Conclusion

As you can see, you can do powerful things with the video element, the canvas element, CSS3, SVG and the new JavaScript engine. You have everything in your hands to create a completely new way to use Video on the web. It’s up to you now — upgrade the web!

References

• LAB Color Space – for color space distances.
• How to use threads in your JavaScript code.
• The centroid definition – good way to compute the center of an object from its pixels.
• TraceMonkey – the new Firefox JavaScript engine
• Pixel manipulation with canvas.
• The “Dynamic Content Injection” demo.
• The canvas.drawImage API – useful to inject the frame of a video into a canvas element.
• How to do transformation in your canvas.
• A good overview of the canvas API.
• Everything you need to know about canvas.
• All the new OpenWeb features in Firefox 3.5.
• Layering a div over a video.
• Transformation with CSS 3.
• Including HTML inside SVG.
• Applying SVG effects to HTML content.
• A 100% OpenVideo player.
• The reference for the video and the audio HTML5 tags.
• The Javascript API for the video and the audio tags.
• The video HTML attributes.
• The video fallback mechanism.
• Media formats supported by the audio and video elements.
• My demos.

This is the article that accompanies the demo below, showing the use of CSS3 selectors implemented in Firefox 3.5 for easy and stylish tables.

View the Demo | CSS3 Selectors for Quick and Pretty Tables

Basic HTML Table

First, we start by writing a simple HTML table. Note: we do not have any classes or ids, which is what makes this all so fantastic:

<table>
  <thead>
    <th>...</th>
    ...
    <th>...</th>
  </thead>
  <tbody>
    <tr>
      <td>...</td>
      ...
      <td>...</td>
    <tr>
    ...
    <tr>
      <td>...</td>
      ...
      <td>...</td>
    <tr>
  </tbody>
</table>

And now, let us write some CSS to make the plain old table look a bit more stylish:

table {
    font: 90%/1.5em "Lucida Grande", Geneva,
                    "DejaVu Sans", "Bitstream Vera Sans", AnjaliOldLipi,
                    "Lucida sans", "Trebuchet MS", Arial, Verdana;
    text-align: center;
    border: 4px black double;
    border-spacing: 0;
    -moz-border-radius: 12px;
    -moz-box-shadow: #6a3d37 5px 5px 6px;
    -webkit-border-radius: 12px;
    -webkit-box-shadow: #6a3d37 5px 5px 6px;
    border-radius: 12px;
    box-shadow: #6a3d37 5px 5px 6px;
    background: #b59d5c
}

The use of border-spacing, border-radius and box-shadow properties make for a quick and easy way to inject a little beauty into the table.

First Selection

Now, we would like to style all the th tags. This is easy with CSS Selectors:

th {
    color: #fff;
    font-size: 110%;
    text-shadow: #6a3d37 2px 2px 2px
}

Reminder: if we write table th we intend to select all the th elements which are child elements of the the table element, and if we write thead > th, we intend to select all the th elements which are direct children of the thead element. Well, it was just a reminder :-)

More Nuanced Selection

The th tag represents a table header. We would like to select the first table header. Hmm… maybe we should use the first-of-type pseudo-class. It represents an element that is the first sibling of its type in the list of children of its parent element. So now we have:

 
th:first-of-type {
font-weight: bold;
font-style: italic
}

Even and odd rows

A recurrent problem with tables is: how to select even and odd rows? The solution is the nth-child() pseudo-class. All these pseudo-classes understand the an+b syntax — to select all the even elements, we use 2n; to select all the odd elements, we use the 2n+1 elements; to select all third elements, we use 3n. In other words, this matches the bth child of an element after all the children have been split into groups of a elements each.

So, let’s style the even and odd rows:

 
tr:nth-child(odd) {
    color: #e0d8cb;
    background: #474644
}
 
tr:nth-child(even) {
    color: #6a3d37
}

Padding on first and last columns

Now, we want to add a padding to the first and the last column. I remind you again that we do not have any classes or IDs, and that the number of columns is unknown.

The solution is the first-of-type and the last-of-type pseudo-classes. We select all the first and last th and td like this:

 
th:first-of-type,
td:first-of-type {
    padding: 0 0 0 4em
}
 
th:last-of-type,
td:last-of-type {
    padding: 0 4em 0 0
}

The two last rows

Now, we would like to combine many pseudo-classes (and introduce a new one).

Let’s say we would like to select row number 1, number 4 and number 7. The mathematical expression is not so simple as we hoped. The tips is to split our rows in groups of 3, like this: 3n. But, that will select rows number 3, 6 and 9. Almost there! In these groups, we will select the first element, so 3n+1 (or 3n-2 if you like to complicate things).

That’s good, but that will select a row in our two last non-opaque rows. Ideally, we should say: “select the first row of each groups of three rows, but not the last row (which will be selected, because the last is the 7th row).” Simple. We are going to use the not pseudo-class, combined with the last-child pseudo-class (or last-of-type, which also works here).

Thus:

 
tr:nth-last-child(-n+2) {
    opacity: .75
}
 
tr:nth-child(3n+1):not(:last-child) td {
    text-shadow: red 0 0 8px
}

The final source is:

table {
    display: table;
    font: 90%/1.5em "Lucida Grande", Geneva,
                    "DejaVu Sans", "Bitstream Vera Sans", AnjaliOldLipi,
                    "Lucida sans", "Trebuchet MS", Arial, Verdana;
    text-align: center;
    border: 4px black double;
    border-spacing: 0;
    -moz-border-radius: 12px;
    -moz-box-shadow: #6a3d37 5px 5px 6px;
    -webkit-border-radius: 12px;
    -webkit-box-shadow: #6a3d37 5px 5px 6px;
    border-radius: 12px;
    box-shadow: #6a3d37 5px 5px 6px;
    background: #b59d5c
}
 
th {
    color: #fff;
    font-size: 110%;
    text-shadow: #6a3d37 2px 2px 2px
}
 
    th:first-of-type {
        font-weight: bold;
        font-style: italic
    }
 
tr:nth-child(odd) {
    color: #e0d8cb;
    background: #474644
}
 
tr:nth-child(even) {
    color: #6a3d37
}
 
th:first-of-type,
td:first-of-type {
    padding: 0 0 0 4em
}
 
th:last-of-type,
td:last-of-type {
    padding: 0 4em 0 0
}
 
tr:nth-last-child(-n+2) {
    opacity: .75
}
 
tr:nth-child(3n+1):not(:last-child) td {
    text-shadow: red 0 0 8px
}

View this demo step by step.

Standardization of many of these features often came after much spirited discussion, and we’re pleased to see the prominent placement of HTML5 as a key strategic initiative by major web development companies. Indeed, exciting new web applications hold a great deal of promise, and really showcase what the future of the web platform holds in store for aspiring developers. Many herald the triumphant arrival of the browser as the computer, an old theme that gets bolstered with the arrival of attractive HTML5 platform features that are implemented across Safari, Chrome, Opera, and of course, Firefox (with IE8 getting an honorable mention for having both some HTML5 features and some ECMAScript, 5th Edition features).

Call it what you will — Web 5.0, Open Web 5th Generation (wince!), or, (R)evolution # 5, the future is now. But lest anyone forget, HTML5 is not a completed standard yet, as the W3C was quick to point out. The editor doesn’t anticipate completion till 2010. The path taken from the start of what is now called HTML5 to the present-day era of (very welcome) hype has been a long one, and Mozilla has been part of the journey from the very beginning.

For one thing, we were there to point out, in no uncertain terms, that the W3C had perhaps lost its way. Exactly 5 summers ago (again, with that magic number!), it became evident that the W3C was no longer able to serve as sole custodian of the standards governing the open web of browser-based applications, so Mozilla, along with Opera, started the WHATWG. Of course, back then, we didn’t call it HTML5, and while Firefox itself made a splash in 2004, the steps taken towards standardization were definitive but tentative. Soon, other browser vendors joined us, and by the time the reconciliation with W3C occurred two years later, the innovations introduced into the web platform via the movement initiated by Mozilla had gained substantial momentum.

The net result is a specification that is not yet complete called “HTML5″ which is implemented piecemeal by most modern browsers. The features we choose to implement as an industry are in response to developers, and our modus operandi is (for the most part) in the open. Mozilla funds the HTML5 Validator, producing the first real HTML5 parser, which now drives W3C’s markup validation for HTML5. That parser has made its way back into Firefox. It’s important to note that capabilities that are of greatest interest (many of which are showcased on this blog) are not only developed within the HTML5 specification, but also as part of the W3C Geolocation WG, the Web Apps WG, and the CSS WG.

The release of Firefox 3.5, along with updates to other modern browsers, seems to declare that HTML5 has arrived. But with the foresight that comes with having been around this for a while, we also know that we have a lot of work ahead of us. For one thing, we’ve got to finish HTML5, or at least publish a subset of it that we all agree is ready for implementation, soon. We’ve also got to ensure that accessibility serves as an important design principle in the emerging web platform, and resolve sticky differences here. Also, an open standard does not an open platform make, as debates about web fonts and audio/video codecs show. We’ve got a lot of work ahead of us, but for now, 5 years after the summer we started the ball rolling, we’re enjoying the hype around (R)evolution Number 5.

CSS Media Queries were originally a proposal submitted to the CSS Working Group by Opera Software and are now implemented in Firefox 3.5. In short, Media Queries extend the media declaration attached to a stylesheet to allow matching based on the rendering device’s intrinsic properties.

Let’s take a link element declaring a stylesheet inside an HTML document:

<link rel="stylesheet" type="text/css" href="style.css"
      media="screen">

Now imagine you want this stylesheet to apply to the document if and only if the width of the content window is less than 300 pixels… CSS Media Queries make it simple to declare:

<link rel="stylesheet" type="text/css" href="style.css"
      media="screen and (max-width: 300px)">

Available properties include viewport’s width and height, device’s width and height, orientation (portrait or landscape), viewport’s aspect ratio, device’s aspect ratio, colormap, resolution and type of device.

It’s then very easy to have one single web page ready for consumption on a wide variety of devices, ranging from mobile devices to monochrome tty displays.

When viewing the demo, please don’t forget to resize the window from large to very very small (less than 100px!) to see it in action.

View the Demo in Firefox 3.5

Loading via planet, RSS or a reader without JavaScript? Click here instead.

first-of-type and last-of-type

These two pseudo-classes allow you to select the first and last item in a list of siblings within a particular element. You can use this to color the first item in a list, use opacity to make the last paragraph on a page fade out or a number of other things. Here’s an example that sets small caps on the first paragraph of a document:

#type-ex1 div:first-of-type {
    font-variant: small-caps;
}

only-of-type

only-of-type is similar to the previous two, but only selects an element if it has no siblings with the same name. Here’s a somewhat contrived example* that will hide single images inside of a div. If there’s more than one image in the div they will be visible:

.type-ex2 img:only-of-type {
    display: none;
}

And that’s it. Enjoy!

[ * Note: if someone can come up with a better example for only-of-type I'm all ears. There are very few examples of where this is useful. ]

The demo uses XML to describe the various segments of a piece of music – Florence Baschet’s StreicherKreis (Circle of Strings). The music itself is a combination of stringed instruments and electronic effects. From the XML, SVG is generated for each section of the music. You can click on each section to listen to that part of the piece and a description is shown on how that particular section was created.

As far as demos go, this is relatively simple. But it’s worth highlighting because it shows how easy it is to build a timeline around a piece of music and add descriptive information. In this case, it’s information meant to teach people how a particular effect was created. But it could be anything, from showing different camera angles of people playing the music to links about different covers of a popular piece. Opening up media to the web means that we can combine it with text, images and other media. This is just a small example.

View the Demo in Firefox 3.5

In the past we’ve always published a list of web developer features for a release. But the feature list for developers for 3.5 is so long that we thought it might be useful to try and break it down into easy-to-swallow bites. And my suspicion, given the number of comments and the number of daily views this weblog gets, is that it’s actually been useful to people.

We’ve been writing articles on features to get people informed on what the changes are, but we’ve also been featuring demos to get people inspired. Some of them have been built by people at the Mozilla core, but the really great thing that we’ve seen is that people have started to show up and build demos for our project without us having to ask. I think that this points to the fact that people are really bullish on the future of the web. People want to see change, they want to see improvements and they understand that every individual has a role to play in that. To everyone involved, I say thank you. You’re helping to improve the web, one action at a time.

We’re about half way through our 35 days posts and we will continue to make posts. And we’re still looking for help. If you want to get involved, you’re more than welcome to join us. There’s always something that you can do – follow our twitter account, write a demo, document a feature or, really, just start using this stuff. We’re helping to write the future of the web and everyone has a part to play.

In HTML, the script element allows authors to include dynamic script in their documents. The defer attribute is boolean attribute that indicates how the script should be executed. If the defer attribute is present, then the script is executed when the page has finished parsing. The element is added to the end of the list of scripts that will execute when the document has finished parsing. Think about a FIFO processing queue : the first script element to be added to the queue will be the first script to be executed, then processing proceeds sequentially in the same order.

There is one very good reason for using the defer attribute: performance. If you include a script element in your HTML page the script must be evaluated immediately while the page is being parsed. This means that objects have to be created, styles must be flushed, etc. This can make page loading slower. The defer attribute implies that the script has no side effects on the document as it’s being loaded and can safely be evaluated at the end of the page load.

The defer attribute was first introduced in Internet Explorer 4, and added in the HTML 4 specification.

A simple test.

Here is a simple first test to see how the attribute works. The following lines are in the head element of a page:

‹script›
	var test1 = "Test 1 : fail";
‹/script›
‹script defer›
  	console.log(test1);
‹/script›
‹script›
	test1 = "Test 1 : pass";
‹/script›

If the defer attribute for the script element is correctly implemented the browser will:

1. Render the page.
2. Execute the second script element after all the others.
3. Display “Test 1 : pass” on the Firebug console.

If the console displays “Test 1 : fail” it’s because the scripts are executed in the same order as in the source code.

Note that the correct syntax for XHTML documents is:

<script defer="defer"></script>

A more advanced test

This second test is a way to see how the feature works in a webpage with multiple script elements inserted:

• Inline in the head and body elements
• External via src attribute in head and body elements
• With dynamic DOM insertion

Here is partial source code of a webpage that tests how defer affects script loading and parsing order:

‹!doctype html›
‹html›
    ‹head›
        ‹title› Test 2 ‹/title›
        ‹script› var test2 = "Test 2 :\n\n"; ‹/script›
        ‹script› document.addEventListener("DOMContentLoaded",
                function(){
                        test2 += "\tDOMContentLoaded\n";
                }, false);
        ‹/script›
        ‹script defer› test2 += "\tInline HEAD deferred\n"; ‹/script›
        ‹script› test2 += "\tInline HEAD\n"; ‹/script›
        ‹script src="script1.js" defer›
                // External HEAD deferred (script1.js) 
        ‹/script›
        ‹script src="script2.js"›
                // External HEAD  (script2.js)
        ‹/script›
	‹script›
            // Dynamic DOM insertion of a script (script3.js)
            head = document.getElementsByTagName('head')[0];
            script3 = document.createElement('script');
            script3.setAttribute('src', 'script3.js');
            head.appendChild(script3);
            // Dynamic DOM insertion of a deferred script (script4.js)
            script4 = document.createElement('script');
            script4.setAttribute('defer', 'defer');
            script4.setAttribute('src', 'script4.js');
            head.appendChild(script4);
	‹/script›
	‹script defer›
            // Deferred dynamic DOM insertion of a script (script5.js)
            head = document.getElementsByTagName('head')[0];
            script5 = document.createElement('script');
            script5.setAttribute('src', 'script5.js');
            head.appendChild(script5);
            // Deferred dynamic DOM insertion of a deferred script
            // (script6.js)
            script6 = document.createElement('script');
            script6.setAttribute('defer', 'defer');
            script6.setAttribute('src', 'script6.js');
            head.appendChild(script6);
	‹/script›
    ‹/head›
    ‹body onload="test2 += '\tBody onLoad\n';"›
        ‹script defer› test2 += "\tInline BODY deferred\n"; ‹/script›
        ‹script› test2 += "\tInline BODY\n"; ‹/script›
 
	... other body content ...
 
		<a onclick="alert(test2);">Launch test 2</a>
 
	... other body content ...
 
        ‹script src="script7.js" defer›
                // External BODY deferred (script7.js)
        ‹/script›
        ‹script src="script8.js"›
                // External BODY (script8.js)
        ‹/script›
    ‹/body›
‹/html›

When you click on the “Launch test 2″ link in the document a pop-up appears with a list in it. This list shows the order of script elements loaded during the session.

The test also displays the DOMContentLoaded and body.onload events when they are fired.

If the defer attribute is correctly implemented in the browser, all the deferred lines should be near the bottom of the list.

Results of the second test for each browser are below (deferred scripts are in green color) :

• The defer attribute behavior in the Firefox 3.5 browser is correct:
  1. Inline HEAD
  2. External HEAD (script2.js)
  3. Dynamic DOM insertion of a script (script3.js)
  4. Inline BODY
  5. External BODY (script8.js)
  6. Inline HEAD deferred
  7. External HEAD deferred (script1.js)
  8. Dynamic DOM insertion of a deferred script (script4.js)
  9. Inline BODY deferred
  10. External BODY deferred (script7.js)
  11. Deferred dynamic DOM insertion of a script (script5.js)
  12. Deferred dynamic DOM insertion of a deferred script (script6.js)
  13. DOMContentLoaded
  14. Body onLoad

• The defer attribute behavior in the IE 8 browser is erratic: the order is different at each reload :
  1. Inline HEAD
  2. External HEAD (script2.js)
  3. Inline BODY
  4. External BODY (script8.js)
  5. Dynamic DOM insertion of a script (script3.js)
  6. Dynamic DOM insertion of a deferred script (script4.js)
  7. Inline HEAD deferred
  8. External HEAD deferred (script1.js)
  9. Inline BODY deferred
  10. External BODY deferred (script7.js)
  11. Body onLoad
  12. Deferred dynamic DOM insertion of a script (script5.js)
  13. Deferred dynamic DOM insertion of a deferred script (script6.js)

• The defer attribute behavior in a WebKit browser (Safari 4.0) is erratic : the order is different at each reload :
  1. Inline HEAD deferred
  2. Inline HEAD
  3. External HEAD deferred (script1.js)
  4. External HEAD (script2.js)
  5. Inline BODY deferred
  6. Inline BODY
  7. External BODY deferred (script7.js)
  8. Deferred dynamic DOM insertion of a script (script5.js)
  9. Dynamic DOM insertion of a deferred script (script4.js)
  10. Deferred dynamic DOM insertion of a deferred script (script6.js)
  11. Dynamic DOM insertion of a script (script3.js)
  12. External BODY (script8.js)
  13. DOMContentLoaded
  14. Body onLoad

• The defer attribute behavior in the Opera 10.00 Beta browser:
  1. Inline HEAD deferred
  2. Inline HEAD
  3. External HEAD deferred (script1.js)
  4. External HEAD (script2.js)
  5. Dynamic DOM insertion of a script (script3.js)
  6. Dynamic DOM insertion of a deferred script (script4.js)
  7. Deferred dynamic DOM insertion of a script (script5.js)
  8. Deferred dynamic DOM insertion of a deferred script (script6.js)
  9. Inline BODY deferred
  10. Inline BODY
  11. External BODY deferred (script7.js)
  12. External BODY (script8.js)
  13. DOMContentLoaded
  14. Body onLoad

We hope that this has been a useful introduction to how the defer attribute works in Firefox 3.5. The tests above will also help you predict behavior in other browsers as well.

Resources

• The fixed bug 28293 – (defer) scripts with defer attribute not deferred : bug 28293
• The “JavaScript: Defer Execution” article on the WebsiteOptimization website : http://www.websiteoptimization.com/speed/tweak/defer/
• The script article on Mozilla Developer Center : https://developer.mozilla.org/En/HTML/Element/Script
• The “script element” part of the HTML 5 Draft : http://www.whatwg.org/specs/web-apps/current-work/#script
• The “Parsing HTML documents : the end” part of the HTML 5 Draft : http://www.whatwg.org/specs/web-apps/current-work/#the-end

This demo contains another interesting effect that can be done with the HTML5 elements present in Firefox 3.5. What better way to create a cool effect than mixing the open web with pop art? This demo uses only JavaScript, <video> and <canvas>. We’ll use them to create an effect similar to the famous Andy Warhol style art on a live video.

View the Demo in Firefox 3.5

The effect is quite simple. What we want to do is to create a stamp effect based on the brightness of the image. We calculate the brightness of each pixel and separate them into 3 groups, from darkest to brightest. Them, for each of our four frames we use a different color palette to convert the original pixels into our desired colors for the best effect.

With that, we already have a cool effect, but it lacks a bit of detail:

So we mix our colors with 30% of the original image, and now we have the final version, with more shades of color while preserving the nice colorization in Andy Warhol style.

We can access the pixels of each frame of the video using <video> and <canvas>, creating a live effect while the video plays.