The final code is available on GitHub and you can see an online demo here. There is also a screencast available on YouTube:

Step 1: Getting the files into the browser

The first step to resize images in the browser is to somehow get them. For this, we can just add an element in the page and assign drag and drop event handlers to it:

s.addEventListener( 'dragover', function ( evt ) {
  evt.preventDefault();
}, false );
s.addEventListener( 'drop', getfiles, false );

Notice that we only prevent the default behaviour when we drag things over the element. This is to prevent the browser from just showing the images when we drag them in.

The getfiles() function then does the hard work of reading all the files in and sending them on to the functions that do the resizing and image generation:

function getfiles( ev ) {
  var files = ev.dataTransfer.files;
  if ( files.length > 0 ) {
    var i = files.length;
    while ( i-- ) {
      var file = files[ i ];
      if ( file.type.indexOf( 'image' ) === -1 ) { continue; }
      var reader = new FileReader();
      reader.readAsDataURL( file );
      reader.onload = function ( ev ) {
        var img = new Image();
        img.src = ev.target.result;
        img.onload = function() {
        imagetocanvas( this, thumbwidth, thumbheight, crop, background );
        };
      };
    }
  }
  ev.preventDefault();
};

The drop event gives us a property called dataTransfer which contains a list of all the files that have been dropped. We make sure that there was at least one file in the drop and then iterate over them.

If the file type was not an image (or in other words the type property of the file does not contain the string “image”) we don’t do anything with the file and continue the loop.

If the file is an image we instantiate a new FileReader and tell it to read the file as a Data URL. When the reader successfully loaded the file it fires its onload handler.

In this handler we create a new image and set its src attribute to the result of the file transfer. We then send this image to the imagetocanvas() function with the parameters to resize (in the demo these come from the form):

function imagetocanvas( img, thumbwidth, thumbheight, crop, background ) {
  c.width = thumbwidth;
  c.height = thumbheight;
  var dimensions = resize( img.width, img.height, thumbwidth, thumbheight );
  if ( crop ) {
    c.width = dimensions.w;
    c.height = dimensions.h;
    dimensions.x = 0;
    dimensions.y = 0;
  }
  if ( background !== 'transparent' ) {
    cx.fillStyle = background;
    cx.fillRect ( 0, 0, thumbwidth, thumbheight );
  }
  cx.drawImage( 
    img, dimensions.x, dimensions.y, dimensions.w, dimensions.h 
  );
  addtothumbslist( jpeg, quality );
};

This function gets the desired thumbnail size and resizes the canvas to these dimensions. This has the added benefit of wiping the canvas so that no old image data would be added to our thumbnail. We then resize the image to fit into the thumbnail using a resize() function. You can see for yourself what this one does in the source code, it just means the image gets resized to fit. The function returns an object with the width and the height of the new image and the x and y position where it should be positioned onto the canvas.

If we don’t want the full-size thumbnail but instead crop it we resize the canvas accordingly and reset x and y to 0.

If the user requested a background we fill the canvas with the colour. After that we put the image on the canvas with the x and y coordinates and the new width and height.

This takes care of creating a new thumbnail on the canvas, but we haven’t got it as an image in the document yet. To this end, we call addtothumbslist():

function addtothumbslist( jpeg, quality ) {
  var thumb = new Image(),
      url = jpeg ? c.toDataURL( 'image/jpeg' , quality ) : c.toDataURL();
  thumb.src = url;
  thumb.title = Math.round( url.length / 1000 * 100 ) / 100 + ' KB';
  o.appendChild( thumb );
};

This one creates a new image and checks if the users wanted a JPG or PNG image to be created. PNG images tend to be better quality but also bigger in file size. If a JPG was requested we call the canvas’ toDataURL() method with two parameters: the requested JPEG mime type and the quality of the image (ranging between 0 and 1 with 1 being best quality). If a PNG is wanted, we can just call toDataURL() without any parameters as this is the default.

We set the src of the image to the generated url string and add a title showing the size of the image in KB (rounded to two decimals). All that is left then is to add the thumb to the output element on the page.

That’s it, you can now drag and drop images into the browser to generate thumbnails. Right now, we can only save them one at a time (or if you have some download add-ons all at once). Would be fun to add Zip.js to the mix to offer them as a zip. I dare you! :)

More reading:

• Drag and Drop on MDN
• FileReader on MDN
• Canvas on MDN

You can see this in action using Chrome, Safari or Firefox Aurora/Nightly. Older browsers should just show a normal roll-over (and yes, the first example looks weird due to the logo transparency but it makes the effect much cooler in supporting browsers).

As people asked how this is done I thought it easiest to create a testing tool where you can play with 3D CSS yourself. Here’s a screencast where I explain how to use it and how the roll-over effects were done.

I found a few interesting bugs while creating this and will file them now. This is the benefit of playing with new tech. Enjoy!

More reading on this:

• An introduction to CSS 3D transforms
• 3D transforms at surfin safari
• The W3C draft spec

As an extra bonus, we thought it would be fun to document the step-by-step development of the calendar and release it for you to re-use or get inspired by.

You can get a white-labeled version of the calendar on GitHub and there is a two-part step-by-step instruction how it was built available: Part 1 – Part 2.

It is an example of how to build something server-side (so you can’t cheat by setting your operating system calendar ahead), enhance it with JavaScript and make it smooth by using CSS transitions. This should work for everybody and by playing each technology to its strengths, the code is very small indeed.

Enjoy, and see you tomorrow and the day after, and the day after that and…

The image example is pretty simple and was actually written by Paul Rouget as a demo in the original Firefox bug request. The main core is the HTML of it:

<section id="noninteractive" contextmenu="imagemenu">
 
  <img src="html5.png" alt="HTML5" id="menudemo">
 
    <menu type="context" id="imagemenu">
      <menuitem label="rotate" onclick="rotate()"
                icon="arrow_rotate_clockwise.png">
      </menuitem>
      <menuitem label="resize" onclick="resize()"
                icon="image-resize.png">
      </menuitem>
      <menu label="share">
        <menuitem label="twitter" onclick="alert('not yet')"></menuitem>
        <menuitem label="facebook" onclick="alert('not yet')"></menuitem>
      </menu>
    </menu>
 
</section>

As you can see you link the menu element to an element via its ID. The contextmenu attribute then points to this one. Each menu can have several menuitems. Each of those gets a textual label and a possible icon. You can also nest menu elements to create multiple layer menus. Here, we add inline onclick handlers to point to different JavaScript functions to call when the menu item gets activated. The resulting context menu looks like this:

The functionality is simple, all the rotate() and resize() functions do is add class names to the image using querySelector and classList:

function rotate() {
  document.querySelector('#menudemo').classList.toggle('rotate'); 
}
function resize() {   
  document.querySelector('#menudemo').classList.toggle('resize'); 
}

The real effect is in CSS transforms and transitions. As the image has an ID of menudemo here is what is needed in CSS to rotate and resize:

#menudemo { 
  -moz-transition: 0.2s; 
  width:200px;
}
#menudemo.rotate { 
  -moz-transform: rotate(90deg); 
}
#menudemo.resize { 
  -moz-transform: scale(0.7); 
}
#menudemo.resize.rotate { 
  -moz-transform: scale(0.7) rotate(90deg); 
}

Notice that in a real product we should of course add the other browser prefixes and go prefix-less but as the functionality now only works in Firefox, this is enough for this demo.

Detecting support and visual hinting

Now, as this is extending the normal user offerings in the browser we need to make it obvious that there is a right-click menu available. In CSS3, there is a context-menu cursor available to us. When context menus are available, this should be shown:

.contextmenu #menudemo, .contextmenu .demo {
  cursor: context-menu;
}

We test the browser for support by checking for contextmenu on the body element and for HTMLMenuItemElement in the window (this has been added as a pull request to Modernizr, too).

if ('contextMenu' in document.body && 'HTMLMenuItemElement' in window) {
  document.documentElement.classList.add('contextmenu');
} else {
  return;
}

Wouldn’t HTMLMenuItemElement be enough? Yes, but a real context menu should only offer functionality when it is sensible, and that is where contextMenu comes in.

Turning menuitems on and off depending on functionality

As a slightly more complex example, let’s add a “count words” functionality to the document. For this, we generate a counter element that will become a tooltip when the words were counted:

var counter = document.createElement('span');
counter.id = 'counter';
counter.className = 'hide';
document.body.appendChild(counter);
 
counter.addEventListener('click', function(ev){
  this.className = 'hide';
},false);

This one is hidden by default and becomes visible when the hide class is removed. To make it smooth, we use a transition:

#counter{
  position: absolute;
  background: rgba(0,0,0,0.7);
  padding:.5em 1em;
  color: #fff;
  font-weight:bold;
  border-radius: 5px;
  -moz-transition: opacity 0.4s;
}
#counter.hide{
  opacity: 0;
}

We start with two sections with context menus:

<section id="noninteractive" contextmenu="countmenu">
  <menu type="context" id="countmenu">
      <menuitem class="wordcount" label="count words"></menuitem>
  </menu>
</section>
 
<section id="interactive" contextmenu="countmenuinteractive">
  <menu type="context" id="countmenuinteractive">
      <menuitem class="wordcount" label="count words"></menuitem>
  </menu>
</section>

We then loop through all the menuitems with the class wordcount and apply the functionality.

var wordcountmenus = document.querySelectorAll('.wordcount'),
    i = wordcountmenus.length;
 
while (i--) {
  wordcountmenus[i].addEventListener('click', function(ev){
    // add functionality 
  }, false);
}

We need to find out what has been selected in the page. We do this by using getSelection() and splitting its string version at whitespace. We then show the counter by removing the hide class name.

var wordcountmenus = document.querySelectorAll('.wordcount'),
    i = wordcountmenus.length;
 
while (i--) {
  wordcountmenus[i].addEventListener('click', function(ev){
    var text = document.getSelection(),
        count = text.toString().split(/\s/).length;
    counter.innerHTML = count + ' words';
    counter.className = '';
  }, false);
}

You can see this in action in the second context menu demo. Now, the issue with this (as explained in the screencast) is that it always counts the words, regardless of the user having selected some text. What we want is the menu only to be active when there is text selected.

So in order to make our menu only become available when it makes sense we check if there is a selection in the document. Every context menu fires an event called contextmenu when it opens. So all we need to do is to subscribe to this event.

When something is selected in the document document.getSelection().isCollapsed is true. Otherwise it is false, so all we need to do is to enable or disable the menu item accordingly:

document.querySelector('#interactive').addEventListener(
  'contextmenu', function(ev) {
    this.querySelector('.wordcount').disabled =
    document.getSelection().isCollapsed;  
  },
false);

The last thing to solve is the position of the mouse to position the counter element. As the menu selection event doesn’t give us the mouse position we need to add a contextmenu handler to the whole document that positions the counter invisibly behind the menu when it is opened:

document.body.addEventListener(
  'contextmenu', function(ev) {
    counter.style.left = ev.pageX + 'px';
    counter.style.top = ev.pageY + 'px';
    counter.className = 'hide';
  }, 
false);

Further reading and resources

• The full code demo
• The Firefox bug request with lots of discussion
• Context Menus on the WHATWG
• A longer discussion on Google+ about context menus
• A jQuery Polyfill for contextmenu

The screencast is on YouTube:

The main procedure to achieve the effect is simple. First we need a semantically valuable way to show these images, in our case a HTML list with figures and figcaptions. Notice that images still need an alternative text as the figcaption can apply to several images:

<ul>
  <li class="image3d">
    <figure>
      <img src="mittens.jpg" alt="Mittens the cat">
      <figcaption>
        <p>
          <strong>Mittens</strong> loves to play with yarn and stuff.
        </p>
      </figcaption>
    </figure>
  </li>
<!-- repeated -->
</ul>

We then position the caption absolutely inside the list item, give the list item dimensions and an overflow of hidden and move the caption outside of the visible space. When the user hovers over the list item (or focuses it with a keyboard) we move the caption to the right place and lower the opacity of the image.

.positioned {
  list-style: none;
  width: 300px;
  height: 200px;
  position: relative;
}
.positioned figcaption {
  position: absolute;
  background: rgba( 0, 0, 0, 0.7 );
  color: #fff;
  border-radius: 5px;
  left: 65px;
  bottom: 10px;
  width: 230px;
}
.positioned img {
  opacity:0.4 ;    
}
.positioned figcaption strong {
  display: block;
  color: lime;
  font-weight: normal;
  font-size:22px;
  padding: 5px 0;
}
.positioned figcaption p {
  display: block;
  color: white;
  padding: 5px;
}
.interactive {
  overflow: hidden;
}
.interactive img {
  opacity: 1;
}
.interactive figcaption {
  left: 300px;
}  
.interactive:hover img, .interactive:focus img {
  opacity: 0.4;
}
.interactive:hover figcaption, .interactive:focus figcaption {
  left: 75px;
}

We make the effect smooth by adding transitions. Notice that it makes sense to list all the browser prefixes and set a prefix-less fallback. That way we don’t need to re-write code when a new browser supports them.

.smooth img {
  -webkit-transition: all 1s;
  -moz-transition:    all 1s;
  -o-transition:      all 1s;
  -ms-transition:     all 1s;
  transition:         all 1s;
}
.smooth figcaption {
  -webkit-transition: all 1s;
  -moz-transition:    all 1s;
  -ms-transition:     all 1s;
  -o-transition:      all 1s;
  transition:         all 1s;
}

In order to rotate the kitty in 3D space, we need to give the list item a perspective and rotate the image and shift it in 3D space with translate3D to avoid clipping:

.threed {
  -webkit-perspective: 800px;
  -moz-perspective:    800px;
  -ms-perspective:     800px;
  -o-perspective:      800px;
  perspective:         800px;
}
.threed:hover img, .threed:focus img {
  -webkit-transform: rotateY( 50deg ) rotateX( 10deg )
                     translate3d( 80px, -20px, -100px );
  -moz-transform:    rotateY( 50deg ) rotateX( 10deg )
                     translate3d( 80px, -20px, -100px );
  -o-transform:      rotateY( 50deg ) rotateX( 10deg )
                     translate3d( 80px, -20px, -100px );
  -ms-transform:     rotateY( 50deg ) rotateX( 10deg )
                     translate3d( 80px, -20px, -100px );
  transform:         rotateY( 50deg ) rotateX( 10deg )
                     translate3d( 80px, -20px, -100px );
}

That’s it. By adding all the browser prefixes, falling back to a simple rollover and by making sure you do the effect on hover and on focus you support all the browsers and bring the cool to the newest ones.

More reading and resources:

• Standalone example
• Step-by-step example
• GitHub source
• Detailed blog post

Today we show you a screencast of how easy BrowserID makes it to login to a web site. For this, we’ll look at how the login flow for an existing BrowserID user works the first time they log in on a new website. Our example website is OpenPhoto, a hot new photo sharing app that keeps users in control of their data.

Screencast of logging into OpenPhoto with BrowserID

Get involved:

BrowserID needs your help to grow and become a weapon of choice in the fight against insecure and annoying login systems. The great thing is that now is the time where you can be part of this.

• Implement BrowserID
• Install OpenPhoto
• Contribute to the Field Guide

You can see them in action in this demo of a rotating HTML5 logo and the screencast below:

This means now that we need your support in trying out CSS 3D examples in Firefox and add other extensions than -webkit- to your CSS 3D products and demos. To show that this is possible, we took the well-known webkit-only “poster circle” demo and made it work with Firefox nightly by adding the -moz- (and of course the other prefixes and one set of instructions without browser prefixes). Here is a slight excerpt:

-webkit-transform-style: preserve-3d;
-moz-transform-style: preserve-3d;
-o-transform-style: preserve-3d;
-ms-transform-style: preserve-3d;
transform-style: preserve-3d;

You can see this in action in the screencast below alongside Chrome and you try the demo out yourself. The slight jerkiness is actually my MacBook Air impersonating a starting jet every time I use ScreenFlow and not the browser.

To celebrate the release and to show how CSS 3D can be applied as subtle effect, have a game of pairs using your favourite browsers (and a cat) :

Oleg Romashin also spent some time to convert a few CSS 3D demos to work with Mozilla and you can check the 3D city for more “wow”.

If you are new to CSS 3D transformations here’s a good beginner course and a tool to create them.

The rotating HTML5 logo demo also shows how you can check if the currently used browser supports 3D transforms. Instead of repeating the animation frames for all the prefixes we test in JavaScript and create the CSS on the fly:

function checksupport() {
  var props = ['perspectiveProperty', 'WebkitPerspective',
               'MozPerspective', 'OPerspective', 'msPerspective'],
      i = 0,
      support = false;
  while (props[i]) {
    if (props[i] in form.style) {
      support = true;
      pfx = props[i].replace('Perspective','');
      pfx = pfx.toLowerCase();
      break;
    }
    i++;
  } 
  return support;
}
if (checksupport()) {
  var s = '';
  styles = document.createElement('style');
  s += '#stage{-'+ pfx +'-perspective: 300px;}'+
       '#logo{-'+ pfx +'-transform-style: preserve-3d;position:relative;}'+  
       '#logo.spin{-'+ pfx +'-animation: spin 3s infinite linear;}'+
       '@-'+ pfx +'-keyframes spin {'+  
       '0% {'+ 
       '-'+ pfx +'-transform: rotateX(0deg) rotateY(0deg) rotateZ(0deg);'+   
       '}'+
       '100% {'+   
       '-'+ pfx +'-transform: rotateX(0deg) rotateY(360deg)'+
       ' rotateZ(360deg);'+ 
       '}}';
  styles.innerHTML = s;
  document.querySelector('head').appendChild(styles);
}

For more information on creating your own pages that use 3D transformations, take a look at the draft specification

As always, If you find any bugs, please report them at bugzilla.mozilla.org!

So please reward our hard work bringing the third dimension to Firefox’s CSS engine by supporting and testing. Cheers!

Since the first alpha version of Tilt was announced (a Firefox extension focused on creating a 3D visualization of a webpage), a lot of work has been done to add a great number of developer-oriented features. These focus on debugging the structure of a webpage, inspecting styling and attributes for each node and seamlessly refreshing the visualization when the DOM structure changes or after contents of document are repainted.

Solve nesting problems

Tilt is useful when searching problems in the HTML structure (like finding unclosed DIV elements for example) by providing the extra third dimension, layering each node based on nesting in the DOM tree. Stacks of elements visually represent branches in the DOM, and each node can be inspected for the inner HTML contents, its computed CSS style and the attributes.

Clicking anywhere on the visualization highlights a color-coded rectangle surrounding the corresponding node. Double click shows up the source preview for that node. Tilt also tries to show the most relevant information when needed (one is most likely to inspect the attributes of an input, button or image element, for example, but can easily switch between HTML, CSS and attributes view at any time).

Minidom map

The “minidom” is a tree view representation showing a minimalistic snapshot of the document object model. Each node is assigned a color associated by tag name (blue for div, green for span etc.) and represented as a strip, along with visual markers for the id and/or class if available. Each one of these strips also has a width relative to the type, id and class name length for the respective element, and the corresponding 3D stack in the visualization has color-coded margins. The coloring for individual elements is easily changeable using the color picker near to the minidom legend.

Clicking a strip in the tree view (or directly a stack on the 3D document visualization mesh) also highlights the node with a colored quad. This behavior is a good way to relate with the Style Inspector, and a more unified interaction between Tilt and other Developer Tools is planned in the future. All of these additions make it easier to analyze the bounds of each node, along with the HTML, computed CSS and attributes.

Realtime editing

Because Tilt is able to detect when a webpage’s DOM structure changes or when a repaint is necessary, integration is seamless with existing Developer Tools. Using Tilt and Firebug or Style Editor at the same time is easy. One can enable or disable CSS properties, changing the style of a node, and the visualization changes accordingly.

To enable realtime updates for the 3D webpage, go to the Options menu and check “Refresh visualization”.

Useful for learning

Developer tools such as “view source” have always been used to help people learn about web development. The 3D view highlights the structure of a page better than a flat view, thus anyone can immediately understand the parent-child relationship between nodes in a webpage, their positioning and how the layout is influenced.

One use case for this is the Hackasaurus mashup. The X-Ray Goggles is a nice and fun tool designed to make it easier to learn about the different document node types, the “building blocks” which create a webpage.

Export

A requested feature was the ability to export the visualization as a 3D mesh, to be used in games or other 3D editors. Tilt adds the ability to export to .obj, along with a material .mtl file and a .png texture (a screenshot of the entire webpage). The open .obj format ensures the fact that the mesh can be opened with almost any editor. Here’s a ray-traced rendering of hacks.mozilla.org in Blender:

Fun with experiments

As soon as it was released, many people found clever and interesting alternative ways to interact with Tilt. One experiment was creating a 3D visualization of an image, by exporting chunks of pixels to a HTML representation. The result was a voxel-like representation, with node blocks and stacks instead of pixels. A simple Image2Tilt converter was written in JavaScript, and you can try it directly in the browser.

Accelerometer support was another addition based on community request. This shows how easy it is to add functionality that wasn’t originally planned.

You can view the source code, fork it and also contribute to the addon with ideas or feature requests on Github, at github.com/victorporof/Tilt.

Available as an addon

The latest version of Tilt can be found on Github, but you can also download Tilt as an addon from addons.mozilla.org.

For compatibility, Tilt requires WebGL capabilities. Go to get.webgl.org to check availability and troubleshoot any issues. The current version works with Firefox 6.0 to latest 10.0 Nightly releases (latest Nightly builds now also support WebGL anti-aliasing, working great with Tilt).

To start Tilt, hit Control+Shift+M (or Command+Shift+M if you’re on Mac OS), or go to Web Developer -> Tilt, available in the Firefox application menu (or the Tools menu on Mac OS). You can modify this hotkey (and other properties) from the Options menu after starting Tilt.

More information about Tilt, the development process and milestone updates can be found on blog.mozilla.com/tilt.

Future

Tilt has become an active Developer Tools project, and an ongoing effort is made to integrate it with other existing tools like Style Inspector and Style Editor (source code and latest builds). As the 3D view of a webpage has proven to be useful for debugging, this main functionality will gradually become part of Firefox in future releases.

Once you are up to speed (or refreshed your memory) why not go and build something for this month’s Developer Derby?

You can see the video on any HTML5 enabled device here (courtesy of vid.ly).

• The simple geolocation example shown in the screencast
• The W3C geolocation specs
• A polyfill to add support to older browsers
• The distance between your IP location and the browser geolocation
• The “things around you” demo shown in the screencast

We asked people from the demo scene to have a go at web technologies and (with WebGL being the absolute winner of course) managed to collect over 100 submissions. Now the judges have spoken and we picked the winners in the categories of Main Demo, Single Effect, Audio Demo, Animated GIF and pure CSS demo.

Amongst other great examples of using technology in a purely creative way unhindered by real life application needs here are the winners of Demo and Single Effect:

Main Demo: Akemi

Single Effect: WebGL Water Simulation

Demo Link or screen capture:

We congratulate all the winners and thank all those who contributed. Playing with technology is a big part of making it interesting for everyone to use.

Today I want to quickly show how you can do (well simulate) something rather complex with it, like a calculated drop shadow on an element. To see what I mean with this, check out the following demo which is also available on the Demo Studio:

See animated version on JSFiddle

(This is using JSFiddle to show you the demos, so you can click the different tabs to see the JavaScript and CSS needed for the effects. All of the demos are also available on GitHub.)

As you can see, the shadow becomes more blurred and less pronounced the further away the “sun” is from it. You can use the mouse to see the effect in the following demo:

See mouse enabled demo on JSFiddle

Let’s take a look how that is done. The first step is to have a canvas we can paint on – you do this simply by having a mouse detection script (which we used for years and years) and a canvas with access to its API:

See step one on JSFiddle

Click the play button of the above example and you see that you can paint on the canvas. However, the issue is that you keep painting on the canvas instead of only having the orb follow the cursor. To do this, we need to wipe the canvas every time the mouse moves. You do this with clearRect()

See step two on JSFiddle

Running the above example now shows that the orb moves with the mouse. Cool, so this is going to be our “sun”. Now we need to place an object on the canvas to cast a shadow. We could just plot it somewhere, but what we really want is it to be in the middle of the canvas and the shadow to go left and right from that. You can move the origin of the canvas’ coordinate system using translate(). Which means though that our orb is now offset from the mouse:

See step three on JSFiddle

If you check the “fix mouse position” checkbox you see that this is fixed. As we move the coordinate system to the half of the width of the canvas and half of its height, we also need to substract these values from the mouse x and y position.

Now we can draw a line from the centre of the canvas to the mouse position to see the distance using c.moveTo( 0, 0 );c.lineTo( distx, disty ); where distx and disty are the mouse position values after the shifting:

See step four on JSFiddle

In order to find out the distance of the shadow, all we need to do is multiply the mouse coordinates with -1 – in this demo shown as a red line:

See step five on JSFiddle

This gives us a shadow distance from the centre opposite of the mouse position, but we don’t want the full length. Therefore we can apply a factor to the length, in our case 0.6 or 60%:

See step six on JSFiddle

Now we are ready for some drop shadow action. You can apply shadows to canvas objects using shadowColor and its distance is shadowOffsetX and shadowOffsetY. In our case, this is the end of the red line, the inversed and factored distance from the mouse position to the centre of the canvas:

See step seven on JSFiddle

Now, let’s blur the shadow. Blurring is done using the shadowBlur property and it is a number starting from 0 to the strength of the blur. We now need to find a way to calculate the blur strength from the distance of the mouse to the centre of the canvas. Luckily enough, Pythagoras found out for us years ago how to do it. As the x and y coordinate of the mouse are the catheti of a right-angled triangle, we can calculate the length of the hypothenuse (the distance of the point from the centre of the canvas) using the Square root of the squares of the coordinates or Math.sqrt( ( distx * distx ) + ( disty * disty ) ).

This gives us the distance in pixels, but what the really want is a number much lower. Therefore we can again calculate a factor for the blur strength – here we use an array for the weakest and strongest blur blur = [ 2, 9 ]. As the canvas itself also has a right-angled triangle from the centre to the top edge points we can calculate the longest possible distance from the center using longest = Math.sqrt( ( hw * hw ) + ( hh * hh ) ) where hw is half the width of the canvas and hh half the height. Now all we need to do is to calculate the factor to multiply the distance with as blurfactor = blur[1] / longest. The blur during the drawing of the canvas is the distance of the mouse position multiplied with the factor or currentblur = parseInt( blurfactor * realdistance, 10 );. We disregard blur values below the range we defined earlier and we have our blurred shadow:

See step eight on JSFiddle

In order to make the shadow weaker the further away the mouse is we can use the alpha value of its rgba() colour. The same principle applies as with the blur, we set our edge values as shadowalpha = [ 3, 8 ] and after calculating them from the distance we apply their inverse as the alpha value with c.shadowColor = 'rgba(0,0,0,' + (1 - currentalpha / 10) + ')';. This blurs and weakens the shadow:

See step nine on JSFiddle

You can do a lot more with this, for example we could also scale the sun orb the further out it gets or use a second shape to resize and blur it. You can also go completely overboard.

Found a way to optimise this? Tell us about it!

Compiling an existing speech synthesis engine to JavaScript is a good way to avoid writing a complicated project like eSpeak from scratch. Once compiled, the eSpeak code in speak.js doesn’t know it’s running on the web: speak.js uses the Emscripten emulated filesystem to ‘fake’ the normal file reading and writing calls that the eSpeak C++ code has (fopen, fread, etc.). This allows the normal eSpeak datafiles to be used (either through an xhr, or by converting them to JSON and bundling them with the script file). The result of running the compiled eSpeak code is that it ‘writes’ a .wav file with the generated audio to the emulated filesystem. speak.js then takes that data, encodes it using base64, and creates a data URL. That URL is then loaded in an HTML5 audio element, letting the browser handle playback. (Note that while that is a very simple way to do things, it isn’t the most efficient. speak.js has not yet focused on speed, but with some additional work it could be much faster, if that turns out to be an issue.)

Why would you want TTS in JavaScript? Well, with speak.js you can bundle a single .js file in your website, and then generating speech is about as simple as writing

speak("hello world")

(see the speak.js website for instructions). The generated speech will be exactly the same on all platforms, unlike if your users each did TTS in their own way (using an OS capability, or a separate program). speak.js can also be used to build browser addons in a straightforward way, since it’s pure JavaScript – no need for platform dependent binaries, and the addon will work the same on all OSes.

A few more comments:

• JavaScript is getting more and more capable all the time. The development versions of the top JavaScript engines today can run code compiled from C++ only 3-5X slower than a fast C++ compiler, and getting even better. As a consequence, expanding the capabilities of the web platform can in many cases be done in JavaScript or by compiling to JavaScript, instead of adding new code to the browsers themselves, which inevitably takes longer – especially if you wait for all browsers to implement a particular feature.
• While speak.js uses only standards-based APIs, due to browser limitations it can’t work everywhere yet. It won’t work in IE, Safari or Opera since they don’t support typed arrays, nor in Chrome since it doesn’t support WAV data URLs. So currently speak.js only works properly in Firefox. However, the missing features just mentioned are not huge and hopefully those browser makers will implement them soon. It is also possible to implement workarounds in speak.js for these issues (see next comment).
• Help with improving speak.js is very welcome! One important thing we need is to implement workarounds for the issues that prevent speak.js from running on the browsers it currently can’t run on. Another goal is to build browser addons using speak.js. Please get in touch on github if you want to help out.
• eSpeak supports multiple languages so speak.js can too. You do need to include the additional language files though. Here is an experimental build where you can switch between English and French support (note that it is an unoptimized build, so it will run slower).

James is the tech lead for mozilla.com on the Web Development team. James is passionate about interactive graphics on the open web.

Today he explains how you can create 3D objects using CSS without having 3D transforms support. Take it away, James.

Recently I was tinkering with CSS3 and I discovered that it enabled me to do primitive 3D rendering, which I found fascinating! This led to the creation of dom3d, a JavaScript library that uses CSS to render basic 3D objects.

Now the question is: why? Aren’t canvas, WebGL, and even SVG better technologies to work with for this? Possibly. However, CSS is becoming a powerful language for describing complex effects and shapes, and we should experiment.

Keep that in mind, because CSS definitely isn’t intended to do this, but it’s worth trying to see where we should take CSS in the future.

Advantages

Although this is more of an experiment, it has a few real world benefits:

All rendering libraries available for the web (canvas, WebGL, SVG) require a canvas, which is a constrained box on the page with a specific width and height. It is not possible to render anything outside this box. The canvas also captures all DOM events (like clicks), even completely transparent sections. Theoretically, this could make it difficult to do effects that overlay large parts of the page or are somehow deeply integrated to the content.

Using CSS, we aren’t constrained to a box, and the effect can overlay large portions of the page without covering any the the links or other content requiring interaction.

Other advantages include no need to initialize canvas 2D or WebGL, and a simplistic API making it easy to pick up even if you don’t know much about 3D. It might be easier for kids to start playing around with this before they jump into WebGL or something else. Also, because it’s just a dump of DOM elements you can embed it anywhere (without animation).

So keep in mind that this is a hack, but with the above advantages. This might be good for certain effects: 3D cursor, nav transitions, and others.

How it works

Three-D objects are just a bunch of triangles put together, so let’s start with one simple triangle. If we get that working, it’s a simple step forward to render multiple triangles to form a 3D object.

Rendering a 3d triangle on a 2D screen involves something called “projection”. This is the act of taking a 3D point and projecting it onto a 2D screen. Plug in a 3D triangle to a simple math equation, and you get a 2D triangle representing how the 3D one would look on the screen.

The math is remarkably simple but may seem weird if you aren’t familiar with linear algebra. You can take a look at the renderer code.

Now comes the fun part: can you render any 2D triangle simply with CSS3 transforms? Turns out you can! It just takes some fiddling to figure out which transforms to generate. CSS3 transforms are composed of translate, scale, rotate, and skew values, and we need a few equations to compute these values for a specific 2D triangle.

First, let’s take a simple DOM element and turn it into a triangle. We can do this with the linear-gradient background image (another way is border triangles).

Now let’s draw the following blue triangle with the points [20, 20], [50, 120], and [120, 30]. A vital step is to set a few initial reference points which set everything in the same space. Our equations will assume these coordinate spaces. This is how the points A, B, C and the side AB are related.

If we take a closer look at this, we can derive the transform values. First, get an idea of which angles and values we need and then use geometry to form the equations (in pseudo-code). The red box represents the DOM element, the form AB represents the side formed by points A and B, and rotation occurs clockwise.

rotation = atan2(AB.x, AB.y)
AC' = rotate(AC, -rotation)
width = AC'.x
height = length(AB)
skew = atan2(AC'.y, AC'.x)
translate = A

Awesome! Let’s try it out. Here is a live DOM element being transformed by applying each of our equations:

The resulting triangle matches our target triangle! Here is the final CSS:

width: 93px;
height: 104px;
background: -moz-linear-gradient(-0.727211rad, #0000FF 50%, transparent 0pt);
-moz-transform: translate(20px, 20px) rotate(-0.291457rad) skewY(0.391125rad);
-moz-transform-origin: top left;

Note: The tranform-origin: top left line is important. Normally transforms happen relative to the center of the element, but our equations assume the top left.

Note: dom3d also generates code with the -webkit and -o prefixes for WebKit and Opera support.

You can view the implementation of these equations. It turns out that these equations work for any triangle, as long as the points given are in counter-clockwise order, which is standard in the graphics world.

Taking it all the way

Since we can project a 3D triangle into 2D space and render it with CSS, all we have to do now is apply that to several 3D triangles to form a 3D object!

We need some 3D data at this point. I used Blender to export a teapot into the simple OBJ file format and wrote a script to dump the data as JavaScript. Rendering all those triangles with this technique produces the following:

Teapot! However, we can do much better. A big part of the 3D effect is shading. If we calculate normals, a vector representing where the triangle is facing, and specify a light direction, we can take the dot product of the normal and light for each triangle to get flat shading. View the code for flat shading.

There are many tweaks that take this even further. For example, the above objects have z-indexing enabled. Without this, a triangle that is supposed to be behind another may actually appear on top because it was rendered later. The dom3d uses a heap to render the triangles from back to front.

Real-time animation can be achieved with a setTimeout or requestAnimationFrame function that continually renders the object. The dom3d supports the scale, translate, yaw, and pitch transformations, but there’s nothing stopping you from modifying the object data however you like between renderings. See some examples over at the dom3d website.

Here is the code which renders the teapot animation with dom3d:

It’s more appropriate for webpages to update an animation in response to user interaction instead of constantly rendering and hogging the CPU. See the pole example on the dom3d site for an example.

Improvements and last thoughts

The most interesting possibility with this is to include actual page elements as part of 3D objects. A navigation item could pop out and swirl around in 3d space, and the nav item is seamlessly transformed along with it.

That’s where this hack starts to show its faults, though. Unfortunately this is a little too hacky to provide an appropriate web experience. Because it tricks DIVs into fake triangles, it removes the possibility of integrating any page elements with it. With the coming of 3D CSS transforms though, we can start building true 3D objects made up of any kind of page elements. The only restriction with 3D transforms is that the 3D objects need to built with rectangles instead of triangles.

Other people have already experimented with 3D transforms, like building a pure CSS 3D city. There’s another cool library, Sprite3D, which provides a JavaScript API for building basic 3d objects from page elements.

The most glaring problem with dom3d is the seams in the object, which appear in all browsers. Apparently there are a few bugs in rendering engines when stressing their CSS3 transforms and using linear-gradients!

The dom3d library provides an API to work with all of this, but is hasn’t been documented very well yet. Feel free to browse the README and code on github. These APIs could be improved as well. It also provides an SVG rendering backend, seen here, but I don’t this is the right direction to take. We should focus on building basic 3D objects with page elements.

This was a fun experiment and I’m excited by how fast and capable browsers are becoming. The web is an exciting platform and getting richer and more powerful every year!

Tilt is a fun new Firefox extension focused on creating a 3D visualization of a webpage.

Since the DOM is essentially a tree-like representation of a document, this tool layers each node based on the nesting in the tree, creating stacks of elements, each having a corresponding depth and being textured according to the webpage rendering itself.

Unlike other developer tools or inspectors, Tilt allows for instant analysis of the relationship between various parts of a webpage in a graphical way, but also making it easy for someone to see obscured or out-of-page elements. Moreover, besides the 3D stacks, various information is available on request, regarding each node’s type, id, class, or other attributes if available, providing a way to inspect (and edit) the inner HTML and other properties.

Based on WebGL

The visualization is drawn using WebGL, for dynamic, fast, in-browser rendering. At initialization, Tilt creates individual 3D objects (structures describing how the webpage geometry looks like) using the DOM, with the BODY as the lowest layer and the base of the document upon which descendant nodes are layered. For each successive level, another platform is built, adding depth to the 3D webpage mesh. For example, stacks are built from DIVs, ULs, or any containing node with children.

Controls

Controlling the visualization is achieved using a virtual trackball (arcball), which rotates around the X and Y axes. Other mouse events exist to control yaw, pitch, roll, pan, zoom, as well as various additional keyboard shortcuts. The controller is not tied to these peripherals only however, making it accessible and easily scalable for other input methods or devices. Double clicking a node brings up the Ace Cloud9 IDE editor, showing more useful information about the node and the inner HTML.

Try it

You can find the Tilt source code and the latest extension builds on Github, and a development blog with milestone updates on blog.mozilla.com/tilt.

For now, to test the extension, just download the latest stable build (tilt.xpi: download the file, then open it with Firefox or drag’n drop it on Firefox), install it and search for Tilt inside the Tools menu. Or, you can use Ctrl+Shift+L (or Cmd+Shift+L if you’re on a Mac) to start the visualization. Close it at any time with the Esc key. Tilt works with any webpage, so you can even inspect this blog to see how it looks in 3D.

Future

More features are soon to be added, some of which include: modifying and updating the 3D webpage mesh on the fly (as the webpage changes, exposing CSS transforms for each node, plus customizing stack spacing, thickness, transparency etc.), rendering elements with absolute position or floats differently (e.g., hovering above the webpage based on their z-index), creating a more developer-friendly environment and better integration with the Ace editor and the Firefox Developer Tools. (highlighting the currently selected node, instant 3D preview), exporting the visualization to other browsers or applications (as a 3D object file, probably .obj and/or COLLADA).

The greatest milestone will be achieving seamless 3D navigation between webpages, as in a normal 2D environment.

For more information about upcoming tasks visit the TODO.md list.

To illustrate what you can achieve with CSS3 Animations, we have been working on demo with Anthony Calzadilla (@acalzadilla), famous for his awesome Animation projects.

Check out the demo on the Mozilla Demo Studio.

And it works on Firefox Mobile too:

The whole animation is orchestrated in CSS (keyframe) and the moves are animated transformations (transforms). The images are nested divs. If you translated a div and rotate its child, the transformations are combined. You can see the elements being transformed (bounding boxes) if you activate the debug mode.

#arm-rt {
  /* ARM  SLIDING OUT FROM BODY */
  transform-origin: 0 50%;
  /* The syntax is:
   animation: name duration timing-function delay count direction
  */
  animation: arm-rt-action-01 60s ease-out 10s 1 both; 
}
@keyframes arm-rt-action-01 {
  /* This part of the animation starts after 10s and lasts for 60s */
  0% { transform : translate(-100px,0) rotate(0deg); }
  5% { transform : translate(0,0) rotate(0deg); }
  6% { transform : translate(0,0) rotate(-16deg); }
  21% { transform : translate(0,0) rotate(-16deg); }
  22% { transform : translate(-100px,0) rotate(0deg); }
  100% { transform : translate(-100px,0) rotate(0deg); }
}

Tip: If you want to avoid some performance issues, we encourage you to use bitmap images. SVG images can make the animation a bit shoppy.

Want to see more CSS3 Animations? Check out Anthony’s website: www.anthonycalzadilla.com. And feel free to submit your CSS3 Animations demos to the Mozilla Demo Studio.

Creating a wall/feed for a social app…

…in a few lines of code (full project available on Github).

The messages

The server will send two kinds of messages:
 ● simple messages, starting on a new line prefixed with “data:”
 ● messages with specific event names, similar to simple messages but with “event: <anEventName>” on the previous line

In this case, simple messages are treated as users’ statuses and specific events will be inserted in the timeline with specific colors, although they could appear in different places on the page. The message data will be sent as JSON, although it could be flat text strings.

The server

The server will be a dummy .php script that reads sample statuses from a text files and stream them, one at a time, to the client, using appropriate headers.

The Client

The client will create an event source and register event handlers for each specific event name, as well as an onmessage handler for simple messages.

The missing pieces of the code are available on Github.

Fallbacks

Here is a short list of polyfills/fallbacks available for other browsers:
 ● Remy Sharp’s polyfill
 ● Yaffle’s polyfill
 ● Rick Waldron’s jquery plugin

Have you got examples of EventSource based Web app to share?

MarbleRun was the winner of the last Mozilla Game On Challenge, among other exciting HTML5 games (check them out), and had been added to Web O’ Wonder.

The authors of MarbleRun visited us a couple of weeks ago and gave the following talk about the story of their game:

The slides

Detailed notes

Currently there is a lot of confusion what HTML5 is – a lot of companies abuse the term for marketing of all kind of products and some of the demos showcasing HTML 5 don’t use it at all.

Today I will promise not to try to sell you any operating system or hardware with it, I won’t show any comparisons between browsers (how many Goldfish they can show at the same time) and I promise not to go native (although that is the newest fad – calling HTML5 native).

What HTML5 means to a large degree is that the web is the platform – more than ever. We used the web with web services and for storage for years but now we move towards a world where we deliver both data and the applications on it.

The browser is the platform and is everything we need to help our end users fulfil the tasks they set out to do. Instead of having software packages for everything we just go to a web app and use this one. The browser offers us APIs and is fast enough nowadays to deliver great experiences. Webmail, collaborative document editing and image manipulation is just a first step in this direction.

The information how to do all that is available for you for free. A lot of HTML5 resources are out there on the web with full documentation and examples of what can be done. The openness of web development makes it unnecessary to go to expensive corporate training courses or buy any software – all you need is a text editor and a browser.

People are the key. If you look around you, you will find that the leading lights in the HTML5 world are not from one company or were known for years. Instead a lot of them come out of the woodwork totally unexpected which shows that this is much more about enthusiasm than about corporate backing.

We have so many new toys as developers! If I had had all the technologies we can play with across browsers in the past, I am sure my career would have been twice as fast.

Native HTML5 video and audio allows you to manipulate video in any which way and make it talk to the rest of the document. For example:

• You can rotate and zoom Video
• You can sync page content and video content
• You can overlay videos over animated maps
• You can analyse and manipulate video with Canvas to inject content

The latter is very interesting as Canvas and SVG (which is not HTML5 but a “friends” technology) allows you to go very rich in the interface. Using Canvas and SVG you can also enhance data in the page to create beautiful visualisations.

Another thing HTML5 offers are richer forms for the web including datepickers, sliders and autocomplete controls with in-built validation. This should safe us a lot of client side validation code that always gets out of sync and is hard to localise.

Other technologies do very much add to what HTML5 gives us:

Offline storage and local storage allows us to build applications that recognise the user’s connection state and keep them working whilst the connection is down.

CSS3 is the natural progression of CSS and means that we don’t have to create gradients and rounded corners as graphics but can make them easy to change and maintain in our code itself.

WebGL allows us to create 3D interfaces and animations in the browser. Google’s Body Browser is a great example of how rich that can get.

All this becomes very powerful when you mix and match the technologies. You can clip, transform and run filters like blur over video content, you can simulate green screen in video+canvas, apply shape detection to video content or even detect nudity in photos.

Give this technology to creative people and give them some time and beautiful things can happen. One example is Nike’s better world which looks and acts like a Flash interface but it clean HTML5. Other, ongoing examples of great use of HTML5 technology can be found at the Mozilla demo gallery.

Don’t think this is not you! We need everyone to play with HTML5 to make it work and be used. So if you have a cool HTML5 demo, please send it to us so we can pimp it for you!

The future of the web is a team effort and can and should not be in the hands of a chosen few following a corporate agenda, so please help us and join in! HTML5 is a debate and the more voices coming from different angles, the better the solution will be.

Firefox 4 Desktop, Firefox 4 Mobile: same engine!

And this is awesome! It means you will find the same feature in mobile and desktop: HTML5, CSS3 and modern JavaScript APIs. And this is what we want to show you in our new round of demos in Web O Wonder (This new round also includes more 3D Demos, see this dedicated blog post).

Meet 3 new demos:

• Runfield: a HTML5 Canvas game. We published this game last week for Desktop last week.
• CSS3 Holo is a little experiment with CSS3 transforms and Orientation Events. We create a parallax effect that simulates depth and offers a 3D illusion.
• HTML5 Dashboard is the mobile version of our previously released HTML5 Dashboard for Desktop.

Youtube link.

In the fall, after finishing Flight of the Navigator, our team of audio and WebGL hackers was looking for a new challenge. We’d finished the new Audio API in time for Firefox 4, and were each maintaining various open web libraries, exploiting the new features of HTML5, Audio, JavaScript, and WebGL. We wanted to take another shot at testing the limits of Firefox 4 – then, still in beta.

Seth Bindernagel had the answer. He’d been in contact with a DJ and producer friend named Kraddy, who had just finished an amazing new album. “What if we tried to do something with his sound?” The idea was too good to pass up, and with Kraddy’s support, we dove into the tracks and started imagining what these songs might look like, when interpreted through the medium of the web.

«The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike»

Kraddy’s music was easy to demo because of its complex nature, with plenty of emphatic transitions and cue points–this music wants to be visualized! The music for No Comply also provided a dark and introspective sound on which to build a narrative. On his blog, Kraddy had already written about how he understood the album’s meaning:

This EP is about Theseus’ decision to be a hero and his decent into the Labyrinth to kill the Minotaur. In a broader sense the EP is about the battle we all face when we challenge ourselves as people. We must enter the Labyrinth of our minds and at the center we find our greatest fears. To defeat those fears we must kill a part of ourselves. And in killing a part of ourselves we create the potential to grow into a more developed person.

Kraddy’s vision informed our early outlines and storyboards. We knew that we wanted to play on the story of the Minotaur and the Maze, and the idea of facing down ones’ own fears. Together we came up with the idea of re-telling the story using a mixture of real-life video and 8-bit video game styling. Because the album was deeply personal to Kraddy, we decided to feature him in the demo. Kraddy agreed to be filmed, and Brett Gaylor used the footage to create the opening and closing video sequences. We also used Kraddy as the inspiration for the demo’s main video game character.

The launch of Firefox 4 brings a lot to the web, not least WebGL. As the web shifts from a 2D-only to a 2D and 3D space, we wanted to explore the intersection of these two familiar graphical paradigms. Rather than picking just one, we chose to create a hybrid, dream world, composed of 3D and 2D elements. Many people will recognize in our 2D characters and graphics an homage to much earlier video games, like Double Dragon. We wanted to celebrate the fact that these two paradigms can now exist together in a simple web page–everything we do in the demo is one web page, whether audio, video, 2D, 3D, or text.

Like the Flight of the Navigator(FOTN) demo before it, we chose the CubicVR.js engine to drive all the 3D graphics. Over the months leading up to the demo, Charles J. Cliffe had begun the painstaking process of porting features from his C++ engine over to JavaScript. The simple environment of WebGL and JavaScript allowed for features that even his C++ version did not yet posses to be quickly prototyped. Many bottlenecks had to be overcome during iterations of the demo, as we wanted to push the limits further than before. The biggest hurdle was visibility and lighting. Luckily, Bobby Richter came to the rescue. Using his experience with Octrees, he was able to work with Charles to produce a visibility and lighting pipeline which provides impressive performance for the task. In contrast, FOTN has no visibility system and was shaded by a single global directional light and ambient surface textures (for window lights, etc.) to simulate the rest. In No Comply we were able to push the limits with high poly counts and many overlapping point lights and were still able to reach the framerate cap.

Creating a 3D world like the one in this demo requires a lot of original content creation, which in turn requires some sophisticated tools. Instead of developing our own, and in the open-nature of our group, we decided to use existing technology like Blender. The community that develops Blender and creates content with it is rich and diverse, and because it’s an open tool, we could add the features we needed when they weren’t already present.

Our preference for open technologies also meant that the COLLADA scene format was an obvious choice. Unfortunately, as of version 2.49, Blender exports an Autodesk-inspired format of COLLADA, which isn’t quite up to the official standard, missing many important bits of information. Fixing this directly in Blender (with a little bit of Python hacking) let CubicVR stay standards-compliant, and let us milk Blender for all of the scene information we could think of using.

The demo’s 3D modelling, while important, comprises perhaps only half of No Comply’s original content. An incredible undertaking on the part of Omar Noory provided the textures for the rich environment through which Kraddy rumbles and tumbles. Frequently, spontaneous requests for “an 8 bit trash can,” “a cool sign with our names on it,” or, “some beefy bad lookin’ dudes” were answered almost instantly by Omar’s gracious and masterful digital pen. You may have recognized Omar’s name from his claim to meme-fame with “Haters Gonna Hate”.

Adding the perfect amount of flare to the graphics pipeline is Al MacDonald’s Burst animation engine. Al not only wrote our sprite animation engine, but also the web-based toolset we used to create the animations. The 8-bit Kraddy and all of No Comply’s 8-bit baddies are driven by animation paths prepared with Burst, and engineered with a set of tools that work right inside the browser.

In addition to cutting edge graphics with WebGL and <canvas>, we also wanted to explore how far we could push the new Firefox 4 Audio API we’d developed. The Audio Data API allows us to do many new things with the HTML5 <audio> and <video> tags, such as outputting generated audio and revealing realtime audio data to JavaScript. Libraries like Corban Brook’s DSP.js and and Charles’ BeatDetektor.js were used to analyze the audio in realtime and trigger various effects and animation sequences. Tracks of audio triggers were also recorded for tighter sequencing of key elements in the song we wanted to emphasize. One of the really new techniques we played with a lot in the demo was controlling GLSL shaders and lighting directly with audio, punching in and out with every beat and clap. Unlike most treatments of audio on the web, in this demo the song isn’t a background element, but is woven into the fabric of all the visuals and effects.

Getting a demo of this scale to work in the browser means figuring out how to make every bit of it work fast, and keep framerates high. Everything we do in the demo, from loading and parsing massive COLLADA models, to controlling 3D scene graphs, to analyzing real-time audio data, is done with JavaScript. We think it’s important to point this out because so many people begin with the assumption that JavaScript isn’t fast enough for the kind of work we’re presenting. The truth is that modern JavaScript, like that in Firefox 4, has been so heavily optimized that we all need to rethink what is and isn’t possible on the web.

We’ve taken advantage of a bunch of Firefox 4′s new performance features, as well as new HTML5 goodies, in order to make this all possible. For example Web Workers let us move heavy resource parsing off the main thread, freeing it for audio analysis and 3D effects. While a large portion of each second is consumed by simply pushing information to the video card, it isn’t necessary for the browser to wait for that to happen. In the background, we can use other threads to load and parse data, so that it’s ready to draw when the main thread needs it. Of course, a host of problems arise immediately whenever concurrency is involved, but we managed to draw a large performance and overall stability increase by utilizing Web Workers.

Another performance trick was using JavaScript Typed Arrays, which give us a tremendous speed boost when working with audio and pixel data. When you’re analyzing slices of audio data hundreds of bytes wide as fast as possible, your Fourier Transform code needs to be blazingly quick. Thanks to Corban’s highly optimized dsp.js library, this was hardly on our minds.

Next, we spent a lot of time optimizing our JavaScript so that it could take advantage of Firefox’s Tracing and Method JIT. Writing code that can be easily byte-compiled by the browser makes sure that anything we write runs as fast as possible. This is a fairly new and surprising concept, especially to those who remember the JavaScript of yesterday.

Part of what appealed to us about writing this demo was that it let those of us who are browser developers, and those of us who are web developers, work together on a single project. Most of the technology showcased in this demo was made on bleeding edge Firefox nightlies and our development process involved lots of feedback about performance or stability issues in the browser. Dave Humphrey focused on the internals of the Audio API, instrumenting and profiling our JavaScript, and helped us work closely with Mozilla’s JavaScript, graphics, and WebGL engineers. People like Benoit Jacob and Boris Zbarsky, among others, were indispensable as we worked to fix various bottlenecks. Part of what makes Mozilla such a successful project is that their engineers are not locked away, unable to work with web developers. Having engineers at our beck and call was essential to our success with such a demanding schedule, and we were proud to be able to help Mozilla test and improve Firefox 4 along the way.

Beyond the technical aspects of the demo, it also points to the spirit of how these technologies are meant to be used. We worked as a distributed team during evenings and on weekends, to plan and code and create everything, from the tools we needed to the graphical resources to the demo’s final code. Some of our team are browser developers, some web and audio hackers, others are graphic designers or filmmakers, still others storytellers and writers–everyone had a place around the table, and a role to play. We think this is part of what makes the web such a powerful platform for creative and collaborative work: there isn’t one right way to be, no single technology you need to know, and the techniques and tools are democratized and open to anyone willing to pick them up. The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike.