In additional to the normal improvements that you’ve come to expect like performance, security and bug fixes, Aurora 10 focuses in HTML5 enhancements.

New additions

• HTML5 Visibility API
• createProcessingInstruction
• WebGL antialiasing
• 3D Transforms
• Visibility API
• Document.mozFullScreenEnabled\

Developer Tools

• Code Editor
• CSS Rules View
• Page Inspector/Highlighter
• Console Object Completion
• Style Inspector

Aurora 10 also implements incremental enhancements like IndexedDB setVersion API changes. Ongoing detailed attention to evolving specifications help to keep Firefox at the front of the Web revolution. (Read more about IndexedDB on MDN.)

DOM

• We now fire a “load” event on stylesheet linking when the sheet load finishes or “error” if the load fails.
• We turn the POSTDATA prompt into an information page (when navigating in session history).
• We only forward event attributes on body/frameset to the window if we also forward the corresponding on* property.
• We no longer allow more than one call to window.open() when we allow popups.
• We fixed a bug where a success callback never fired when a position update is triggered after getCurrentPosition().
• We removed replaceWholeText().
• We fixed an error with createPattern(zero-size canvas).
• We now handle putImageData(nonfinite) correctly.
• We now throw INVALID_STATE_ERR when dispatching uninitialized events.
• We’ve made Document.documentURI readonly.
• We fixed document.importNode to comply with optional argument omitted.

Web workers

• We now allow data URLs.
• We implemented event.stopImmediatePropagation in workers.
• We made XHR2 response/responseType work in Web Workers.

Graphics

• We implement the WebGL OES_standard_derivatives extension.
• We implement minimal-capabilities WebGL mode.

JavaScript

• The function caller property no longer skips over eval frames.
• We fixed E4X syntax so that it is not accepted in ES5 strict mode.
• weakmap.set no longer returns itself instead of undefined.
• We implemented the battery API.

Offline: IndexedDB enhancements

• IndexedDB setVersion API changes
• Added support for IDBObjectStore/IDBIndex.count
• Various methods accept both keys and KeyRanges.
• Added support for IDBCursor.advance.
• Implemented deleteDatabase.
• objectStoreNames are no longer updated on closed databases when another connection adds or removes object stores
• IDBObjectStore.delete and IDBCursor.delete now return undefined.
• No longer throws an error if there are unknown properties in the options objects to createObjectStore/createIndex.
• We now the errorCode to “ABORT_ERR” for all pending requests when IDBTransaction.abort() is called.
• Fixed the sort order for indexes.

Layout

• We have updated the current rule for handling malformed media queries.
• We now support the HTML5 <bdi> element and CSS property unicode-bidi: isolate.
• The CSS3 implementation now supports unicode-bidi: plaintext.

Media

• Implemented Document.mozFullScreenEnabled.
• Enabled the DOM full-screen API on desktop Firefox by default.

Most prominently, this feature allows for impressive 3D mapping applications such as Google MapsGL and Nokia Maps 3D.

What happened

Earlier this year, the Editor’s Draft WebGL specification got updated in response to a security concern. The additions were:

1. A mandatory clause disallowing usage of cross-domain elements as WebGL textures in the general case.
2. A non-normative clause specifically allowing cross-domain elements that have CORS approval. For that occasion, the HTML specification on the <img> element got updated to add a new crossorigin attribute.

The first got implemented in Firefox 5, the second is now in Firefox 8.

How to use this feature

There are two CORS modes: “anonymous” and “use-credentials”. We’ll focus on “anonymous” as it’s the common case. A great example of images served with anonymous CORS is Google Maps imagery, such as:

http://khm0.googleapis.com/kh?v=95&x=0&y=0&z=0

In order to load it with CORS as a WebGL texture, we set the crossOrigin attribute on it:

var earthImage = new Image();
earthImage.crossOrigin = "anonymous";

Now we load it as usual:

    earthImage.onload = function() {
      // whatever you usually to do load WebGL textures
    };
    earthImage.src = "http://khm0.googleapis.com/kh?v=95&x=0&y=0&z=0";

That’s it! Aside from setting the crossOrigin attribute, we didn’t have to do anything. Here is the full self-contained example.

The HTTP headers

If we study the HTTP headers for this image (using, for example, Firefox’s Web Console), we find that the Request Headers contain

Origin: null

which is the effect of having set this crossOrigin attribute on the img element. And the Response Headers contain

Access-Control-Allow-Origin: null

which is the effect of the server supporting CORS for this file.

Doing this in HTML

Of course, one could also set this attribute in HTML, in which case it’s case-insensitive:

<img src="http://khm0.googleapis.com/kh?v=95&x=0&y=0&z=0" crossorigin="anonymous">

And since “anonymous” is both the missing-value-default and the invalid-value-default for the crossorigin attribute, we can pass any invalid value for it, or even just omit its value:

<img src="http://khm0.googleapis.com/kh?v=95&x=0&y=0&z=0" crossorigin>

Coming soon: CORS approval for Canvas 2D drawImage

What if you first draw a CORS-approved cross-domain image onto a 2D
canvas, and then use that canvas as the source of a WebGL texture? The
good news is that this will work in Firefox 9, which is hitting the Beta
channel soon. This fix means that demos like this will work really
nicely in Firefox 9.

As we were in Poland anyways to attend Frontrow it was easy to visit Sabre and give a one hour presentation on the ins and outs of HTML5 and where the web might go. The day after the same talk was repeated over Skype for the German office of SAP.

In the one hour presentation we covered:

• the basics of HTML5
• what it meant as an evolution of markup
• what HTML5 is not
• common HTML5 myths
• “Friends of HTML5″ – related technologies and
• what the future of web technologies could look like

The slides are available online or embedded below (cursor keys to navigate, press N to show and hide notes and cursor down to proceed on slides with bullet points):

The audio recording of the talk is available on archive.org.

The slides and the audio is licensed with Creative Commons, so feel free to re-use and distribute them.

Since the first alpha version of Tilt was announced (a Firefox extension focused on creating a 3D visualization of a webpage), a lot of work has been done to add a great number of developer-oriented features. These focus on debugging the structure of a webpage, inspecting styling and attributes for each node and seamlessly refreshing the visualization when the DOM structure changes or after contents of document are repainted.

Solve nesting problems

Tilt is useful when searching problems in the HTML structure (like finding unclosed DIV elements for example) by providing the extra third dimension, layering each node based on nesting in the DOM tree. Stacks of elements visually represent branches in the DOM, and each node can be inspected for the inner HTML contents, its computed CSS style and the attributes.

Clicking anywhere on the visualization highlights a color-coded rectangle surrounding the corresponding node. Double click shows up the source preview for that node. Tilt also tries to show the most relevant information when needed (one is most likely to inspect the attributes of an input, button or image element, for example, but can easily switch between HTML, CSS and attributes view at any time).

Minidom map

The “minidom” is a tree view representation showing a minimalistic snapshot of the document object model. Each node is assigned a color associated by tag name (blue for div, green for span etc.) and represented as a strip, along with visual markers for the id and/or class if available. Each one of these strips also has a width relative to the type, id and class name length for the respective element, and the corresponding 3D stack in the visualization has color-coded margins. The coloring for individual elements is easily changeable using the color picker near to the minidom legend.

Clicking a strip in the tree view (or directly a stack on the 3D document visualization mesh) also highlights the node with a colored quad. This behavior is a good way to relate with the Style Inspector, and a more unified interaction between Tilt and other Developer Tools is planned in the future. All of these additions make it easier to analyze the bounds of each node, along with the HTML, computed CSS and attributes.

Realtime editing

Because Tilt is able to detect when a webpage’s DOM structure changes or when a repaint is necessary, integration is seamless with existing Developer Tools. Using Tilt and Firebug or Style Editor at the same time is easy. One can enable or disable CSS properties, changing the style of a node, and the visualization changes accordingly.

To enable realtime updates for the 3D webpage, go to the Options menu and check “Refresh visualization”.

Useful for learning

Developer tools such as “view source” have always been used to help people learn about web development. The 3D view highlights the structure of a page better than a flat view, thus anyone can immediately understand the parent-child relationship between nodes in a webpage, their positioning and how the layout is influenced.

One use case for this is the Hackasaurus mashup. The X-Ray Goggles is a nice and fun tool designed to make it easier to learn about the different document node types, the “building blocks” which create a webpage.

Export

A requested feature was the ability to export the visualization as a 3D mesh, to be used in games or other 3D editors. Tilt adds the ability to export to .obj, along with a material .mtl file and a .png texture (a screenshot of the entire webpage). The open .obj format ensures the fact that the mesh can be opened with almost any editor. Here’s a ray-traced rendering of hacks.mozilla.org in Blender:

Fun with experiments

As soon as it was released, many people found clever and interesting alternative ways to interact with Tilt. One experiment was creating a 3D visualization of an image, by exporting chunks of pixels to a HTML representation. The result was a voxel-like representation, with node blocks and stacks instead of pixels. A simple Image2Tilt converter was written in JavaScript, and you can try it directly in the browser.

Accelerometer support was another addition based on community request. This shows how easy it is to add functionality that wasn’t originally planned.

You can view the source code, fork it and also contribute to the addon with ideas or feature requests on Github, at github.com/victorporof/Tilt.

Available as an addon

The latest version of Tilt can be found on Github, but you can also download Tilt as an addon from addons.mozilla.org.

For compatibility, Tilt requires WebGL capabilities. Go to get.webgl.org to check availability and troubleshoot any issues. The current version works with Firefox 6.0 to latest 10.0 Nightly releases (latest Nightly builds now also support WebGL anti-aliasing, working great with Tilt).

To start Tilt, hit Control+Shift+M (or Command+Shift+M if you’re on Mac OS), or go to Web Developer -> Tilt, available in the Firefox application menu (or the Tools menu on Mac OS). You can modify this hotkey (and other properties) from the Options menu after starting Tilt.

More information about Tilt, the development process and milestone updates can be found on blog.mozilla.com/tilt.

Future

Tilt has become an active Developer Tools project, and an ongoing effort is made to integrate it with other existing tools like Style Inspector and Style Editor (source code and latest builds). As the 3D view of a webpage has proven to be useful for debugging, this main functionality will gradually become part of Firefox in future releases.

We asked people from the demo scene to have a go at web technologies and (with WebGL being the absolute winner of course) managed to collect over 100 submissions. Now the judges have spoken and we picked the winners in the categories of Main Demo, Single Effect, Audio Demo, Animated GIF and pure CSS demo.

Amongst other great examples of using technology in a purely creative way unhindered by real life application needs here are the winners of Demo and Single Effect:

Main Demo: Akemi

Single Effect: WebGL Water Simulation

Demo Link or screen capture:

We congratulate all the winners and thank all those who contributed. Playing with technology is a big part of making it interesting for everyone to use.

Weekly links August 18th 2011

As usual, anything you think we should read or be aware of, don’t hesitate to post a comment, contact us on Twitter or through any other mean.
Here comes this week’s treats!

Christian Heilmann

This water implementation in WebGL is incredibly impressive. Turn the box and also don’t forget to drag the ball around. Beautifully done and it shows that interactive WebGL experiences are very engaging. Imagine this paired with touch interfaces and you have the introductory screen Microsoft’s Surface table had.

If you want to read more tips or discuss the web with Christian, he’s available on Twitter as @codepo8.

Janet Swisher

HTML5 and the document outlining algorithm. As dear as this topic is to my tech writer heart, it’s also important for anybody creating content for the web.

If you want to read more tips or discuss the web with Janet, she’s available on Twitter as @jmswisher.

Jay Patel

Anatomy of a HTML5 Mobile App Tries to answer the question: “We all agree HTML5 can get the job, but can it DO the job?” A interesting look at all the pieces to developing an HTML5 mobile app vs going native.

Jay can be found on Twitter as @jaybhai

Jeff Griffiths

Firefox Electrolysis 101 (part 1) Tim Taubert started an excellent series of posts on Electrolysis, Firefox’s upcoming multi-process architectural shift. Highly recommended for add-on developers or people interested in the future of Firefox development.

If you want to read more tips or discuss the web with Jeff, he’s available on Twitter as @canuckistani.

Louis-Rémi Babe

Andrea Giammarchi compares the different ways of loading javascript in a Web page and introduces a new minimalistic loader: yal.js. Andrea is a javascript ninja and you can trust him to produce state of the art solutions when tackling such problems.

If you want to read more tips or discuss the web with Louis-Rémi, he’s available on Twitter as @Louis_Remi.

Rob Hawkes

Forever is a command line interface that lets you keep your Node apps running for, well, ever! Never again do you have to worry about a random crash taking your app down until you next check on it.

If you want to read more tips or discuss the web with Rob, he’s available on Twitter as @robhawkes.

Robert Nyman

Understanding Hardware Acceleration on Mobile Browsers – Explaining hardware acceleration and how web browsers are affected by it.

If you want to read more tips or discuss the web with Robert, he’s available on Twitter as @robertnyman.

Stormy Peters

Mozilla Brown Bag on BrowserID & Privacy – It really worries me when people login to sites with their Facebook identity. They are giving control over their identity on the web to a non neutral third party. Mozilla is trying to fix this problem with BrowserID. It will let you login to websites with one ID – in ways that preserve your privacy.

Stormy can be found on Twitter as @storming

Tristan Nitot

The new release of Firefox has brought us ScratchPad, a very cool new tool for Web devs, a bit like the Web Console on steroids. Kevin Dangoor has a great post on this: Introducing ScratchPad. Web devs who want to know what is coming in this department with future Firefox versions should follow the Devtools blog!

If you want to read more tips or discuss the web with Tristan, he’s available on Twitter as @nitot.

Introducing Mozilla Hacks Weekly

The idea behind Mozilla Hacks Weekly is to each Thursday present you with a link from each of our team members. It could be something news-worthy, or an article, resource or blog post that really helped us out recently. It is also our way of giving the proper credits to people who share such important and helpful information, and additionally, a way for you to getting to know us and our interests.

Weekly links July 28th 2011

Here are the picks for this week from the team members:

Christian Heilmann

In HTML5 notes for analysts and journalists Bruce Lawson cuts through some of the misinformation that is being spread about HTML5 and asks for clearer reporting on the subject.

Christian can be found on Twitter as @codepo8

Havi Hoffman

We (Mozilla) Fight For the User – In this 12 minute podcast, Mozilla CTO Brendan Eich goes deep on the future of Mozilla and the Web, including thoughts on what’s ahead for open web app stores, mobile interface/devices, and technology change in support of user sovereignty.

Havi can be found on Twitter as @freshelectrons

Janet Swisher

Weapons of Mass Assignment: A high-profile Ruby on Rails app highlights some serious, yet easily avoided, security vulnerabilities. The official Rails security guide and the Open Web Application Security Project (OWASP) list of Web application vulnerabilities would have helped catch all of the issues discussed in this article.

Janet can be found on Twitter as @jmswisher

Jay Patel

11 HTML5 Video Players You Should Try

Jay can be found on Twitter as @jaybhai

John Karahalis

Dive into HTML5, a freely available book on HTML5 web development

John can be found on Twitter as @openjck

Louis-Rémi Babe

Ericson’s take on Web Real-Time Communication

Louis-Rémi can be found on Twitter as @Louis_Remi

Rob Hawkes

Car physics demo in WebGL: It’s not perfect, but this is a great demo of vehicle physics using 3D models in WebGL.

Rob can be found on Twitter as @robhawkes

Robert Nyman

HTML5 Audio support in web browsers – A walk-through of HTML5 codecs and their support in different web browsers, with a compatibility table.

Robert can be found on Twitter as @robertnyman

Stormy Peters

Simple HTML slides. It’s really easy to create full feature slides in HTML these days. You can even have people follow along real time on the web.

Stormy can be found on Twitter as @storming

Will Bamberg

Protecting Browsers from Extension Vulnerabilities: the design here formed the basis of Chrome’s extension system and influenced that of Mozilla’s Add-on SDK.

What did you read?

If you have any thoughts about the above links, or anything else you would want to share with us, feel more than free to do that in a comment below!

We would love to hear your takes and get tips on what to read and pay notice to.

Tilt is a fun new Firefox extension focused on creating a 3D visualization of a webpage.

Since the DOM is essentially a tree-like representation of a document, this tool layers each node based on the nesting in the tree, creating stacks of elements, each having a corresponding depth and being textured according to the webpage rendering itself.

Unlike other developer tools or inspectors, Tilt allows for instant analysis of the relationship between various parts of a webpage in a graphical way, but also making it easy for someone to see obscured or out-of-page elements. Moreover, besides the 3D stacks, various information is available on request, regarding each node’s type, id, class, or other attributes if available, providing a way to inspect (and edit) the inner HTML and other properties.

Based on WebGL

The visualization is drawn using WebGL, for dynamic, fast, in-browser rendering. At initialization, Tilt creates individual 3D objects (structures describing how the webpage geometry looks like) using the DOM, with the BODY as the lowest layer and the base of the document upon which descendant nodes are layered. For each successive level, another platform is built, adding depth to the 3D webpage mesh. For example, stacks are built from DIVs, ULs, or any containing node with children.

Controls

Controlling the visualization is achieved using a virtual trackball (arcball), which rotates around the X and Y axes. Other mouse events exist to control yaw, pitch, roll, pan, zoom, as well as various additional keyboard shortcuts. The controller is not tied to these peripherals only however, making it accessible and easily scalable for other input methods or devices. Double clicking a node brings up the Ace Cloud9 IDE editor, showing more useful information about the node and the inner HTML.

Try it

You can find the Tilt source code and the latest extension builds on Github, and a development blog with milestone updates on blog.mozilla.com/tilt.

For now, to test the extension, just download the latest stable build (tilt.xpi: download the file, then open it with Firefox or drag’n drop it on Firefox), install it and search for Tilt inside the Tools menu. Or, you can use Ctrl+Shift+L (or Cmd+Shift+L if you’re on a Mac) to start the visualization. Close it at any time with the Esc key. Tilt works with any webpage, so you can even inspect this blog to see how it looks in 3D.

Future

More features are soon to be added, some of which include: modifying and updating the 3D webpage mesh on the fly (as the webpage changes, exposing CSS transforms for each node, plus customizing stack spacing, thickness, transparency etc.), rendering elements with absolute position or floats differently (e.g., hovering above the webpage based on their z-index), creating a more developer-friendly environment and better integration with the Ace editor and the Firefox Developer Tools. (highlighting the currently selected node, instant 3D preview), exporting the visualization to other browsers or applications (as a 3D object file, probably .obj and/or COLLADA).

The greatest milestone will be achieving seamless 3D navigation between webpages, as in a normal 2D environment.

For more information about upcoming tasks visit the TODO.md list.

The event was one-track with 8 speakers, and personally I always prefer when it’s just one track for follow-up discussions and that everyone have seen and heard the same thing. The topics were ranging broadly between script loaders and HTML5 in one end, and voice-controlled telephony applications in the other.

My presentation

My talk was about HTML5 APIs in general, to give an introduction to them but also to inspire people to try things out and also give feedback to both working groups and web browser vendors about current implementations.

Slides can also be downloaded at SlideShare.

Additionally to the APIs covered in my London Ajax Mobile Event presentation, I went through Web Sockets, File API, HTML5 video, canvas and WebGL. Also, if you are more interested in the <canvas> element, my colleague Rob Hawkes recently released the Foundation HTML5 Canvas book.

What I especially liked talking about is services like vid.ly which helps you take control over the problem of different video codec support in different web browsers, by storing various formats and then deliver the most suitable one depending on the web browser/device accessing it.

Another favorite is Universal Subtitles, which is an excellent tool for everyone to be able to add subtitles to a video clip, empowering users with varying language skills to take part of a video and its content and sharing it with the world.

An option to make the content of a web site richer, there is Popcorn.js to sync key events in the video playing to what kind of text or other information you want to present to go with that. To complement that, the Butterapp is an editor to create that kind of content syncing, currently in alpha.

I also mentioned videograbber for taking easy video screenshots in the web browser.

Dev Derby <video> challenge

I also want to take the opportunity to remind you that Mozilla Dev Derby has a challenge for what you can accomplish with the <video> that goes till the end of July, so please submit anything if you have a good idea!

The security rules about cross-domain images

A cross-domain image is an image coming from a different domain than the canvas. A basic rule of Web security is that scripts must not be able to read pixel data from cross-domain images — they can only “blindly” display them. In more concrete terms, imagine that you currently have a session open on your bank’s website, allowing you to download scanned copies of cheques you’ve written. You don’t want scripts loaded in other tabs, from other websites, to be able to read your scanned cheques! Similarly, when you draw a cross-domain image onto a 2D canvas, using drawImage(), the canvas becomes “tainted” so that it’s no longer possible for scripts to read its pixels. This prevents a loophole whereby a canvas would be used as a proxy to read cross-domain images.

The problem with cross-domain images as WebGL textures

When a cross-domain image was used as a WebGL texture, the WebGL canvas was “tainted” so that reading from it was no longer possible. Theoretically, that eliminated the concern. But a while ago, a researcher wrote to the public WebGL list with a possible attack that could still allow reading pixels from WebGL textures. The idea was to paint the texture one pixel at a time with a WebGL fragment shader crafted to take an amount of time proportional to its brightness, and then time how long it takes: that would conceivably allow to get an approximation of the original image. Initially this attack seemed difficult to execute practically, but since then further research, including a proof-of-concept has been published which shows the attack to be more realistic than initially expected.

The response

The WebGL spec has been updated to disallow using cross-domain images as WebGL textures, and Mozilla’s implementation in Firefox 5 has been updated accordingly. A non-normative section has also been added allowing cross-domain images that have CORS approval. Using CORS in this case is a way for servers to explicitly say when an image is OK to be read by cross-domain scripts. CORS support for WebGL is a high priority for us and will be implemented very soon.

A wiki page explains some more details. Affected scripts will generate a DOM_SECURITY_ERR exception and, just before that, a JS warning explaining what happened and linking to that wiki page.

In the fall, after finishing Flight of the Navigator, our team of audio and WebGL hackers was looking for a new challenge. We’d finished the new Audio API in time for Firefox 4, and were each maintaining various open web libraries, exploiting the new features of HTML5, Audio, JavaScript, and WebGL. We wanted to take another shot at testing the limits of Firefox 4 – then, still in beta.

Seth Bindernagel had the answer. He’d been in contact with a DJ and producer friend named Kraddy, who had just finished an amazing new album. “What if we tried to do something with his sound?” The idea was too good to pass up, and with Kraddy’s support, we dove into the tracks and started imagining what these songs might look like, when interpreted through the medium of the web.

«The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike»

Kraddy’s music was easy to demo because of its complex nature, with plenty of emphatic transitions and cue points–this music wants to be visualized! The music for No Comply also provided a dark and introspective sound on which to build a narrative. On his blog, Kraddy had already written about how he understood the album’s meaning:

This EP is about Theseus’ decision to be a hero and his decent into the Labyrinth to kill the Minotaur. In a broader sense the EP is about the battle we all face when we challenge ourselves as people. We must enter the Labyrinth of our minds and at the center we find our greatest fears. To defeat those fears we must kill a part of ourselves. And in killing a part of ourselves we create the potential to grow into a more developed person.

Kraddy’s vision informed our early outlines and storyboards. We knew that we wanted to play on the story of the Minotaur and the Maze, and the idea of facing down ones’ own fears. Together we came up with the idea of re-telling the story using a mixture of real-life video and 8-bit video game styling. Because the album was deeply personal to Kraddy, we decided to feature him in the demo. Kraddy agreed to be filmed, and Brett Gaylor used the footage to create the opening and closing video sequences. We also used Kraddy as the inspiration for the demo’s main video game character.

The launch of Firefox 4 brings a lot to the web, not least WebGL. As the web shifts from a 2D-only to a 2D and 3D space, we wanted to explore the intersection of these two familiar graphical paradigms. Rather than picking just one, we chose to create a hybrid, dream world, composed of 3D and 2D elements. Many people will recognize in our 2D characters and graphics an homage to much earlier video games, like Double Dragon. We wanted to celebrate the fact that these two paradigms can now exist together in a simple web page–everything we do in the demo is one web page, whether audio, video, 2D, 3D, or text.

Like the Flight of the Navigator(FOTN) demo before it, we chose the CubicVR.js engine to drive all the 3D graphics. Over the months leading up to the demo, Charles J. Cliffe had begun the painstaking process of porting features from his C++ engine over to JavaScript. The simple environment of WebGL and JavaScript allowed for features that even his C++ version did not yet posses to be quickly prototyped. Many bottlenecks had to be overcome during iterations of the demo, as we wanted to push the limits further than before. The biggest hurdle was visibility and lighting. Luckily, Bobby Richter came to the rescue. Using his experience with Octrees, he was able to work with Charles to produce a visibility and lighting pipeline which provides impressive performance for the task. In contrast, FOTN has no visibility system and was shaded by a single global directional light and ambient surface textures (for window lights, etc.) to simulate the rest. In No Comply we were able to push the limits with high poly counts and many overlapping point lights and were still able to reach the framerate cap.

Creating a 3D world like the one in this demo requires a lot of original content creation, which in turn requires some sophisticated tools. Instead of developing our own, and in the open-nature of our group, we decided to use existing technology like Blender. The community that develops Blender and creates content with it is rich and diverse, and because it’s an open tool, we could add the features we needed when they weren’t already present.

Our preference for open technologies also meant that the COLLADA scene format was an obvious choice. Unfortunately, as of version 2.49, Blender exports an Autodesk-inspired format of COLLADA, which isn’t quite up to the official standard, missing many important bits of information. Fixing this directly in Blender (with a little bit of Python hacking) let CubicVR stay standards-compliant, and let us milk Blender for all of the scene information we could think of using.

The demo’s 3D modelling, while important, comprises perhaps only half of No Comply’s original content. An incredible undertaking on the part of Omar Noory provided the textures for the rich environment through which Kraddy rumbles and tumbles. Frequently, spontaneous requests for “an 8 bit trash can,” “a cool sign with our names on it,” or, “some beefy bad lookin’ dudes” were answered almost instantly by Omar’s gracious and masterful digital pen. You may have recognized Omar’s name from his claim to meme-fame with “Haters Gonna Hate”.

Adding the perfect amount of flare to the graphics pipeline is Al MacDonald’s Burst animation engine. Al not only wrote our sprite animation engine, but also the web-based toolset we used to create the animations. The 8-bit Kraddy and all of No Comply’s 8-bit baddies are driven by animation paths prepared with Burst, and engineered with a set of tools that work right inside the browser.

In addition to cutting edge graphics with WebGL and <canvas>, we also wanted to explore how far we could push the new Firefox 4 Audio API we’d developed. The Audio Data API allows us to do many new things with the HTML5 <audio> and <video> tags, such as outputting generated audio and revealing realtime audio data to JavaScript. Libraries like Corban Brook’s DSP.js and and Charles’ BeatDetektor.js were used to analyze the audio in realtime and trigger various effects and animation sequences. Tracks of audio triggers were also recorded for tighter sequencing of key elements in the song we wanted to emphasize. One of the really new techniques we played with a lot in the demo was controlling GLSL shaders and lighting directly with audio, punching in and out with every beat and clap. Unlike most treatments of audio on the web, in this demo the song isn’t a background element, but is woven into the fabric of all the visuals and effects.

Getting a demo of this scale to work in the browser means figuring out how to make every bit of it work fast, and keep framerates high. Everything we do in the demo, from loading and parsing massive COLLADA models, to controlling 3D scene graphs, to analyzing real-time audio data, is done with JavaScript. We think it’s important to point this out because so many people begin with the assumption that JavaScript isn’t fast enough for the kind of work we’re presenting. The truth is that modern JavaScript, like that in Firefox 4, has been so heavily optimized that we all need to rethink what is and isn’t possible on the web.

We’ve taken advantage of a bunch of Firefox 4′s new performance features, as well as new HTML5 goodies, in order to make this all possible. For example Web Workers let us move heavy resource parsing off the main thread, freeing it for audio analysis and 3D effects. While a large portion of each second is consumed by simply pushing information to the video card, it isn’t necessary for the browser to wait for that to happen. In the background, we can use other threads to load and parse data, so that it’s ready to draw when the main thread needs it. Of course, a host of problems arise immediately whenever concurrency is involved, but we managed to draw a large performance and overall stability increase by utilizing Web Workers.

Another performance trick was using JavaScript Typed Arrays, which give us a tremendous speed boost when working with audio and pixel data. When you’re analyzing slices of audio data hundreds of bytes wide as fast as possible, your Fourier Transform code needs to be blazingly quick. Thanks to Corban’s highly optimized dsp.js library, this was hardly on our minds.

Next, we spent a lot of time optimizing our JavaScript so that it could take advantage of Firefox’s Tracing and Method JIT. Writing code that can be easily byte-compiled by the browser makes sure that anything we write runs as fast as possible. This is a fairly new and surprising concept, especially to those who remember the JavaScript of yesterday.

Part of what appealed to us about writing this demo was that it let those of us who are browser developers, and those of us who are web developers, work together on a single project. Most of the technology showcased in this demo was made on bleeding edge Firefox nightlies and our development process involved lots of feedback about performance or stability issues in the browser. Dave Humphrey focused on the internals of the Audio API, instrumenting and profiling our JavaScript, and helped us work closely with Mozilla’s JavaScript, graphics, and WebGL engineers. People like Benoit Jacob and Boris Zbarsky, among others, were indispensable as we worked to fix various bottlenecks. Part of what makes Mozilla such a successful project is that their engineers are not locked away, unable to work with web developers. Having engineers at our beck and call was essential to our success with such a demanding schedule, and we were proud to be able to help Mozilla test and improve Firefox 4 along the way.

Beyond the technical aspects of the demo, it also points to the spirit of how these technologies are meant to be used. We worked as a distributed team during evenings and on weekends, to plan and code and create everything, from the tools we needed to the graphical resources to the demo’s final code. Some of our team are browser developers, some web and audio hackers, others are graphic designers or filmmakers, still others storytellers and writers–everyone had a place around the table, and a role to play. We think this is part of what makes the web such a powerful platform for creative and collaborative work: there isn’t one right way to be, no single technology you need to know, and the techniques and tools are democratized and open to anyone willing to pick them up. The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike.

And to celebrate the launch, we have released another round of demos on Web O’ Wonder, with 3 awesome WebGL demos! (This new round also introduces mobile-specific demos, see this dedicated blog post).

WebGL: It’s 3D and Web Content together.

Demo by Cédric Pinson and Guillaume Lecollinet.

GlobeTweeter is a perfect example of how you can mix 3D and Web Content. In this page, you can see real-time geo-located twitter activity represented on the planet earth.

WebGL animations

Demo by The Audio Team.

No-Comply is a WebGL animation. With JägerMonkey (Firefox’s new JavaScript engine) and the experimental animation scheduler (mozRequestAnimationFrame), we can now create complex WebGL animations.

Learn more about the no comply demo.

This demo has been developed by the audio team who has also created the Flight Of The Navigator demo, where you can find Videos and live Flickr and Twitter content in a 3D city, all build with WebGL:

Firefox 4 automatically disables the hardware acceleration and WebGL features if the graphics driver on your system has bugs that cause Firefox to crash. You still get all the other benefits of Firefox 4, of course, just not the newest graphics features. But for best results, you need an up-to-date graphics driver that fixes those bugs.

• Windows users with Intel cards: you need a very recent driver.
• Windows users with NVIDIA cards: you need driver version 257.21 or newer.
• Windows users with ATI cards: you need driver version 10.6 or newer.
• Instructions are also given for Mac and X11 (Linux/Unix).

If you’re planning to develop using WebGL, you need to also spread this message to your users, so they will be able to experience the awesome results of your hard work.

Flight of the Navigator is a WebGL + Audio API demo developed by a team of Mozilla volunteers.

You can see the demo online here (you need a WebGL compatible browser).

More information about this demo on David Humphrey’s blog (one of the developer of this demo) and more about WebGL in Firefox Beta8 on Vlad’s blog.

The screencast:

David’s work is important because it’s showing where the web is going, and where Mozilla is helping to take it. It’s not enough that we’re working on HTML5, which we’re about finished with, but we’re trying to figure out what’s next. Mozilla’s platform, Gecko, is a huge part of why we’re able to experiment and learn as fast as we can. And that’s reflected with what’s possible here. It’s a web you can see, touch and interact with in new ways.

David’s post follows:

I’m working with an ever growing group of web, audio, and Mozilla developers on a project to expose audio spectrum data to JavaScript from Firefox’s audio and video elements. Today we show what we did at www2010.

I’m in Raleigh, North Carolina, with Al MacDonald for the www2010 conference. We’re here to present our work on exposing audio data in the browser. Over the past month Corban, Charles, and a bunch of other friends have been working with us to refine the API and get new types of demos ready. We ended-up with 11 demos, some of which I’ve shown here before. Here are the others.

The first was done by Jacob Seidelin, and shows many cool 2D visualizations of audio using our API. You can see the live version on his site, or check out this video:

The second and third demos where done by Charles Cliffe, and show 3D visualizations using WebGL and his CubicVR engine. These also show off his JavaScript beat detection code. Is JavaScript fast enough to do real-time analysis of audio and synchronized 3D graphics? Yes, yes it is. The live versions are here and here, and here are some videos:

The fourth demo was done by Corban Brook and shows how audio data can be mixed live using script. Here he mutes the main audio, plays it, passes the data through a low pass filter written in JavaScript, then dumps the modified frames into a second audio element to be played. It’s a technique we need to apply more widely, as it holds a lot of potential. Here’s the live version, and here’s a video (check out his updated JavaScript synthesizer, which we also presented):

The fifth and sixth demos were done by Al (with the help of many). When I was last in Boston, for the Processing.js meetup at Bocoup, we met with Doug Schepers from the W3C. He loved our stuff, and was talking to us about ideas that would be cool to build. He pulled out his iPhone and showed us Brian Eno’s Bloom audio app. “It would be cool to do this in the browser.” Yeah, it is cool, and here it is, written in a few hundred lines of JavaScript and Processing.js (video 1, video 2):

This demo also showcases the awesome work of Felipe Gomes, who has a patch to add multi-touch DOM events to Firefox. The method we’ve used here can be taken a lot further. Imagine being able to connect multiple browsers together for collaborative music creation, layering other audio underneath, mixing fragments vs. just oscillators, etc. We built this one in a week, and the web is capable of a lot more.

One of the main points of our talk was to emphasize that what we’re talking about here isn’t just a concept, and it isn’t some far away future. This is real code, running in a real browser, and it’s all being done in HTML5 and JavaScript. The web is fast enough to do real-time audio processing now, powerful enough and expressive enough to create music. And the community of digital music and audio hackers, visualizers, etc. are hungry for it. So hungry that they are seeking us out, downloading our hacked builds and creating gorgeous web audio applications.

We want to keep going, and we need help. We need help from those within Mozilla, the W3C, and other browsers to get this stuff into shipping browsers. We need the audio, digital music, accessibility, and web communities to come together in order to help us build js audio libraries and more sample applications. Yesterday Joe Hewitt was talking on twitter about how web browser vendors need to experiment more with non-standard APIs. I couldn’t agree more, and here’s a chance for people to put their money where their mouth is. Let’s make audio a scriptable part of the open web.

I’m currently creating new builds of our updated patch for Firefox, and will post links to them here when I’m done. You can read more about the technical details of our work here, and get involved in the bug here. You can talk more with me on irc in the processing.js channel (I’m humph on moznet), or talk to me on twitter (@humphd) or by email. One way or another, get in touch so you can help us push this forward.

A Mozilla Developer Preview of improvements in the Gecko layout engine is now available for download. This is a pre-release version of the Gecko 1.9.3 platform, which forms the core of rich Internet applications such as Firefox. Please note that this release is intended for developers and testers only. As always, we appreciate any feedback you may have and encourage users to help us by filing bugs.

This developer preview introduces several new features, including:

• Support for CSS Transitions. This support is not quite complete: support for animation of transforms and gradients has not yet been implemented.
• Support for SMIL Animation in SVG. Support for animating some SVG attributes is still under development and the animateMotion element isn’t supported yet.
• Support for WebGL, which is disabled by default but can be enabled by changing a preference. See this blog post and this blog post for more details.
• Support for the getClientRects and getBoundingClientRect methods on Range objects. See bug 396392 for details.
• Support for the setCapture and releaseCapture methods on DOM elements. See bug 503943 for details.
• Support for the HTML5 History.pushState() and History.replaceState() methods and the popstate event. See bug 500328 for details.
• Support for the -moz-image-rect() value for background-image. See bug 113577 for more details.

and several other significant changes, including:

• On Mac OS X, we render text using Core Text rather than ATSUI.
• We rewrote major parts of the code for handling scrolling. See bug 526394 for details.
• We rewrote the way a snapshot of a document is taken in order to print or print preview. See bug 487667 for details.
• We made significant changes to table border handling. See bug 452319 and bug 43178 for details.
• We made various architectural changes to improve Web page performance.

More information on these changes is in the release notes, as well as the Upcoming Firefox features for developers article on the Mozilla Developer Center.

Please use the following links when downloading this Mozilla Developer Preview:

• Installer for Windows
• DMG for Mac OS X (10.5 or better required)
• Archive for Linux

For one thing, it means more developers can get involved in the evolution of WebGL. There’s a public mailing list set up, so that you can engage directly with members of the WebGL Working Group, as well as a web forum. It’s important to note that the specification is not yet finalized. Participation from the web community is essential towards finalizing the specification, which we hope to do in the first quarter of 2010.

It also means that there are implementations of the draft specification that you can begin to test. You can obtain a Firefox nightly that implements the WebGL draft specification, and can turn on WebGL support in that build by following these steps:

• Type “about:config” in your browser’s URL bar
• Do a search for “webgl” in the Filter field
• Double-click the “enabled_for_all_sites” preference to change it to “true”

Other browsers that support the draft WebGL specification are listed on the WebGL Wiki.

Now of course, this is hardware-accelerated 3D graphics in an early beta, and for now requires an OpenGL 2.0-capable GPU and drivers. In particular, most flavors of Intel’s integrated GPUs will not work straight out of the box (such as the GMA900/GMA950 often found in laptops). Developers who build nightly builds of the browser can induce software rendering using Mesa, and should check out Vlad’s blog post for doing this on Windows. Caveat emptor: building software rendering using Mesa into a Firefox nightly is likely to yield slower performance, and is only for the intrepid.

WebGL is a royalty-free, cross-platform API that brings OpenGL ES 2.0 to the web as a 3D drawing context within HTML5′s Canvas element, exposed as low-level interfaces within the Document Object Model.

Developers familiar with the Canvas 2D context will recognize that WebGL is another context for Canvas:

// get the canvas element in the DOM
var canvas = document.getElementById("canvas3D");
var gl = canvas.getContext("experimental-webgl");

Note that till the specification is finalized, the context is called experimental-webgl.

WebGL uses the OpenGL shading language, GLSL ES, and can be cleanly combined with other web content that is layered on top or underneath the 3D content. It is an emerging web standard, and is designed to be used with other parts of the web platform. The release of the draft specification is one step in bringing about a plugin free 3D API to the web, usable straight out of the box. People have already begun using it to create compelling libraries. Check out X3DOM, which is a JavaScript library using WebGL that allows users to author content in X3D. We expect developer ingenuity to surprise and awe us, as it always has.

References

• Review the WebGL draft specification
• Get involved! WebGL Homepage on Khronos
• Read articles and contribute on the WebGL Wiki
• A blog about learning WebGL
• Find 3D recipes in the learning WebGL Cookbook

For more information about WebGL check back here or follow planet WebGL.

I’ve been spending a lot of time lately thinking about 3D in the browser. It’s to the point that I have to blog about it, and try to get it out of my head and onto the web where it belongs. Part of the reason I’m writing this tonight is that my colleague, Cathy Leung, has just put up a demo so awesome I’m compelled to share it (video here):

She put the video together after this article was written about her work to make 3D accessible to web developers via C3DL. Cathy has been working with a team of students at Seneca College for the past few years, focused on an experimental 3D technology created by Mozilla. Her work proves how valuable 3D in the browser can be. I think a lot of people misunderstand its significance: “Doom in the browser?” Sure, but it’s way more than that. There is no shortage of tea pots on the web these days, as various browser vendors start to show-off early releases of WebGL enabled builds. However, Mozilla has been at this game a long time. I know because we’ve been using their stuff as long as they’ve been building it. And it’s really reached a new level now.

As Mozilla pushes WebGL further, and Cathy continues her work on C3DL, I’m focused on another project to harness the power of 3D in the browser. Together with Al MacDonald, I’m leading a Mozilla Foundation project to complete the work begun by John Resig to port the Processing language to JavaScript (aka processing.js). The processing.js port, while unfinished, is already amazing. Imagine if we took eight students, finished it, and added 3D support from C3DL? I know, exactly.

Part of what makes me so excited about all this work is the deep intersection of things I love. I’ve long been a fan of the work of Processing creators, Ben Fry and Casey Reas, every since they were part of the Aesthetics and Computation Group. I’ve always thought that their work belonged on the web instead of on the desktop. At the same time, my own work with Mozilla and the open web has proven to me the need for richer ways of working with things like open data. The web is at an interesting point in its evolution. We now take for granted that it can do what was once only possible via desktop apps. Looking forward I can already see a time when the web will also be the proper medium of artists, designers, data visualizers, and others pushing the limits of the second and third dimensions. I’m thrilled to be a small part of this effort now.

If you’d like to get involved with any of the work discussed above, please do get in touch with me. The most exciting aspect of everything I’ve told you remains that it’s all community developed, from Mozilla to C3DL to processing to processing.js. That’s how you build the software stack of the future.

A port of Puls to WebGL:

Escher-Droste effect in WebGL:

Metatunnel by FRequency:

(via Mark Steele)