Highlights

• To quickly edit the outerHTML of an element, we have now added support for element.outerHTML.
• The Battery API
• Storing files in IndexedDB
• Support for loop attributes for media elements
• SPDY (off by preferences)
• WebSockets no longer needs a prefix when you call them

All new features

Below is a list of all the new improvements in Aurora 11, grouped by category together with a link to each bug in bugzilla if you are interested in reading up more about it and its process.

DOM

• Support element.outerHTML property
• SVGSVGElement.getElementById not implemented
• IndexedDB: support indexedDB.cmp
• IndexedDB: multientry indexes
• IndexedDB now support all key types, including floating point numbers, dates and arrays!
• Battery API
• Support HTML parsing in XMLHttpRequest per XMLHttpRequest Level 2
• Remove the no-argument form of requestAnimationFrame
• requestAnimationFrame callback function name should be “sample”, not “onBeforePaint”
• Allow mozRequestAnimationFrame requests to be cancelable
• Use cancelRequestAnimationFrame where appropriate instead of boolean flags
• Add mozCancelAnimationFrame
• Implement Event constructors
• Implement HTML event ctors
• Implement MouseEvent and UIEvent ctors
• Enable storing files in IndexedDB
• IndexedDB: Allow passing an array as keypath

GFX

• Add Graphite2 to gecko

Layout

• -moz-text-size-adjust CSS property
• [css3-conditional] allow @-rules inside of @media and @-moz-document

Media

• Implement the looping attributes in media elements

Network

• SPDY (Preffed off)
• Enable Extended Protection (channel and service binding) for NTLM authentication
• add workaround for broken Outlook Web App (OWA) attachment handling
• HTTP content type charset parameter accepts single quotes
• XMLHttpRequest can fire an abort event after a load event, but should not
• Disallow responseType and withCredentials for sync XHR
• Allow Cross-Origin URLs in EventSource (Server-Sent Events)
• Support unprefixed responseType == “json” in XMLHttpRequest
• Implement Binary Messages for Websockets
• Add HSTS support for websockets
• No longer dispatch incoming WebSocket messages in CLOSING state
• Set WebSocket message size limit to 2 GB
• Tracking bug: unprefix WebSockets

Performance

• Lots of animated gifs swamp us with paint events

Compiling an existing speech synthesis engine to JavaScript is a good way to avoid writing a complicated project like eSpeak from scratch. Once compiled, the eSpeak code in speak.js doesn’t know it’s running on the web: speak.js uses the Emscripten emulated filesystem to ‘fake’ the normal file reading and writing calls that the eSpeak C++ code has (fopen, fread, etc.). This allows the normal eSpeak datafiles to be used (either through an xhr, or by converting them to JSON and bundling them with the script file). The result of running the compiled eSpeak code is that it ‘writes’ a .wav file with the generated audio to the emulated filesystem. speak.js then takes that data, encodes it using base64, and creates a data URL. That URL is then loaded in an HTML5 audio element, letting the browser handle playback. (Note that while that is a very simple way to do things, it isn’t the most efficient. speak.js has not yet focused on speed, but with some additional work it could be much faster, if that turns out to be an issue.)

Why would you want TTS in JavaScript? Well, with speak.js you can bundle a single .js file in your website, and then generating speech is about as simple as writing

speak("hello world")

(see the speak.js website for instructions). The generated speech will be exactly the same on all platforms, unlike if your users each did TTS in their own way (using an OS capability, or a separate program). speak.js can also be used to build browser addons in a straightforward way, since it’s pure JavaScript – no need for platform dependent binaries, and the addon will work the same on all OSes.

A few more comments:

• JavaScript is getting more and more capable all the time. The development versions of the top JavaScript engines today can run code compiled from C++ only 3-5X slower than a fast C++ compiler, and getting even better. As a consequence, expanding the capabilities of the web platform can in many cases be done in JavaScript or by compiling to JavaScript, instead of adding new code to the browsers themselves, which inevitably takes longer – especially if you wait for all browsers to implement a particular feature.
• While speak.js uses only standards-based APIs, due to browser limitations it can’t work everywhere yet. It won’t work in IE, Safari or Opera since they don’t support typed arrays, nor in Chrome since it doesn’t support WAV data URLs. So currently speak.js only works properly in Firefox. However, the missing features just mentioned are not huge and hopefully those browser makers will implement them soon. It is also possible to implement workarounds in speak.js for these issues (see next comment).
• Help with improving speak.js is very welcome! One important thing we need is to implement workarounds for the issues that prevent speak.js from running on the browsers it currently can’t run on. Another goal is to build browser addons using speak.js. Please get in touch on github if you want to help out.
• eSpeak supports multiple languages so speak.js can too. You do need to include the additional language files though. Here is an experimental build where you can switch between English and French support (note that it is an unoptimized build, so it will run slower).

The latest thing that make me go “wow” is jsmad (source on GitHub) by Amos Wenger, Jens Nockert and Matthias Georgi. JSMad is an MP3 decoder in JavaScript!

“So what”, you say? Well, having JSMad means that now Firefox can play MP3 files without any Flash. It also means that you can listen to MP3 in the browser without the 64bit issues on Linux. With JSMad we can dive deep into the MP3 format and not only play the song but also get information about it. It allows us to build a lot of native dj-mixers, samplers and sequencers in the nearer future.

Right now JSMad works in Firefox 4+ and on Chrome 13.0+, if you enable the Web Audio API in ‘about:flags’.

I remember when MP3 came out and my computer back then was too slow to encode it without locking up in WinAmp. Back then a scene player also helped me out. Now we do the same inside a browser rather than desktop applications.

In the fall, after finishing Flight of the Navigator, our team of audio and WebGL hackers was looking for a new challenge. We’d finished the new Audio API in time for Firefox 4, and were each maintaining various open web libraries, exploiting the new features of HTML5, Audio, JavaScript, and WebGL. We wanted to take another shot at testing the limits of Firefox 4 – then, still in beta.

Seth Bindernagel had the answer. He’d been in contact with a DJ and producer friend named Kraddy, who had just finished an amazing new album. “What if we tried to do something with his sound?” The idea was too good to pass up, and with Kraddy’s support, we dove into the tracks and started imagining what these songs might look like, when interpreted through the medium of the web.

«The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike»

Kraddy’s music was easy to demo because of its complex nature, with plenty of emphatic transitions and cue points–this music wants to be visualized! The music for No Comply also provided a dark and introspective sound on which to build a narrative. On his blog, Kraddy had already written about how he understood the album’s meaning:

This EP is about Theseus’ decision to be a hero and his decent into the Labyrinth to kill the Minotaur. In a broader sense the EP is about the battle we all face when we challenge ourselves as people. We must enter the Labyrinth of our minds and at the center we find our greatest fears. To defeat those fears we must kill a part of ourselves. And in killing a part of ourselves we create the potential to grow into a more developed person.

Kraddy’s vision informed our early outlines and storyboards. We knew that we wanted to play on the story of the Minotaur and the Maze, and the idea of facing down ones’ own fears. Together we came up with the idea of re-telling the story using a mixture of real-life video and 8-bit video game styling. Because the album was deeply personal to Kraddy, we decided to feature him in the demo. Kraddy agreed to be filmed, and Brett Gaylor used the footage to create the opening and closing video sequences. We also used Kraddy as the inspiration for the demo’s main video game character.

The launch of Firefox 4 brings a lot to the web, not least WebGL. As the web shifts from a 2D-only to a 2D and 3D space, we wanted to explore the intersection of these two familiar graphical paradigms. Rather than picking just one, we chose to create a hybrid, dream world, composed of 3D and 2D elements. Many people will recognize in our 2D characters and graphics an homage to much earlier video games, like Double Dragon. We wanted to celebrate the fact that these two paradigms can now exist together in a simple web page–everything we do in the demo is one web page, whether audio, video, 2D, 3D, or text.

Like the Flight of the Navigator(FOTN) demo before it, we chose the CubicVR.js engine to drive all the 3D graphics. Over the months leading up to the demo, Charles J. Cliffe had begun the painstaking process of porting features from his C++ engine over to JavaScript. The simple environment of WebGL and JavaScript allowed for features that even his C++ version did not yet posses to be quickly prototyped. Many bottlenecks had to be overcome during iterations of the demo, as we wanted to push the limits further than before. The biggest hurdle was visibility and lighting. Luckily, Bobby Richter came to the rescue. Using his experience with Octrees, he was able to work with Charles to produce a visibility and lighting pipeline which provides impressive performance for the task. In contrast, FOTN has no visibility system and was shaded by a single global directional light and ambient surface textures (for window lights, etc.) to simulate the rest. In No Comply we were able to push the limits with high poly counts and many overlapping point lights and were still able to reach the framerate cap.

Creating a 3D world like the one in this demo requires a lot of original content creation, which in turn requires some sophisticated tools. Instead of developing our own, and in the open-nature of our group, we decided to use existing technology like Blender. The community that develops Blender and creates content with it is rich and diverse, and because it’s an open tool, we could add the features we needed when they weren’t already present.

Our preference for open technologies also meant that the COLLADA scene format was an obvious choice. Unfortunately, as of version 2.49, Blender exports an Autodesk-inspired format of COLLADA, which isn’t quite up to the official standard, missing many important bits of information. Fixing this directly in Blender (with a little bit of Python hacking) let CubicVR stay standards-compliant, and let us milk Blender for all of the scene information we could think of using.

The demo’s 3D modelling, while important, comprises perhaps only half of No Comply’s original content. An incredible undertaking on the part of Omar Noory provided the textures for the rich environment through which Kraddy rumbles and tumbles. Frequently, spontaneous requests for “an 8 bit trash can,” “a cool sign with our names on it,” or, “some beefy bad lookin’ dudes” were answered almost instantly by Omar’s gracious and masterful digital pen. You may have recognized Omar’s name from his claim to meme-fame with “Haters Gonna Hate”.

Adding the perfect amount of flare to the graphics pipeline is Al MacDonald’s Burst animation engine. Al not only wrote our sprite animation engine, but also the web-based toolset we used to create the animations. The 8-bit Kraddy and all of No Comply’s 8-bit baddies are driven by animation paths prepared with Burst, and engineered with a set of tools that work right inside the browser.

In addition to cutting edge graphics with WebGL and <canvas>, we also wanted to explore how far we could push the new Firefox 4 Audio API we’d developed. The Audio Data API allows us to do many new things with the HTML5 <audio> and <video> tags, such as outputting generated audio and revealing realtime audio data to JavaScript. Libraries like Corban Brook’s DSP.js and and Charles’ BeatDetektor.js were used to analyze the audio in realtime and trigger various effects and animation sequences. Tracks of audio triggers were also recorded for tighter sequencing of key elements in the song we wanted to emphasize. One of the really new techniques we played with a lot in the demo was controlling GLSL shaders and lighting directly with audio, punching in and out with every beat and clap. Unlike most treatments of audio on the web, in this demo the song isn’t a background element, but is woven into the fabric of all the visuals and effects.

Getting a demo of this scale to work in the browser means figuring out how to make every bit of it work fast, and keep framerates high. Everything we do in the demo, from loading and parsing massive COLLADA models, to controlling 3D scene graphs, to analyzing real-time audio data, is done with JavaScript. We think it’s important to point this out because so many people begin with the assumption that JavaScript isn’t fast enough for the kind of work we’re presenting. The truth is that modern JavaScript, like that in Firefox 4, has been so heavily optimized that we all need to rethink what is and isn’t possible on the web.

We’ve taken advantage of a bunch of Firefox 4′s new performance features, as well as new HTML5 goodies, in order to make this all possible. For example Web Workers let us move heavy resource parsing off the main thread, freeing it for audio analysis and 3D effects. While a large portion of each second is consumed by simply pushing information to the video card, it isn’t necessary for the browser to wait for that to happen. In the background, we can use other threads to load and parse data, so that it’s ready to draw when the main thread needs it. Of course, a host of problems arise immediately whenever concurrency is involved, but we managed to draw a large performance and overall stability increase by utilizing Web Workers.

Another performance trick was using JavaScript Typed Arrays, which give us a tremendous speed boost when working with audio and pixel data. When you’re analyzing slices of audio data hundreds of bytes wide as fast as possible, your Fourier Transform code needs to be blazingly quick. Thanks to Corban’s highly optimized dsp.js library, this was hardly on our minds.

Next, we spent a lot of time optimizing our JavaScript so that it could take advantage of Firefox’s Tracing and Method JIT. Writing code that can be easily byte-compiled by the browser makes sure that anything we write runs as fast as possible. This is a fairly new and surprising concept, especially to those who remember the JavaScript of yesterday.

Part of what appealed to us about writing this demo was that it let those of us who are browser developers, and those of us who are web developers, work together on a single project. Most of the technology showcased in this demo was made on bleeding edge Firefox nightlies and our development process involved lots of feedback about performance or stability issues in the browser. Dave Humphrey focused on the internals of the Audio API, instrumenting and profiling our JavaScript, and helped us work closely with Mozilla’s JavaScript, graphics, and WebGL engineers. People like Benoit Jacob and Boris Zbarsky, among others, were indispensable as we worked to fix various bottlenecks. Part of what makes Mozilla such a successful project is that their engineers are not locked away, unable to work with web developers. Having engineers at our beck and call was essential to our success with such a demanding schedule, and we were proud to be able to help Mozilla test and improve Firefox 4 along the way.

Beyond the technical aspects of the demo, it also points to the spirit of how these technologies are meant to be used. We worked as a distributed team during evenings and on weekends, to plan and code and create everything, from the tools we needed to the graphical resources to the demo’s final code. Some of our team are browser developers, some web and audio hackers, others are graphic designers or filmmakers, still others storytellers and writers–everyone had a place around the table, and a role to play. We think this is part of what makes the web such a powerful platform for creative and collaborative work: there isn’t one right way to be, no single technology you need to know, and the techniques and tools are democratized and open to anyone willing to pick them up. The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike.

During the week we covered topics like WebGL for 3D development, basics of JavaScript, debugging and performance, canvas development, offline development and local storage and multimedia on the web. We’ll make these slides available in the comments to this blog post.

The slides of the multimedia talk are available here:

The topics covered were:

• Quick history of Multimedia on the web
• Annoyances with Flash
• HTML5 audio and video
• Codecs and conversion
• Embedding video and audio
• Controlling video
• Transforming video
• Making realtime changes
• Awesome new audio possibilities

We showed and explained a few demos like:

• The Media events demo page at the W3C web site
• The Spirit of Indiana demo syncing a Google Maps animation with a video
• Transforming Video to create a player that allows you to zoom inside a video and rotate it using CSS transformations
• Paul Rouget’s round video demo showing how to create a play/pause button in SVG, greyscale a video and rotate it using CSS transformations
• Paul’s Video Mashup demo showing how to apply SVG filters like Gaussian Blur, masking and how to apply CSS transforms to a running video
• Remy Sharp’s round video demo showing how to create a rotating, bouncing circle that contains a video
• Paul Rouget’s Green demo showing how to simulate a green screen effect with open web technologies.
• The dynamic content injection demo showing how to embed content into a video after analysing it for injection hotspots.
• The Tracker demo showing how to detect human shapes and movement in a video
• The edge detection demo shows how to detect edges in a live video using JavaScript
• The HTML5 Guitar Tab generator
• The sfxr sound effect library

I also provided extensive notes for the whole talk with code examples on my own blog.

Leggi la traduzione in italiano

Today we are featuring Remy Sharp co-author of Introducing HTML5 and organiser of the Full Frontal conference in Brighton, England.

Remy is one of those ubiquitous people of HTML5. Whenever something needed fixing, there is probably something on GitHub that Remy wrote that helps you. He is also very English and doesn’t mince his words much.

You can find Remy on Twitter as @rem.

The video interview

Watch the video on YouTube or Download it from Archive.org as MP4 (98 MB), OGG (70 MB) or WebM (68MB)

Ten questions about HTML5 for Remy Sharp

1) Reading “Introducing HTML5″ it seems to me that you were more of the API – focused person and Bruce the markup guy. Is that a fair assumption? What is your background and passion?

That’s spot on. Bruce asked me to join the project as the “JavaScript guy” – which is the slogan I wear under my clothes and frequently reveal in a superman ‘spinning around’ fashion (often at the surprise of clients).

My background has always been coding – even from a young age, my dad had me copying out listings from old spectrum magazines only to result in hours of typing and some random error that I could never debug.

As I got older I graduated to coding in C but those were the days the SDKs were 10Mb downloaded over a 14kb modem, and compile in to some really odd environment. Suffice to say I didn’t get very far.

Then along came JavaScript. A programming language that didn’t require any special development environment. I could write the code in Notepad on my dodgy Window 95 box, and every machine came with the runtime: the browser. Score!

From that point on the idea of instant gratification from the browser meant that I was converted – JavaScript was the way for me.

Since then I’ve worked on backend environments too (yep, I’m a Perl guy, sorry!), but always worked and played in the front end in some way or another. However, since started on my own in 2006, it’s allowed me to move focus almost entirely on the front end, and specialise in JavaScript. Basically, work-wise: I’m a pig in shit [Ed: for our non-native English readers, he means "happy")].

2) From a programmer’s point of view, what are the most exciting bits about the HTML5 standard? What would you say is something every aspiring developer should get their head around first?

For me, the most exciting aspects of HTML5 is the depth of the JavaScript APIs. It’s pretty tricky to explain to Joe Bloggs that actually this newly spec’ed version of HTML isn’t mostly HTML; it’s mostly JavaScript.

I couldn’t put my finger on one single part of the spec, only because it’s like saying which is your favourite part of CSS (the :target selector – okay, so I can, but that’s not the point!). What’s most exciting to me is that HTML5 is saying that the browser is the platform that we can deliver real applications – take this technology seriously.

If an aspiring developer wanted something quick and impressive, I’d say play around with the video API – by no means is this the best API, just an easy one.

If they really wanted to blow people away with something amazing using HTML5, I’d say learn JavaScript (I’m assuming they’re already happy with HTML and CSS). Get a book like JavaScript: The Good Parts and then get JavaScript Patterns and master the language. Maybe, just maybe, then go buy Introducing HTML5, it’s written by two /really/ good looking (naked) guys: http://flic.kr/p/8iyQTE and http://flic.kr/p/8iy6Z1 [Ed: maybe NSFW, definitely disturbing].

3) In your book you wrote a nice step-by-step video player for HTML5 video. What do you think works well with the Video APIs and what are still problems that need solving?

The media API is dirt simple, so it means working with video and audio is a doddle. For me, most of it works really well (so long as you understand the loading process and the events).

Otherwise what’s really quite neat, is the fact I can capture the video frames and mess with them in a canvas element – there’s lots of fun that can be had there (see some of Paul Rouget’s demos for that!).

What sucks, and sucks hard, is the spec asks vendors, ie. browser makers, *not* to implement full screen mode. It uses security concerns as the reason (which I can understand), but Flash solved this long ago – so why not follow their lead on this particular problem? If native video won’t go full screen, it will never be a competitive alternative to Flash for video.

That all said, I do like that the folks behind WebKit went and ignored the spec, and implemented full screen. The specs are just guidelines, and personally, I think browsers should be adding this feature.

4) Let’s talk a bit about non-HTML5 standards, like Geolocation. I understand you did some work with that and found that some parts of the spec work well whilst others less so. Can you give us some insight?

On top of HTML5 specification there’s a bunch more specs that make the browser really, really exciting. If we focus on the browser being released today (IE9 included) there’s a massive amount that can be done that we couldn’t do 10 years ago.

There’s the “non-HTML5″ specs that actually were part of HTML5, but split out for good reason (so they can be better managed), like web storage, 2D canvas API and Web Sockets, but there’s also the /really/ “nothing-to-do-with-HTML5″ APIs (NTDWH5API!) like querySelector, XHR2 and the Device APIs. I’m super keen to try all of these out even if they’re not fully there in all the browsers.

Geolocation is a great example of cherry picking technology. Playing against the idea that the technology isn’t fully implemented. Something I find myself ranting on and on about when it comes to the question of whether a developer should use HTML5. Only 50% of Geolocation is implemented in the browsers supporting it, in that they don’t have altitude, heading or speed – all of which are part of the spec. Does that stop mainstream apps like Google Maps from using the API? (clue: no).

The guys writting the specs have done a pretty amazing job, and in particular there are few cases where the specs have been retrospectively written. XHR is one of these and now we’ve got a stable API being added in new browsers (i.e. IE6 sucks, yes, we all know that). Which leads us to drag and drop. The black sheep of the APIs. In theory a really powerful API that could make our applications rip, but the technical implementation is a shambles. PPK (Peter-Paul Koch) tore the spec a bit of a ‘new one’. It’s usable, but it’s confusing, and lacking.

Generally, I’ve found the “non-HTML5″ specs to be a bit of mixed bag. Some are well supported in new browsers, some not at all. SVG is an oldie and now really viable with the help of JavaScript libraries such as Raphaël.js or SVGWeb (a Flash based solution). All in all, there’s lots of options available in JavaScript API nowadays compared to back in the dark ages.

5) Let’s talk Canvas vs. SVG for a bit. Isn’t Canvas just having another pixel-based rectangle in the page much like Java Applets used to be? SVG, on the other hand is Vector based and thus would be a more natural tool to do something with open standards that we do in Flash now. When would you pick SVG instead of Canvas and vice versa?

Canvas, in a lot of ways is just like the Flash drawing APIs. It’s not accessible and a total black box. The thing is, in the West, there’s a lot of businesses, rightly or wrongly, that want their fancy animation thingy to work on iOS. Since Flash doesn’t work there, canvas is a superb solution.

However, you must, MUST, decide which technology to use. Don’t just use canvas because you saw a Mario Kart demo using it. Look at the pros and cons of each. SVG and the canvas API are not competitive technologies, they’re specially shaped hammers for specific jobs.

Brad Neuberg did a superb job of summarising the pros and cons of each, and I’m constantly referring people to it (here’s the video).

So it really boils down to:

• Canvas: pixel manipulation, non-interactive and high animation
• SVG: interactive, vector based

Choose wisely young padawan!

6) What about performance? Aren’t large Canvas solutions very slow, especially on mobile devices? Isn’t that a problem for gaming? What can be done to work around that?

Well…yes and no. I’m finishing a project that has a large canvas animation going on, and it’s not slow on mobile devices (not that it was designed for those). The reason it’s not slow is because of the way the canvas animates. It doesn’t need to be constantly updating at 60fps.

Performance depends on your application. Evaluate the environment, the technologies and make a good decision. I personally don’t think using a canvas for a really high performance game on a mobile is quite ready. I don’t think the devices have the ommph to get the job done – but there’s a hidden problem – the browser in the device isn’t quite up to it. Hardware acceleration is going to help, a lot, but today, right now, I don’t think we’ll see games like Angry Birds written in JavaScript.

That said… I’ve seriously considered how you could replicate a game like Canabalt using a mix of canvas, DIVs and CSS. I think it might be doable ::throws gauntlet::

I think our community could actually learn a lot from the Flash community. They’ve been through all of this already. Trying to make old versions of Flash from years back do things that were pushing the boundaries. People like Seb Lee-Delisle (@seb_ly / http://seb.ly) are doing an amazing job of teaching both the Flash and JavaScript community.

7) A feature that used to be HTML5 and is now an own spec is LocalStorage and its derivatives Session Storage or the full-fledged WebSQL and IndexedDB. Another thing is offline storage. There seems to be a huge discussion in developer circles about what to use when and if NoSQL solutions client side are the future or not. What are your thoughts? When can you use what and what are the drawbacks?

Personally I love seeing server-less applications. Looking at the storage solutions I often find it difficult to see why you wouldn’t use WebStorage every time.

In a way it acts like (in my limited experience of) NoSQL, in that you lookup a key and get a result.

Equally, I think SQL in the browser is over the top. Like you’re trying to use the storage method *you* understand and forcing it into the browser. Seems like too much work for too little win.

Offline Apps, API-wise, ie. the application cache is /really/ sexy. Like sexy with chocolate on top sexy. The idea that our applications can run without the web, or switch when it detects it’s offline is really powerful. The only problem is that the events are screwed. The event to say your app is now offline requires the user to intervene via the browser menu, telling the browser to “work in offline mode”. A total failure of experience. What’s worse is that, as far as I know, there’s no plan to make offline event fire properly :-(

That all said, cookies are definitely dead for me. I’ve yet to find a real solution for cookies since I found the Web Storage API – and there’s a good decent number of polyfills for Web Storage – so there’s really no fear of using the API.

8) Changing the track a bit, you’ve built the HTML5shiv to make HTML5elements styleable in IE. This idea sparked quite a lot of other solutions to make IE6 work with the new technologies (or actually simulate them). Where does this end? Do you think it is worth while to write much more code just to have full IE6 support?

There’s two things here:

1. Supporting IE6 (clue: don’t)
2. Polyfills

IE6, seriously, and for the umpteenth time, look at your users. Seriously. I know the project manager is going to say they don’t know what the figures are, in that case: find out! Then, once you’ve got the usage stats in hand, you know your audience and you know what technology they support.

If they’re mostly IE6 users, then adding native video with spinning and dancing canvas effect isn’t going to work – not even with shims and polyfills. IE6 is an old dog that just isn’t up to doing the mileage he used to be able to do back in his prime. But enough on this subject – the old ‘do I, or don’t I developer for IE6′ is long in the tooth.

Polyfills – that’s a different matter. They’re not there to support IE6, they’re there to bring browsers up to your expectations as a developer. However, I’d ask that you carefully consider them before pulling them in. The point of these scripts is they plug missing APIs in those older browsers. “Old browsers” doesn’t particularly mean IE6. For example, the Web Sockets API has a polyfill by way of Flash. If native Web Sockets aren’t there, Flash fills the gap, but the API is exposed in exactly the same manner, meaning that you don’t have to fork your code.

I don’t think people should be pulling in scripts just for the hell of it. You should consider what you’re trying to achieve and decide whether X technology is the right fit. If it is, and you know (or expect) your users have browsers that don’t support X technology – should you plug it with JavaScript or perhaps should you consider a different technology?

This exact same argument rings true for when someone adds jQuery just to add or remove a class from an element. It’s simply not worth it – but clearly that particular developer didn’t really understand what they needed to do. So is education the solution? I should hope so.

9) Where would you send people if they want to learn about HTML5? What are tutorials that taught you a lot? Where should interested people hang out?

HTML5 Doctor – fo sho’. :)

I tend to also direct people to my http://html5demos.com simply to encourage viewing source, and hacking away.

Regarding what tutorials taught me – if I’m totally honest, the place I’ve learnt the most from is actually HTML5Doctor.com. There’s some pretty good JavaScript / API tutorials coming from the chaps at http://bocoup.com. Otherwise, I actually spend a lot of time just snooping through the specifications, looking for bits that I’ve not seen before and generally poking them with a stick.

10) You have announced that you are concentrating on building a framework to make Websockets easy to work with. How is that getting on and what do you see Websockets being used for in the future? In other words, why the fascination?

Concentrating is a strong word ;-) but it is true, I’ve started working on a product that abstracts Web Sockets to a service. Not the API alone, since it’s so damn simple, but the server setup: creating sessions, user control flow, waiting for users and more.

The service is called Förbind. Swedish for “connect”, ie. to connect your users. It’s still early days, but I hope to release alpha access to forbind.net this month.

I used to work in finance web sites and real-time was the golden egg: to get that data as soon as it was published. So now that it’s available in a native form in the browser, I’m all over it!

What’s more, I love the idea of anonymous users. I created a bunch of demos where the user can contribute to something without ever really revealing themselves, and when the users come, you start to see how creative people are without really trying. Sure, you get a lot of cocks being drawn, but you also see some impressive ideas – my business 404 page for example allows people to leave a drawing, one of the most impressive is a Super Mario in all his glory. Anonymous users really interest me because as grey as things can seem sometimes, a stranger can easily inspire you.

Do you know anyone I should interview for “People of HTML5″? Tell me on Twitter: @codepo8

I created a mashup using the Firefox 4 Audio Data API, MusicXML, and Vexflow to create a HTML 5 Guitar Tab Player.

This is a Youtube video. It uses the HTML5 <video> tag if you activate it here.

• View demo.

• Source code.

Using JavaScript, this guitar tab player converts the music note data from a MusicXML file to frequencies playable by the Audio Data API. This note data is also converted to a format that can be read by Vexflow to display the guitar tab.

I have broken down this functionality into 4 steps. Here’s how the HTML5 Guitar Tab Player works:

MusicXML

When a tab (a tablature) is selected, the jQuery ajax() function loads the MusicXML file, parses through the contents, and appends all the formatted note data to a <script> tag. The format for each note is: playNote(note, octave, duration).

playNote(notes.C, 4, 1);

The tablature notation information in the MusicXML file is also processed and appended to a <div> tag for Vexflow.

Here is the structure of the nodes I am using from the MusicXML file:

<measure number="1">
    <attributes>
        <divisions>1024</divisions>
    </attributes>
    <note>
        <pitch>
            <step>C</step>
            <octave>4</octave>
        </pitch>
        <duration>1024</duration>
        <notations>
            <technical>
                <fret>1</fret>
                <string>2</string>
            </technical>
        </notations>
    </note>
    the note node repeats for the rest of the notes in the measure…
</measure>

And here is the generated source that is produced by the ajax() function.

For the audio:

<script id="audio" type="text/javascript">
	function loadAudio(){
	  playNote(notes.C, 4, 1);
	  playNote(notes.C, 4, 1);
	  playNote(notes.G, 4, 1);
	  playNote(notes.G, 4, 1);
	  …
	}
</script>

For the tab:

<div id="tab" class="vex-tabdiv" width=780 scale=1.0 editor="false">
tabstave notation=true
	notes :q 1/2 :q 1/2 :q 3/1 :q 3/1 | :q 5/1
   …
</div>

Frequency Calculation

When the Play button is clicked the music data <script id=”audio”> tag is evaluated and the note information is converted to frequencies using the following formula (assumes equal temperament):

frequency = 440 * 2^(n/12)

440 is the frequency for the note A at the 4th octave.

n = distance away from A4 (440)

I am using an array to store the distance between a certain note and the base note “A”

notes = {C:-9, Cs:-8, D:-7, Ds:-6, E:-5, F:-4, Fs:-3, G:-2, Gs:-1, A:0, As:1, B:2};

And to incorporate that a note can be played at different octaves I have adapted the above formula to the following:

frequency = 440 * 2^((octave-4) * 12 + distance)/12);

Audiodata.js

I am using the Audiodata.js library in the guitar tab player which makes it easy to play continuous audio using a single array. This library encapsulates the Audio Data API methods. Audiodata.js is available on GitHub here: https://github.com/notmasteryet/audiodata. The Audiodata.js project contains a score example with “Twinkle, Twinkle, Little Star” which inspired me to build this guitar tab player.

Vexflow and the Animated Cursor

Vexflow is an open-source web-based music notation rendering API based on the HTML5 canvas element (http://www.vexflow.com/). I am using Vexflow to display the tab for each MusicXML file. I have added an additional canvas element on top of the Vexflow canvas to control the red cursor that links the audio to the tablature. The speed of the audio is controlled by the tempo which is measured in beats per minute. By converting this tempo to milliseconds I am able to use it for the redraw speed of the second canvas. Every time this canvas is redrawn the red cursor is moved 5 pixels to the right to highlight which note is currently being played.
Keep in mind that this HTML5 Guitar Tab Player is just a proof of concept and can only play single notes.

If you have recommendations on how to make this tab player better or would like to contribute to this project check out the following post: http://www.gregjopa.com/2010/12/html5-guitar-tab-player-with-firefox-audio-data-api/

Flight of the Navigator is a WebGL + Audio API demo developed by a team of Mozilla volunteers.

You can see the demo online here (you need a WebGL compatible browser).

More information about this demo on David Humphrey’s blog (one of the developer of this demo) and more about WebGL in Firefox Beta8 on Vlad’s blog.

The screencast:

See the demo online

You can download the animation demo for yourself and try it out locally – all you need is a browser that supports HTML5 video. I know – the music is not quite the same as in the movies, but at least this one is not copyright infringing and it came from the heart (5 minutes in a meeting room in the Mozilla office).

So how was this done and what are problems that needed solving? Here’s how and what.

Step 1: Find the movie and get it to the right format

That was the easy part. Archive.org has a lot of awesome public domain movies available for you and they are already in the formats needed to use in an HTML5 video element. In this case, I took the short movie of Charles Lindbergh taking off for his record breaking flight from New York to fly to Paris in 1927.

Step 2: Displaying the video

Using the video is pretty simple:

<div id="stage">
  <video>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff_512kb.mp4" 
type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff.ogv"
 type='video/ogg; codecs="theora, vorbis"'> 
  </video>
</div>

The MP4 format will be used by Webkit based browsers and the Ogg version by Firefox and others. As we want to control the video we omit the controls attribute on the video element – instead we create a button to play the video with JavaScript:

window.addEventListener('load', 
  function() {
    var stage = document.getElementById('stage');
    var v = document.getElementsByTagName('video')[0];
    but = document.createElement('button');
    but.innerHTML = 'Click to see Lindbergh\'s flight';
    stage.appendChild(but);
    but.addEventListener('click',function(e) {
      v.play();
      e.preventDefault();
    },false);
  }, 
false);

As the video is markup we can do whatever we please with it – the power of open technologies. For example as we will do here we can set its opacity in CSS and put in on top of a map.

Step 3: Create the map path animation

Talking of which, let’s get that moving path done. Google Earth has an API to do that, but it needs a special plugin. Google Maps allows you to paint paths on maps (which actually are SVG, another open standard). Put that in a recursive function and you get the desired effect:

In essence, what I did was take the latitude and longitude of the start and end points and calculate as many points in between the two as I need for the duration of the animation. I store the points in an array called pos and then paint a path from the start to the current point and move the map centre to this point on every iteration.

spirit.draw = function(){
  var path = new google.maps.Polyline({
        path: [startpos,pos[now]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
  });
  path.setMap(map);
  map.panTo(pos[now])
  now = now + 1;
  if(now < animationend-1){
    setTimeout(spirit.draw,200);
  }
}

Check the highly commented source of the map example for the details. Now, we could use this kind of animation and play the video over it – the problem though is that they may get out of sync. When the movie stalls (as it frequently does on this hotel wireless connection) we don’t want the animation to keep moving, right?

Step 4: Syncing video and the map movement

Instead of having two sources of timing information we have to limit ourselves to one source of truth. This is the time stamp of the currently playing movie.

By the way – you might have noticed that I wrapped the map code in a tilesloaded event handler. This is another safeguard for keeping things in sync. I found that on slow connections the tile loading can delay the whole interface immensely (because of all the subdomain lookups), so I make the whole interface dependent on the loading of the map and only proceed when the tiles have finished loading. As the tilesloaded event also fires when the map pans we need to use a boolean to stop it from starting the effect several times:

google.maps.event.addListener(map,'tilesloaded',function(){
  if(played === false){
    // [...other code...]
    played = true;
  }
});

You can read the current timestamp of a video with video.currentTime and whilst the movie is playing it constantly fires an event called timeupdate. As the event fires a lot we need to throttle it somehow. The trick here is to only take the full seconds and increase a counter when a new second is reached. You can see the timestamp and the second interval firing in the video syncing demo:

var now = 0;
v.addEventListener('timeupdate',function(o){
  log.innerHTML = v.currentTime; /* logging the real timestamp */
  var full = parseInt(v.currentTime);
  if(full >= now) {
    seqlog.innerHTML = now;  /* logging the seconds firing */
    now = now + 1;
  }
},false);

That way the movie can lag in between and the sequence still stays in sync. Check the source of this demo on Github.

Putting it all together

And that was about it – all I had to do is to set the movie’s opacity at a certain time stamp, start the sound at another and show and hide the copyright complaint at another. As we rely on the timestamp for the other effects we needed a boolean switch to avoid repeated firing:

v.addEventListener('timeupdate',function(o){
  full = parseInt(v.currentTime);
  if(full === now-1){
    mapelm.style.opacity = .8;
    v.style.opacity = .4;
  }
  if(full === animationstart+1 && audioplay === false){
    a.play();
    audioplay = true;
  }
  if(full === animationstart+2 && hidden === true){
    drmbedamned.style.display = 'block';
    hidden = false;
  }
  if(full === animationstart+3 && hidden === false){
    drmbedamned.style.display = 'none';
    hidden = true;
  }
  if(full >= now) {
    path = new google.maps.Polyline({
        path: [startpos,pos[full]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
    });
    path.setMap(map);
    map.panTo(pos[full])
    now = now + 1;
  }
},false);

Another event we needed to subscribe to was the movie ending so we can stop the music and start to roll the credits:

v.addEventListener('ended',function(o){
  a.pause();
  spirit.credslist.parentNode.style.display = 'block';
  spirit.creds();
},false)

As the theme is too short for the whole animation we need to loop it. This can be done by testing for the ended event and rolling back the time to 0:

a.addEventListener('ended', function(o) {
  a.currentTime = 0;
},false);

Summary

And there you have it – Indiana Jones style maps using open services and open technologies. A workaround for the copyrighted audio (recorded, edited and converted with the free Audacity sound editor) and using Google’s Web Fonts as graphics.

You can now take this and change it for even more awesome:

• Replace Google Maps with Openstreetmap to avoid going over the limit
• Add a slight curve to the path from NYC to Paris to make it more accurate (but then again the time is not accurate either – it took charles a tad longer)
• Use a static map and paint the path with Canvas to speed up and smoothen the animation

Why not have a go – it is free and fun to play.

Simply, the demo uses the mouse button for a crash cymbal, mouse actions as an arpeggiator, and the accelerometer (Orientation events) to apply a low-pass filter. I remember Thai from BarCamp Bangkok 2 where he made a Wiimote demo with Linux (and he was 14 then.)

If you also have a demo and some code to share, please let us know: @mozhacks (twitter account) or leave us a comment.

Late last week I landed support on Firefox trunk for the HTML5 video ‘preload’ attribute. This replaces the ‘autobuffer’ attribute, which we previously supported. If you were previously using the autobuffer attribute on your HTLM5 videos, you need to update to using the preload attribute as well.

The preload attribute provides a hint to the browser as to how much downloading is sensible for a given media. The preload attribute can have three values:

1. “none” – suggests to the browser that it doesn’t need to load this media at all until the user plays the resource. The browser will delay any network traffic required to load the media until the users tries to play the resource, or explicitly loads the resource. I suggest using this preload value, along with the poster attribute, when it’s unlikely that the user will play the resource. This is probably most useful in a mobile environment, where data can be expensive.
2. “metadata” – suggests to the browser that it isn’t necessary to load the entire resource in advance. The browser will suspend the load after loading metadata, displaying the first video frame (if there’s no poster image), and ensuring it can play the media. This is the default behaviour, and prevents excess network traffic when the web developer isn’t certain the video will definitely be played. If you don’t specify a preload value, Firefox will automatically do this. This was also the default behaviour in Firefox 3.5 and 3.6 in the absence of the autobuffer attribute. This default behaviour is a suitable compromise between bandwidth saving and user convenience.
3. “auto” – suggests to the browser that it should load as much of the resource as possible. As long as the browser’s own media cache isn’t full, it will keep on downloading. I suggest this is most useful in the “YouTube” case, when you’ve got a media which the user is almost certainly going to watch, and so having the user download the media is not likely to be wasting server bandwidth. This behaviour is the same as using the autobuffer attribute in Firefox 3.5 and 3.6

Since the autobuffer attribute is no longer present in Firefox 4, and the preload attribute is not present in Firefox 3.5 and 3.6, if you want a media to download completely, you should include both preload=”auto” and autobuffer in the video element, e.g.:

<video autobuffer preload=”auto” src=”video.ogg”></video>

If you want the video to only be downloaded if the user actually plays the video, you should not include either the preload or the autobuffer attribute. The default behaviour in Firefox 3.5, 3.6, and the upcoming Firefox 4 is to only load up to the first frame and then suspend the download.

Recently I landed support for the HTML5 video ‘buffered’ property in Firefox. This is cool because we can now accurately determine which time-segments of a video we can play and seek into without needing to pause playback to download more data. Previously you could only get the byte position the download had reached, which often doesn’t map to the time ranges which are playable very well, especially in a variable bit rate video. This also can’t tell you if there are chunks which we skipped downloading before the downloaded byte position. Once the video controls UI is updated, users will be able to know exactly which segments of their video are downloaded and playable and can be seeked into without pausing playback to download more data.

To see this in action, download a current Firefox nightly build , and point your browser at my video ‘buffered’ property demo. You’ll see something like the screenshot below, including an extra progress bar (implemented using canvas) showing the time ranges which are buffered.

I’ve implemented the ‘buffered’ property for the Ogg and WAV backends. Support for the ‘buffered’ property for WebM is being worked on by Matthew Gregan, and is well underway. At the moment we return empty ranges for the ‘buffered’ property on video elements playing WebM and raw video.

My checkin just missed the cutoff for Firefox 4 Beta 3, so the first beta release that the video ‘buffered’ property will appear in is Firefox 4 Beta 4.

I think that my favourite demo by far this time around is one that I’ve been waiting to see since we first began these experiments. I’ve written in the past that our work could be used to solve many web accessibility problems. A few weeks ago I mentioned on irc that someone should take a shot at building a text to speech engine in JavaScript, now that we have typed arrays. Yury quietly went off and built one based on the flite engine. When you run this, remember that you’re watching a browser speak with no plugins of any kind. This is all done in JavaScript.

Web Audio Data API – Text to Speech Demo from David Humphrey on Vimeo.

David’s work is important because it’s showing where the web is going, and where Mozilla is helping to take it. It’s not enough that we’re working on HTML5, which we’re about finished with, but we’re trying to figure out what’s next. Mozilla’s platform, Gecko, is a huge part of why we’re able to experiment and learn as fast as we can. And that’s reflected with what’s possible here. It’s a web you can see, touch and interact with in new ways.

David’s post follows:

I’m working with an ever growing group of web, audio, and Mozilla developers on a project to expose audio spectrum data to JavaScript from Firefox’s audio and video elements. Today we show what we did at www2010.

I’m in Raleigh, North Carolina, with Al MacDonald for the www2010 conference. We’re here to present our work on exposing audio data in the browser. Over the past month Corban, Charles, and a bunch of other friends have been working with us to refine the API and get new types of demos ready. We ended-up with 11 demos, some of which I’ve shown here before. Here are the others.

The first was done by Jacob Seidelin, and shows many cool 2D visualizations of audio using our API. You can see the live version on his site, or check out this video:

The second and third demos where done by Charles Cliffe, and show 3D visualizations using WebGL and his CubicVR engine. These also show off his JavaScript beat detection code. Is JavaScript fast enough to do real-time analysis of audio and synchronized 3D graphics? Yes, yes it is. The live versions are here and here, and here are some videos:

The fourth demo was done by Corban Brook and shows how audio data can be mixed live using script. Here he mutes the main audio, plays it, passes the data through a low pass filter written in JavaScript, then dumps the modified frames into a second audio element to be played. It’s a technique we need to apply more widely, as it holds a lot of potential. Here’s the live version, and here’s a video (check out his updated JavaScript synthesizer, which we also presented):

The fifth and sixth demos were done by Al (with the help of many). When I was last in Boston, for the Processing.js meetup at Bocoup, we met with Doug Schepers from the W3C. He loved our stuff, and was talking to us about ideas that would be cool to build. He pulled out his iPhone and showed us Brian Eno’s Bloom audio app. “It would be cool to do this in the browser.” Yeah, it is cool, and here it is, written in a few hundred lines of JavaScript and Processing.js (video 1, video 2):

This demo also showcases the awesome work of Felipe Gomes, who has a patch to add multi-touch DOM events to Firefox. The method we’ve used here can be taken a lot further. Imagine being able to connect multiple browsers together for collaborative music creation, layering other audio underneath, mixing fragments vs. just oscillators, etc. We built this one in a week, and the web is capable of a lot more.

One of the main points of our talk was to emphasize that what we’re talking about here isn’t just a concept, and it isn’t some far away future. This is real code, running in a real browser, and it’s all being done in HTML5 and JavaScript. The web is fast enough to do real-time audio processing now, powerful enough and expressive enough to create music. And the community of digital music and audio hackers, visualizers, etc. are hungry for it. So hungry that they are seeking us out, downloading our hacked builds and creating gorgeous web audio applications.

We want to keep going, and we need help. We need help from those within Mozilla, the W3C, and other browsers to get this stuff into shipping browsers. We need the audio, digital music, accessibility, and web communities to come together in order to help us build js audio libraries and more sample applications. Yesterday Joe Hewitt was talking on twitter about how web browser vendors need to experiment more with non-standard APIs. I couldn’t agree more, and here’s a chance for people to put their money where their mouth is. Let’s make audio a scriptable part of the open web.

I’m currently creating new builds of our updated patch for Firefox, and will post links to them here when I’m done. You can read more about the technical details of our work here, and get involved in the bug here. You can talk more with me on irc in the processing.js channel (I’m humph on moznet), or talk to me on twitter (@humphd) or by email. One way or another, get in touch so you can help us push this forward.

HTML5 Canvas and Audio Experiment

Please comment on their cool experiment at their blog: HTML5 Canvas Experiment.

The demo uses XML to describe the various segments of a piece of music – Florence Baschet’s StreicherKreis (Circle of Strings). The music itself is a combination of stringed instruments and electronic effects. From the XML, SVG is generated for each section of the music. You can click on each section to listen to that part of the piece and a description is shown on how that particular section was created.

As far as demos go, this is relatively simple. But it’s worth highlighting because it shows how easy it is to build a timeline around a piece of music and add descriptive information. In this case, it’s information meant to teach people how a particular effect was created. But it could be anything, from showing different camera angles of people playing the music to links about different covers of a popular piece. Opening up media to the web means that we can combine it with text, images and other media. This is just a small example.

View the Demo in Firefox 3.5

View the Demo in Firefox 3.5

But what’s really interesting about this demo is not so much that it plays audio – lots of people have built audio players – but how it works. If you look at the source code for the page what you’ll find looks something like this:

<div id="jai">
  <canvas id="jai-transport" width="320" height="20"></canvas>
  <ul class="playlist">
    <li>
      <a href="@F1LT3R - Cryogenic Unrest.ogg">
        F1LT3R - Cryogenic Unrest
      </a>
      <audio src="@F1LT3R - Cryogenic Unrest.ogg"/>.
    <li>
      <a href="@F1LT3R - Ghosts in HyperSpace.ogg">
        F1LT3R - Ghosts in HyperSpace
      </a>
      <audio src="@F1LT3R - Ghosts in HyperSpace.ogg"/>.       
  </ul>    
</div>

That’s right – the player above is just a simple HTML unordered list that happens to include audio elements and is styled with CSS. You’ll notice that if you right click on one of them that it has all the usual items – save as, bookmark this link, copy this link location, etc. You can even poke at it with Firebug.

The JavaScript driver that Al has written will look for a <div> element with the jai ID and then look for any audio elements that are inside it. It then will draw the playback interface in the canvas at the top of the list. The playback interface is built with simple JS canvas calls and an SVG-derived font.

Using this driver it’s super-easy to add an audio player to any web site by just defining a canvas and a list. Much like what we’ve seen on a lot of the web with the rise of useful libraries like jQuery, this library can add additional value to easily-defined markup. Another win for HTML5 and the library model.

Al has a much larger write-up on the same page as the demo. If you haven’t read through it you should now.

(Also? Al wrote the music himself. So awesome.)

Recent developer nightlies of Google Chrome support these codecs and a future version of Opera will also support them. Theora and Vorbis also work in Safari if you install the Xiph Qt component. We’re quickly reaching the point where all modern browsers support these open codecs with full support for the video tag.

You’ll note that Greg’s post doesn’t have the tone of a marketing document – it’s not meant to. Nor is this a comparison against HD-sized, high-bitrate video. Instead it’s an attempt to give an honest comparison of how the open codecs fare against commonly-used formats and sizes used on the world’s largest video site. I think you’ll agree with Greg’s conclusions at the bottom of the document, especially with audio where Vorbis really shines.

Greg’s post follows.

Purpose

On Jun 13th 2009 Chris DiBona of Google made a remarkable claim on the WhatWG mailing list:

“If [youtube] were to switch to theora and maintain even a semblance of the current youtube quality it would take up most available bandwidth across the Internet.”

Unfortunately, open video formats have been subjected to FUD so frequently that people are willing to believe bold claims like these without demanding substantiation.

In this comparison I will demonstrate that this claim was unfair and unreasonable. Using a simple test case I show that Theora is competitive and even superior to some of the files that Google is distributing today on YouTube.

Theora isn’t the most efficient video codec available right now. But it is by no means bad, and it is substantially better than many other widely used options. By conventional criteria Theora is competitive. It also has the substantial advantage of being unencumbered, reasonable in computational complexity, and entirely open source. People are often confused by the correct observation that Theora doesn’t provide the state of the art in bitrate vs quality, and take that to mean that Theora does poorly when in reality it does quite well. Also, the Theora encoder has improved a lot lately so some older problems no longer apply.

While different files may produce different results, the allegation made on WhatWG was so expansive that I believe a simple comparison can reliably demonstrate its falsehood.

I do not believe Chris intended to deceive anyone, only that he is a victim of the same outdated and/or simply inaccurate information that has fooled many others. Automotive enthusiasts may make a big deal about a 5 horsepower difference between two cars, but these kinds of raw performance differences are not relevant to most car buyers nor are they even the most important criteria to people who race. Likewise, videophiles nitpick the quality of compression formats and this nitpicking is important for the advancement of the art. But I believe that people are mistaking these kinds of small differences for something which is relevant to their own codec selection.

Results

A 499kbit/sec H.264+AAC output and a 327kbit/sec H.263(Sorensen Spark)+MP3 output were available via the download service. The YouTube-encoded files are available on the YouTube site. Because the files on YouTube may change and the web player does not disclose the underlying bitrate, I have made the two encoded files available.

~499kbit/sec comparison

YouTube

Download (H.264+AAC; 17MiB)

Ogg/Theora+Vorbis

Download / Watch (Ogg/Theora+Vorbis; 17MiB)

~327kbit/sec comparison

YouTube

Download (H.263+MP3; 12MiB)

Ogg/Theora+Vorbis

Download / Watch (Ogg/Theora+Vorbis; 12MiB)

A slightly lower bitrate was used for the Theora+Vorbis test cases to avoid any question of quality improvement resulting from larger outputs.
For a fair comparison you must compare the audio as well. Even without audio differences, still image comparisons are a poor proxy for video quality.
I provided this random frame still image comparison only because I expect that people will not bother watching the examples without evidence that the results are interesting.

Methodology

In order to avoid any possible bias in the selection of H.264 encoders and encoding options, and to maximize the relevance for this particular issue, I’ve used YouTube itself as the H.264 encoder. This is less than ideal because YouTube does not accept lossless input, but it does accept arbitrarily high bitrate inputs.

I utilized the Blender Foundation’s Big Buck Bunny as my test case because of its clear licensing status, because it’s a real world test case, and because I have it available in a lossless format. I am not aware of any reason why this particular clip would favor either Theora or H.264.

I chose to use a test case with a soundtrack because most real usage has sound. No one implements HTML5 video without audio, and no one is implementing either of Theora or Vorbis without the other. Vorbis’s state-of-the-art performance is a contributor to the overall Ogg/Theora+Vorbis solution.

• Obtain the lossless 640×360 Big Buck Bunny source PNGs and FLACs from media.xiph.org.
• Resample the images to 480×270 using ImageMagick’s convert utility.
• Use gstreamer’s jpegenc, produce a quality=100 mjpeg + PCM audio stream. The result is around 1.5Gbytes with a bitrate of around 20Mbit/sec.
• Truncate the file to fit under the YouTube 1Gbyte limit, resulting in input_mjpeg.avi (706MiB).
• Upload the file to YouTube and wait for it to transcode.
• Download the FLV and H.264 files produced by YouTube using one of the many web downloading services. (I used keepvid)
• Using libtheora 1.1a2 and Vorbis aoTuv 5.7 produce a file of comparable bitrate to the youtube 499kbit/sec from the same file uploaded to YouTube (input_mjpeg.avi).
• Resample the file uploaded to YouTube to 400×226.
• Using libtheora 1.1a2 and Vorbis aoTuv 5.7 produce a file of comparable bitrate to the youtube 327kbit/sec from the 400×226 downsampled copy of input_mjpeg.avi.

I later discovered that YouTube sometimes offers additional sizes. I tried the youtube-dl utility and it indicated that these other sizes were not available for my file. Otherwise I would have also included them in this comparison.

A keyframe interval of 250 frames was used for the Theora encoding. The theora 1.1a2 encoder software used is available from theora.org. The Vorbis encoder used is available from the aoTuV website. No software modifications were performed.

My conclusions

It can be difficult to compare video at low bitrates, and even YouTube’s higher bitrate option is not high enough to achieve good quality. The primary challenge is that all files at these rates will have problems, so the reviewer is often forced to decide which of two entirely distinct flaws is worse. Sometimes people come to different conclusions.

That said, I believe that the Theora+Vorbis results are substantially better than the YouTube 327kbit/sec. Several other people have expressed the same view to me, and I expect you’ll also reach the same conclusion. This is unsurprising since we’ve been telling people that Theora is better than H.263, especially at lower bitrates, for some time now and YouTube only uses a subset of H.263.

The low bitrate case is also helped by Vorbis’ considerable superiority over MP3. For example, the crickets at the beginning are inaudible in the low rate YouTube clip but sound fine in the Ogg/Theora+Vorbis version.

In the case of the 499kbit/sec H.264 I believe that under careful comparison many people would prefer the H.264 video. However, the difference is not especially great. I expect that most casual users would be unlikely to express a preference or complain about quality if one was substituted for another and I’ve had several people perform a casual comparison of the files and express indifference. Since Theora+Vorbis is providing such comparable results, I think I can confidently state that reports of the internet’s impending demise are greatly exaggerated.

Of course, YouTube may be using an inferior processing chain, or encoding options which trade off quality for some other desirable characteristic (like better seeking granularity, encoding speed, or a specific rate control pattern). But even if they are, we can conclude that adopting an an open unencumbered format in addition to or instead of their current offerings would not cause problems on the basis of quality or bitrate.

But please— see and hear for yourself.