Firefox has a few features up its sleeve when it comes to HTML5 video playback you might not be aware of:

• Firefox‘s seeking is now accurate to millisecondsmicroseconds, there is visual feedback when the video has stalled and clicking the whole video pauses and plays it
• Firefox Beta has specialised controls when you watch video on small devices and watching HTML5 video shows a pleasing background rather than a brutal grey
• Firefox Aurora has fullscreen, statistics overlay, saving of snapshots and controls appearing when the video ended
• Firefox Nightly has a full-screen button, fading video controls after 2 seconds of non-interaction, no loading throbber on audio, error reporting when a video could not be loaded on the video, loop attribute support, and resizing of videos larger than the browser window when you watch them directly

Planned features for Firefox are an overlay play button like YouTube when the video is not set to autoplay and turning off screensavers during fullscreen playback.

Check out Jared’s post for more information.

We had 15 awesome demos submitted and while there was a lot of entertainment and innovation happening, we had to narrow them down to 5 finalists:

Facial Recognition and Analytics with HTML5′s Video Tag
HTML5 Video Voting
HTML5 VJing Tool
jQuery VideoBG
Punch the TV!

There are so many cool things you can do with HTML5 video on the Web, and our winners circle definitely reflects the possibilities. So join us in congratulating our July Dev Derby winners!

1st Place: HTML5 Video Voting by Jordan Humphreys
2nd Place: HTML5 VJing Tool by spite
3rd Place: Facial Recognition and Analytics with HTML5′s Video Tag by dsg (@dsg: Please update your profile with an email address so we can contact you! Thanks.)

Runners-up:
jQuery VideoBG
Punch the TV!

Thanks to everyone that submitted their HTML5 video demos for the July Dev Derby. On to the next race!

Right now, there are only a few days left to submit your History API demo for August. And coming up next we have Geolocation for September and CSS Media Queries for October. So start experimenting and hack on.

The event was one-track with 8 speakers, and personally I always prefer when it’s just one track for follow-up discussions and that everyone have seen and heard the same thing. The topics were ranging broadly between script loaders and HTML5 in one end, and voice-controlled telephony applications in the other.

My presentation

My talk was about HTML5 APIs in general, to give an introduction to them but also to inspire people to try things out and also give feedback to both working groups and web browser vendors about current implementations.

Slides can also be downloaded at SlideShare.

Additionally to the APIs covered in my London Ajax Mobile Event presentation, I went through Web Sockets, File API, HTML5 video, canvas and WebGL. Also, if you are more interested in the <canvas> element, my colleague Rob Hawkes recently released the Foundation HTML5 Canvas book.

What I especially liked talking about is services like vid.ly which helps you take control over the problem of different video codec support in different web browsers, by storing various formats and then deliver the most suitable one depending on the web browser/device accessing it.

Another favorite is Universal Subtitles, which is an excellent tool for everyone to be able to add subtitles to a video clip, empowering users with varying language skills to take part of a video and its content and sharing it with the world.

An option to make the content of a web site richer, there is Popcorn.js to sync key events in the video playing to what kind of text or other information you want to present to go with that. To complement that, the Butterapp is an editor to create that kind of content syncing, currently in alpha.

I also mentioned videograbber for taking easy video screenshots in the web browser.

Dev Derby <video> challenge

I also want to take the opportunity to remind you that Mozilla Dev Derby has a challenge for what you can accomplish with the <video> that goes till the end of July, so please submit anything if you have a good idea!

Moving pictures have always fascinated people. From the first zoetropes to the multi million dollar blockbusters of today – seeing things move grabs our attention as humans much better than any clever copy or imagery could. Video was alien to the web for a long time. Well, not alien, but we needed plugins to make them play. With newer browsers and HTML5, we now have the <video> element to natively embed video in a web page. This allows us to, amongst other things:

• Style video elements with CSS and SVG
• Make the video interact with other elements of the document and vice versa
• Use Canvas to grab frames from videos and manipulate them
• Create accessible video as the controls can be used with a keyboard without us having to write that functionality

Using this arsenal, you can do pretty cool things. You can:

• write a player that rotates and zooms video (source on GitHub)
• Build your own Indiana Jones style maps (GitHub)
• Put your colleagues in a washing machine
• Simulate a green screen
• Apply CSS3 and SVG filters and transitions to video
• Inject other content into videos
• Stretch a video over the whole screen
• Use video in a 3D environment
• Allow users to take screengrabs of videos (source on GitHub)

..and many more things.

Now it is your turn to show us what you can do with HTML5 video! We don’t expect you to build another Wilderness Downtown or 3 Dreams of Black, but we are very excited to offer you the chance to shine in July’s Mozilla Dev Derby.

Why not use CSS3 and Canvas to redesign the Outer Limits intro? Why not make the video do the things Diana Ross talks about in Upside down? Why not build a video cropping tool in Canvas? How about you have a crack at making closed captioning and subtitling work with HTML5 video? There are many ideas floating about – it just needs a bit of drive and a text editor to go for it. So, show us what you can do!

There are a lot of tools out there to make your life easier with HTML5 video. Vid.ly makes it easy to convert videos, Popcorn.js is a library to sync page content and video and there are many tutorials on video available.

To get you into the mood and see some examples in action, we are running an HTML5 video webinar on the 14th of July and you can ask some questions you have upfront using Google Moderator.

Bring the web screen alive, I dare you! I am off to watch Cinema Paradiso again.

Update 2011-07-20: The video recording of this webinar is now available:

On Thursday, July 14th, at 16:00 UTC, Chris Heilmann will give second in the (so far sporadic) Mozilla Developer Engagement webinar series, discussing syncing HTML5 video with Google maps, using his “Spirit of Indiana Jones” demo as an example. Chris has already blogged about this demo, so this webinar is a chance for you to take part in a conversation, rather than just receive a broadcast.

Take a look at the demo code and the blog post, and then bring your questions to the webinar session. If you have a Google account, you can post questions (or respond to others’) ahead of time in this Google Moderator series; if not, ask your questions in IRC during the session.

The webinar will be broadcast on Air Mozilla, with text chat on #airmozilla on irc.mozilla.org. (There’s an IRC widget on the Air Mozilla page if you need it.)

Add this webinar to your calendar:

Oh, by the way, HTML5 video is the theme of the Mozilla Dev Derby for July. Whether you’re new to working with HTML5 video or an “old pro” (if there is such a thing), submit a demo during July to show off your stuff and win cool prizes.

The Mozilla Developer Engagement team is experimenting with a slightly different approach to presenting webinars. We’re decoupling the presentation part from the Q&A part. Instead of a real-time presentation with a little bit of Q&A, we’re publishing a screencast, and scheduling time for online chat about it later.

You can view the screencast on your own time, as much as you need to, and peruse and play with the code under discussion. Then, for those who have questions, the presenter will be on IRC during scheduled times to discuss them. You’ll also be able to chat with other participants, which usually isn’t possible in a typical webinar.

Here is the first in an intended series of pseudo-webinars: Christian Heilmann talking about syncing page content with HTML5 video.

The screencast

The chats

Discussion of this screencast will take place in the #mdnlearning channel on irc.mozilla.org at the following times:

• 10:00 UTC, Monday, 21 March
• 19:00 UTC, Monday, 21 March

If you don’t have an IRC client installed (such as the Chatzilla add-on for Firefox), you can access the #mdnlearning channel via this Mibbit widget.

We picked the times for the chat sessions to cover as many time zones as possible. We know they won’t be ideal for everybody. Please let us know in the comments how much lead time you’d like between publishing screencasts and the scheduled chats.

The code

The code that Chris demos in the screencast is available at https://github.com/codepo8/Syncing-Video.

Seeing that the audience consisted almost exclusively of students and small agencies I thought I write a talk about HTML5 explaining that is not only a plaything of large companies and “web development rockstars” but that everybody can be part of the cause and that we really need people’s input and real life implementations to make the new open tech revolution a reality.

Here are the video recording and slides of my talk. You can also find the detailed notes of the slides here.

Video

You can see “Multimedia on the web” on any HTML5 enabled device here (courtesy of vid.ly) or embedded here:

Slides

The slides are available on Slideshare:

More videos of Heart and Sole

You can see the other videos of Heart and Sole on Vimeo including a great Introduction to Canvas by Rob Hawkes, musings on Designing WordPress by John O’Nolan and Making Ajax userfriendly with the History API by Syd Lawrence.

There will be another Heart and Sole, so keep your eyes open.

One of the biggest annoyance of using HTML5 video is to convert your movie to various formats supported by different browsers. Vid.ly does this job for you: it converts the video you upload into 14 different formats for different browsers and, more importantly, devices. The interface is very intuitive and you can stand by and watch the conversion happen or simply let it run whilst you go offline or do other things. Once the service finished creating the different formats for you, you get an email. You then can visit the vid.ly URL on different devices and you get the video in the correct format.

Interview with Jeff Malkin about vid.ly

To learn more about the service, I got Jeff Malkin, president of Encoding.com on the phone and asked him a few things about the service. You can listen to the interview:

In the interview Jeff answered a few questions and explained a few of my concerns:

• Vid.ly uses the same cloud backend as encoding.com, consisting of Amazon’s EC2 and S3 services and fallbacks
• The conversion happens with ffmpeg and a few other, commercial products as vid.ly also creates a high quality FLV for you
• The videos are tested on the real devices and they are also using services that give them access to more exotic physical devices on demand
• In addition to simply converting the videos to MP4 and Ogg (which can be done using Archive.org), vid.ly also creates iOS versions that support adaptive streaming and webm versions.
• There is no API to make part of a build process in the free version but they are working on a pro version that will allow you to batch process videos and have more granular access to the encoding parameters
• Right now, you can’t send the converted videos back to a hosting platform of your own choice but this is also in the making.

First impressions

When trying out the service with a video, I was very impressed with the interface. After entering your code you get a video URL. Pressing start asks you for an email (to notify you once the conversion is done) and offers you an interface to upload your video, either from FTP, HTTP, S3 or your hard drive. A log on the page tells you at any time what is happening:

The whole page is a progress bar. Once you uploaded the video the progress starts from left to right and tells you in the log box what is going on. You can stand by or close the page – vid.ly emails you once the conversion is done.

Under the hood

Once the conversion is done, you get the email with your URL and you can try it on different devices. I checked it on my Nexus One and my iPod and got working videos on each without any redirect, ad display or other annoyances.

The embed code vid.ly sends you can be improved a bit but was built for ease of use:

<video id="vidly-video" controls="controls" 
width="640" height="390">
<source
src="http://vid.ly/{ID}?content=video" />
<script id="vidjs" language="javascript" 
src="http://m.vid.ly/js/html5.js">
</script>
</video>

Instead of linking the different sources, vid.ly adds a JavaScript that re-writes the source accordingly. If you check the script you see that it includes the VideoJS player by Zencoder and that there is a lot of detection going on – both using the HTML5 Video API and browser sniffing for Quicktime support or to fall back to Flash.

This seems a bit overkill and could be improved. The other annoyance are IDs on the video and script as that prevents you from using several vid.ly videos in the same document. But then again, the power of open tech is that you can work around that:

The email is that it tells you that there is a high quality FLV file created for you at http://vidly.s3.amazonaws.com/{ID}/flv.flv. This means that your videos are stored in Amazon’s S3 with a bucket with the ID of your video. If you rename the flv.flv to ogv.ogv, webm.webm and mp4/mp4 accordingly you use these URLs to create your own video embed without JavaScript or you can download the different versions.

Try it out yourself

You can try vid.ly yourself using the HNY2011 code. What do you think?

There are dozens of video players that allow you to do all the normal things with videos: play, pause, jump to a certain time and so on. More advanced ones also allow you to fast forward and reverse the video and support subtitles.

One thing I haven’t found yet though is zooming and rotation of a video. Granted, an edge use case, but sometimes it is cool to be able to zoom into a detail like a blooper (boom mic in the background) or an easter egg (check for the term “A 113″ in every Pixar movie – an homage to the classroom most of the original Pixar cast learned their trade in).

Rotation might come in handy when you recorded your video in portrait instead of landscape on your camera and you don’t want to re-encode it before you put it on the web.

The HTML5 video tag allows you to style the video with CSS and CSS3 transforms allow both for scaling and for rotation. So let’s put those together.

Embedding a video is as simple as this (see this in action on the demo page):

<video controls>
  <source src="http://www.archive.org/download/AnimatedMechanicalArtPiecesAtMit/P1120973_512kb.mp4" type="video/mp4">         
  <source src="http://www.archive.org/download/AnimatedMechanicalArtPiecesAtMit/P1120973.ogv" type="video/ogg">
  <p>Your browser doesn't support the HTML5 video tag it seems. 
     You can see this video as part of a collection <a href="http://www.archive.org/download/AnimatedMechanicalArtPiecesAtMit/">at archive.org</a>.</p>         
</video>

Now, to scale this video you can use CSS3 transform:scale:

video{
  display:block;
  width:400px;
  height:300px;
}
video.scale2{
  -moz-transform:scale(1.5);
  -webkit-transform:scale(1.5);
  -o-transform:scale(1.5);
  -ms-transform:scale(1.5);
  transform:scale(1.5);
}

You can rotate with CSS3 transform:rotate:

video{
  -moz-transform:rotate(20deg);
  -webkit-transform:rotate(20deg);
  -o-transform:rotate(20deg);
  -ms-transform:rotate(20deg);
  transform:rotate(20deg);
}

And you can do both:

video{
  -moz-transform:scale(1.5) rotate(20deg);
  -webkit-transform:scale(1.5) rotate(20deg);
  -o-transform:scale(1.5) rotate(20deg);
  -ms-transform:scale(1.5) rotate(20deg);
  transform:scale(1.5) rotate(20deg);
}

Now, this doesn’t make much sense though as it changes the dimension of the video (in the demo page I needed to change the margin for each video accordingly). To really provide a “zoom” functionality, we’d need to keep the original size and cut off the parts we don’t need. We could do this using a CANVAS element, but why go that far when a simple DIV does the job for us?

All we need to do to keep the space is nest our videos in a DIV with the class stage:

<div class="stage"><video controls>
  <source src="http://www.archive.org/download/AnimatedMechanicalArtPiecesAtMit/P1120973_512kb.mp4" type="video/mp4">         
  <source src="http://www.archive.org/download/AnimatedMechanicalArtPiecesAtMit/P1120973.ogv" type="video/ogg">         
  <p>Your browser doesn't support the HTML5 video tag it seems. 
     You can see this video as part of a collection <a href="http://www.archive.org/download/AnimatedMechanicalArtPiecesAtMit/">at archive.org</a>.</p>         
</video></div>

The CSS to make the cropping work is the following:

.stage{
  width:400px;
  height:300px;
  position:relative;
}
video{
  width:400px;
  height:300px;
  position:absolute;
  top:0;
  left:0;
}

Absolutely positioned elements are taken out of the normal flow of the document. If you put an absolutely positioned element inside one that is relatively positioned, its top and left values are relative to the other element. In other words: the video now covers the stage completely.

If we rotate the video we still have triangular spaces showing like in this example:

.demostage{
  width:400px;
  height:300px;
  position:relative;
}
.demovideo{
  position:absolute;
  top:0;
  left:0;
  -moz-transform:rotate(20deg);
  -webkit-transform:rotate(20deg);
  -o-transform:rotate(20deg);
  -ms-transform:rotate(20deg);
  transform:rotate(20deg);
  width:400px;
  height:300px;
}

To work around this, we must add a overflow:hidden to the stage DIV:

.demostage{
  width:400px;
  height:300px;
  position:relative;
  overflow:hidden;
}
.demovideo{
  position:absolute;
  top:0;
  left:0;
  -moz-transform:rotate(20deg);
  -webkit-transform:rotate(20deg);
  -o-transform:rotate(20deg);
  -ms-transform:rotate(20deg);
  transform:rotate(20deg);
  width:400px;
  height:300px;
}

In order to provide a zoom and rotate functionality, we need to use JavaScript and buttons for the end user. The first hurdle here is – as you probably already realised from the CSS – browser differences in the syntax. Therefore we need to detect which of the transformations the current browser supports. The safest way is to ask the browser:

var properties = ['transform', 'WebkitTransform', 'MozTransform',
                  'msTransform', 'OTransform'];
var prop = properties[0];
for(var i=0,j=properties.length;i<j;i++){
  if(typeof v.style[properties[i]] !== 'undefined'){
    prop = properties[i];
    break;
  }
}

Once this runs we can set a transformation with the following JavaScript syntax:

var zoom = 1.5;
var rotate = 20;
v.style[prop]='rotate('+rotate+'deg) scale('+zoom+')';

Of course it doesn’t make much sense to rotate the controls with the video. Therefore you need to provide your own. You can use any of the aforementioned players for that or roll your own. To demonstrate, I just built one with a single button allowing you to play and pause the video:

The full source is available on GitHub, read the comments to see what is going on here.

Just a quick example of what you can do with open technologies.

During the week we covered topics like WebGL for 3D development, basics of JavaScript, debugging and performance, canvas development, offline development and local storage and multimedia on the web. We’ll make these slides available in the comments to this blog post.

The slides of the multimedia talk are available here:

The topics covered were:

• Quick history of Multimedia on the web
• Annoyances with Flash
• HTML5 audio and video
• Codecs and conversion
• Embedding video and audio
• Controlling video
• Transforming video
• Making realtime changes
• Awesome new audio possibilities

We showed and explained a few demos like:

• The Media events demo page at the W3C web site
• The Spirit of Indiana demo syncing a Google Maps animation with a video
• Transforming Video to create a player that allows you to zoom inside a video and rotate it using CSS transformations
• Paul Rouget’s round video demo showing how to create a play/pause button in SVG, greyscale a video and rotate it using CSS transformations
• Paul’s Video Mashup demo showing how to apply SVG filters like Gaussian Blur, masking and how to apply CSS transforms to a running video
• Remy Sharp’s round video demo showing how to create a rotating, bouncing circle that contains a video
• Paul Rouget’s Green demo showing how to simulate a green screen effect with open web technologies.
• The dynamic content injection demo showing how to embed content into a video after analysing it for injection hotspots.
• The Tracker demo showing how to detect human shapes and movement in a video
• The edge detection demo shows how to detect edges in a live video using JavaScript
• The HTML5 Guitar Tab generator
• The sfxr sound effect library

I also provided extensive notes for the whole talk with code examples on my own blog.

Leggi la traduzione in italiano

Today we are featuring Remy Sharp co-author of Introducing HTML5 and organiser of the Full Frontal conference in Brighton, England.

Remy is one of those ubiquitous people of HTML5. Whenever something needed fixing, there is probably something on GitHub that Remy wrote that helps you. He is also very English and doesn’t mince his words much.

You can find Remy on Twitter as @rem.

The video interview

Watch the video on YouTube or Download it from Archive.org as MP4 (98 MB), OGG (70 MB) or WebM (68MB)

Ten questions about HTML5 for Remy Sharp

1) Reading “Introducing HTML5″ it seems to me that you were more of the API – focused person and Bruce the markup guy. Is that a fair assumption? What is your background and passion?

That’s spot on. Bruce asked me to join the project as the “JavaScript guy” – which is the slogan I wear under my clothes and frequently reveal in a superman ‘spinning around’ fashion (often at the surprise of clients).

My background has always been coding – even from a young age, my dad had me copying out listings from old spectrum magazines only to result in hours of typing and some random error that I could never debug.

As I got older I graduated to coding in C but those were the days the SDKs were 10Mb downloaded over a 14kb modem, and compile in to some really odd environment. Suffice to say I didn’t get very far.

Then along came JavaScript. A programming language that didn’t require any special development environment. I could write the code in Notepad on my dodgy Window 95 box, and every machine came with the runtime: the browser. Score!

From that point on the idea of instant gratification from the browser meant that I was converted – JavaScript was the way for me.

Since then I’ve worked on backend environments too (yep, I’m a Perl guy, sorry!), but always worked and played in the front end in some way or another. However, since started on my own in 2006, it’s allowed me to move focus almost entirely on the front end, and specialise in JavaScript. Basically, work-wise: I’m a pig in shit [Ed: for our non-native English readers, he means "happy")].

2) From a programmer’s point of view, what are the most exciting bits about the HTML5 standard? What would you say is something every aspiring developer should get their head around first?

For me, the most exciting aspects of HTML5 is the depth of the JavaScript APIs. It’s pretty tricky to explain to Joe Bloggs that actually this newly spec’ed version of HTML isn’t mostly HTML; it’s mostly JavaScript.

I couldn’t put my finger on one single part of the spec, only because it’s like saying which is your favourite part of CSS (the :target selector – okay, so I can, but that’s not the point!). What’s most exciting to me is that HTML5 is saying that the browser is the platform that we can deliver real applications – take this technology seriously.

If an aspiring developer wanted something quick and impressive, I’d say play around with the video API – by no means is this the best API, just an easy one.

If they really wanted to blow people away with something amazing using HTML5, I’d say learn JavaScript (I’m assuming they’re already happy with HTML and CSS). Get a book like JavaScript: The Good Parts and then get JavaScript Patterns and master the language. Maybe, just maybe, then go buy Introducing HTML5, it’s written by two /really/ good looking (naked) guys: http://flic.kr/p/8iyQTE and http://flic.kr/p/8iy6Z1 [Ed: maybe NSFW, definitely disturbing].

3) In your book you wrote a nice step-by-step video player for HTML5 video. What do you think works well with the Video APIs and what are still problems that need solving?

The media API is dirt simple, so it means working with video and audio is a doddle. For me, most of it works really well (so long as you understand the loading process and the events).

Otherwise what’s really quite neat, is the fact I can capture the video frames and mess with them in a canvas element – there’s lots of fun that can be had there (see some of Paul Rouget’s demos for that!).

What sucks, and sucks hard, is the spec asks vendors, ie. browser makers, *not* to implement full screen mode. It uses security concerns as the reason (which I can understand), but Flash solved this long ago – so why not follow their lead on this particular problem? If native video won’t go full screen, it will never be a competitive alternative to Flash for video.

That all said, I do like that the folks behind WebKit went and ignored the spec, and implemented full screen. The specs are just guidelines, and personally, I think browsers should be adding this feature.

4) Let’s talk a bit about non-HTML5 standards, like Geolocation. I understand you did some work with that and found that some parts of the spec work well whilst others less so. Can you give us some insight?

On top of HTML5 specification there’s a bunch more specs that make the browser really, really exciting. If we focus on the browser being released today (IE9 included) there’s a massive amount that can be done that we couldn’t do 10 years ago.

There’s the “non-HTML5″ specs that actually were part of HTML5, but split out for good reason (so they can be better managed), like web storage, 2D canvas API and Web Sockets, but there’s also the /really/ “nothing-to-do-with-HTML5″ APIs (NTDWH5API!) like querySelector, XHR2 and the Device APIs. I’m super keen to try all of these out even if they’re not fully there in all the browsers.

Geolocation is a great example of cherry picking technology. Playing against the idea that the technology isn’t fully implemented. Something I find myself ranting on and on about when it comes to the question of whether a developer should use HTML5. Only 50% of Geolocation is implemented in the browsers supporting it, in that they don’t have altitude, heading or speed – all of which are part of the spec. Does that stop mainstream apps like Google Maps from using the API? (clue: no).

The guys writting the specs have done a pretty amazing job, and in particular there are few cases where the specs have been retrospectively written. XHR is one of these and now we’ve got a stable API being added in new browsers (i.e. IE6 sucks, yes, we all know that). Which leads us to drag and drop. The black sheep of the APIs. In theory a really powerful API that could make our applications rip, but the technical implementation is a shambles. PPK (Peter-Paul Koch) tore the spec a bit of a ‘new one’. It’s usable, but it’s confusing, and lacking.

Generally, I’ve found the “non-HTML5″ specs to be a bit of mixed bag. Some are well supported in new browsers, some not at all. SVG is an oldie and now really viable with the help of JavaScript libraries such as Raphaël.js or SVGWeb (a Flash based solution). All in all, there’s lots of options available in JavaScript API nowadays compared to back in the dark ages.

5) Let’s talk Canvas vs. SVG for a bit. Isn’t Canvas just having another pixel-based rectangle in the page much like Java Applets used to be? SVG, on the other hand is Vector based and thus would be a more natural tool to do something with open standards that we do in Flash now. When would you pick SVG instead of Canvas and vice versa?

Canvas, in a lot of ways is just like the Flash drawing APIs. It’s not accessible and a total black box. The thing is, in the West, there’s a lot of businesses, rightly or wrongly, that want their fancy animation thingy to work on iOS. Since Flash doesn’t work there, canvas is a superb solution.

However, you must, MUST, decide which technology to use. Don’t just use canvas because you saw a Mario Kart demo using it. Look at the pros and cons of each. SVG and the canvas API are not competitive technologies, they’re specially shaped hammers for specific jobs.

Brad Neuberg did a superb job of summarising the pros and cons of each, and I’m constantly referring people to it (here’s the video).

So it really boils down to:

• Canvas: pixel manipulation, non-interactive and high animation
• SVG: interactive, vector based

Choose wisely young padawan!

6) What about performance? Aren’t large Canvas solutions very slow, especially on mobile devices? Isn’t that a problem for gaming? What can be done to work around that?

Well…yes and no. I’m finishing a project that has a large canvas animation going on, and it’s not slow on mobile devices (not that it was designed for those). The reason it’s not slow is because of the way the canvas animates. It doesn’t need to be constantly updating at 60fps.

Performance depends on your application. Evaluate the environment, the technologies and make a good decision. I personally don’t think using a canvas for a really high performance game on a mobile is quite ready. I don’t think the devices have the ommph to get the job done – but there’s a hidden problem – the browser in the device isn’t quite up to it. Hardware acceleration is going to help, a lot, but today, right now, I don’t think we’ll see games like Angry Birds written in JavaScript.

That said… I’ve seriously considered how you could replicate a game like Canabalt using a mix of canvas, DIVs and CSS. I think it might be doable ::throws gauntlet::

I think our community could actually learn a lot from the Flash community. They’ve been through all of this already. Trying to make old versions of Flash from years back do things that were pushing the boundaries. People like Seb Lee-Delisle (@seb_ly / http://seb.ly) are doing an amazing job of teaching both the Flash and JavaScript community.

7) A feature that used to be HTML5 and is now an own spec is LocalStorage and its derivatives Session Storage or the full-fledged WebSQL and IndexedDB. Another thing is offline storage. There seems to be a huge discussion in developer circles about what to use when and if NoSQL solutions client side are the future or not. What are your thoughts? When can you use what and what are the drawbacks?

Personally I love seeing server-less applications. Looking at the storage solutions I often find it difficult to see why you wouldn’t use WebStorage every time.

In a way it acts like (in my limited experience of) NoSQL, in that you lookup a key and get a result.

Equally, I think SQL in the browser is over the top. Like you’re trying to use the storage method *you* understand and forcing it into the browser. Seems like too much work for too little win.

Offline Apps, API-wise, ie. the application cache is /really/ sexy. Like sexy with chocolate on top sexy. The idea that our applications can run without the web, or switch when it detects it’s offline is really powerful. The only problem is that the events are screwed. The event to say your app is now offline requires the user to intervene via the browser menu, telling the browser to “work in offline mode”. A total failure of experience. What’s worse is that, as far as I know, there’s no plan to make offline event fire properly :-(

That all said, cookies are definitely dead for me. I’ve yet to find a real solution for cookies since I found the Web Storage API – and there’s a good decent number of polyfills for Web Storage – so there’s really no fear of using the API.

8) Changing the track a bit, you’ve built the HTML5shiv to make HTML5elements styleable in IE. This idea sparked quite a lot of other solutions to make IE6 work with the new technologies (or actually simulate them). Where does this end? Do you think it is worth while to write much more code just to have full IE6 support?

There’s two things here:

1. Supporting IE6 (clue: don’t)
2. Polyfills

IE6, seriously, and for the umpteenth time, look at your users. Seriously. I know the project manager is going to say they don’t know what the figures are, in that case: find out! Then, once you’ve got the usage stats in hand, you know your audience and you know what technology they support.

If they’re mostly IE6 users, then adding native video with spinning and dancing canvas effect isn’t going to work – not even with shims and polyfills. IE6 is an old dog that just isn’t up to doing the mileage he used to be able to do back in his prime. But enough on this subject – the old ‘do I, or don’t I developer for IE6′ is long in the tooth.

Polyfills – that’s a different matter. They’re not there to support IE6, they’re there to bring browsers up to your expectations as a developer. However, I’d ask that you carefully consider them before pulling them in. The point of these scripts is they plug missing APIs in those older browsers. “Old browsers” doesn’t particularly mean IE6. For example, the Web Sockets API has a polyfill by way of Flash. If native Web Sockets aren’t there, Flash fills the gap, but the API is exposed in exactly the same manner, meaning that you don’t have to fork your code.

I don’t think people should be pulling in scripts just for the hell of it. You should consider what you’re trying to achieve and decide whether X technology is the right fit. If it is, and you know (or expect) your users have browsers that don’t support X technology – should you plug it with JavaScript or perhaps should you consider a different technology?

This exact same argument rings true for when someone adds jQuery just to add or remove a class from an element. It’s simply not worth it – but clearly that particular developer didn’t really understand what they needed to do. So is education the solution? I should hope so.

9) Where would you send people if they want to learn about HTML5? What are tutorials that taught you a lot? Where should interested people hang out?

HTML5 Doctor – fo sho’. :)

I tend to also direct people to my http://html5demos.com simply to encourage viewing source, and hacking away.

Regarding what tutorials taught me – if I’m totally honest, the place I’ve learnt the most from is actually HTML5Doctor.com. There’s some pretty good JavaScript / API tutorials coming from the chaps at http://bocoup.com. Otherwise, I actually spend a lot of time just snooping through the specifications, looking for bits that I’ve not seen before and generally poking them with a stick.

10) You have announced that you are concentrating on building a framework to make Websockets easy to work with. How is that getting on and what do you see Websockets being used for in the future? In other words, why the fascination?

Concentrating is a strong word ;-) but it is true, I’ve started working on a product that abstracts Web Sockets to a service. Not the API alone, since it’s so damn simple, but the server setup: creating sessions, user control flow, waiting for users and more.

The service is called Förbind. Swedish for “connect”, ie. to connect your users. It’s still early days, but I hope to release alpha access to forbind.net this month.

I used to work in finance web sites and real-time was the golden egg: to get that data as soon as it was published. So now that it’s available in a native form in the browser, I’m all over it!

What’s more, I love the idea of anonymous users. I created a bunch of demos where the user can contribute to something without ever really revealing themselves, and when the users come, you start to see how creative people are without really trying. Sure, you get a lot of cocks being drawn, but you also see some impressive ideas – my business 404 page for example allows people to leave a drawing, one of the most impressive is a Super Mario in all his glory. Anonymous users really interest me because as grey as things can seem sometimes, a stranger can easily inspire you.

Do you know anyone I should interview for “People of HTML5″? Tell me on Twitter: @codepo8

See the demo online

You can download the animation demo for yourself and try it out locally – all you need is a browser that supports HTML5 video. I know – the music is not quite the same as in the movies, but at least this one is not copyright infringing and it came from the heart (5 minutes in a meeting room in the Mozilla office).

So how was this done and what are problems that needed solving? Here’s how and what.

Step 1: Find the movie and get it to the right format

That was the easy part. Archive.org has a lot of awesome public domain movies available for you and they are already in the formats needed to use in an HTML5 video element. In this case, I took the short movie of Charles Lindbergh taking off for his record breaking flight from New York to fly to Paris in 1927.

Step 2: Displaying the video

Using the video is pretty simple:

<div id="stage">
  <video>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff_512kb.mp4" 
type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff.ogv"
 type='video/ogg; codecs="theora, vorbis"'> 
  </video>
</div>

The MP4 format will be used by Webkit based browsers and the Ogg version by Firefox and others. As we want to control the video we omit the controls attribute on the video element – instead we create a button to play the video with JavaScript:

window.addEventListener('load', 
  function() {
    var stage = document.getElementById('stage');
    var v = document.getElementsByTagName('video')[0];
    but = document.createElement('button');
    but.innerHTML = 'Click to see Lindbergh\'s flight';
    stage.appendChild(but);
    but.addEventListener('click',function(e) {
      v.play();
      e.preventDefault();
    },false);
  }, 
false);

As the video is markup we can do whatever we please with it – the power of open technologies. For example as we will do here we can set its opacity in CSS and put in on top of a map.

Step 3: Create the map path animation

Talking of which, let’s get that moving path done. Google Earth has an API to do that, but it needs a special plugin. Google Maps allows you to paint paths on maps (which actually are SVG, another open standard). Put that in a recursive function and you get the desired effect:

In essence, what I did was take the latitude and longitude of the start and end points and calculate as many points in between the two as I need for the duration of the animation. I store the points in an array called pos and then paint a path from the start to the current point and move the map centre to this point on every iteration.

spirit.draw = function(){
  var path = new google.maps.Polyline({
        path: [startpos,pos[now]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
  });
  path.setMap(map);
  map.panTo(pos[now])
  now = now + 1;
  if(now < animationend-1){
    setTimeout(spirit.draw,200);
  }
}

Check the highly commented source of the map example for the details. Now, we could use this kind of animation and play the video over it – the problem though is that they may get out of sync. When the movie stalls (as it frequently does on this hotel wireless connection) we don’t want the animation to keep moving, right?

Step 4: Syncing video and the map movement

Instead of having two sources of timing information we have to limit ourselves to one source of truth. This is the time stamp of the currently playing movie.

By the way – you might have noticed that I wrapped the map code in a tilesloaded event handler. This is another safeguard for keeping things in sync. I found that on slow connections the tile loading can delay the whole interface immensely (because of all the subdomain lookups), so I make the whole interface dependent on the loading of the map and only proceed when the tiles have finished loading. As the tilesloaded event also fires when the map pans we need to use a boolean to stop it from starting the effect several times:

google.maps.event.addListener(map,'tilesloaded',function(){
  if(played === false){
    // [...other code...]
    played = true;
  }
});

You can read the current timestamp of a video with video.currentTime and whilst the movie is playing it constantly fires an event called timeupdate. As the event fires a lot we need to throttle it somehow. The trick here is to only take the full seconds and increase a counter when a new second is reached. You can see the timestamp and the second interval firing in the video syncing demo:

var now = 0;
v.addEventListener('timeupdate',function(o){
  log.innerHTML = v.currentTime; /* logging the real timestamp */
  var full = parseInt(v.currentTime);
  if(full >= now) {
    seqlog.innerHTML = now;  /* logging the seconds firing */
    now = now + 1;
  }
},false);

That way the movie can lag in between and the sequence still stays in sync. Check the source of this demo on Github.

Putting it all together

And that was about it – all I had to do is to set the movie’s opacity at a certain time stamp, start the sound at another and show and hide the copyright complaint at another. As we rely on the timestamp for the other effects we needed a boolean switch to avoid repeated firing:

v.addEventListener('timeupdate',function(o){
  full = parseInt(v.currentTime);
  if(full === now-1){
    mapelm.style.opacity = .8;
    v.style.opacity = .4;
  }
  if(full === animationstart+1 && audioplay === false){
    a.play();
    audioplay = true;
  }
  if(full === animationstart+2 && hidden === true){
    drmbedamned.style.display = 'block';
    hidden = false;
  }
  if(full === animationstart+3 && hidden === false){
    drmbedamned.style.display = 'none';
    hidden = true;
  }
  if(full >= now) {
    path = new google.maps.Polyline({
        path: [startpos,pos[full]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
    });
    path.setMap(map);
    map.panTo(pos[full])
    now = now + 1;
  }
},false);

Another event we needed to subscribe to was the movie ending so we can stop the music and start to roll the credits:

v.addEventListener('ended',function(o){
  a.pause();
  spirit.credslist.parentNode.style.display = 'block';
  spirit.creds();
},false)

As the theme is too short for the whole animation we need to loop it. This can be done by testing for the ended event and rolling back the time to 0:

a.addEventListener('ended', function(o) {
  a.currentTime = 0;
},false);

Summary

And there you have it – Indiana Jones style maps using open services and open technologies. A workaround for the copyrighted audio (recorded, edited and converted with the free Audacity sound editor) and using Google’s Web Fonts as graphics.

You can now take this and change it for even more awesome:

• Replace Google Maps with Openstreetmap to avoid going over the limit
• Add a slight curve to the path from NYC to Paris to make it more accurate (but then again the time is not accurate either – it took charles a tad longer)
• Use a static map and paint the path with Canvas to speed up and smoothen the animation

Why not have a go – it is free and fun to play.

Late last week I landed support on Firefox trunk for the HTML5 video ‘preload’ attribute. This replaces the ‘autobuffer’ attribute, which we previously supported. If you were previously using the autobuffer attribute on your HTLM5 videos, you need to update to using the preload attribute as well.

The preload attribute provides a hint to the browser as to how much downloading is sensible for a given media. The preload attribute can have three values:

1. “none” – suggests to the browser that it doesn’t need to load this media at all until the user plays the resource. The browser will delay any network traffic required to load the media until the users tries to play the resource, or explicitly loads the resource. I suggest using this preload value, along with the poster attribute, when it’s unlikely that the user will play the resource. This is probably most useful in a mobile environment, where data can be expensive.
2. “metadata” – suggests to the browser that it isn’t necessary to load the entire resource in advance. The browser will suspend the load after loading metadata, displaying the first video frame (if there’s no poster image), and ensuring it can play the media. This is the default behaviour, and prevents excess network traffic when the web developer isn’t certain the video will definitely be played. If you don’t specify a preload value, Firefox will automatically do this. This was also the default behaviour in Firefox 3.5 and 3.6 in the absence of the autobuffer attribute. This default behaviour is a suitable compromise between bandwidth saving and user convenience.
3. “auto” – suggests to the browser that it should load as much of the resource as possible. As long as the browser’s own media cache isn’t full, it will keep on downloading. I suggest this is most useful in the “YouTube” case, when you’ve got a media which the user is almost certainly going to watch, and so having the user download the media is not likely to be wasting server bandwidth. This behaviour is the same as using the autobuffer attribute in Firefox 3.5 and 3.6

Since the autobuffer attribute is no longer present in Firefox 4, and the preload attribute is not present in Firefox 3.5 and 3.6, if you want a media to download completely, you should include both preload=”auto” and autobuffer in the video element, e.g.:

<video autobuffer preload=”auto” src=”video.ogg”></video>

If you want the video to only be downloaded if the user actually plays the video, you should not include either the preload or the autobuffer attribute. The default behaviour in Firefox 3.5, 3.6, and the upcoming Firefox 4 is to only load up to the first frame and then suspend the download.

Recently I landed support for the HTML5 video ‘buffered’ property in Firefox. This is cool because we can now accurately determine which time-segments of a video we can play and seek into without needing to pause playback to download more data. Previously you could only get the byte position the download had reached, which often doesn’t map to the time ranges which are playable very well, especially in a variable bit rate video. This also can’t tell you if there are chunks which we skipped downloading before the downloaded byte position. Once the video controls UI is updated, users will be able to know exactly which segments of their video are downloaded and playable and can be seeked into without pausing playback to download more data.

To see this in action, download a current Firefox nightly build , and point your browser at my video ‘buffered’ property demo. You’ll see something like the screenshot below, including an extra progress bar (implemented using canvas) showing the time ranges which are buffered.

I’ve implemented the ‘buffered’ property for the Ogg and WAV backends. Support for the ‘buffered’ property for WebM is being worked on by Matthew Gregan, and is well underway. At the moment we return empty ranges for the ‘buffered’ property on video elements playing WebM and raw video.

My checkin just missed the cutoff for Firefox 4 Beta 3, so the first beta release that the video ‘buffered’ property will appear in is Firefox 4 Beta 4.

Fast JavaScript is a cornerstone of the modern web. In the past, application authors had to wait for browser developers to implement any complex functionality in the browser itself, so that they could access it from script code. Today, many of those functions can move straight into JavaScript itself. This has many advantages for application authors: there’s no need to wait for a new version of a browser before you can develop or ship your app, you can tailor the functionality to exactly what you need, and you can improve it directly (make it faster, higher quality, more precise, etc.).

Here are two examples that show off what can be done with the improved JS engine and capabilities that will be present in Firefox 4. The first example shows a simple web-based Darkroom that allows you to perform color correction on an image. The HTML+JS is around 700 lines of code, not counting jQuery. This is based on a demo that’s included with Google’s Native Client (NaCl) SDK; in that demo, the color correction work is done inside native code going through NaCl. That demo (originally presented as “too slow to run in JavaScript”) is a few thousand lines of code, and involves downloading and installing platform-specific compilers, multiple steps to test/deploy code, and installing a plugin on the browser side.

I get about 15-16 frames per second with the default zoomed out image (around 5 million pixels per second — that number won’t be affected by image size) on my MacBook Pro, which is definitely fast enough for live manipulation. The algorithm could be tightened up to make this faster still. Further optimizations to the JS engine could help here as well; for example, I noticed that we spend a lot of time doing floating point to integer conversions for writing the computed pixels back to the display canvas, due to how the canvas API specifies image data handling.

The Web Darkroom tool also supports drag & drop, so you can take any image from your computer and drop it onto the canvas to load it. A long (long!) time ago, back in 2006, I wrote an addon called “Croppr!”. It was intended to be used with Flickr, allowing users to play around with custom crops of any image, and then leave crop suggestions in comments to be viewed using Croppr. It almost certainly doesn’t work any more, but it would be neat to update it: this time with both cropping and color correction. Someone with the addon (perhaps a Jetpack now!) could then visit a Flickr photo and experiment, and leave suggestions for the photographer.

The second example is based on some work that Dave Humphrey and others have been doing to bring audio manipulation to the web platform. Originally, their spec included a pre-computed FFT with each audio frame delivered to the web app. I suggested that there’s no need for this — while a FFT is useful for some applications, for others it would be wasted work. Those apps that want a FFT could implement one in JS. Some benchmark numbers backed this up — using the typed arrays originally created for WebGL, computing an FFT in JS was approaching the speed of native code. Again, both could be sped up (perhaps using SSE2 or something like Mono.Simd on the JS side), but it’s fast enough to be useful already.

The demo shows this in action. A numeric benchmark isn’t really all that interesting, so instead I take a video clip, and as it’s playing, I extract a portion of the green channel of each frame and compute its 2D FFT, which is then displayed. The original clip plays at 24 frames per second, so that’s the upper bound of this demo. Using Float32 typed arrays, the computation and playback proceeds at around 22-24fps for me.

You can grab the video controls and scrub to a specific frame. (The frame rate calculation is only correct while the video is playing normally, not while you’re scrubbing.) The video source uses Theora, so you’ll need a browser that can play Theora content. (I didn’t have a similar clip that uses WebM, or I could have used that.)

These examples are demonstrating the strength of the trace-based JIT technique that Firefox has used for accelerating JavaScript since Firefox 3.5. However, not all code can see such dramatic speedups from that type of acceleration. Because of that, we’ll be including a full method-based JIT for Firefox 4 (for more details, see David Anderson’s blog, as well as David Mandelin’s blog). This will provide significantly faster baseline JS performance, with the trace JIT becoming a turbocharger for code that it would naturally apply to.

Combining fast JavaScript performance alongside new web platform technologies such as WebGL and Audio will make for some pretty exciting web apps, and I’m looking forward to seeing what developers do with them!

Edit: Made some last-minute changes to the demos, which ended up pulling in a slightly broken version of jQuery UI that wasn’t all that happy with Safari. Should be fixed now!

1. Google will be releasing VP8 under an open source and royalty-free basis. VP8 is a high-quality video codec that Google acquired when they purchased the company On2. The VP8 codec represents a vast improvement in quality-per-bit over Theora and is comparable in quality to H.264.

2. The VP8 codec will be combined with the Vorbis audio codec and a subset of the Matroska container format to build a new standard for Open Video on the web called WebM. You can find out more about the project at its new site: http://www.webmproject.org/.

3. We will include support for WebM in Firefox. You can get super-early WebM builds of Firefox 4 pre-alpha today. WebM will also be included in Google Chrome and Opera.

4. Every video on YouTube will be transcoded into WebM. They have about 1.2 million videos available today and will be working through their back catalog over time. But they have committed to supporting everything.

5. This is something that is supported by many partners, not just Google and others. Content providers like Brightcove have signed up to support WebM as part of a full HTML5 video solution. Hardware companies, encoding providers and other parts of the video stack are all part of the list of companies backing WebM. Even Adobe will be supporting WebM in Flash. Firefox, with its market share and principled leadership and YouTube, with its video reach are the most important partners in this solution, but we are only a small part of the larger ecosystem of video.

We’re extremely excited to see Google joining us to support Open Video. They are making technology available on terms consistent with the Open Web and the W3C Royalty-Free licensing terms. And – most importantly – they are committing to support a full open video stack on the world’s largest video site. This changes the landscape for video and moves the baseline for what other sites have to do to maintain parity and keep up with upcoming advances in video technology, not to mention compatibility with the set of browsers that are growing their userbase and advancing technology on the web.

At Mozilla, we’ve wanted video on the web to move as fast as the rest of the web. That has required a baseline of open technology to build on. Theora was a good start, but VP8 is better. Expect us to start pushing on video innovation with vigor. We’ll innovate like the web has, moving from the edges in, with dozens of small revolutions that add up to something larger than the sum of those parts. VP8 is one of those pieces, HTML5 is another. If you watch this weblog, you can start to see those other pieces starting to emerge as well. The web is creeping into more and more technologies, with Firefox leading the way. We intend to keep leading the web beyond HTML5 to the next place it needs to be.

Today is a day of great change. Tomorrow will be another.

David’s work is important because it’s showing where the web is going, and where Mozilla is helping to take it. It’s not enough that we’re working on HTML5, which we’re about finished with, but we’re trying to figure out what’s next. Mozilla’s platform, Gecko, is a huge part of why we’re able to experiment and learn as fast as we can. And that’s reflected with what’s possible here. It’s a web you can see, touch and interact with in new ways.

David’s post follows:

I’m working with an ever growing group of web, audio, and Mozilla developers on a project to expose audio spectrum data to JavaScript from Firefox’s audio and video elements. Today we show what we did at www2010.

I’m in Raleigh, North Carolina, with Al MacDonald for the www2010 conference. We’re here to present our work on exposing audio data in the browser. Over the past month Corban, Charles, and a bunch of other friends have been working with us to refine the API and get new types of demos ready. We ended-up with 11 demos, some of which I’ve shown here before. Here are the others.

The first was done by Jacob Seidelin, and shows many cool 2D visualizations of audio using our API. You can see the live version on his site, or check out this video:

The second and third demos where done by Charles Cliffe, and show 3D visualizations using WebGL and his CubicVR engine. These also show off his JavaScript beat detection code. Is JavaScript fast enough to do real-time analysis of audio and synchronized 3D graphics? Yes, yes it is. The live versions are here and here, and here are some videos:

The fourth demo was done by Corban Brook and shows how audio data can be mixed live using script. Here he mutes the main audio, plays it, passes the data through a low pass filter written in JavaScript, then dumps the modified frames into a second audio element to be played. It’s a technique we need to apply more widely, as it holds a lot of potential. Here’s the live version, and here’s a video (check out his updated JavaScript synthesizer, which we also presented):

The fifth and sixth demos were done by Al (with the help of many). When I was last in Boston, for the Processing.js meetup at Bocoup, we met with Doug Schepers from the W3C. He loved our stuff, and was talking to us about ideas that would be cool to build. He pulled out his iPhone and showed us Brian Eno’s Bloom audio app. “It would be cool to do this in the browser.” Yeah, it is cool, and here it is, written in a few hundred lines of JavaScript and Processing.js (video 1, video 2):

This demo also showcases the awesome work of Felipe Gomes, who has a patch to add multi-touch DOM events to Firefox. The method we’ve used here can be taken a lot further. Imagine being able to connect multiple browsers together for collaborative music creation, layering other audio underneath, mixing fragments vs. just oscillators, etc. We built this one in a week, and the web is capable of a lot more.

One of the main points of our talk was to emphasize that what we’re talking about here isn’t just a concept, and it isn’t some far away future. This is real code, running in a real browser, and it’s all being done in HTML5 and JavaScript. The web is fast enough to do real-time audio processing now, powerful enough and expressive enough to create music. And the community of digital music and audio hackers, visualizers, etc. are hungry for it. So hungry that they are seeking us out, downloading our hacked builds and creating gorgeous web audio applications.

We want to keep going, and we need help. We need help from those within Mozilla, the W3C, and other browsers to get this stuff into shipping browsers. We need the audio, digital music, accessibility, and web communities to come together in order to help us build js audio libraries and more sample applications. Yesterday Joe Hewitt was talking on twitter about how web browser vendors need to experiment more with non-standard APIs. I couldn’t agree more, and here’s a chance for people to put their money where their mouth is. Let’s make audio a scriptable part of the open web.

I’m currently creating new builds of our updated patch for Firefox, and will post links to them here when I’m done. You can read more about the technical details of our work here, and get involved in the bug here. You can talk more with me on irc in the processing.js channel (I’m humph on moznet), or talk to me on twitter (@humphd) or by email. One way or another, get in touch so you can help us push this forward.

We realize this post isn’t strictly web developer-facing, but it’s interesting enough for those who want to know how this stuff works under the covers.

Theora on N900: Or, how to play full-screen Theora video on the N900 with 80% idle CPU.

The C64x+ DSP is often found in systems built upon TI’s OMAP3 SoC, such as the Palm Pre, Motorola Droid, and Nokia N900. Last year, Mozilla funded a port, named Leonora, of Xiph’s Theora video codec to the TI C64x+ DSP. David Schleef conducted the port impressively quickly and published his results. The intention of this project was to provide a high quality set of royalty free media codecs for a common mobile computing platform. The initial focus is Firefox Mobile on the N900, so I am working on integrating David’s work into Firefox. To experiment with other facilities Firefox could use to accelerate video playback and test integration, I’ve been hacking on a branch of a stand-alone Ogg Theora and Vorbis player originally written by Chris Double called plogg.

Decoding and playing video can be a CPU intensive process, especially when all of the steps are fighting for time on a single CPU. The expensive parts of the playback process can be broken down into a few coarse pieces, in approximate descending order of cost:

1. Video frame decode
2. Video colour-space conversion (Y’CbCr to RGB)
3. Video frame display
4. Audio block decode
5. Audio block playback

David’s DSP work enables item 1 to be off-loaded from the CPU completely, effectively providing “hardware accelerated” video decoding. Most devices have some way to off-load items 2 and 3 to the graphics hardware, but it can be difficult to make use of this while integrating with an existing graphics rendering pipeline.

The N900 has a 800×480 pixel display, so my hope was to play a 800×480 video full-screen at 30 frames per second with low CPU use and good battery life.

The ARM CPU in the N900 is quite fast. Doing a pure video-decode-only test, the original Theora library, which currently does not have ARM specific optimizations, is able to decode a 640×360 video at 76 frames per second, and it can even decode an 800×480 video at 32 frames per second. With the ARM optimized port by Robin Watts, those numbers become 110 FPS and 47 FPS. David’s DSP port produces 78 FPS and 39 FPS, and it leaves the CPU completely idle because the entire decode is off-loaded to the DSP. With these numbers, it’s clear that the N900 is up to the task of playing back video smoothly if we can get the bits on the screen fast enough.

I am using plogg as a basis for experimentation using techniques applicable in the Firefox rendering engine. This requirement immediately excludes some techniques. For example, using hardware Y’CbCr overlays to display the video frames is excluded because it is not possible for Firefox to render arbitrary HTML content over the top of the overlay.

Chris’s original version of plogg used SDL’s Y’CbCr overlay API, which uses a fast direct overlay path on most systems. This provided a baseline for playback performance. Decoding my 800×480 test video with the DSP, it was possible to play back at 33 FPS with around 20% CPU idle. Unlike the decode-only benchmarks mentioned above, the plogg benchmarks are playing both audio and video with correct A/V synchronisation. With 44.1kHz stereo audio, I observed that 10-15% of the device’s CPU is used by PulseAudio. This indicates that audio playback may constitute a significant amount of processor time with some configurations.

Because there was already work underway to provide OpenGL accelerated compositing in Firefox with the newly conceived Layers API, it seemed logical to try using a GLSL fragment shader to off-load colour-space conversion to the GPU. This turned out to be too slow to play back a full-screen video.

Looking at the list of vendor-specific OpenGL extensions available on the N900, I discovered the texture streaming API. This allows a program to directly map texture memory and copy Y’CbCr data into that memory without having to perform an expensive texture upload or colour-space conversion. The colour-space conversion is off-loaded to dedicated graphics hardware inaccessible via the standard OpenGL APIs. Using this and the modified bc-cat kernel module from the gst-bc-cat project, it’s possible to play back at 26 FPS with 81% CPU idle.

One drawback of the current bc-cat kernel driver is that there is a very limited set of texture formats supported (NV12, UYVY, RGB565, and YUYV), and none of them are the same as what Theora produces. To work around this, a format conversion is required to convert planar Y’CbCr to packed 4:2:2 UYVY. Fortunately, this conversion is much simpler than a full colour-space conversion. Timothy Terriberry sent me a couple of patches to off-load this conversion work to the DSP. If it’s possible to extend the bc-cat driver to support texture formats compatible with Theora’s output, performance can be further improved.

The test files used for benchmarking were: Big Buck Bunny (video: 640×360 @ 500 kbps 24 FPS, audio: 64 kbps 48kHz stereo, 9m 56s, 40MB) and the movie trailer for 9 (video: 800×480 @ 2 Mbps 23.98 FPS, audio: 44.1kHz stereo, 2m 30s, 30MB). Benchmarks were run with the CPU frequency fixed at 600MHz.

In summary, it’s possible to play full-screen Ogg Theora videos on the N900 at full frame rates with low CPU use by off-loading video decoding to the DSP and colour-space conversion and painting to the GPU. There are opportunities for optimization left, tuning for battery life needs to be investigated, and the integration into Firefox still needs to be done.

Video decode-only:

Decoder	FPS (800×480)	Idle CPU
libtheora 1.1	32	0.7%
TheorARM 0.04	47	0.4%
leonora (DSP)	39	99.0%

Playback (video decode + paint, audio decode + not played). DSP decoding video:

Method	FPS (800×480)	Idle CPU
SDL/overlay	33	20.0%
OpenGL/bc-cat	26	81.4%

]]> http://hacks.mozilla.org/2010/04/theora-on-n900/feed/ 4 http://hacks.mozilla.org/2009/12/autobuffering-video-in-firefox/ http://hacks.mozilla.org/2009/12/autobuffering-video-in-firefox/#comments Tue, 22 Dec 2009 00:56:05 +0000 Christopher Blizzard http://hacks.mozilla.org/?p=3103 John Gruber recently wrote up an article titled Why the HTML5 ‘Video’ Element Is Effectively Unusable, Even in the Browsers Which Support It

He’s mostly upset that browsers don’t respect the autobuffer attribute. Or, really, that browsers autobuffer by default. Safari and Chrome do apparently autobuffer by default, but he incorrectly says that Firefox does as well. Just to be clear: Firefox does not autobuffer by default, nor does it autoplay by default. I’m not sure how his testing led him to believe that it does, but I wrote up some examples to show that it does not. Here are three examples:

• Video test without the autobuffer attribute. (Load this first to avoid caching issues!)
• Video test with the autobuffer attibute.
• Video test with one of each.

The best way to test this is to load your favorite browser and clear its cache. Then load your favorite network monitor. Then load the page without the autobuffer attribute set. You can see how much bandwidth the browser is actually using via your external tool.

Of the browsers I tried, only Firefox gives useful feedback on how much is actually buffered as part of its native control set so you have to go to external tools to be able to tell what’s going on with anything else. If you load the page with two videos – one with autobuffer set and one without - you can mouse over them in Firefox and see how one has buffered and one hasn’t.

The video on that test page is pretty big – around 115mb – so you can really tell when it’s buffering even on a fast network.

Here’s what my testing shows:

• Safari for desktops downloads a lot of data. (I don’t have an iPhone so I can’t test that.)
• Chrome downloads a lot of data.
• Firefox loads a small amount of data and then stops.
• Update: According to Philip Jägenstedt, Opera will also do what Firefox does when they include open video support.

This is consistent with what I know about Safari. I talked with one of the Apple WebKit engineers a few months ago and he said that Safari for desktops buffered by default and said that Safari for mobile does not.

There might be some confusion because Firefox does make a couple of network requests when you include a video tag. This is done for two reasons:

• To get the first frame of the video and it’s size. It will only download a small part of the beginning of the video to get the first frame and then stop downloading.
• If your server supports byte range requests, it seeks to the end of the video to try and determine the duration of the video. If your server doesn’t support byterange requests, we don’t get the duration. (Ogg was originally designed as a format for streaming, not static files and as such doesn’t include duration information in the header of the file, although there is work underway to add this functionality.)

If you want to get a sense of how Firefox uses bandwidth, try this bandwidth test I wrote a few months ago. It uses a Firefox-only feature to read progress information on a video as it’s downloaded and give you a rough sense of the amount of bandwidth that particular video element is using. It shows a little traffic when you load the video and then shows how it starts using bandwidth once you press the play button.

But this shows that we understand John’s point of view and we made the decision not to auto buffer by default for the very reasons he points out.

The 1.1 release is largely an improvement to the Theora encoder. This post will attempt to give people a high-level overview of the changes and what they mean to web developers and people who are thinking of deploying Theora to support HTML5 video. Theora is an important technology to web developers – it’s the only competitive codec that currently complies with the W3C patent policy.

Here’s a quick list of important things that have changed in this release. We’ll go into more detail on each of these items.

• Video quality between Theora 1.0 and Theora 1.1 has been improved.
• Rate control for live streaming now works well.
• A two-pass mode has been added to the encoder that can create rate controlled videos with very predictable bandwidth requirements.
• CPU usage during encoding is much more consistent.
• Decoder performance has been improved.

Video quality between Theora 1.0 and Theora 1.1 has been improved.

One issue that people had with the Theora 1.0 encoder was that it produced video that appeared fuzzy. The 1.1 improvements are clear in these two images provided by Monty, one of the Xiph Developers. Open each of these images in new tabs and flip between them. You can really see the difference.

This was also very visible at the edges of text. Here’s an example taken from one of our Firefox 3.5 promotional videos. The first is with the 1.0 encoder (9.0MB) and the second is with the 1.1 encoder (8.2MB). You will notice that not only are the edges more defined but there’s a lot less noise in the area around the edges of the text. Once again, if you open them in tabs and flip between them you can see the difference.

Note that the original video is nearly 17MB. That was done largely to get the text crisp. With these changes we can likely use a much lower-bandwidth version of the video, probably as small as 9.9MB. That’s a pretty big difference.

Note that we’re talking about an improvement of quality at the same video bitrate. This means that we’re either able to produce higher quality videos at the same file size or we’re able to reduce the file size and keep the same quality – either way it’s a big win.

Rate control for live streaming now works well.

Before describing this change, something important must be described. This is the difference between videos encoded with a variable bitrate (VBR) and a constant bitrate (CBR).

In variable bitrate encoding the amount of data that’s required to represent the difference between two frames in a video is allowed to grow. This happens most often when shifting from a scene where there isn’t much movement to a scene where there’s a lot of motion. You could easily go from requiring 40Kb/sec to 400Kb/sec because the entire background moves.

In constant bitrate encoding the amount of data that you’re allowed to use to represent a change from one frame to the next is pinned at some maximum value. If you’ve got a low maximum value and there’s a set of frames that requires a lot of bits to represent the changes from one to the next you will need to sacrifice something in order to stay inside of that maximum value. Very often it’s some amount of video quality or the encoder will start dropping frames in order to keep under the watermark.

This leads to a pretty simple rule: If you want the highest quality video possible, you should be using variable rate encoding. This means that when you’re encoding a video you should be using quality settings (0-99, low/medium/high, 1-10) instead of picking bitrates (60Kb/sec, 200Kb/sec.) For most use cases on the web VBR-encoded videos actually work very well because users are allowed to buffer quite a bit of video out ahead of their current position so these bursts of data don’t affect the user’s experience.

But there are some use cases where having a constant bitrate is very important. These include:

1. Live, low delay streaming over HTTP with a lot of clients.
2. Streaming large files where a large read-ahead buffer is not desired.
3. Situations where large bursts of data result in large bursts in CPU to handle them.

For live, low-delay streaming over HTTP it’s important to realize what happens when there’s a sudden burst of data to handle. HTTP runs over TCP. In TCP it takes a while for a connection to increase its bandwidth. (And by “a while” I mean “not that long” but it’s long enough to affect the low latency connection that we want for this use case. This is why many low-latency applications don’t use TCP. But we’re talking about delivering video over HTTP.) If you’ve got a big burst of data and the TCP window takes a long time to open up you start building up a big send buffer on the server. (And remember in this use case you’ve got a lot of clients connected!) That requires a lot of memory to hold the send buffers for each client. What happens then is that servers will start closing connections en masse because it needs to save memory or because it thinks that the client has become somehow unreachable. This is made worse by the fact that even if the connection scales up and then scales back down it re-settles at the low rate and the process has to be repeated. The user’s experience is that the video stream stops and restarts or just stops working altogether when the server hangs up. The solution? Using a constant rate that doesn’t require the TCP window to open up suddenly and doesn’t require large send buffers for each client.

For the use case where you’re streaming large files it might not be reasonable for the client to cache a lot of data. You also might be serving up a lot of data to a lot of clients and you might want to avoid the large send buffer problem as well, just for different reasons.

And for the last use case where you’re in a CPU-constrained environment the bursting nature of variable bitrate videos means it often takes a large bursts of CPU to handle those bursts. While CPUs do burst up faster than TCP does, you might be talking to constrained processors (think mobile) or you might be serving up files near HD-sized content, which CPUs often struggle to decode.

In any case there are a number of use cases for constant bitrate encoding. Back to the question of what’s improved in Theora 1.1.

In Theora 1.0 the rate controlled encoding mode was very very broken. This resulted in two things:

1. People trying to do live streaming ran into problems.
2. People who used rate controlled settings to compare overall Theora quality to the quality of other encoders saw worse results than the format actually represented.

The first issue is clear – it was broken, it should be fixed. And it has been. The new encoder does a pretty good job of maintaining bitrates, changes quality on the fly, drops frames and even includes a “soft-target” mode so that bitrates can fluctuate a little bit to maintain quality while occasionally breaking the bandwidth rules.

The encoder also has a wonderful new piece of functionality that people will find very useful. It’s now possible to specify a maximum rate ceiling for video encoding while also specifying a minimum quality floor. What this means is that the encoder will try and maintain very crisp video frames within rate constraints. This means that it will aggressively drop frames instead of creating frame deltas that are fuzzy or low-quality. While this might sound like a poor trade-off it’s actually very useful. If you’re showing a live video of a presentation you usually want a crisp video of the slides and having a lower frame update rate is very acceptable.

The second issue that was caused by the bad rate control in Theora 1.0 is an issue of marketing. People would often use the encoder with the fixed bitrate mode instead of the quality mode and dismiss the results as a reflection of the format instead of problems with the encoder. We hope that people find better results with the new encoder.

A two-pass mode has been added to the encoder that can create rate controlled videos with very predictable bandwidth requirements.

In addition to fixing the single pass rate controlled encoder in 1.1 a two-pass encoding option has been added. This means that if you are transcoding a file (as opposed to doing a live stream) you can create a very consistent bitrate in a file if you want. This is because the encoder can look ahead in the stream to allocate bits efficiently. Monty from Xiph made a graph that shows one example of the bitrate in a file with one pass and two pass.

Above: graph of quantizer choice (which roughly corresponds to the encoding quality) when using default two-pass bitrate management as opposed to one-pass (with –soft-target) when encoding [the Matrix movie clip] at 300kbps. Both encodes resulted in the same bitrate. The quality of a one-pass encode varies considerably as the encoder has no way to plan ahead.

CPU usage during encoding is much more consistent.

People who were doing live streaming often saw huge spikes in CPU usage during high-motion events. This has been fixed and now CPU usage is much more consistent during single pass rate constrained encoding making it much easier to live stream video.

Decoder performance has been improved.

And last but not least the decoder has been made faster during the 1.1 release. How much faster depends quite a bit on the clip, but people are reporting that the new encoder is anywhere from 1.5-2x faster than the 1.0 of release of libtheora.

Coming soon to a product near you.

This release is a library release. It’s not a product in itself, but is instead something that other products include. So over the next days and weeks we’ll see other products pick up and start using this as part of their releases.

Enjoy!