As explained in this blog post by Jared Wein of the Firefox team there are quite a few new features in Firefox when it comes to playing HTML5 video. As an Aurora user, I am most excited about the option to go full-screen, the ability to overlay video statistics and to save a snapshot of the current frame as a JPG. You can see me talking about and showing them in this short video:
Firefox has a few features up its sleeve when it comes to HTML5 video playback you might not be aware of:
Firefox‘s seeking is now accurate to millisecondsmicroseconds, there is visual feedback when the video has stalled and clicking the whole video pauses and plays it
Firefox Beta has specialised controls when you watch video on small devices and watching HTML5 video shows a pleasing background rather than a brutal grey
Firefox Aurora has fullscreen, statistics overlay, saving of snapshots and controls appearing when the video ended
Firefox Nightly has a full-screen button, fading video controls after 2 seconds of non-interaction, no loading throbber on audio, error reporting when a video could not be loaded on the video, loop attribute support, and resizing of videos larger than the browser window when you watch them directly
Planned features for Firefox are an overlay play button like YouTube when the video is not set to autoplay and turning off screensavers during fullscreen playback.
Yesterday at the London Ajax Meetup I spoke about “Rethinking User Registration with BrowserID”. In the (roughly) half hour talk I covered the need for new login systems on the web, what is broken now, how BrowserID is a solution for these issues and showed how easy it is to create a BrowserID login (both from a developer and from a user point of view).
The video of the talk can be seen on the Skillsmatter website (HD on and fullscreen is your friend as the embedded version is a bit small).
CSS 3D transforms as supported in the latest Aurora allow us to do some nice effects that in the past were only possible in Flash or with a lot of trickery using skewing and filters. I was asked to show a small demo the other day and thought it would be fun to spice up the classic image rollover:
You can see this in action using Chrome, Safari or Firefox Aurora/Nightly. Older browsers should just show a normal roll-over (and yes, the first example looks weird due to the logo transparency but it makes the effect much cooler in supporting browsers).
A few days ago I was asked to deliver the first talk of the amazing Beyond Tellerand conference in Dusseldorf, Germany. The talk Breaking the barriers – moving browsers and the web forward introduced a lot of new ideas and technologies that are worked on my Mozilla and others to make the web of the future better.
Here is the video of the presentation with jump points and links to more information. If you want to see the slides of the talk, they are available here and there is also an audio recording on archive.org.
Modern web technologies of HTML5 and friends that can be used right now (with 64 myself as a demo) [03:50 - 07:19]
Rich HTML semantics (HTML5)
Self-validating forms (HTML5)
Richer form controls with fallbacks (HTML5)
Canvas for painting in the browser (HTML5)
CSS gradients, multiple backgrounds, animation and transition
CSS 3D transforms
Local storage and offline storage
SVG for scalable and interactive graphics
RequestAnimationFrame for secure animation
History API
WebGL
Taking on challenges – we need you to show the world that web technology is good enough to do jobs that in the past were only possible with native or server-side code (with Joe Lambert’s image unshredder as the example) [07:20 - 08:04]
Breaking the browser mould – showing that the browser interfaces can be manipulated with HTML and JS (with browser menus, context menus and the Fullscreen API as examples) [08:05 - 10:54]
Online identity and issues with current login systems (with BrowserID as a solution to a lot of the problems we have right now) [17:25 - 29:35]
Apps, the shortcomings and myths of native apps and the opportunity to build hybrid apps with web technologies (with Open Web Apps and Web Intents as examples) [29:36 - 37:34]
As you might be aware, Mozilla is working hard on making mobile development possible with purely open technologies. For this, we are defining a set of APIs to access the hardware of mobile devices called the Web APIs. John Hammink of Mozilla published a blog post on the subject on Monday outlining the current state of affairs. A good chance to have a chat about where we are and to ask what you can do to help Mozilla in this effort.
1) John, what is your involvement in the Web APIs project?
I am the lead QA. I am also assuming an evangelist role – we need to get the community involved as early as possible around testing these. Not to mention building apps around these! Wanna help?
2) What are the APIs that are ready to play at the moment and what do people need to try them out?
I’ve made a few atomic testpages around testing the webapis available. These are: Battery, Camera, IndexedDB, Vibrator and SMS. More are coming soon – but perhaps require combination with other APIs (for example, Telephony requires Audio); that and other APIs are specific to Boot to Gecko (B2G).
3) So the battery API is one of the first ones to land. Whilst interesting as an information piece on your phone, it is not quite that fascinating for developers. What uses cases can you think of? Maybe reading the battery before your massive WebGL animation and reading it afterwards?
Maybe I have a WebGL simulation or, perhaps, a game that requires a certain power output or power availability. We can, for example, trigger notifications when power goes below a certain level. If it’s a productivity app, maybe we want to trigger the user to save their work.
4) Camera APIs, now we are getting somewhere! How is the support for Canvas on Fennec and mobile devices? The first thing that would get me interested is to do an Instagram in HTML5 by changing the photos.
Canvas works identically to desktop. Taking the picture – which JS then renders – is the bulk of the use-case.
Arguably you could implement zooming so that you can check the picture in more detail. Another thing we have yet to try is checking the rendered image for EXIF metadata.
5) You mention IndexDB as an alternative to store local data as that could be dangerous to the file system. The main benefit of localStorage() is that the API is dead easy though. Is there a chance to have a wrapper to use the localStorage() API but store in the DB?
The problem is that localstorage() is sync – which makes it easy to use but, as you can imagine, introduces tremendous performance issues. As our Indexeddb is an async API – you can’t wrap a sync api around an async one.
6) Good vibrations. The vibrator API allows you to vibrate the device. I can see that working really well for games and notifications. Or with the orientation API telling you when you tilt too far. Is there a way to detect if the vibration is turned on or off? I know quite a few people that turned off device vibration.
I can’t wait to see where my name shows up in search results thanks to this one. There is no API for detecting this – it’s been theorized that you can deduce its presence using the orientation API – i.e. it would see the vibrations. But this would vary from device to device; as not all devices have vibrator.
7) Sending SMS with JavaScript is the last API you mention. This is working as far as I understand it. What is the most requested use case there? Is there a throttling mechanism to stop people from sending SMS spam or mass mailings?
Aside from the about:config setting dom.sms.enabled, there is no specific throttling mechanism. It is to be a “trusted” API, that is, we won’t expose it to all pages. As with many of these APIs there are inherent security threats – in the case of this one, you can for example, end up sending text messages to paid-for services.
8) This is all pretty cool. What do you want from people now and where can they go to get all that to play with?
Check out my blogpost and try out the testpages themselves on my people share! I’d recommend using latest XUL nightlies – for SMS, you need a phone with a sim card and a special build which is linked in the blogpost. Play around! Note that these are presently atomic APIs, each representing a single function. Make suggestions via bugzilla -and write bugs! And it goes without saying also to try to combine these (and the others as they land) into working apps of your own. Our challenge will not only be to find out how these APIs work on their own but also in combination with another. I would say that IndexDB at this stage represents a particular challenge, due to its size and complexity.
Report bugs with any of the demos and/or APIs themselves (make sure to include your device model and Android version, as well as the nightly fennec version you are using, in the bug.).
Tomorrow we will release the MDN advent calendar at http://thewebrocks.com/calendar with a daily link on a web technology product, a MDN wiki page or a great demo collected by us over the last few days. You can get a preview of how the calendar will look and work here:
As an extra bonus, we thought it would be fun to document the step-by-step development of the calendar and release it for you to re-use or get inspired by.
It is an example of how to build something server-side (so you can’t cheat by setting your operating system calendar ahead), enhance it with JavaScript and make it smooth by using CSS transitions. This should work for everybody and by playing each technology to its strengths, the code is very small indeed.
Enjoy, and see you tomorrow and the day after, and the day after that and…
And here we are in a quick interview, chatting about how he approached the problem and why he used web technologies:
1) The specs of the competition said you can use Python, Ruby or C++ and you went instead for JavaScript and Canvas. Why?
JavaScript is my language of choice and what I use most of the time. I also find it the most fun as its easy to post your creations online and show people immediately, no downloads or dependencies to worry about. I wasn’t really looking to get hired by Instagram so adhering strictly to the rules wasn’t a big deal. Essentially it was an appealing problem that I thought could be solved in a browser, so wanted to prove that was the case.
I actually knew from the start that creating the algorithm to unshred the image was only half of what I wanted to do. That in itself was going to be a challenge and quite satisfying to solve but I really wanted to get to a point where I could make the slices move into the right place. I wouldn’t have been able to create this effect as quickly in any other environment.
2) How did you approach the problem of unshredding the images? Did you do similar things before? What is the logic you use?
I wasn’t too worried about the most efficient algorithm so I just started by thinking about how a human might solve this type of problem. I’ve played around with image data a little before but nothing that would have been directly useful to this particular problem.
If I were solving this in the physical world I’d start by picking up a slice at random then pick up each other shredded slice and see if it looked like it fitted to the left or right of the piece in my hand. I’d then repeat this till the image was back to its original state.
This gave an indication of the kind of structure my code should have, I just needed to work out how to computationally measure whether two strips ought to be next to one another. For this I just compared the pixels closest to each edge and measured the euclidean distance between the colour values. It seemed to work pretty well!
3) I see you are not using typed arrays in your solution yet. It seems to perform well the way it is now, do you think they’d make a difference?
I’m not sure, efficiency wasn’t high on my priorities for this challenge so I didn’t really look to optimise it but as you mention it seems to perform quite well. I haven’t tried the algorithm with larger images but I suspect it would perform not so well.
4) What was the feedback so far? Has Instagram contacted you?
The feedback has been really positive, especially via Twitter. I think generally the competition has had quite a high profile which certainly helped.
Instagram haven’t been in contact yet but it would be good to hear from them. I suspect they’re focusing on all the people who used the recommended languages
5) Seeing how easy this was can you see other challenges with images and canvas? Could you think of ways how to make it easier for developers, for example by extending the API?
The way you access pixel data in Canvas is a little complicated, it might be useful to have some helper functions to let you get access to a single pixel via X/Y coordinates. I ended up writing a little helper function to do this for this challenge, I’m going to try get the code up on GitHub shortly so this might be of use for others.
It would also have been useful to have been able to call toDataURL() on a CanvasPixelArray object. My implementation used the DOM and CSS3 transitions to move the images into their correct places after the algorithm had solved the order, being able to access a Data URI for each slice would have be handy.
If you want to chat with Joe, he is available on Twitter as @joelambert.
Luke gave an overview on the history of HTML5 vs. XHTML and the approaches of the W3C and the WHATWG. He explains how standards are produced and how this can be daunting. He then shows how “shipping code wins” and showcases lots of examples of new technology in action.
He concludes with a nice comparison on how much simpler it is to build rich form interfaces with HTML5 and polyfills than with native code for various platforms.
All in all a presentation very worth while from someone who is in the trenches and who is not afraid to use new technology and wait for the market to catch up.
You may not know it, but the HTML5 specifications go beyond what we put in the pages and also define how parts of the browser should become available to developers with HTML, CSS and JavaScript. One of these parts of the specs are context menus, or “right click menus”. Using HTML5 and a menu element you can add new options to these without having to write a browser add-on. In Firefox 8 (the current one) we have support for those. See the following screencast for a context menu demo.
The image example is pretty simple and was actually written by Paul Rouget as a demo in the original Firefox bug request. The main core is the HTML of it:
As you can see you link the menu element to an element via its ID. The contextmenu attribute then points to this one. Each menu can have several menuitems. Each of those gets a textual label and a possible icon. You can also nest menu elements to create multiple layer menus. Here, we add inline onclick handlers to point to different JavaScript functions to call when the menu item gets activated. The resulting context menu looks like this:
The functionality is simple, all the rotate() and resize() functions do is add class names to the image using querySelector and classList:
function rotate(){
document.querySelector('#menudemo').classList.toggle('rotate');}function resize(){
document.querySelector('#menudemo').classList.toggle('resize');}
The real effect is in CSS transforms and transitions. As the image has an ID of menudemo here is what is needed in CSS to rotate and resize:
Notice that in a real product we should of course add the other browser prefixes and go prefix-less but as the functionality now only works in Firefox, this is enough for this demo.
Detecting support and visual hinting
Now, as this is extending the normal user offerings in the browser we need to make it obvious that there is a right-click menu available. In CSS3, there is a context-menu cursor available to us. When context menus are available, this should be shown:
We test the browser for support by checking for contextmenu on the body element and for HTMLMenuItemElement in the window (this has been added as a pull request to Modernizr, too).
Wouldn’t HTMLMenuItemElement be enough? Yes, but a real context menu should only offer functionality when it is sensible, and that is where contextMenu comes in.
Turning menuitems on and off depending on functionality
As a slightly more complex example, let’s add a “count words” functionality to the document. For this, we generate a counter element that will become a tooltip when the words were counted:
var counter = document.createElement('span');
counter.id='counter';
counter.className='hide';
document.body.appendChild(counter);
counter.addEventListener('click',function(ev){this.className='hide';},false);
This one is hidden by default and becomes visible when the hide class is removed. To make it smooth, we use a transition:
We then loop through all the menuitems with the class wordcount and apply the functionality.
var wordcountmenus = document.querySelectorAll('.wordcount'),
i = wordcountmenus.length;
while (i--){
wordcountmenus[i].addEventListener('click',function(ev){// add functionality },false);}
We need to find out what has been selected in the page. We do this by using getSelection() and splitting its string version at whitespace. We then show the counter by removing the hide class name.
var wordcountmenus = document.querySelectorAll('.wordcount'),
i = wordcountmenus.length;
while (i--){
wordcountmenus[i].addEventListener('click',function(ev){var text = document.getSelection(),
count = text.toString().split(/\s/).length;
counter.innerHTML= count +' words';
counter.className='';},false);}
You can see this in action in the second context menu demo. Now, the issue with this (as explained in the screencast) is that it always counts the words, regardless of the user having selected some text. What we want is the menu only to be active when there is text selected.
So in order to make our menu only become available when it makes sense we check if there is a selection in the document. Every context menu fires an event called contextmenu when it opens. So all we need to do is to subscribe to this event.
When something is selected in the document document.getSelection().isCollapsed is true. Otherwise it is false, so all we need to do is to enable or disable the menu item accordingly:
The last thing to solve is the position of the mouse to position the counter element. As the menu selection event doesn’t give us the mouse position we need to add a contextmenu handler to the whole document that positions the counter invisibly behind the menu when it is opened:
Here’s a quick screencast how to create a 3D image rollover and still give a useful interface to browsers that do not support 3D transforms. If you want to see the effect in Firefox get the latest Aurora or Nightly. Check the following video to see what it looks like (first with a browser without CSS 3D transport, then with a newer one):
The screencast is on YouTube:
The main procedure to achieve the effect is simple. First we need a semantically valuable way to show these images, in our case a HTML list with figures and figcaptions. Notice that images still need an alternative text as the figcaption can apply to several images:
<ul><liclass="image3d"><figure><imgsrc="mittens.jpg"alt="Mittens the cat"><figcaption><p><strong>Mittens</strong> loves to play with yarn and stuff.
</p></figcaption></figure></li><!-- repeated --></ul>
We then position the caption absolutely inside the list item, give the list item dimensions and an overflow of hidden and move the caption outside of the visible space. When the user hovers over the list item (or focuses it with a keyboard) we move the caption to the right place and lower the opacity of the image.
We make the effect smooth by adding transitions. Notice that it makes sense to list all the browser prefixes and set a prefix-less fallback. That way we don’t need to re-write code when a new browser supports them.
.smooth img {
-webkit-transition: all 1s;
-moz-transition: all 1s;
-o-transition: all 1s;
-ms-transition: all 1s;
transition: all 1s;}.smooth figcaption {
-webkit-transition: all 1s;
-moz-transition: all 1s;
-ms-transition: all 1s;
-o-transition: all 1s;
transition: all 1s;}
In order to rotate the kitty in 3D space, we need to give the list item a perspective and rotate the image and shift it in 3D space with translate3D to avoid clipping:
That’s it. By adding all the browser prefixes, falling back to a simple rollover and by making sure you do the effect on hover and on focus you support all the browsers and bring the cool to the newest ones.
Today we have a quickie for you: 
