Last Saturday in London, England the State of the browser conference brought together developer advocates from almost all browser vendors to give the audience an overview of what is going on in the world of browsers.
My involvement was to talk about the state of HTML5 when seen from a native market’s perspective, show some cool new technologies that need our input and take part in the browser panel to discuss current issues. Here are the talks and screencasts. Videos recorded by the organisers should follow soon.
Talk “Broken HTML5 promises – are we ‘appy?”
The main Mozilla presentation was about feedback on HTML5 we got at Mobile World Congress from mobile developers, how we as an HTML5 community fail to answer their questions and get tangled up in petty bickering over details instead and what Mozilla does to make HTML5 work across the board.
Breakout session: “The bleeding edge of HTML needs blood donors”
The breakout session (which was repeated twice) was much less of a “talk” but more of a show and tell in a smaller room. Therefore the screencast is a bit more raw but shows what you can do right now.
All in all the conference was great value for money. All the speakers had great information to give and there was no “marketing talk” promising things that don’t work outside lab environments.
Michael Mahemoff did a great job introducing the day with a “native vs. web knockout” talk.
Paul Kinlan showed what is coming in Chrome and how Web Intents can change the way we solve app communication over the web
Martin Beeby gave a glimpse of how the web can merge with newer devices and UX needs of users
Seb Lee-Delisle took all the browsers to the performance test to end all performance tests by animating millions of 3D particles and seeing which browser would be the one that can show the most without slowing down. In the end Firefox was the winner with 3695244 particles at 10FPS. Of course this is not a real measure (especially seeing IE10 was run in a VM) but it is always fun to see Seb code live.
I guess my favourite piece about the conference was that the browser panel was very much about answering people’s questions instead of trying to beat each other in being the browser that people should use. British understatement at its best.
It is not often that you find yourself in a disused nuclear reactor from the 50s to talk about state-of-the-art web technology. For about a hundred developers and designers this is exactly what happened last Saturday in Stockholm, Sweden.
The R1 reactor played host to the Mosync hackathon organised to get developers to try out the Wormhole and Reload technologies, both of which make it very easy to build apps based on HTML5 or C++ for both feature and smartphones.
Mosync asked Mozilla to participate after a quick brownbag in their office on HTML5 a few weeks ago. So we went and gave an introduction on “HTML5 and the near-future of the web”. You can read the slides here and see a screencast with audio on YouTube.
And as I had some time and brought my trusty Competition Pro joystick, I thought I should give the Gamepad API a whirl and created the world’s first joystick powered kitten cube (maybe).
When we think of sound in an HTML application we might think of two things: We remember all those sites that started playing loud obnoxious background music when the page loads and then we think about music playing apps.
Sound can however be much more: when building immersive app experiences it can be a crucial attribute. It can enhance tactile feedback or communicate activity or changes in state to the user. A ping sound when a new email arrives or a dismissive sound when there was an error can make things much more obvious for the end user.
Prior to HTML5 most developers had to resort to plug-in technologies like Flash, Quicktime, Real player or Windows Media Player to play audio. These, of course, required that these technologies were installed on the users’ machines and the plugins being active.
With HTML5, we have an audio element that natively supports sound playback. As with any HTML element, you can even play nice with older technologies by providing fallback content. For example by simply linking to the audio file:
<audio controls src="intro.mp3"><ahref="intro.mp3">Introduction to HTML5 (10:12) - MP3 - 3.2MB</a></audio>
As not all browsers support the same audio formats (MP3 not being a free format makes it impossible to decode it in an open source browser) you can provide the same audio in different formats:
When your application checked that HTML5, Canvas and all the other things needed for your functionality is supported then backward compatibility is less of a concern, however you may still have cross-browser compatibility concerns since browser vendors are not fully converged on common feature implementation. That said, basic support for audio is available across all major browsers.
You don’t need to have an audio element in your HTML, you can also create them on the fly in your JavaScript:
var aSound = document.createElement('audio');
aSound.setAttribute('src','PlayMe.ogg');
aSound.load()
However, there may be advantages to using the tag in your HTML.
Using the tag adds to the semantic integrity of your markup.
It makes your code easier to read and understand.
The tag can display controls so the user can play the audio and seek in it with native controls that are also keyboard accessible
The tag has an optional preload attribute that tells to the browser to load the audio before users start playing it.
Browsers have a distinct loading sequence of referenced assets. By leveraging that sequence, it iss possible for you to help improve the performance of your app.
Here are some examples for using the tag in HTML5.
These controls differ from browser to browser and operating system to operating system, but all have the same features as shown in the following image:
You can easily hide or display the audio element’s controls whenever appropriate (like when the UI state changes) with a simple bit of JavaScript:
var myAudio = document.getElementById("TimerBellSound");if( myAudio.hasAttribute("controls")){
myAudio.removeAttribute("controls");}else{
myAudio.setAttribute("controls","controls")}
As Terrill Thompson explains in his blog post HERE, we can easily create a custom player as well. Not only does this provide us with the flexibility of defining our own user interface but it lets us address accessibility concerns as well. His player looks like this and has a consistent look and feel across browsers and operating systems:
So what could sound do in your app? As an example, consider the follow application prototype:
This application will be a timer for athletes. When in use, the athletes won’t be sitting in front of the device that is running the app. It will be running on their computer, tablet or phone and, while they may glance at it to check the time, for the most part they will rely on audible feedback for when to start working out, when to rest, and when to increase or decrease the intensity of their workout.
The audio element in HTML5 makes adding sound to your app both easy and straight forward.
As we’re having a HTML5 Audio developer derby this month, I thought it fun to play with audio again. And I found it sadly enough pretty frustrating.
One thing I proposed in a lot of talks is using the idea of CSS sprites and apply them to HTML5 audio. You’ll get the same benefits – loading one file in one HTTP request instead of many, avoiding failure as files might not get loaded and so on.
Clicking the different buttons should play the part of the music file and nothing more. This works fine in Firefox, Chrome and Opera on my computer here. Safari, however, fails to preload the audio and the setting of the current time is off. The code is simple enough that this should work:
// get the audio element and the buttons container// define a sprite object with the names and the start and end times // of the different sounds.var a = document.querySelector('audio'),
buttoncontainer = document.querySelector('#buttons'),
audiosprite ={'all':[0,5],'boing':[0,1.3],'boomtchack':[2,2.5],'peng':[4,5]},
end =0;// when the audio data is loaded, create the buttons // this way non-HTML5 browsers don't get any buttons
a.addEventListener('loadeddata',function(ev){for(var i in audiosprite){
buttoncontainer.innerHTML+='<button onclick="play(\''+
i +'\')">'+ i +'</button>';}},false);// If the time of the file playing is updated, compare it // to the current end time and stop playing when this one // is reached
a.addEventListener('timeupdate',function(ev){if(a.currentTime> end){
a.pause();}},false);// Play the current audio sprite by setting the currentTimefunction play(sound){if( audiosprite[sound]){
a.currentTime= audiosprite[sound][0];
end = audiosprite[sound][1];
a.play();}}
Now, this is nothing new, Remy Sharp wrote about audio sprites in 2010 and lamented especially the buggy support in iOS (audio won’t load at all until you activate it with a touch – something that sounds horribly like the “click to active” Flash has on IE).
All in all it seems HTML5 audio still needs a lot of work which is why a lot of Games released lately under the banner of HTML5 use Flash audio or no audio at all. This is sad and needs fixing.
Interestingly enough there are some great projects that you could be part of. Are we playing yet? by Soundcloud and others for example is a test suite for audio support in browsers. You can write own tests on GitHub and report results to the browser makers.
The jPlayer team has a great HTML5 Media Event Inspector showing just how many of the HTML5 media events are supported in your current browser.
If you want to be safe, you can use SoundManager 2 by Scott Schiller to have an API that uses HTML5 when possible and falls back to Flash when the browser doesn’t have any support. It also fixes a few issues for you.
All in all it would be interesting to hear what you think of the state of HTML5 audio:
Did the companies that heralded HTML5 as the end of plugins drop the ball?
Is it really sensible to have an API that returns probably or maybe or ” when you ask it if the browser can play a certain type of media?
What could be done to work around these issues?
Let’s re-ignite the discussion on HTML5 audio, after all we need it for the future of messaging in the browser and telephony, too.
Oh and another thing. Of course there is the Audio Data API of Firefox and the web audio proposal from Webkit available but getting those running in mobile devices will be a much bigger change. If you want to know more about those and libraries to work around their differences, there is a great overview post available on Happyworm.
HTML5 needs spokespeople to work. There are a lot of people out there who took on this role, and here at Mozilla we thought it is a good idea to introduce some of them to you with a series of interviews and short videos. The format is simple – we send the experts 10 questions to answer and then do a quick video interview to let them introduce themselves and ask for more detail on some of their answers.
Today we are featuring Andrew Betts, director of Assanka.
1) You worked on the HTML5 application for the financial times. What made you go HTML5 and not native? How many people were involved at all?
The FT has an agnostic approach when it comes to the formats and technologies we can use to display our content. The choice has to suit the content, the reader, and our business model. Before we launched the HTML5 app there was already an FT native app on iOS, and it received an Apple design award, but HTML5 offered several significant benefits, the most important of which was to allow us to maintain a direct relationship with the reader. Secondary benefits include a faster update process, an investment in future cross-platform compatibility (HTML5 is yet to fully realise the dream of write-once, run everywhere), and freedom from restrictions and rules that are (or may be in future) imposed by app store operators.
2) What would you say was the easiest part and what was the thing that you had to spend the most time on to make it work?
Interactions are the hardest thing to crack. Matching behaviour to user expectations is frighteningly complex, especially when you realise that instinctive expectations of swipe behaviour is actually quite varied and inconsistent. One example of this complexity is swiping horizontally from the bottom of a section page, which takes you to the top of the next one. If you then swipe back, do you end up where you were, or at the top? So dealing with interactions is both technically hard and architecturally challenging.
Offline behaviour is also very tough. We get a lot of “User x can’t use the app offline” style support requests, and those are very hard to debug. We log every report but the priority is to figure out the scale of the problem.
The easiest thing to deal with is probably layout. We send all legacy browsers to m.ft.com so we only have to cope with the most recent major versions, and that means that generally we get good standards support for CSS.
3) How did you deal with the differences in interfaces? A mobile has much less space than a tablet and again less than a desktop. Are you using mediaqueries to give different experiences?
We group devices into one of four sizes based on media queries, and we vary which resources we use slightly based on the reported DPI, but most of our responsive layout logic is done in JavaScript. We make a few assumptions right now to make development easier, such as that the user cannot change the size of the browser window, except by rotating the device, so we only need to change the layout on orientation change, not in response to arbitrary resizing of the window. We also currently assume the user has no mouse cursor, so there are no hover effects.
4) How does input get into it? Is it simple enough to build touch interfaces in HTML5 or did you have to use some trickery?
Quite a lot of trickery is involved, unfortunately. The browser has to tread a fine line between on the one hand offering a touch API to web developers and getting out of their way to let them use it, and on the other, dealing with the fact that almost all websites are still built for keyboard and mouse, so the browser needs to help the user deal with that. We developed a polyfill called Fastclick which makes click events fire as fast as touch events, and that certainly helps us provide an experience that is as snappy as using a native app.
We also deal with swiping in a few quite distinctly different ways. Swiping between sections is an endless carousel, and we implement the tracking and sliding ourselves, by maintaining hidden page containers to the left and right of the current view and applying CSS 3D transforms. But paging through an article requires different scroll mechanics – it has a fixed length, and we have to lay out the whole article in one go to know how many pages there are. Pages that scroll vertically are a different challenge again, and there we use a combination of leaving the browser to do scrolling natively, and simulating it where we need slightly different behaviour.
You’ll also notice that the pages don’t ‘bounce’ when you hit the top and bottom of the page, despite that being the normal behaviour in the iOS browser. We’ve gone to quite some lengths to ensure that the user experience mirrors an app and doesn’t feel like the device ‘coping’ with a site that hasn’t been designed to be viewed using a device with touch input.
5) You said in your talk at Blackberry Devcon that you are using localStorage to cache your JavaScript libraries and some of the images. Why not use WebSQL or IndexDB for that? Isn’t the synchronous nature of localStorage slowing things down?
No, actually localStorage is considerably faster than WebSQL (we are yet to use IndexedDB because it’s not supported on some of our target platforms). The time required for a lookup by key in localStorage is many times less than the equivalent SELECT statement on a WebSQL table. It’s true that it’s synchronous, but if the critical aspect affecting perceived performance is the time required to fetch data, which it often is, then localStorage offers a much faster response. it’s also typically reliable and relatively stable compared to the behaviour of WebSQL and particularly the HTML5 app cache.
6) You also said that localStorage uses UTF-16 instead of UTF-8. This means you can store two ASCII characters in one UTF-16 one and thus double the storage capacity, right? Can you share some code? This would be incredible to use for everybody out there.
Both localStorage and WebSQL use UTF-16 in webkit browsers, it’s one of many aspects of the implementation of these technologies that don’t make a lot of sense! We’re experimenting with some algorithms for compressing our data so we can make more efficient use of that storage, and one of those is to combine two characters into a single UTF-16 one. We’re not using that in production yet, but when we do we’re certainly hoping to open source it.
There are simpler things to be done as well. A cursory poke and prod of WebSQL tables suggest that column names are stored along with the value in every cell, so some attempts to increase efficiency are as simple as using single character column names. Clearly we ought not to have to do all this, but it’s the price you pay for dealing with bleeding edge features in browsers.
7) What would you consider the best start for someone when they want to build an HTML5 app? What are the first things to get sorted?
Decide ahead of time what you’re trying to achieve. I could be a pedant and say that making your site HTML5 is really just a matter of changing the DOCTYPE, so an ‘HTML5 app’ is really whatever you define it to be, and having a clear idea of that is a good place to begin. Are you building something just for touch, or for keyboard and mouse as well? Does it need to work offline? Will it be crawlable by search spiders? Will it work with JavaScript disabled?
There are some great resources like MDN, but we often find ourselves reading the HTML5 specs, and sometimes even the webkit source code to find out how something is actually implemented.
8) Can native and web apps be the same thing?
Technically, I guess not – strictly speaking a native app is complied code running directly on a platform, but the hybrid model seems to be getting ever more popular, and I often see apps which are thinly veiled web browsers. If you look at it from the user perspective and say, can a web app feel exactly like a native app, I don’t see why not. The challenges are greater because of the diversity of uses that web technologies are designed for, and the diversity of platforms on which they are used. And for uses such as gaming, native code is always likely to run faster and have deeper access to the OS.
But for applications like publishing, which after all was the original purpose of the web, web technologies do provide support for most of the user experience concepts that we need.
9) What about testing? How did you approach this? Do you have an array of hardware to play with?
It’s a nightmare. We had a local electrician built us a charging station where we could store all our devices and keep them charged all the time. We have around 45 individual devices in our team (that’s around 4 times more devices than we have people). We’re constantly looking at ways of improving our testing process, and we keep revisiting automated, on device testing, but we’re not using it in our build cycle as yet. Right now we have a team of very dedicated testers who poke and prod devices all day.
10) As someone who went through the process of building a big HTML5 app, what would you like to have from browser vendors to make your life easier?
A blithe answer is ‘How long do you have’, but in practice we have to accept we’re using freshly minted technologies and there might be teething troubles. The main things on our shopping list are: more frequent and aggressive browser updates (especially on Android, where the browser is the problem child of the mobile web world), a better and more reliable app cache, and hardware acceleration of CSS transforms.
HTML5 is really more than one thing. In the strictest sense, HTML5 is fifth major revision of the W3C specification of the markup language that is used to create web pages.
But in a practical sense, HTML5 is far more than that. For developers, HTML is a wave of technologies that are evolving and gaining browser implementations along roughly the same time-line as the markup language itself. Many of the related technologies are being spearheaded by the Web Hypertext Application Technology Working Group (whatwg.org), but still other technologies in this genre are driven from elsewhere. For example WebGL, the 3D rendering engine for Web Apps, is driven at Khronos, and Mozilla’s WebAPI team is leading a collection of relevant API work including a comprehensive array of device-specific APIs. And the W3C was a Device APIs Working Group
Mozilla is also investing heavily in an initiative to facilitate the use of standards-based Web technologies to build applications that can be installed and used in the same way a “native” applications would be.
There are many reasons that doing so might be desirable, not the least of which is the ability to build much or all of your application for use on many different devices and platforms using a single set of development technologies.
Developers can already build HTML Applications (with help from the HTML5 Apps documentation ) and submit their applications to the Mozilla Marketplace.
App versus Page ?
When people start talking about building apps with HTML5, one of the early questions that comes to the conversation is “What’s the difference between an app and a page?”
This is bit of a difficult question to answer because of the very wide range of answers which can all be true, but which may be completely different.
While your HTML5 app should be more than a simple HTML5 page, technically speaking it doesn’t have to be. Using the steps outlined in the guide “Getting Started with making (HTML5) apps”, you could simply create an apps manifest file and submit your HTML page as an app.
When an Open Web app runs, the Mozilla Web Runtime installs local copies of the HTML5 markup and all of the other web assets that are specified using an HTML5 Cache Manifest ( read here ).
If a Web page is dynamically generated, as in an ASP.NET, PHP, etc., application, any cached version represents a snapshot of a single point in time. When submitted as an app, the page actually “becomes” an app, getting a launcher item similar to native apps on whatever the host operating system is. When the app is launched, the local assets are used if there is no Internet connection or if the caching mechanism determines that the assets have not changed since they were last downloaded.
By doing the above we have not added any additional functionality to our application. All we have done by declaring our page to be an app is to get the run-time to copy the bits locally and to run those local bits if the device is off line or if the bits haven’t changed.
Does this get us anything ?
Is there value in just this simple step ?
Yes, it gets us three valuable improvements:
Users can see our page even if they are not connected because the run-time renders the cached version if there is no Internet connection.
Processor cycles and bandwidth consumption from our infrastructure or hosting are greatly reduced because new bits are only fetched when they are needed.
Performance improves because resources are loaded locally by default.
Many organizations are starting with this approach because very little technical effort is necessary and it can be done very quickly. In most cases all you need to do is to create an app manifest file ( more info here ) and to submit the app to the marketplace. Developers can submit apps now at the Mozilla marketplace Developer Submission site so that they will be available to consumers when the “buy side” of the marketplace opens later this year.)
But if that’s all you do with your app, you are missing the opportunity.
Take a look at http://caniuse.com/ for a snapshot view of technologies that are available to developers using the modern web runtime.
Whether we are building a real Open Web App from scratch or we are converting an HTML page to an app and then adding modern features, HTML5 era technologies let us really start to developer client / server applications that utilize the browser and its host resources, thereby reducing our server requirements while improving the end user’s experience.
Developers can do things like :
Read and write data locally and sync with a server when appropriate.
Run multiple threads of execution at the same time.
Exchange content with other applications, or share logic with them.
Interact with the device on which their app is running.
Touch user interface
GPS
Camera
Richly integrate multimedia.
And much more…..
Since apps’ source code is deployed to a single URL and is updated on the device on which the app runs only when updates are necessary, versioning is low cost and friction free. This makes it easy to start by making a page and then iteratively adding features one after another.
Lets do it !
Since this model makes it so easy to get started, why not give at a try?
BrowserQuest is a tribute to classic video-games with a multiplayer twist. You play as a young warrior driven by the thrill of adventure. No princess to save here, just a dangerous world filled with treasures to discover. And it’s all done in glorious HTML5 and JavaScript.
BrowserQuest can be played by thousands of simultaneous players, distributed across different instances of the in-game world. Click on the population counter at any time to know exactly how many total players are currently online.
Players can see and interact with each other by using an in-game chat system. They can also team up and fight enemies together.
BrowserQuest is a game of exploration: the more dangerous the places you go, the better the rewards.
Powered by WebSockets
WebSockets are a new technology enabling bi-directional communication between a browser and a server on the web.
BrowserQuest is a demo of how this technology can be used today to create a real-time multiplayer game in a single webpage. When you start to play, your browser opens up a WebSocket connection to one of several load-balanced game servers. Each server hosts multiple world instances and handles the player synchronization and game logic within all instances. Because the server code is running on Node.js, both the server and client codebases share a small portion of the same JavaScript source code.
BrowserQuest makes extensive use of different web technologies, such as:
HTML5 Canvas, which powers the 2D tile-based graphics engine.
Web workers, allowing to initialize the large world map without slowing down the homepage UI.
localStorage, in which the progress of your character is continually saved.
CSS3 Media Queries, so that the game can resize itself and adapt to many devices.
HTML5 audio, so you can hear that rat or skeleton die!
Available everywhere
Since BrowserQuest is written in HTML5/JavaScript, it is available across a lot of different browsers and platforms. The game can be played in Firefox, Chrome and Safari. With WebSockets enabled, it’s also playable in Opera. Moreover, it’s compatible with iOS devices, as well as tablets and phones running Firefox for Android.
The mobile versions are more experimental than the desktop experience, which has richer features and performance, but it’s an early glimpse of what kind of games will be coming to the mobile Web in the future. Give it a try with your favorite mobile device!
Join the adventure
Want to be part of BrowserQuest? Create your own character and venture into the world. Fight enemies by yourself or with friends to get your hands on new equipment and items. You might even stumble upon a couple of surprises along the way…
We in Mozilla’s Developer Engagement team keep on reading, and as usual on Thursdays, we’d like to share some good links and read-worthy web sites with you!
Today we are featuring 
