Mozilla

Articles by Chris Heilmann

Sort by:

View:

  1. HTML5 audio and audio sprites – this should be simple

    As we’re having a HTML5 Audio developer derby this month, I thought it fun to play with audio again. And I found it sadly enough pretty frustrating.

    One thing I proposed in a lot of talks is using the idea of CSS sprites and apply them to HTML5 audio. You’ll get the same benefits – loading one file in one HTTP request instead of many, avoiding failure as files might not get loaded and so on.

    To test this out I wrote the following small demo using the awesome Music Non Stop by Kraftwerk.

    Clicking the different buttons should play the part of the music file and nothing more. This works fine in Firefox, Chrome and Opera on my computer here. Safari, however, fails to preload the audio and the setting of the current time is off. The code is simple enough that this should work:

    <div id="buttons"></div>
    <audio preload controls>
      <source src="boing-boomchack-peng.mp3" type="audio/mp3"></source>
      <source src="boing-boomchack-peng.ogg" type="audio/ogg"></source>
    </audio>
    // get the audio element and the buttons container
    // define a sprite object with the names and the start and end times 
    // of the different sounds.
    var a = document.querySelector('audio'),
        buttoncontainer = document.querySelector('#buttons'),
        audiosprite = {
          'all': [ 0, 5 ],
          'boing': [ 0, 1.3 ],
          'boomtchack': [ 2, 2.5 ],
          'peng': [ 4, 5 ]
        },
        end = 0;
     
    // when the audio data is loaded, create the buttons 
    // this way non-HTML5 browsers don't get any buttons 
    a.addEventListener('loadeddata', function(ev) {
      for (var i in audiosprite) {
        buttoncontainer.innerHTML += '<button onclick="play(\'' +
                                      i + '\')">' + i + '</button>';
      }
    }, false);
     
    // If the time of the file playing is updated, compare it 
    // to the current end time and stop playing when this one 
    // is reached
    a.addEventListener('timeupdate', function(ev) {
      if (a.currentTime > end) {
        a.pause();
      }
    },false);
     
    // Play the current audio sprite by setting the currentTime
    function play(sound) {
      if ( audiosprite[sound] ) {
        a.currentTime = audiosprite[sound][0];
        end = audiosprite[sound][1];
        a.play();
      }
    }

    Now, this is nothing new, Remy Sharp wrote about audio sprites in 2010 and lamented especially the buggy support in iOS (audio won’t load at all until you activate it with a touch – something that sounds horribly like the “click to active” Flash has on IE).

    Other issues are looping and latency of HTML5 audio. As reported by Robert O’Callahan there is a work-around by cloning the audio element before playing it (with an incredibly annoying test) and this fix has been used in the Gladius HTML5 game engine.

    All in all it seems HTML5 audio still needs a lot of work which is why a lot of Games released lately under the banner of HTML5 use Flash audio or no audio at all. This is sad and needs fixing.

    Interestingly enough there are some great projects that you could be part of. Are we playing yet? by Soundcloud and others for example is a test suite for audio support in browsers. You can write own tests on GitHub and report results to the browser makers.

    The jPlayer team has a great HTML5 Media Event Inspector showing just how many of the HTML5 media events are supported in your current browser.

    If you want to be safe, you can use SoundManager 2 by Scott Schiller to have an API that uses HTML5 when possible and falls back to Flash when the browser doesn’t have any support. It also fixes a few issues for you.

    Speaking of Scott Schiller, he continually gives good insight on the state of audio. There is a 51 minute video of his article on 24 ways “Probably, Maybe, No: The State of HTML5 Audio“.

    A shorter and more recent talk on the same subject is also available:

    All in all it would be interesting to hear what you think of the state of HTML5 audio:

    • Did the companies that heralded HTML5 as the end of plugins drop the ball?
    • Is it really sensible to have an API that returns probably or maybe or ” when you ask it if the browser can play a certain type of media?
    • What could be done to work around these issues?

    Let’s re-ignite the discussion on HTML5 audio, after all we need it for the future of messaging in the browser and telephony, too.

    Oh and another thing. Of course there is the Audio Data API of Firefox and the web audio proposal from Webkit available but getting those running in mobile devices will be a much bigger change. If you want to know more about those and libraries to work around their differences, there is a great overview post available on Happyworm.

  2. Porting “Me & My Shadow” to the Web – C++ to JavaScript/Canvas via Emscripten

    Editors note: This is a guest post by Alon Zakai of the Mozilla Emscripten team. Thanks Alon!

    Me & My Shadow is an open source 2D game, with clever gameplay in which you control not one character but two. I happened to hear about it recently when they released a 0.3 version:

    Since I’m looking for games to port to the web, I figured this was a good candidate. It was quite easy to port, here is the result: Me & My Shadow on the Web

    Me and my shadow

    You can also get the source on GitHub.

    The port was done automatically by compiling the original code to JavaScript using Emscripten, an open-source C++ to JavaScript compiler that uses LLVM. Using a compiler like this allows the game to just be compiled, instead of manually rewriting it in JavaScript, so the process can take almost no time.

    The compiled game works almost exactly like the desktop version does on the machines and browsers I’ve tested on. Interestingly, performance looks very good. In this case, it’s mainly because most of what the game does is blit images. It uses the cross-platform SDL API, which is a wrapper library for things like opening a window, getting input, loading images, rendering text, etc. (so it is exactly what a game like this needs). Emscripten supports SDL through native canvas calls, so when you compile a game that uses SDL into JavaScript, it will use Emscripten’s SDL implementation. That implementation implements SDL blit operations using drawImage calls and so forth, which browsers generally hardware accelerate these days, so the game runs as fast as it would natively.

    For example, if the C++ code has

    SDL_BlitSurface(sprite, NULL, screen, position)

    then that means to blit the entire bitmap represented by sprite into the screen, at a specific position. Emscripten’s SDL implementation does some translation of arguments, and then calls

    ctx.drawImage(src.canvas, sr.x, sr.y, sr.w, sr.h, dr.x, dr.y, sr.w, sr.h);

    which draws the sprite, contained in src.canvas, into the context representing the screen, at the correct position and size. In other words, the C++ code is translated automatically into code that uses native HTML canvas operations in an efficient manner.

    There are some caveats though. The main problem is browser support for necessary features, the main problems I ran into here are typed arrays and the Blob constructor:

    • Typed arrays are necessary to run compiled C++ code quickly and with maximum compatibility. Emscripten can compile code without them, but the result is slower and needs manual correction for compatibility. Thankfully, all browsers are getting typed arrays. Firefox, Chrome and Opera already have them, Safari was only missing FloatArray64 until recently I believe, and IE will get them in IE10.
    • The Blob constructor is necessary because this game uses Emscripten’s new compression option. It takes all the datafiles (150 or so), packs them into a single file, does LZMA on that, and then the game in the browser downloads that, decompresses, and splits it up. This makes the download much smaller (but does mean there is a short pause to decompress). The issue though is that we end up with data for each file in a typed array. It’s easy to use the BlobBuilder for images, but for audio, they need the mimetype set or they fail to decode, and only the Blob constructor supports that. It looks like only Firefox has the Blob constructor so far, I’ve been told on Twitter there might be a workaround for Chrome that I am hoping to hear more about. Not sure about other browsers. But, the game should still work, just without sound effects and music.

    Another caveat is that there is some unavoidable amount of manual porting necessary:

    JavaScript main loops must be written in an asynchronous way: A callback for each frame. Thankfully, games are usually written in a way that the main loop can easily be refactored into a function that does one iteration, and that was the case here. Then that function that does one main loop iteration is called each frame from JavaScript. However, there are other cases of synchronous code that are more annoying, for example fadeouts that happen when a menu item is picked are done synchronously (draw, SDL_Delay, draw, etc.). This same problem turned up when I ported Doom, I guess it’s a common code pattern. So I just disabled those fadeouts for now; if you do want them in a game you port, you’d need to refactor them to be asynchronous.

    Aside from that, everything pretty much just worked. (The sole exception was that this code fell prey to an LLVM LTO bug, but Rafael fixed it.) So in conclusion I would argue that there is no reason not to run games like these on the web: They are easy to port, and they run nice and fast.

  3. Launching Evangelism Reps – getting the army of awesome ready to take the stage

    Today the Developer Engagement Team has launched the Evangelism Reps program – a special interest group within ReMo. Each year, we get thousands of requests to send Mozilla speakers around the world to talk about HTML5, new web technologies, Mozilla’s mission, our projects, products and more. Now, we would love for you to join the effort and become a Mozilla speaker too!

    This program is open to paid staff and Mozilla Reps of all skill levels and capabilities. If you are a new speaker and have always wanted to represent Mozilla at events, you can take advantage of our advanced speaker training where you can learn from people like Christian Heilmann and Robert Nyman on how to give effective presentations and get access to their best practices. People who are veteran speakers can also benefit by having the tools and resources available to host events, prepare stunning screen casts and be mentors to new Evangelism Reps.

    We encourage all current Mozilla speakers to please join our Speaker Database even if you don’t join the Evangelism Reps program. This will help us know who you are and match you up with the speaking opportunities that fit you best.

    The Evangelism Reps program information is on the wiki: https://wiki.mozilla.org/ReMo/SIGs/Evangelism_Reps

    Please contact us at make-me-a-speaker@mozilla.com if you are interested in joining the team. We anticipate our first training to be held in May so stay tuned for this exciting opportunity.

  4. People of HTML5: Andrew Betts on building the FT.com HTML5 app

    HTML5 needs spokespeople to work. There are a lot of people out there who took on this role, and here at Mozilla we thought it is a good idea to introduce some of them to you with a series of interviews and short videos. The format is simple – we send the experts 10 questions to answer and then do a quick video interview to let them introduce themselves and ask for more detail on some of their answers.

    Andrew Betts Today we are featuring Andrew Betts, director of Assanka.

    We came across Andrew because of his talk “We’ve Got a Website for That – The FT Web App and the Future of the Mobile Web” at the BlackBerry Devcon in Amsterdam where he did a great job allowing us to peek under the hood of the Financial Times web app. You can reach Andrew on Twitter as @triblondon.

    The video interview

    You can see the video on any HTML5 enabled device here (courtesy of vid.ly).

    10 Questions for Andrew Betts

    1) You worked on the HTML5 application for the financial times. What made you go HTML5 and not native? How many people were involved at all?

    The FT has an agnostic approach when it comes to the formats and technologies we can use to display our content. The choice has to suit the content, the reader, and our business model. Before we launched the HTML5 app there was already an FT native app on iOS, and it received an Apple design award, but HTML5 offered several significant benefits, the most important of which was to allow us to maintain a direct relationship with the reader. Secondary benefits include a faster update process, an investment in future cross-platform compatibility (HTML5 is yet to fully realise the dream of write-once, run everywhere), and freedom from restrictions and rules that are (or may be in future) imposed by app store operators.

    2) What would you say was the easiest part and what was the thing that you had to spend the most time on to make it work?

    Interactions are the hardest thing to crack. Matching behaviour to user expectations is frighteningly complex, especially when you realise that instinctive expectations of swipe behaviour is actually quite varied and inconsistent. One example of this complexity is swiping horizontally from the bottom of a section page, which takes you to the top of the next one. If you then swipe back, do you end up where you were, or at the top? So dealing with interactions is both technically hard and architecturally challenging.

    Offline behaviour is also very tough. We get a lot of “User x can’t use the app offline” style support requests, and those are very hard to debug. We log every report but the priority is to figure out the scale of the problem.

    The easiest thing to deal with is probably layout. We send all legacy browsers to m.ft.com so we only have to cope with the most recent major versions, and that means that generally we get good standards support for CSS.

    3) How did you deal with the differences in interfaces? A mobile has much less space than a tablet and again less than a desktop. Are you using mediaqueries to give different experiences?

    We group devices into one of four sizes based on media queries, and we vary which resources we use slightly based on the reported DPI, but most of our responsive layout logic is done in JavaScript. We make a few assumptions right now to make development easier, such as that the user cannot change the size of the browser window, except by rotating the device, so we only need to change the layout on orientation change, not in response to arbitrary resizing of the window. We also currently assume the user has no mouse cursor, so there are no hover effects.

    4) How does input get into it? Is it simple enough to build touch interfaces in HTML5 or did you have to use some trickery?

    Quite a lot of trickery is involved, unfortunately. The browser has to tread a fine line between on the one hand offering a touch API to web developers and getting out of their way to let them use it, and on the other, dealing with the fact that almost all websites are still built for keyboard and mouse, so the browser needs to help the user deal with that. We developed a polyfill called Fastclick which makes click events fire as fast as touch events, and that certainly helps us provide an experience that is as snappy as using a native app.

    We also deal with swiping in a few quite distinctly different ways. Swiping between sections is an endless carousel, and we implement the tracking and sliding ourselves, by maintaining hidden page containers to the left and right of the current view and applying CSS 3D transforms. But paging through an article requires different scroll mechanics – it has a fixed length, and we have to lay out the whole article in one go to know how many pages there are. Pages that scroll vertically are a different challenge again, and there we use a combination of leaving the browser to do scrolling natively, and simulating it where we need slightly different behaviour.

    You’ll also notice that the pages don’t ‘bounce’ when you hit the top and bottom of the page, despite that being the normal behaviour in the iOS browser. We’ve gone to quite some lengths to ensure that the user experience mirrors an app and doesn’t feel like the device ‘coping’ with a site that hasn’t been designed to be viewed using a device with touch input.

    5) You said in your talk at Blackberry Devcon that you are using localStorage to cache your JavaScript libraries and some of the images. Why not use WebSQL or IndexDB for that? Isn’t the synchronous nature of localStorage slowing things down?

    No, actually localStorage is considerably faster than WebSQL (we are yet to use IndexedDB because it’s not supported on some of our target platforms). The time required for a lookup by key in localStorage is many times less than the equivalent SELECT statement on a WebSQL table. It’s true that it’s synchronous, but if the critical aspect affecting perceived performance is the time required to fetch data, which it often is, then localStorage offers a much faster response. it’s also typically reliable and relatively stable compared to the behaviour of WebSQL and particularly the HTML5 app cache.

    6) You also said that localStorage uses UTF-16 instead of UTF-8. This means you can store two ASCII characters in one UTF-16 one and thus double the storage capacity, right? Can you share some code? This would be incredible to use for everybody out there.

    Both localStorage and WebSQL use UTF-16 in webkit browsers, it’s one of many aspects of the implementation of these technologies that don’t make a lot of sense! We’re experimenting with some algorithms for compressing our data so we can make more efficient use of that storage, and one of those is to combine two characters into a single UTF-16 one. We’re not using that in production yet, but when we do we’re certainly hoping to open source it.

    There are simpler things to be done as well. A cursory poke and prod of WebSQL tables suggest that column names are stored along with the value in every cell, so some attempts to increase efficiency are as simple as using single character column names. Clearly we ought not to have to do all this, but it’s the price you pay for dealing with bleeding edge features in browsers.

    7) What would you consider the best start for someone when they want to build an HTML5 app? What are the first things to get sorted?

    Decide ahead of time what you’re trying to achieve. I could be a pedant and say that making your site HTML5 is really just a matter of changing the DOCTYPE, so an ‘HTML5 app’ is really whatever you define it to be, and having a clear idea of that is a good place to begin. Are you building something just for touch, or for keyboard and mouse as well? Does it need to work offline? Will it be crawlable by search spiders? Will it work with JavaScript disabled?

    There are some great resources like MDN, but we often find ourselves reading the HTML5 specs, and sometimes even the webkit source code to find out how something is actually implemented.

    8) Can native and web apps be the same thing?

    Technically, I guess not – strictly speaking a native app is complied code running directly on a platform, but the hybrid model seems to be getting ever more popular, and I often see apps which are thinly veiled web browsers. If you look at it from the user perspective and say, can a web app feel exactly like a native app, I don’t see why not. The challenges are greater because of the diversity of uses that web technologies are designed for, and the diversity of platforms on which they are used. And for uses such as gaming, native code is always likely to run faster and have deeper access to the OS.

    But for applications like publishing, which after all was the original purpose of the web, web technologies do provide support for most of the user experience concepts that we need.

    9) What about testing? How did you approach this? Do you have an array of hardware to play with?

    It’s a nightmare. We had a local electrician built us a charging station where we could store all our devices and keep them charged all the time. We have around 45 individual devices in our team (that’s around 4 times more devices than we have people). We’re constantly looking at ways of improving our testing process, and we keep revisiting automated, on device testing, but we’re not using it in our build cycle as yet. Right now we have a team of very dedicated testers who poke and prod devices all day.

    10) As someone who went through the process of building a big HTML5 app, what would you like to have from browser vendors to make your life easier?

    A blithe answer is ‘How long do you have’, but in practice we have to accept we’re using freshly minted technologies and there might be teething troubles. The main things on our shopping list are: more frequent and aggressive browser updates (especially on Android, where the browser is the problem child of the mobile web world), a better and more reliable app cache, and hardware acceleration of CSS transforms.

  5. Making the Dino roar – syncing audio and CSS transitions

    It started with Brian King setting up our Google+ page using this round MDN logo by John Slater. I thought this looks cool and reminded me of the famous MGM intro so I wondered if I could turn it into an intro for our video tutorials (not sure if we will do that though). And, some photoshop and sound work later and with a sprinkle of HTML5 audio and CSS transitions, here we are (source on GitHub):

    I started with the sound. If you need Creative Commons licensed sounds, Freesound is a good resource. So I took Chinese Fanfare by Nick-Nack and Roar by CGEffex and put them together in Audacity.

    Saving them as OGG and MP3 gave me an audio element that I could tie into. All I needed was to listen to the timeupdate event and compare the currentTime to trigger the animations. The animations (rotation of the dino and opening and closing of the jaw) are CSS transitions triggered by classes on the parent element. The main trick was to store both the dino and the jaw inside a div and transition them separately. The jaw animation also needed a change in transformation origin as we don’t rotate the image around its center.

    If you got seven minutes to spare, here is a blow-by-blow screencast explaining what is going on:

  6. Getting you started for the CSS 3D transform Dev Derby (15 minute screencast)

    This month’s Mozilla Dev Derby is about CSS 3D transformations and as a reminder and inspiration we thought it a good idea to give you a walk-through of a simple demo: a rotating cube with content on each side. That’s simple? Yes it is! Check the video to see the result:

    You can also see it in action and grab the source on GitHub. So here is a 15 minute rough and ready walkthrough of what is going on in this code:

    I hope that got you inspired a bit and we are looking forward to your Dev Derby contributions.

    Read more:

  7. There is no simple solution for local storage

    TL;DR: we have to stop advocating localStorage as a great opportunity for storing data as it performs badly. Sadly enough the alternatives are not nearly as supported or simple to implement.

    When it comes to web development you will always encounter things that sound too good to be true. Sometimes they are good, and all that stops us from using them is our notion of being conspicuous about *everything* as developers. In a lot of cases, however, they really are not as good as they seem but we only find out after using them for a while that we are actually “doing it wrong”.

    One such case is local storage. There is a storage specification (falsely attributed to HTML5 in a lot of examples) with an incredibly simple API that was heralded as the cookie killer when it came out. All you have to do to store content on the user’s machine is to access the navigator.localStorage (or sessionStorage if you don’t need the data to be stored longer than the current browser session):

    localStorage.setItem( 'outofsight', 'my data' );
    console.log( localStorage.getItem( 'outofsight' ) ); // -> 'my data'

    This local storage solution has a few very tempting features for web developers:

    • It is dead simple
    • It uses strings for storage instead of complex databases (and you can store more complex data using JSON encoding)
    • It is well supported by browsers
    • It is endorsed by a lot of companies (and was heralded as amazing when iPhones came out)

    A few known issues with it are that there is no clean way to detect when you reach the limit of local storage and there is no cross-browser way to ask for more space. There are also more obscure issues around sessions and HTTPS, but that is just the tip of the iceberg.

    The main issue: terrible performance

    LocalStorage also has a lot of drawbacks that aren’t quite documented and certainly not covered as much in “HTML5 tutorials”. Especially performance oriented developers are very much against its use.

    When we covered localStorage a few weeks ago using it to store images and files in localStorage it kicked off a massive thread of comments and an even longer internal mailing list thread about the evils of localStorage. The main issues are:

    • localStorage is synchronous in nature, meaning when it loads it can block the main document from rendering
    • localStorage does file I/O meaning it writes to your hard drive, which can take long depending on what your system does (indexing, virus scanning…)
    • On a developer machine these issues can look deceptively minor as the operating system cached these requests – for an end user on the web they could mean a few seconds of waiting during which the web site stalls
    • In order to appear snappy, web browsers load the data into memory on the first request – which could mean a lot of memory use if lots of tabs do it
    • localStorage is persistent. If you don’t use a service or never visit a web site again, the data is still loaded when you start the browser

    This is covered in detail in a follow-up blog post by Taras Glek of the Mozilla performance team and also by Andrea Giammarchi of Nokia.

    In essence this means that a lot of articles saying you can use localStorage for better performance are just wrong.

    Alternatives

    Of course, browsers always offered ways to store local data, some you probably never heard of as shown by evercookie (I think my fave when it comes to the “evil genius with no real-world use” factor is the force-cached PNG image to be read out in canvas). In the internal discussions there was a massive thrust towards advocating IndexedDB for your solutions instead of localStorage. We then published an article how to store images and files in IndexedDB and found a few issues – most actually related to ease-of-use and user interaction:

    • IndexedDB is a full-fledged DB that requires all the steps a SQL DB needs to read and write data – there is no simple key/value layer like localStorage available
    • IndexedDB asks the user for permission to store data which can spook them
    • The browser support is not at all the same as localStorage, right now IndexedDB is supported in IE10, Firefox and Chrome and there are differences in their implementations
    • Safari, Opera, iOS, Opera Mobile, Android Browser favour WebSQL instead (which is yet another standard that has been officially deprecated by the W3C)

    As always when there are differences in implementation someone will come up with an abstraction layer to work around that. Parashuram Narasimhan does a great job with that – even providing a jQuery plugin. It feels wrong though that we as implementers have to use these. It is the HTML5 video debate of WebM vs. H264 all over again.

    Now what?

    There is no doubt that the real database solutions and their asynchronous nature are the better option in terms of performance. They are also more matured and don’t have the “shortcut hack” feeling of localStorage. On the other hand they are hard to use in comparison, we already have a lot of solutions out there using localStorage and asking the user to give us access to storing local files is unacceptable for some implementations in terms of UX.

    The answer is that there is no simple solution for storing data on the end users’ machines and we should stop advocating localStorage as a performance boost. What we have to find is a solution that makes everybody happy and doesn’t break the current implementations. This might prove hard to work around. Here are some ideas:

    • Build a polyfill library that overrides the localStorage API and stores the content in IndexedDB/WebSQL instead? This is dirty and doesn’t work around the issue of the user being asked for permission
    • Implement localStorage in an asynchronous fashion in browsers – actively disregarding the spec? (this could set a dangerous precedent though)
    • Change the localStorage spec to store asynchronously instead of synchronously? We could also extend it to have a proper getStorageSpace interface and allow for native JSON support
    • Define a new standard that allows browser vendors to map the new API to the existing supported API that matches the best for the use case?

    We need to fix this as it doesn’t make sense to store things locally and sacrifice performance at the same time. This is a great example of how new web standards give us much more power but also make us face issues we didn’t have to deal with before. With more access to the OS, we also have to tread more carefully.

  8. Presentation: HTML5 and friends at Mobile World Congress 2012

    As part of the WIP Jam at the Mobile World Congress in Barcelona, Spain we were asked to give and introduction to HTML5 so we sent Chris Heilmann and Joe Stagner to set the record straight in terms of what HTML5 is and what you can do with it.

    The presentation was on the last day of the conference and both of us had spent long hours explaining all Mozilla technologies on our booth and really not enough hours sleeping in between. In addition to that the room was a boiling 30 degrees and packed with lots of developers. It seems we did a good job regardless as nobody left and we had a lot of great questions afterwards.

    The slides (well, the HTML5 document) are available for you to see:

    Slides for HTML5 and friends

    However, they make much more sense when you see them in context, which is why we also provide you with a screencast of the talk:

    All in all it was a lot of fun talking to people at MWC and Boot to Gecko showed that HTML5 is the future of mobile as much as it is already the present technology of choice on the desktop.

  9. Friday fun: Trigger Rally in WebGL

    As reported by Creative JS, Jasmine Kent ported her open source Linux racing game Trigger Rally to WebGL to run in HTML5 browsers. In the following video you can see it running pretty smoothly on my MacBook air that is already running 12% of CPU without it starting (as ScreenFlow is recording and VLC was playing music whilst TweetDeck used quite a chunk, too):

    If you are the artistic type, you can even design your own levels for the game.

    You can play play Trigger Rally online here: http://triggerrally.com/ or follow it on Twitter @triggerrally.