Mozilla

Articles

Sort by:

View:

  1. Video, Mobile, and the Open Web

    [Also posted at brendaneich.com.]

    I wrote The Open Web and Its Adversaries just over five years ago, based on the first SXSW Browser Wars panel (we just had our fifth, it was great — thanks to all who came).

    Some history

    The little slideshow I presented is in part quaint. WPF/E and Adobe Apollo, remember those? (Either the code names, or the extant renamed products?) The Web has come a long way since 2007.

    But other parts of my slideshow are still relevant, in particular the part where Mozilla and Opera committed to an unencumbered <video> element for HTML5:

    • Working with Opera via WHATWG on <video>
      • Unencumbered Ogg Theora decoder in all browsers
      • Ogg Vorbis for <audio>
      • Other formats possible
      • DHTML player controls

    We did what we said we would. We fought against the odds. We carried the unencumbered HTML5 <video> torch even when it burned our hands.

    We were called naive (no) idealists (yes). We were told that we were rolling a large stone up a tall hill (and how!). We were told that we could never overcome the momentum behind H.264 (possibly true, but Mozilla was not about to give up and pay off the patent rentiers).

    Then in 2009 Google announced that it would acquire On2 (completed in 2010), and Opera and Mozilla had a White Knight.

    At Google I/O in May 2010, Adobe announced that it would include VP8 (but not all of WebM?) support in an upcoming Flash release.

    On January 11, 2011, Mike Jazayeri of Google blogged:

    … we are changing Chrome’s HTML5 <video> support to make it consistent with the codecs already supported by the open Chromium project. Specifically, we are supporting the WebM (VP8) and Theora video codecs, and will consider adding support for other high-quality open codecs in the future. Though H.264 plays an important role in video, as our goal is to enable open innovation, support for the codec will be removed and our resources directed towards completely open codec technologies.

    These changes will occur in the next couple months….

    A followup post three days later confirmed that Chrome would rely on Flash fallback to play H.264 video.

    Where we are today

    It is now March 2012 and the changes promised by Google and Adobe have not been made.

    What’s more, any such changes are irrelevant if made only on desktop Chrome — not on Google’s mobile browsers for Android — because authors typically do not encode twice (once in H.264, once in WebM), they instead write Flash fallback in an <object> tag nested inside the <video> tag. Here’s an example adapted from an Opera developer document:

    <video controls poster="video.jpg" width="854" height="480">
     <source src="video.mp4" type="video/mp4">
     <object type="application/x-shockwave-flash" data="player.swf"
             width="854" height="504">
      <param name="allowfullscreen" value="true">
      <param name="allowscriptaccess" value="always">
      <param name="flashvars" value="file=video.mp4">
      <!--[if IE]><param name="movie" value="player.swf"><![endif]-->
      <img src="video.jpg" width="854" height="480" alt="Video">
      <p>Your browser can't play HTML5 video.
     </object>
    </video>
    

    The Opera doc nicely carried the unencumbered video torch by including

     <source src="video.webm" type="video/webm">
    

    after the first <source> child in the <video> container (after the first, because of an iOS WebKit bug, the Opera doc said), but most authors do not encode twice and host two versions of their video (yes, you who do are to be commended; please don’t spam my blog with comments, you’re not typical — and YouTube is neither typical nor yet completely transcoded [1]).

    Of course the ultimate fallback content could be a link to a video to download and view in a helper app, but that’s not “HTML5 video” and it is user-hostile (profoundly so on mobile). Flash fallback does manage to blend in with HTML5, modulo the loss of expressiveness afforded by DHTML playback controls.

    Now, consider carefully where we are today.

    Firefox supports only unencumbered formats from Gecko’s <video> implementation. We rely on Flash fallback that authors invariably write, as shown above. Let that sink in: we, Mozilla, rely on Flash to implement H.264 for Firefox users.

    Adobe has announced that it will not develop Flash on mobile devices.

    In spite of the early 2011 Google blog post, desktop Chrome still supports H.264 from <video>. Even if it were to drop that support, desktop Chrome has a custom patched Flash embedding, so the fallback shown above will work well for almost all users.

    Mobile matters most

    Android stock browsers (all Android versions), and Chrome on Android 4, all support H.264 from <video>. Given the devices that Android has targeted over its existence, where H.264 hardware decoding is by far the most power-efficient way to decode, how could this be otherwise? Google has to compete with Apple on mobile.

    Steve Jobs may have dealt the death-blow to Flash on mobile, but he also championed and invested in H.264, and asserted that “[a]ll video codecs are covered by patents”. Apple sells a lot of H.264-supporting hardware. That hardware in general, and specifically in video playback quality, is the gold standard.

    Google is in my opinion not going to ship mobile browsers this year or next that fail to play H.264 content that Apple plays perfectly. Whatever happens in the very long run, Mozilla can’t wait for such an event. Don’t ask Google why they bought On2 but failed to push WebM to the exclusion of H.264 on Android. The question answers itself.

    So even if desktop Chrome drops H.264 support, Chrome users almost to a person won’t notice, thanks to Flash fallback. And Apple and Google, along with Microsoft and whomever else might try to gain mobile market share, will continue to ship H.264 support on all their mobile OSes and devices — hardware-implemented H.264, because that uses far less battery than alternative decoders.

    Here is a chart of H.264 video in HTML5 content on the Web from MeFeedia:

    MeFeedia.com, December 2011

    And here are some charts showing the rise of mobile over desktop from The Economist:

    The Economist, October 2011

    These charts show Bell’s Law of Computer Classes in action. Bell’s Law predicts that the new class of computing devices will replace older ones.

    In the face of this shift, Mozilla must advance its mission to serve users above all other agendas, and to keep the Web — including the “Mobile Web” — open, interoperable, and evolving.

    What Mozilla is doing

    We have successfully launched Boot to Gecko (B2G) and we’re preparing to release a new and improved Firefox for Android, to carry our mission to mobile users.

    What should we do about H.264?

    Andreas Gal proposes to use OS- and hardware-based H.264 decoding capabilities on Android and B2G. That thread has run to over 240 messages, and spawned some online media coverage.

    Some say we should hold out longer for someone (Google? Adobe?) to change something to advance WebM over H.264.

    MozillaMemes.tumblr.com/post/19415247873

    Remember, dropping H.264 from <video> only on desktop and not on mobile doesn’t matter, because of Flash fallback.

    Others say we should hold out indefinitely and by ourselves, rather than integrate OS decoders for encumbered video.

    I’ve heard people blame software patents. I hate software patents too, but software isn’t even the issue on mobile. Fairly dedicated DSP hardware takes in bits and puts out pixels. H.264 decoding lives completely in hardware now.

    Yes, some hardware also supports WebM decoding, or will soon. Too little, too late for HTML5 <video> as deployed and consumed this year or (for shipping devices) next.

    As I wrote in the newsgroup thread, Mozilla has never ignored users or market share. We do not care only about market share, but ignoring usability and market share can easily lead to extinction. Without users our mission is meaningless and our ability to affect the evolution of open standards goes to zero.

    Clearly we have principles that prohibit us from abusing users for any end (e.g., by putting ads in Firefox’s user interface to make money to sustain ourselves). But we have never rejected encumbered formats handled by plugins, and OS-dependent H.264 decoding is not different in kind from Flash-dependent H.264 decoding in my view.

    We will not require anyone to pay for Firefox. We will not burden our downstream source redistributors with royalty fees. We may have to continue to fall back on Flash on some desktop OSes. I’ll write more when I know more about desktop H.264, specifically on Windows XP.

    What I do know for certain is this: H.264 is absolutely required right now to compete on mobile. I do not believe that we can reject H.264 content in Firefox on Android or in B2G and survive the shift to mobile.

    Losing a battle is a bitter experience. I won’t sugar-coat this pill. But we must swallow it if we are to succeed in our mobile initiatives. Failure on mobile is too likely to consign Mozilla to decline and irrelevance. So I am fully in favor of Andreas’s proposal.

    Our mission continues

    Our mission, to promote openness, innovation, and opportunity on the Web, matters more than ever. As I said at SXSW in 2007, it obligates us to develop and promote unencumbered video. We lost one battle, but the war goes on. We will always push for open, unencumbered standards first and foremost.

    In particular we must fight to keep WebRTC unencumbered. Mozilla and Opera also lost the earlier skirmish to mandate an unencumbered default format for HTML5 <video>, but WebRTC is a new front in the long war for an open and unencumbered Web.

    We are researching downloadable JS decoders via Broadway.js, but fully utilizing parallel and dedicated hardware from JS for battery-friendly decoding is a ways off.

    Can we win the long war? I don’t know if we’ll see a final victory, but we must fight on. Patents expire (remember the LZW patent?). They can be invalidated. (Netscape paid to do this to certain obnoxious patents, based on prior art.) They can be worked around. And patent law can be reformed.

    Mozilla is here for the long haul. We will never give up, never surrender.

    /be

    [1] Some points about WebM on YouTube vs. H.264:

    • Google has at best transcoded only about half the videos into WebM. E.g., this YouTube search for “cat” gives ~1.8M results, while the same one for WebM videos gives 704K results.
    • WebM on YouTube is presented only for videos that lack ads, which is a shrinking number on YouTube. Anything monetizable (i.e., popular) has ads and therefore is served as H.264.
    • All this is moot when you consider mobile, since there is no Flash on mobile, and as of yet no WebM hardware, and Apple’s market-leading position.

  2. Hello Chrome, it's Firefox calling!

    Mozilla is excited to announce that we’ve achieved a major milestone in WebRTC development: WebRTC RTCPeerConnection interoperability between Firefox and Chrome. This effort was made possible because of the close collaboration between the open Web community and engineers from both Mozilla and Google.

    RTCPeerConnection (also known simply as PeerConnection or PC) interoperability means that developers can now create Firefox WebRTC applications that make direct audio/video calls to Chrome WebRTC applications without having to install a third-party plugin. Because the functionality is now baked into the browser, users can avoid problems with first-time installs and buggy plugins, and developers can deploy their apps much more easily and universally.

    To help celebrate this momentous milestone, we thought it would be fun to call up our friends at Google to discuss it with them. Check out this Firefox-Chrome demonstration call between Mozilla’s Chief Innovation Officer, Todd Simpson, and Google’s Director of Product Management, Hugh Finnan, and read what Google had to say about this momentous occasion in their blog post.

    This milestone builds on an earlier demo we showed late last year of WebRTC integrated with Social API. There we demonstrated an industry first with our implementation of DataChannels, a powerful component of WebRTC that can combined with an audio/video chat to allow users to share almost anything on their computer or device. Send vacation photos, memorable videos, links news stories etc., simply by dragging the item into your video chat window. Look out for more on this to come.

    The purpose of WebRTC, an open standard being defined jointly at the W3C and IETF standards organizations, is to provide a common platform for all user devices to communicate and share audio, video and data in real-time. This is a first step toward that vision of interoperability and true, open, real-time communication on the web.

    Posted by:
    Serge Lachapelle, Chrome Product Manager and Maire Reavy, Firefox Media Product Lead

    Start Developing Using RTCPeerConnection in Firefox

    For JavaScript developers who haven’t tried RTCPeerConnection in Firefox yet (since it is a brand new feature for us), you can try this out using the most recent Firefox Nightly by setting the media.peerconnection.enabled pref to “true” (browse to about:config and search for the media.peerconnection.enabled pref in the list of prefs). Here is a snippet of code from a sample app that shows off how to initiate, accept, and end a WebRTC call in Firefox using RTCPeerConnection:

    function initiateCall(user) {
      document.getElementById("main").style.display = "none";
      document.getElementById("call").style.display = "block";
     
      // Here's where you ask user permission to access the camera and microphone streams
      navigator.mozGetUserMedia({video:true, audio:true}, function(stream) {
        document.getElementById("localvideo").mozSrcObject = stream;
        document.getElementById("localvideo").play();
        document.getElementById("localvideo").muted = true;
     
        // Here's where you set up a Firefox PeerConnection
        var pc = new mozRTCPeerConnection();
        pc.addStream(stream);
     
        pc.onaddstream = function(obj) {
          log("Got onaddstream of type " + obj.type);
          document.getElementById("remotevideo").mozSrcObject = obj.stream;
          document.getElementById("remotevideo").play();
          document.getElementById("dialing").style.display = "none";
          document.getElementById("hangup").style.display = "block";
        };
     
        pc.createOffer(function(offer) {
          log("Created offer" + JSON.stringify(offer));
          pc.setLocalDescription(offer, function() {
            // Send offer to remote end.
            log("setLocalDescription, sending to remote");
            peerc = pc;
            jQuery.post(
              "offer", {
                to: user,
                from: document.getElementById("user").innerHTML,
                offer: JSON.stringify(offer)
              },
              function() { console.log("Offer sent!"); }
            ).error(error);
          }, error);
        }, error);
      }, error);
    }
     
    function acceptCall(offer) {
      log("Incoming call with offer " + offer);
      document.getElementById("main").style.display = "none";
      document.getElementById("call").style.display = "block";
     
      // Here's where you ask user permission to access the camera and microphone streams
      navigator.mozGetUserMedia({video:true, audio:true}, function(stream) {
        document.getElementById("localvideo").mozSrcObject = stream;
        document.getElementById("localvideo").play();
        document.getElementById("localvideo").muted = true;
     
        // Here's where you set up a Firefox PeerConnection
        var pc = new mozRTCPeerConnection();
        pc.addStream(stream);
     
        pc.onaddstream = function(obj) {
          document.getElementById("remotevideo").mozSrcObject = obj.stream;
          document.getElementById("remotevideo").play();
          document.getElementById("dialing").style.display = "none";
          document.getElementById("hangup").style.display = "block";
        };
     
        pc.setRemoteDescription(JSON.parse(offer.offer), function() {
          log("setRemoteDescription, creating answer");
          pc.createAnswer(function(answer) {
            pc.setLocalDescription(answer, function() {
              // Send answer to remote end.
              log("created Answer and setLocalDescription " + JSON.stringify(answer));
              peerc = pc;
              jQuery.post(
                "answer", {
                  to: offer.from,
                  from: offer.to,
                  answer: JSON.stringify(answer)
                },
                function() { console.log("Answer sent!"); }
              ).error(error);
            }, error);
          }, error);
        }, error);
      }, error);
    }
     
    function endCall() {
      log("Ending call");
      document.getElementById("call").style.display = "none";
      document.getElementById("main").style.display = "block";
     
      document.getElementById("localvideo").mozSrcObject.stop();
      document.getElementById("localvideo").mozSrcObject = null;
      document.getElementById("remotevideo").mozSrcObject = null;
     
      peerc.close();
      peerc = null;
    }

    You’ll notice that Firefox still prefixes the RTCPeerConnection API call as mozRTCPeerConnection because the standards committee is not yet done defining it. Chrome prefixes it as webkitRTCPeerConnection. Once the standards committee finishes its work, we will remove the prefixes and use the same API, but in the meantime, you’ll want to support both prefixes so that your app works in both browsers.

    Trying Interop Yourself

    For those eager to give interop a try, here are instructions and information about “trying this at home”.

    This is Firefox’s and Chrome’s first version of PeerConnection interoperability. As with most early releases, there are still bugs to fix, and interop isn’t supported yet in every network environment. But this is a major step forward for this new web feature and for the Web itself. We thank the standards groups and every contributor to the WebRTC community. While there’s more work to do, we hope you’ll agree that the Web is about to get a lot more awesome.

  3. Upgrading your ZTE Open to Firefox 1.1 or 1.2 (fastboot enabled)

    FoxLove

    If you are a ZTE Open owner, you may have already upgraded to Firefox OS 1.1 following the instructions from our previous post. If so, you probably realized that the latest build from ZTE had a problem: fastboot wasn’t enabled anymore. For those of you who didn’t upgrade because of that, ZTE has put a new build of 1.1 with fastboot enabled on their site.

    Depending on the version of the phone you purchased, you need either the US or the UK (European) version of the build. Download the files from the ZTE support site by clicking the “Downloads” tab on one of these pages: US version or UK version. The zip file you’ll download will also contain documentation for the upgrade, or you can follow our earlier instructions.

    Upgrading to Firefox 1.2

    We are pleased to announce that ZTE has also made available a version of Firefox OS 1.2. If you want to upgrade to it, you’ll first need to install the version of 1.1 that has fastboot enabled (or be on an older version of the OS with fastboot enabled). Next, you need to verify that you can establish a connection with the phone via USB. This post describes how to configure Windows, Linux and Mac machines for a USB connection (something you’ll want to do anyway if you are pushing apps to your phone during development).

    Finally, you’ll need to have have fastboot from the Android Developer Toolkit installed on your desktop machine. It is not necessary to install the entire toolkit. adb and fastboot are found in the /platform-tools/ folder. They can be copied to /usr/bin of your Linux or Mac OS X machine, or copied to another folder as long as that folder is added to your $PATH.

    After your phone and your desktop computer are properly configured, connect your phone to your computer via USB cable and try to restart your device with this command from the console (in Windows, open a command prompt window):

    fastboot reboot

    If your phone reboots itself, you are good to go for the upgrade. Download the appropriate version of the build from the Dropbox account ZTE has set up: US version or UK version. For Windows users, you can also download special instructions, and an upgrade tool to help you to install the new version easily. In this post, however, I provide steps for all operating systems–Linux, OS X, and Windows–without relying on the special tool.

    Once you have the file, extract the contents, and open a console. Note that those steps will erase your personal data, so be sure to make a backup first. Then navigate to the folder where the files are. Still in the console, type this command:

    adb reboot bootloader

    Wait for your phone to reboot, and while it is rebooting, execute those commands:

    fastboot flash boot boot.img
    fastboot flash userdata userdata.img
    fastboot flash system system.img
    fastboot flash recovery recovery.img
    fastboot erase cache
    fastboot reboot
    ZTE Open upgraded to Firefox OS 1.2

    The software version you see after upgrading

    If everything went well, your phone should reboot at this point. You’ll find that this version of 1.2 from ZTE includes many test applications, which you can remove if you like.

    Congratulations, your phone is now running Firefox OS 1.2! You will benefit from all the bug fixes, new features for users, and new features for developers of the latest released build of Firefox OS.

    If you have any questions related to the upgrade, I invite you to submit them on our StackOverflow Q&A. You’ll benefit from the expertise of thousands of people, including our Technical Evangelist team.

    2014-02-05 Updates

    We updated the process as some users had booting issues after upgrading to 1.2. Some users also reported us being able to execute the command, even if the fastboot test (fastboot restart) wasn’t working: I can’t confirm, but please try, and let us know if it’s working.

  4. HTML5 mythbusting

    The ongoing discussion about the “readiness” of HTML5 is based on a lot of false assumptions. These lead to myths about HTML5 that get uttered once and then continuously repeated – a lot of times without checking their validity at all.

    HTML5 doesn’t perform?

    The big thing everybody who wants to talk about the problems with HTML5 is performance. The main problem here is that almost every single comparison misses the fact that you are comparing apples and pears (no pun intended).

    Comparing an HTML5 application’s performance with a native App is comparing a tailored suit with one bought in a shop. Of course the tailored suit will fit you like a glove and looks amazing but if you ever want to sell it or hand it over to someone else you are out of luck. It will not be the same for the next person.

    IMG_4197.jpg

    That is what native Apps are – they are built and optimized for one single environment and purpose and are fixed in their state – more on that later.

    HTML5, on the other hand by its very definition is a web technology that should run independent of environment, display or technology. It has to be as flexible as possible in order to be a success on the web. In its very definition the web is for everybody, not just for a small group of lucky people who can afford a very expensive piece of hardware and are happy to get locked into a fixed environment governed by a single company.

    Native applications need to be written for every single device and every new platform from scratch whereas an HTML5 App allows you to support mobiles, tablets and desktops with the same product. Instead of having fixed dimensions and functionality an HTML5 App can test what is supported and improve the experience for people on faster and newer devices whilst not locking out others that can not buy yet another phone.

    Native Apps on the other hand do in a lot of cases need an upgrade and force the end user to buy new hardware or they’ll not get the product at all. From a flexibility point of view, HTML5 Apps perform admirably whilst native applications make you dependent on your hardware and leave you stranded when there is an upgrade you can’t afford or don’t want to make. A great example of this is the current switch from Apple to their own maps on iOS. Many end users are unhappy and would prefer to keep using Google Maps but can not.


    HexGL – a WebGL powered racing game

    Seeing that HTML5 is perfectly capable on Desktop to exceed in performance, from scrolling performance to analyzing and changing video on the fly up to running full 3D games at a very high frame rate and have high speed racing games we have to ask ourselves where the problem with its performance lies.

    The answer is hardware access. HTML5 applications are treated by mobile hardware developed for iOS and Android as second class citizens and don’t get access to the parts that allow for peak performance. A web view in iOS is hindered by the operating system to perform as fast as a native App although it uses the same principles. On Android both Chrome and Firefox show how fast browsers can perform whereas the stock browser crawls along in comparison.

    The stock browser on Android reminds us of the Internet Explorer of the 90s which threatened to be set in stone for a long time and hinder the world wide web from evolving – the very reason Mozilla and Firefox came into existence.

    In essence HTML5 is a Formula 1 car that has to drive on a dirt road whilst dragging a lot of extra payload given to it by the operating system without a chance to work around that – for now.

    HTML5 can not be monetized?

    HTML5 is a technology stack based on open web technologies. Saying that HTML5 has no monetization model is like saying the web can not be monetized (which is especially ironic when this is written on news sites that show ads).

    Whilst on the first glance a closed App-market is a simple way to sell your products there is a lot of hype about their success and in reality not many developers manage to make a living with a single app on closed App markets. As discovery and find-ability is getting increasingly harder in App markets a lot of developers don’t build one App but hundreds of the same App (talking dog, talking cat, talking donkey…) as it is all about being found quickly and being on the first page of search results in the market.

    This is where closed App markets with native Apps are a real disadvantage for developers: Apps don’t have an address on the web (URL) and can not be found outside the market. You need to manually submit each of the Apps in each of the markets, abide to their review and submission process and can not update your App easily without suffering outages in your offering.

    An HTML5 App is on the web and has a URL, it can also get packaged up with products like Adobe PhoneGap to become a native application for iOS or Android. The other way around is not possible.

    In the long term that begs the question what is the better strategy for developers: betting on one closed environment that can pull your product any time it wants or distributing over a world-wide, open distribution network and cover the closed shops as well?

    Many apps in the Android and iOS store are actually HTML5 and got converted using PhoneGap. The biggest story about this was the Financial Times releasing their app as HTML5 and making a better profit than with the native one. And more recently the New York Times announced it was following suit with its Web app.

    HTML5 can not be offline?

    As HTML5 is a web technology stack the knee-jerk reaction is thinking that you would have to be online all the time to use them. This is plain wrong. There are many ways to store content offline in a HTML5 application. The simplest way is the Web Storage API which is supported across all modern browsers (excluding Opera mini which is a special case as it sends content via a cloud service and has its own storage tools). You can also store the application itself offline using AppCache which is supported by all but Internet Explorer. If you have more complex data to store than Web Storage provides you can use either IndexedDB (for Chrome and Firefox) or WebSQL (for iOS and Safari). To work around the issues there are libraries like Lawnchair available to make it easy for developers to use.

    HTML5 has no development environment?

    One concern often mentioned is that HTML5 lacks in tooling for developers. Strangely enough you never hear that argument from developers but from people who want to buy software to make their developers more effective instead of letting them decide what makes them effective.

    HTML5 development at its core is web development and there is a quite amazingly practical development environment for that available. Again, the main issue is a misunderstanding of the web. You do not build a product that looks and performs the same everywhere – this would rob the web of its core strengths. You build a product that works for everybody and excels on a target platform. Therefore your development environment is a set of tools, not a single one doing everything for you. Depending on what you build you choose to use many of them or just one.

    The very success of the web as a media is based on the fact that you do not need to be a developer to put content out – you can use a blogging platform, a CMS or even a simple text editor that comes with your operating system to start your first HTML page. As you progress in your career as a developer you find more and more tools you like and get comfortable and effective with but there is no one tool to rule them all. Some developers prefer IDEs like Visual Studio, or Eclipse. Others want a WYSIWYG style editor like Dreamweaver but the largest part of web developers will have a text editor or other of some sorts. From Sublime Text, Notepad++ up to VIM or emacs on a Linux computer, all of these are tools that can be used and are used by millions of developers daily to build web content.

    When it comes to debugging and testing web developers are lucky these days as the piece of software our end users have to see what we build – the browser – is also the debugging and testing environment. Starting with Firefox having Firebug as an add-on to see changes live and change things on the fly, followed by Opera’s Dragonfly and Safari and Chrome’s Devtools, all browsers now also have a lot of functionality that is there especially for developers. Firefox’s new developer tools go even further and instead of simply being a debugging environment are a set of tools in themselves that developers can extend to their needs.

    Remote debugging is another option we have now. This means we can as developers change applications running on a phone on our development computers instead of having to write them, send them to the phone, install them, test them, find a mistake and repeat. This speeds up development time significantly.


    Remote debugging in Firefox

    For the more visual developers Adobe lately released their Edge suite which brings WYSIWYG style development to HTML5, including drag and drop from Photoshop. Adobe’s Edge Inspect and PhoneGap makes it easy to test on several devices at once and send HTML5 Apps as packaged native Apps to iOS and Android.

    In terms of deployment and packaging Google just released their Yeoman project which makes it dead easy for web developers to package and deploy their web products as applications with all the necessary steps to make them perform well.

    All in all there is no fixed development environment for HTML5 as that would neuter the platform – this is the web, you can pick and choose what suits you most.

    Things HTML5 can do that native Apps can not

    In essence a lot of the myths of HTML5 are based on the fact that the comparison was between something explicitly built for the platform it was tested on versus something that is also supported on it. Like comparing the performance of speedboat and a hovercraft would result in the same predictable outcome. The more interesting question is what makes HTML5 great for developers and end users, that native applications can or do not do:

    • Write once, deploy anywhere – HTML5 can run in browsers, on tablets and desktops and you can convert it to native code to support iOS and Android. This is not possible the other way around.
    • Share over the web – as HTML5 apps have a URL they can be shared over the web and found when you search the web. You don’t need to go to a market place and find it amongst the crowded, limited space but the same tricks how to promote other web content apply. The more people like and link to your app, the easier it will be found.
    • Built on agreed, multi-vendor standards – HTML5 is a group effort of the companies that make the web what it is now, not a single vendor that can go into a direction you are not happy with
    • Millions of developers – everybody who built something for the web in the last years is ready to write apps. It is not a small, specialized community any longer
    • Consumption and development tool are the same thing – all you need to get started is a text editor and a browser
    • Small, atomic updates – if a native app needs an upgrade, the whole App needs to get downloaded again (new level of Angry Birds? Here are 23MB over your 3G connection). HTML5 apps can download data as needed and store it offline, thus making updates much less painful.
    • Simple functionality upgrade – native apps need to ask you for access to hardware when you install them and can not change later on which is why every app asks for access to everything upfront (which of course is a privacy/security risk). An HTML5 app can ask for access to hardware and data on demand without needing an update or re-installation.
    • Adaptation to the environment – an HTML5 app can use responsive design to give the best experience for the environment without having to change the code. You can switch from Desktop to mobile to tablet seamlessly without having to install a different App on each.

    Let’s see native Apps do that.

    Breaking the hardware lockout and making monetization easier

    The main reason why HTML5 is not the obvious choice for developers now is the above mentioned lockout when it comes to hardware. An iOS device does not allow different browser engines and does not allow HTML5 to access the camera, the address book, vibration, the phone or text messaging. In other words, everything that makes a mobile device interesting for developers and very necessary functionality for Apps.

    To work around this issue, Mozilla and a few others have created a set of APIs to define access to these in a standardized way called Web APIs. This allows every browser out there to get access to the hardware in a secure way and breaks the lockout.

    The first environment to implement these is the Firefox OS with devices being shipped next year. Using a Firefox OS phone you can build applications that have the same access to hardware native applications have. Developers have direct access to the hardware and thus can build much faster and – more importantly – much smaller Apps. For the end user the benefit is that the devices will be much cheaper and Firefox OS can run on very low specification hardware that can for example not be upgraded to the newest Android.

    In terms of monetization Mozilla is working on their own marketplace for HTML5 Apps which will not only allow HTML5 Apps to be submitted but also to be discovered on the web with a simple search. To make it easier for end users to buy applications we partner with mobile providers to allow for billing to the mobile contract. This allows end users without a credit card to also buy Apps and join the mobile web revolution.

    How far is HTML5?

    All in all HTML5 is going leaps and bounds to be a very interesting and reliable platform for app developers. The main barriers we have to remove is the hardware access and with the WebAPI work and systems like PhoneGap to get us access these are much less of a stopper than we anticipated.

    The benefits of HTML5 over native apps mentioned above should be reason enough for developers to get involved and start with HTML5 instead of spending their time building a different code base for each platform. If all you want to support is one special platform you don’t need to go that way, but then it is also pointless to blame HTML5 issues for your decision.

    HTML5 development is independent of platform and browser. If you don’t embrace that idea you limit its potential. Historically closed platforms came and went and the web is still going strong and allows you to reach millions of users world-wide and allows you to start developing without asking anyone for permission or having to install a complex development environment. This was and is the main reason why people start working with the web. And nobody is locked out, so have a go.

  5. There is no simple solution for local storage

    TL;DR: we have to stop advocating localStorage as a great opportunity for storing data as it performs badly. Sadly enough the alternatives are not nearly as supported or simple to implement.

    When it comes to web development you will always encounter things that sound too good to be true. Sometimes they are good, and all that stops us from using them is our notion of being conspicuous about *everything* as developers. In a lot of cases, however, they really are not as good as they seem but we only find out after using them for a while that we are actually “doing it wrong”.

    One such case is local storage. There is a storage specification (falsely attributed to HTML5 in a lot of examples) with an incredibly simple API that was heralded as the cookie killer when it came out. All you have to do to store content on the user’s machine is to access the navigator.localStorage (or sessionStorage if you don’t need the data to be stored longer than the current browser session):

    localStorage.setItem( 'outofsight', 'my data' );
    console.log( localStorage.getItem( 'outofsight' ) ); // -> 'my data'

    This local storage solution has a few very tempting features for web developers:

    • It is dead simple
    • It uses strings for storage instead of complex databases (and you can store more complex data using JSON encoding)
    • It is well supported by browsers
    • It is endorsed by a lot of companies (and was heralded as amazing when iPhones came out)

    A few known issues with it are that there is no clean way to detect when you reach the limit of local storage and there is no cross-browser way to ask for more space. There are also more obscure issues around sessions and HTTPS, but that is just the tip of the iceberg.

    The main issue: terrible performance

    LocalStorage also has a lot of drawbacks that aren’t quite documented and certainly not covered as much in “HTML5 tutorials”. Especially performance oriented developers are very much against its use.

    When we covered localStorage a few weeks ago using it to store images and files in localStorage it kicked off a massive thread of comments and an even longer internal mailing list thread about the evils of localStorage. The main issues are:

    • localStorage is synchronous in nature, meaning when it loads it can block the main document from rendering
    • localStorage does file I/O meaning it writes to your hard drive, which can take long depending on what your system does (indexing, virus scanning…)
    • On a developer machine these issues can look deceptively minor as the operating system cached these requests – for an end user on the web they could mean a few seconds of waiting during which the web site stalls
    • In order to appear snappy, web browsers load the data into memory on the first request – which could mean a lot of memory use if lots of tabs do it
    • localStorage is persistent. If you don’t use a service or never visit a web site again, the data is still loaded when you start the browser

    This is covered in detail in a follow-up blog post by Taras Glek of the Mozilla performance team and also by Andrea Giammarchi of Nokia.

    In essence this means that a lot of articles saying you can use localStorage for better performance are just wrong.

    Alternatives

    Of course, browsers always offered ways to store local data, some you probably never heard of as shown by evercookie (I think my fave when it comes to the “evil genius with no real-world use” factor is the force-cached PNG image to be read out in canvas). In the internal discussions there was a massive thrust towards advocating IndexedDB for your solutions instead of localStorage. We then published an article how to store images and files in IndexedDB and found a few issues – most actually related to ease-of-use and user interaction:

    • IndexedDB is a full-fledged DB that requires all the steps a SQL DB needs to read and write data – there is no simple key/value layer like localStorage available
    • IndexedDB asks the user for permission to store data which can spook them
    • The browser support is not at all the same as localStorage, right now IndexedDB is supported in IE10, Firefox and Chrome and there are differences in their implementations
    • Safari, Opera, iOS, Opera Mobile, Android Browser favour WebSQL instead (which is yet another standard that has been officially deprecated by the W3C)

    As always when there are differences in implementation someone will come up with an abstraction layer to work around that. Parashuram Narasimhan does a great job with that – even providing a jQuery plugin. It feels wrong though that we as implementers have to use these. It is the HTML5 video debate of WebM vs. H264 all over again.

    Now what?

    There is no doubt that the real database solutions and their asynchronous nature are the better option in terms of performance. They are also more matured and don’t have the “shortcut hack” feeling of localStorage. On the other hand they are hard to use in comparison, we already have a lot of solutions out there using localStorage and asking the user to give us access to storing local files is unacceptable for some implementations in terms of UX.

    The answer is that there is no simple solution for storing data on the end users’ machines and we should stop advocating localStorage as a performance boost. What we have to find is a solution that makes everybody happy and doesn’t break the current implementations. This might prove hard to work around. Here are some ideas:

    • Build a polyfill library that overrides the localStorage API and stores the content in IndexedDB/WebSQL instead? This is dirty and doesn’t work around the issue of the user being asked for permission
    • Implement localStorage in an asynchronous fashion in browsers – actively disregarding the spec? (this could set a dangerous precedent though)
    • Change the localStorage spec to store asynchronously instead of synchronously? We could also extend it to have a proper getStorageSpace interface and allow for native JSON support
    • Define a new standard that allows browser vendors to map the new API to the existing supported API that matches the best for the use case?

    We need to fix this as it doesn’t make sense to store things locally and sacrifice performance at the same time. This is a great example of how new web standards give us much more power but also make us face issues we didn’t have to deal with before. With more access to the OS, we also have to tread more carefully.

  6. Firefox 5 is here

    Today, three months after the release of Firefox 4, we release Firefox 5, thanks to our new development cycle. Developers will be able to create richer animations using CSS3 Animations. This release comes with various improvements, performance optimization and bug fixes.

    CSS3 Animations

    CSS Animations (check out the documentation) are a new way to create animations using CSS. Like CSS Transitions, they are efficient and run smoothly (see David Baron’s article), and the developers have a better controls over the intermediate steps (keyframes), and can now create much more complex animations.

    Notable changes

    Other Bug Fixes and Performance Improvements:

    HTML

    Canvas improvements

    • The <canvas> 2D drawing context now supports specifying an ImageData object as the input to the createImageData() method; this creates a new ImageData object initialized with the same dimensions as the specified object, but still with all pixels preset to transparent black.
    • Specifying non-finite values when adding color stops through a call to the CanvasGradient method addColorStop() now correctly throws INDEX_SIZE_ERR instead of SYNTAX_ERR.
    • The HTMLCanvasElement method toDataURL() now correctly lower-cases the specified MIME type before matching.
    • getImageData() now correctly accepts rectangles that extend beyond the bounds of the canvas; pixels outside the canvas are returned as transparent black.
    • drawImage() and createImageData() now handle negative arguments in accordance with the specification, by flipping the rectangle around the appropriate axis.
    • Specifying non-finite values when calling createImageData() now properly throws a NOT_SUPPORTED_ERR exception.
    • createImageData() and getImageData() now correctly return at least one pixel’s worth of image data if a rectangle smaller than one pixel is specified.
    • Specifying a negative radius when calling createRadialGradient() now correctly throws INDEX_SIZE_ERR.
    • Specifying a null or undefined image when calling createPattern() or drawImage() now correctly throws a TYPE_MISMATCH_ERR exception.
    • Specifying invalid values for globalAlpha no longer throws a SYNTAX_ERR exception; these are now correctly silently ignored.
    • Specifying invalid values when calling translate(), transform(), rect(), clearRect(), fillRect(), strokeRect(), lineTo(), moveTo(), quadraticCurveTo(), or arc() no longer throws an exception; these calls are now correctly silently ignored.
    • Setting the value of shadowOffsetX, shadowOffsetY, or shadowBlur to an invalid value is now silently ignored.
    • Setting the value of rotate or scale to an invalid value is now silently ignored.

    CSS

    • Support for CSS animations has been added, using the -moz- prefix for now.

    DOM

    • The selection object’s modify() method has been changed so that the “word” selection granularity no longer includes trailing spaces; this makes it more consistent across platforms and matches the behavior of WebKit’s implementation.
    • The window.setTimeout() method now clamps to send no more than one timeout per second in inactive tabs. In addition, it now clamps nested timeouts to the smallest value allowed by the HTML5 specification: 4 ms (instead of the 10 ms it used to clamp to).
    • Similarly, the window.setInterval() method now clamps to no more than one interval per second in inactive tabs.
    • XMLHttpRequest now supports the loadend event for progress listeners. This is sent after any transfer is finished (that is, after the abort, error, or load event). You can use this to handle any tasks that need to be performed regardless of success or failure of a transfer.
    • The Blob and, by extension, the File objects’ slice() method has been removed and replaced with a new, proposed syntax that makes it more consistent with Array.slice() and String.slice() methods in JavaScript. This method is named mozSlice() for now.
    • The value of window.navigator.language is now determined by looking at the value of the Accept-Language HTTP header.

    JavaScript

    • Regular expressions are no longer callable as if they were functions; this change has been made in concert with the WebKit team to ensure compatibility (see WebKit bug 28285).
    • The Function.prototype.isGenerator() method is now supported; this lets you determine if a function is a generator.

    SVG

    • The class SVG attribute can now be animated.
    • The following SVG-related DOM interfaces representing lists of objects are now indexable and can be accessed like arrays; in addition, they have a length property indicating the number of items in the lists: SVGLengthList , SVGNumberList , SVGPathSegList , and SVGPointList.

    HTTP

    • Firefox no longer sends the “Keep-Alive” HTTP header; we weren’t formatting it correctly, and it was redundant since we were also sending the Connection: or Proxy-Connection: header with the value “keep-alive” anyway.
    • The HTTP transaction model has been updated to be more intelligent about reusing connections in the persistent connection pool; instead of treating the pool as a FIFO queue, Necko now attempts to sort the pool with connections with the largest congestion window (CWND) first. This can reduce the round-trip time (RTT) of HTTP transactions by avoiding the need to grow connections’ windows in many cases.
    • Firefox now handles the Content-Disposition HTTP response header more effectively if both the filename and filename* parameters are provided; it looks through all provided names, using the filename* parameter if one is available, even if a filename parameter is included first. Previously, the first matching parameter would be used, thereby preventing a more appropriate name from being used. See bug 588781 .

    MathML

    Developer tools

    • The Web Console’s Console object now has a debug() method, which is an alias for its log() method; this improves compatibility with certain existing sites.

  7. Why no FileSystem API in Firefox?

    A question that I get asked a lot is why Firefox doesn’t support the FileSystem API. Usually, but not always, they are referring specifically to the FileSystem and FileWriter specifications which Google is implementing in Chrome, and which they have proposed for standardization in W3C.

    The answer is somewhat complex, and depends greatly on what exact capabilities of the above two specifications the person is actually wanting to use. The specifications are quite big and feature full, so it’s no surprise that people are wanting to do very different things with it. This blog post is an attempt at giving my answer to this question and explain why we haven’t implemented the above two specifications. But note that this post represents my personal opinion, intended to spur more conversation on this topic.

    As stated above, people asking for “FileSystem API support” in Firefox are actually often interested in solving many different problems. In my opinion most, but so far not all, of these problems have better solutions than the FileSystem API. So let me walk through them below.

    Storing resources locally

    Probably the most common thing that people want to do is to simply store a set of resources so that they are available without having to use the network. This is useful if you need quick access to the resources, or if you want to be able to access them even if the user is offline. Games are a very common type of application where this is needed. For example an enemy space ship might have a few associated images, as well as a couple of associated sounds, used when the enemy is moving around the screen and shooting. Today, people generally solve this by storing the images and sound files in a file system, and then store the file names of those files along with things like speed and firepower of the enemy.

    However it seems a bit non-optimal to me to have to store some data separated from the rest. Especially when there is a solution which can store both structured data as well as file data. IndexedDB treats file data just like any other type of data. You can write a File or a Blob into IndexedDB just like you can store strings, numbers and JavaScript objects. This is specified by the IndexedDB spec and so far implemented in both the Firefox and IE implementations of IndexedDB. Using this, you can store all information that you need in one place, and a single query to IndexedDB can return all the data you need. So for example, if you were building a web based email client, you could store an object like:

    {
      subject: "Hi there",
      body: "Hi Sven,\nHow are you doing...",
      attachments: [blob1, blob2, blob3]
    }

    Another advantage here is that there’s no need to make up file names for resources. Just store the File or Blob object. No name needed.

    In Firefox’s IndexedDB implementation (and I believe IE’s too) the files are transparently stored outside of the actual database. This means that performance of storing a file in IndexedDB is just as good as storing the file in a filesystem. It does not bloat the database itself slowing down other operations, and reading from the file means that the implementation just reads from an OS file, so it’s just as fast as a filesystem.

    Firefox IndexedDB implementation is even smart enough that if you store the same Blob multiple files in a IndexedDB database it just creates one copy of the file. Writing further references to the same Blob just adds to an internal reference counter. This is completely transparent to the web page, the only thing it will notice is faster writes and less resource use. However I’m not sure if IE does the same, so check there first before relying on it.

    Access pictures and music folders

    The second most common thing that people ask for related to a file system APIs is to be able to access things like the user’s picture or music libraries. This is something that the FileSystem API submitted to W3C doesn’t actually provide, though many people seems to think it does. To satisfy that use-case we have the DeviceStorage API. This API allows full file system capabilities for “user files”. I.e. files that aren’t specific to a website, but rather resources that are managed and owned by the user and that the user might want to access through several apps. Such as photos and music. The DeviceStorage API is basically a simple file system API mostly optimized for these types of files.

    We’re still in the process of specifying and implementing this API. It’s available to test with in recent nightly builds, but so far isn’t enabled by default. The main problem with exposing this functionality to the web is security. You wouldn’t want just any website to read or modify your images. We could put up a prompt like we do with the GeoLocation API, given that this API potentially can delete all your pictures from the last 10 years, we probably want something more. This is something we are actively working on. But it’s definitely the case here that security is the hard part here, not implementing the low-level file operations.

    Low-level file manipulation

    A less common request is the ability to do low-level create, read, update and delete (CRUD) file operations. For example being able to write 10 bytes in the middle of a 10MB file. This is not something IndexedDB supports right now, it only allows adding and removing whole files. This is supported by the FileWriter specification draft. However I think this part of this API has some pretty fundamental problems. Specifically there are no locking capabilities, so there is no way to do multiple file operations and be sure that another tab didn’t modify or read the file in between those operations. There is also no way to do fsync which means that you can’t implement ACID type applications on top of FileWriter, such as a database.

    We have instead created an API with the same goal, but which has capabilities for locking a file and doing multiple operations. This is done in a way to ensure that there is no risk that pages can forget to unlock a file, or that deadlocks can occur. The API also allows fsync operations which should enable doing things like databases on top of FileHandle. However most importantly, the API is done in such a way that you shouldn’t need to nest asynchronous callbacks as much as with FileWriter. In other words it should easier to use for authors. You can read more about FileHandle at

    https://wiki.mozilla.org/WebAPI/FileHandleAPI

    The filesystem URL scheme

    There is one more capability that exist in the FileSystem API not covered above. The specification introduces a new filesystem: URL scheme. When loading URLs from filesystem: it returns the contents of files in stored using the FileSystem API. This is a very cool feature for a couple of reasons. First of all these URLs are predictable. Once you’ve stored a file in the file system, you always know which URL can be used to load from it. And the URL will continue to work as long as the file is stored in the file system, even if the web page is reloaded. Second, relative URLs work with the filesystem: scheme. So you can create links from one resource stored in the filesystem to another resource stored in the filesystem.

    Firefox does support the blob: URL scheme, which does allow loading data from a Blob anywhere where URLs can be used. However it doesn’t have the above mentioned capabilities. This is something that I’d really like to find a solution for. If we can’t find a better solution, implementing the Google specifications is definitely an option.

    Conclusions

    As always when talking about features to be added to the web platform it’s important to talk about use cases and capabilities, and not jump directly to a particular solution. Most of the use cases that the FileSystem API aims to solve can be solved in other ways. In my opinion many times in better ways.

    This is why we haven’t prioritized implementing the FileSystem API, but instead focused on things like making our IndexedDB implementation awesome, and coming up with a good API for low-level file manipulation.

    Focusing on IndexedDB has also meant that we very soon have a good API for basic file storage available in 3 browsers: IE10, Firefox and Chrome.

    On a related note, we just fixed the last known spec compliance issues in our IndexedDB implementation, so Firefox 16 will ship with IndexedDB unprefixed!

    As always, we’re very interested in getting feedback from other people, especially from web developers. Do you think that FileSystem API is something we should prioritize? If so, why?

  8. Beyond HTML5: Database APIs and the Road to IndexedDB

    IndexedDB is an evolving web standard for the storage of significant amounts of structured data in the browser and for high performance searches on this data using indexes. Mozilla has submitted substantial technical feedback on the specification, and we plan to implement it in Firefox 4. We spoke to prominent web developers about evolving an elegant structured storage API for the web. While versions of Safari, Chrome, and Opera support a technology called Web SQL Database, which uses SQL statements as string arguments passed to a JavaScript API, we think developer aesthetics are an important consideration, and that this is a particularly inelegant solution for client-side web applications. We brought developer feedback to the editor of the IndexedDB specification, and also spoke with Microsoft, who agree with us that IndexedDB is a good option for the web. With additional implementations from the Chrome team in the offing, we think it is worth explaining our design choices, and why we think IndexedDB is a better solution for the web than Web SQL Database.

    Web applications can already take advantage of localStorage and sessionStorage in IE 8+, Safari 4+, Chrome 4+, Opera 10.5+ and Firefox 2+ to store key-value pairs with a simple JavaScript API. The Web Storage standard (encompassing localStorage and sessionStorage), now widely implemented, is useful for storing smaller amounts of data, but less useful for storing larger amounts of structured data. While many server-side databases use SQL to programmatically store structured data and to meaningfully query it, on the client-side, the use of SQL in a JavaScript API has been contentious.

    SQL? Which SQL?

    Many web developers certainly are familiar with SQL, since many developers touch just as much server-side code (e.g. PHP and database operations) as client-side code (e.g. JavaScript, CSS, and markup). However, despite the ubiquity that SQL enjoys, there isn’t a single normative SQL standard that defines the technology. In particular, SQLite supports most of SQL-92, with some notable omissions, and is what the WebDatabase API is based on. But SQLite itself isn’t a specification — it’s a release-ready technology! And the best definition of what constitutes the supported subset of SQL that SQLite uses is the SQLite manual. In order to really get Web SQL Database right, we’d have to first start with defining a meaningful subset of SQL for web applications. Why define a whole other language, when more elegant solutions exist within JavaScript itself?

    The Benefits and Pitfalls of SQLite

    We think SQLite is an extremely useful technology for applications, and make it available for Firefox extensions and trusted code. We don’t think it is the right basis for an API exposed to general web content, not least of all because there isn’t a credible, widely accepted standard that subsets SQL in a useful way. Additionally, we don’t want changes to SQLite to affect the web later, and don’t think harnessing major browser releases (and a web standard) to SQLite is prudent. IndexedDB does not have this problem; even though our underlying implementation of IndexedDB may be based on SQLite, we keep developers insulated from changes to SQLite by exposing an API that isn’t based on SQLite’s supported syntax.

    Aesthetics and Web Developers

    Last year, we held a summit at the Mozilla campus to discuss storage on the web. We invited web developers to speak to us about a desirable structured storage API on the web. Many did express resigned acceptance of a SQLite-based API, since they had already experimented with releases of Web SQL Database in some browsers, and claimed that something in circulation was better than a collection of ideas. Yet, all voiced enthusiasm for better design choices, and how a simpler model would make life easier for them. We watched as developers whiteboarded a simple BTree API that addressed their application storage needs, and this galvanized us to consider other options. We were resolved that using strings representing SQL commands lacked the elegance of a “web native” JavaScript API, and started looking at alternatives. Along with Microsoft, we sent feedback about the IndexedDB proposal and actively became involved in the standardization effort.

    In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter. IndexedDB leaves room for a third-party JavaScript library to straddle the underlying primitives with a BTree API, and we look forward to seeing initiatives like BrowserCouch built on top of IndexedDB. Intrepid web developers can even build a SQL API on top of IndexedDB. We’d particularly welcome an implementation of the Web SQL Database API on top of IndexedDB, since we think that this is technically feasible. Starting with a SQL-based API for use with browser primitives wasn’t the right first step, but certainly there’s room for SQL-based APIs on top of IndexedDB.

    We want to continue the discussion with web developers about storage on the web, since that helps us structure our thoughts about product features and future web standards. We look forward to seeing the next generation of web applications with support for high performance searches on indexed data, and to seeing web applications work even more robustly in “airplane mode.”

    Links