Mozilla

JavaScript Articles

Sort by:

View:

  1. WebRTC in Firefox 38: Multistream and renegotiation

    Building on the JSEP (Javascript Session Establishment Protocol) engine rewrite introduced in 37, Firefox 38 now has support for multistream (multiple tracks of the same type in a single PeerConnection), and renegotiation (multiple offer/answer exchanges in a single PeerConnection). As usual with such things, there are caveats and limitations, but the functionality seems to be pretty solid.

    Multistream and renegotiation features

    Why are these things useful, you ask? For example, now you can handle a group video call with a single PeerConnection (multistream), and do things like add/remove these streams on the fly (renegotiation). You can also add screensharing to an existing video call without needing a separate PeerConnection. Here are some advantages of this new functionality:

    • Simplifies your job as an app-writer
    • Requires fewer rounds of ICE (Interactive Connectivity Establishment – the protocol for establishing connection between the browsers), and reduces call establishment time
    • Requires fewer ports, both on the browser and on TURN relays (if using bundle, which is enabled by default)

    Now, there are very few WebRTC services that use multistream (the way it is currently specified, see below) or renegotiation. This means that real-world testing of these features is extremely limited, and there will probably be bugs. If you are working with these features, and are having difficulty, do not hesitate to ask questions in IRC at irc.mozilla.org on #media, since this helps us find these bugs.

    Also, it is important to note that Google Chrome’s current implementation of multistream is not going to be interoperable; this is because Chrome has not yet implemented the specification for multistream (called “unified plan” – check on their progress in the Google Chromium Bug tracker). Instead they are still using an older Google proposal (called “plan B”). These two approaches are mutually incompatible.

    On a related note, if you maintain or use a WebRTC gateway that supports multistream, odds are good that it uses “plan B” as well, and will need to be updated. This is a good time to start implementing unified plan support. (Check the Appendix below for examples.)

    Building a simple WebRTC video call page

    So let’s start with a concrete example. We are going to build a simple WebRTC video call page that allows the user to add screen sharing during the call. As we are going to dive deep quickly you might want to check out our earlier Hacks article, WebRTC and the Early API, to learn the basics.

    First we need two PeerConnections:

    pc1 = new mozRTCPeerConnection();
    pc2 = new mozRTCPeerConnection();

    Then we request access to camera and microphone and attach the resulting stream to the first PeerConnection:

    let videoConstraints = {audio: true, video: true};
    navigator.mediaDevices.getUserMedia(videoConstraints)
      .then(stream1) {
        pc1.addStream(stream1);
      });

    To keep things simple we want to be able to run the call just on one machine. But most computers today don’t have two cameras and/or microphones available. And just having a one-way call is not very exciting. So let’s use a built-in testing feature of Firefox for the other direction:

    let fakeVideoConstraints = {video: true, fake: true };
    navigator.mediaDevices.getUserMedia(fakeVideoConstraints)
      .then(stream2) {
        pc2.addStream(stream2);
      });

    Note: You’ll want to call this part from within the success callback of the first getUserMedia() call so that you don’t have to track with boolean flags if both getUserMedia() calls succeeded before you proceed to the next step.
    Firefox also has a built-in fake audio source (which you can turn on like this {audio: true, fake: true}). But listening to an 8kHz tone is not as pleasant as looking at the changing color of the fake video source.

    Now we have all the pieces ready to create the initial offer:

    pc1.createOffer().then(step1, failed);

    Now the WebRTC typical offer – answer flow follows:

    function step1(offer) {
      pc1_offer = offer;
      pc1.setLocalDescription(offer).then(step2, failed);
    }
     
    function step2() {
      pc2.setRemoteDescription(pc1_offer).then(step3, failed);
    }

    For this example we take a shortcut: Instead of passing the signaling message through an actual signaling relay, we simply pass the information into both PeerConnections as they are both locally available on the same page. Refer to our previous hacks article WebRTC and the Early API for a solution which actually uses FireBase as relay instead to connect two browsers.

    function step3() {
      pc2.createAnswer().then(step4, failed);
    }
     
    function step4(answer) {
      pc2_answer = answer;
      pc2.setLocalDescription(answer).then(step5, failed);
    }
     
    function step5() {
      pc1.setRemoteDescription(pc2_answer).then(step6, failed);
    }
     
    function step6() {
      log("Signaling is done");
    }

    The one remaining piece is to connect the remote videos once we receive them.

    pc1.onaddstream = function(obj) {
      pc1video.mozSrcObject = obj.stream;
    }

    Add a similar clone of this for our PeerConnection 2. Keep in mind that these callback functions are super trivial — they assume we only ever receive a single stream and only have a single video player to connect it. The example will get a little more complicated once we add the screen sharing.

    With this we should be able to establish a simple call with audio and video from the real devices getting sent from PeerConnection 1 to PeerConnection 2 and in the opposite direction a fake video stream that shows slowly changing colors.

    Implementing screen sharing

    Now let’s get to the real meat and add screen sharing to the already established call.

    function screenShare() {
      let screenConstraints = {video: {mediaSource: "screen"}};
     
      navigator.mediaDevices.getUserMedia(screenConstraints)
        .then(stream) {
          stream.getTracks().forEach(track) {
            screenStream = stream;
            screenSenders.push(pc1.addTrack(track, stream));
          });
        });
    }

    Two things are required to get screen sharing working:

    1. Only pages loaded over HTTPS are allowed to request screen sharing.
    2. You need to append your domain to the user preference  media.getusermedia.screensharing.allowed_domains in about:config to whitelist it for screen sharing.

    For the screenConstraints you can also use ‘window‘ or ‘application‘ instead of ‘screen‘ if you want to share less than the whole screen.
    We are using getTracks() here to fetch and store the video track out of the stream we get from the getUserMedia call, because we need to remember the track later when we want to be able to remove screen sharing from the call. Alternatively, in this case you could use the addStream() function used before to add new streams to a PeerConnection. But the addTrack() function gives you more flexibility if you want to handle video and audio tracks differently, for instance. In that case, you can fetch these tracks separately via the getAudioTracks() and getVideoTracks() functions instead of using the getTracks() function.

    Once you add a stream or track to an established PeerConnection this needs to be signaled to the other side of the connection. To kick that off, the onnegotiationneeded callback will be invoked. So your callback should be setup before adding a track or stream. The beauty here — from this point on we can simply re-use our signaling call chain. So the resulting screen share function looks like this:

    function screenShare() {
      let screenConstraints = {video: {mediaSource: "screen"}};
     
      pc1.onnegotiationneeded = function (event) {
        pc1.createOffer(step1, failed);
      };
     
      navigator.mediaDevices.getUserMedia(screenConstraints)
        .then(stream) {
          stream.getTracks().forEach(track) {
            screenStream = stream;
            screenSenders.push(pc1.addTrack(track, stream));
          });
        });
    }

    Now the receiving side also needs to learn that the stream from the screen sharing was successfully established. We need to slightly modify our initial onaddstream function for that:

    pc2.onaddstream = function(obj) {
      var stream = obj.stream;
      if (stream.getAudioTracks().length == 0) {
        pc3video.mozSrcObject = obj.stream;
      } else {
        pc2video.mozSrcObject = obj.stream;
      }
    }

    The important thing to note here: With multistream and renegotiation onaddstream can and will be called multiple times. In our little example onaddstream is called the first time we establish the connection and PeerConnection 2 starts receiving the audio and video from the real devices. And then it is called a second time when the video stream from the screen sharing is added.
    We are simply assuming here that the screen share will have no audio track in it to distinguish the two cases. There are probably cleaner ways to do this.

    Please refer to the Appendix for a bit more detail on what happens here under the hood.

    As the user probably does not want to share his/her screen until the end of the call let’s add a function to remove it as well.

    function stopScreenShare() {
      screenStream.stop();
      screenSenders.forEach(sender) {
        pc1.removeTrack(sender);
      });
    }

    We are holding on to a reference to the original stream to be able to call stop() on it to release the getUserMedia permission we got from the user. The addTrack() call in our screenShare() function returned us an RTCRtpSender object, which we are storing so we can hand it to the removeTrack() function.

    All of the code combined with some extra syntactic sugar can be found on our MultiStream test page.

    If you are going to build something which allows both ends of the call to add screen share, a more realistic scenario than our demo, you will need to handle special cases. For example, multiple users might accidentally try to add another stream (e.g. the screen share) exactly at the same time and you may end up with a new corner-case for renegotiation called “glare.” This is what happens when both ends of the WebRTC session decide to send new offers at the same time. We do not yet support the “rollback” session description type that can be used to recover from glare (see Jsep draft and the Firefox bug). Probably the best interim solution to prevent glare is to announce via your signaling channel that the user did something which is going to kick off another round of renegotiation. Then, wait for the okay from the far end before you call createOffer() locally.

    Appendix

    This is an example renegotiation offer SDP from Firefox 39 when adding the screen share:

    v=0
    o=mozilla...THIS_IS_SDPARTA-39.0a1 7832380118043521940 1 IN IP4 0.0.0.0
    s=-
    t=0 0
    a=fingerprint:sha-256 4B:31:DA:18:68:AA:76:A9:C9:A7:45:4D:3A:B3:61:E9:A9:5F:DE:63:3A:98:7C:E5:34:E4:A5:B6:95:C6:F2:E1
    a=group:BUNDLE sdparta_0 sdparta_1 sdparta_2
    a=ice-options:trickle
    a=msid-semantic:WMS *
    m=audio 9 RTP/SAVPF 109 9 0 8
    c=IN IP4 0.0.0.0
    a=candidate:0 1 UDP 2130379007 10.252.26.177 62583 typ host
    a=candidate:1 1 UDP 1694236671 63.245.221.32 54687 typ srflx raddr 10.252.26.177 rport 62583
    a=sendrecv
    a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
    a=ice-pwd:3aefa1a552633717497bdff7158dd4a1
    a=ice-ufrag:730b2351
    a=mid:sdparta_0
    a=msid:{d57d3917-64e9-4f49-adfb-b049d165c312} {920e9ffc-728e-0d40-a1b9-ebd0025c860a}
    a=rtcp-mux
    a=rtpmap:109 opus/48000/2
    a=rtpmap:9 G722/8000/1
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=setup:actpass
    a=ssrc:323910839 cname:{72b9ff9f-4d8a-5244-b19a-bd9b47251770}
    m=video 9 RTP/SAVPF 120
    c=IN IP4 0.0.0.0
    a=candidate:0 1 UDP 2130379007 10.252.26.177 62583 typ host
    a=candidate:1 1 UDP 1694236671 63.245.221.32 54687 typ srflx raddr 10.252.26.177 rport 62583
    a=sendrecv
    a=fmtp:120 max-fs=12288;max-fr=60
    a=ice-pwd:3aefa1a552633717497bdff7158dd4a1
    a=ice-ufrag:730b2351
    a=mid:sdparta_1
    a=msid:{d57d3917-64e9-4f49-adfb-b049d165c312} {35eeb34f-f89c-3946-8e5e-2d5abd38c5a5}
    a=rtcp-fb:120 nack
    a=rtcp-fb:120 nack pli
    a=rtcp-fb:120 ccm fir
    a=rtcp-mux
    a=rtpmap:120 VP8/90000
    a=setup:actpass
    a=ssrc:2917595157 cname:{72b9ff9f-4d8a-5244-b19a-bd9b47251770}
    m=video 9 RTP/SAVPF 120
    c=IN IP4 0.0.0.0
    a=sendrecv
    a=fmtp:120 max-fs=12288;max-fr=60
    a=ice-pwd:3aefa1a552633717497bdff7158dd4a1
    a=ice-ufrag:730b2351
    a=mid:sdparta_2
    a=msid:{3a2bfe17-c65d-364a-af14-415d90bb9f52} {aa7a4ca4-189b-504a-9748-5c22bc7a6c4f}
    a=rtcp-fb:120 nack
    a=rtcp-fb:120 nack pli
    a=rtcp-fb:120 ccm fir
    a=rtcp-mux
    a=rtpmap:120 VP8/90000
    a=setup:actpass
    a=ssrc:2325911938 cname:{72b9ff9f-4d8a-5244-b19a-bd9b47251770}

    Note that each track gets its own m-section, denoted by the msid attribute.

    As you can see from the BUNDLE attribute, Firefox offers to put the new video stream, with its different msid value, into the same bundled transport. That means if the answerer agrees we can start sending the video stream over the already established transport. We don’t have to go through another ICE and DTLS round. And in case of TURN servers we save another relay resource.

    Hypothetically, this is what the previous offer would look like if it used plan B (as Chrome does):

    v=0
    o=mozilla...THIS_IS_SDPARTA-39.0a1 7832380118043521940 1 IN IP4 0.0.0.0
    s=-
    t=0 0
    a=fingerprint:sha-256 4B:31:DA:18:68:AA:76:A9:C9:A7:45:4D:3A:B3:61:E9:A9:5F:DE:63:3A:98:7C:E5:34:E4:A5:B6:95:C6:F2:E1
    a=group:BUNDLE sdparta_0 sdparta_1
    a=ice-options:trickle
    a=msid-semantic:WMS *
    m=audio 9 RTP/SAVPF 109 9 0 8
    c=IN IP4 0.0.0.0
    a=candidate:0 1 UDP 2130379007 10.252.26.177 62583 typ host
    a=candidate:1 1 UDP 1694236671 63.245.221.32 54687 typ srflx raddr 10.252.26.177 rport 62583
    a=sendrecv
    a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
    a=ice-pwd:3aefa1a552633717497bdff7158dd4a1
    a=ice-ufrag:730b2351
    a=mid:sdparta_0
    a=rtcp-mux
    a=rtpmap:109 opus/48000/2
    a=rtpmap:9 G722/8000/1
    a=rtpmap:0 PCMU/8000
    a=rtpmap:8 PCMA/8000
    a=setup:actpass
    a=ssrc:323910839 msid:{d57d3917-64e9-4f49-adfb-b049d165c312} {920e9ffc-728e-0d40-a1b9-ebd0025c860a}
    a=ssrc:323910839 cname:{72b9ff9f-4d8a-5244-b19a-bd9b47251770}
    m=video 9 RTP/SAVPF 120
    c=IN IP4 0.0.0.0
    a=candidate:0 1 UDP 2130379007 10.252.26.177 62583 typ host
    a=candidate:1 1 UDP 1694236671 63.245.221.32 54687 typ srflx raddr 10.252.26.177 rport 62583
    a=sendrecv
    a=fmtp:120 max-fs=12288;max-fr=60
    a=ice-pwd:3aefa1a552633717497bdff7158dd4a1
    a=ice-ufrag:730b2351
    a=mid:sdparta_1
    a=rtcp-fb:120 nack
    a=rtcp-fb:120 nack pli
    a=rtcp-fb:120 ccm fir
    a=rtcp-mux
    a=rtpmap:120 VP8/90000
    a=setup:actpass
    a=ssrc:2917595157 msid:{d57d3917-64e9-4f49-adfb-b049d165c312} {35eeb34f-f89c-3946-8e5e-2d5abd38c5a5}
    a=ssrc:2917595157 cname:{72b9ff9f-4d8a-5244-b19a-bd9b47251770}
    a=ssrc:2325911938 msid:{3a2bfe17-c65d-364a-af14-415d90bb9f52} {aa7a4ca4-189b-504a-9748-5c22bc7a6c4f}
    a=ssrc:2325911938 cname:{72b9ff9f-4d8a-5244-b19a-bd9b47251770}

    Note that there is only one video m-section, with two different msids, which are part of the ssrc attributes rather than in their own a lines (these are called “source-level” attributes). Continued…

  2. An analytics primer for developers

    There are three kinds of lies: lies, damned lies, and statistics – Mark Twain

    Deciding what to track (all the things)

    When you are adding analytics to a system you should try to log everything. At some point in the future if you need to pull information out of a system it’s much better to have every piece of information to hand, rather than realising that you need some data that you don’t yet track. Here are some guidelines and suggestion for collecting and analysing information about how people are interacting with your website or app.

    Grouping your stats as a best practice

    Most analytics platforms allow you to tag an event with metadata. This lets you analyse stats against each other and makes it easier to compare elements in a user interaction.

    For example, if you are logging clicks on a menu, you could track each menu item differently e.g.:

    track("Home pressed");
    track("Cart pressed");
    track("Logout pressed");

    Doing this makes it harder to answer questions such as which button is most popular etc. Using metadata you can make most analytics platforms perform calculations like this for you:

    track("Menu pressed","Home button");
    track("Menu pressed","Cart button");
    track("Menu pressed","Logout button");

    The analytics above mean you now have a total of all menu presses, and you can find the most/least popular of the menu items with no extra effort.

    Optimising your funnel

    A conversion funnel is a term of art derived from a consumer marketing model. The metaphor of the funnel describes the flow of steps a user goes through as they engage more deeply with your software. Imagine you want to know how many users clicked log in and then paid at the checkout? If you track events such as “Checkout complete” and “User logged in” you can then ask your analytics platform what percentage of users did both within a certain time frame (a day for instance).

    Imagine the answer comes out to be 10%, this tells you useful information about the behaviour of your users (bear in mind this funnel is not order sensitive, i.e., it does not matter in which order the events happen in login -> cart -> pay or cart -> login -> pay). Thus, you can start to optimise parts of your app and use this value to determine whether or not you are converting more of your users to make a purchase or otherwise engage more deeply.

    Deciding what to measure

    Depending on your business, different stats will have different levels of importance. Here are some common stats of interest to developers of apps or online services:

    Number of sessions
    The total number of sessions on your product (the user opening your product, using it, then closing it = 1 session)
    Session length
    How long each session lasts (can be mode, mean, median)
    Retention
    How many people come back to your product having used it before (there are a variety of metrics such as rolling retention, 30 day retention etc)
    MAU
    Monthly active users: how may users use the app once a month
    DAU
    Daily active users: how may users use the app once a day
    ARPU
    Average revenue per user: how much money you make per person
    ATV
    Average transaction value: how much money you make per sale
    CAC
    Customer acquisition cost: how much it costs to get one extra user (normally specified by the channel for getting them)
    CLV
    Customer lifetime value: total profit made from a user (usually projected)
    Churn
    The number of people who leave your product in a given time (usually given as a percentage of total user base)
    Cycle time
    The the time it takes for one user to refer another

    Choosing an analytics tool or platform

    There are plenty of analytics providers, listed below are some of the best known and most widely used:

    Google Analytics

    Website
    Developer documentation

    Quick event log example:

    ga('send', 'event', 'button', 'click');

    Pros:

    • Free
    • Easy to set up

    Cons:

    • Steep learning curve for using the platform
    • Specialist training can be required to get the most out of the platform

    Single page apps:

    If you are making a single page app/website, you need to keep Google informed that the user is still on your page and hasn’t bounced (gone to your page/app and left without doing anything):

    ga('set' , 'page', location.pathname + location.search + location.hash);
    ga('send', 'pageview');

    Use the above code every time a user navigates to a new section of your app/website to let Google know the user is still browsing your site/app.

    Flurry

    Website
    Developer documentation

    Quick event log example:

    FlurryAgent.logEvent("Button clicked");
    FlurryAgent.logEvent("Button clicked",{more : 'data'});

    Pros:

    • Free
    • Easy to set up

    Cons:

    • Data normally 24 hours behind real time
    • Takes ages to load the data

    Mixpanel

    Website
    Developer documentation

    Quick event log example:

    mixpanel.track("Button clicked");
    mixpanel.track("Button clicked",{more : 'data'});

    Pros:

    • Free trial
    • Easy to set up
    • Real-time data

    Cons:

    • Gets expensive after the free trial
    • If you are tracking a lot of points, the interface can get cluttered

    Speeding up requests

    When you are loading an external JS file you want to do it asynchronously if possible to speed up the page load.

    <script type="text/javascript" async> ... </script>

    The above code will cause the JavaScript to load asynchronously but assumes the user has a browser supporting HTML5.

    //jQuery example
    $.getScript('https://cdn.flurry.com/js/flurry.js', 
    function(){
       ...
    });

    This code will load the JavaScript asynchronously with greater browser support.

    The next problem is that you could try to add an analytic even though the framework does not exist yet, so you need to check to see if the variable framework first:

    if(typeof FlurryAgent != "undefined"){
       ...
    }

    This will prevent errors and will also allow you to easily disable analytics during testing. (You can just stop the script from being loaded – and the variable will never be defined.)

    The problem here is that you might be missing analytics whilst waiting for the script to load. Instead, you can make a queue to store the events and then post them all when the script loads:

    var queue = [];
     
    if(typeof FlurryAgent != "undefined"){
       ...
    }else{
       queue.push(["data",{more : data}]);
    }
     
    ...
     
    //jQuery example
    $.getScript('https://cdn.flurry.com/js/flurry.js', 
    function(){
       ...
     
       for(var i = 0;i < queue.length;i++)
       {
          FlurryAgent.logEvent(queue[i][0],queue[i][1]);
       }
       queue = [];
    });

    Analytics for your Firefox App

    You can use any of the above providers above with Firefox OS, but remember when you paste a script into your code they generally are protocol agnostic: they start //myjs.com/analytics.js and you need to choose either http: or https:https://myjs.com/analytics.js (This is required only if you are making a packaged app.)

    Let us know how it goes.

  3. asm.js Speedups Everywhere

    asm.js is an easy-to-optimize subset of JavaScript. It runs in all browsers without plugins, and is a good target for porting C/C++ codebases such as game engines – which have in fact been the biggest adopters of this approach, for example Unity 3D and Unreal Engine.

    Obviously, developers porting games using asm.js would like them to run well across all browsers. However, each browser has different performance characteristics, because each has a different JavaScript engine, different graphics implementation, and so forth. In this post, we’ll focus on JavaScript execution speed and see the significant progress towards fast asm.js execution that has been happening across the board. Let’s go over each of the four major browsers now.

    Chrome

    Already in 2013, Google released Octane 2.0, a new version of their primary JavaScript benchmark suite, which contained a new asm.js benchmark, zlib. Benchmarks define what browsers optimize: things that matter are included in benchmarks, and browsers then compete to achieve the best scores. Therefore, adding an asm.js benchmark to Octane clearly signaled Google’s belief that asm.js content is important to optimize for.

    A further major development happened more recently, when Google landed TurboFan, a new work-in-progress optimizing compiler for Chrome’s JavaScript engine, v8. TurboFan has a “sea of nodes” architecture (which is new in the JavaScript space, and has been used very successfully elsewhere, for example in the Java server virtual machine), and aims to reach even higher speeds than CrankShaft, the first optimizing compiler for v8.

    While TurboFan is not yet ready to be enabled on all JavaScript content, as of Chrome 41 it is enabled on asm.js. Getting the benefits of TurboFan early on asm.js shows the importance of optimizing asm.js for the Chrome team. And the benefits can be quite substantial: For example, TurboFan speeds up Emscripten‘s zlib benchmark by 13%, and fasta by 24%.

    Safari

    During the last year, Safari’s JavaScript Engine, JavaScriptCore, introduced a new JIT (Just In Time compiler) called FTL. FTL stands for “Fourth Tier LLVM,” as it adds a fourth level of optimization above the three previously-existing ones, and it is based on LLVM, a powerful open source compiler framework. This is exciting because LLVM is a top-tier general-purpose compiler, with many years of optimizations put into it, and Safari gets to reuse all those efforts. As shown in the blogposts linked to earlier, the speedups that FTL provides can be very substantial.

    Another interesting development from Apple this year was the introduction of a new JavaScript benchmark, JetStream. JetStream contains several asm.js benchmarks, an indication that Apple believes asm.js content is important to optimize for, just as when Google added an asm.js benchmark to Octane.

    Internet Explorer

    The JavaScript engine inside Internet Explorer is named Chakra. Last year, the Chakra team blogged about a suite of optimizations coming to IE in Windows 10 and pointed to significant improvements in the scores on asm.js workloads in Octane and JetStream. This is yet another example of how having asm.js workloads in common benchmarks drives measurement and optimization.

    The big news, however, is the recent announcement by the Chakra team that they are working on adding specific asm.js optimizations, to arrive in Windows 10 together with the other optimizations mentioned earlier. These optimizations haven’t made it to the Preview channel yet, so we can’t measure and report on them here. However, we can speculate on the improvements based on the initial impact of landing asm.js optimizations in Firefox. As shown in this benchmark comparisons slide containing measurements from right after the landing, asm.js optimizations immediately brought Firefox to around 2x slower than native performance (from 5-12x native before). Why should these wins translate to Chakra? Because, as explained in our previous post, the asm.js spec provides a predictable way to validate asm.js code and generate high-quality code based on the results.

    So, here’s looking forward to good asm.js performance in Windows 10!

    Firefox

    As we mentioned before, the initial landing of asm.js optimizations in Firefox generally put Firefox within 2x of native in terms of raw throughput. By the end of 2013, we were able to report that the gap had shrunk to around 1.5x native – which is close to the amount of variability that different native compilers have between each other anyhow, so comparisons to “native speed” start to be less meaningful.

    At a high-level, this progress comes from two kinds of improvements: compiler backend optimizations and new JavaScript features. In the area of compiler backend optimizations, there has been a stream of tiny wins (specific to particular code patterns or hardware) making it difficult to point to any one thing. Two significant improvements stand out, though:

    Along with backend optimization work, two new JavaScript features have been incorporated into asm.js which unlock new performance capabilities in the hardware. The first feature, Math.fround, may look simple but it enables the compiler backend to generate single-precision floating-point arithmetic when used carefully in JS. As described in this post, the switch can result in anywhere from a 5% – 60% speedup, depending on the workload. The second feature is much bigger: SIMD.js. This is still a stage 1 proposal for ES7 so the new SIMD operations and the associated asm.js extensions are only available in Firefox Nightly. Initial results are promising though.

    Separate from all these throughput optimizations, there have also been a set of load time optimizations in Firefox: off-main-thread and parallel compilation of asm.js code as well as caching of the compiled machine code. As described in this post, these optimizations significantly improve the experience of starting a Unity- or Epic-sized asm.js application. Existing asm.js workloads in the benchmarks mentioned above do not test this aspect of asm.js performance so we put together a new benchmark suite named Massive that does. Looking at Firefox’s Massive score over time, we can see the load-time optimizations contributing to a more than 6x improvement (more details in the Hacks post introducing the Massive benchmark).

    The Bottom Line

    What is most important, in the end, are not the underlying implementation details, nor even specific performance numbers on this benchmark or that. What really matters is that applications run well. The best way to check that is to actually run real-world games! A nice example of an asm.js-using game is Dead Trigger 2, a Unity 3D game:

    The video shows the game running on Firefox, but as it uses only standard web APIs, it should work in any browser. We tried it now, and it renders quite smoothly on Firefox, Chrome and Safari. We are looking forward to testing it on the next Preview version of Internet Explorer as well.

    Another example is Cloud Raiders:

    As with Unity, the developers of Cloud Raiders were able to compile their existing C++ codebase (using Emscripten) to run on the web without relying on plugins. The result runs well in all four of the major browsers.

    In conclusion, asm.js performance has made great strides over the last year. There is still room for improvement – sometimes performance is not perfect, or a particular API is missing, in one browser or another – but all major browsers are working to make sure that asm.js runs quickly. We can see that by looking at the benchmarks they are optimizing on, which contain asm.js, and in the new improvements they are implementing in their JavaScript engines, which are often motivated by asm.js. As a result, games that not long ago would have required plugins are quickly getting to the point where they can run well without them, in modern browsers across the web.

  4. Synchronous Execution and Filesystem Access in Emscripten

    Emscripten helps port C and C++ code to run on the Web. When doing such porting, we have to work around limitations of the web platform, one of which is that code must be asynchronous: you can’t have long-running code on the Web, it must be split up into events, because other important things – rendering, input, etc. – can’t happen while your code is running. But, it is common to have C and C++ code that is synchronous! This post will review how Emscripten helps handle this problem, using a variety of methods. We’ll look at preloading a virtual filesystem as well as a recently-added option to execute your compiled code in a special interpreter. We’ll also get the chance to play some Doom!

    First, let’s take a more concrete look at  the problem. Consider, for example,

    FILE *f = fopen("data.txt", "rb");
    fread(buffer, 100, 1, f);
    fclose(f);

    This C code opens a file and reads from it synchronously. Now, in the browser we don’t have local filesystem access (content is sandboxed, for security), so when reading a file, we might be issuing a remote request to a server, or loading from IndexedDB – both of which are asynchronous! How, then, does anything get ported at all? Let’s go over three approaches to handling this problem.

    1. Preloading to Emscripten’s virtual filesystem

    The first tool Emscripten has is a virtual in-memory filesystem, implemented in JavaScript (credit goes to inolen for most of the code), which can be pre-populated before the program runs. If you know which files will be accessed, you can preload them (using emcc’s –preload-file option), and when the code executes, copies of the files are already in memory, ready for synchronous access.

    On small to medium amounts of data, this is a simple and useful technique. The compiled code doesn’t know it’s using a virtual filesystem, everything looks normal and synchronous to it. Things just work. However, with large amounts of data, it can be too expensive to preload it all into memory. You might only need each file for a short time – for example, if you load it into a WebGL shader, and then forget about it on the CPU side – but if it’s all preloaded, you have to hold it all in memory at once. Also, the Emscripten virtual filesystem works hard to be as POSIX-compliant as it can, supporting things like permissions, mmap, etc., which add overhead that might be unnecessary in some applications.

    How much of a problem this is depends not just on the amount of data you load, but also the browser and the operating system. For example, on a 32-bit browser you are generally limited to 4GB of virtual address space, and fragmentation can be a problem. For these reasons, 64-bit browsers can sometimes succeed in running applications that need a lot of memory while 32-bit browsers fail (or fail some of the time). To some extent you can try to work around memory fragmentation problems by splitting up your data into separate asset bundles, by running Emscripten’s file packager separately several times, instead of using –preload-file once for everything. Each bundle is a combination of JavaScript that you load on your page, and a binary file with the data of all the files you packaged in that asset bundle, so in this way you get multiple smaller files rather than one big one. You can also run the file packager with –no-heap-copy, which will keep the downloaded asset bundle data in separate typed arrays instead of copying them into your program’s memory. However, even at best, these things can only help some of the time with memory fragmentation, in an unpredictable manner.

    Preloading all the data is therefore not always a viable solution: With large amounts of data, we might not have enough memory, or fragmentation might be a problem. Also, we might not know ahead of time which files we will need. And in general, even if preloading works for a project, we would still like to avoid it so that we can use as little memory as possible, as things generally run faster that way. That’s why we need the 2 other approaches to handling the problem of synchronous code, which we will discuss now.

    2. Refactor code to be asynchronous

    The second approach is to refactor your code to turn synchronous into asynchronous code. Emscripten provides asynchronous APIs you can use for this purpose, for example, the fread() in the example above could be replaced with an asynchronous network download (emscripten_async_wget, emscripten_async_wget_data), or an asynchronous access of locally-cached data in IndexedDB (emscripten_idb_async_load, emscripten_idb_async_store, etc.).

    And if you have synchronous code doing something other than filesystem access, for example rendering, Emscripten provides a generic API to do an asynchronous callback (emscripten_async_call). For the common case of a main loop which should be called once per frame from the browser’s event loop, Emscripten has a main loop API (emscripten_set_main_loop, etc.).

    Concretely, an fread() would be replaced with something like

    emscripten_async_wget_data("filename.txt", 0, onLoad, onError);

    where the first parameter is the filename on the remote server, then an optional void* argument (that will be passed to the callbacks), then callbacks on load and on error. The tricky thing is that the code that should execute right after the fread() would need to be in the onLoad callback – that’s where the refactoring comes in. Sometimes this is easy to do, but it might not be.

    Refactoring code to be asynchronous is generally the optimal thing to do. It makes your application use the APIs that are available on the Web in the way they are intended to be used. However, it does require changes to your project, and may require that the entire thing be designed in an event-friendly manner, which can be difficult if it wasn’t already structured that way. For these reasons, Emscripten has one more approach that can help you here.

    3. The Emterpreter: Run synchronous code asynchronously, automatically

    The Emterpreter is a fairly new option in Emscripten that was initially developed for startup-time reasons. It compiles your code into a binary bytecode, and ships it with a little interpreter (written in JavaScript, of course), in which the code can be executed. Code running in an interpreter is “manually executed” by us, so we can control it more easily than normal JavaScript, and we can add the capability to pause and resume, which is what we need to turn synchronous code into asynchronous code. Emterpreter-Async, the Emterpreter plus support for running synchronous code asynchronously, was therefore fairly easy to add on top of the existing Emterpreter option.

    The idea of an automatic transformation from synchronous to asynchronous code was experimented with by Lu Wang during his internship over the summer of 2014: the Asyncify option. Asyncify rewrites code at the LLVM level to support pausing and resuming execution: you write synchronous code, and the compiler rewrites it to run asynchronously. Returning to the fread() example from before, Asyncify would automatically break up the function around that call, and put the code after the call into a callback function – basically, it does what we suggested you do manually in the “Refactor code to be asynchronous” section above. This can work surprisingly well: For example, Lu ported vim, a large application with a lot of synchronous code in it, to the Web. And it works! However, we hit significant limitations in terms of increased code size because of how Asyncify restructures your code.

    The Emterpreter’s async support avoids the code size problem that Asyncify hit because it is an interpreter running bytecode: The bytecode is always the same size (in fact, smaller than asm.js), and we can manipulate control flow on it manually in the interpreter, without instrumenting the code.

    Of course, running in an interpreter can be quite slow, and this one is no exception – speed can be significantly slower than usual. Therefore, this is not a mode in which you want to run most of your code. But, the Emterpreter gives you the option to decide which parts of your codebase are interpreted and which are not, and this is crucial to productive use of this option, as we will now see.

    Let’s make this concrete by showing the option in practice on the Doom codebase. Here is a normal port of Doom (specifically Boon:, the Doom code with Freedoom open art assets). That link is just Doom compiled with Emscripten, not using synchronous code or the Emterpreter at all, yet. It looks like the game works in that link – do we even need anything else? It turns out that we need synchronous execution in two places in Doom: First, for filesystem access. Since Doom is from 1993, the size of the game is quite small compared to today’s hardware. We can preload all of the data files and things just work (that’s what happens in that link). So far, so good!

    The second problem, though, is trickier: For the most part Doom renders a whole frame in each iteration of the main loop (which we can call from the browser’s event loop one at a time), however it also does some visual effects using synchronous code. Those effects are not shown in that first link – Doom fans may have noticed something was missing! :)

    Here is a build with the Emterpreter-Async option enabled. This runs the entire application as bytecode in the interpreter, and it’s quite slow, as expected. Ignoring speed for now, you might notice that when you start a game, there is a “wipe” effect right before you begin to play, that wasn’t in the previous build. It looks kind of like a descending wave. Here’s a screenshot:

    22297That effect is written synchronously (note the screen update and sleep). The result is that in the initial port of the game, the wipe effect code is executed, but the JavaScript frame doesn’t end yet so no rendering happens. For this reason, we don’t see the wipe in the first build! But we do see it in the second, because we enabled the Emterpreter-Async option, which supports synchronous code.

    The second build is slow. What can we do? The Emterpreter lets you decide which code runs normally, as full-speed asm.js, and which is interpreted. We want to run only what we absolutely must run in the interpreter, and everything else in asm.js, so things are as fast as possible. For purposes of synchronous code, the code we must interpret is anything that is on the stack during a synchronous operation. To understand what that means, imagine that the callstack currently looks like this:

    main() => D_DoomMain() => D_Display() => D_Wipe() => I_uSleep()

    and the last of those does a call to sleep. Then the Emterpreter turns this synchronous operation into an asynchronous operation by saving where execution is right now in the current method (this is easy using the interpreter’s program counter, as well as since all local variables are already stored in a stack on a global typed array), then doing the same for the methods calling it, and while doing so to exit them all (which is also easy, each call to the interpreter is a call to a JavaScript method, which just returns). After that, we can do a setTimeout() for when we want to resume. So far, we have saved what we were doing, stopped, set an asynchronous callback for some time in the future, and we can then return control to the browser’s event loop, so it can render and so forth.

    When the asynchronous callback fires sometime later, we reverse the first part of the process: We call into the interpreter for main(), jump to the right position in it, then continue to do so for the rest of the call stack – basically, recreating the call stack exactly as it was before. At this point we can resume execution in the interpreter, and it is as if we never left: synchronous execution has been turned asynchronous.

    That means that if D_Wipe() does a synchronous operation, it must be interpreted, and anything that can call it as well, and so forth, recursively. The good news is that often such code tends to be small and doesn’t need to be fast: it’s typically event-loop handling code, and not code actually doing hard work. Talking abstractly, it’s common to see callstacks like these in games:

    main() => MainLoop() => RunTasks() => PhysicsTask() => HardWork()

    and

    main() => MainLoop() => RunTasks() => IOTask() => LoadFile()

    Assuming LoadFile() does a synchronous read of a file, it must be interpreted. As we mentioned above, this means everything that can be on the stack together with it must also be interpreted: main(), MainLoop(), RunTasks(), and IOTask() – but not any of the physics methods. In other words, if you never have physics and networking on the stack at the same time (a network event calling something that ends up calling physics, or a physics event that somehow decides to do a network request all of a sudden), then you can run networking in the interpreter, and physics at full speed. This is the case in Doom, and also other real-world codebases (and even in ones that are tricky, as in Em-DOSBox which has recursion in a crucial method, sometimes a solution can be found).

    Here is a build of Doom with that optimization enabled – it only interprets what we absolutely must interpret. It runs at about the same speed as the original, optimized build and it also has the wipe effect fully working. Also, the wipe effect is nice and smooth, which it wasn’t before: even though the wipe method itself must be interpreted – because it calls sleep() – the rendering code it calls in between sleeping can run at full speed, as that rendering code is never on the stack while sleeping!

    To have synchronous code working properly while the project stays at full speed, it is crucial to run exactly the right methods in the interpreter. Here is a list of the methods we need in Doom (in the ‘whitelist’ option there) – only 15 out of 1,425, or ~1%. To help you find a list for your project, the Emterpreter provides both static and dynamic tools, see the docs for more details.

    Conclusion

    Emscripten is often used to port code that contains synchronous portions, but long-running synchronous code is not possible on the Web. As described in this article, there are three approaches to handling that situation:

    • If the synchronous code just does file access, then preloading everything is a simple solution.
    • However, if there is a great amount of data, or you don’t know what you’ll need ahead of time, this might not work well. Another option is to refactor your code to be asynchronous.
    • If that isn’t an option either, perhaps because the refactoring is too extensive, then Emscripten now offers the Emterpreter option to run parts of your codebase in an interpreter which does support synchronous execution.

    Together, these approaches provide a range of options for handling synchronous code, and in particular the common case of synchronous filesystem access.

  5. What’s new in Web Audio

    Introduction

    It’s been a while since we said anything on Hacks about the Web Audio API. However, with Firefox 37/38 hitting our Developer Edition/Nightly browser channels, there are some interesting new features to talk about!

    This article presents you with some new Web Audio tricks to watch out for, such as the new StereoPannerNode, promise-based methods, and more.

    Simple stereo panning

    Firefox 37 introduces the StereoPannerNode interface, which allows you to add a stereo panning effect to an audio source simply and easily. It takes a single property: pan—an a-rate AudioParam that can accept numeric values between -1.0 (full left channel pan) and 1.0 (full right channel pan).

    But don’t we already have a PannerNode?

    You may have already used the older PannerNode interface, which allows you to position sounds in 3D. Connecting a sound source to a PannerNode causes it to be “spatialised”, meaning that it is placed into a 3D space and you can then specify the position of the listener inside. The browser then figures out how to make the sources sound, applying panning and doppler shift effects, and other nice 3D “artifacts” if the sounds are moving over time, etc:

    var audioContext = new AudioContext();
    var pannerNode = audioContext.createPanner();
     
    // The listener is 100 units to the right of the 3D origin
    audioContext.listener.setPosition(100, 0, 0);
     
    // The panner is in the 3D origin
    pannerNode.setPosition(0, 0, 0);

    This works well with WebGL-based games as both environments use similar units for positioning—an array of x, y, z values. So you could easily update the position, orientation, and velocity of the PannerNodes as you update the position of the entities in your 3D scene.

    But what if you are just building a conventional music player where the songs are already stereo tracks, and you actually don’t care at all about 3D? You have to go through a more complicated setup process than should be necessary, and it can also be computationally more expensive. With the increased usage of mobile devices, every operation you don’t perform is a bit more battery life you save, and users of your website will love you for it.

    Enter StereoPannerNode

    StereoPannerNode is a much better solution for simple stereo use cases, as described above. You don’t need to care about the listener’s position; you just need to connect source nodes that you want to spatialise to a StereoPannerNode instance, then use the pan parameter.

    To use a stereo panner, first create a StereoPannerNode using createStereoPanner(), and then connect it to your audio source. For example:

    var audioCtx = window.AudioContext();
    // You can use any type of source
    var source = audioCtx.createMediaElementSource(myAudio);
    var panNode = audioCtx.createStereoPanner();
     
    source.connect(panNode);
    panNode.connect(audioCtx.destination);

    To change the amount of panning applied, you just update the pan property value:

    panNode.pan.value = 0.5; // places the sound halfway to the right
    panNode.pan.value = 0.0; // centers it
    panNode.pan.value = -0.5; // places the sound halfway to the left

    You can see http://mdn.github.io/stereo-panner-node/ for a complete example.

    Also, since pan is an a-rate AudioParam you can design nice smooth curves using parameter automation, and the values will be updated per sample. Trying to do this kind of change over time would sound weird and unnatural if you updated the value over multiple requestAnimationFrame calls. And you can’t automate PannerNode positions either.

    For example, this is how you could set up a panning transition from left to right that lasts two seconds:

    panNode.pan.setValueAtTime(-1, audioContext.currentTime);
    panNode.pan.linearRampToValueAtTime(1, audioContext.currentTime + 2);

    The browser will take care of updating the pan value for you. And now, as of recently, you can also visualise these curves using the Firefox Devtools Web Audio Editor.

    Detecting when StereoPannerNode is available

    It might be the case that the Web Audio implementation you’re using has not implemented this type of node yet. (At the time of this writing, it is supported in Firefox 37 and Chrome 42 only.) If you try to use StereoPannerNode in these cases, you’re going to generate a beautiful undefined is not a function error instead.

    To make sure StereoPannerNodes are available, just check whether the createStereoPanner() method exists in your AudioContext:

    if (audioContext.createStereoPanner) {
        // StereoPannerNode is supported!
    }

    If it doesn’t, you will need to revert back to the older PannerNode.

    Changes to the default PannerNode panning algorithm

    The default panning algorithm type used in PannerNodes used to be HRTF, which is a high quality algorithm that rendered its output using a convolution with human-based data (thus it’s very realistic). However it is also very computationally expensive, requiring the processing to be run in additional threads to ensure smooth playback.

    Authors often don’t require such a high level of quality and just need something that is good enough, so the default PannerNode.type is now equalpower, which is much cheaper to compute. If you want to go back to using the high quality panning algorithm instead, you just need to change the type:

    pannerNodeInstance.type = 'HRTF';

    Incidentally, a PannerNode using type = 'equalpower' results in the same algorithm that StereoPannerNode uses.

    Promise-based methods

    Another interesting feature that has been recently added to the Web Audio spec is Promise-based versions of certain methods. These are OfflineAudioContext.startRendering() and AudioContext.decodeAudioData.

    The below sections show how the method calls look with and without Promises.

    OfflineAudioContext.startRendering()

    Let’s suppose we want to generate a minute of audio at 44100 Hz. We’d first create the context:

    var offlineAudioContext = new OfflineAudioContext(2, 44100 * 60, 44100);

    Classic code

    offlineAudioContext.addEventListener('oncomplete', function(e) {
        // rendering complete, results are at `e.renderedBuffer`
    });
    offlineAudioContext.startRendering();

    Promise-based code

    offlineAudioContext.startRendering().then(function(renderedBuffer) {
        // rendered results in `renderedBuffer`
    });

    AudioContext.decodeAudioData

    Likewise, when decoding an audio track we would create the context first:

    var audioContext = new AudioContext();

    Classic code

    audioContext.decodeAudioData(data, function onSuccess(decodedBuffer) {
        // decoded data is decodedBuffer
    }, function onError(e) {
        // guess what... something didn't work out well!
    });

    Promise-based code

    audioContext.decodeAudioData(data).then(function(decodedBuffer) {
        // decoded data is decodedBuffer
    }, function onError(e) {
        // guess what... something didn't work out well!
    });

    In both cases the differences don’t seem major, but if you’re composing the results of promises sequentially or if you’re waiting on an event to complete before calling several other methods, promises are really helpful to avoid callback hell.

    Detecting support for Promise-based methods

    Again, you don’t want to get the dreaded undefined is not a function error message if the browser you’re running your code on doesn’t support these new versions of the methods.

    A quick way to check for support: look at the returned type of these calls. If they return a Promise, we’re in luck. If they don’t, we have to keep using the old methods:

    if((new OfflineAudioContext(1, 1, 44100)).startRendering() != undefined) {
        // Promise with startRendering is supported
    }
     
    if((new AudioContext()).decodeAudioData(new Uint8Array(1)) != undefined) {
        // Promise with decodeAudioData is supported
    }

    Audio workers

    Although the spec has not been finalised and they are not implemented in any browser yet, it is also worth giving a mention to Audio Workers, which —you’ve guessed it— are a specialised type of web worker for use by Web Audio code.

    Audio Workers will replace the almost-obsolete ScriptProcessorNode. Originally, this was the way to run your own custom nodes inside the audio graph, but they actually run on the main thread causing all sorts of problems, from audio glitches (if the main thread becomes stalled) to unresponsive UI code (if the ScriptProcessorNodes aren’t fast enough to process their data).

    The biggest feature of audio workers is that they run in their own separate thread, just like any other Worker. This ensures that audio processing is prioritised and we avoid sound glitches, which human ears are very sensitive to.

    There is an ongoing discussion on the w3c web audio list; if you are interested in this and other Web Audio developments, you should go check it out.

    Exciting times for audio on the Web!

  6. Embedding an HTTP Web Server in Firefox OS

    Nearing the end of last year, Mozilla employees were gathered together for a week of collaboration and planning. During that week, a group was formed to envision what the future of Firefox OS might be surrounding a more P2P-focused Web. In particular, we’ve been looking at harnessing technologies to collectively enable offline P2P connections such as Bluetooth, NFC and WiFi Direct.

    Since these technologies only provide a means to communicate between devices, it became immediately clear that we would also need a protocol for apps to send and receive data. I quickly realized that we already have a standard protocol for transmitting data in web apps that we could leverage – HTTP.

    By utilizing HTTP, we would already have everything we’d need for apps to send and receive data on the client side, but we would still need a web server running in the browser to enable offline P2P communications. While this type of HTTP server functionality might be best suited as part of a standardized WebAPI to be baked into Gecko, we actually already have everything we need in Firefox OS to implement this in JavaScript today!

    navigator.mozTCPSocket

    Packaged apps have access to both raw TCP and UDP network sockets, but since we’re dealing with HTTP, we only need to work with TCP sockets. Access to the TCPSocket API is exposed through navigator.mozTCPSocket which is currently only exposed to “privileged” packaged apps with the tcp-socket permission:

    "type": "privileged",
    "permissions": {
      "tcp-socket": {}
    },

    In order to respond to incoming HTTP requests, we need to create a new TCPSocket that listens on a known port such as 8080:

    var socket = navigator.mozTCPSocket.listen(8080);

    When an incoming HTTP request is received, the TCPSocket needs to handle the request through the onconnect handler. The onconnect handler will receive a TCPSocket object used to service the request. The TCPSocket you receive will then call its own ondata handler each time additional HTTP request data is received:

    socket.onconnect = function(connection) {
      connection.ondata = function(evt) {
        console.log(evt.data);
      };
    };

    Typically, an HTTP request will result in a single calling of the ondata handler. However, in cases where the HTTP request payload is very large, such as for file uploads, the ondata handler will be triggered each time the buffer is filled, until the entire request payload is delivered.

    In order to respond to the HTTP request, we must send data to the TCPSocket we received from the onconnect handler:

    connection.ondata = function(evt) {
      var response = 'HTTP/1.1 200 OK\r\n';
      var body = 'Hello World!';
     
      response += 'Content-Length: ' + body.length + '\r\n';
      response += '\r\n';
      response += body;
     
      connection.send(response);
      connection.close();
    };

    The above example sends a proper HTTP response with “Hello World!” in the body. Valid HTTP responses must contain a status line which consists of the HTTP version HTTP/1.1, the response code 200 and the response reason OK terminated by a CR+LF \r\n character sequence. Immediately following the status line are the HTTP headers, one per line, separated by a CR+LF character sequence. After the headers, an additional CR+LF character sequence is required to separate the headers from the body of the HTTP response.

    FxOS Web Server

    Now, it is likely that we will want to go beyond simple static “Hello World!” responses to do things like parsing the URL path and extracting parameters from the HTTP request in order to respond with dynamic content. It just so happens that I’ve already implemented a basic-featured HTTP server library that you can include in your own Firefox OS apps!

    FxOS Web Server can parse all parts of the HTTP request for various content types including application/x-www-form-urlencoded and multipart/form-data. It can also gracefully handle large HTTP requests for file uploads and can send large binary responses for serving up content such as images and videos. You can either download the source code for FxOS Web Server on GitHub to include in your projects manually or you can utilize Bower to fetch the latest version:

    bower install justindarc/fxos-web-server --save

    Once you have the source code downloaded, you’ll need to include dist/fxos-web-server.js in your app using a <script> tag or a module loader like RequireJS.

    Simple File Storage App

    Next, I’m going to show you how to use FxOS Web Server to build a simple Firefox OS app that lets you use your mobile device like a portable flash drive for storing and retrieving files. You can see the source code for the finished product on GitHub.

    Before we get into the code, let’s set up our app manifest to get permission to access DeviceStorage and TCPSocket:

    {
      "version": "1.0.0",
      "name": "WebDrive",
      "description": "A Firefox OS app for storing files from a web browser",
      "launch_path": "/index.html",
      "icons": {
        "128": "/icons/icon_128.png"
      },
      "type": "privileged",
      "permissions": {
        "device-storage:sdcard": { "access": "readwrite" },
        "tcp-socket": {}
      }
    }

    Our app won’t need much UI, just a listing of files in the “WebDrive” folder on the device, so our HTML will be pretty simple:

    <!DOCTYPE html>
    <html>
    <head>
      <meta charset="utf-8">
      <title>WebDrive</title>
      <meta name="description" content="A Firefox OS app for storing files from a web browser">
      <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1">
      <script src="bower_components/fxos-web-server/dist/fxos-web-server.js"></script>
      <script src="js/storage.js"></script>
      <script src="js/app.js"></script>
    </head>
    <body>
      <h1>WebDrive</h1>
      <hr>
      <h3>Files</h3>
      <ul id="list"></ul>
    </body>
    </html>

    As you can see, I’ve included fxos-web-server.js in addition to app.js. I’ve also included a DeviceStorage helper module called storage.js since enumerating files can get somewhat complex. This will help keep the focus on our code specific to the task at hand.

    The first thing we’ll need to do is create new instances of the HTTPServer and Storage objects:

    var httpServer = new HTTPServer(8080);
    var storage = new Storage('sdcard');

    This will initialize a new HTTPServer on port 8080 and a new instance of our Storage helper pointing to the device’s SD card. In order for our HTTPServer instance to be useful, we must listen for and handle the “request” event. When an incoming HTTP request is received, the HTTPServer will emit a “request” event that passes the parsed HTTP request as an HTTPRequest object to the event handler.

    The HTTPRequest object contains various properties of an HTTP request including the HTTP method, path, headers, query parameters and form data. In addition to the request data, an HTTPResponse object is also passed to the “request” event handler. The HTTPResponse object allows us to send our response as a file or string and set the response headers:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      // Handle request here...
    });

    When a user requests the root URL of our web server, we’ll want to present them with a listing of files stored in the “WebDrive” folder on the device along with a file input for uploading new files. For convenience, we’ll create two helper functions to generate the HTML string to send in our HTTP response. One will just generate the listing of files which we’ll reuse to display the files on the device locally and the other will generate the entire HTML document to send in the HTTP response:

    function generateListing(callback) {
      storage.list('WebDrive', function(directory) {
        if (!directory || Object.keys(directory).length === 0) {
          callback('<li>No files found</li>');
          return;
        }
     
        var html = '';
        for (var file in directory) {
          html += `<li><a href="/${encodeURIComponent(file)}" target="_blank">${file}</a></li>`;
        }
     
        callback(html);
      });
    }
     
    function generateHTML(callback) {
      generateListing(function(listing) {
        var html =
    `<!DOCTYPE html>
    <html>
    <head>
      <meta charset="utf-8">
      <title>WebDrive</title>
    </head>
    <body>
      <h1>WebDrive</h1>
      <form method="POST" enctype="multipart/form-data">
        <input type="file" name="file">
        <button type="submit">Upload</button>
      </form>
      <hr>
      <h3>Files</h3>
      <ul>${listing}</ul>
    </body>
    </html>`;
     
        callback(html);
      });
    }

    You’ll notice that we’re using ES6 Template Strings for generating our HTML. If you’re not familiar with Template Strings, they allow us to have multi-line strings that automatically include whitespace and new lines and we can do basic string interpolation that automatically inserts values inside the ${} syntax. This is especially useful for generating HTML because it allows us to span multiple lines so our template markup remains highly readable when embedded within JavaScript code.

    Now that we have our helper functions, let’s send our HTML response in our “request” event handler:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      generateHTML(function(html) {
        response.send(html);
      });
    });

    As of right now, our “request” event handler will always respond with an HTML page listing all the files in the device’s “WebDrive” folder. However, we must first start the HTTPServer before we can receive any requests. We’ll do this once the DOM is ready and while we’re at it, let’s also render the file listing locally:

    window.addEventListener('DOMContentLoaded', function(evt) {
      generateListing(function(listing) {
        list.innerHTML = listing;
      });
     
      httpServer.start();
    });

    We should also be sure to stop the HTTPServer when the app is terminated, otherwise the network socket may never be freed:

    window.addEventListener('beforeunload', function(evt) {
      httpServer.stop();
    });

    At this point, our web server should be up and running! Go ahead and install the app on your device or simulator using WebIDE. Once installed, launch the app and point your desktop browser to your device’s IP address at port 8080 (e.g.: http://10.0.1.12:8080).

    You should see our index page load in your desktop browser, but the upload form still isn’t wired up and if you have any files in your “WebDrive” folder on your device, they cannot be downloaded yet. Let’s first wire up the file upload by first creating another helper function to save files received in an HTTPRequest:

    function saveFile(file, callback) {
      var arrayBuffer = BinaryUtils.stringToArrayBuffer(file.value);
      var blob = new Blob([arrayBuffer]);
     
      storage.add(blob, 'WebDrive/' + file.metadata.filename, callback);
    }

    This function will first convert the file’s contents to an ArrayBuffer using the BinaryUtils utility that comes with fxos-web-server.js. We then create a Blob that we pass to our Storage helper to save it to the SD card in the “WebDrive” folder. Note that the filename can be extracted from the file’s metadata object since it gets passed to the server using ‘multipart/form-data’ encoding.

    Now that we have a helper for saving an uploaded file, let’s wire it up in our “request” event handler:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      if (request.method === 'POST' && request.body.file) {
        saveFile(request.body.file, function() {
          generateHTML(function(html) {
            response.send(html);
          });
     
          generateListing(function(html) {
            list.innerHTML = html;
          });
        });
     
        return;
      }
     
      generateHTML(function(html) {
        response.send(html);
      });
    });

    Now, anytime an HTTP POST request is received that contains a “file” parameter in the request body, we will save the file to the “WebDrive” folder on the SD card and respond with an updated file listing index page. At the same time, we’ll also update the file listing on the local device to display the newly-added file.

    The only remaining part of our app to wire up is the ability to download files. Once again, let’s update the “request” event handler to do this:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      if (request.method === 'POST' && request.body.file) {
        saveFile(request.body.file, function() {
          generateHTML(function(html) {
            response.send(html);
          });
     
          generateListing(function(html) {
            list.innerHTML = html;
          });
        });
     
        return;
      }
     
      var path = decodeURIComponent(request.path);
      if (path !== '/') {
        storage.get('WebDrive' + path, function(file) {
          if (!file) {
            response.send(null, 404);
            return;
          }
     
          response.headers['Content-Type'] = file.type;
          response.sendFile(file);
        });
     
        return;
      }
     
      generateHTML(function(html) {
        response.send(html);
      });
    });

    This time, our “request” event handler will check the requested path to see if a URL other than the root is being requested. If so, we assume that the user is requesting to download a file which we then proceed to get using our Storage helper. If the file cannot be found, we return an HTTP 404 error. Otherwise, we set the “Content-Type” in the response header to the file’s MIME type and send the file with the HTTPResponse object.

    You can now re-install the app to your device or simulator using WebIDE and once again point your desktop browser to your device’s IP address at port 8080. Now, you should be able to upload and download files from your device using your desktop browser!

    The possible use cases enabled by embedding a web server into Firefox OS apps are nearly limitless. Not only can you serve up web content from your device to a desktop browser, as we just did here, but you can also serve up content from one device to another. That also means that you can use HTTP to send and receive data between apps on the same device! Since its inception, FxOS Web Server has been used as a foundation for several exciting experiments at Mozilla:

    • wifi-columns

      Guillaume Marty has combined FxOS Web Server with his amazing jsSMS Master System/Game Gear emulator to enable multi-player gaming across two devices in conjunction with WiFi Direct.

    • sharing

      Several members of the Gaia team have used FxOS Web Server and dns-sd.js to create an app that allows users to discover and share apps with friends over WiFi.

    • firedrop

      I have personally used FxOS Web Server to build an app that lets you share files with nearby users without an Internet connection using WiFi Direct. You can see the app in action here:

    I look forward to seeing all the exciting things that are built next with FxOS Web Server!

  7. From Map/Reduce to JavaScript Functional Programming

    Since ECMAScript 5.1, Array.prototype.map & Array.prototype.reduce were introduced to major browsers. These two functions not only allow developers to describe a computation more clearly, but also to simplify the work of writing loops for traversing an array; especially when the looping code actually is for mapping the array to a new array, or for the accumulation, checksum, and other similar reducing operations.

    Left: using ordinary loop; Right: using map & reduce

    Left: using ordinary loop; Right: using map & reduce

     

    Map/Reduce

    Map actually means to compute things with the original array without doing structural changes to the output. For example, when map receives an array, you can make sure the output will be another array, and the only difference is that the elements inside it may be transformed from the original value/type to another value/type. So we can say the doMap function from the above example comes with the following type signature:

    doMap signature

    The signature reveals that [Number] means this is an array of numbers. So we now can read the signature as:

    doMap is a function, which would turn an array of numbers to an array of booleans

    On the other hand, the reducing operation means we may change the structure of the input data type to a new one. For example, the signature of the doReduce is:

    doReduce signature

    Here, the Array of [Number] is gone. So we can see the major difference between map and reduce1.

    Functional Programming

    In fact, the concepts of map and reduce are older even than JavaScript and are widely used in other functional programming languages, such as Lisp or Haskell2. This observation is noted in the famous article ‘JavaScript: The World’s Most Misunderstood Programming Language’ by Douglas Crockford 3:

    JavaScript’s C-like syntax, including curly braces and the clunky for statement, makes it appear to be an ordinary procedural language. This is misleading because JavaScript has more in common with functional languages like Lisp or Scheme than with C or Java.

    This is one reason why JavaScript can do some functional-like things which other orthogonal OOP languages can’t or won’t do. For example, before Java 8 4 5, if we wanted to do some ‘callback’ things common in JavaScript, we’d need to create a redundant ‘anonymous class.':

    Button button =
      (Button) findViewById(R.id.button);
    button.setOnClickListener(
      new OnClickListener() {
        public void onClick(View v) {
          // do something
        }
      }
    );

    Of course, using anonymous callbacks or not in JavaScript is always controversial. We may encounter callback hell especially when the component keeps growing. However, first-class functions can do lots of things beyond the callback. In Haskell, we can organize our whole GUI program similar to the Quake-like games6 with only functions7. That is, we can even make it without the classes, methods, inheritance, templates and other stuff8 people usually expect to have when a program needs to be constructed.

    Frag

    Frag, the Quake-like game in Haskell

    Therefore, in the JavaScript world, it’s possible to follow similar patterns to construct our programs, rather than hurriedly implementing our own ‘class’ and ‘class system,’ as programmers often do when starting on a problem9. Adding some functional flavor in JavaScript is not so bad after all, especially when features like map and reduce are supported by native APIs. To adopt this approach also means we can write more concise code10 by combining features instead of redefining them. The only limitation is the language itself is still not functional enough, so we may run into trouble if we play too many tricks, although this should be solvable with the right library11.

    map and reduce receive other functions as arguments and output them as results. This is important because in this way they present the basic idea of composing computations in the functional world, and are able to glue small pieces together with flexibility and scalability. For example, let’s take a look at the signature of our map expression mentioned above:

    map signature

    You’ll notice that the second argument indicates a function with type Number -> Boolean. In fact, you can give it any function with a -> b type. This may not be so odd in the world of JavaScript — we write tons of callbacks in our daily work. However, the point is that higher-order functions are functions as well. They can be composed into larger ones till we generate the complete program with only first-class functions and some powerful high-order functions like id, reduce, curry, uncurry, arrow and bind12.

    Map/Reduce in Practice

    Since we may encounter language limitations, we can’t write our JavaScript code in fully functional style; however, we can borrow the idea of types and composition to do lots of things. For example, when you think in types, you will find that map is not only for data handling:

    map & fold signatures

    This is what the type signatures for map and reduce would look like in Haskell. We can substitute the a and b with anything. So, what if a becomes SQL and b becomes IO x ? Remember, we’re thinking in type, and IO x is nothing more than an ordinary type like Int or URL:

    -- Let's construct queries from SQL statements.
    makeQueries strs = map str  prepare conn str
    doQuery qrys = foldl (results query  results >> query) (return ()) qrys 
    -- Do query and get results.
    let stmts = [ "INSERT INTO Articles ('Functional JavaScript')"
                , "INSERT INTO Gecko VALUES ('30.a1')"
                , "DELETE FROM Articles WHERE version='deprecated'"
                ]
    main = execute (doQuery (makeQuery stmts))`

    (Note: This is a simplified Haskell example for demo only. It can’t actually be executed.)

    The example creates a makeQueries function with map, which will turn the SQL into IO ()13; this also means we generate several actions which can be performed.

    makeQueries signature

    And then, the doQuery function, which is actually a reducing operation, will execute the queries:

    doQueries signature

    Note its reducing operation performs IO action with the help of the bind function (>>) of the specific Monad. This topic is not covered in this article, but readers should imagine this as a way to compose functions to execute them step by step, just as a Promise does24.

    The technique is useful not only in Haskell but also in JavaScript. We can use this idea with Promises and ES6 arrow functions to organize a similar computation:

      // Use a Promise-based library to do IO.
      var http = require("q-io/http")
         ,noop = new Promise(()=>{})
         ,prepare =
            (str)=> http.read('http://www.example.com/' + str)
                      .then((res)=> res.body.toString())
                      // the 'then' is equal to the '>>'
         ,makeQuery = 
            (strs)=> strs.map((str)=> prepare(str))
         ,doQuery = 
            (qrys)=> qrys.reduce((results, qry)=> results.then(qry), noop)
         ,stmts = [ "articles/FunctionalJavaScript"
                  , "blob/c745ef73-ece9-46da-8f66-ebes574789b1"
                  , "books/language/Haskell"
                  ]
         ,main = doQuery(makeQuery(stmts));

    (NOTE: In Node.js, the similar querying code with map/reduce and Promise would not run like the Haskell version, since we need Lazy Promise14 and Lazy Evaluation15)

    We’re very close to what we want: define computations with functions, and then combine them to perform it later, although the idea of ‘later’ is not actually true since we don’t have real lazy evaluation in JavaScript. This can be accomplished if we play the trick of keeping an undone Promise — a resolve function which is only resolved when we want to do so. However, even this is tricky, and there are still some unsolvable issues.

    Another thing to note is that our program doesn’t need variable variables, but some computation results are transformed and forwarded at every step of our program. In fact, this is just one reason why functional languages can stay pure, and thus they can benefit from the optimization and getting rid of unexpected side-effects 1617.

    More on Functional Programming

    Map/reduce are the most common functional features in JavaScript. With other not-so-functional features like Promise, we can use tricks like Monad-style computation control, or we can easily define curried functions with ES6’s fat-arrow functions18 and so on. Also, there are some excellent libraries which provide nice functional features19 20 21, and some Domain Specific Languages (DSLs) are even born with functional spirit 22 23. Of course, the best way to understand functional programming is to learn a language designed for it, like Haskell, ML or OCaml. Scala, F#, and Erlang are good choices, too.


    1. In fact, map can be implemented with reduce. The most basic operation for structure like this is reduce.
    https://github.com/timoxley/functional-javascript-workshop/blob/440da6737f34b4550301ba3a77b2e4b7721e99fd/problems/implement_map_with_reduce/solution.js#L11

    2. http://en.wikipedia.org/wiki/Lisp_(programming_language)#Control_structures

    3. http://javascript.crockford.com/javascript.html

    4. Java 8 now includes the lambda function: https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html

    5. C++ traditionally has not been a functional language, but C++11 introduces lambda functions: http://en.wikipedia.org/wiki/C%2B%2B11#Lambda_functions_and_expressions

    6. https://www.haskell.org/haskellwiki/Frag

    7. Haskell can represent data structure in function sense, even though declaring a function and a data type are still not the same thing: http://book.realworldhaskell.org/read/data-structures.html

    8. Yes, I’m cheating: we have Typeclass, Functor, instance and type variable in Haskell.

    9. For those who can’t live without classes, ES6 is in your future: http://wiki.ecmascript.org/doku.php?id=strawman:maximally_minimal_classes

    10. I’ve found that some ‘bad functional code’ can be refactored as concisely as possible by strictly following some functional patterns. The most problematic ‘functional’ code happens when the coder mixes two programming styles badly. This can mix problems from two paradigms in a way that makes the code more complicated.

    11. I always hit a wall when I want to have nice Monad and lazy Promise in JavaScript. However, if you don’t mind a ‘crazy’ implementation, these are doable, and we can even have ‘Monad Transformer’ in JavaScript. Other features, like tail-recursion optimization and real lazy-evaluation, are impossible to do without runtime support.

    12. The functions arrow and bind are actually >>> and >>= in Haskell. They’re the keys In Haskell to compose our computation and program with specific effects; hence we can have state machine, networking, event handling, IO statements, and asynchronous flow control. Importantly, these are still plain functions.

    13. The type IO () means ‘do IO without any value returned.’ The IO a means some IO actions may get value a when the function had been performed, although some actions only get (). For example, the function to get a string from user input would be: ask:: IO String, while the function to print string out is print:: String -> IO String.

    14. http://www.jroller.com/vaclav/entry/promises_getting_lazy

    15. http://www.haskell.org/haskellwiki/Lazy_evaluation

    16. JavaScript can do this with a library for structures like map, set, and list. Facebook created a library of immutable data structures called immutable-js for this: https://github.com/facebook/immutable-js

    17. You can do almost the same thing with immutable-js and convince everyone to use only let and const to define variables.

    18. http://wiki.ecmascript.org/doku.php?id=harmony:arrow_function_syntax

    19. wu.js: http://fitzgen.github.io/wu.js/

    20. Ramda: http://ramdajs.com/ramdocs/docs/

    21. transducer.js: http://jlongster.com/Transducers.js–A-JavaScript-Library-for-Transformation-of-Data

    22. LiveScript: http://livescript.net/

    23. Elm: http://elm-lang.org/

    24. No, they’re not really the same, but you *could* implement Promise in Monad

  8. You can’t go wrong watching JavaScript talks

    Late last week, I was collecting suggestions for year-end Hacks blog posts. As she headed out for the winter holidays, apps engineer Soledad Penadés gifted me “a bunch of cool talks I watched this year.”

    In fact, it’s a curated collection of presentations from JSConf, JSConf EU, and other recent developer conferences. Presenters include notable Mozillians and opinionated friends: hackers, JavaScripters, teachers, preachers, and hipsters. Many talks come with links to slides. They cover diverse topics: software development, developer culture and practices, developer tools, open source, JavaScript, CSS, new APIs.

    Part 1: Sole’s list

    Part 2: Making video better on Firefox

    Binge-watching JavaScript talks not your idea of a holiday break? If you’re watching video at all, Mozilla could use your help improving the video experience on Firefox.

    Here’s how you can contribute: Download Firefox Nightly, and help us test the implementation of Media Source Extensions (MSE) to support YouTube videos running in HTML5. Over recents weeks, Firefox engineers have been working hard addressing top bugs. To get this into the hands of users as quickly as possible, we need more help testing the experience. Here’s a detailed description of the testing task, thanks to Bugmaster Liz Henry from the QA team.

    File bugs using this link. Describe your platform, OS, and the steps needed to reproduce the problem.

    Thank you for helping to make the Firefox Desktop video experience awesome. See you in the new year.

  9. Pseudo elements, promise inspection, raw headers, and much more – Firefox Developer Edition 36

    Firefox 36 was just uplifted to the Developer Edition channel, so let’s take a look at the most important Developer Tools changes in this release. We will also cover some changes from Firefox 35 since it was released shortly before the initial Developer Edition announcement. There is a lot to talk about, so let’s get right to it.

    Inspector

    You can now inspect ::before and ::after pseudo elements.  They behave like other elements in the markup tree and inspector sidebars. (35, development notes)

    before-after-inspection

    There is a new “Show DOM Properties” context menu item in the markup tree. (35, development notes, documentation about this feature on MDN)

    show-dom-properties

    The box model highlighter now works on remote targets, so there is a full-featured highlighter even when working with pages on Firefox for Android or apps on Firefox OS. (36, development notes, and technical documentation for building your own custom highlighter)

    Shadow DOM content is now visible in the markup tree; note that you will need to set dom.webcomponents.enabled to true to test this feature (36, development notes, and see bug 1053898 for more work in this space).

    We borrowed a useful feature from Firebug and are now allowing more paste options when right clicking a node in the markup tree.  (36, development notes, documentation about this feature on MDN)

    paste-options

    Some more changes to the Inspector included in Firefox 35 & 36:

    • Deleting a node now selects the previous sibling instead of the parent (36, development notes)
    • There is higher contrast for the currently hovered node in the markup view (36, development notes)
    • Hover over a CSS selector in the computed view to highlight all the nodes that match that selector on the page. (35, development notes)

    Debugger / Console

    DOM Promises are now inspectable. You can inspect the promises state and value at any moment. (36, development notes)

    promise-inspection

    The debugger now recognizes and works with eval’ed sources. (36, development notes)

    Eval’ed sources support the //# sourceURL=path.js syntax, which will make them appear as a normal file in the debugger and in stack traces reported by Error.prototype.stack. See this post: http://fitzgeraldnick.com/weblog/59/ for much more information. (36, development notes,  more development notes)

    Console statements now include the column number they originated from. (36, development notes)

    WebIDE

    You are now able to connect to Firefox for Android from the WebIDE.  See documentation for debugging firefox for android on MDN.  (36, development notes).

    We also made some changes to improve the user experience in the WebIDE:

    • Bring up developer tools by default when I select a runtime app / tab (35, development notes)
    • Re-select project on connect if last project is runtime app (35, development notes)
    • Auto-select and connect to last used runtime if available (35, development notes)
    • Font resizing (36, development notes)
    • You can now adding a hosted app project by entering the base URL (eg: “http://example.com”) instead of requiring the full path to the manifest.webapp file (35, development notes)

    Network Monitor

    We added a plain request/response headers view to make it easier to view and copy the raw headers on a request. (35, development notes)

    net-headers-raw

    Here is a list of all the bugs resolved for Firefox 35 and all the bugs resolved for Firefox 36.

    Do you have feedback, bug reports, feature requests, or questions? As always, you can comment here, add/vote for ideas on UserVoice or get in touch with the team at @FirefoxDevTools on Twitter.

  10. Introducing the JavaScript Internationalization API

    Firefox 29 issued half a year ago, so this post is long overdue. Nevertheless I wanted to pause for a second to discuss the Internationalization API first shipped on desktop in that release (and passing all tests!). Norbert Lindenberg wrote most of the implementation, and I reviewed it and now maintain it. (Work by Makoto Kato should bring this to Android soon; b2g may take longer due to some b2g-specific hurdles. Stay tuned.)

    What’s internationalization?

    Internationalization (i18n for short — i, eighteen characters, n) is the process of writing applications in a way that allows them to be easily adapted for audiences from varied places, using varied languages. It’s easy to get this wrong by inadvertently assuming one’s users come from one place and speak one language, especially if you don’t even know you’ve made an assumption.

    function formatDate(d)
    {
      // Everyone uses month/date/year...right?
      var month = d.getMonth() + 1;
      var date = d.getDate();
      var year = d.getFullYear();
      return month + "/" + date + "/" + year;
    }
     
    function formatMoney(amount)
    {
      // All money is dollars with two fractional digits...right?
      return "$" + amount.toFixed(2);
    }
     
    function sortNames(names)
    {
      function sortAlphabetically(a, b)
      {
        var left = a.toLowerCase(), right = b.toLowerCase();
        if (left > right)
          return 1;
        if (left === right)
          return 0;
        return -1;
      }
     
      // Names always sort alphabetically...right?
      names.sort(sortAlphabetically);
    }

    JavaScript’s historical i18n support is poor

    i18n-aware formatting in traditional JS uses the various toLocaleString() methods. The resulting strings contained whatever details the implementation chose to provide: no way to pick and choose (did you need a weekday in that formatted date? is the year irrelevant?). Even if the proper details were included, the format might be wrong e.g. decimal when percentage was desired. And you couldn’t choose a locale.

    As for sorting, JS provided almost no useful locale-sensitive text-comparison (collation) functions. localeCompare() existed but with a very awkward interface unsuited for use with sort. And it too didn’t permit choosing a locale or specific sort order.

    These limitations are bad enough that — this surprised me greatly when I learned it! — serious web applications that need i18n capabilities (most commonly, financial sites displaying currencies) will box up the data, send it to a server, have the server perform the operation, and send it back to the client. Server roundtrips just to format amounts of money. Yeesh.

    A new JS Internationalization API

    The new ECMAScript Internationalization API greatly improves JavaScript’s i18n capabilities. It provides all the flourishes one could want for formatting dates and numbers and sorting text. The locale is selectable, with fallback if the requested locale is unsupported. Formatting requests can specify the particular components to include. Custom formats for percentages, significant digits, and currencies are supported. Numerous collation options are exposed for use in sorting text. And if you care about performance, the up-front work to select a locale and process options can now be done once, instead of once every time a locale-dependent operation is performed.

    That said, the API is not a panacea. The API is “best effort” only. Precise outputs are almost always deliberately unspecified. An implementation could legally support only the oj locale, or it could ignore (almost all) provided formatting options. Most implementations will have high-quality support for many locales, but it’s not guaranteed (particularly on resource-constrained systems such as mobile).

    Under the hood, Firefox’s implementation depends upon the International Components for Unicode library (ICU), which in turn depends upon the Unicode Common Locale Data Repository (CLDR) locale data set. Our implementation is self-hosted: most of the implementation atop ICU is written in JavaScript itself. We hit a few bumps along the way (we haven’t self-hosted anything this large before), but nothing major.

    The Intl interface

    The i18n API lives on the global Intl object. Intl contains three constructors: Intl.Collator, Intl.DateTimeFormat, and Intl.NumberFormat. Each constructor creates an object exposing the relevant operation, efficiently caching locale and options for the operation. Creating such an object follows this pattern:

    var ctor = "Collator"; // or the others
    var instance = new Intl[ctor](locales, options);

    locales is a string specifying a single language tag or an arraylike object containing multiple language tags. Language tags are strings like en (English generally), de-AT (German as used in Austria), or zh-Hant-TW (Chinese as used in Taiwan, using the traditional Chinese script). Language tags can also include a “Unicode extension”, of the form -u-key1-value1-key2-value2..., where each key is an “extension key”. The various constructors interpret these specially.

    options is an object whose properties (or their absence, by evaluating to undefined) determine how the formatter or collator behaves. Its exact interpretation is determined by the individual constructor.

    Given locale information and options, the implementation will try to produce the closest behavior it can to the “ideal” behavior. Firefox supports 400+ locales for collation and 600+ locales for date/time and number formatting, so it’s very likely (but not guaranteed) the locales you might care about are supported.

    Intl generally provides no guarantee of particular behavior. If the requested locale is unsupported, Intl allows best-effort behavior. Even if the locale is supported, behavior is not rigidly specified. Never assume that a particular set of options corresponds to a particular format. The phrasing of the overall format (encompassing all requested components) might vary across browsers, or even across browser versions. Individual components’ formats are unspecified: a short-format weekday might be “S”, “Sa”, or “Sat”. The Intl API isn’t intended to expose exactly specified behavior.

    Date/time formatting

    Options

    The primary options properties for date/time formatting are as follows:

    weekday, era
    "narrow", "short", or "long". (era refers to typically longer-than-year divisions in a calendar system: BC/AD, the current Japanese emperor’s reign, or others.)
    month
    "2-digit", "numeric", "narrow", "short", or "long"
    year
    day
    hour, minute, second
    "2-digit" or "numeric"
    timeZoneName
    "short" or "long"
    timeZone
    Case-insensitive "UTC" will format with respect to UTC. Values like "CEST" and "America/New_York" don’t have to be supported, and they don’t currently work in Firefox.

    The values don’t map to particular formats: remember, the Intl API almost never specifies exact behavior. But the intent is that "narrow", "short", and "long" produce output of corresponding size — “S” or “Sa”, “Sat”, and “Saturday”, for example. (Output may be ambiguous: Saturday and Sunday both could produce “S”.) "2-digit" and "numeric" map to two-digit number strings or full-length numeric strings: “70” and “1970”, for example.

    The final used options are largely the requested options. However, if you don’t specifically request any weekday/year/month/day/hour/minute/second, then year/month/day will be added to your provided options.

    Beyond these basic options are a few special options:

    hour12
    Specifies whether hours will be in 12-hour or 24-hour format. The default is typically locale-dependent. (Details such as whether midnight is zero-based or twelve-based and whether leading zeroes are present are also locale-dependent.)

    There are also two special properties, localeMatcher (taking either "lookup" or "best fit") and formatMatcher (taking either "basic" or "best fit"), each defaulting to "best fit". These affect how the right locale and format are selected. The use cases for these are somewhat esoteric, so you should probably ignore them.

    Locale-centric options

    DateTimeFormat also allows formatting using customized calendaring and numbering systems. These details are effectively part of the locale, so they’re specified in the Unicode extension in the language tag.

    For example, Thai as spoken in Thailand has the language tag th-TH. Recall that a Unicode extension has the format -u-key1-value1-key2-value2.... The calendaring system key is ca, and the numbering system key is nu. The Thai numbering system has the value thai, and the Chinese calendaring system has the value chinese. Thus to format dates in this overall manner, we tack a Unicode extension containing both these key/value pairs onto the end of the language tag: th-TH-u-ca-chinese-nu-thai.

    For more information on the various calendaring and numbering systems, see the full DateTimeFormat documentation.

    Examples

    After creating a DateTimeFormat object, the next step is to use it to format dates via the handy format() function. Conveniently, this function is a bound function: you don’t have to call it on the DateTimeFormat directly. Then provide it a timestamp or Date object.

    Putting it all together, here are some examples of how to create DateTimeFormat options for particular uses, with current behavior in Firefox.

    var msPerDay = 24 * 60 * 60 * 1000;
     
    // July 17, 2014 00:00:00 UTC.
    var july172014 = new Date(msPerDay * (44 * 365 + 11 + 197));

    Let’s format a date for English as used in the United States. Let’s include two-digit month/day/year, plus two-digit hours/minutes, and a short time zone to clarify that time. (The result would obviously be different in another time zone.)

    var options =
      { year: "2-digit", month: "2-digit", day: "2-digit",
        hour: "2-digit", minute: "2-digit",
        timeZoneName: "short" };
    var americanDateTime =
      new Intl.DateTimeFormat("en-US", options).format;
     
    print(americanDateTime(july172014)); // 07/16/14, 5:00 PM PDT

    Or let’s do something similar for Portuguese — ideally as used in Brazil, but in a pinch Portugal works. Let’s go for a little longer format, with full year and spelled-out month, but make it UTC for portability.

    var options =
      { year: "numeric", month: "long", day: "numeric",
        hour: "2-digit", minute: "2-digit",
        timeZoneName: "short", timeZone: "UTC" };
    var portugueseTime =
      new Intl.DateTimeFormat(["pt-BR", "pt-PT"], options);
     
    // 17 de julho de 2014 00:00 GMT
    print(portugueseTime.format(july172014));

    How about a compact, UTC-formatted weekly Swiss train schedule? We’ll try the official languages from most to least popular to choose the one that’s most likely to be readable.

    var swissLocales = ["de-CH", "fr-CH", "it-CH", "rm-CH"];
    var options =
      { weekday: "short",
        hour: "numeric", minute: "numeric",
        timeZone: "UTC", timeZoneName: "short" };
    var swissTime =
      new Intl.DateTimeFormat(swissLocales, options).format;
     
    print(swissTime(july172014)); // Do. 00:00 GMT

    Or let’s try a date in descriptive text by a painting in a Japanese museum, using the Japanese calendar with year and era:

    var jpYearEra =
      new Intl.DateTimeFormat("ja-JP-u-ca-japanese",
                              { year: "numeric", era: "long" });
     
    print(jpYearEra.format(july172014)); // 平成26年

    And for something completely different, a longer date for use in Thai as used in Thailand — but using the Thai numbering system and Chinese calendar. (Quality implementations such as Firefox’s would treat plain th-TH as th-TH-u-ca-buddhist-nu-latn, imputing Thailand’s typical Buddhist calendar system and Latin 0-9 numerals.)

    var options =
      { year: "numeric", month: "long", day: "numeric" };
    var thaiDate =
      new Intl.DateTimeFormat("th-TH-u-nu-thai-ca-chinese", options);
     
    print(thaiDate.format(july172014)); // ๒๐ 6 ๓๑

    Calendar and numbering system bits aside, it’s relatively simple. Just pick your components and their lengths.

    Number formatting

    Options

    The primary options properties for number formatting are as follows:

    style
    "currency", "percent", or "decimal" (the default) to format a value of that kind.
    currency
    A three-letter currency code, e.g. USD or CHF. Required if style is "currency", otherwise meaningless.
    currencyDisplay
    "code", "symbol", or "name", defaulting to "symbol". "code" will use the three-letter currency code in the formatted string. "symbol" will use a currency symbol such as $ or £. "name" typically uses some sort of spelled-out version of the currency. (Firefox currently only supports "symbol", but this will be fixed soon.)
    minimumIntegerDigits
    An integer from 1 to 21 (inclusive), defaulting to 1. The resulting string is front-padded with zeroes until its integer component contains at least this many digits. (For example, if this value were 2, formatting 3 might produce “03”.)
    minimumFractionDigits, maximumFractionDigits
    Integers from 0 to 20 (inclusive). The resulting string will have at least minimumFractionDigits, and no more than maximumFractionDigits, fractional digits. The default minimum is currency-dependent (usually 2, rarely 0 or 3) if style is "currency", otherwise 0. The default maximum is 0 for percents, 3 for decimals, and currency-dependent for currencies.
    minimumSignificantDigits, maximumSignificantDigits
    Integers from 1 to 21 (inclusive). If present, these override the integer/fraction digit control above to determine the minimum/maximum significant figures in the formatted number string, as determined in concert with the number of decimal places required to accurately specify the number. (Note that in a multiple of 10 the significant digits may be ambiguous, as in “100” with its one, two, or three significant digits.)
    useGrouping
    Boolean (defaulting to true) determining whether the formatted string will contain grouping separators (e.g. “,” as English thousands separator).

    NumberFormat also recognizes the esoteric, mostly ignorable localeMatcher property.

    Locale-centric options

    Just as DateTimeFormat supported custom numbering systems in the Unicode extension using the nu key, so too does NumberFormat. For example, the language tag for Chinese as used in China is zh-CN. The value for the Han decimal numbering system is hanidec. To format numbers for these systems, we tack a Unicode extension onto the language tag: zh-CN-u-nu-hanidec.

    For complete information on specifying the various numbering systems, see the full NumberFormat documentation.

    Examples

    NumberFormat objects have a format function property just as DateTimeFormat objects do. And as there, the format function is a bound function that may be used in isolation from the NumberFormat.

    Here are some examples of how to create NumberFormat options for particular uses, with Firefox’s behavior. First let’s format some money for use in Chinese as used in China, specifically using Han decimal numbers (instead of much more common Latin numbers). Select the "currency" style, then use the code for Chinese renminbi (yuan), grouping by default, with the usual number of fractional digits.

    var hanDecimalRMBInChina =
      new Intl.NumberFormat("zh-CN-u-nu-hanidec",
                            { style: "currency", currency: "CNY" });
     
    print(hanDecimalRMBInChina.format(1314.25)); // ¥ 一,三一四.二五

    Or let’s format a United States-style gas price, with its peculiar thousandths-place 9, for use in English as used in the United States.

    var gasPrice =
      new Intl.NumberFormat("en-US",
                            { style: "currency", currency: "USD",
                              minimumFractionDigits: 3 });
     
    print(gasPrice.format(5.259)); // $5.259

    Or let’s try a percentage in Arabic, meant for use in Egypt. Make sure the percentage has at least two fractional digits. (Note that this and all the other RTL examples may appear with different ordering in RTL context, e.g. ٤٣٫٨٠٪ instead of ٤٣٫٨٠٪.)

    var arabicPercent =
      new Intl.NumberFormat("ar-EG",
                            { style: "percent",
                              minimumFractionDigits: 2 }).format;
     
    print(arabicPercent(0.438)); // ٤٣٫٨٠٪

    Or suppose we’re formatting for Persian as used in Afghanistan, and we want at least two integer digits and no more than two fractional digits.

    var persianDecimal =
      new Intl.NumberFormat("fa-AF",
                            { minimumIntegerDigits: 2,
                              maximumFractionDigits: 2 });
     
    print(persianDecimal.format(3.1416)); // ۰۳٫۱۴

    Finally, let’s format an amount of Bahraini dinars, for Arabic as used in Bahrain. Unusually compared to most currencies, Bahraini dinars divide into thousandths (fils), so our number will have three places. (Again note that apparent visual ordering should be taken with a grain of salt.)

    var bahrainiDinars =
      new Intl.NumberFormat("ar-BH",
                            { style: "currency", currency: "BHD" });
     
    print(bahrainiDinars.format(3.17)); // د.ب.‏ ٣٫١٧٠

    Collation

    Options

    The primary options properties for collation are as follows:

    usage
    "sort" or "search" (defaulting to "sort"), specifying the intended use of this Collator. (A search collator might want to consider more strings equivalent than a sort collator would.)
    sensitivity
    "base", "accent", "case", or "variant". This affects how sensitive the collator is to characters that have the same “base letter” but have different accents/diacritics and/or case. (Base letters are locale-dependent: “a” and “ä” have the same base letter in German but are different letters in Swedish.) "base" sensitivity considers only the base letter, ignoring modifications (so for German “a”, “A”, and “ä” are considered the same). "accent" considers the base letter and accents but ignores case (so for German “a” and “A” are the same, but “ä” differs from both). "case" considers the base letter and case but ignores accents (so for German “a” and “ä” are the same, but “A” differs from both). Finally, "variant" considers base letter, accents, and case (so for German “a”, “ä, “ä” and “A” all differ). If usage is "sort", the default is "variant"; otherwise it’s locale-dependent.
    numeric
    Boolean (defaulting to false) determining whether complete numbers embedded in strings are considered when sorting. For example, numeric sorting might produce "F-4 Phantom II", "F-14 Tomcat", "F-35 Lightning II"; non-numeric sorting might produce "F-14 Tomcat", "F-35 Lightning II", "F-4 Phantom II".
    caseFirst
    "upper", "lower", or "false" (the default). Determines how case is considered when sorting: "upper" places uppercase letters first ("B", "a", "c"), "lower" places lowercase first ("a", "c", "B"), and "false" ignores case entirely ("a", "B", "c"). (Note: Firefox currently ignores this property.)
    ignorePunctuation
    Boolean (defaulting to false) determining whether to ignore embedded punctuation when performing the comparison (for example, so that "biweekly" and "bi-weekly" compare equivalent).

    And there’s that localeMatcher property that you can probably ignore.

    Locale-centric options

    The main Collator option specified as part of the locale’s Unicode extension is co, selecting the kind of sorting to perform: phone book (phonebk), dictionary (dict), and many others.

    Additionally, the keys kn and kf may, optionally, duplicate the numeric and caseFirst properties of the options object. But they’re not guaranteed to be supported in the language tag, and options is much clearer than language tag components. So it’s best to only adjust these options through options.

    These key-value pairs are included in the Unicode extension the same way they’ve been included for DateTimeFormat and NumberFormat; refer to those sections for how to specify these in a language tag.

    Examples

    Collator objects have a compare function property. This function accepts two arguments x and y and returns a number less than zero if x compares less than y, 0 if x compares equal to y, or a number greater than zero if x compares greater than y. As with the format functions, compare is a bound function that may be extracted for standalone use.

    Let’s try sorting a few German surnames, for use in German as used in Germany. There are actually two different sort orders in German, phonebook and dictionary. Phonebook sort emphasizes sound, and it’s as if “ä”, “ö”, and so on were expanded to “ae”, “oe”, and so on prior to sorting.

    var names =
      ["Hochberg", "Hönigswald", "Holzman"];
     
    var germanPhonebook = new Intl.Collator("de-DE-u-co-phonebk");
     
    // as if sorting ["Hochberg", "Hoenigswald", "Holzman"]:
    //   Hochberg, Hönigswald, Holzman
    print(names.sort(germanPhonebook.compare).join(", "));

    Some German words conjugate with extra umlauts, so in dictionaries it’s sensible to order ignoring umlauts (except when ordering words differing only by umlauts: schon before schön).

    var germanDictionary = new Intl.Collator("de-DE-u-co-dict");
     
    // as if sorting ["Hochberg", "Honigswald", "Holzman"]:
    //   Hochberg, Holzman, Hönigswald
    print(names.sort(germanDictionary.compare).join(", "));

    Or let’s sort a list Firefox versions with various typos (different capitalizations, random accents and diacritical marks, extra hyphenation), in English as used in the United States. We want to sort respecting version number, so do a numeric sort so that numbers in the strings are compared, not considered character-by-character.

    var firefoxen =
      ["FireFøx 3.6",
       "Fire-fox 1.0",
       "Firefox 29",
       "FÍrefox 3.5",
       "Fírefox 18"];
     
    var usVersion =
      new Intl.Collator("en-US",
                        { sensitivity: "base",
                          numeric: true,
                          ignorePunctuation: true });
     
    // Fire-fox 1.0, FÍrefox 3.5, FireFøx 3.6, Fírefox 18, Firefox 29
    print(firefoxen.sort(usVersion.compare).join(", "));

    Last, let’s do some locale-aware string searching that ignores case and accents, again in English as used in the United States.

    // Comparisons work with both composed and decomposed forms.
    var decoratedBrowsers =
      [
       "A\u0362maya",  // A͢maya
       "CH\u035Brôme", // CH͛rôme
       "FirefÓx",
       "sAfàri",
       "o\u0323pERA",  // ọpERA
       "I\u0352E",     // I͒E
      ];
     
    var fuzzySearch =
      new Intl.Collator("en-US",
                        { usage: "search", sensitivity: "base" });
     
    function findBrowser(browser)
    {
      function cmp(other)
      {
        return fuzzySearch.compare(browser, other) === 0;
      }
      return cmp;
    }
     
    print(decoratedBrowsers.findIndex(findBrowser("Firêfox"))); // 2
    print(decoratedBrowsers.findIndex(findBrowser("Safåri")));  // 3
    print(decoratedBrowsers.findIndex(findBrowser("Ãmaya")));   // 0
    print(decoratedBrowsers.findIndex(findBrowser("Øpera")));   // 4
    print(decoratedBrowsers.findIndex(findBrowser("Chromè")));  // 1
    print(decoratedBrowsers.findIndex(findBrowser("IË")));      // 5

    Odds and ends

    It may be useful to determine whether support for some operation is provided for particular locales, or to determine whether a locale is supported. Intl provides supportedLocales() functions on each constructor, and resolvedOptions() functions on each prototype, to expose this information.

    var navajoLocales =
      Intl.Collator.supportedLocalesOf(["nv"], { usage: "sort" });
    print(navajoLocales.length > 0
          ? "Navajo collation supported"
          : "Navajo collation not supported");
     
    var germanFakeRegion =
      new Intl.DateTimeFormat("de-XX", { timeZone: "UTC" });
    var usedOptions = germanFakeRegion.resolvedOptions();
    print(usedOptions.locale);   // de
    print(usedOptions.timeZone); // UTC

    Legacy behavior

    The ES5 toLocaleString-style and localeCompare functions previously had no particular semantics, accepted no particular options, and were largely useless. So the i18n API reformulates them in terms of Intl operations. Each method now accepts additional trailing locales and options arguments, interpreted just as the Intl constructors would do. (Except that for toLocaleTimeString and toLocaleDateString, different default components are used if options aren’t provided.)

    For brief use where precise behavior doesn’t matter, the old methods are fine to use. But if you need more control or are formatting or comparing many times, it’s best to use the Intl primitives directly.

    Conclusion

    Internationalization is a fascinating topic whose complexity is bounded only by the varied nature of human communication. The Internationalization API addresses a small but quite useful portion of that complexity, making it easier to produce locale-sensitive web applications. Go use it!

    (And a special thanks to Norbert Lindenberg, Anas El Husseini, Simon Montagu, Gary Kwong, Shu-yu Guo, Ehsan Akhgari, the people of #mozilla.de, and anyone I may have forgotten [sorry!] who provided feedback on this article or assisted me in producing and critiquing the examples. The English and German examples were the limit of my knowledge, and I’d have been completely lost on the other examples without their assistance. Blame all remaining errors on me. Thanks again!)