Mozilla

WebAPI Articles

Sort by:

View:

  1. Flash-Free Clipboard for the Web

    As part of our effort to grow the Web platform and make it accessible to new devices, we are trying to reduce the Web’s dependence on Flash. As part of that effort, we are standardizing and exposing useful features which are currently only available to Flash to the entirety of the Web platform.

    One of the reasons why many sites still use Flash is because of its copy and cut clipboard APIs. Flash exposes an API for programmatically copying text to the user’s clipboard on a button press. This has been used to implement handy features, such as GitHub’s “clone URL” button. It’s also useful for things such as editor UIs, which want to expose a button for copying to the clipboard, rather than requiring users to use keyboard shortcuts or the context menu.

    Unfortunately, Web APIs haven’t provided the functionality to copy text to the clipboard through JavaScript, which is why visiting GitHub with Flash disabled shows an ugly grey box where the button is supposed to be. Fortunately, we have a solution. The editor APIs provide document.execCommand as an entry point for executing editor commands. The "copy" and cut" commands have previously been disabled for web pages, but with Firefox 41, which is currently in Beta, and slated to move to release in mid-September, it is becoming available to JavaScript within user-action initiated callbacks.

    Using execCommand("cut"/"copy")

    The execCommand("cut"/"copy") API is only available during a user-triggered callback, such as a click. If you try to call it at a different time, execCommand will return false, meaning that the command failed to execute. Running execCommand("cut") will copy the current selection to the clipboard, so let’s go about implementing a basic copy-to-clipboard button.

    // button which we are attaching the event to
    var button = ...;
    // input containing the text we want to copy 
    var input = ...;
    
    button.addEventListener("click", function(event) {
      event.preventDefault();
      // Select the input node's contents
      input.select();
      // Copy it to the clipboard
      document.execCommand("copy");
    });

    That code will trigger a copy of the text in the input to the clipboard upon the click of the button in Firefox 41 and above. However, you probably want to also handle failure situations, potentially to fallback to another Flash-based approach such as ZeroClipboard, or even just to tell the user that their browser doesn’t support the functionality.

    The execCommand method will return false if the action failed, for example, due to being called outside of a user-initiated callback, but on older versions of Firefox, we would also throw a security exception if you attempted to use the "cut" or "copy" APIs. Thus, if you want to be sure that you capture all failures, make sure to surround the call in a try-catch block, and also interpret an exception as a failure.

    // button which we are attaching the event to
    var button = ...;
    // input containing the text we want to copy
    var input = ...;
    
    button.addEventListener("click", function(event) {
      event.preventDefault();
      input.select(); // Select the input node's contents
      var succeeded;
      try {
        // Copy it to the clipboard
        succeeded = document.execCommand("copy");
      } catch (e) {
        succeeded = false;
      }
      if (succeeded) {
        // The copy was successful!
      } else {
        // The copy failed :(
      }
    });

    The "cut" API is also exposed to web pages through the same mechanism, so just s/copy/cut/, and you’re all set to go!

    Feature testing

    The editor APIs provide a method document.queryCommandSupported("copy") intended to allow API consumers to determine whether a command is supported by the browser. Unfortunately, in versions of Firefox prior to 41, we returned true from document.queryCommandSupported("copy") even though the web page was unable to actually perform the copy operation. However, attempting to execute document.execCommand("copy") would throw a SecurityException. So, attempting to copy on load, and checking for this exception is probably the easiest way to feature-detect support for document.execCommand("copy") in Firefox.

    var supported = document.queryCommandSupported("copy");
    if (supported) {
      // Check that the browser isn't Firefox pre-41
      try {
        document.execCommand("copy");
      } catch (e) {
        supported = false;
      }
    }
    if (!supported) {
      // Fall back to an alternate approach like ZeroClipboard
    }

    Support in other browsers

    Google Chrome and Internet Explorer both also support this API. Chrome uses the same restriction as Firefox (that it must be run in a user-initiated callback). Internet Explorer allows it to be called at any time, except it first prompts the user with a dialog, asking for permission to access the clipboard.

    For more information about the API and browser support, see MDN documentation for document.execCommand().

  2. Pointer Events now in Firefox Nightly

    This past February Pointer Events became a W3C Recommendation. In the intervening time Microsoft Open Tech and Mozilla have been working together to implement the specification. As consumers continue to expand the range of devices that are used to explore the web with different input mechanisms such as touch, pen or mouse, it is important to provide a unified API that developers can use within their applications. In this effort we have just reached a major milestone: Pointer events are now available in Firefox Nightly. We are very excited about this effort which represents a great deal of cooperation across several browser vendors in an effort to produce a high quality industry standard API with growing support.

    Be sure to download Firefox Nightly and give it a try and give us your feedback on the implementation either using the dev-platform mailing list or the mozilla.dev.platform group. If you have feedback on the specification please send those to public-pointer-events@w3.org.

    The intent of this specification is to expand the open web to support a variety of input mechanisms beyond the mouse, while maintaining compatibility with most web-based content, which is built around mouse events. The API is designed to create one solution that will handle a variety of input devices, with a focus on pointing devices (mouse, pens, and touch). The pointer is defined in the spec as a hardware-agnostic device that can target a specific set of screen coordinates. Pointer events are intentionally similar to the current set of events associated with mouse events.

    In the current Nightly build, pointer events for mouse input are now supported. Additionally, if you’re using Windows, once you’ve set two preferences, touch events can be enabled now. The first property, Async Pan & Zoom (APZ) is enabled by setting the layers.async-pan-zoom.enabled Firefox configuration preference to true. The dom.w3c_touch_events.enabled should also be enabled by setting this value to 1 in your preferences.

    This post covers some of the basic features of the new API.

    Using the Pointer API

    Before getting started with the Pointer API, it’s important to test whether your current browser supports the API. This can be done with code similar to this example:

    if (window.PointerEvent) {
      .....
    }else{
      // use mouse events
    }

    The Pointer API provides support for pointerdown, pointerup, pointercancel, pointermove, pointerover, pointerout, gotpointercapture, and lostpointercapture events. Most of these should be familiar to you if you have coded event handling for mouse input before. For example, if you need a web app to move an image around a canvas when touched or clicked on, you can use the following code:

    function DragImage() {
        var imageGrabbed = false;
        var ctx;
        var cnv;
        var myImage;
        var x = 0;
        var y = 0;
        var rect;
        this.imgMoveEvent = function(evt) {
            if (imageGrabbed) {
                ctx.clearRect(0, 0, cnv.width, cnv.height);
                x = evt.clientX - rect.left;
                y = evt.clientY - rect.top;
                ctx.drawImage(myImage, x, y, 30, 30);
     
            }
        }
        this.imgDownEvent = function(evt) {
            //Could use canvas hit regions
            var xcl = evt.clientX - rect.left;
            var ycl = evt.clientY - rect.top;
            if (xcl > x && xcl < x + 30 && ycl > y && ycl < y + 30) {
                imageGrabbed = true;
            }
        }
        this.imgUpEvent = function(evt) {
            imageGrabbed = false;
        }
        this.initDragExample = function() {
            if (window.PointerEvent) {
                cnv = document.getElementById("myCanvas");
                ctx = cnv.getContext('2d');
                rect = cnv.getBoundingClientRect();
                x = 0;
                y = 0;
                myImage = new Image();
                myImage.onload = function() {
                    ctx.drawImage(myImage, 0, 0, 30, 30);
                };
                myImage.src = 'images/ff.jpg';
                cnv.addEventListener("pointermove", this.imgMoveEvent, false);
                cnv.addEventListener("pointerdown", this.imgDownEvent, false);
                cnv.addEventListener("pointerup", this.imgUpEvent, false);
            }
        }
    }

    PointerCapture events are used when there’s the possibility that a pointer device could leave the region of an existing element while tracking the event. For example, suppose you’re using a slider and your finger slips off the actual element –you’ll want to continue to track the pointer movements. You can set PointerCapture by using code similar to this:

    var myElement = document.getElementById("myelement");
    myelement.addEventListener("pointerdown", function(e) {
        if (this.setPointerCapture) {
        //specify the id of the point to capture
            this.setPointerCapture(e.pointerId);
        }
    }, false);

    This code guarantees that you still get pointermove events, even if you leave the region of myelement. If you do not set the PointerCapture, the pointer move events will not be called for the containing element once your pointer leaves its area. You can also release the capture by calling releasePointerCapture. The browser does this automatically when a pointerup or pointercancel event occurs.

    The Pointer Event interface

    The PointerEvent interface extends the MouseEvent interface and provides a few additional properties. These properties include pointerId, width, height, pressure, tiltX, tiltY, pointerType and isPrimary.

    The pointerId property provides a unique id for the pointer that initiates the event. The height and width properties provide respective values in CSS pixels for the contact geometry of the pointer. When the pointer happens to be a mouse, these values are set to 0. The pressure property contains a floating point value from 0 to 1 to indicate the amount of pressure applied by the pointer, where 0 is the lowest and 1 is the highest. For pointers that do not support pressure, the value is set to 0.5.

    The tiltY property contains the angle value between the X-Z planes of the pointer and the screen and ranges between -90 and 90 degrees. This property is most useful when using a stylus pen for pointer operations. A value of 0 degrees would indicate the pointer touched the surface at an exact perpendicular angle with respect to the Y-axis. Likewise the tiltX property contains the angle between the Y-Z planes.

    The pointType property contains the device type represented by the pointer. Currently this value will be set to mouse, touch, pen, unknown or an empty string.

    var myElement = document.getElementById("myelement");
    myElement.addEventListener("pointerdown", function(e) {
        switch(e.pointerType) {
            case "mouse":
                console.log("Mouse Pointer");
                break;
            case "pen":
                console.log("Pen Pointer");
                break;
            case "touch":
                console.log("Touch Pointer");
                break;
            default:
                console.log("Unknown Pointer");
        }
    }, false);

    The isPrimary property is either true or false and indicates whether the pointer is the primary pointer. A primary pointer property is required when supporting multiple touch points to provide multi-touch input and gesture support. Currently this property will be set to true for each specific pointer type (mouse, touch, pen) when the pointer first makes contact with an element that is tracking pointer events. If you are using one touch point and a mouse pointer simultaneously both will be set to true. The isPrimary property will be set to false for an event if a different pointer is already active with the same pointerType.

    var myElement = document.getElementById("myelement");
    myelement.addEventListener("pointerdown", function(e) {
        if( e.pointerType == "touch" ){
             if( e.isPrimary ){
                 //first touch
             }else{
                 //handle multi-touch
             }
        }
     
    }, false);

    Handling multi-touch

    As stated earlier, touch pointers are currently implemented only for Firefox Nightly running on Windows with layers.async-pan-zoom.enabled and dom.w3c_touch_events.enabled preferences enabled. You can check to see whether multi-touch is supported with the following code.

    if( window.maxTouchPoints && window.maxTouchPoints > 1 ){
    //supports multi-touch
    }

    Some browsers provide default functionality for certain touch interactions such as scrolling with a swipe gesture, or using a pinch gesture for zoom control. When these default actions are used, the events for the pointer will not be fired. To better support different applications, Firefox Nightly supports the CSS property touch-action. This property can be set to auto, none, pan-x, pan-y, and manipulation. Setting this property to auto will not change any default behaviors of the browser when using touch events. To disable all of the default behaviors and allow your content to handle all touch input using pointer events instead, you can set this value to none. Setting this value to either pan-x or pan-y invokes all pointer events when not panning/scrolling in a given direction. For instance, pan-x will invoke pointer event handlers when not panning/scrolling in the horizontal direction. When the property is set to manipulation, pointer events are fired if panning/scrolling or manipulating the zoom are not occurring.

    This element receives pointer events when not panning in the horizontal direction.
    // Very Simplistic pinch detector with little error detection,
    // using only x coordinates of a pointer event
     
    // Currently active pointers
    var myPointers = [];
    var lastDif = -1;
     
    function myPointerDown(evt) {
        myPointers.push(evt);
        this.setPointerCapture(evt.pointerId);
        console.log("current pointers down = " + myPointers.length);
    }
     
    //remove touch point from array when touch is released
    function myPointerUp(evt) {
        // Remove pointer from array
        for (var i = 0; i < myPointers.length; i++) {
            if (myPointers[i].pointerId == evt.pointerId) {
                myPointers.splice(i, 1);
                break;
            }
        }
        console.log("current pointers down = " + myPointers.length);
     
        if (myPointers.length < 2) {
            lastDif = -1;
        }
    }
     
    //check for a pinch using only the first two touchpoints
    function myPointerMove(evt) {
        // Update pointer position.
        for (var i = 0; i < myPointers.length; i++) {
            if (evt.pointerId = myPointers[i].pointerId) {
                myPointers[i] = evt;
                break;
            }
        }
     
        if (myPointers.length >= 2) {
            // Detect pinch gesture.
            var curDif = Math.abs(myPointers[0].clientX - myPointers[1].clientX);
            if (lastDif > 0) {
                if (curDif > lastDif) { console.log("Zoom in"); }
                if (curDif < lastDif) { console.log("Zoom out"); }
            }
            lastDif = curDif;
        }
    }

    You can test the example code here. For some great examples of the Pointer Events API in action, see Patrick H. Lauke’s collection of Touch and Pointer Events experiments on GitHub. Patrick is a member of the W3C Pointer Events Working Group, the W3C Touch Events Community Group, and Senior Accessibility Consultant for The Paciello Group.

    Conclusion

    In this post we covered some of the basics that are currently implemented in Firefox Nightly. To track the progress of this API, check out the Gecko Touch Wiki page. You can also follow along on the main feature bug and be sure to report any issues you find while testing the new Pointer API.

  3. How fast are web workers?

    The next version of Firefox OS, the mobile operating system, will unleash the power of devices by taking full advantage of their multi-core processors. Classically, JavaScript has been executed on a single thread, but web workers offer a way to execute code in parallel. Doing so frees the browser of anything that may get in the way of the main thread so that it can smoothly animate the UI.

    A brief introduction to web workers

    There are several types of web workers:

    They each have specific properties, but share a similar design. The code running in a worker is executed in its own separate thread and runs in parallel with the main thread and other workers. The different types of workers all share a common interface.

    Web workers

    Dedicated web workers are instantiated by the main process and they can communicate only with it.

    Shared workers

    Shared workers can be reached by all processes running on the same origin (different browser tabs, iframes or other shared workers).

    Service workers

    Service workers have gained a lot of attention recently. They make it possible to proxy a web server programmatically to deliver specific content to the requester (e.g. the main thread). One use case for service workers is to serve content while offline. Service workers are a very new API, not fully implemented in all browsers, and are not covered in this article.

    In order to verify that web workers make Firefox OS run faster, we need to validate their speed by benchmarking them.

    The cost of creating web workers

    This article focuses on Firefox OS. All measurement are made on a Flame device, powered by middle-end hardware.

    The first set of benchmarks will look at the time it takes to create web workers. To do that, we set up a script that instantiates a web worker and sends a minimal message, to which the worker replies immediately. Once the response is received by the main thread, the time that the operation takes is calculated. The web worker is destroyed and the operation is repeated enough times to get a good idea of how long it takes on average to get a functional web worker. Instantiating a web worker is as easy as:

    // Start a worker.
    var worker = new Worker('worker-script.js');
     
    // Terminate a worker.
    worker.terminate();

    The same method is applied to the creation of broadcast channel:

    // Open a broadcast channel.
    var channel = new window.BroadcastChannel('channel-name');
     
    // Close a broadcast channel.
    channel.close();

    Shared workers can’t really be benchmarked here because once they are created, the developer can’t destroy them. The browser is entirely responsible for their lifetime. For that reason, we can’t create and destroy shared workers at will to get a meaningful benchmark.

    Web workers take about 40 ms to be instantiated. Also, this time is pretty stable with variations of only a few milliseconds. Setting up a broadcast channel is usually done within 1 ms.

    Under normal circumstances, the browser UI is refreshed at a rate of 60 frames per second. This means that no JavaScript code should run longer than the time needed by a frame, i.e., 16.66ms (60 frames per second). Otherwise, you may introduce jankiness and lag in your application.

    Instantiating web workers is pretty efficient, but still may not fit in the time allocated for a single frame. That’s why it’s important to create as few web workers as possible and reuse them.

    Message latency

    A critical aspect of web workers is having fast communication between your main thread and the workers. There are two different ways the main browser thread can communicate with a web worker.

    postMessage

    This API is the default and preferred way to send and receive messages from a web worker. postMessage() is easy to use:

    // Send a message to the worker.
    worker.postMessage(myMessage);
     
    // Listen to messages from the worker.
    worker.onmessage = evt => {
      var message = evt.data;
    };

    Broadcast Channel

    This is a newly implemented API, only available in Firefox at the time of this writing. It lets us broadcast messages to all contexts sharing the same origin. All browser tabs, iframes, or workers served from the same origin can emit and receive messages:

    // Send a message to the broadcast channel.
    channel.postMessage(myMessage);
     
    // Listen to messages from the broadcast channel.
    channel.onmessage = evt => {
      var message = evt.data;
    };

    To benchmark this, we use a script similar to the one described above, except that the web worker is not destroyed and reused at each operation. The time to get a round trip response should be divided by two.

    As you might expect, the simple postMessage is fast. It usually takes between 0 to 1 ms to send a message, whether to a web or shared worker. Broadcast channel API takes about 1 to 2 ms.

    Under normal circumstances, exchanging messages with workers is fast and you should not feel too concerned about speed here. However, larger messages can take longer.

    The size of messages

    There are 2 ways to send messages to web workers:

    • Copying the message
    • Transferring the message

    In the first case, the message is serialized, copied, and sent over. In the latter, the data is transferred. This means that the original sender can no longer use it once sent. Transferring data is almost instantaneous, so there is no real point in benchmarking that. However, only ArrayBuffer is transferable.

    As expected, serializing, copying, and de-serializing data adds significant overhead to the message transmission. The bigger the message, the longer it takes to be sent.

    The benchmark here sends a typed array to a web worker. Its size is progressively increased at each iteration. There is a linear correlation between size of the message and transfer time. For each measurement, we can divide the size (in kilobytes) by the time (in milliseconds) to get the transfer speed in kb/ms.

    Typically, on a Flame, the transfer speed is 80 kB/ms for postMessage and 12 kB/ms using broadcast channel. This means that if you want your message to fit in a single frame, you should keep it under 1,300 kB with postMessage and under 200 kB when using the broadcast channel. Otherwise, it may introduce frame drop in your application.

    In this benchmark, we use typed arrays, because it makes it possible to determine their size in kilobytes precisely. You can also transfer JavaScript objects, but due to the serialization process, they take longer to post. For small objects, this doesn’t really matter, but if you need to send huge objects, you may as well serialize them to a binary format. You can use something similar to Protocol Buffer.

    Web workers are fast if used correctly

    Here is a quick summary of various benchmarks related to web workers, as measured on a Flame:

    Operation Value
    Instantiation of a web worker 40 ms
    Instantiation of a broadcast channel 1 ms
    Communication latency with postMessage 0.5 ms
    Communication latency with broadcast channel 1.5 ms
    Communication speed with postMessage 80 kB/ms
    Communication speed with broadcast channel 12 kB/ms
    Maximum message size with postMessage 1,300 kB
    Maximum message size with broadcast channel 200 kB

    Benchmarking is the only way to make sure that the solution you are implementing is fast. This process takes much of the guesswork out of web development.

    If you want to run these benchmarks on a specific device, the app I built to make these measurements, web workers benchmark, is open source. You are also welcome to contribute by submitting new types of benchmarks.

  4. Streaming media on demand with Media Source Extensions

    Introducing MSE

    Media Source Extensions (MSE) is a new addition to the Web APIs available in all major browsers.  This API allows for things like adaptive bitrate streaming of video directly in our browser, free of plugins. Where previously we may have used proprietary solutions like RTSP (Real Time Streaming Protocol) and Flash, we can now use simpler protocols like HTTP to fetch content, and MSE to smoothly stitch together video segments of varied quality.

    All browsers that support HTMLMediaElements, such as audio and video tags, already make byte-range requests for subsequent segments of media assets.  One problem is that it’s up to each browser’s implementation of a media engine to decide when and how much to fetch.  It’s also tough to stitch together or deliver smooth playback of segments of different quality without pauses, gaps, flashes, clicks, or pops.  MSE gives us finer-grained control at the application level for fetching and playing back content.

    In order to begin streaming, we need to figure out how to transcode our assets into a meaningful byte format for browsers’ media engines, determine what abstractions MSE provides, and figure out how to instruct the browser to play them back.

    Having multiple resolutions of content allows us to switch between them while maintaining a constant viewport size.  This is known as upscaling, and it’s a common technique for real-time rendering in video games to meet a required frame time.  By switching to a lower quality video resolution, we can meet bandwidth limitations at the cost of fidelity.  The loss of fidelity causes such artifacts as aliasing, in which curves appear jagged and blocky.  This technique can often be seen by Netflix subscribers during peak viewing hours.

    Rather than having an advanced protocol like RTSP handle bandwidth estimates, we can use a simpler network protocol like HTTP and move the advanced logic up one level into the application logic.

    Transcoding

    My recommended tools, ffmpeg and Bento4, are both free and open-source software (FOSS). ffmpeg is our Swiss army knife of transcoding, and Bento4 is a collection of great tools for working with mp4.  While I’m partial to non-licensed codecs like webm/vp8-9/opus, current browser support for those containers and codecs is rather poor, so in this post we’ll just be working with mp4/h.264/aac.  Both of the tools I’m working with are command line utilities; if you have nice GUI tools in your asset pipeline you’d like to recommend to our readers, let us know in the comments below.

    We’ll start with a master of some file, and end up transcoding it into multiple files each of smaller resolutions, then segmenting the smaller-res whole files into a bunch of tiny files.  Once we have a bunch of small files (imagine splitting your video into a bunch of 10-second segments), the client can use more advanced heuristics for fetching the preferred next segment.

    MSE multiple resolutions

    Our smaller-res copies of the master asset

     Proper fragmentation

    When working with mp4 and MSE, it helps to know that the mp4 files should be structured so that metadata is fragmented across pieces of the container, and across the actual audio/video streams being fragmented, instead of clustered together.  This is specified in the ISO BMFF Byte Stream Format spec, section 3:

    “An ISO BMFF initialization segment is defined in this specification as a single File Type Box (ftyp) followed by a single Movie Header Box (moov).”

    This is really important: Simply transcoding to an mp4 container in ffmpeg does not have the expected format and thus fails when trying to play back in a browser with MSE.  To check and see if your mp4 is properly fragmented, you can run Bento4’s mp4dump on your mp4.

    If you see something like:

      $ ./mp4dump ~/Movies/devtools.mp4 | head
      [ftyp] size=8+24
        ...
      [free] size=8+0
      [mdat] size=8+85038690
      [moov] size=8+599967
        ...

    Then your mp4 won’t be playable since the [ftyp] “atom” is not followed immediately by a [moov] “atom.”  A properly fragmented mp4 looks something like this —

      $ ./mp4fragment ~/Movies/devtools.mp4 devtools_fragmented.mp4
      $ ./mp4dump devtools_fragmented.mp4 | head
      [ftyp] size=8+28
        ...
      [moov] size=8+1109
        ...
      [moof] size=8+600
        ...
      [mdat] size=8+138679
      [moof] size=8+536
        ...
      [mdat] size=8+24490
        ...
      ...

    — where mp4fragment is another Bento4 utility.  The properly fragmented mp4 has the [ftyp] followed immediately by a [moov], then subsequent [moof]/[mdat] pairs.

    It’s possible to skip the need for mp4fragment by using the -movflags frag_keyframe+empty_moov flags when transcoding to an mp4 container with ffmpeg, then checking with mp4dump:

      $ ffmpeg -i bunny.y4m -movflags frag_keyframe+empty_moov bunny.mp4
    Creating multiple resolutions

    If we want to switch resolutions, we can then run our fragmented mp4 through Bento4’s mp4-dash-encode.py script to get multiple resolutions of our video.  This script will fire up ffmpeg and other Bento4 tools, so make sure they are both available in your $PATH environment variable.

    $ python2.7 mp4-dash-encode.py -b 5 bunny.mp4
    $ ls
    video_00500.mp4 video_00875.mp4 video_01250.mp4 video_01625.mp4 video_02000.mp4
    Segmenting

    We now have 5 different copies of our video with various bit rates and resolutions. To be able to switch between them easily during playback, based on our effective bandwidth that changes constantly over time, we need to segment the copies and produce a manifest file to facilitate playback on the client.  We’ll create a Media Presentation Description (MPD)-style manifest file. This manifest file containing info about the segments, such as the threshold effective bandwidth for fetching the requisite segment.

    Bento4’s mp4-dash.py script can take multiple input files, perform the segmentation, and emit a MPD manifest that most DASH clients/libraries understand.

    $ python2.7 mp4-dash.py --exec-dir=. video_0*
    ...
    $ tree -L 1 output
    output
    ├── audio
    │   └── und
    ├── stream.mpd
    └── video
        ├── 1
        ├── 2
        ├── 3
        ├── 4
        └── 5
    
    8 directories, 1 file

    We should now have a folder with segmented audio and segmented video of various resolutions.

    MSE & Playback

    With an HTMLMediaElement such as an audio or video tag, we simply assign a URL to the element’s src attribute and the browser handles fetching and playback.  With MSE, we will fetch the content ourselves with XMLHttpRequests (XHRs) treating the response as an ArrayBuffer (raw bytes), and assigning the src attribute of the media element to a URL that points to a MediaSource object.  We may then append SourceBuffer objects to the MediaSource.

    Pseudocode for the MSE workflow might look like:

    let m = new MediaSource
    m.onsourceopen = () =>
      let s = m.addSourceBuffer('codec')
      s.onupdateend = () =>
        if (numChunks === totalChunks)
          m.endOfStream()
        else
          s.appendBuffer(nextChunk)
      s.appendBuffer(arrayBufferOfContent)
    video.src = URL.createObjectURL(m)
    

    Here’s a trick to get the size of a file: make an XHR with the HTTP HEAD method.  A response to a HEAD request will have the content-length header specifying the body size of the response, but unlike a GET, it does not actually have a body.  You can use this to preview the size of a file without actually requesting the file contents.  We can naively subdivide the video and fetch the next segment of video when we’re 80% of the way through playback of the current segment.  Here’s a demo of this in action and a look at the code.

    Note: You’ll need the latest Firefox Developer Edition browser to view the demo and test the code. More information below in the Compatibility section. The MSE primer from WebPlatform.org docs is another great resource to consult.

    My demo is a little naive and has a few issues:

    • It doesn’t show how to properly handle seeking during playback.
    • It assumes bandwidth is constant (always fetching the next segment at 80% playback of the previous segment), which it isn’t.
    • It starts off by loading only one segment (it might be better to fetch the first few, then wait to fetch the rest).
    • It doesn’t switch between segments of varying resolution, instead only fetching segments of one quality.
    • It doesn’t remove segments (part of the MSE API), although this can be helpful on memory constrained devices. Unfortunately, this requires you to re-fetch content when seeking backwards.

    These issues can all be solved with smarter logic on the client side with Dynamic Adaptive Streaming over HTTP (DASH).

    Compatibility

    Cross-browser codec support is a messy story right now; we can use MediaSource.isTypeSupported to detect codec support.  You pass isTypeSupported a string of the MIME type of the container you’re looking to play.  mp4 has the best compatibility currently. Apparently, for browsers that use the Blink rendering engine, MediaSource.isTypeSupported requires the full codec string to be specified.  To find this string, you can use Bento4’s mp4info utility:

    ./mp4info bunny.mp4| grep Codec
        Codecs String: avc1.42E01E

    Then in our JavaScript:

    if (MediaSource.isTypeSupported('video/mp4; codecs="avc1.42E01E, mp4a.40.2"')) {
    // we can play this
    }

    — where mp4a.40.2 is the codec string for low complexity AAC, the typical audio codec used in an mp4 container.

    Some browsers also currently whitelist certain domains for testing MSE, or over-aggressively cache CORS content, which makes testing frustratingly difficult.  Consult your browser for how to disable the whitelist or CORS caching when testing.

    DASH

    Using the MPD file we created earlier, we can grab a high quality DASH client implemented in JavaScript such as Shaka Player or dash.js.  Both clients implement numerous features, but could use more testing, as there are some subtle differences between media engines of various browsers.  Advanced clients like Shaka Player use an exponential moving average of three to ten samples to estimate the bandwidth, or even let you specify your own bandwidth estimator.

    If we serve our output directory created earlier with Cross Origin Resource Sharing (CORS) enabled, and point either DASH client to http://localhost:<port>/output/stream.mpd, we should be able to see our content playing.  Enabling video cycling in Shaka, or clicking the +/- buttons in dash.js should allow us to watch the content quality changing.  For more drastic/noticeable changes in quality, try encoding fewer bitrates than the five we demonstrated.

    Shaka Player in Firefox Dev Edition

    Shaka Player in Firefox Developer Edition

    dash.js running in Firefox Developer Edition

    dash.js in Firefox Developer Edition

    In conclusion

    In this post, we looked at how to prep video assets for on-demand streaming by pre-processing and transcoding.  We also took a peek at the MSE API, and how to use more advanced DASH clients.  In an upcoming post, we’ll explore live content streaming using the MSE API, so keep an eye out.  I recommend you use Firefox Developer Edition to test out MSE; lots of hard work is going into our implementation.

    Here are some additional resources for exploring MSE:

  5. Let’s get charged: Updates to the Battery Status API

    Web APIs provide a way for Open Web Apps to access device hardware, data and sensors through JavaScript, and open the doors to a number of possibilities especially for mobile devices, TVs, interactive kiosks, and Internet of Things (IoT) applications.

    Knowing the battery status of a device can be useful in a number of situations or use cases. Here are some examples:

    • Utility apps that collect statistics on battery usage or simply inform the user if the device is charged enough to play a game, watch a movie, or browse the Web.
    • High-quality apps that optimize battery consumption: for example, an email client may check the server for new email less frequently if the device is low on battery.
    • A word processor could save changes automatically before the battery runs out in order to prevent data loss.
    • A system checking if an interactive kiosk or TV installed in a showroom of an event is charging or if something wrong happened with the cables
    • A module that checks the battery status of a drone in order to make it come back to the base before it runs out of power.

    This article looks at a standardized way to manage energy consumption: The Battery Status API.

    The Battery Status API

    Open Web Apps can retrieve battery status information thanks to the Battery Status API, a W3C Recommendation supported by Firefox since version 16. The API is also supported by Firefox OS, and recently by Chrome, Opera, and the Android browser, so now it can be used in production across many major platforms.

    Also, the W3C Recommendation has recently been improved, introducing Promise Objects, and the ability to handle multiple batteries installed on the same device.

    At the time of writing, this W3C update has not yet been implemented by Firefox: please check the following bugs for implementation updates or in case you want to contribute to Gecko development:

    • [1050749] Expose BatteryManager via getBattery() returning a Promise instead of a synchronous accessor (navigator.battery)
    • [1050752] BatteryManager: specify the behavior when a host device has more than one battery

    Below we will look at using the Battery Status API in an instant messaging app running on Firefox OS and all the browsers that currently support the API.

    Demo: Low Energy Messenger

    Low Energy Messenger is an instant messaging demo app that pays close attention to battery status. The app has been built with HTML + CSS + Javascript (no libraries) and uses static data. It does not include web services running on the Internet, but includes real integration with the Battery Status API and has a realistic look & feel.

    You’ll find a working demo of Low Energy Messenger, along with the demo code on Github, and an >MDN article called Retrieving Battery status information that explains the code step-by-step.

    Low Energy Messenger

    Low Energy Messenger has the following features:

    • A battery status bar, containing battery status information.
    • A chat section, containing all the messages received or sent.
    • An action bar, containing a text field, a button to send a message, a button to take a photo, and a button to install the app on Firefox OS
    • In order to preserve battery life when the power level is low, the app doesn’t allow users to take photos when the device is running out of battery.

    The visual representation of the battery, in the app’s status bar, changes depending on the charge level. For example:

    13% Discharging: 0:23 remaining
    40% Discharging: 1:19 remaining
    92% Charging: 0:16 until full

    Low Energy Messenger includes a module called EnergyManager.js that uses the Battery Status API to get the information displayed above and perform checks.

    The battery object, of type BatteryManager, is provided by the navigator.getBattery method, using Promises, or by the deprecated navigator.battery property, part of a previous W3C specification and currently used by Firefox and Firefox OS. As mentioned above, follow this bug for implementation updates or if you want to contribute to Gecko development.

    The EnergyManager.js module eliminates this difference in API implementation in the following way:

    /* EnergyManager.js */
     init: function(callback) {
         var _self = this;
        /* Initialize the battery object */
        if (navigator.getBattery) {
           navigator.getBattery().then(function(battery) {
               _self.battery = battery;
               callback();
           });
        } else if (navigator.battery || navigator.mozBattery) { // deprecated battery objects
            _self.battery = navigator.battery || navigator.mozBattery;
            callback();
        }
     }

    The navigator.getBattery method returns a battery promise, which is resolved in a BatteryManager object providing events you can handle to monitor the battery status. The deprecated navigator.battery attribute returns the BatteryManager object directly; the implementation above checks for vendor prefixes as well, for even older, experimental, API implementations carried on by Mozilla in earlier stages of the specification.

    Logging into the Web Console of a browser is a useful way to understand how the Battery Status API actually works:

    /* EnergyManager.js */
     log: function(event) {
        if (event) {
            console.warn(event);
        }
        console.log('battery.level: ' + this.battery.level);
        console.log('battery.charging: ' + this.battery.charging);
        console.log('battery.chargingTime: ' + this.battery.chargingTime);
        console.log('battery.dischargingTime: ' + this.battery.dischargingTime);
     }

    Here is how the logs appear on the Web Console:

    Web Console

    Every time an event (dischargingtimechange, levelchange, etc.) gets fired, the BatteryManager object provides updated values that can be used by the application for any purpose.

    Conclusions

    The Battery Status API is a standardized way to access the device hardware and is ready to be used in production, even if at the time of writing some compatibility checks still have to be performed on Firefox. Also, the W3C specification is generic enough to be used in different contexts, thus the API covers a good number of real-world use cases.

  6. How TV Functionality Leverages Web Technology

    The convergence of Internet-based IPTV, Video-on-Demand (VoD) and traditional broadcasting is happening now. As more and more web technology comes to television, the gap between web apps and native apps is rapidly narrowing.

    Firefox OS now supports the TV Manager API, a baseline of the W3C TV Control API (the editor’s draft driven by the TV Control API Community Group), which allows web-based applications to acquire information such as the Electronic Program Guide (or EPG) from service providers, and also to manage native TV hardware such as tuners and remotes. Firefox OS also relies on multiple API specifications to fully support TV functionality on a web platform. Some APIs are still on the road to becoming standards. Here is an overview of some common TV functions which can be implemented today via these Web APIs.

    FxOS_TV

    Tuner, channel, and EPG management

    One of the primary goals of the TV Control API is to enable standard use cases, such as channel switching and other activities generally available and expected in conventional remote controls. The API exposes specific tuner, channel, and EPG information to the web app and also allows indirect interactions with the tuner modules. Some applications, like virtual remote control, are easy to enable with this API. And the web content has the flexibility to manipulate multiple tuners if resource-permitted. In addition, event-driven notifications about channel scanning status and EPG updates provide a dynamic way to access broadcasting information.

    The following example shows how to get the available tuners and set the current playing source via the API.

        // Retrieve all the available tuners.
        navigator.tv.getTuners().then(function(tuners) {
          // Just use the first tuner.
          var tuner = tuners[0];
    
          // Set the 'oncurrentsourcechanged' event handler.
          tuner.oncurrentsourcechanged = function(event) {
            // The current source has changed.
          };
    
          // Get the supported source types for the tuner. 
          var sourceTypes = tuner.getSupportedSourceTypes();
    
          // Just use the first source.
          tuner.setCurrentSource(sourceTypes[0]).then(function() {
            // Succeeded to set the source.
          }, function(error) {
            // Failed to set the source.
          });
        }, function(error) {
          // Failed to get tuners.
        });
    

    Streaming

    When it comes to TV streaming, multiple web specifications work together. The Media Capture and Streams API enables access to multimedia streams from local devices, and it defines a MediaStream interface to carry video/audio tracks and other stream-relevant attributes. The current streaming data for each tuner can be wrapped in a MediaStream instance and retrieved via the TV Control API. Then the web content can start playing the stream by assigning it to HTML5 <video> elements. Here’s an example:

        // Retrieve all the available TV tuners and then play
        // the stream for the first tuner.
        navigator.tv.getTuners().then(function(tuners) {
          // Assuming a video element is already accessible to
          // variable |video|, just assign the stream to it.
          video.srcObject = tuners[0].stream;
        });
    

    The Media Capture and Streams API and the HTML5 <video> element provide attributes and methods to help manage streaming contents. Combining the APIs and HTML5 elements makes the development of some advanced TV streaming scenarios, such as concurrent display for multiple streams (i.e., Picture-in-Picture), easier to accomplish and more flexible.

    Track management

    While playing video streams, sometimes the media content provides multiple audio channels and even includes subtitles. For instance, a film or a sports show may be broadcast with multiple supported languages available for the viewer to select. Thus, there are two lists of VideoTrack and AudioTrack objects in the HTML5 <video> element to help the web content utilize available video and audio tracks respectively. Meanwhile, the Media Capture and Streams API also provides certain methods to dynamically manage associated media tracks. With this flexibility, web content is able to play a video stream that’s originated from another connected device, such as a camera.

    The HTML5 <video> element has a list of TextTrack instances. These tracks can be associated with a specific video tag to represent different instances, such as subtitles, captions, and metadata. In addition, multiple TextTrackCue objects containing time-sensitive data can be affiliated with them, and used to control timing of the display for subtitles and captions.

    Recording

    The MediaStream Recording API provides a MediaRecorder interface to fulfill recording requirements over the MediaStream. It lets the web content select the appropriate supported encoding format for the stream, and also enables the recorded media to be accessed either as a whole blob or in smaller buffers.

    In cooperation with the Task Scheduler API and Service Workers, you can even add DVR-style scheduling functionality to your web app or web-driven device. On the other hand, the EPG data retrieved via TV Control API may also help the application record specific TV programs.

    Emergency alert

    Sometimes a public alert may be sent out via TV broadcasting or other communication systems when an emergency or natural disaster occurs such an earthquake, tsunami, or hurricane. A corresponding event-triggering mechanism included with the TV Control API can be used to help a web application manage an alert.

       // Assuming variable |currentSource| is associated with
        // the playing TV signal.
        currentSource.onemergencyalerted = function (event) {
          var type = event.type; // i.e. “Earthquake”.
          var severityLevel = event.severityLevel;
          var description = event.description;
          // Display the emergency information.
          // (More attributes could be available.)
        };
    

    Final thoughts

    TV broadcast mechanisms and applications are on a fast track to evolve rapidly, as is the web technology. Since many of the web specifications and APIs mentioned are still in the standardization progress, more functionality or further improvements might be added to make television experience more webby. As always, you can contribute to the evolution of these Web APIs by following along on relevant standards mailing lists and filing bugs when you find them.

  7. This API is so Fetching!

    For more than a decade the Web has used XMLHttpRequest (XHR) to achieve asynchronous requests in JavaScript. While very useful, XHR is not a very nice API. It suffers from lack of separation of concerns. The input, output and state are all managed by interacting with one object, and state is tracked using events. Also, the event-based model doesn’t play well with JavaScript’s recent focus on Promise- and generator-based asynchronous programming.

    The Fetch API intends to fix most of these problems. It does this by introducing the same primitives to JS that are used in the HTTP protocol. In addition, it introduces a utility function fetch() that succinctly captures the intention of retrieving a resource from the network.

    The Fetch specification, which defines the API, nails down the semantics of a user agent fetching a resource. This, combined with ServiceWorkers, is an attempt to:

    1. Improve the offline experience.
    2. Expose the building blocks of the Web to the platform as part of the extensible web movement.

    As of this writing, the Fetch API is available in Firefox 39 (currently Nightly) and Chrome 42 (currently dev). Github has a Fetch polyfill.

    Feature detection

    Fetch API support can be detected by checking for Headers,Request, Response or fetch on the window or worker scope.

    Simple fetching

    The most useful, high-level part of the Fetch API is the fetch() function. In its simplest form it takes a URL and returns a promise that resolves to the response. The response is captured as a Response object.

    fetch("/data.json").then(function(res) {
      // res instanceof Response == true.
      if (res.ok) {
        res.json().then(function(data) {
          console.log(data.entries);
        });
      } else {
        console.log("Looks like the response wasn't perfect, got status", res.status);
      }
    }, function(e) {
      console.log("Fetch failed!", e);
    });

    Submitting some parameters, it would look like this:

    fetch("http://www.example.org/submit.php", {
      method: "POST",
      headers: {
        "Content-Type": "application/x-www-form-urlencoded"
      },
      body: "firstName=Nikhil&favColor=blue&password=easytoguess"
    }).then(function(res) {
      if (res.ok) {
        alert("Perfect! Your settings are saved.");
      } else if (res.status == 401) {
        alert("Oops! You are not authorized.");
      }
    }, function(e) {
      alert("Error submitting form!");
    });

    The fetch() function’s arguments are the same as those passed to the
    Request() constructor, so you may directly pass arbitrarily complex requests to fetch() as discussed below.

    Headers

    Fetch introduces 3 interfaces. These are Headers, Request and
    Response. They map directly to the underlying HTTP concepts, but have
    certain visibility filters in place for privacy and security reasons, such as
    supporting CORS rules and ensuring cookies aren’t readable by third parties.

    The Headers interface is a simple multi-map of names to values:

    var content = "Hello World";
    var reqHeaders = new Headers();
    reqHeaders.append("Content-Type", "text/plain"
    reqHeaders.append("Content-Length", content.length.toString());
    reqHeaders.append("X-Custom-Header", "ProcessThisImmediately");

    The same can be achieved by passing an array of arrays or a JS object literal
    to the constructor:

    reqHeaders = new Headers({
      "Content-Type": "text/plain",
      "Content-Length": content.length.toString(),
      "X-Custom-Header": "ProcessThisImmediately",
    });

    The contents can be queried and retrieved:

    console.log(reqHeaders.has("Content-Type")); // true
    console.log(reqHeaders.has("Set-Cookie")); // false
    reqHeaders.set("Content-Type", "text/html");
    reqHeaders.append("X-Custom-Header", "AnotherValue");
     
    console.log(reqHeaders.get("Content-Length")); // 11
    console.log(reqHeaders.getAll("X-Custom-Header")); // ["ProcessThisImmediately", "AnotherValue"]
     
    reqHeaders.delete("X-Custom-Header");
    console.log(reqHeaders.getAll("X-Custom-Header")); // []

    Some of these operations are only useful in ServiceWorkers, but they provide
    a much nicer API to Headers.

    Since Headers can be sent in requests, or received in responses, and have various limitations about what information can and should be mutable, Headers objects have a guard property. This is not exposed to the Web, but it affects which mutation operations are allowed on the Headers object.
    Possible values are:

    • “none”: default.
    • “request”: guard for a Headers object obtained from a Request (Request.headers).
    • “request-no-cors”: guard for a Headers object obtained from a Request created
      with mode “no-cors”.
    • “response”: naturally, for Headers obtained from Response (Response.headers).
    • “immutable”: Mostly used for ServiceWorkers, renders a Headers object
      read-only.

    The details of how each guard affects the behaviors of the Headers object are
    in the specification. For example, you may not append or set a “request” guarded Headers’ “Content-Length” header. Similarly, inserting “Set-Cookie” into a Response header is not allowed so that ServiceWorkers may not set cookies via synthesized Responses.

    All of the Headers methods throw TypeError if name is not a valid HTTP Header name. The mutation operations will throw TypeError if there is an immutable guard. Otherwise they fail silently. For example:

    var res = Response.error();
    try {
      res.headers.set("Origin", "http://mybank.com");
    } catch(e) {
      console.log("Cannot pretend to be a bank!");
    }

    Request

    The Request interface defines a request to fetch a resource over HTTP. URL, method and headers are expected, but the Request also allows specifying a body, a request mode, credentials and cache hints.

    The simplest Request is of course, just a URL, as you may do to GET a resource.

    var req = new Request("/index.html");
    console.log(req.method); // "GET"
    console.log(req.url); // "http://example.com/index.html"

    You may also pass a Request to the Request() constructor to create a copy.
    (This is not the same as calling the clone() method, which is covered in
    the “Reading bodies” section.).

    var copy = new Request(req);
    console.log(copy.method); // "GET"
    console.log(copy.url); // "http://example.com/index.html"

    Again, this form is probably only useful in ServiceWorkers.

    The non-URL attributes of the Request can only be set by passing initial
    values as a second argument to the constructor. This argument is a dictionary.

    var uploadReq = new Request("/uploadImage", {
      method: "POST",
      headers: {
        "Content-Type": "image/png",
      },
      body: "image data"
    });

    The Request’s mode is used to determine if cross-origin requests lead to valid responses, and which properties on the response are readable. Legal mode values are "same-origin", "no-cors" (default) and "cors".

    The "same-origin" mode is simple, if a request is made to another origin with this mode set, the result is simply an error. You could use this to ensure that
    a request is always being made to your origin.

    var arbitraryUrl = document.getElementById("url-input").value;
    fetch(arbitraryUrl, { mode: "same-origin" }).then(function(res) {
      console.log("Response succeeded?", res.ok);
    }, function(e) {
      console.log("Please enter a same-origin URL!");
    });

    The "no-cors" mode captures what the web platform does by default for scripts you import from CDNs, images hosted on other domains, and so on. First, it prevents the method from being anything other than “HEAD”, “GET” or “POST”. Second, if any ServiceWorkers intercept these requests, they may not add or override any headers except for these. Third, JavaScript may not access any properties of the resulting Response. This ensures that ServiceWorkers do not affect the semantics of the Web and prevents security and privacy issues that could arise from leaking data across domains.

    "cors" mode is what you’ll usually use to make known cross-origin requests to access various APIs offered by other vendors. These are expected to adhere to
    the CORS protocol. Only a limited set of headers is exposed in the Response, but the body is readable. For example, you could get a list of Flickr’s most interesting photos today like this:

    var u = new URLSearchParams();
    u.append('method', 'flickr.interestingness.getList');
    u.append('api_key', '<insert api key here>');
    u.append('format', 'json');
    u.append('nojsoncallback', '1');
     
    var apiCall = fetch('https://api.flickr.com/services/rest?' + u);
     
    apiCall.then(function(response) {
      return response.json().then(function(json) {
        // photo is a list of photos.
        return json.photos.photo;
      });
    }).then(function(photos) {
      photos.forEach(function(photo) {
        console.log(photo.title);
      });
    });

    You may not read out the “Date” header since Flickr does not allow it via
    Access-Control-Expose-Headers.

    response.headers.get("Date"); // null

    The credentials enumeration determines if cookies for the other domain are
    sent to cross-origin requests. This is similar to XHR’s withCredentials
    flag, but tri-valued as "omit" (default), "same-origin" and "include".

    The Request object will also give the ability to offer caching hints to the user-agent. This is currently undergoing some security review. Firefox exposes the attribute, but it has no effect.

    Requests have two read-only attributes that are relevant to ServiceWorkers
    intercepting them. There is the string referrer, which is set by the UA to be
    the referrer of the Request. This may be an empty string. The other is
    context which is a rather large enumeration defining what sort of resource is being fetched. This could be “image” if the request is from an tag in the controlled document, “worker” if it is an attempt to load a worker script, and so on. When used with the fetch() function, it is “fetch”.

    Response

    Response instances are returned by calls to fetch(). They can also be created by JS, but this is only useful in ServiceWorkers.

    We have already seen some attributes of Response when we looked at fetch(). The most obvious candidates are status, an integer (default value 200) and statusText (default value “OK”), which correspond to the HTTP status code and reason. The ok attribute is just a shorthand for checking that status is in the range 200-299 inclusive.

    headers is the Response’s Headers object, with guard “response”. The url attribute reflects the URL of the corresponding request.

    Response also has a type, which is “basic”, “cors”, “default”, “error” or
    “opaque”.

    • "basic": normal, same origin response, with all headers exposed except
      “Set-Cookie” and “Set-Cookie2″.
    • "cors": response was received from a valid cross-origin request. Certain headers and the body may be accessed.
    • "error": network error. No useful information describing the error is available. The Response’s status is 0, headers are empty and immutable. This is the type for a Response obtained from Response.error().
    • "opaque": response for “no-cors” request to cross-origin resource. Severely
      restricted

    The “error” type results in the fetch() Promise rejecting with TypeError.

    There are certain attributes that are useful only in a ServiceWorker scope. The
    idiomatic way to return a Response to an intercepted request in ServiceWorkers is:

    addEventListener('fetch', function(event) {
      event.respondWith(new Response("Response body", {
        headers: { "Content-Type" : "text/plain" }
      });
    });

    As you can see, Response has a two argument constructor, where both arguments are optional. The first argument is a body initializer, and the second is a dictionary to set the status, statusText and headers.

    The static method Response.error() simply returns an error response. Similarly, Response.redirect(url, status) returns a Response resulting in
    a redirect to url.

    Dealing with bodies

    Both Requests and Responses may contain body data. We’ve been glossing over it because of the various data types body may contain, but we will cover it in detail now.

    A body is an instance of any of the following types.

    In addition, Request and Response both offer the following methods to extract their body. These all return a Promise that is eventually resolved with the actual content.

    • arrayBuffer()
    • blob()
    • json()
    • text()
    • formData()

    This is a significant improvement over XHR in terms of ease of use of non-text data!

    Request bodies can be set by passing body parameters:

    var form = new FormData(document.getElementById('login-form'));
    fetch("/login", {
      method: "POST",
      body: form
    })

    Responses take the first argument as the body.

    var res = new Response(new File(["chunk", "chunk"], "archive.zip",
                           { type: "application/zip" }));

    Both Request and Response (and by extension the fetch() function), will try to intelligently determine the content type. Request will also automatically set a “Content-Type” header if none is set in the dictionary.

    Streams and cloning

    It is important to realise that Request and Response bodies can only be read once! Both interfaces have a boolean attribute bodyUsed to determine if it is safe to read or not.

    var res = new Response("one time use");
    console.log(res.bodyUsed); // false
    res.text().then(function(v) {
      console.log(res.bodyUsed); // true
    });
    console.log(res.bodyUsed); // true
     
    res.text().catch(function(e) {
      console.log("Tried to read already consumed Response");
    });

    This decision allows easing the transition to an eventual stream-based Fetch API. The intention is to let applications consume data as it arrives, allowing for JavaScript to deal with larger files like videos, and perform things like compression and editing on the fly.

    Often, you’ll want access to the body multiple times. For example, you can use the upcoming Cache API to store Requests and Responses for offline use, and Cache requires bodies to be available for reading.

    So how do you read out the body multiple times within such constraints? The API provides a clone() method on the two interfaces. This will return a clone of the object, with a ‘new’ body. clone() MUST be called before the body of the corresponding object has been used. That is, clone() first, read later.

    addEventListener('fetch', function(evt) {
      var sheep = new Response("Dolly");
      console.log(sheep.bodyUsed); // false
      var clone = sheep.clone();
      console.log(clone.bodyUsed); // false
     
      clone.text();
      console.log(sheep.bodyUsed); // false
      console.log(clone.bodyUsed); // true
     
      evt.respondWith(cache.add(sheep.clone()).then(function(e) {
        return sheep;
      });
    });

    Future improvements

    Along with the transition to streams, Fetch will eventually have the ability to abort running fetch()es and some way to report the progress of a fetch. These are provided by XHR, but are a little tricky to fit in the Promise-based nature of the Fetch API.

    You can contribute to the evolution of this API by participating in discussions on the WHATWG mailing list and in the issues in the Fetch and ServiceWorker specifications.

    For a better web!

    The author would like to thank Andrea Marchesini, Anne van Kesteren and Ben
    Kelly for helping with the specification and implementation.

  8. Birdsongs, Musique Concrète, and the Web Audio API

    In January 2015, my friend and collaborator Brian Belet and I presented Oiseaux de Même — an audio soundscape app created from recordings of birds — at the first Web Audio Conference. In this post I’d like to describe my experience of implementing this app using the Web Audio API, Twitter Bootstrap, Node.js, and REST APIs.

    Screenshot showing Birds of a Feather, a soundscape created with field recordings of birds that are being seen in your vicinity.

    Screenshot showing Birds of a Feather, a soundscape created with field recordings of birds that are being seen in your vicinity.

    What is it? Musique Concrète and citizen science

    We wanted to create a web-based Musique Concrète, building an artistic sound experience by processing field recordings. We decided to use xeno-canto — a library of over 200,000 recordings of 9,000 different bird species — as our source of recordings. Almost all the recordings are licensed under Creative Commons by their generous recordists. We select recordings from this library based on data from eBird, a database of tens of millions of bird sightings contributed by bird watchers everywhere. By using the Geolocation API to retrieve eBird sightings near to the listeners’ location, our soundscape can consist of recordings of bird species that bird watchers have reported recently near the listener — each user gets a personalized soundscape that changes daily.

    Use of the Web Audio API

    We use the browser’s Web Audio API to play back the sounds from xeno-canto. The Web Audio API allows developers to play back, record, analyze, and process sound by creating AudioNodes that are connected together, like an old modular synthesizer.

    Our soundscape is implemented using four AudioBuffer nodes, each of which plays a field recording in a loop. These loops are placed in a stereo field using Panner nodes, and mixed together before being sent to the listener’s speakers or headphones.

    Controls

    After all the sounds have loaded and begin playing, we offer users several controls for manipulating the sounds as they play:

    • The Pan button randomizes the spatial location of the sound in 3D space.
    • The Rate button randomizes the playback rate.
    • The Reverse button reverses the direction of sound playback.
    • Finally, the Share button lets you capture the state of the soundscape and save that snapshot for later.

    The controls described above are implemented as typical JavaScript event handlers. When the Pan button is pressed, for example, we run this handler:

    // sets the X,Y,Z position of the Panner to random values between -1 and +1
    BirdSongPlayer.prototype.randomizePanner = function() {
      this.resetLastActionTime();
      // NOTE: x = -1 is LEFT
      this.panPosition = { x: 2 * Math.random() - 1, y: 2 * Math.random() - 1, z: 2 * Math.random() - 1}
      this.panner.setPosition( this.panPosition.x, this.panPosition.y, this.panPosition.z);
    }

    Some parts of the Web Audio API are write-only

    I had a few minor issues where I had to work around shortcomings in the Web Audio API. Other authors have already documented similar experiences; I’ll summarize mine briefly here:

    • Can’t read Panner position: In the event handler for the Share button, I want to retrieve and store the current Audio Buffer playback rate and Panner position. However, the current Panner node does not allow retrieval of the position after setting it. Hence, I store the new Panner position in an instance variable in addition to calling setPosition().

      This has had a minimal impact on my code so far. My longer-term concern is that I’d rather store the position in the Panner and retrieve it from there, instead of storing a copy elsewhere. In my experience, multiple copies of the same information becomes a readability and maintainability problem as code grows bigger and more complex.

    • Can’t read AudioBuffer’s playbackRate: The Rate button described above calls linearRampToValueAtTime() on the playbackRate AudioParam. As far as I can tell, AudioParams don’t let me retrieve their values after calling linearRampToValueAtTime(), so I’m obliged to keep a duplicate copy of this value in my JS object.
    • Can’t read AudioBuffer playback position: I’d like to show the user the current playback position for each of my sound loops, but the API doesn’t provide this information. Could I compute it myself? Unfortunately, after a few iterations of ramping an AudioBuffer’s playbackRate between random values, it is very difficult to compute the current playback position within the buffer. Unlike some API users, I don’t need a highly accurate position, I just want to visualize for my users when the current sound loop restarts.

    Debugging with the Web Audio inspector

    Firefox’s Web Audio inspector shows how Audio Nodes are connected to one another.

    Firefox’s Web Audio inspector shows how Audio Nodes are connected to one another.



    I had great success using Firefox’s Web Audio inspector to watch my Audio Nodes being created and interconnected as my code runs.

    In the screenshot above, you can see the four AudioBufferSources, each feeding through a GainNode and PannerNode before being summed by an AudioDestination. Note that each recording is also connected to an AnalyzerNode; the Analyzers are used to create the scrolling amplitude graphs for each loop.

    Visualizing sound loops

    As the soundscape evolves, users often want to know which bird species is responsible for a particular sound they hear in the mix. We use a scrolling visualization for each loop that shows instantaneous amplitude, creating distinctive shapes you can correlate with what you’re hearing. The visualization uses the Analyzer node to perform a fast Fourier transform (FFT) on the sound, which yields the amplitude of the sound at every frequency. We compute the average of all those amplitudes, and then draw that amplitude at the right edge of a Canvas. As the contents of the Canvas shift sideways on every animation frame, the result is a horizontally scrolling amplitude graph.

    BirdSongPlayer.prototype.initializeVUMeter = function() {
      // set up VU meter
      var myAnalyser = this.analyser;
      var volumeMeterCanvas = $(this.playerSelector).find('canvas')[0];
      var graphicsContext = volumeMeterCanvas.getContext('2d');
      var previousVolume = 0;
     
      requestAnimationFrame(function vuMeter() {
        // get the average, bincount is fftsize / 2
        var array =  new Uint8Array(myAnalyser.frequencyBinCount);
        myAnalyser.getByteFrequencyData(array);
        var average = getAverageVolume(array);
        average = Math.max(Math.min(average, 128), 0);
     
        // draw the rightmost line in black right before shifting
        graphicsContext.fillStyle = 'rgb(0,0,0)'
        graphicsContext.fillRect(258, 128 - previousVolume, 2, previousVolume);
     
        // shift the drawing over one pixel
        graphicsContext.drawImage(volumeMeterCanvas, -1, 0);
     
        // clear the rightmost column state
        graphicsContext.fillStyle = 'rgb(245,245,245)'
        graphicsContext.fillRect(259, 0, 1, 130);
     
        // set the fill style for the last line (matches bootstrap button)
        graphicsContext.fillStyle = '#5BC0DE'
        graphicsContext.fillRect(258, 128 - average, 2, average);
     
        requestAnimationFrame(vuMeter);
        previousVolume = average;
      });
    }

    What’s next

    I’m continuing to work on cleaning up my JavaScript code for this project. I have several user interface improvements suggested by my Mozillia colleagues that I’d like to try. And Prof. Belet and I are considering what other sources of geotagged sounds we can use to make more soundscapes with. In the meantime, please try Oiseaux de Même for yourself and let us know what you think!

  9. What’s new in Web Audio

    Introduction

    It’s been a while since we said anything on Hacks about the Web Audio API. However, with Firefox 37/38 hitting our Developer Edition/Nightly browser channels, there are some interesting new features to talk about!

    This article presents you with some new Web Audio tricks to watch out for, such as the new StereoPannerNode, promise-based methods, and more.

    Simple stereo panning

    Firefox 37 introduces the StereoPannerNode interface, which allows you to add a stereo panning effect to an audio source simply and easily. It takes a single property: pan—an a-rate AudioParam that can accept numeric values between -1.0 (full left channel pan) and 1.0 (full right channel pan).

    But don’t we already have a PannerNode?

    You may have already used the older PannerNode interface, which allows you to position sounds in 3D. Connecting a sound source to a PannerNode causes it to be “spatialised”, meaning that it is placed into a 3D space and you can then specify the position of the listener inside. The browser then figures out how to make the sources sound, applying panning and doppler shift effects, and other nice 3D “artifacts” if the sounds are moving over time, etc:

    var audioContext = new AudioContext();
    var pannerNode = audioContext.createPanner();
     
    // The listener is 100 units to the right of the 3D origin
    audioContext.listener.setPosition(100, 0, 0);
     
    // The panner is in the 3D origin
    pannerNode.setPosition(0, 0, 0);

    This works well with WebGL-based games as both environments use similar units for positioning—an array of x, y, z values. So you could easily update the position, orientation, and velocity of the PannerNodes as you update the position of the entities in your 3D scene.

    But what if you are just building a conventional music player where the songs are already stereo tracks, and you actually don’t care at all about 3D? You have to go through a more complicated setup process than should be necessary, and it can also be computationally more expensive. With the increased usage of mobile devices, every operation you don’t perform is a bit more battery life you save, and users of your website will love you for it.

    Enter StereoPannerNode

    StereoPannerNode is a much better solution for simple stereo use cases, as described above. You don’t need to care about the listener’s position; you just need to connect source nodes that you want to spatialise to a StereoPannerNode instance, then use the pan parameter.

    To use a stereo panner, first create a StereoPannerNode using createStereoPanner(), and then connect it to your audio source. For example:

    var audioCtx = window.AudioContext();
    // You can use any type of source
    var source = audioCtx.createMediaElementSource(myAudio);
    var panNode = audioCtx.createStereoPanner();
     
    source.connect(panNode);
    panNode.connect(audioCtx.destination);

    To change the amount of panning applied, you just update the pan property value:

    panNode.pan.value = 0.5; // places the sound halfway to the right
    panNode.pan.value = 0.0; // centers it
    panNode.pan.value = -0.5; // places the sound halfway to the left

    You can see http://mdn.github.io/stereo-panner-node/ for a complete example.

    Also, since pan is an a-rate AudioParam you can design nice smooth curves using parameter automation, and the values will be updated per sample. Trying to do this kind of change over time would sound weird and unnatural if you updated the value over multiple requestAnimationFrame calls. And you can’t automate PannerNode positions either.

    For example, this is how you could set up a panning transition from left to right that lasts two seconds:

    panNode.pan.setValueAtTime(-1, audioContext.currentTime);
    panNode.pan.linearRampToValueAtTime(1, audioContext.currentTime + 2);

    The browser will take care of updating the pan value for you. And now, as of recently, you can also visualise these curves using the Firefox Devtools Web Audio Editor.

    Detecting when StereoPannerNode is available

    It might be the case that the Web Audio implementation you’re using has not implemented this type of node yet. (At the time of this writing, it is supported in Firefox 37 and Chrome 42 only.) If you try to use StereoPannerNode in these cases, you’re going to generate a beautiful undefined is not a function error instead.

    To make sure StereoPannerNodes are available, just check whether the createStereoPanner() method exists in your AudioContext:

    if (audioContext.createStereoPanner) {
        // StereoPannerNode is supported!
    }

    If it doesn’t, you will need to revert back to the older PannerNode.

    Changes to the default PannerNode panning algorithm

    The default panning algorithm type used in PannerNodes used to be HRTF, which is a high quality algorithm that rendered its output using a convolution with human-based data (thus it’s very realistic). However it is also very computationally expensive, requiring the processing to be run in additional threads to ensure smooth playback.

    Authors often don’t require such a high level of quality and just need something that is good enough, so the default PannerNode.type is now equalpower, which is much cheaper to compute. If you want to go back to using the high quality panning algorithm instead, you just need to change the type:

    pannerNodeInstance.type = 'HRTF';

    Incidentally, a PannerNode using type = 'equalpower' results in the same algorithm that StereoPannerNode uses.

    Promise-based methods

    Another interesting feature that has been recently added to the Web Audio spec is Promise-based versions of certain methods. These are OfflineAudioContext.startRendering() and AudioContext.decodeAudioData.

    The below sections show how the method calls look with and without Promises.

    OfflineAudioContext.startRendering()

    Let’s suppose we want to generate a minute of audio at 44100 Hz. We’d first create the context:

    var offlineAudioContext = new OfflineAudioContext(2, 44100 * 60, 44100);

    Classic code

    offlineAudioContext.addEventListener('oncomplete', function(e) {
        // rendering complete, results are at `e.renderedBuffer`
    });
    offlineAudioContext.startRendering();

    Promise-based code

    offlineAudioContext.startRendering().then(function(renderedBuffer) {
        // rendered results in `renderedBuffer`
    });

    AudioContext.decodeAudioData

    Likewise, when decoding an audio track we would create the context first:

    var audioContext = new AudioContext();

    Classic code

    audioContext.decodeAudioData(data, function onSuccess(decodedBuffer) {
        // decoded data is decodedBuffer
    }, function onError(e) {
        // guess what... something didn't work out well!
    });

    Promise-based code

    audioContext.decodeAudioData(data).then(function(decodedBuffer) {
        // decoded data is decodedBuffer
    }, function onError(e) {
        // guess what... something didn't work out well!
    });

    In both cases the differences don’t seem major, but if you’re composing the results of promises sequentially or if you’re waiting on an event to complete before calling several other methods, promises are really helpful to avoid callback hell.

    Detecting support for Promise-based methods

    Again, you don’t want to get the dreaded undefined is not a function error message if the browser you’re running your code on doesn’t support these new versions of the methods.

    A quick way to check for support: look at the returned type of these calls. If they return a Promise, we’re in luck. If they don’t, we have to keep using the old methods:

    if((new OfflineAudioContext(1, 1, 44100)).startRendering() != undefined) {
        // Promise with startRendering is supported
    }
     
    if((new AudioContext()).decodeAudioData(new Uint8Array(1)) != undefined) {
        // Promise with decodeAudioData is supported
    }

    Audio workers

    Although the spec has not been finalised and they are not implemented in any browser yet, it is also worth giving a mention to Audio Workers, which —you’ve guessed it— are a specialised type of web worker for use by Web Audio code.

    Audio Workers will replace the almost-obsolete ScriptProcessorNode. Originally, this was the way to run your own custom nodes inside the audio graph, but they actually run on the main thread causing all sorts of problems, from audio glitches (if the main thread becomes stalled) to unresponsive UI code (if the ScriptProcessorNodes aren’t fast enough to process their data).

    The biggest feature of audio workers is that they run in their own separate thread, just like any other Worker. This ensures that audio processing is prioritised and we avoid sound glitches, which human ears are very sensitive to.

    There is an ongoing discussion on the w3c web audio list; if you are interested in this and other Web Audio developments, you should go check it out.

    Exciting times for audio on the Web!

  10. Embedding an HTTP Web Server in Firefox OS

    Nearing the end of last year, Mozilla employees were gathered together for a week of collaboration and planning. During that week, a group was formed to envision what the future of Firefox OS might be surrounding a more P2P-focused Web. In particular, we’ve been looking at harnessing technologies to collectively enable offline P2P connections such as Bluetooth, NFC and WiFi Direct.

    Since these technologies only provide a means to communicate between devices, it became immediately clear that we would also need a protocol for apps to send and receive data. I quickly realized that we already have a standard protocol for transmitting data in web apps that we could leverage – HTTP.

    By utilizing HTTP, we would already have everything we’d need for apps to send and receive data on the client side, but we would still need a web server running in the browser to enable offline P2P communications. While this type of HTTP server functionality might be best suited as part of a standardized WebAPI to be baked into Gecko, we actually already have everything we need in Firefox OS to implement this in JavaScript today!

    navigator.mozTCPSocket

    Packaged apps have access to both raw TCP and UDP network sockets, but since we’re dealing with HTTP, we only need to work with TCP sockets. Access to the TCPSocket API is exposed through navigator.mozTCPSocket which is currently only exposed to “privileged” packaged apps with the tcp-socket permission:

    "type": "privileged",
    "permissions": {
      "tcp-socket": {}
    },

    In order to respond to incoming HTTP requests, we need to create a new TCPSocket that listens on a known port such as 8080:

    var socket = navigator.mozTCPSocket.listen(8080);

    When an incoming HTTP request is received, the TCPSocket needs to handle the request through the onconnect handler. The onconnect handler will receive a TCPSocket object used to service the request. The TCPSocket you receive will then call its own ondata handler each time additional HTTP request data is received:

    socket.onconnect = function(connection) {
      connection.ondata = function(evt) {
        console.log(evt.data);
      };
    };

    Typically, an HTTP request will result in a single calling of the ondata handler. However, in cases where the HTTP request payload is very large, such as for file uploads, the ondata handler will be triggered each time the buffer is filled, until the entire request payload is delivered.

    In order to respond to the HTTP request, we must send data to the TCPSocket we received from the onconnect handler:

    connection.ondata = function(evt) {
      var response = 'HTTP/1.1 200 OK\r\n';
      var body = 'Hello World!';
     
      response += 'Content-Length: ' + body.length + '\r\n';
      response += '\r\n';
      response += body;
     
      connection.send(response);
      connection.close();
    };

    The above example sends a proper HTTP response with “Hello World!” in the body. Valid HTTP responses must contain a status line which consists of the HTTP version HTTP/1.1, the response code 200 and the response reason OK terminated by a CR+LF \r\n character sequence. Immediately following the status line are the HTTP headers, one per line, separated by a CR+LF character sequence. After the headers, an additional CR+LF character sequence is required to separate the headers from the body of the HTTP response.

    FxOS Web Server

    Now, it is likely that we will want to go beyond simple static “Hello World!” responses to do things like parsing the URL path and extracting parameters from the HTTP request in order to respond with dynamic content. It just so happens that I’ve already implemented a basic-featured HTTP server library that you can include in your own Firefox OS apps!

    FxOS Web Server can parse all parts of the HTTP request for various content types including application/x-www-form-urlencoded and multipart/form-data. It can also gracefully handle large HTTP requests for file uploads and can send large binary responses for serving up content such as images and videos. You can either download the source code for FxOS Web Server on GitHub to include in your projects manually or you can utilize Bower to fetch the latest version:

    bower install justindarc/fxos-web-server --save

    Once you have the source code downloaded, you’ll need to include dist/fxos-web-server.js in your app using a <script> tag or a module loader like RequireJS.

    Simple File Storage App

    Next, I’m going to show you how to use FxOS Web Server to build a simple Firefox OS app that lets you use your mobile device like a portable flash drive for storing and retrieving files. You can see the source code for the finished product on GitHub.

    Before we get into the code, let’s set up our app manifest to get permission to access DeviceStorage and TCPSocket:

    {
      "version": "1.0.0",
      "name": "WebDrive",
      "description": "A Firefox OS app for storing files from a web browser",
      "launch_path": "/index.html",
      "icons": {
        "128": "/icons/icon_128.png"
      },
      "type": "privileged",
      "permissions": {
        "device-storage:sdcard": { "access": "readwrite" },
        "tcp-socket": {}
      }
    }

    Our app won’t need much UI, just a listing of files in the “WebDrive” folder on the device, so our HTML will be pretty simple:

    <!DOCTYPE html>
    <html>
    <head>
      <meta charset="utf-8">
      <title>WebDrive</title>
      <meta name="description" content="A Firefox OS app for storing files from a web browser">
      <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1">
      <script src="bower_components/fxos-web-server/dist/fxos-web-server.js"></script>
      <script src="js/storage.js"></script>
      <script src="js/app.js"></script>
    </head>
    <body>
      <h1>WebDrive</h1>
      <hr>
      <h3>Files</h3>
      <ul id="list"></ul>
    </body>
    </html>

    As you can see, I’ve included fxos-web-server.js in addition to app.js. I’ve also included a DeviceStorage helper module called storage.js since enumerating files can get somewhat complex. This will help keep the focus on our code specific to the task at hand.

    The first thing we’ll need to do is create new instances of the HTTPServer and Storage objects:

    var httpServer = new HTTPServer(8080);
    var storage = new Storage('sdcard');

    This will initialize a new HTTPServer on port 8080 and a new instance of our Storage helper pointing to the device’s SD card. In order for our HTTPServer instance to be useful, we must listen for and handle the “request” event. When an incoming HTTP request is received, the HTTPServer will emit a “request” event that passes the parsed HTTP request as an HTTPRequest object to the event handler.

    The HTTPRequest object contains various properties of an HTTP request including the HTTP method, path, headers, query parameters and form data. In addition to the request data, an HTTPResponse object is also passed to the “request” event handler. The HTTPResponse object allows us to send our response as a file or string and set the response headers:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      // Handle request here...
    });

    When a user requests the root URL of our web server, we’ll want to present them with a listing of files stored in the “WebDrive” folder on the device along with a file input for uploading new files. For convenience, we’ll create two helper functions to generate the HTML string to send in our HTTP response. One will just generate the listing of files which we’ll reuse to display the files on the device locally and the other will generate the entire HTML document to send in the HTTP response:

    function generateListing(callback) {
      storage.list('WebDrive', function(directory) {
        if (!directory || Object.keys(directory).length === 0) {
          callback('<li>No files found</li>');
          return;
        }
     
        var html = '';
        for (var file in directory) {
          html += `<li><a href="/${encodeURIComponent(file)}" target="_blank">${file}</a></li>`;
        }
     
        callback(html);
      });
    }
     
    function generateHTML(callback) {
      generateListing(function(listing) {
        var html =
    `<!DOCTYPE html>
    <html>
    <head>
      <meta charset="utf-8">
      <title>WebDrive</title>
    </head>
    <body>
      <h1>WebDrive</h1>
      <form method="POST" enctype="multipart/form-data">
        <input type="file" name="file">
        <button type="submit">Upload</button>
      </form>
      <hr>
      <h3>Files</h3>
      <ul>${listing}</ul>
    </body>
    </html>`;
     
        callback(html);
      });
    }

    You’ll notice that we’re using ES6 Template Strings for generating our HTML. If you’re not familiar with Template Strings, they allow us to have multi-line strings that automatically include whitespace and new lines and we can do basic string interpolation that automatically inserts values inside the ${} syntax. This is especially useful for generating HTML because it allows us to span multiple lines so our template markup remains highly readable when embedded within JavaScript code.

    Now that we have our helper functions, let’s send our HTML response in our “request” event handler:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      generateHTML(function(html) {
        response.send(html);
      });
    });

    As of right now, our “request” event handler will always respond with an HTML page listing all the files in the device’s “WebDrive” folder. However, we must first start the HTTPServer before we can receive any requests. We’ll do this once the DOM is ready and while we’re at it, let’s also render the file listing locally:

    window.addEventListener('DOMContentLoaded', function(evt) {
      generateListing(function(listing) {
        list.innerHTML = listing;
      });
     
      httpServer.start();
    });

    We should also be sure to stop the HTTPServer when the app is terminated, otherwise the network socket may never be freed:

    window.addEventListener('beforeunload', function(evt) {
      httpServer.stop();
    });

    At this point, our web server should be up and running! Go ahead and install the app on your device or simulator using WebIDE. Once installed, launch the app and point your desktop browser to your device’s IP address at port 8080 (e.g.: http://10.0.1.12:8080).

    You should see our index page load in your desktop browser, but the upload form still isn’t wired up and if you have any files in your “WebDrive” folder on your device, they cannot be downloaded yet. Let’s first wire up the file upload by first creating another helper function to save files received in an HTTPRequest:

    function saveFile(file, callback) {
      var arrayBuffer = BinaryUtils.stringToArrayBuffer(file.value);
      var blob = new Blob([arrayBuffer]);
     
      storage.add(blob, 'WebDrive/' + file.metadata.filename, callback);
    }

    This function will first convert the file’s contents to an ArrayBuffer using the BinaryUtils utility that comes with fxos-web-server.js. We then create a Blob that we pass to our Storage helper to save it to the SD card in the “WebDrive” folder. Note that the filename can be extracted from the file’s metadata object since it gets passed to the server using ‘multipart/form-data’ encoding.

    Now that we have a helper for saving an uploaded file, let’s wire it up in our “request” event handler:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      if (request.method === 'POST' && request.body.file) {
        saveFile(request.body.file, function() {
          generateHTML(function(html) {
            response.send(html);
          });
     
          generateListing(function(html) {
            list.innerHTML = html;
          });
        });
     
        return;
      }
     
      generateHTML(function(html) {
        response.send(html);
      });
    });

    Now, anytime an HTTP POST request is received that contains a “file” parameter in the request body, we will save the file to the “WebDrive” folder on the SD card and respond with an updated file listing index page. At the same time, we’ll also update the file listing on the local device to display the newly-added file.

    The only remaining part of our app to wire up is the ability to download files. Once again, let’s update the “request” event handler to do this:

    httpServer.addEventListener('request', function(evt) {
      var request  = evt.request;
      var response = evt.response;
     
      if (request.method === 'POST' && request.body.file) {
        saveFile(request.body.file, function() {
          generateHTML(function(html) {
            response.send(html);
          });
     
          generateListing(function(html) {
            list.innerHTML = html;
          });
        });
     
        return;
      }
     
      var path = decodeURIComponent(request.path);
      if (path !== '/') {
        storage.get('WebDrive' + path, function(file) {
          if (!file) {
            response.send(null, 404);
            return;
          }
     
          response.headers['Content-Type'] = file.type;
          response.sendFile(file);
        });
     
        return;
      }
     
      generateHTML(function(html) {
        response.send(html);
      });
    });

    This time, our “request” event handler will check the requested path to see if a URL other than the root is being requested. If so, we assume that the user is requesting to download a file which we then proceed to get using our Storage helper. If the file cannot be found, we return an HTTP 404 error. Otherwise, we set the “Content-Type” in the response header to the file’s MIME type and send the file with the HTTPResponse object.

    You can now re-install the app to your device or simulator using WebIDE and once again point your desktop browser to your device’s IP address at port 8080. Now, you should be able to upload and download files from your device using your desktop browser!

    The possible use cases enabled by embedding a web server into Firefox OS apps are nearly limitless. Not only can you serve up web content from your device to a desktop browser, as we just did here, but you can also serve up content from one device to another. That also means that you can use HTTP to send and receive data between apps on the same device! Since its inception, FxOS Web Server has been used as a foundation for several exciting experiments at Mozilla:

    • wifi-columns

      Guillaume Marty has combined FxOS Web Server with his amazing jsSMS Master System/Game Gear emulator to enable multi-player gaming across two devices in conjunction with WiFi Direct.

    • sharing

      Several members of the Gaia team have used FxOS Web Server and dns-sd.js to create an app that allows users to discover and share apps with friends over WiFi.

    • firedrop

      I have personally used FxOS Web Server to build an app that lets you share files with nearby users without an Internet connection using WiFi Direct. You can see the app in action here:

    I look forward to seeing all the exciting things that are built next with FxOS Web Server!