Mozilla

Articles

Sort by:

View:

  1. Keyboard events in Firefox OS TV

    Getting started

    The behavior of input events via hardware keys in Firefox OS varies widely from app to app. Early smartphones came with a limited number of keys — Power, Home, Volume up, Volume down — so it was easy for the software to determine an appropriate response for each keypress event. However, Smart TV remotes now come with many hardware keys, and defining the appropriate behavior when a key is pressed has become an important issue on the Firefox OS TV platform. If a hardware key on a smart remote can be used both by apps and by the system, it’s important to determine which response is triggered when the key is pressed.

    This post will introduce the challenges of programming a TV remote to manage keyboard events on the Firefox OS Smart TV platform. We’ll classify keyboard events into four scenarios and describe these dispatch scenarios and how they interact. This is the first of two posts about keyboard events for Firefox OS Smart TV. In a followup Part 2 post, we’ll drill down into the implementation details and and take a look at sample code for event handling on the TV remote.

    f1

    [Fig 1]

    We begin with the ‘Info’ key on a TV remote. Often, it’s used by the hardware to display system information, although it’s possible for an application to use the same key to display app information. When a user presses the key, what action will be shown on screen — system info or app info?

    Four keyboard event scenarios

    To determine the appropriate behavior when hardware keys are pressed, we start by describing four scenarios for keyboard events.

    Scenario Description Event order
    SYSTEM-ONLY For keys which should be handled by mozbrowser-iframe-host-page only. system
    SYSTEM-FIRST For keys which can be handled by mozbrowser-iframe-host-page first and can then also be handled by mozbrowser-iframe-embedded-page. system->app
    APP-CANCELLED For keys which should be handled by the mozbrowser-iframe-embedded-page only. app
    APP-FIRST For keys which can be handled by mozbrowser-iframe-embedded-page first and can then also be handled by mozbrowser-iframe-host-page. app->system

    [Table 1]

    [Fig 2]

    [Fig 2]

    The mozbrowser-iframe-host-page and mozbrowser-iframe-embedded-page mentioned in Table 1 are illustrated in Fig.2 above. If A.html represents a host page whose source is B.html, then A.html is the mozbrowser-iframe-host-page, and B.html is mozbrowser-iframe-embedded-page. mozbrowser uses the non-standard Firefox Browser API, built for the implementation of key features and content experiences in Firefox OS apps. Learn more about mozbrowser on MDN.

    Suitable responses for any given keyboard event depend on the scenario. In the case illustrated above, let’s suppose that the ‘Info’ keyboard event is categorized as APP-FIRST and the default action set by system is to show system information. Thus, when we press the ‘Info’ key with app Z in the foreground, there are two possible results:

      1. If app Z has an event handler that tells the ‘Info’ key to show app information, then app information will appear on screen when the user presses the ‘Info’ key on the remote.
      2. If app Z doesn’t set an event handler for the ‘Info’ key, the default action is triggered. That is, the screen will show the system information.

    How to implement the four scenarios

    To implement the four keyboard event scenarios described above, we’ve introduced four new keyboard events:

        • mozbrowserbeforekeydown – fired before the keydown event
        • mozbrowserafterkeydown – fired after the keydown event
        • mozbrowserbeforekeyup – fired before the keyup event
        • mozbrowserafterkeyup – fired after the keyup event

    These four keyboard events are only received by the window that embeds a mozbrowser-iframe.

    The keyboard events occur in a specific sequence over time: mozbrowserbeforekeydown, mozbrowserafterkeydown, mozbrowserbeforekeyup, keyup, mozbrowserafterkeyup.

    This gives developers a way to implement the four scenarios mentioned above. Conceptually, the scenarios SYSTEM-ONLY, SYSTEM-FIRST and APP-CANCELLED, APP-FIRST can be implemented by setting proper handlers for the mozbrowserbeforekey* and mozbrowserafterkey* events. The SYSTEM-ONLY and SYSTEM-FIRST scenarios can be implemented by setting proper handlers for mozbrowserbeforekey* events and the APP-CANCELLED and APP-FIRST scenarios can be implemented via mozbrowserafterkey* events.

    iframe structure in Firefox OS

    [Fig 3]

    [Fig 3]

    To understand how to implement the four scenarios, let’s first take a look at iframe structure in Firefox OS. The outermost iframe in Firefox OS is shell.html. It embeds an in-process iframe which is sourced from system/index.html. The system app (system/index.html) contains several web apps (essentially iframes) which can be in-process (remote=”false”) or out-of-process (remote=”true”). As a result, the relationship of these three layers can be displayed in the following table:

    mozbrowser iframe host page mozbrowser iframe embedded page
    shell.html system/index.html
    system/index.html web apps(essentially iframes)

    [Table 2]

    Dispatch order of keyboard events

    [Fig 4]

    [Fig 4]

    When a keydown event is sent to some element in a mozbrowser-iframe-embedded-page, the owner of the embedded iframe, i.e., the mozbrowser-iframe-host-page, will receive the mozbrowserbeforekeydown event before the keydown event is sent and the mozbrowserafterkeydown event after the event is sent to the mozbrowser-iframe-embedded-page.

    In Gecko, once there is one keydown event with the target in an out-of-process iframe embedded in an HTML document, the keydown event is duplicated to the HTML document as well. The target of this duplicated event is set as the embedded iframe element.

    This results in the keyboard event sequence shown in Fig 4. It illustrates all related keydown events and their relationship when a keydown event with a target in a mozbrowser-iframe-embedded-page needs to be dispatched. In brief, events follow this sequence:

        1. Before dispatching any keydown event, the mozbrowserbeforekeydown event is first dispatched to the window of mozbrowser-iframe-host-page.
        2. Then, the original keydown event (with a target in a mozbrowser-iframe-embedded-page) will be duplicated to the mozbrowser-iframe-host-page HTML document. Its target will be set to be the iframe that contains the mozbrowser-iframe-embedded-page.
        3. Next, the original keydown event will be dispatched to its target.
        4. After the original keydown event dispatch is complete, the mozbrowserafterkeydown event will be dispatched to the window of mozbrowser-iframe-host-page.

    Notice that the event dispatch process described above follows the DOM tree event flow. Event sequence and event targets can be organized into the following table:

    Order event target
    1 mozbrowserbeforekeydown window in mozbrowser-iframe-host-page
    2 keydown iframe that contains the mozbrowser-iframe-embedded-page in mozbrowser-iframe-host-page
    3 keydown original one in mozbrowser-iframe-embedded-page
    4 mozbrowserafterkeydown window in mozbrowser-iframe-host-page

    [Table 3]

    f5

    [Fig 5]

    The keyboard events mozbrowserbeforekeydown, keydown, and mozbrowserafterkeydown, can be extended to nested mozbrowser iframes, like the iframe structure in Firefox OS described in Table 2. In this case, the mozbrowserbeforekeydown and mozbrowserafterkeydown events will be dispatched to the innermost mozbrowser-iframe-host-page as well as the outer one. Thus, in Firefox OS, mozbrowserkeydown and mozbrowserafterkeydown will be dispatched to the window of system/index.html and the window of shell.html. Fig. 5 illustrates the whole dispatch sequence of related events when a keydown event is dispatched to a web app. The sequence of events is demonstrated in Table 4.

    Order event target
    1 mozbrowserbeforekeydown window in shell.html
    2 mozbrowserbeforekeydown window in system/index.html
    3 keydown iframe that contains the web app in system/index.html
    4 keydown original one in web app
    5 mozbrowserafterkeydown window in system/index.html
    6 mozbrowserafterkeydown window in shell.html

    [Table 4]

    f6

    [Fig 6]

    Although the keyup event must be fired after keydown, the keydown event and the keyup event are independent of each other. Moreover, the path mozbrowserbeforekeyup, keyup, mozbrowserafterkeyup is independent of the path mozbrowserbeforekeydown, keydown, mozbrowserafterkeydown. Therefore, it’s possible for these two paths to cross each other. The mozbrowserbeforekeyup may arrive before the keydown event.

    In Firefox OS, most apps run out-of-process. This means that the app runs on its own process, not on the main process. After dispatching a given key* event (the duplicate) to the system app, it takes time to send the original key* event to the process where the mozbrowser-iframe-embedded-page is located. In a similar manner, after a given key* event is dispatched to the mozbrowser-iframe-embedded-page’s process, time is required to send mozbrowserafterkey* back to the process where the mozbrowser-iframe-host-page is located.

    Consequently, the mozbrowserbeforekeyup event may arrive in the main Firefox OS process (where the system app lives), before the keydown event is dispatched to the app’s own process. Common results of the order of the key* events are demonstrated above in Fig. 6. The yellow series represents the keydown path, and the blue series show the keyup path. And yes, these two paths may cross each other.

    Next steps

    If you’re looking for implementation details for the four scenarios we’ve described, as well as event handler sample code to help with your implementation of keyboard events for smart TV remotes, keep an eye out for the followup post, and stay tuned for more complete documentation arriving very soon on MDN.

  2. ES6 In Depth: Classes

    ES6 In Depth is a series on new features being added to the JavaScript programming language in the 6th Edition of the ECMAScript standard, ES6 for short.

    Today, we get a bit of a respite from the complexity that we’ve seen in previous posts in this series. There are no new never-before-seen ways of writing code with Generators; no all-powerful Proxy objects which provide hooks into the inner algorithmic workings of the JavaScript language; no new data structures that obviate the need for roll-your-own solutions. Instead, we get to talk about syntactic and idiomatic cleanups for an old problem: object constructor creation in JavaScript.

    The Problem

    Say we want to create the most quintessential example of object-oriented design principles: the Circle class. Imagine we are writing a Circle for a simple Canvas library. Among other things, we might want to know how to do the following:

    • Draw a given Circle to a given Canvas.
    • Keep track of the total number of Circles ever made.
    • Keep track of the radius of a given Circle, and how to enforce invariants on its value.
    • Calculate the area of a given Circle.

    Current JS idioms say that we should first create the constructor as a function, then add any properties we might want to the function itself, then replace the prototype property of that constructor with an object. This prototype object will contain all of the properties that instance objects created by our constructor should start with. For even a simple example, by the time you get it all typed out, this ends up being a lot of boilerplate:

    function Circle(radius) {
        this.radius = radius;
        Circle.circlesMade++;
    }
    
    Circle.draw = function draw(circle, canvas) { /* Canvas drawing code */ }
    
    Object.defineProperty(Circle, "circlesMade", {
        get: function() {
            return !this._count ? 0 : this._count;
        },
    
        set: function(val) {
            this._count = val;
        }
    });
    
    Circle.prototype = {
        area: function area() {
            return Math.pow(this.radius, 2) * Math.PI;
        }
    };
    
    Object.defineProperty(Circle.prototype, "radius", {
        get: function() {
            return this._radius;
        },
    
        set: function(radius) {
            if (!Number.isInteger(radius))
                throw new Error("Circle radius must be an integer.");
            this._radius = radius;
        }
    });
    

    Not only is the code cumbersome, it’s also far from intuitive. It requires having a non-trivial understanding of the way functions work, and how various installed properties make their way onto created instance objects. If this approach seems complicated, don’t worry. The whole point of this post is to show off a much simpler way of writing code that does all of this.

    Method Definition Syntax

    In a first attempt to clean this up, ES6 offered a new syntax for adding special properties to an object. While it was easy to add the area method to Circle.prototype above, it felt much heavier to add the getter/setter pair for radius. As JS moved towards a more object-oriented approach, people became interested in designing cleaner ways to add accessors to objects. We needed a new way of adding “methods” to an object exactly as if they had been added with obj.prop = method, without the weight of Object.defineProperty. People wanted to be able to do the following things easily:

    1. Add normal function properties to an object.
    2. Add generator function properties to an object.
    3. Add normal accessor function properties to an object.
    4. Add any of the above as if you had done it with [] syntax on the finished object. We’ll call these Computed property names.

    Some of these things couldn’t be done before. For example, there is no way to define a getter or setter with assignments to obj.prop. Accordingly, new syntax had to be added. You can now write code that looks like this:

    var obj = {
        // Methods are now added without a function keyword, using the name of the
        // property as the name of the function.
        method(args) { ... },
    
        // To make a method that's a generator instead, just add a '*', as normal.
        *genMethod(args) { ... },
    
        // Accessors can now go inline, with the help of |get| and |set|. You can
        // just define the functions inline. No generators, though.
    
        // Note that a getter installed this way must have no arguments
        get propName() { ... },
    
        // Note that a setter installed this way must have exactly one argument
        set propName(arg) { ... },
    
        // To handle case (4) above, [] syntax is now allowed anywhere a name would
        // have gone! This can use symbols, call functions, concatenate strings, or
        // any other expression that evaluates to a property id. Though I've shown
        // it here as a method, this syntax also works for accessors or generators.
        [functionThatReturnsPropertyName()] (args) { ... }
    };
    

    Using this new syntax, we can now rewrite our snippet above:

    function Circle(radius) {
        this.radius = radius;
        Circle.circlesMade++;
    }
    
    Circle.draw = function draw(circle, canvas) { /* Canvas drawing code */ }
    
    Object.defineProperty(Circle, "circlesMade", {
        get: function() {
            return !this._count ? 0 : this._count;
        },
    
        set: function(val) {
            this._count = val;
        }
    });
    
    Circle.prototype = {
        area() {
            return Math.pow(this.radius, 2) * Math.PI;
        },
    
        get radius() {
            return this._radius;
        },
        set radius(radius) {
            if (!Number.isInteger(radius))
                throw new Error("Circle radius must be an integer.");
            this._radius = radius;
        }
    };
    

    Pedantically, this code isn’t exactly identical to the snippet above. Method definitions in object literals are installed as configurable and enumerable, while the accessors installed in the first snippet will be non-configurable and non-enumerable. In practice, this is rarely noticed, and I decided to elide enumerability and configurability above for brevity.

    Still, it’s getting better, right? Unfortunately, even armed with this new method definition syntax, there’s not much we can do for the definition of Circle, as we have yet to define the function. There’s no way to get properties onto a function as you’re defining it.

    Class Definition Syntax

    Though this was better, it still didn’t satisfy people who wanted a cleaner solution to object-oriented design in JavaScript. Other languages have a construct for handling object-oriented design, they argued, and that construct is called a class.

    Fair enough. Let’s add classes, then.

    We want a system that will allow us to add methods to a named constructor, and add methods to its .prototype as well, so that they will appear on constructed instances of the class. Since we have our fancy new method definition syntax, we should definitely use it. Then, we only need a way to differentiate between what is generalized over all instances of the class, and what functions are specific to a given instance. In C++ or Java, the keyword for that is static. Seems as good as any. Let’s use it.

    Now it would be useful to have a way to designate one of the methods of the bunch to be the function that gets called as the constructor. In C++ or Java, that would be named the same as the class, with no return type. Since JS doesn’t have return types, and we need a .constructor property anyway, for backwards compatibility, let’s call that method constructor.

    Putting it together, we can rewrite our Circle class as it was always meant to be:

    class Circle {
        constructor(radius) {
            this.radius = radius;
            Circle.circlesMade++;
        };
    
        static draw(circle, canvas) {
            // Canvas drawing code
        };
    
        static get circlesMade() {
            return !this._count ? 0 : this._count;
        };
        static set circlesMade(val) {
            this._count = val;
        };
    
        area() {
            return Math.pow(this.radius, 2) * Math.PI;
        };
    
        get radius() {
            return this._radius;
        };
        set radius(radius) {
            if (!Number.isInteger(radius))
                throw new Error("Circle radius must be an integer.");
            this._radius = radius;
        };
    }
    

    Wow! Not only can we group everything related to a Circle together, but everything looks so… clean. This is definitely better than what we started with.

    Even so, some of you are likely to have questions or to find edge cases. I’ll try to anticipate and address some of these below:

    • What’s with the semicolons? – In an attempt to “make things look more like traditional classes,” we decided to go with a more traditional separator. Don’t like it? It’s optional. No delimiter is required.

    • What if I don’t want a constructor, but still want to put methods on created objects? – That’s fine. The constructor method is totally optional. If you don’t supply one, the default is as if you had typed constructor() {}.

    • Can constructor be a generator? – Nope! Adding a constructor that’s not a normal method will result in a TypeError. This includes both generators and accessors.

    • Can I define constructor with a computed property name? – Unfortunately not. That would be really hard to detect, so we don’t try. If you define a method with a computed property name that ends up being named constructor, you will still get a method named constructor, it just won’t be the class’s constructor function.

    • What if I change the value of Circle? Will that cause new Circle to misbehave? – Nope! Much like function expressions, classes get an internal binding of their given name. This binding cannot be changed by external forces, so no matter what you set the Circle variable to in the enclosing scope, Circle.circlesMade++ in the constructor will function as expected.

    • OK, but I could pass an object literal directly as a function argument. This new class thing looks like it won’t work anymore. – Luckily, ES6 also adds class expressions! They can be either named or unnamed, and will behave exactly the same way as described above, except they won’t create a variable in the scope in which you declare them.

    • What about those shenanigans above with enumerability and so on? – People wanted to make it so that you could install methods on objects, but that when you enumerated the object’s properties, you only got the added data properties of the object. Makes sense. Because of this, installed methods in classes are configurable, but not enumerable.

    • Hey, wait… what..? Where are my instance variables? What about static constants? – You caught me. They currently don’t exist in class definitions in ES6. Good news, though! Along with others involved in the spec process, I am a strong proponent of both static and const values being installable in class syntax. In fact, it’s already come up in spec meetings! I think we can look forward to more discussion of this in the future.

    • OK, even still, these are awesome! Can I use them yet? – Not exactly. There are polyfill options (especially Babel) so that you can play around with them today. Unfortunately, it’s going to be a little while before they are natively implemented in all major browsers. I’ve implemented everything we discussed here today in the Nightly version of Firefox, and it’s implemented but not enabled by default in Edge and Chrome. Unfortunately, it looks like there’s no current implementation in Safari.

    • Java and C++ have subclassing and a super keyword, but there’s nothing mentioned here. Does JS have that? – It does! However, that’s a whole other post’s worth of discussion. Check back with us later for an update about subclassing, where we’ll discuss more about the power of JavaScript classes.

    I would not have been able to implement classes without the guidance and enormous code review responsiblity of Jason Orendorff and Jeff Walden.

    Next week, Jason Orendorff returns from a week’s vacation and takes up the subject of let and const.

  3. ES6 In Depth: Proxies

    ES6 In Depth is a series on new features being added to the JavaScript programming language in the 6th Edition of the ECMAScript standard, ES6 for short.

    Here is the sort of thing we are going to do today.

    var obj = new Proxy({}, {
      get: function (target, key, receiver) {
        console.log(`getting ${key}!`);
        return Reflect.get(target, key, receiver);
      },
      set: function (target, key, value, receiver) {
        console.log(`setting ${key}!`);
        return Reflect.set(target, key, value, receiver);
      }
    });
    

    That’s a little complicated for a first example. I’ll explain all the parts later. For now, check out the object we created:

    > obj.count = 1;
        setting count!
    > ++obj.count;
        getting count!
        setting count!
        2
    

    What’s going on here? We are intercepting property accesses on this object. We are overloading the "." operator.

    How it’s done

    The best trick in computing is called virtualization. It’s a very general-purpose technique for doing astonishing things. Here’s how it works.

    1. Take any picture.

      (picture of a coal power plant)

      Photo credit: Martin Nikolaj Bech
    2. Draw an outline around something in the picture.

      (same photo, with the power plant circled)
    3. Now replace either everything inside the outline, or everything outside the outline, with something totally unexpected. There is just one rule, the Rule of Backwards Compatibility. Your replacement must behave enough like what was there before that nobody on the other side of the line notices that anything has changed.

      (the circled part is replaced with a wind farm)

      Photo credit: Beverley Goodwin.

    You’ll be familiar with this kind of hack from classic computer science films such as The Truman Show and The Matrix, where a person is inside the outline, and the rest of the world has been replaced with an elaborate illusion of normalcy.

    In order to satisfy the Rule of Backwards Compatibility, your replacement may need to be cunningly designed. But the real trick is in drawing the right outline.

    By outline, I mean an API boundary. An interface. Interfaces specify how two bits of code interact and what each part expects of the other. So if an interface is designed into the system, the outline is already drawn for you. You know you can replace either side, and the other side won’t care.

    It’s when there’s not an existing interface that you have to get creative. Some of the coolest software hacks of all time have involved drawing an API boundary where previously there was none, and bringing that interface into existence via a prodigious engineering effort.

    Virtual memory, Hardware virtualization, Docker, Valgrind, rr—to various degrees all of these projects involved driving new and rather unexpected interfaces into existing systems. In some cases, it took years and new operating system features and even new hardware to make the new boundary work well.

    The best virtualization hacks bring with them a new understanding of whatever’s being virtualized. To write an API for something, you have to understand it. Once you understand, you can do amazing things.

    ES6 introduces virtualization support for JavaScript’s most fundamental concept: the object.

    What is an object?

    No, really. Take a moment. Think it over. Scroll down when you know what an object is.

    (picture of Auguste Rodin’s sculpture, The Thinker)

    Photo credit: Joe deSousa.

    This question is too hard for me! I’ve never heard a really satisfying definition.

    Is that surprising? Defining fundamental concepts is always hard—check out the first few definitions in Euclid’s Elements sometime. The ECMAScript language specification is in good company, therefore, when it unhelpfully defines an object as a “member of the type Object.”

    Later, the spec adds that “An Object is a collection of properties.” That’s not bad. If you want a definition, that will do for now. We’ll come back to it later.

    I said before that to write an API for something, you have to understand it. So in a way, I’ve promised that if we get through all this, we’re going to understand objects better, and we’ll be able to do amazing things.

    So let’s follow in the footsteps of the ECMAScript standard committee and see what it would take to define an API, an interface, for JavaScript objects. What sort of methods do we need? What can objects do?

    That depends somewhat on the object. DOM Element objects can do certain things; AudioNode objects do other things. But there are a few fundamental abilities all objects share:

    • Objects have properties. You can get and set properties, delete them, and so on.
    • Objects have prototypes. This is how inheritance works in JS.
    • Some objects are functions or constructors. You can call them.

    Almost everything JS programs do with objects is done using properties, prototypes, and functions. Even the special behavior of an Element or AudioNode object is accessed by calling methods, which are just inherited function properties.

    So when the ECMAScript standard committee defined a set of 14 internal methods, the common interface for all objects, it should come as no surprise that they ended up focusing on these three fundamental things.

    The full list can be found in tables 5 and 6 of the ES6 standard. Here I’ll just describe a few. The weird double brackets, [[ ]], emphasize that these are internal methods, hidden from ordinary JS code. You can’t call, delete, or overwrite these like ordinary methods.

    • obj.[[Get]](key, receiver) – Get the value of a property.

      Called when JS code does: obj.prop or obj[key].

      obj is the object currently being searched; receiver is the object where we first started searching for this property. Sometimes we have to search several objects. obj might be an object on receiver’s prototype chain.

    • obj.[[Set]](key, value, receiver) – Assign to a property of an object.

      Called when JS code does: obj.prop = value or obj[key] = value.

      In an assignment like obj.prop += 2, the [[Get]] method is called first, and the [[Set]] method afterwards. Same goes for ++ and --.

    • obj.[[HasProperty]](key) – Test whether a property exists.

      Called when JS code does: key in obj.

    • obj.[[Enumerate]]() – List obj’s enumerable properties.

      Called when JS code does: for (key in obj) ....

      This returns an iterator object, and that’s how a forin loop gets an object’s property names.

    • obj.[[GetPrototypeOf]]() – Return obj’s prototype.

      Called when JS code does: obj.__proto__ or Object.getPrototypeOf(obj).

    • functionObj.[[Call]](thisValue, arguments) – Call a function.

      Called when JS code does: functionObj() or x.method().

      Optional. Not every object is a function.

    • constructorObj.[[Construct]](arguments, newTarget) – Invoke a constructor.

      Called when JS code does: new Date(2890, 6, 2), for example.

      Optional. Not every object is a constructor.

      The newTarget argument plays a role in subclassing. We’ll cover it in a future post.

    Maybe you can guess at some of the other seven.

    Throughout the ES6 standard, wherever possible, any bit of syntax or builtin function that does anything with objects is specified in terms of the 14 internal methods. ES6 drew a clear boundary around the brains of an object. What proxies let you do is replace the standard kind of brains with arbitrary JS code.

    When we start talking about overriding these internal methods in a moment, remember, we’re talking about overriding the behavior of core syntax like obj.prop, builtin functions like Object.keys(), and more.

    Proxy

    ES6 defines a new global constructor, Proxy. It takes two arguments: a target object and a handler object. So a simple example would look like this:

    var target = {}, handler = {};
    var proxy = new Proxy(target, handler);
    

    Let’s set aside the handler object for a moment and focus on how proxy and target are related.

    I can tell you how proxy is going to behave in one sentence. All of proxy’s internal methods are forwarded to target. That is, if something calls proxy.[[Enumerate]](), it’ll just return target.[[Enumerate]]().

    Let’s try it out. We’ll do something that causes proxy.[[Set]]() to be called.

    proxy.color = "pink";
    

    OK, what just happened? proxy.[[Set]]() should have called target.[[Set]](), so that should have made a new property on target. Did it?

    > target.color
        "pink"
    

    It did. And the same goes for all the other internal methods. This proxy will, for the most part, behave exactly the same as its target.

    There are limits to the fidelity of the illusion. You’ll find that proxy !== target. And a proxy will sometimes flunk type checks that the target would pass. Even if a proxy’s target is a DOM Element, for example, the proxy isn’t really an Element; so something like document.body.appendChild(proxy) will fail with a TypeError.

    Proxy handlers

    Now let’s return to the handler object. This is what makes proxies useful.

    The handler object’s methods can override any of the proxy’s internal methods.

    For example, if you’d like to intercept all attempts to assign to an object’s properties, you can do that by defining a handler.set() method:

    var target = {};
    var handler = {
      set: function (target, key, value, receiver) {
        throw new Error("Please don't set properties on this object.");
      }
    };
    var proxy = new Proxy(target, handler);
    
    > proxy.name = "angelina";
        Error: Please don't set properties on this object.
    

    The full list of handler methods is documented on the MDN page for Proxy. There are 14 methods, and they line up with the 14 internal methods defined in ES6.

    All handler methods are optional. If an internal method is not intercepted by the handler, then it’s forwarded to the target, as we saw before.

    Example: “Impossible” auto-populating objects

    We now know enough about proxies to try using them for something really weird, something that’s impossible without proxies.

    Here’s our first exercise. Make a function Tree() that can do this:

    > var tree = Tree();
    > tree
        { }
    > tree.branch1.branch2.twig = "green";
    > tree
        { branch1: { branch2: { twig: "green" } } }
    > tree.branch1.branch3.twig = "yellow";
        { branch1: { branch2: { twig: "green" },
                     branch3: { twig: "yellow" }}}
    

    Note how all the intermediate objects branch1, branch2, and branch3, are magically autocreated when they’re needed. Convenient, right? How could it possibly work?

    Until now, there’s no way it could work. But with proxies this is only a few lines of code. We just need to tap into tree.[[Get]](). If you like a challenge, you might want to try implementing this yourself before reading on.

    (picture of a tap in a maple tree)

    Not the right way to tap into a tree in JS. Photo credit: Chiot’s Run.

    Here’s my solution:

    function Tree() {
      return new Proxy({}, handler);
    }
    
    var handler = {
      get: function (target, key, receiver) {
        if (!(key in target)) {
          target[key] = Tree();  // auto-create a sub-Tree
        }
        return Reflect.get(target, key, receiver);
      }
    };
    

    Note the call to Reflect.get() at the end. It turns out there’s an extremely common need, in proxy handler methods, to be able to say “now just do the default behavior of delegating to target.” So ES6 defines a new Reflect object with 14 methods on it that you can use to do exactly that.

    Example: A read-only view

    I think I may have given the false impression that proxies are easy to use. Let’s do one more example to see if that’s true.

    This time our assignment is more complex: we have to implement a function, readOnlyView(object), that takes any object and returns a proxy that behaves just like that object, except without the ability to mutate it. So, for example, it should behave like this:

    > var newMath = readOnlyView(Math);
    > newMath.min(54, 40);
        40
    > newMath.max = Math.min;
        Error: can't modify read-only view
    > delete newMath.sin;
        Error: can't modify read-only view
    

    How can we implement this?

    The first step is to intercept all internal methods that would modify the target object if we let them through. There are five of those.

    function NOPE() {
      throw new Error("can't modify read-only view");
    }
    
    var handler = {
      // Override all five mutating methods.
      set: NOPE,
      defineProperty: NOPE,
      deleteProperty: NOPE,
      preventExtensions: NOPE,
      setPrototypeOf: NOPE
    };
    
    function readOnlyView(target) {
      return new Proxy(target, handler);
    }
    

    This works. It prevents assignment, property definition, and so on via the read-only view.

    Are there any loopholes in this scheme?

    The biggest problem is that the [[Get]] method, and others, may still return mutable objects. So even if some object x is a read-only view, x.prop may be mutable! That’s a huge hole.

    To plug it, we must add a handler.get() method:

    var handler = {
      ...
    
      // Wrap other results in read-only views.
      get: function (target, key, receiver) {
        // Start by just doing the default behavior.
        var result = Reflect.get(target, key, receiver);
    
        // Make sure not to return a mutable object!
        if (Object(result) === result) {
          // result is an object.
          return readOnlyView(result);
        }
        // result is a primitive, so already immutable.
        return result;
      },
    
      ...
    };
    

    This is not sufficient either. Similar code is needed for other methods, including getPrototypeOf and getOwnPropertyDescriptor.

    Then there are further problems. When a getter or method is called via this kind of proxy, the this value passed to the getter or method will typically be the proxy itself. But as we saw earlier, many accessors and methods perform a type check that the proxy won’t pass. It would be better to substitute the target object for the proxy here. Can you figure out how to do it?

    The lesson to draw from this is that creating a proxy is easy, but creating a proxy with intuitive behavior is quite hard.

    Odds and ends

    • What are proxies really good for?

      They’re certainly useful whenever you want to observe or log accesses to an object. They’ll be handy for debugging. Testing frameworks could use them to create mock objects.

      Proxies are useful if you need behavior that’s just slightly past what an ordinary object can do: lazily populating properties, for example.

      I almost hate to bring this up, but one of the best ways to see what’s going on in code that uses proxies… is to wrap a proxy’s handler object in another proxy that logs to the console every time a handler method is accessed.

      Proxies can be used to restrict access to an object, as we did with readOnlyView. That sort of use case is rare in application code, but Firefox uses proxies internally to implement security boundaries between different domains. They’re a key part of our security model.

    • Proxies ♥ WeakMaps. In our readOnlyView example, we create a new proxy every time an object is accessed. It could save a lot of memory to cache every proxy we create in a WeakMap, so that however many times an object is passed to readOnlyView, only a single proxy is created for it.

      This is one of the motivating use cases for WeakMap.

    • Revocable proxies. ES6 also defines another function, Proxy.revocable(target, handler), that creates a proxy, just like new Proxy(target, handler), except this proxy can be revoked later. (Proxy.revocable returns an object with a .proxy property and a .revoke method.) Once a proxy is revoked, it simply doesn’t work anymore; all its internal methods throw.

    • Object invariants. In certain situations, ES6 requires proxy handler methods to report results that are consistent with the target object’s state. It does this in order to enforce rules about immutability across all objects, even proxies. For example, a proxy can’t claim to be inextensible unless its target really is inextensible.

      The exact rules are too complex to go into here, but if you ever see an error message like "proxy can't report a non-existent property as non-configurable", this is the cause. The most likely remedy is to change what the proxy is reporting about itself. Another possibility is to mutate the target on the fly to reflect whatever the proxy is reporting.

    What is an object now?

    I think where we left it was: “An Object is a collection of properties.”

    I’m not totally happy with this definition, even taking for granted that we throw in prototypes and callability as well. I think the word “collection” is too generous, given how poorly defined a proxy can be. Its handler methods could do anything. They could return random results.

    By figuring out what an object can do, standardizing those methods, and adding virtualization as a first-class feature that everyone can use, the ECMAScript standard committee has expanded the realm of possibilities.

    Objects can be almost anything now.

    Maybe the most honest answer to the question “What is an object?” now is to take the 12 required internal methods as a definition. An object is something in a JS program that has a [[Get]] operation, a [[Set]] operation, and so on.


    Do we understand objects better after all that? I’m not sure! Did we do amazing things? Yeah. We did things that were never possible in JS before.

    Can I use Proxies today?

    Nope! Not on the Web, anyway. Only Firefox and Microsoft Edge support proxies, and there is no polyfill.

    Using proxies in Node.js or io.js requires both an off-by-default option (--harmony_proxies) and the harmony-reflect polyfill, since V8 implements an older version of the Proxy specification. (A previous version of this article had incorrect information about this. Thanks to Mörre and Aaron Powell for correcting my mistakes in the comments.)

    So feel free to experiment with proxies! Create a hall of mirrors where there seem to be thousands of copies of every object, all alike, and it’s impossible to debug anything! Now is the time. There’s little danger of your ill-advised proxy code escaping into production… yet.

    Proxies were first implemented in 2010, by Andreas Gal, with code reviews by Blake Kaplan. The standard committee then completely redesigned the feature. Eddy Bruel implemented the new spec in 2012.

    I implemented Reflect, with code reviews by Jeff Walden. It’ll be in Firefox Nightly starting this weekend—all except Reflect.enumerate(), which is not implemented yet.

    Next up, we’ll be talking about the most controversial feature in ES6, and who better to present it than the person who’s implementing it in Firefox? So please join us next week as Mozilla engineer Eric Faust presents ES6 classes in depth.

  4. ES6 In Depth: Generators, continued

    ES6 In Depth is a series on new features being added to the JavaScript programming language in the 6th Edition of the ECMAScript standard, ES6 for short.

    Welcome back to ES6 In Depth! I hope you had as much fun as I did during our summer break. But the life of a programmer cannot be all fireworks and lemonade. It’s time to pick up where we left off—and I’ve got the perfect topic to resume with.

    Back in May, I wrote about generators, a new kind of function introduced in ES6. I called them the most magical feature in ES6. I talked about how they might be the future of asynchronous programming. And then I wrote this:

    There is more to say about generators… But I think this post is long and bewildering enough for now. Like generators themselves, we should pause, and take up the rest another time.

    Now is the time.

    You can find part 1 of this article here. It’s probably best to read that before reading this. Go on, it’s fun. It’s… a little long and bewildering. But there’s a talking cat!

    A quick revue

    Last time, we focused on the basic behavior of generators. It’s a little strange, perhaps, but not hard to understand. A generator-function is a lot like a regular function. The main difference is that the body of a generator-function doesn’t run all at once. It runs a little bit at a time, pausing each time execution reaches a yield expression.

    There’s a detailed explanation in part 1, but we never did a thorough worked example of how all the parts come together. Let’s do that now.

    function* someWords() {
      yield "hello";
      yield "world";
    }
    
    for (var word of someWords()) {
      alert(word);
    }
    

    This script is straightforward enough. But if you could observe everything that happens here, as though all the various bits of code were characters in a play, that would be a rather different script. It might go something like this:

    SCENE - INTERIOR COMPUTER, DAY
    
    FOR LOOP stands alone onstage, wearing a hard hat and
    carrying a clipboard, all business.
    
                              FOR LOOP
                             (calling)
                            someWords()!
                                  
    The GENERATOR appears: a tall, brass, clockwork gentleman.
    It looks friendly enough, but it's still as a statue.
    
                              FOR LOOP
                   (clapping her hands smartly)
               All right! Let's get some stuff done.
                         (to the generator)
                              .next()!
    
    The GENERATOR springs to life.
    
                             GENERATOR
                   {value: "hello", done: false}
    
    It freezes in a goofy pose.
    
                              FOR LOOP
                               alert!
    
    Enter ALERT at a sprint, wide-eyed and breathless. We sense
    he's always like that.
    
                              FOR LOOP
                       Tell the user "hello".
    
    ALERT turns around and sprints offstage.
    
                               ALERT
                       (offstage, screaming)
                          Stop everything!
              The web page at hacks.mozilla.org says,
                              "hello"!
    
    A few seconds' pause, then ALERT races back on, crossing all
    the way over to FOR LOOP and skidding to a halt.
    
                               ALERT
                         The user says OK.
    
                              FOR LOOP
                   (clapping her hands smartly)
               All right! Let's get some stuff done.
                  (turning back to the GENERATOR)
                              .next()!
    
    The GENERATOR again springs to life.
    
                             GENERATOR
                   {value: "world", done: false}
    
    It freezes in a different goofy pose.
    
                              FOR LOOP
                               alert!
    
                               ALERT
                         (already running)
                               On it!
                       (offstage, screaming)
                          Stop everything!
              The web page at hacks.mozilla.org says,
                              "world"!
    
    Again, a pause, then ALERT trudges back onstage, suddenly
    crestfallen.
    
                               ALERT
                   The user says OK again, but...
                 but please prevent this page from
                   creating additional dialogues.
    
    He exits, pouting.
    
                              FOR LOOP
                   (clapping her hands smartly)
               All right! Let's get some stuff done.
                  (turning back to the GENERATOR)
                              .next()!
    
    The GENERATOR springs to life a third time.
    
                             GENERATOR
                           (with dignity)
                   {value: undefined, done: true}
    
    Its head comes to rest on its chest and the lights go out of
    its eyes. It will never move again.
    
                              FOR LOOP
                      Time for my lunch break.
    
    She exits.
    
    After a while, the GARBAGE COLLECTOR enters, picks up the
    lifeless GENERATOR, and carries it offstage.
    

    All right—it’s not exactly Hamlet. But you get the picture.

    As you can see in the play, when a generator object first appears, it is paused. It wakes up and runs for a bit each time its .next() method is called.

    The action is synchronous and single-threaded. Note that only one of these characters is actually doing anything at any given time. The characters never interrupt each other or talk over one another. They take turns speaking, and whoever’s talking can go on as long as they want. (Just like Shakespeare!)

    And some version of this drama unfolds each time a generator is fed to a forof loop. There is always this sequence of .next() method calls that do not appear anywhere in your code. Here I’ve put it all onstage, but for you and your programs, all this will happen behind the scenes, because generators and the forof loop were designed to work together, via the iterator interface.

    So to summarize everything up to this point:

    • Generator objects are polite brass robots that yield values.
    • Each robot’s programming consists of a single chunk of code: the body of the generator function that created it.

    How to shut down a generator

    Generators have several fiddly extra features that I didn’t cover in part 1:

    • generator.return()
    • the optional argument to generator.next()
    • generator.throw(error)
    • yield*

    I skipped them mainly because without understanding why those features exist, it’s hard to care about them, much less keep them all straight in your head. But as we think more about how our programs will use generators, we’ll see the reasons.

    Here’s a pattern you’ve probably used at some point:

    function doThings() {
      setup();
      try {
        // ... do some things ...
      } finally {
        cleanup();
      }
    }
    
    doThings();
    

    The cleanup might involve closing connections or files, freeing system resources, or just updating the DOM to turn off an “in progress” spinner. We want this to happen whether our work finishes successfully or not, so it goes in a finally block.

    How would this look in a generator?

    function* produceValues() {
      setup();
      try {
        // ... yield some values ...
      } finally {
        cleanup();
      }
    }
    
    for (var value of produceValues()) {
      work(value);
    }
    

    This looks all right. But there is a subtle issue here: the call work(value) isn’t inside the try block. If it throws an exception, what happens to our cleanup step?

    Or suppose the forof loop contains a break or return statement. What happens to the cleanup step then?

    It executes anyway. ES6 has your back.

    When we first discussed iterators and the forof loop, we said the iterator interface contains an optional .return() method which the language automatically calls whenever iteration exits before the iterator says it’s done. Generators support this method. Calling myGenerator.return() causes the generator to run any finally blocks and then exit, just as if the current yield point had been mysteriously transformed into a return statement.

    Note that the .return() is not called automatically by the language in all contexts, only in cases where the language uses the iteration protocol. So it is possible for a generator to be garbage collected without ever running its finally block.

    How would this feature play out on stage? The generator is frozen in the middle of a task that requires some setup, like building a skyscraper. Suddenly someone throws an error! The for loop catches it and sets it aside. She tells the generator to .return(). The generator calmly dismantles all its scaffolding and shuts down. Then the for loop picks the error back up, and normal exception handling continues.

    Generators in charge

    So far, the conversations we’ve seen between a generator and its user have been pretty one-sided. To break with the theater analogy for a second:

    (A fake screenshot of iPhone text messages between a generator and its user, with the user just saying 'next' repeatedly and the generator replying with values.)

    The user is in charge. The generator does its work on demand. But this isn’t the only way to program with generators.

    In part 1, I said that generators could be used for asynchronous programming. Things you currently do with asynchronous callbacks or promise chaining could be done with generators instead. You may have wondered how exactly that is supposed to work. Why is the ability to yield (which after all is a generator’s only special power) sufficient? After all, asynchronous code doesn’t just yield. It makes stuff happen. It calls for data from files and databases. It fires off requests to servers. And then it returns to the event loop to wait for those asynchronous processes to finish. How exactly will generators do this? And without callbacks, how does the generator receive data from those files and databases and servers when it comes in?

    To start working toward the answer, consider what would happen if we just had a way for the .next() caller to pass a value back into the generator. With just this one change, we could have a whole new kind of conversation:

    (A fake screenshot of iPhone text messages between a generator and its caller; each value the generator yields is an imperious demand, and the caller passes whatever the generator wants as an argument the next time it calls .next().)

    And a generator’s .next() method does in fact take an optional argument, and the clever bit is that the argument then appears to the generator as the value returned by the yield expression. That is, yield isn’t a statement like return; it’s an expression that has a value, once the generator resumes.

      var results = yield getDataAndLatte(request.areaCode);
    

    This does a lot of things for a single line of code:

    • It calls getDataAndLatte(). Let’s say that function returns the string "get me the database records for area code..." that we saw in the screenshot.
    • It pauses the generator, yielding the string value.
    • At this point, any amount of time could pass.
    • Eventually, someone calls .next({data: ..., coffee: ...}). We store that object in the local variable results and continue on the next line of code.

    To show that in context, here’s code for the entire conversation shown above:

    function* handle(request) {
      var results = yield getDataAndLatte(request.areaCode);
      results.coffee.drink();
      var target = mostUrgentRecord(results.data);
      yield updateStatus(target.id, "ready");
    }
    

    Note how yield still just means exactly what it meant before: pause the generator and pass a value back to the caller. But how things have changed! This generator expects very specific supportive behavior from its caller. It seems to expect the caller to act like an administrative assistant.

    Ordinary functions are not usually like that. They tend to exist to serve their caller’s needs. But generators are code you can have a conversation with, and that makes for a wider range of possible relationships between generators and their callers.

    What might this administrative assistant generator-runner look like? It doesn’t have to be all that complicated. It might look like this.

    function runGeneratorOnce(g, result) {
      var status = g.next(result);
      if (status.done) {
        return;  // phew!
      }
    
      // The generator has asked us to fetch something and
      // call it back when we're done.
      doAsynchronousWorkIncludingEspressoMachineOperations(
        status.value,
        (error, nextResult) => runGeneratorOnce(g, nextResult));
    }
    

    To get the ball rolling, we would have to create a generator and run it once, like this:

      runGeneratorOnce(handle(request), undefined);
    

    In May, I mentioned Q.async() as an example of a library that treats generators as asynchronous processes and automatically runs them as needed. runGeneratorOnce is that sort of thing. In practice, generator will not yield strings spelling out what they need the caller to do. They will probably yield Promise objects.

    If you already understand promises, and now you understand generators, you might want to try modifying runGeneratorOnce to support promises. It’s a difficult exercise, but once you’re done, you’ll be able to write complex asynchronous algorithms using promises as straight-line code, not a .then() or a callback in sight.

    How to blow up a generator

    Did you notice how runGeneratorOnce handles errors? It ignores them!

    Well, that’s not good. We would really like to report the error to the generator somehow. And generators support this too: you can call generator.throw(error) rather than generator.next(result). This causes the yield expression to throw. Like .return(), the generator will typically be killed, but if the current yield point is in a try block, then catch and finally blocks are honored, so the generator may recover.

    Modifying runGeneratorOnce to make sure .throw() gets called appropriately is another great exercise. Keep in mind that exceptions thrown inside generators are always propagated to the caller. So generator.throw(error) will throw error right back at you unless the generator catches it!

    This completes the set of possibilities when a generator reaches a yield expression and pauses:

    • Someone may call generator.next(value). In this case, the generator resumes execution right where it left off.
    • Someone may call generator.return(), optionally passing a value. In this case, the generator does not resume whatever it was doing. It executes finally blocks only.
    • Someone may call generator.throw(error). The generator behaves as if the yield expression were a call to a function that threw error.
    • Or, maybe nobody will do any of those things. The generator might stay frozen forever. (Yes, it is possible for a generator to enter a try block and simply never execute the finally block. A generator can even be reclaimed by the garbage collector while it’s in this state.)

    This is not much more complicated than a plain old function call. Only .return() is really a new possibility.

    In fact, yield has a lot in common with function calls. When you call a function, you’re temporarily paused, right? The function you called is in control. It might return. It might throw. Or it might just loop forever.

    Generators working together

    Let me show off one more feature. Suppose we write a simple generator-function to concatenate two iterable objects:

    function* concat(iter1, iter2) {
      for (var value of iter1) {
        yield value;
      }
      for (var value of iter2) {
        yield value;
      }
    }
    

    ES6 provides a shorthand for this:

    function* concat(iter1, iter2) {
      yield* iter1;
      yield* iter2;
    }
    

    A plain yield expression yields a single value; a yield* expression consumes an entire iterator and yields all values.

    The same syntax also solves another funny problem: the problem of how to call a generator from within a generator. In ordinary functions, we can scoop up a bunch of code from one function and refactor it into a separate function, without changing behavior. Obviously we’ll want to refactor generators too. But we’ll need a way to call the factored-out subroutine and make sure that every value we were yielding before is still yielded, even though it’s a subroutine that’s producing those values now. yield* is the way to do that.

    function* factoredOutChunkOfCode() { ... }
    
    function* refactoredFunction() {
      ...
      yield* factoredOutChunkOfCode();
      ...
    }
    

    Think of one brass robot delegating subtasks to another. You can see how important this idea is for writing large generator-based projects and keeping the code clean and organized, just as functions are crucial for organizing synchronous code.

    Exeunt

    Well, that’s it for generators! I hope you enjoyed that as much as I did, too. It’s good to be back.

    Next week, we’ll talk about yet another mind-blowing feature that’s totally new in ES6, a new kind of object so subtle, so tricky, that you may end up using one without even knowing it’s there. Please join us next week for a look at ES6 proxies in depth.

  5. Developer Edition 41: View source in a tab, screenshot elements, HAR files, and more

    When we introduced the new Performance tools a few weeks ago, we also talked about how the Firefox Dev Tools team had spent a lot of time focusing on user feedback and what we call ‘polish’ bugs – things reported via our UserVoice feedback channel and Bugzilla. Even though the Firefox 41 was a short release cycle for us, this focus on user feedback continues to pay off — several new features that our community had been asking for landed in time for the release. Here’s a closer look:

    Screenshot the selected node in the Inspector

    New contributor Léon McGregor implemented an interesting suggestion that was posted in UserVoice. This functionality has been available via the gcli ‘screenshot’ command for quite some time, but is much more discoverable and useful as a context menu item. When the screenshot is created, Firefox copies it to your configured downloads directory.

    Create a screenshot of the currently selected element.

    Create a screenshot of the currently selected element.

    View source in tab

    Starting with Firefox 41, when you right-click and select View Page Source, the html source view will open in a tab instead of a new window. This was a hugely popular request and we would have shipped it earlier but what started out as a seemingly simple change was actually quite involved: See the bug linked below for all the gory details. Importantly, we have also ensured that View Page Source provides you with the source of the page as-is from Firefox’s cache – we do not fetch a new version.

    'View source' always opens in a tab now.

    View Page Source always opens in a tab now.

    Add Rules button

    It’s very convenient to be able to add a new rule to the Inspector as you work, and this is a feature from Firebug that users have requested for some time. During this last cycle, we spent some time polishing our implementation, and provided the convenience of a UI button in addition to the context menu command.

    We've added a button to the inspector so you can quickly add a new css rule.

    We’ve added a button to the inspector so you can quickly add a new css rule.

    “Copy as HAR” and “Save all as HAR”

    Another feature from Firebug that is particularly popular with Selenium users is the ability to export HAR archives for the current page.

    You can now export HAR archives directly from the network monitor.

    You can now export HAR archives directly from the network monitor.

    Other notable changes

    In total, 140 Developer Tools bugs have been fixed in Firefox since June 1st. On behalf of the team, I’d like to thank all of the people who reported bugs, tested patches, and spent many hours working to improve this version of Firefox Developer Edition, and especially these contributors that fixed bugs: edoardo.putti, fayolle-florent, 15electronicmotor, veeti.paananen, sr71pav, sjakthol, ntim, MattN, lemcgregor3, and indiasuny000. Thanks!.

    • Bug 1164210 – $$() should return a true Array
    • Bug 1077339 – Display keyboard shortcuts when hovering panel tabs
    • Bug 1163183 – Show HTML5 Forms pseudo elements in the rule view
    • Bug 1165576 – Netmonitor theme refresh
    • Bug 1049888 – Make the storage actor work in e10s and Firefox OS
    • Bug 987365 – Add pseudo-class lock options to rule view
    • Bug 1059882 – Frame selection command button should be visible by default
    • Bug 1143224 – Opening the netmonitor slows down requests on the page
    • Bug 1119133 – Keyboard shortcut to toggle devtools docking mode between last two positions
    • Bug 1024693 – Copy CSS declarations
    • Bug 1050691 – Click on a function on the console should go to the debugger

    Download Firefox Developer Edition 41 now. Let us know what you think and what you’d like to see in future releases. We’re paying attention.

  6. Compacting Garbage Collection in SpiderMonkey

    Overview

    Compacting is a new feature of our garbage collector, released in Firefox 38, that allows us to reduce external fragmentation in the JavaScript heap. The aim is to use less memory in general and to be able to recover from more out-of-memory situations. So far, we have only implemented compacting for JavaScript objects, which are one of several kinds of garbage-collected cells in the heap.

    The problem

    The JavaScript heap is made up of 4K blocks of memory called arenas, each of which is divided into fixed-size cells. Different arenas are used to allocate different kinds of cells; each arena only contains cells of the same size and kind.

    The heap contains various kinds of cell, including those for JavaScript objects, strings, and symbols, as well as several internal kinds such as scripts (used to represent units of JS code), shapes (used to determine the layout of object properties in memory), and jitcode (compiled JIT code). Of these, object cells usually take up the most memory.

    An arena cannot be freed while it contains any live cells. Cells allocated at the same time may have different lifetimes and so a heap may end up in a state where there are many arenas that contain only a few cells. New cells of the same kind can be allocated into this space, but the space cannot be used for cells of a different kind or returned to the operating system if memory is low.

    Here is a simplified diagram of some data on the heap showing arenas containing two different kinds of cell:

    heap layout smallest

    Note that if the free space in arena 3 were used to hold the cells in arena 5, we could free up a whole arena.

    Measuring wasted heap space

    You can see how much memory these free cells are taking up by navigating to about:memory and hitting the ‘Measure’ button. The totals for the different kinds of cell are shown under the section js-main-runtime-gc-heap-committed/unused/gc-things. (If you’re not used to interpreting the about:memory reports, there is some documentation here).

    Here’s a screenshot of the whole js-main-runtime-gc-heap-committed section with compacting GC disabled, showing the difference between ‘used’ and ‘unused’ sizes:

    unused heap screenshot

    I made some rough measurements of my normal browsing profile with and without compacting GC (details of how to do this are below at the end of the post). The profile consisted of Google Mail, Calendar, many bugzilla tabs and various others (~50 tabs total), and I obtained the following readings:

    Total explicit allocations Unused cells
    Before compacting 1,324.46 MiB 69.58 MiB
    After compacting 1,296.28 MiB 40.18 MiB

    This shows a reduction of 29.4MiB (mebibytes) worth of explicit allocations. That’s only about 2% of total allocations, but accounts for over 8% of the space taken up by the JS heap.

    How does compacting work?

    To free up this space we have to allow the GC to move cells between arenas. That way it can consolidate the live cells in fewer arenas and reuse the unused space. Of course, this is easier said than done, as every pointer to a moved cell must be updated. Missing a single one is a sure-fire way to make the browser crash!

    Also, this is a potentially expensive operation as we have to scan many cells to find the pointers we need to update. Therefore the idea is to compact the heap only when memory is low or the user is inactive.

    The algorithm works in three phases:

    1. Select the cells to move.
    2. Move the cells.
    3. Update the pointers to those cells.

    Selecting the cells to move

    We want to move the minimum amount of data and we want to do it without allocating any more memory, as we may be doing this when we don’t have any free memory. To do this, we take all the arenas with free space in them and put them in a list arranged in decreasing order of the number of free cells they contain. We split this list into two parts at the first point at which the preceding arenas have enough free cells to contain the used cells in the subsequent arenas. We will move all the cells out of the subsequent arenas.

    Moving the cells

    We allocate a new cell from one of the arenas we are not moving. The previous step ensures there is always enough space for this. Then we copy the data over from the original location.

    In some cases, we know the cell contains pointers to itself and these are updated at this point. The browser may have external references to some kinds of object and so we also call an optional hook here to allow these to be updated.

    When we have moved a cell, we update the original location with a forwarding pointer to the new location, so we can find it later. This also marks the cell, indicating to the GC that the cell has been moved, when updating pointers in the next phase.

    Updating pointers to moved cells

    This is the most demanding part of the compacting process. In general, we don’t know which cells may contain pointers to cells we have moved, so it seems we have to iterate through all cells in the heap. This would be very expensive.

    We cut down this cost in a number of ways. Firstly, note that the heap is split into several zones (there is a zone per browser tab, and others for system use). Compacting is performed per-zone, since in general cells do not have cross-zone pointers (these are handled separately). Compacting per zone allows us to spread the total cost over many incremental slices.

    Secondly, not every kind of cell can contain pointers to every other kind of cell (indeed not all kinds of cells can contain pointers) so some kinds of cell can be excluded from the search.

    Finally, we can parallelise this work and make use of all CPU resources available.

    It’s important to note that this work was enabled by our shift to exact stack rooting, described in this blog post. It is only possible to move objects if we know which stack locations are roots, otherwise we could overwrite unrelated data on the stack if it happened to look like a moved cell pointer.

    Scheduling heap compaction

    As mentioned earlier, compacting GC doesn’t run every time we collect. Currently it is triggered on three events:

    • We ran out of memory and we’re performing a last-ditch attempt to free up some space
    • The OS has sent us a memory pressure event
    • The user has been inactive for some length of time (currently 20 seconds)

    The first two should allow us to avoid some out-of-memory situations, while the last aims to free up memory without affecting the user’s browsing experience.

    Conclusion

    Hopefully this has explained the problem compacting GC is trying to solve, and how it’s done.

    One unexpected benefit of implementing compacting GC is that it showed us a couple of places where we weren’t correctly tracing cell pointers. Errors like this can cause hard-to-reproduce crashes or potential security vulnerabilities, so this was an additional win.

    Ideas for future work

    The addition of compacting is an important step in improving our GC, but it’s not the end by any means. There are several ways in which we can continue to develop this:

    Currently we only compact cells corresponding to JavaScript objects, but there are several other kinds of cells in the heap. Moving these would bring greater memory savings.

    Is it possible to determine in advance which cells contain pointers to cells we want to move? If we had this information we could cut the cost of compacting. One possibility is to scan the heap in the background to determine this information, but we would need to be able to detect changes made by the mutator.

    The current algorithm mixes together cells allocated at different times. Cells with similar lifetimes are often allocated at the same time, so this may not be the best strategy.

    If compacting can be made fast enough, we might be able to do it whenever the collector sees a certain level of fragmentation in the heap.

    How to measure heap space freed up by compacting

    To measure roughly how much space is freed by compacting, you can perform the following steps:

    1. Disable compacting by navigating to about:config and setting javascript.options.mem.gc_compacting to false.
    2. It makes it easier to disable multiprocess Firefox as well at this point. This can be done from the main Preferences page.
    3. Restart the browser and open some tabs. I used ‘Reload all tabs’ to open all my pages from last time. Wait for everything to load.
    4. Open about:memory and force a full GC by clicking ‘Minimize memory usage’ and then click ‘Measure.’ Since memory usage can take a while to settle down, I repeated this a few times until I got a consistent number.
    5. Note the total ‘explicit’ size and that of js-main-runtime-gc-heap-committed/unused/gc-things.
    6. Enable compacting again by setting javascript.options.mem.gc_compacting to true. There is no need to restart for this to take effect.
    7. Click ‘Minimize memory usage’ again and then ‘Measure.’
    8. Compare the new readings to the previous.

    This does not give precise readings as all kinds of things might be happening in the background, but it can provide a good ballpark figure.

  7. How fast are web workers?

    The next version of Firefox OS, the mobile operating system, will unleash the power of devices by taking full advantage of their multi-core processors. Classically, JavaScript has been executed on a single thread, but web workers offer a way to execute code in parallel. Doing so frees the browser of anything that may get in the way of the main thread so that it can smoothly animate the UI.

    A brief introduction to web workers

    There are several types of web workers:

    They each have specific properties, but share a similar design. The code running in a worker is executed in its own separate thread and runs in parallel with the main thread and other workers. The different types of workers all share a common interface.

    Web workers

    Dedicated web workers are instantiated by the main process and they can communicate only with it.

    Shared workers

    Shared workers can be reached by all processes running on the same origin (different browser tabs, iframes or other shared workers).

    Service workers

    Service workers have gained a lot of attention recently. They make it possible to proxy a web server programmatically to deliver specific content to the requester (e.g. the main thread). One use case for service workers is to serve content while offline. Service workers are a very new API, not fully implemented in all browsers, and are not covered in this article.

    In order to verify that web workers make Firefox OS run faster, we need to validate their speed by benchmarking them.

    The cost of creating web workers

    This article focuses on Firefox OS. All measurement are made on a Flame device, powered by middle-end hardware.

    The first set of benchmarks will look at the time it takes to create web workers. To do that, we set up a script that instantiates a web worker and sends a minimal message, to which the worker replies immediately. Once the response is received by the main thread, the time that the operation takes is calculated. The web worker is destroyed and the operation is repeated enough times to get a good idea of how long it takes on average to get a functional web worker. Instantiating a web worker is as easy as:

    // Start a worker.
    var worker = new Worker('worker-script.js');
     
    // Terminate a worker.
    worker.terminate();

    The same method is applied to the creation of broadcast channel:

    // Open a broadcast channel.
    var channel = new window.BroadcastChannel('channel-name');
     
    // Close a broadcast channel.
    channel.close();

    Shared workers can’t really be benchmarked here because once they are created, the developer can’t destroy them. The browser is entirely responsible for their lifetime. For that reason, we can’t create and destroy shared workers at will to get a meaningful benchmark.

    Web workers take about 40 ms to be instantiated. Also, this time is pretty stable with variations of only a few milliseconds. Setting up a broadcast channel is usually done within 1 ms.

    Under normal circumstances, the browser UI is refreshed at a rate of 60 frames per second. This means that no JavaScript code should run longer than the time needed by a frame, i.e., 16.66ms (60 frames per second). Otherwise, you may introduce jankiness and lag in your application.

    Instantiating web workers is pretty efficient, but still may not fit in the time allocated for a single frame. That’s why it’s important to create as few web workers as possible and reuse them.

    Message latency

    A critical aspect of web workers is having fast communication between your main thread and the workers. There are two different ways the main browser thread can communicate with a web worker.

    postMessage

    This API is the default and preferred way to send and receive messages from a web worker. postMessage() is easy to use:

    // Send a message to the worker.
    worker.postMessage(myMessage);
     
    // Listen to messages from the worker.
    worker.onmessage = evt => {
      var message = evt.data;
    };

    Broadcast Channel

    This is a newly implemented API, only available in Firefox at the time of this writing. It lets us broadcast messages to all contexts sharing the same origin. All browser tabs, iframes, or workers served from the same origin can emit and receive messages:

    // Send a message to the broadcast channel.
    channel.postMessage(myMessage);
     
    // Listen to messages from the broadcast channel.
    channel.onmessage = evt => {
      var message = evt.data;
    };

    To benchmark this, we use a script similar to the one described above, except that the web worker is not destroyed and reused at each operation. The time to get a round trip response should be divided by two.

    As you might expect, the simple postMessage is fast. It usually takes between 0 to 1 ms to send a message, whether to a web or shared worker. Broadcast channel API takes about 1 to 2 ms.

    Under normal circumstances, exchanging messages with workers is fast and you should not feel too concerned about speed here. However, larger messages can take longer.

    The size of messages

    There are 2 ways to send messages to web workers:

    • Copying the message
    • Transferring the message

    In the first case, the message is serialized, copied, and sent over. In the latter, the data is transferred. This means that the original sender can no longer use it once sent. Transferring data is almost instantaneous, so there is no real point in benchmarking that. However, only ArrayBuffer is transferable.

    As expected, serializing, copying, and de-serializing data adds significant overhead to the message transmission. The bigger the message, the longer it takes to be sent.

    The benchmark here sends a typed array to a web worker. Its size is progressively increased at each iteration. There is a linear correlation between size of the message and transfer time. For each measurement, we can divide the size (in kilobytes) by the time (in milliseconds) to get the transfer speed in kb/ms.

    Typically, on a Flame, the transfer speed is 80 kB/ms for postMessage and 12 kB/ms using broadcast channel. This means that if you want your message to fit in a single frame, you should keep it under 1,300 kB with postMessage and under 200 kB when using the broadcast channel. Otherwise, it may introduce frame drop in your application.

    In this benchmark, we use typed arrays, because it makes it possible to determine their size in kilobytes precisely. You can also transfer JavaScript objects, but due to the serialization process, they take longer to post. For small objects, this doesn’t really matter, but if you need to send huge objects, you may as well serialize them to a binary format. You can use something similar to Protocol Buffer.

    Web workers are fast if used correctly

    Here is a quick summary of various benchmarks related to web workers, as measured on a Flame:

    Operation Value
    Instantiation of a web worker 40 ms
    Instantiation of a broadcast channel 1 ms
    Communication latency with postMessage 0.5 ms
    Communication latency with broadcast channel 1.5 ms
    Communication speed with postMessage 80 kB/ms
    Communication speed with broadcast channel 12 kB/ms
    Maximum message size with postMessage 1,300 kB
    Maximum message size with broadcast channel 200 kB

    Benchmarking is the only way to make sure that the solution you are implementing is fast. This process takes much of the guesswork out of web development.

    If you want to run these benchmarks on a specific device, the app I built to make these measurements, web workers benchmark, is open source. You are also welcome to contribute by submitting new types of benchmarks.

  8. Streaming media on demand with Media Source Extensions

    Introducing MSE

    Media Source Extensions (MSE) is a new addition to the Web APIs available in all major browsers.  This API allows for things like adaptive bitrate streaming of video directly in our browser, free of plugins. Where previously we may have used proprietary solutions like RTSP (Real Time Streaming Protocol) and Flash, we can now use simpler protocols like HTTP to fetch content, and MSE to smoothly stitch together video segments of varied quality.

    All browsers that support HTMLMediaElements, such as audio and video tags, already make byte-range requests for subsequent segments of media assets.  One problem is that it’s up to each browser’s implementation of a media engine to decide when and how much to fetch.  It’s also tough to stitch together or deliver smooth playback of segments of different quality without pauses, gaps, flashes, clicks, or pops.  MSE gives us finer-grained control at the application level for fetching and playing back content.

    In order to begin streaming, we need to figure out how to transcode our assets into a meaningful byte format for browsers’ media engines, determine what abstractions MSE provides, and figure out how to instruct the browser to play them back.

    Having multiple resolutions of content allows us to switch between them while maintaining a constant viewport size.  This is known as upscaling, and it’s a common technique for real-time rendering in video games to meet a required frame time.  By switching to a lower quality video resolution, we can meet bandwidth limitations at the cost of fidelity.  The loss of fidelity causes such artifacts as aliasing, in which curves appear jagged and blocky.  This technique can often be seen by Netflix subscribers during peak viewing hours.

    Rather than having an advanced protocol like RTSP handle bandwidth estimates, we can use a simpler network protocol like HTTP and move the advanced logic up one level into the application logic.

    Transcoding

    My recommended tools, ffmpeg and Bento4, are both free and open-source software (FOSS). ffmpeg is our Swiss army knife of transcoding, and Bento4 is a collection of great tools for working with mp4.  While I’m partial to non-licensed codecs like webm/vp8-9/opus, current browser support for those containers and codecs is rather poor, so in this post we’ll just be working with mp4/h.264/aac.  Both of the tools I’m working with are command line utilities; if you have nice GUI tools in your asset pipeline you’d like to recommend to our readers, let us know in the comments below.

    We’ll start with a master of some file, and end up transcoding it into multiple files each of smaller resolutions, then segmenting the smaller-res whole files into a bunch of tiny files.  Once we have a bunch of small files (imagine splitting your video into a bunch of 10-second segments), the client can use more advanced heuristics for fetching the preferred next segment.

    MSE multiple resolutions

    Our smaller-res copies of the master asset

     Proper fragmentation

    When working with mp4 and MSE, it helps to know that the mp4 files should be structured so that metadata is fragmented across pieces of the container, and across the actual audio/video streams being fragmented, instead of clustered together.  This is specified in the ISO BMFF Byte Stream Format spec, section 3:

    “An ISO BMFF initialization segment is defined in this specification as a single File Type Box (ftyp) followed by a single Movie Header Box (moov).”

    This is really important: Simply transcoding to an mp4 container in ffmpeg does not have the expected format and thus fails when trying to play back in a browser with MSE.  To check and see if your mp4 is properly fragmented, you can run Bento4’s mp4dump on your mp4.

    If you see something like:

      $ ./mp4dump ~/Movies/devtools.mp4 | head
      [ftyp] size=8+24
        ...
      [free] size=8+0
      [mdat] size=8+85038690
      [moov] size=8+599967
        ...

    Then your mp4 won’t be playable since the [ftyp] “atom” is not followed immediately by a [moov] “atom.”  A properly fragmented mp4 looks something like this —

      $ ./mp4fragment ~/Movies/devtools.mp4 devtools_fragmented.mp4
      $ ./mp4dump devtools_fragmented.mp4 | head
      [ftyp] size=8+28
        ...
      [moov] size=8+1109
        ...
      [moof] size=8+600
        ...
      [mdat] size=8+138679
      [moof] size=8+536
        ...
      [mdat] size=8+24490
        ...
      ...

    — where mp4fragment is another Bento4 utility.  The properly fragmented mp4 has the [ftyp] followed immediately by a [moov], then subsequent [moof]/[mdat] pairs.

    It’s possible to skip the need for mp4fragment by using the -movflags frag_keyframe+empty_moov flags when transcoding to an mp4 container with ffmpeg, then checking with mp4dump:

      $ ffmpeg -i bunny.y4m -movflags frag_keyframe+empty_moov bunny.mp4
    Creating multiple resolutions

    If we want to switch resolutions, we can then run our fragmented mp4 through Bento4’s mp4-dash-encode.py script to get multiple resolutions of our video.  This script will fire up ffmpeg and other Bento4 tools, so make sure they are both available in your $PATH environment variable.

    $ python2.7 mp4-dash-encode.py -b 5 bunny.mp4
    $ ls
    video_00500.mp4 video_00875.mp4 video_01250.mp4 video_01625.mp4 video_02000.mp4
    Segmenting

    We now have 5 different copies of our video with various bit rates and resolutions. To be able to switch between them easily during playback, based on our effective bandwidth that changes constantly over time, we need to segment the copies and produce a manifest file to facilitate playback on the client.  We’ll create a Media Presentation Description (MPD)-style manifest file. This manifest file containing info about the segments, such as the threshold effective bandwidth for fetching the requisite segment.

    Bento4’s mp4-dash.py script can take multiple input files, perform the segmentation, and emit a MPD manifest that most DASH clients/libraries understand.

    $ python2.7 mp4-dash.py --exec-dir=. video_0*
    ...
    $ tree -L 1 output
    output
    ├── audio
    │   └── und
    ├── stream.mpd
    └── video
        ├── 1
        ├── 2
        ├── 3
        ├── 4
        └── 5
    
    8 directories, 1 file

    We should now have a folder with segmented audio and segmented video of various resolutions.

    MSE & Playback

    With an HTMLMediaElement such as an audio or video tag, we simply assign a URL to the element’s src attribute and the browser handles fetching and playback.  With MSE, we will fetch the content ourselves with XMLHttpRequests (XHRs) treating the response as an ArrayBuffer (raw bytes), and assigning the src attribute of the media element to a URL that points to a MediaSource object.  We may then append SourceBuffer objects to the MediaSource.

    Pseudocode for the MSE workflow might look like:

    let m = new MediaSource
    m.onsourceopen = () =>
      let s = m.addSourceBuffer('codec')
      s.onupdateend = () =>
        if (numChunks === totalChunks)
          m.endOfStream()
        else
          s.appendBuffer(nextChunk)
      s.appendBuffer(arrayBufferOfContent)
    video.src = URL.createObjectURL(m)
    

    Here’s a trick to get the size of a file: make an XHR with the HTTP HEAD method.  A response to a HEAD request will have the content-length header specifying the body size of the response, but unlike a GET, it does not actually have a body.  You can use this to preview the size of a file without actually requesting the file contents.  We can naively subdivide the video and fetch the next segment of video when we’re 80% of the way through playback of the current segment.  Here’s a demo of this in action and a look at the code.

    Note: You’ll need the latest Firefox Developer Edition browser to view the demo and test the code. More information below in the Compatibility section. The MSE primer from WebPlatform.org docs is another great resource to consult.

    My demo is a little naive and has a few issues:

    • It doesn’t show how to properly handle seeking during playback.
    • It assumes bandwidth is constant (always fetching the next segment at 80% playback of the previous segment), which it isn’t.
    • It starts off by loading only one segment (it might be better to fetch the first few, then wait to fetch the rest).
    • It doesn’t switch between segments of varying resolution, instead only fetching segments of one quality.
    • It doesn’t remove segments (part of the MSE API), although this can be helpful on memory constrained devices. Unfortunately, this requires you to re-fetch content when seeking backwards.

    These issues can all be solved with smarter logic on the client side with Dynamic Adaptive Streaming over HTTP (DASH).

    Compatibility

    Cross-browser codec support is a messy story right now; we can use MediaSource.isTypeSupported to detect codec support.  You pass isTypeSupported a string of the MIME type of the container you’re looking to play.  mp4 has the best compatibility currently. Apparently, for browsers that use the Blink rendering engine, MediaSource.isTypeSupported requires the full codec string to be specified.  To find this string, you can use Bento4’s mp4info utility:

    ./mp4info bunny.mp4| grep Codec
        Codecs String: avc1.42E01E

    Then in our JavaScript:

    if (MediaSource.isTypeSupported('video/mp4; codecs="avc1.42E01E, mp4a.40.2"')) {
    // we can play this
    }

    — where mp4a.40.2 is the codec string for low complexity AAC, the typical audio codec used in an mp4 container.

    Some browsers also currently whitelist certain domains for testing MSE, or over-aggressively cache CORS content, which makes testing frustratingly difficult.  Consult your browser for how to disable the whitelist or CORS caching when testing.

    DASH

    Using the MPD file we created earlier, we can grab a high quality DASH client implemented in JavaScript such as Shaka Player or dash.js.  Both clients implement numerous features, but could use more testing, as there are some subtle differences between media engines of various browsers.  Advanced clients like Shaka Player use an exponential moving average of three to ten samples to estimate the bandwidth, or even let you specify your own bandwidth estimator.

    If we serve our output directory created earlier with Cross Origin Resource Sharing (CORS) enabled, and point either DASH client to http://localhost:<port>/output/stream.mpd, we should be able to see our content playing.  Enabling video cycling in Shaka, or clicking the +/- buttons in dash.js should allow us to watch the content quality changing.  For more drastic/noticeable changes in quality, try encoding fewer bitrates than the five we demonstrated.

    Shaka Player in Firefox Dev Edition

    Shaka Player in Firefox Developer Edition

    dash.js running in Firefox Developer Edition

    dash.js in Firefox Developer Edition

    In conclusion

    In this post, we looked at how to prep video assets for on-demand streaming by pre-processing and transcoding.  We also took a peek at the MSE API, and how to use more advanced DASH clients.  In an upcoming post, we’ll explore live content streaming using the MSE API, so keep an eye out.  I recommend you use Firefox Developer Edition to test out MSE; lots of hard work is going into our implementation.

    Here are some additional resources for exploring MSE:

  9. Trainspotting: Firefox 39

    Trainspotting is a series of articles highlighting features in the lastest version of Firefox. A new version of Firefox is shipped every six weeks – we at Mozilla call this pattern “release trains.”

    A new version of Firefox is here, and with it come some great improvements and additions to the Web platform and developer tools. This post will call out a few highlights.

    For a full list of changes and additions, take a look at the Firefox 39 release notes.

    DevTools Love

    The Firefox Developer Tools are constantly getting better. We’re listening to developers on UserVoice, and using their feedback to make tools that are more powerful and easier to use. One requested feature was the ability to re-order elements in the Inspector:

    Editing and tweaking CSS Animations is easier than ever – Firefox 39 lets developers pause, restart, slow down, and preview new timings without having to switch applications.

    Menu of animation easing presets in the Inspector

    CSS Scroll Snap Points

    CSS Scroll Snap Points in action

    CSS Scroll Snap Points let web developers instruct the browser to smoothly snap element scrolling to specific points along an axis, creating smoother, easier to interact with interfaces with fewer lines of code.

    Improvements to Firefox on Mac OS X

    Firefox gets some Mac- specific improvements and updates in version 39:

    • Project Silk enabled – Improves scrolling and animation performance by more closely timing painting with hardware vsync. Read more about Project Silk.
    • Unicode 8.0 skin tone emoji – Fixed a bug in the rendering of skin tone modifiers for emoji.
    • Dashed line performance – Rendering of dotted and dashed lines is vastly improved. Check out the fixed bug for more information.

    Service Workers Progress

    Firefox’s implementation of the Service Workers API continues – fetch is enabled for workers and is now generally available to web content, and the Cache and CacheStorage are now available behind a flag.

    There’s lots more changes and improvements in Firefox 39 – check out the Developer Release Notes for developer-oriented changes or the full list of bugs fixed in this release. Enjoy!

  10. Performance Testing Firefox OS With Raptor

    When we talk about performance for the Web, a number of familiar questions may come to mind:

    • Why does this page take so long to load?
    • How can I optimize my JavaScript to be faster?
    • If I make some changes to this code, will that make this app slower?

    I’ve been working on making these types of questions easier to answer for Gaia, the UI layer for Firefox OS, a completely web-centric mobile device OS. Writing performant web pages for the desktop has its own idiosyncrasies, and writing native applications using web technologies takes the challenge up an order of magnitude. I want to introduce the challenges I’ve faced in making performance an easier topic to tackle in Firefox OS, as well as document my solutions and expose holes in the Web’s APIs that need to be filled.

    From now on, I’ll refer to web pages, documents, and the like as applications, and while web “documents” typically don’t need the performance attention I’m going to give here, the same techniques could still apply.

    Fixing the lifecycle of applications

    A common question I get asked in regard to Firefox OS applications:

    How long did the app take to load?

    Tough question, as we can’t be sure we are speaking the same language. Based on UX and my own research at Mozilla, I’ve tried at adopt this definition for determining the time it takes an application to load:

    The amount of time it takes to load an application is measured from the moment a user initiates a request for the application to the moment the application appears ready for user interaction.

    On mobile devices, this is generally from the time the user taps on an icon to launch an app, until the app appears visually loaded; when it looks like a user can start interacting with the application. Some of this time is delegated to the OS to get the application to launch, which is outside the control of the application in question, but the bulk of the loading time should be within the app.

    So window load right?

    With SPAs (single-page applications), Ajax, script loaders, deferred execution, and friends, window load doesn’t hold much meaning anymore. If we could merely measure the time it takes to hit load, our work would be easy. Unfortunately, there is no way to infer the moment an application is visually loaded in a predictable way for everyone. Instead we rely on the apps to imply these moments for us.

    For Firefox OS, I helped develop a series of conventional moments that are relevant to almost every application for implying its loading lifecycle (also documented as a performance guideline on MDN):

    navigation loaded (navigationLoaded)

    The application designates that its core chrome or navigation interface exists in the DOM and has been marked as ready to be displayed, e.g. when the element is not display: none or any other functionality that would affect the visibility of the interface element.

    navigation interactive (navigationInteractive)

    The application designates that the core chrome or navigation interface has its events bound and is ready for user interaction.

    visually loaded (visuallyLoaded)

    The application designates that it is visually loaded, i.e., the “above-the-fold” content exists in the DOM and has been marked as ready to be displayed, again not display: none or other hiding functionality.

    content interactive (contentInteractive)

    The application designates that it has bound the events for the minimum set of functionality to allow the user to interact with “above-the-fold” content made available at visuallyLoaded.

    fully loaded (fullyLoaded)

    The application has been completely loaded, i.e., any relevant “below-the-fold” content and functionality have been injected into the DOM, and marked visible. The app is ready for user interaction. Any required startup background processing is complete and should exist in a stable state barring further user interaction.

    The important moment is visually loaded. This correlates directly with what the user perceives as “being ready.” As an added bonus, using the visuallyLoaded metric pairs nicely with camera-based performance verifications.

    Denoting moments

    With a clearly-defined application launch lifecycle, we can denote these moments with the User Timing API, available in Firefox OS starting with v2.2:

    window.performance.mark( string markName )
    

    Specifically during a startup:

    performance.mark('navigationLoaded');
    performance.mark('navigationInteractive');
    ...
    performance.mark('visuallyLoaded');
    ...
    performance.mark('contentInteractive');
    performance.mark('fullyLoaded');
    

    You can even use the measure() method to create a measurement between start and another mark, or even 2 other marks:

    // Denote point of user interaction
    performance.mark('tapOnButton');
    
    loadUI();
    
    // Capture the time from now (sectionLoaded) to tapOnButton
    performance.measure('sectionLoaded', 'tapOnButton');
    

    Fetching these performance metrics is pretty straightforward with getEntries, getEntriesByName, or getEntriesByType, which fetch a collection of the entries. The purpose of this article isn’t to cover the usage of User Timing though, so I’ll move on.

    Armed with the moment an application is visually loaded, we know how long it took the application to load because we can just compare it to—oh, wait, no. We don’t know the moment of user intent to launch.

    While desktop sites may be able to easily procure the moment at which a request was initiated, doing this on Firefox OS isn’t as simple. In order to launch an application, a user will typically tap an icon on the Homescreen. The Homescreen lives in a process separate from the app being launched, and we can’t communicate performance marks between them.

    Solving problems with Raptor

    Without the APIs or interaction mechanisms available in the platform to be able to overcome this and other difficulties, we’ve build tools to help. This is how the Raptor performance testing tool originated. With it, we can gather metrics from Gaia applications and answer the performance questions we have.

    Raptor was built with a few goals in mind:

    • Performance test Firefox OS without affecting performance. We shouldn’t need polyfills, test code, or hackery to get realistic performance metrics.
    • Utilize web APIs as much as possible, filling in gaps through other means as necessary.
    • Stay flexible enough to cater to the many different architectural styles of applications.
    • Be extensible for performance testing scenarios outside the norm.
    Problem: Determining moment of user intent to launch

    Given two independent applications — Homescreen and any other installed application — how can we create a performance marker in one and compare it in another? Even if we could send our performance mark from one app to another, they are incomparable. According to High-Resolution Time, the values produced would be monotonically increasing numbers from the moment of the page’s origin, which is different in each page context. These values represent the amount of time passed from one moment to another, and not to an absolute moment.

    The first breakdown in existing performance APIs is that there’s no way to associate a performance mark in one app with any other app. Raptor takes a simplistic approach: log parsing.

    Yes, you read that correctly. Every time Gecko receives a performance mark, it logs a message (i.e., to adb logcat) and Raptor streams and parses the log looking for these log markers. A typical log entry looks something like this (we will decipher it later):

    I/PerformanceTiming( 6118): Performance Entry: clock.gaiamobile.org|mark|visuallyLoaded|1074.739956|0.000000|1434771805380
    

    The important thing to notice in this log entry is its origin: clock.gaiamobile.org, or the Clock app; here the Clock app created its visually loaded marker. In the case of the Homescreen, we want to create a marker that is intended for a different context altogether. This is going to need some additional metadata to go along with the marker, but unfortunately the User Timing API does not yet have that ability. In Gaia, we have adopted an @ convention to override the context of a marker. Let’s use it to mark the moment of app launch as determined by the user’s first tap on the icon:

    performance.mark('appLaunch@' + appOrigin)
    

    Launching the Clock from the Homescreen and dispatching this marker, we get the following log entry:

    I/PerformanceTiming( 5582): Performance Entry: verticalhome.gaiamobile.org|mark|appLaunch@clock.gaiamobile.org|80081.169720|0.000000|1434771804212
    

    With Raptor we change the context of the marker if we see this @ convention.

    Problem: Incomparable numbers

    The second breakdown in existing performance APIs deals with the incomparability of performance marks across processes. Using performance.mark() in two separate apps will not produce meaningful numbers that can be compared to determine a length of time, because their values do not share a common absolute time reference point. Fortunately there is an absolute time reference that all JS can access: the Unix epoch.

    Looking at the output of Date.now() at any given moment will return the number of milliseconds that have elapsed since January 1st, 1970. Raptor had to make an important trade-off: abandon the precision of high-resolution time for the comparability of the Unix epoch. Looking at the previous log entry, let’s break down its output. Notice the correlation of certain pieces to their User Timing counterparts:

    • Log level and tag: I/PerformanceTiming
    • Process ID: 5582
    • Base context: verticalhome.gaiamobile.org
    • Entry type: mark, but could be measure
    • Entry name: appLaunch@clock.gaiamobile.org, the @ convention overriding the mark’s context
    • Start time: 80081.169720,
    • Duration: 0.000000, this is a mark, not a measure
    • Epoch: 1434771804212

    For every performance mark and measure, Gecko also captures the epoch of the mark, and we can use this to compare times from across processes.

    Pros and Cons

    Everything is a game of tradeoffs, and performance testing with Raptor is no exception:

    • We trade high-resolution times for millisecond resolution in order to compare numbers across processes.
    • We trade JavaScript APIs for log parsing so we can access data without injecting custom logic into every application, which would affect app performance.
    • We currently trade a high-level interaction API, Marionette, for low-level interactions using Orangutan behind the scenes. While this provides us with transparent events for the platform, it also makes writing rich tests difficult. There are plans to improve this in the future by adding Marionette integration.

    Why log parsing

    You may be a person that believes log parsing is evil, and to a certain extent I would agree with you. While I do wish for every solution to be solvable using a performance API, unfortunately this doesn’t exist yet. This is yet another reason why projects like Firefox OS are important for pushing the Web forward: we find use cases which are not yet fully implemented for the Web, poke holes to discover what’s missing, and ultimately improve APIs for everyone by pushing to fill these gaps with standards. Log parsing is Raptor’s stop-gap until the Web catches up.

    Raptor workflow

    Raptor is a Node.js module built into the Gaia project that enables the project to do performance tests against a device or emulator. Once you have the project dependencies installed, running performance tests from the Gaia directory is straightforward:

    1. Install the Raptor profile on the device; this configures various settings to assist with performance testing. Note: this is a different profile that will reset Gaia, so keep that in mind if you have particular settings stored.
      make raptor
    2. Choose a test to run. Currently, tests are stored in tests/raptor in the Gaia tree, so some manual discovery is needed. There are plans to improve the command-line API soon.
    3. Run the test. For example, you can performance test the cold launch of the Clock app using the following command, specifying the number of runs to launch it:
      APP=clock RUNS=5 node tests/raptor/launch_test
    4. Observe the console output. At the end of the test, you will be given a table of test results with some statistics about the performance runs completed. Example:
    [Cold Launch: Clock Results] Results for clock.gaiamobile.org
    
    Metric                            Mean     Median   Min      Max      StdDev  p95
    --------------------------------  -------  -------  -------  -------  ------  -------
    coldlaunch.navigationLoaded       214.100  212.000  176.000  269.000  19.693  247.000
    coldlaunch.navigationInteractive  245.433  242.000  216.000  310.000  19.944  274.000
    coldlaunch.visuallyLoaded         798.433  810.500  674.000  967.000  71.869  922.000
    coldlaunch.contentInteractive     798.733  810.500  675.000  967.000  71.730  922.000
    coldlaunch.fullyLoaded            802.133  813.500  682.000  969.000  72.036  928.000
    coldlaunch.rss                    10.850   10.800   10.600   11.300   0.180   11.200
    coldlaunch.uss                    0.000    0.000    0.000    0.000    0.000   n/a
    coldlaunch.pss                    6.190    6.200    5.900    6.400    0.114   6.300
    

    Visualizing Performance

    Access to raw performance data is helpful for a quick look at how long something takes, or to determine if a change you made causes a number to increase, but it’s not very helpful for monitoring changes over time. Raptor has two methods for visualizing performance data over time, in order to improve performance.

    Official metrics

    At raptor.mozilla.org, we have dashboards for persisting the values of performance metrics over time. In our automation infrastructure, we execute performance tests against devices for every new build generated by mozilla-central or b2g-inbound (Note: The source of builds could change in the future.) Right now this is limited to Flame devices running at 319MB of memory, but there are plans to expand to different memory configurations and additional device types in the very near future. When automation receives a new build, we run our battery of performance tests against the devices, capturing numbers such as application launch time and memory at fullyLoaded, reboot duration, and power current. These numbers are stored and visualized many times per day, varying based on the commits for the day.

    Looking at these graphs, you can drill down into specific apps, focus or expand your time query, and do advanced query manipulation to gain insight into performance. Watching trends over time, you can even pick out regressions that have sneaked into Firefox OS.

    Local visualization

    The very same visualization tool and backend used by raptor.mozilla.org is also available as a Docker image. After running the local Raptor tests, data will report to your own visualization dashboard based on those local metrics. There are some additional prerequisites for local visualization, so be sure to read the Raptor docs on MDN to get started.

    Performance regressions

    Building pretty graphs that display metrics is all well and fine, but finding trends in data or signal within noise can be difficult. Graphs help us understand data and make it accessible for others to easily communicate around the topic, but using graphs for finding regressions in performance is reactive; we should be proactive about keeping things fast.

    Regression hunting on CI

    Rob Wood has been doing incredible work in our pre-commit continuous integration efforts surrounding the detection of performance regressions in prospective commits. With every pull request to the Gaia repository, our automation runs the Raptor performance tests against the target branch with and without the patch applied. After a certain number of iterations for statistical accuracy, we have the ability to reject patches from landing in Gaia if a regression is too severe. For scalability purposes we use emulators to run these tests, so there are inherent drawbacks such as greater variability in the metrics reported. This variability limits the precision with which we can detect regressions.

    Regression hunting in automation

    Luckily we have the post-commit automation in place to run performance tests against real devices, and is where the dashboards receive their data from. Based on the excellent Python tool from Will Lachance, we query our historical data daily, attempting to discover any smaller regressions that could have crept into Firefox OS in the previous seven days. Any performance anomalies found are promptly reported to Bugzilla and relevant bug component watchers are notified.

    Recap and next steps

    Raptor, combined with User Timing, has given us the know-how to ask questions about the performance of Gaia and receive accurate answers. In the future, we plan on improving the API of the tool and adding higher-level interactions. Raptor should also be able to work more seamlessly with third-party applications, something that is not easily done right now.

    Raptor has been an exciting tool to build, while at the same time helping us drive the Web forward in the realm of performance. We plan on using it to keep Firefox OS fast, and to stay proactive about protecting Gaia performance.