1. The CSS 3 Flexible Box Model

    This article about the Flexible Box Layout was written by Jérémie Patonnier, French open Web enthusiast.

    The flexible box model

    CSS 3 introduces a brand new box model in addition of the traditional box model from CSS 1 and 2. The flexible box model determines the way boxes are distributed inside other boxes and the way they share the available space.

    You can see the specification here.

    This box model is similar to the one used by XUL (the user interface language used by Firefox). Some others languages use similar box models such as XAML or GladeXML.

    Usually the flexible box model is exactly what you need if you want to create fluid layouts which adapt themselves to the size of the browser window or elastic layouts which adapt themselves to the font size.

    In this article, all my examples are based on the following HTML code:

    <body>
      <div id="box1">1</div>
      <div id="box2">2</div>
      <div id="box3">3</div>
    </body>

    Distributing boxes: so what?

    By default, the traditional CSS box model distributes boxes vertically depending on the HTML flow. With the flexible box model, it’s possible to specify the order explicitly. You can even reverse it. To switch to the flexible box model, set the property display to the value box (or inline-box) on a box which has child boxes.

    display: box;

    Horizontal or vertical distribution

    The box-orient property lets you specify the distribution axis. vertical and horizontal values define how boxes are displayed. Other values (inline-axis and block-axis) have the same effect, but also let you define the baseline alignment itself (basically the boxes are treated like “inline” boxes).

    body{
      display: box;
      box-orient: horizontal;
    }

    Reversed distribution

    The property box-direction allows you to set the order in which the boxes appear. By default–when you simply specify the distribution axis–the boxes follow the HTML flow and are distributed from top to bottom if you are using a vertical axis and from left to right if you are using a horizontal axis. By setting box-direction to reverse, you can reverse the boxes’ distribution order. It acts as if you actually reversed the order of the elements in the HTML.

    Be careful with this property because it changes the way some other properties work, which can produce some unexpected behavior.

    body {
      display: box;
      box-orient: vertical;
      box-direction: reverse;
    }

    Explicit distribution

    The property box-ordinal-group lets you specify the order in which the boxes will be distributed. This is the ultimate customization opportunity, because you can define the order you want, regardless of the HTML flow order. Those groups are defined by a number starting at 1 (which is the default value). So the box model will first distribute those groups, then all the boxes inside each group. The distribution occurs from the lowest value (the group numbered 1) to the highest (the groups numbered 2 and above).

    body {
      display: box;
      box-orient: vertical;
      box-direction : reverse;
    }
    #box1 {
      box-ordinal-group: 2;
    }
    #box2 {
      box-ordinal-group: 2;
    }
    #box3 {
      box-ordinal-group: 1;
    }

    And what about flexibility?

    If changing the natural HTML flow order is huge, the real fun begins when you start to deal with the available space.

    Box sizing

    By default, a box is not flexible. It becomes flexible only if it has the property box-flex with a value of at least 1.

    If a box is not flexible, it will be as wide as possible to make its content visible without any overflow. Its size can be forced with the properties width and height (or their min-*, and max-* alternatives).

    If a box is flexible, its size will be computed as follows:

    1. The explicit size declarations (width, height, min-* and max-*)
    2. The size of the parent box and all the remaining available inner space.

    So, if the boxes haven’t any size declarations, their sizes will fully depend on their parent box’s size. It will work like this: the size of box is equal to the size of its parent multiplied by the value of the its box-flex property divided by the sum of all the box-flex properties values of all boxes included in its parent.

    On the other hand, if one or more boxes have an explicit size statements, the size of all those boxes is computed and all the flexible boxes share the remaining available space on the same principle as above.

    It probably sounds a bit tricky, but with some examples it will become easier.

    All boxes are flexible

    In the next example, box 1 is twice the size of box 2 and box 2 has the same size as box 3. It looks the same as using percentages to set the boxes’ sizes. But there is a big difference. If you add a box, you don’t need to recalculate its size. With the flexible box model, each time you add a box, all the others automatically shrink to make room for the new one.

    body {
      display: box;
      box-orient: horizontal;
    }
    #box1 {
      box-flex: 2;
    }
    #box2 {
      box-flex: 1;
    }
    #box3 {
      box-flex: 1;
    }

    Some boxes have a fixed size

    In the next example, box 3, which is not flexible, is 160px in width. In this case, there’s 240px of free space available for boxes 1 and 2. So, box 1 will be 160px in width (240px x 2/3) and box 2 will be 80px in width (240px x 1/3). If you wish, you can make box 3 flexible as well. In this case the way the size of this box is computed will be almost the same as with the property min-width.

    body {
      display: box;
      box-orient: horizontal;
      width: 400px;
    }
    #box1 {
      box-flex: 2;
    }
    #box2 {
      box-flex: 1;
    }
    #box3 {
      width: 160px;
    }

    Managing overflow

    Because we can mix flexible boxes, inflexible boxes, and flexible boxes which have preset sizes, It’s possible for the sum of all the boxes’ sizes to be larger or smaller than the parent box size. So you can have too much space or not enough.

    I have too much space available; what do I do?

    The available space gets distributed depending on the properties box-align and box-pack

    The property box-pack manages the way the space is distributed on the horizontal axis and can have one of four possible values: start, end, justify, or center

    1. start: All the boxes are on the left side of the parent box and all the remaining space is on the right side.
    2. end: All the boxes are on the right and the remaining space is on the left
    3. justify: The available space is divided evenly in-between each boxes
    4. center: The available space is divided evenly on each side of the parent box

    The property box-align manages the way the space is distributed on the vertical axis and can have one of five values: start, end, center, baseline, and stretch

    1. start: The top edge of each box is placed along the top of the parent box and all the remaining space is placed below.
    2. end: The bottom edge of each box is placed along the bottom of the parent box and all the remaining space is placed above.
    3. center: The available space is divided evenly and placed half above and half below.
    4. baseline: All children are placed with their baselines aligned and the remaining space is placed before or after as necessary (This is a simplification about how this value really works, but you see the point).
    5. stretch: The height of each boxes is adjusted to fit the parent box height

    A warning about how those properties work: they are strongly influenced by the use of the properties box-orient and box-direction. They can cause some unexpected behavior (for example, the behavior of values start and end could be fully reversed). I hope that once the specification is finalized, we’ll have more information about how those properties work together.

    body {
      display: box;
      box-orient: horizontal;
      /* The content of the body is horizontally centered */
      box-pack: center;
      /* and vertically as well ... \o/ */
      box-align: center;
      width: 100%;
      height : 100%;
    }

    What happens if I don’t have enough space?

    Just like with the traditional box model, the overflow property lets you to define the way it’s managed. No surprise here.

    However, you must be careful here too. Indeed, the use of the properties box-orient and box-direction can mess it up. For example, you can see elements overflowed to the right instead of the left or to the top instead of the bottom. Take the time to experiment before trying to use it on a big project or you could go mad.

    You can also avoid overflow by making the boxes run over multiple lines (or columns, depending on the orientation) by setting the property box-lines to multiple.

    Okay, cool, but does it work in real life?

    Yes it does! Both Gecko and WebKit have vendor-prefixed implementations of a box model (Note: The current state of the specification does not reflect Mozilla’s or WebKit’s implementation). This means that Firefox, Safari, Chrome, and any browsers that use one of those rendering engines are able to use the features described in this article. If you use one of those awesome browsers, here is a little demo of the flexible box model in action.

    If you’re not using a browser implementing a box model, this screenshot shows you what it looks like:

    To conclude

    You can start to use this box model to layout your HTML documents with modern web browsers. Be careful though, it’s the really first iteration of a W3C Working Draft. There will certainly be some changes. Anyway, the implementations available in Gecko and Webkit are extremely consistent and mature, so if there are changes, they should not be that troublesome.

    This box model is a very easy and simple way to solve some usual problems in web design (form layout, page footers, vertical centering, disassociation of visual flow from HTML flow, etc.). I strongly suggest you become familiar with it because it could become a standard tool for web designers in the near future (if Microsoft decides to include it in IE, it could become so very fast).

    What is already available is a good start to play with. But at this point, the way the traditional box model and the flexible box model interact is not very clear (for example, it’s impossible to use position:relative with the properties left or top on a box which uses the property box-ordinal-group). This will be improved, but don’t be surprised if your work habits are somewhat undermined. Another tricky point: the way all the properties relative to this new box model interact can be sometimes really confusing. This should remind you of the day you discovered the float property. ;)

    For further information

  2. Account Manager coming to Firefox

    Update: The Account Manager is no longer maintained. Building on this experiment, we have conceived BrowserID. Please consider using it instead.

    Last month Mozilla Labs announced a new concept series on online identity. As part of this exploration, we developed the Account Manager.

    The Account Manager makes it incredibly easy for users to create new accounts with optional randomly generated passwords, and log into and out of them with just a click. As a web developer, adding support for this feature could take as little as fifteen minutes of hacking (in fact, we’ll mention the first 5 people to add support – read below to learn more.).

    We want to make signing into websites easier for all Firefox users, and are looking to ship this feature as soon as possible in Firefox. As part of that process we’re looking for feedback to refine the specification. Now is a really good time to get involved in defining the spec.

    There are three things that you can do right now:

    This feature is currently available as an experimental add-on, available on the Account Manager homepage.

    Here’s a video where you can get a basic idea of how Account Manager works today:

    How Does It Work?

    The Account Manager specification proposes two small changes to Web sites:

    1. The browser needs to know how to register, sign in, and sign out of your site. You will need a static JSON document, automatically discovered by the browser, which describes what methods the site supports and how they should be executed. For example, a web site might describe their support of “connect” (sign in) like this:

        "methods": {
          "username-password-form": {
            "connect": {
              "method": "POST",
              "path": "/accounts/LoginAuth",
              "params": {
                "username": "Email",
                "password": "Passwd"
            }
        }

      This example tells the browser that the site supports signing in with a form POST to /accounts/LoginAuth, and what parameter names to use for the username and password (Email and Passwd respectively).

    2. The browser needs a way to check which user (if any) is currently signed in. To do this, you need to set an HTTP header in the same code where you would set a cookie with a session ID. If you can’t set an HTTP header, you can also supply a URL the browser will ping.

      The header would look like this:

      X-Account-Management-Status: active; name="Joe User"

      That would tell the browser that “Joe User” is now signed in, so it can provide the appropriate UI (to switch users or sign out).

      How do I try it?

      • Install the demo add-on.
      • Place a host-meta document to your website at /.well-known/host-meta (it must be at this location). This tells the browser where to find the JSON file we described above. For examples, check the spec or Yahoo!’s host-meta.
      • Add the JSON file itself to your site. We call this the Account Manager Control Document, or AMCD for short. The AMCD should contain your form end-points for sign-in and sign-out. Note that you don’t need to change the end-points, just describe them. Check the spec for a complete example.
      • Change your site to set the correct headers when users sign in or out.
      • Make sure you have a password saved in the password manager; you may need to sign in manually once to do that if you haven’t already (this requirement will go away in the future).
      • When we add sign-up support in Account Manager, you will likely need to make minor changes to your registration code.

      Update, 5:45PM PST: Just realized while debugging an intrepid first adopter’s site that there is one more requirement:

      • You can send the status header with every request, or if you don’t want to do that, then you need to provide a sessionstatus method (see the spec) that the browser can ping to find out the user’s signed-in status.

      That’s it, folks! Be one of the first to try implementing the specification on your website, and let us know, and let us know how long it took you to add support for it. We’ll put the first five people to implement this on the @mozhacks twitter account with a link to your site!

      Next time we will go into more depth on how discovery works, our plans to support other auth schemes (like HTTP Auth, OpenID, etc), as well as other neat features we plan to add. Stay tuned! And don’t forget to tell us what you think.


      Web Developer FAQ

      • Do I need to redo all of my authentication code?

        No. Account Manager is designed to require minimal server-side changes. You do have a couple of options, but the minimal setup is just a flat file and a couple of extra headers you need to send out.

      • Do I need to redo all of my account creation code?

        Registration will require some small changes to your registration flow, but we have put extra thought into making it as simple as possible for both Web sites and users alike. Check out our discussion group and specification for the details, and let us know what you think!

      • How is this going to help my users?

        Account Manager is great for users. Here are a few highlights:

        • Simple, convenient, user control

          The browser has a couple of advantages when it comes to making this kind of UI. First, it can dedicate a spot in the browser chrome that will look and behave the same for every site, making it a convenient and automatic go-to place for users to check or change their sign-in status.

          The browser also has deep knowledge about the user. For example, the browser could implement fast user switching with just a click. Or think about picking a username: the browser can look at usernames for other accounts and make some pretty good guesses about what usernames are preferred.

        • Secure

          Many security researchers will tell you: one of the biggest security problems on the Web today is that usernames and passwords are often short and easily guessed. Account Manager makes it so that users don’t need to remember their passwords, and in fact can automatically generate strong passwords when signing up.

          Moreover, Account Manager begins the process of abstracting the plumbing of account management from the UI, making it possible in the future to support cryptographically strong protocols without any major UI changes.

        • Works on top of current and emerging solutions

          Lastly, Account Manager is not a new ID for the Web. Rather, it is designed to work on top of current and emerging solutions like OpenID or others, to bring them all under the same user experience. Users shouldn’t have to care what the underlying technology is.

      • How is this going to help me get more users?

        The easier it is to sign up and sign into your site, the more users you will get. It’s pretty much that simple.

        Note that Account Manager doesn’t force your users to make a choice: you can keep all of your current content-based flows intact, so there is really no downside to adding Account Manager support to your site.

      • Do I need to have special content for Firefox only?

        No! First of all, you don’t need to do *any* changes to your current content at all. Account Manager works behind the scenes using a sitemap and headers to communicate with your site and present the right UI to the user.

        On the other hand, we hope that Account Manager will not be a Firefox-only technology. We’re working towards defining the protocol as a formal specification that other Web browsers can implement.

  3. WebSockets in Firefox

    Here’s the pitch for WebSockets: a low-complexity, low-latency, bi-directional communication system that has a pretty simple API for web developers. Let’s break that down, and then talk about if and when we’re going to include it in Firefox:

    Low-complexity

    The WebSocket protocol, which is started via an HTTP-like handshake, has a relatively simple model for sending text packets back and forth. The complexity of the protocol is quite low. There’s no multiplexing, no support for binary data, and once the data connection is set up, the actual transport is quite cheap.

    Note that there are people who feel – possibly correctly – that support for multiplexing and binary data should be added. That would obviously increase the complexity but in some cases might be worth the cost. But more on that later.

    Bi-directional Communication

    One of the key aspects of WebSocket is its support for simple bi-directional communication. Servers can easily send updates to clients and clients can send updates to servers. Many of the comet-style applications that people have built will be much simpler and faster with this model because the protocol and API support it directly.

    While allowing for bi-directional communication it also is bound by the HTTP same-origin model. That is, it’s doesn’t give the browser the ability to connect to any site on any port it wants to.

    So WebSocket is really TCP with the HTTP security model.

    Low-latency

    This is actually the primary value in WebSockets. The key thing is that the cost of sending a small amount of data is also very small. It’s possible to do bi-directional communication today with comet-style applications, but small amounts of data often require a huge amount of overhead to do so.

    A second-hand story to give you a sense of scale: Google Wave, which tries to do real-time communication with keystrokes has a several-kilobyte overhead for just about every keystroke because of the TCP startup, teardown and HTTP message headers involved with what should be only a few bytes sent over the wire.

    I haven’t tried, but I’m going to guess that if you built Quake on top of HTTP comet that the interactive experience would be poor. So this is where WebSockets really shine.

    Simple API

    The actual API that a developer sees for WebSocket is relatively simple. You can send messages, you can get messages, you get events when sockets open, close or there’s an error. Additional complexity, which I’m sure others will develop, will happen in JavaScript and backend libraries.

    (While this might seem like punting on the issue of complexity, this is actually the model for success we’ve seen for other browser standards: build something relatively simple that others can experiment with. When we understand what’s slow or what’s preventing progress, iterate and improve.)

    When will it be in Firefox?

    We really really want to support WebSockets in the next version of Firefox. And a lot of other people want us to include support too.

    The first set of test patches, written by the wonderful Wellington Fernando de Macedo (one of our most excellent contributors!) was first submitted in April of 2009. Since then we’ve been iterating both on the patches themselves and following the spec as it’s changed.

    Unfortunately, the spec itself is still under revision. WebSockets did ship in Chrome with version 4 and I’m told by Chrome developers that it’s going to be included in Chrome 5, without changes. Unfortunately, the version that Google included in Chrome doesn’t reflect the current draft. The handshake for how to start a WebSocket connection has changed, largely to improve security on servers. There’s also a lot of ongoing discussion on the role of integrated compression, direct support for binary data (instead of encoding binary data as strings), whether or not the protocol should support multiplexing and a number of other issues.

    Since WebSocket isn’t used widely yet, and that Chrome has an uptake rate that’s similar to Firefox, the hope is that once a more recent draft makes it through the process that both Chrome and Firefox can include a more recent version at around the same time.

    So in short: we want to ship it because the promise of WebSockets is great, but we’ll have to see if it’s stable and safe enough to do so. Code isn’t the issue – we’ve had that for a while. It’s whether or not there’s consensus on if the protocol is “done enough” to ship it to a few hundred million people.

  4. Beyond HTML5: experiments with interactive audio

    This is a re-post of an important post from David Humphrey who has been doing a lot of experiments on top of Mozilla’s extensible platform and doing experiments with multi-touch, sound, video, WebGL and all sorts of other goodies. It’s worth going through all of the demos below. You’ll find some stuff that will amaze and inspire you.

    David’s work is important because it’s showing where the web is going, and where Mozilla is helping to take it. It’s not enough that we’re working on HTML5, which we’re about finished with, but we’re trying to figure out what’s next. Mozilla’s platform, Gecko, is a huge part of why we’re able to experiment and learn as fast as we can. And that’s reflected with what’s possible here. It’s a web you can see, touch and interact with in new ways.

    David’s post follows:

    I’m working with an ever growing group of web, audio, and Mozilla developers on a project to expose audio spectrum data to JavaScript from Firefox’s audio and video elements. Today we show what we did at www2010.

    I’m in Raleigh, North Carolina, with Al MacDonald for the www2010 conference. We’re here to present our work on exposing audio data in the browser. Over the past month Corban, Charles, and a bunch of other friends have been working with us to refine the API and get new types of demos ready. We ended-up with 11 demos, some of which I’ve shown here before. Here are the others.

    The first was done by Jacob Seidelin, and shows many cool 2D visualizations of audio using our API. You can see the live version on his site, or check out this video:

    The second and third demos where done by Charles Cliffe, and show 3D visualizations using WebGL and his CubicVR engine. These also show off his JavaScript beat detection code. Is JavaScript fast enough to do real-time analysis of audio and synchronized 3D graphics? Yes, yes it is. The live versions are here and here, and here are some videos:

    The fourth demo was done by Corban Brook and shows how audio data can be mixed live using script. Here he mutes the main audio, plays it, passes the data through a low pass filter written in JavaScript, then dumps the modified frames into a second audio element to be played. It’s a technique we need to apply more widely, as it holds a lot of potential. Here’s the live version, and here’s a video (check out his updated JavaScript synthesizer, which we also presented):

    The fifth and sixth demos were done by Al (with the help of many). When I was last in Boston, for the Processing.js meetup at Bocoup, we met with Doug Schepers from the W3C. He loved our stuff, and was talking to us about ideas that would be cool to build. He pulled out his iPhone and showed us Brian Eno’s Bloom audio app. “It would be cool to do this in the browser.” Yeah, it is cool, and here it is, written in a few hundred lines of JavaScript and Processing.js (video 1, video 2):

    This demo also showcases the awesome work of Felipe Gomes, who has a patch to add multi-touch DOM events to Firefox. The method we’ve used here can be taken a lot further. Imagine being able to connect multiple browsers together for collaborative music creation, layering other audio underneath, mixing fragments vs. just oscillators, etc. We built this one in a week, and the web is capable of a lot more.

    One of the main points of our talk was to emphasize that what we’re talking about here isn’t just a concept, and it isn’t some far away future. This is real code, running in a real browser, and it’s all being done in HTML5 and JavaScript. The web is fast enough to do real-time audio processing now, powerful enough and expressive enough to create music. And the community of digital music and audio hackers, visualizers, etc. are hungry for it. So hungry that they are seeking us out, downloading our hacked builds and creating gorgeous web audio applications.

    We want to keep going, and we need help. We need help from those within Mozilla, the W3C, and other browsers to get this stuff into shipping browsers. We need the audio, digital music, accessibility, and web communities to come together in order to help us build js audio libraries and more sample applications. Yesterday Joe Hewitt was talking on twitter about how web browser vendors need to experiment more with non-standard APIs. I couldn’t agree more, and here’s a chance for people to put their money where their mouth is. Let’s make audio a scriptable part of the open web.

    I’m currently creating new builds of our updated patch for Firefox, and will post links to them here when I’m done. You can read more about the technical details of our work here, and get involved in the bug here. You can talk more with me on irc in the processing.js channel (I’m humph on moznet), or talk to me on twitter (@humphd) or by email. One way or another, get in touch so you can help us push this forward.

  5. a series of updates on what’s coming in Firefox 4

    Over the next couple of weeks, people like Paul Rouget and I are going to post a series of updates on a bunch of technologies that are part of our next release. That release, likely called Firefox 4, was underway before the release of Firefox 3.6 and already includes a bunch of new features, bug fixes and performance enhancements. (As an aside, I personally suspect that in a lot of ways Firefox 4 is likely to be a revolutionary release for both users and web developers, but more on that in later posts.)

    Since we’re in the middle of the development cycle it’s important for web developers to know what’s coming and how they can test them. We’ll talk candidly about not only what’s coming, but why they are important. Very often in the current race to improve browsers context is lost on why something is or isn’t included, or why something is higher priority than something else. We’ll try and shine some context on a lot of these things as well as give good technical info about our roadmap to the next Firefox release.

    So stay tuned, it’s going to be a fun ride.

  6. mozilla developer preview 4 ready for testing

    Note: this is a re-post of the entry in the Mozilla Project Development Weblog. There’s some juicy stuff in here for Web Developers that need testing. In particular, this is the first build with the CSS history changes.

    As part of our ongoing platform development work, we’re happy to announce the fourth pre-release of the Gecko 1.9.3 platform. Gecko 1.9.3 will form the core of Firefox and other Mozilla project releases.

    It’s available for download on Mac, Windows or Linux.

    Mozilla expects to release a Developer Preview every 2-3 weeks. If you’ve been running a previous release, you will be automatically updated to the latest version when it is released.

    This preview release contains a lot of interesting stuff that’s worth pointing out, and contains many things that were also in previous releases. Here are the things of note in this release:

    User Interface Changes

    • Open tabs that match searches in the Awesomebar now show up as “Switch to Tab.”
    • This is the first preview release to contain resizable text areas by default.

    Web Developer Changes

    • This is the first preview release to contain changes to CSS :visited that prevent a large class of history sniffing attacks. You can find more information about the details of why this change is important over on the hacks post on the topic and on the Mozilla Security Weblog. Note that this change is likely to break some web sites and requires early testing – please test if you can.
    • SVG Attributes which are mapped to CSS properties can now be animated with SMIL. See the bug or a demo.

    Plugins

    • Out of process plugins support for Windows and Linux continues to improve. This release contains many bug fixes vs. our previous developer preview releases. (In fact, it’s good enough that we’ve ported this code back to the 3.6 branch and have pushed that to beta for a later 3.6.x release.)
    • This is the first release that contains support for out of process plugins for the Mac. If you are running OSX 10.6 and you’re running the latest Flash beta, Flash should run out of process

    Performance

    • One area where people complained about performance was restart performance when applying an update. It turns out that a lot of what made that experience poor wasn’t startup time, it was browser shutdown time. We’ve made a fix since the last preview release that made a whopping 97% improvement in shutdown time. (That’s not a typo, it’s basically free now.)
    • Our work to reduce the amount of I/O on the main thread continues unabated. This preview release will feel much snappier than previous snapshots, and feel much faster than Firefox 3.6.
    • We continue to add hardware acceleration support. If you’re on Windows and you’ve got decent OpenGL 2 drivers, open video will use hardware to scale the video when you’re in full screen mode. For large HD videos this can make a huge difference in the smoothness of the experience and how much power + CPU are used. We’ll be adding OSX and Linux support at some point in the future as well, but we’re starting with Windows.
    • We continue to make improvements and bug fixes to our support for Direct2D. (Not enabled by default. If you want to turn it on see Bas’ post.) If you’re running Alpha 4 on Windows Vista or Windows 7, and you’ve turned on D2D, try running this stress test example in Alpha 4 vs. Firefox 3.6. The difference is pretty amazing. You can also see what this looks like compared to other browsers in this this video. (Thanks to Hans Schmucker for the video and demo.)

    Platform

    • JS-ctypes, our new easy-to-use system for extension authors who want to call into native code now has support for complex types: structures, pointers, and arrays. For more information on this, and how easy it can make calling into native code from JavaScript, see Dan Witte’s post.
    • Mozilla is now sporting an infallible allocator. What is this odd-sounding thing, you ask? It’s basically an allocator that when memory can’t be allocated it aborts instead of returning NULL. This reduces the surface area for an entire class of security bugs related to checking NULL pointers, and also allows us to vastly simplify a huge amount of Gecko’s source code.
  7. Real-time server visualization with canvas and processing.js

    This is a guest blog post by Logan Welliver, Chief Creative at Cloudkick. He is a graphic designer by training and a web designer in practice.

    Cloud management company Cloudkick has released a real-time server monitoring visualization based on canvas and processing.js, that was co-developed with Alastair McDonald of processing.js fame. The product is designed to let users keep a finger on the pulse of their infrastructure, quickly identify problem nodes, and visualize aggregate performance with an easy-to-digest interface.

    The tool uses canvas and processing.js to plot servers as stylized circles in 3 dimensions, with axes mapped to one of three performance metrics: CPU usage, memory usage, and ping latency. Each server’s radius is determined by it’s relative prowess (i.e. an EC2 extra large is bigger than 256mb Slice), and colors are customizable via the Cloudkick dashboard. Each server sparkles when the monitoring system returns data, and servers with problems identify themselves by flashing an angry red.

    Canvas and processing.js take care of all the presentation, powered by a slew of back-end services that do everything from monitoring servers to pushing data in real-time back to the user.

    Here’s a brief overview of the back-end architecture: instances of the Cloudkick Agent (running on individual servers) report metrics to an endpoint, which talks to Reconnoiter, which then publishes messages to RabbitMQ. An internal service called Livedata consumes these messages, finds the ones applicable to an account, and publishes messages back to RabbitMQ. Orbited consumes these messages and sends them to the browser. From agent to browser, the round-trip time is less than a second.

    Cloudkick has partnered with Mozilla to provide the visualization for their addons.mozilla.org servers. You can see how they’re behaving in a live demo of the visualization here: addons.mozilla.org infrastructure in Cloudkick Viz.

    Get the visualization for your own servers. Cloudkick is offering 20% off for the first 100 Mozilla Hacks readers, using promo code “mozhacks01″.

  8. Theora on N900

    This is a re-post from Matthew Gregan’s personal weblog on the work that he’s been doing to bring HTML5 open video to mobile devices. Google recently announced funding for some work to bring Theora to ARM devices via a CPU-driven code path. Mozilla has been funding similar work over the last year or so to do video decoding on DSPs found in mobile devices, leaving the CPU largely idle.

    We realize this post isn’t strictly web developer-facing, but it’s interesting enough for those who want to know how this stuff works under the covers.

    Theora on N900: Or, how to play full-screen Theora video on the N900 with 80% idle CPU.

    The C64x+ DSP is often found in systems built upon TI’s OMAP3 SoC, such as the Palm Pre, Motorola Droid, and Nokia N900. Last year, Mozilla funded a port, named Leonora, of Xiph’s Theora video codec to the TI C64x+ DSP. David Schleef conducted the port impressively quickly and published his results. The intention of this project was to provide a high quality set of royalty free media codecs for a common mobile computing platform. The initial focus is Firefox Mobile on the N900, so I am working on integrating David’s work into Firefox. To experiment with other facilities Firefox could use to accelerate video playback and test integration, I’ve been hacking on a branch of a stand-alone Ogg Theora and Vorbis player originally written by Chris Double called plogg.

    Decoding and playing video can be a CPU intensive process, especially when all of the steps are fighting for time on a single CPU. The expensive parts of the playback process can be broken down into a few coarse pieces, in approximate descending order of cost:

    1. Video frame decode
    2. Video colour-space conversion (Y’CbCr to RGB)
    3. Video frame display
    4. Audio block decode
    5. Audio block playback

    David’s DSP work enables item 1 to be off-loaded from the CPU completely, effectively providing “hardware accelerated” video decoding. Most devices have some way to off-load items 2 and 3 to the graphics hardware, but it can be difficult to make use of this while integrating with an existing graphics rendering pipeline.

    The N900 has a 800×480 pixel display, so my hope was to play a 800×480 video full-screen at 30 frames per second with low CPU use and good battery life.

    The ARM CPU in the N900 is quite fast. Doing a pure video-decode-only test, the original Theora library, which currently does not have ARM specific optimizations, is able to decode a 640×360 video at 76 frames per second, and it can even decode an 800×480 video at 32 frames per second. With the ARM optimized port by Robin Watts, those numbers become 110 FPS and 47 FPS. David’s DSP port produces 78 FPS and 39 FPS, and it leaves the CPU completely idle because the entire decode is off-loaded to the DSP. With these numbers, it’s clear that the N900 is up to the task of playing back video smoothly if we can get the bits on the screen fast enough.

    I am using plogg as a basis for experimentation using techniques applicable in the Firefox rendering engine. This requirement immediately excludes some techniques. For example, using hardware Y’CbCr overlays to display the video frames is excluded because it is not possible for Firefox to render arbitrary HTML content over the top of the overlay.

    Chris’s original version of plogg used SDL’s Y’CbCr overlay API, which uses a fast direct overlay path on most systems. This provided a baseline for playback performance. Decoding my 800×480 test video with the DSP, it was possible to play back at 33 FPS with around 20% CPU idle. Unlike the decode-only benchmarks mentioned above, the plogg benchmarks are playing both audio and video with correct A/V synchronisation. With 44.1kHz stereo audio, I observed that 10-15% of the device’s CPU is used by PulseAudio. This indicates that audio playback may constitute a significant amount of processor time with some configurations.

    Because there was already work underway to provide OpenGL accelerated compositing in Firefox with the newly conceived Layers API, it seemed logical to try using a GLSL fragment shader to off-load colour-space conversion to the GPU. This turned out to be too slow to play back a full-screen video.

    Looking at the list of vendor-specific OpenGL extensions available on the N900, I discovered the texture streaming API. This allows a program to directly map texture memory and copy Y’CbCr data into that memory without having to perform an expensive texture upload or colour-space conversion. The colour-space conversion is off-loaded to dedicated graphics hardware inaccessible via the standard OpenGL APIs. Using this and the modified bc-cat kernel module from the gst-bc-cat project, it’s possible to play back at 26 FPS with 81% CPU idle.

    One drawback of the current bc-cat kernel driver is that there is a very limited set of texture formats supported (NV12, UYVY, RGB565, and YUYV), and none of them are the same as what Theora produces. To work around this, a format conversion is required to convert planar Y’CbCr to packed 4:2:2 UYVY. Fortunately, this conversion is much simpler than a full colour-space conversion. Timothy Terriberry sent me a couple of patches to off-load this conversion work to the DSP. If it’s possible to extend the bc-cat driver to support texture formats compatible with Theora’s output, performance can be further improved.

    The test files used for benchmarking were: Big Buck Bunny (video: 640×360 @ 500 kbps 24 FPS, audio: 64 kbps 48kHz stereo, 9m 56s, 40MB) and the movie trailer for 9 (video: 800×480 @ 2 Mbps 23.98 FPS, audio: 44.1kHz stereo, 2m 30s, 30MB). Benchmarks were run with the CPU frequency fixed at 600MHz.

    In summary, it’s possible to play full-screen Ogg Theora videos on the N900 at full frame rates with low CPU use by off-loading video decoding to the DSP and colour-space conversion and painting to the GPU. There are opportunities for optimization left, tuning for battery life needs to be investigated, and the integration into Firefox still needs to be done.

    Video decode-only:

    Decoder FPS (800×480) Idle CPU
    libtheora 1.1 32 0.7%
    TheorARM 0.04 47 0.4%
    leonora (DSP) 39 99.0%

    Playback (video decode + paint, audio decode + not played). DSP decoding video:

    Method FPS (800×480) Idle CPU
    SDL/overlay 33 20.0%
    OpenGL/bc-cat 26 81.4%