Mozilla

Articles by Robin Hawkes

Sort by:

View:

  1. Lessons learnt building ViziCities

    Just over 2 weeks ago Peter Smart and Robin Hawkes released the first version of ViziCities to the world. It’s free to use and open-sourced under an MIT license.

    In this post I will talk to you about the lessons learnt during the development of ViziCities. From application architecture to fine-detailed WebGL rendering improvements, we learnt a lot in the past year and we hope that by sharing our experiences we can help others avoid the same mistakes.

    What is ViziCities?

    In a rather geeky nutshell, ViziCities is a WebGL application that allows you to visualise anywhere in the world in 3D. It’s primary purpose is to look at urban areas, though it’ll work perfectly fine in other places if they have decent OpenStreetMap coverage.

    Demo

    The best way to explain what ViziCities does is to try it out yourself. You’ll need a WebGL-enabled browser and an awareness that you’re using pre-alpha quality software. Click and drag your way around, zoom in using the mouse wheel, and rotate the camera by clicking the mouse wheel or holding down shift while clicking and dragging.

    You can always take a look at this short video if you’re unable to try the demo right now:

    What’s the point of it?

    We started the project for multiple reasons. One of those reasons is that it’s an exciting technical and design challenge for us – both Peter and I thrive on pushing ourselves to the limits by exploring something unknown to us.

    Another reason is that we were inspired by the latest SimCity game and how they visualise data about the performance of your city – in fact, Maxis, the developers behind SimCity reached out to us to tell us how much they like the project!

    There’s something exciting about creating a way to do that for real-world cities rather than fictional ones. The idea of visualising a city in 3D with up-to-date data about that city overlaid is an appealing one. Imagine if you could see census data about your area, education data, health data, crime data, property information, live transport (trains, buses, traffic, etc), you’d be able to learn and understand so much more about the place you live.

    This is really just the beginning – the possibilities are endless.

    Why 3D?

    A common question we get is “Why 3D?” – the short answer, beyond “because it’s a visually interesting way of looking at a city”, is that 3D allows you to do things and analyse data in ways that you can’t do in a 2D map. For example by using 3D you can take height and depth into consideration, so you can better visualise the sheer volume of stuff that lies above and below you in a city – things like pipes and underground tunnels, or bridges, overpasses, tall buildings, the weather, and planes! On a 2D map, looking at all of this would be a confusing mess, in 3D you get to see it exactly how it would look in the real world – you can easily see how objects within a city relate to each other.

    Core technology

    At the most basic level ViziCities is built using Three.js, a WebGL library that abstracts all the complexity of 3D rendering in the browser. We use a whole range of other technologies too, like Web Workers, each of which serves a specific purpose or solves a specific problem that we’ve encountered along the way.

    Let’s take a look at some of those now.

    Lessons learnt

    Over the past year we’ve come from knowing practically nothing about 3D rendering and geographic data visualisation, to knowing at least enough about each of them to be dangerous. Along the way we’ve hit many roadblocks and have had to spend a significant amount of time working out what’s wrong and coming up with solutions to get around them.

    The process of problem solving is one I absolutely thrive on, but it’s not for everybody and I hope that the lessons I’m about to share will help you avoid these same problems, allowing you to save time and do more important things with your life.

    These lessons are in no particular order.

    Using a modular, decoupled application architecture pays off in the long run

    We originally started out with hacky, prototypal experiments that were dependency heavy and couldn’t easily be pulled apart and used in other experiments. Although it allowed us to learn how everything worked, it was a mess and caused a world of pain when it came to building out a proper application.

    In the end we re-wrote everything based on a simple approach using the Constructor Pattern and the prototype property. Using this allowed us to separate out logic into decoupled modules, making everything a bit more understandable whilst also allowing us to extend and replace functionality without breaking anything else (we use the Underscore _.extend method to extend objects).

    Here’s an example of our use of the Constructor Pattern.

    To communicate amongst modules we use the Mediator Pattern. This allows us to keep things as decoupled as possible as we can publish events without having to know about who is subscribing to them.

    Here’s an example of our use of the Mediator Pattern:

    /* globals window, _, VIZI */
    (function() {
      "use strict";
    
      // Apply to other objects using _.extend(newObj, VIZI.Mediator);
      VIZI.Mediator = (function() {
        // Storage for topics that can be broadcast or listened to
        var topics = {};
    
        // Subscribe to a topic, supply a callback to be executed
        // when that topic is broadcast to
        var subscribe = function( topic, fn ){
          if ( !topics[topic] ){
            topics[topic] = [];
          }
    
          topics[topic].push( { context: this, callback: fn } );
    
          return this;
        };
    
        // Publish/broadcast an event to the rest of the application
        var publish = function( topic ){
          var args;
    
          if ( !topics[topic] ){
            return false;
          }
    
          args = Array.prototype.slice.call( arguments, 1 );
          for ( var i = 0, l = topics[topic].length; i < l; i++ ) {
    
            var subscription = topics[topic][i];
            subscription.callback.apply( subscription.context, args );
          }
          return this;
        };
    
        return {
          publish: publish,
          subscribe: subscribe
        };
      }());
    }());
    

    I’d argue that these 2 patterns are the most useful aspects of the new ViziCities application architecture – they have allowed us to iterate quickly without fear of breaking everything.

    Using promises instead of wrestling with callbacks

    Early on in the project I was talking to my friend Hannah Wolfe (Ghost’s CTO) about how annoying callbacks are, particularly when you want to load a bunch of stuff in order. It didn’t take Hannah long to point out how stupid I was being (thanks Hannah) and that I should be using promises instead of wrestling with callbacks. At the time I brushed them off as another hipster fad but in the end she was right (as always) and from that point onwards I used promises wherever possible to take back control of application flow.

    For ViziCities we ended up using the Q library, though there are plenty others to choose from (Hannah uses when.js for Ghost).

    The general usage is the same whatever library you choose – you set up promises and you deal with them at a later date. However, the beauty comes when you want to queue up a bunch of tasks and either handle them in order, or do something when they’re all complete. We use this in a variety of places, most noticeably when loading ViziCities for the first time (also allowing us to output a progress bar).

    I won’t lie, promises take a little while to get your head around but once you do you’ll never look back. I promise (sorry, couldn’t resist).

    Using a consistent build process with basic tests

    I’ve never been one to care too much about process, code quality, testing, or even making sure things are Done Right™. I’m a tinkerer and I much prefer learning and seeing results than spending what feels like wasted time on building a solid process. It turns out my tinkering approach doesn’t work too well for a large Web application which requires consistency and robustness. Who knew?

    The first step for code consistency and quality was to enable strict mode and linting. This meant that the most glaring of errors and inconsistencies were flagged up early on. As an aside, due to our use of the Constructor Pattern we wrapped each module in an anonymous function so we could enable strict mode per module without necessarily enabling it globally.

    At this point it was still a faff to use a manual process for creating new builds (generating a single JavaScript file with all the modules and external dependencies) and to serve the examples. The break-through was adopting a proper build system using Grunt, thanks mostly to a chat I had with Jack Franklin about my woes at an event last year (he subsequently gave me his cold which took 8 weeks to get rid of, but it was worth it).

    Grunt allows us to run a simple command in the terminal to do things like automatically test, concatenate and minify files ready for release. We also use it to serve the local build and auto-refresh examples if they’re open in a browser. You can look at our Grunt setup to see how we set everything up.

    For automated testing we use Mocha, Chai, Sinon.js, Sinon-Chai and PhantomJS. Each of which serves a slightly different purpose in the testing process:

    • Mocha is used for the overall testing framework
    • Chai is used as an assertion library to allows you to write readable tests
    • Sinon.js is used to fake application logic and track behaviour through the testing process
    • PhantomJS is used to run client-side tests in a headless browser from the terminal

    We’ve already put together some (admittedly basic) tests and we plan to improve and increase the test coverage before releasing 0.1.0.

    Travis CI is used to make sure we don’t break anything when pushing changes to GitHub. It automatically performs linting and runs our tests via Grunt when changes are pushed, including pull requests from other contributors (a life saver). Plus it lets you have a cool badge to put on your GitHub readme that shows everyone whether the current version is building successfully.

    Together, these solutions have made ViziCities much more reliable than it has ever been. They also mean that we can move rapidly by building automatically, and they allow us to not have to worry so much about accidentally breaking something. The peace of mind is priceless.

    Monitoring performance to measure improvements

    General performance in frames-per-second can be monitored using FPSMeter. It’s useful for debugging parts of the application that are locking up the browser or preventing the rendering loop from running at a fast pace.

    You can also use the Three.js renderer.info property to monitor what you’re rendering and how it changes over time.

    It’s worth keeping an eye on this to make sure objects are not being rendered when they move out of the current viewport. Early on in ViziCities we had a lot of issues with this not happening and the only way to be sure we to monitor these values.

    Turning geographic coordinates into 2D pixel coordinates using D3.js

    One of the very first problems we encountered was how to turn geographic coordinates (latitude and longitude) into pixel-based coordinates. The math involved to achieve this isn’t simple and it gets even more complicated if you want to consider different geographic projections (trust me, it gets confusing fast).

    Fortunately, the D3.js library has already solved these problems for you, specifically within its geo module. Assuming you’ve included D3.js, you can convert coordinates like so:

    var geoCoords = [-0.01924, 51.50358]; // Central point as [lon, lat]
    var tileSize = 256; // Pixel size of a single map tile
    var zoom = 15; // Zoom level
    
    var projection = d3.geo.mercator()
      .center(geoCoords) // Geographic coordinates of map centre
      .translate([0, 0]) // Pixel coordinates of .center()
      .scale(tileSize << zoom); // Scaling value
    
    // Pixel location of Heathrow Airport to relation to central point (geoCoords)
    var pixelValue = projection([-0.465567112, 51.4718071791]); // Returns [x, y]
    

    The scale() value is the hardest part of the process to understand. It basically changes the pixel value that’s returned based on how zoomed in you want to be (imagine zooming in on Google Maps). It took me a very long time to understand so I detailed how scale works in the ViziCities source code for others to learn from (and so I can remember!). Once you nail the scaling then you will be in full control of the geographic-to-pixel conversion process.

    Extruding 2D building outlines into 3D objects on-the-fly

    While 2D building outlines are easy to find, turning them into 3D objects turned out to be not quite as easy as we imagined. There’s currently no public dataset containing 3D buildings, which is a shame though it makes it more fun to do it yourself.

    What we ended up using was the THREE.ExtrudeGeometry object, passing in a reference to an array of pixel points (as a THREE.Shape object) representing a 2D building footprint.

    The following is a basic example that would extrude a 2D outline into a 3D object:

    var shape = new THREE.Shape();
    shape.moveTo(0, 0);
    shape.lineTo(10, 0);
    shape.lineTo(10, 10);
    shape.lineTo(0, 10);
    shape.lineTo(0, 0); // Remember to close the shape
    
    var height = 10;
    var extrudeSettings = { amount: height, bevelEnabled: false };
    
    var geom = new THREE.ExtrudeGeometry( shape, extrudeSettings );
    var mesh = new THREE.Mesh(geom);
    

    What turned out interesting was how it actually turned out quicker to generate 3D objects on-the-fly than to pre-render them and load them in. This was mostly due to the fact it would take longer to download a pre-rendered 3D object than downloading the 2D coordinates string and generating it at runtime.

    Using Web Workers to dramatically increase performance and prevent browser lockup

    One thing we did notice with the generation of 3D objects was that it locked up the browser, particularly when processing a large number of shapes at the same time (you know, like an entire city). To work around this we delved into the magical world of Web Workers.

    Web Workers allow you to run parts of your application in a completely separate processor thread to the browser renderer, meaning anything that happens in the Web Worker thread won’t slow down the browser renderer (ie. it won’t lock up). It’s incredibly powerful but it can also be incredibly complicated to get working as you want it to.

    We ended up using the Catiline.js Web Worker library to abstract some of the complexity and allow us to focus on using Web Workers to our advantage, rather than fighting against them. The result is a Web Worker processing script that’s passed 2D coordinate arrays and returns generated 3D objects.

    After getting this working we noticed that while the majority of browser lock-ups were eliminated, there were two new lock-ups introduced. Specifically, there was a lock-up when the 2D coordinates were passed into the Web Worker scripts, and another lock-up when the 3D objects were returned back to the main application.

    The solution to this problem came from the inspiring mind of Vladimir Agafonkin (of LeafletJS fame). He helped me understand that to avoid the latter of the lock-ups (passing the 3D objects back to the application) I needed to use transferrable objects), namely ArrayBuffer objects. Doing this allows you to effectively transfer ownership of objects created within a Web Worker thread to the main application thread, rather than copying them. We implemented this to great effect, eliminating the second lock-up entirely.

    To eliminate the first lock-up (passing 2D coordinates into the Web Worker) we need to take a different approach. The problem lies again with the copying of the data, though in this case you can’t use transferrable objects. The solution instead lies in loading the data into the Web Worker script using the importScripts method. Unfortunately, I’ve not yet worked out a way to do this with dynamic data sourced from XHR requests. Still, this is definitely a solution that would work.

    Using simplify.js to reduce the complexity of 2D shapes before rendering

    Something we found early on was that complex 2D shapes caused a lot of strain when rendered as 3D objects en-masse. To get around this we use Vladimir Agafonkin’s simplify.js library to reduce the quality of 2D shapes before rendering.

    It’s a great little tool that allows you to keep the general shape while dramatically reducing the number of points used, thus reducing its complexity and render cost. By using this method we could render many more objects with little to no change in how the objects look.

    Getting accurate heights for buildings is really difficult

    One problem we never imagined encountering was getting accurate height information for buildings within a city. While the data does exist, it’s usually unfathomably expensive or requires you to be in education to get discounted access.

    The approach we went for uses accurate height data from OpenStreetMap (if available), falling back to a best-guess that uses the building type combined with 2D footprint area. In most cases this will give a far more accurate guess at the height than simply going for a random height (which is how we originally did it).

    Restricting camera movement to control performance

    The original dream with ViziCities was to visualise an entire city in one go, flying around, looking down on the city from the clouds like some kind of God. We fast learnt that this came at a price, a performance price, and a data-size price. Neither of which we were able to afford.

    When we realised this wasn’t going to be possible we looked at how to approach things from a different angle. How can you feel like you’re looking at an entire city without rendering an entire city? The solution was deceptively simple.

    By restricting camera movement to only view a small area at a time (limiting zoom and rotation) you’re able to have much more control over how many objects can possibly be in view at one time. For example, if you prevent someone from being able to tilt a camera to look at the horizon then you’ll never need to render every single object between the camera and the edge of your scene.

    This simple approach means that you can go absolutely anywhere in the world within ViziCities, whether a thriving metropolis or a country-side retreat, and not have to change the way you handle performance. Every situation is predictable and therefore optimisable.

    Tile-based batching of objects to improve loading and rendering performance

    Another approach we took to improve performance was by splitting the entire world into a grid system, exactly like how Google and other map providers do things. This allows you to load data in small chunks that eventually build up to a complete image.

    In the case of ViziCities, we use the tiles to only request JSON data for the geographic area visible to you. This means that you can start outputting 3D objects as each tile loads rather than waiting for everything to load.

    A by-product of this approach is that you get to benefit from frustum culling, which is when objects not within your view are not rendered, dramatically increasing performance.

    Caching of loaded data to save on time and resources when viewing the same location

    Coupled with the tile-based loading is a caching system that means that you don’t request the same data twice, instead pulling the data from a local store. This saves bandwidth but also saves time as it can take a while to download each JSON tile.

    We currently use a dumb local approach that resets the cache on each refresh, but we plan to implement something like localForage to have the cache persist between browser sessions.

    Using the OpenStreetMap Overpass API rather than rolling your own PostGIS database

    Late into the development of ViziCities we realised that it was unfeasible to continue using our own PostGIS database to store and manipulate geographic data. For one, it would require a huge server just to store the entirety of OpenStreetMap in a database, but really it was just a pain to set up and manage and an external approach was required.

    The solution came in the shape of the Overpass API, an external JSON and XML endpoint to OpenStreetMap data. Overpass allows you to send a request for specific OpenStreetMap tags within a bounding box (in our case, a map tile):

    http://overpass-api.de/api/interpreter?data=[out:json];((way(51.50874,-0.02197,51.51558,-0.01099)[%22building%22]);(._;node(w);););out;
    

    And get back a lovely JSON response:

    {
      "version": 0.6,
      "generator": "Overpass API",
      "osm3s": {
        "timestamp_osm_base": "2014-03-02T22:08:02Z",
        "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
      },
      "elements": [
    
    {
      "type": "node",
      "id": 262890340,
      "lat": 51.5118466,
      "lon": -0.0205134
    },
    {
      "type": "node",
      "id": 278157418,
      "lat": 51.5143963,
      "lon": -0.0144833
    },
    ...
    {
      "type": "way",
      "id": 50258319,
      "nodes": [
        638736123,
        638736125,
        638736127,
        638736129,
        638736123
      ],
      "tags": {
        "building": "yes",
        "leisure": "sports_centre",
        "name": "The Workhouse"
      }
    },
    {
      "type": "way",
      "id": 50258326,
      "nodes": [
        638736168,
        638736170,
        638736171,
        638736172,
        638736168
      ],
      "tags": {
        "building": "yes",
        "name": "Poplar Playcentre"
      }
    },
    ...
      ]
    }
    

    The by-product of this was that you get worldwide support out of the box and benefit from minutely OpenSteetMap updates. Seriously, if you edit or add something to OpenStreetMap (please do) it will show up in ViziCities within minutes.

    Limiting the number of concurrent XHR requests

    Something we learnt very recently was that spamming the Overpass API endpoint with a tonne of XHR requests at the same time wasn’t particularly good for us nor for Overpass. It generally caused delays as Overpass rate-limited us so data took a long time to make its way back to the browser. The great thing was that by already using promises to manage the XHR requests we were half-way ready to solve the problem.

    The final piece of the puzzle is to use throat.js to limit the number of concurrent XHR requests so we can take control and load resources without abusing external APIs. It’s beautifully simple and worked perfectly. No more loading delays!

    Using ViziCities in your own project

    I hope that these lessons and tips have helped in some way, and I hope that it encourages you to try out ViziCities for yourself. Getting set up is easy and well documented, just head to the ViziCities GitHub repo and you’ll find everything you need.

    Contributing to ViziCities

    Part of the reason why we opened up ViziCities was to encourage other people to help build it and make it even more awesome than Peter and I could ever make it. Since launch, we’ve had over 1,000 people favourite the project on GitHub, as well as nearly 100 forks. More importantly, we’ve had 9 Pull Requests from members of the community who we didn’t previously know and who we’ve not asked to help. It’s such an amazing feeling to see people help out like that.

    If we were to pick a favourite contribution so far, it would be adding the ability to load anywhere in the world by putting coordinates in the URL. Such a cool feature and one that has made the project much more usable for everyone.

    We’d love to have more people contribute, whether dealing with issues or playing with the visual styling. Read more about how to contribute and give it a go!

    What’s next?

    It’s been a crazy year and an even crazier fortnight since we launched the project. We never imagined it would excite people in the way it has, it’s blown us away.

    The next steps are slowly going through the issues and getting ready for the 0.1.0 release, which will still be alpha quality but will be sort of stable. Aside from that we’ll continue experimenting with exciting new technologies like the Oculus Rift (yes, that’s me with one strapped to my face)…

    Visualising realtime air traffic in 3D…

    And much, much more. Watch this space.

  2. Creating the future of mobile with Firefox OS – resources, docs and more!

    Just under a month ago I wrote a personal post about my thoughts on Firefox OS and why I think there is something ‘magical’ about what it stands for and the possibilities it brings to the table. This post is a follow-up that aims to cover much of the same ground but with extra detail and more of a technical focus.

    What is Firefox OS?

    In short, Firefox OS is about taking the technologies behind the Web, like JavaScript, and using them to produce an entire mobile operating system. It’s effectively a mobile OS powered by JavaScript!

    Firefox OS screenshots

    This is achieved with a custom version of Gecko, the rendering engine in Firefox, that introduces a variety of new JavaScript APIs needed to create a phone-like experience; things like WebSMS to send text messages, and WebTelephony to make phone calls.

    You might be wondering what’s running Gecko, as a phone can’t naturally boot directly into Gecko. To do that, the phone boots into a very lightweight Linux kernel that, in turn, boots the Gecko process. The process is a little more involved than that and much more detail can be found in the B2G Architecture documentation, including how Gecko accesses the radio hardware and other phone-specific functionality.

    The Firefox OS project also aims to combine many of the other projects at Mozilla into a single vision, what we refer to as the Web as the platform. These projects include the Open Web Apps initiative and Persona, our solution to identity and logins on the Web (formally known as BrowserID). It’s the combination of these various technologies that completes Firefox OS.

    If you want to find out more technical information about the OS then definitely check out the Firefox OS pages on MDN.

    Why Firefox OS?

    A couple of common questions that come up are, “Why Firefox OS?” and more specifically, “Why build a mobile OS using JavaScript?” These are incredibly important questions so let’s take a moment to delve into them in a little detail.

    Why build a mobile OS using JavaScript?

    Answering this question can quite simply be boiled down to one sentence; because it’s possible. It’s not the one and only answer but it succinctly handles most of the arguments against JavaScript being used in this way.

    A longer answer is that a JavaScript-powered OS unlocks a whole range of possibilities that aren’t normally or easily available to developers and users with existing operating systems.

    The most obvious of the possibilities is the ability to build applications using the technologies that we already use to build websites; namely JavaScript, CSS, and HTML. While not a truly unique feature of Firefox OS — projects like PhoneGap have done this for years on ‘native’ platforms — it allows developers everywhere to create mobile applications without having to learn native languages and APIs.

    Another draw of JavaScript is that it’s both extremely well documented and free to develop with. Anyone could sit down for a weekend and put together an application without having to pay for a single thing. Obviously that’s not true in the majority of cases, as people tend to buy their own hosting or tooling, but theoretically there is nothing to stop you building with these technologies for free.

    What’s arguably most interesting about JavaScript being used in this way is that it inherently enables physical devices to communicate using the same APIs that we already use on websites. In effect, instead of accessing the Web through a mobile browser the entire phone is now capable of accessing and communicating with any Web API. For example, there is nothing to stop you building an application for Firefox OS that uses WebRTC (once added) to create Skype-like P2P video communication between phones, desktop computers, or anything else that supports WebRTC.

    This really only scrapes the surface of “Why JavaScript?” but it certainly gives you a feel of how this is both interesting and important, beyond the tired debate of ‘native’ vs. Web. If you’re still not convinced, just think for a moment about how you can now customise an entire mobile OS using nothing by JavaScript. You’d be hard pushed to deny that it’s pretty darn interesting!

    OK, but why Firefox OS?

    Effectively, Firefox OS has been built to put our money where our mouth is (so to speak) and prove that JavaScript is capable of doing what we say it can do. However, there is much more to the project than just proving the the technology is fast enough.

    The first reason ‘Why Firefox OS’ is that the mobile ecosystem is overrun with proprietary platforms, most of which prevent you from easily moving between various platforms. What Firefox OS aims to achieve is a truly ‘open’ platform that doesn’t lock you in and inherently makes it as easy and possible to move between devices as and when you choose.

    Mozilla is effectively replicating its success with Firefox, in which it stormed the browser market and showed users that there is an alternative, one that lets them be in control of how they use the Web. In this case, it’s less about browsers and more about mobile platforms and application portability.

    Another reason is that Firefox OS is an attempt to push the Web forward into the world of physical devices. One direct benefit of this is the addition of brand new Web standards and APIs that allow for things like hardware access using JavaScript.

    Plenty of challenges

    It’s fair to say that the Firefox OS journey will contain a number of technical challenges along the way, however that’s part of the fun and the reasons why we’re working on it.

    One of those challenges is how to manage an apps ecosystem that is open and distributed. This is something that we are tackling with the Open Web Apps initiative and the Mozilla Marketplace. It’s a challenge that we are dealing with as things progress and as we learn more about how things work best, as is the nature with new ways of thinking.

    Another of the challenges is making sure that the phone runs as fast as possible, creating the best experience possible. This also relates to questions raised within the developer community around the performance capabilities of JavaScript, particularly when it is used to do things that are perceived to be complex, or when it is compared against ‘native’ technologies. This is a challenge that we are taking very seriously and one which we feel we can overcome. In fact, it’s a challenge that I believe we have already overcome.

    One prime example of how capable JavaScript has become is seeing beautiful JavaScript games running in Firefox OS at near-enough 60 frames per-second, on a low-end, cheap phone.

    Beyond the mobile phone

    While the phone aspect of Firefox OS is immediately interesting, you should consider the wider implications of a JavaScript OS and what possibilities it unlocks. For example, what other devices could benefit from being powered by JavaScript? And, what would a network of JavaScript-powered devices allow us to do — things like Ubiquitous Computing, perhaps?

    These aren’t things that we are exploring directly at Mozilla, but they are things that are now inherently possible as a result of the work that we’re doing. There is nothing to stop you taking the Firefox OS source code from GitHub and porting it to a device that we’ve never even considered.

    We’re already starting to see this happen with examples like a Firefox OS port for the Raspberry Pi, as well as another for the Pandaboard.

    What about a game console powered by Firefox OS? A TV, or set-top box? What about a fridge? Individually, these are all interesting projects, but together they offer something we don’t really have at the moment, a network of different devices powered by the same, open technologies and able to access and communicate across the Web with the same APIs.

    We are a long way away from that kind of world but it is projects like Firefox OS that may pave the way for it to happen. You could even be a part of it!

    Getting started with Firefox OS

    The hope is that by now you’re sufficiently interested in Firefox OS to begin exploring, experimenting and playing with it. The good news is that there are a whole host of ways that you can do that.

    Documentation

    One of the first places to start is the MDN documentation surrounding Firefox OS and its related technologies. This is where you’ll find everything you need to know about the developer-facing aspects of the platform.

    If you’re more interested with the inner-workings of the platform then you’ll want to cast an eye over the B2G wiki, which outlines much of the internals in plenty of detail.

    Source code

    If you’re keen to get to grips with the source code of Firefox OS then you’ll want to head over to GitHub and check it out. The two main repositories that you want are ‘b2g’ (the underlying Gecko engine) and ‘gaia’ (everything you can see, the OS).

    Getting involved

    There are a few ways to get involved with the project. You could check out some of the issues and get involved in fixing them, or perhaps just hang out in the mailing list for B2G, or the one for Gaia, and take part in the discussions there.

    If you just want to ask a few immediate questions then try out the #b2g and #gaia rooms on irc.mozilla.org. We’re all pretty friendly!

    Development options

    If you just want to dig in and make some applications, or perhaps customise the OS, then you’ll need to know about the various development options available to you. They are covered in some detail on MDN but here is a brief overview.

    The simplest method to get started is running Gaia (the visual side of Firefox OS) within Firefox Nightly. This doesn’t give you a true representation of a phone environment but it will allow you to install applications and use all of the developer tools within the browser that you’re already used to.

    Slightly more involved than Nightly is using the desktop B2G client. This is effectively a chromeless build of Firefox that looks phone-like has some added APIs that aren’t normally available in standard Firefox. This doesn’t replicate phone hardware but it’s the next best thing before starting to develop on an actual device.

    Setting up the desktop B2G client isn’t too hard, but it could be made easier. In the meantime, projects like r2d2b2g aim to make the process super simple. Definitely worth checking out.

    The last method, and arguably the most important one, is developing on an actual Firefox OS device. This is the only method that will give you a true representation of how your application will perform. It is also the only method that will give you access to the all the new APIs that come with Firefox OS.

    Right now, you’ll need to build and install Firefox OS on one of the supported devices. In the future you will be able to skip this step and get access to devices that already run Firefox OS. We don’t have any dates for that just yet.

    Go forth and be part of something big

    My hope is that by now you should now have enough inspiration and information to go forth and begin building for this new platform, powered by the technologies you already use. We hope you do and we’d love to see what you come up with.

    It’s not every day that you get the opportunity to be a part of something that could quite literally change the way we do things.

  3. Mozilla supports the Liberated Pixel Cup for open games

    We’re excited to announce that Mozilla is supporting the Liberated Pixel Cup, a fantastic competition aiming to spark the creation of artwork and code for games that are free and available for others to use.

    The Liberated Pixel Cup is the brainchild of OpenGameArt, a long-standing community of artists that provide graphics and sound effects for others to use in their games. Joining us in supporting the competition are the Creative Commons, Free Software Foundation, as well as the hundreds of everyday supporters who donated to the prize fund.

    Getting involved with the Liberated Pixel Cup was an obvious choice for us. Over the past year we’ve vastly improved our performance for games in Firefox, we’ve added new features for games, we’ve hosted a work week to discuss the future of games, we’ve attended and represented the open Web at important game-related events like GDC, as well as much, much more. To put it simply, we think games are pretty darn important to the future of the Web!

    As part of our involvement, we’ll be:

    • Supporting the HTML5 category of the competition
    • Using our various spaces around the world to host Liberated Pixel Cup events
    • Donating funds to support the running of the competition

    The art side of the competition starts in June, with the code side starting in July; so mark those dates in your diary and get ready to create some game-related goodness!

    Check out the Liberated Pixel Cup website for more information.

  4. Come and make games with us at GameHack

    This coming weekend (28–29th April) we’ll be at GameHack, hosted at Pinewood Studios in the UK.

    GameHack

    The weekend-long event is focussed on creating amazing games on a variety of platforms. We’re working with the highly-experienced guys at Turbulenz to encourage the creation of HTML5 games.

    And it’s not just about having fun making games (thought that’s mostly the point). If your game is good enough then you could win yourself one of our Boot to Gecko phones!

    More information can be found on the GameHack website. Tickets are still available and cost £5, with all proceeds going to charity.

    I’ll be the dude in the black Firefox hoodie, and I’ll give you a demo of Boot to Gecko if you ask nicely. I look forward to seeing you there.

  5. Rounding-up the first Mozilla HTML5 games work week

    A few weeks ago I wrote about the first Mozilla HTML5 games work week in Toronto and why it is important. In this post I summarise the week’s events and highlight some key observations.

    DSCF0062

    Before I start I should point out that I won’t be making any dramatic announcements from Mozilla or going into too much detail about some of the decisions that were made during the work week. While some important discussions and decisions did occur, some aren’t quite ready to be made public yet. However, we will be posting specifically about some of these things in the coming weeks so keep an eye open. The purpose of this post is to give an overview of the week’s events and some of the discussions that were made.

    So with that out of the way, let’s crack on.

    Why a HTML5 games work week?

    With the amount of buzz happening around HTML5 and JavaScript, it could be seen as surprising that we hadn’t organised a get-together of all the interested parties within Mozilla sooner. The long and short of it is that it has only been recently that we have begun to construct a consistent vision and commitment to HTML5 games.

    To create a vision we need to find out who within the Mozilla community is actually interested in games and how we can work together to lay the path towards the future. The HTML5 games work week was set up to start the ball rolling. It’s safe to say that Mozilla cares a lot about games and this work week is just the beginning.

    Aside from the internal interest and passion to build games, there are an increasing number of HTML5 games being released in the Web community each and every week. This shows that there is desire for these types of games and for browsers to work out how to support them.

    DSCF0068

    It’s also clear that there are areas of improvement that need to be addressed within all browsers so that HTML5 games can flourish in the future. The point of the work week has been to bring all interested parties together in one place to make stuff happen and actually make decisions about how to address these issues.

    From another perspective, the work week gave Mozilla the chance to listen to external developers and hear about the issues they are experiencing in the real world of HTML5 game development. We don’t develop games every day, so it is of absolute importance for us to fully understand the concerns of the people who do. Without this type of feedback we simply won’t be able to make our improvements as useful as they could be.

    Who was there?

    The work week was split into 2 parts: the first was attended by Mozilla staff and key Mozilla contributors, and the second included some invited external guests.

    DSCF0083

    It was clear from the beginning that we had a whole bunch of people within Mozilla who were passionate about games and wanted to see us commit to them. I already knew a handful of these people, but I had absolutely no idea that so many Mozillians from so many different areas of the organisation were so interested. It great to see all those people in one (big) room working together to make awesome stuff happen.

    The invited guests for the second part of the work week included developers from game studios like EA, as well as individual developers who have proven themselves experts in specific technologies, like WebGL. We worked with these guests to hear about issues from the guys dealing with this stuff every single day. It was absolutely fascinating to hear their concerns and work with them to find ways of improving the situation for games on the Web.

    What was discussed?

    Throughout the week we discussed a whole variety of topics ranging from how to better engage with the game developer community all the way to narrowing down fine-grained performance improvements to be made in JavaScript.

    DSCF0082

    Overall, both the internal Mozilla staff and invited guests covered the following areas and many more (in no particular order):

    • Looking into the issue of mobile and how its requirements differ from those on the desktop
    • Working out what tools are needed for authoring HTML5 games
    • Graphics, specifically Canvas and WebGL
    • Delving into the issues surrounding DRM and the protection of assets
    • NaCl and Emscripten as methods of porting existing games to the Web
    • Being clear about who exactly wants HTML5 games
    • Looking at what dev tools are required to ease the debugging of games
    • Learning the business model behind games from those with years of experience in this area
    • Working out how to better engage with the game development community
    • Producing a roadmap for the Firefox platform specifically related to games
    • Looking at ways JavaScript performance could be increased
    • Discussing the situation with asset management and how it can be solved
    • Uncovering any new WebAPIs that might become necessary
    • Delving deep into common audio issues and looking at its future
    • Hearing how high-precision timers would be used by game developers
    • Learning about WebRTC and how it will provide the functionality for peer-to-peer communication
    • Looking at how games tie into the Mozilla Marketplace and the questions that Mozilla needs to answer in this area

    All in all we covered a huge amount of stuff in the space of just a few days. I won’t list each and every decision that was made but rest assured that you’ll be hearing about them in the near-future as they get finalised and released.

    In the meantime, please check out the Are We Fun Yet site as it’s where you’ll be able to keep tabs on the status of game-related technology within Mozilla. There are also some links to the detailed feedback and notes taken during this work week.

    External demos and code review

    During the final 2 days of the work week we saw demos from our external guests and worked with them to review specific issues that they have been experiencing in Firefox.

    DSCF0080

    These demos proved invaluable for those within Mozilla to get a better idea of how professional game developers are using the technologies on the Web. We were inspired and amazed by the quality of these demos and the kind of games that could be created by pushing JavaScript right to the very limits.

    As a result of these demos and code reviews, issues have been pinned down, test-cases written, and in some cases even fixes produced there and then. It was really a very useful aspect of the work week that we clearly need to explore further in the future.

    What’s next?

    So, what does that future look like for Mozilla and game development? Frankly, very bright indeed. There is so much interest and desire within Mozilla to further the Web as a platform for games. It’s clear that we have the knowledge and resources, we just need to work out how best to apply them.

    While we work on refining our position on games and exactly how we’re going to tackle them, there are a few things that I can say for sure about the future.

    The first is that we’re dedicated to helping developers learn how to use Web technologies, that’s the reason why MDN and this blog exist. We’ve started to work on ways to help developers learn how to use Web technologies to create games. We’re not quite there yet but it’s a priority.

    Stack of gamepads

    Aside from MDN, we are going to be reaching out to developers at game-related events. At some events we’ll be even bringing along hardware for you to play with and hack with JavaScript APIs, like gamepads for the Gamepad API. We’ll also be present at some of the larger events like GDC so we can keep up to speed with the game world and talk to developers about the issues they’re facing.

    You’ll also see games as a focus on this year’s Mozilla Festival. We’re refining the program but we’re working on tying in elements like the successful Game On competition that we ran last year.

    Documenting the game-related decisions and discussions within Mozilla is key, and it is something that we’re working on improving. For now the best place to keep up to date is this blog, the #games channel on irc.mozilla.org, and the HTML5 games page on the Mozilla wiki.

    With recent announcements it’s clear that Boot to Gecko (B2G) and games will be an area that Mozilla and the developer community will explore in the near future. It will be interesting to see what people do with a mobile platform as open and hackable as B2G.

    I could go on but it’s safe to say that there is a whole bunch of stuff happening around games at Mozilla, it’s an exciting time indeed. No doubt we’ll have another work week in the coming months to take stock and take more steps in the right direction.

  6. Gaming and the Mozilla Labs Apps Project

    In this post I give a quick overview of the Mozilla Labs Apps project and how it and the other technologies at Mozilla relate to gaming. We really are at a point where amazing games can be created on the Web with nothing but open technologies.

    A few days ago we launched the developer preview of the Mozilla Labs Apps project, our vision of a distributed Web application platform. This apps project is directly related to our overall mission to better the Web, as recently described by Ragavan Srinivasan:

    As part of Mozilla’s mission to make the Web better we believe that apps should be available on any modern Web-connected device, operating system or browser, allowing billions of people worldwide to use and interact with apps.

    Today, apps are tied to closed ecosystems. Apps also can’t be used across all devices; data within apps is silo-ed and inaccessible to other apps or Web services; and user choice is limited and undervalued.

    In an open ecosystem, developers should have the ability to build apps using open Web technologies and resources. They should then be able to distribute and sell those apps with the reach, flexibility and choice that the Web provides.

    It’s important to remember that although we often talk about apps as pieces of software that each serve a particular task, they can also be games. After all, if you pull away the story and graphics, a game is really no different to what you would normally consider as an application — it uses the same platform and programming languages, HTML and JavaScript in this case.

    So how does the Mozilla Labs Apps project and the other technologies that we’re working on relate to open Web games? I hope to shed some light on this question in the brief overviews that follow.

    Rest assured, Mozilla is very much focused on making the Web a better platform for open Web gaming.

    Identifying players with BrowserID

    Just like how iOS has services like OpenFeint and the Apple Game Center, games on the Web need open and reliable methods of identifying players. BrowserID is one of our solutions to this problem. It allows players to log into your game using their existing email address, providing a password only once per browser session.

    Implementing BrowserID within your game is a simple 3-step process. I advise you to check out the documentation about installation, where you’ll also find some fancy BrowserID buttons to show your players that you support distributed authentication. Right now BrowserID is provided via a JavaScript library. However in the near-future it will be built into Firefox for a seamless experience.

    Identifying players in this way is just the first step in providing all sorts of functionality with your game, like friends lists, leader boards, chat, and multiplayer.

    Creating immersive gameplay with the Full Screen API

    Something that prevents current games on the Web from feeling immersive is that they look like they’re just tiny boxes embedded into another website. Why do they feel like that? Well, because they are just tiny boxes embedded into other websites. The odd 5-minute puzzle game during your lunch hour might feel OK in a tiny box surrounded by browser UI and other distractions, but a first-person shooter or driving game certainly wouldn’t.

    Fortunately, the Full Screen API has arrived to solve this problem. It allows you to make any DOM element fill the player’s entire screen, something normally only considered for videos. Using this in a game can make the difference between 5 minutes of relative fun and hours of immersive delight.

    Like most of the JavaScript APIs, it’s pretty easy to implement full-screen functionality in your game. All you need to do is call the requestFullScreen method on the desired DOM element:

    var elem = document.getElementById("myGame"); if (elem.requestFullScreen) { elem.requestFullScreen(); } else if (elem.mozRequestFullScreen) { elem.mozRequestFullScreen(); } else if (elem.webkitRequestFullScreen) { elem.webkitRequestFullScreen(); } 

    Taming mouse input with the Mouse Lock API

    An issue related to input in games is that of misbehaving cursors, where the mouse is used to rotate a player around a fixed point, like moving the viewpoint in a 3D first-person shooter, or rotating the player in a top-down 2D game.

    In both of these situations, the mouse cursor is visible at all times, which is generally annoying and ruins the experience. However, the most debilitating problem is that when the mouse cursor leaves the browser window all movement stops. This same behaviour occurs in full-screen mode when the mouse cursor hits the edge of the screen. It’s a horribly simple problem for players that completely ruins the experience.

    The good news is that the Mouse Lock API has been created to solve this problem, and it just landed in experimental builds of Firefox Nightly. Its sole purpose is to tame the mouse by hiding the cursor and locking it in place so it doesn’t hit the edges of the screen. This means that instead of relying on x and y coordinate values for mouse position in related to the top-left corner of the browser, you instead rely on x and y distance values from the position that the mouse was locked to.

    Using the Mouse Lock API is really just a case of calling the navigator.pointer.lock method. However this must be done in full screen mode to work properly (and remember to use the experimental build of Firefox linked previously).

    The following example expands a <div> element to full screen and locks the mouse in place at the same time:

    <script> function fullAndLock() { var div = document.getElementById("div"); document.addEventListener("mozfullscreenchange", function() { var pointer = navigator.pointer; if (document.mozFullScreen && document.mozFullScreenElement === div && !pointer.islocked()) { function onMouseLockSuccess() { console.log("Mouse successfully locked"); } function onMouseLockError() { console.log("Error locking mouse"); } pointer.lock(div, onMouseLockSuccess, onMouseLockError); } }, false); div.mozRequestFullScreen(); } </script> <button onclick="fullAndLock();">Mouse Lock</button> <div id="div">Element to receive mouse lock</div> 

    Here is a quick video of the Mouse Lock API in action, with comparison to non-mouse lock game-play to see the improvement:

    Providing a console-like experience with the Gamepad API

    Another input-related improvement that is coming to the Web is that of the Gamepad API. No longer are the keyboard and mouse the only options available for your players to engage with your game. The Gamepad API now allows for all sorts of gamepads to be accessed via JavaScript. This even includes some of the console controllers like those on the xBox 360 and Playstation 3 (with third-party drivers)!

    Like the Mouse Lock API, the Gamepad API has just landed in experimental builds of Firefox Nightly and it’s nice and simple to use. Coupled with the Full Screen API, gamepad support can really change the experience of your game from that of a game within a website to that of a desktop game or console.

    To find out when a gamepad has been connected, you need only listen to the MozGamepadConnected event on the window object (remember to use the experimental build linked previously):

    <div id="gamepads"></div> <script> function gamepadConnected(e) { var gamepads = document.getElementById("gamepads"), gamepadId = e.gamepad.id; gamepads.innerHTML += " Gamepad Connected (id=" + gamepadId + ")"; } window.addEventListener("MozGamepadConnected", gamepadConnected, false); </script> 

    Receiving updates about all the buttons and joysticks on a gamepad is also a case of listening for events, or you can manually poll the status of buttons and joysticks whenever you want. Check out the Gamepad API documentation and the input.js library for more information.

    The video below shows the Gamepad API in action, controlling a player in the game Rawkets:

    Adding real-time multiplayer game-play with WebSockets

    If you’re thinking of creating a multiplayer game then before now you would either have put up with the latency involved in constant AJAX requests, or you would have moved to Flash. Neither option is ideal. What’s cool is that since last year this is no longer the case, WebSockets have now arrived to allow for real-time bi-directional communication between the browser and a server.

    But why is bi-directional real-time communication important for games? Well, this means that you can now literally stream data to and from a player’s browser in real time. One obvious benefit to this is that it saves bandwidth by no longer requiring constant AJAX requests to check for new data. Instead the WebSocket connection is left open and data is instantly pushed to the player or server as soon as it is needed. This is perfect for fast-paced games that require updates every few milliseconds. On top of this, the bi-directional nature of WebSockets means that data can be instantly sent both from the server to the player and from the player to the server at the same time.

    Using WebSockets is a case of connecting to the socket server by initialising the WebSocket object:

    var mySocket = new WebSocket("http://www.example.com/socketserver",); 

    From there you can send data using the send method of the WebSocket object:

    mySocket.send("Here's some text that the server is urgently awaiting!"); 

    And you can receive data by listening for the onmessage event on the WebSocket object:

    mySocket.onmessage = function(e) { console.log(e.data); } 

    A full code example for game communication would be too much for this post but you should definitely check out the tutorial that that I made a couple months ago. In it I detail exactly how to create a real-time multiplayer game using Node.js and WebSockets.

    Using games offline with local storage and the application cache

    Creating games on the Web is all well and good, but what about if you want to play that game offline? Or, what if the player’s Internet connection drops out half way through an epic gaming session? Most open Web games today would at worst simply stop working as soon as an Internet connection fails, and at best they would stop sending data to your server and saving player data. When your player refreshes the page that the game is on, they’ll just see a blank page and all their hard work achieved while offline will have been lost. That probably won’t make your players very happy, and unhappy players are not ideal.

    There are a few solutions available today that can help solve these issues. The first is the application cache, which allows you to use a cache manifest to declare particular assets (like HTML, CSS, images, and JavaScript) that you would like the browser to cache for offline use. Not only that, the browser uses the cached versions of the files when when online to speed up the loading process.

    You can include a cache manifest within your game by using the manifest attribute of the <html> element:

    <html manifest="example.appcache"> <h1>Application Cache Example</h1> </html> 

    Writing a cache manifest is straightforward and requires you to follow a simple format to declare the files that you want cached. For example, the manifest below looks for /example/bar online and falls back to the locally-cached example.html file if your player is offline or experiencing network issues:

    CACHE MANIFEST FALLBACK: example/bar/ example.html 

    There is much more to cache manifest than this, so I advise you to read the documentation to discover it in detail.

    Another technique you can use is to store a player’s game data locally and periodically sync it with the game server. Normally you wouldn’t be able to store enough data in the browser to achieve this (like with cookies) but with Local Storage and IndexedDB, you can now store many megabytes of data in a structured way.

    You can also add functionality to your game so it is alerted when a player’s Internet connection goes offline. The navigator.onLine property allows you to use JavaScript to see if your player is currently online or not. You can also use the offline and online events to trigger automatic behaviour in your game when a change in connection occurs, like stopping all WebSockets communication and caching player data locally until the connection is back.

    Creating native OS applications with WebRT

    One of the more profound aspects of the Mozilla Labs Apps project is the integration of a Web run-time (WebRT). This allows players to install your game “natively” on their chosen operating system (Windows, Mac, and Android right now), with a launch icon just like standard OS applications.

    Installing Web apps natively

    WebRT also runs your game using an app-centric user agent (in contrast to browser-centric user agents like Firefox) and runs your game using a separate user profile and OS process to your player’s normal Firefox that they use for browsing.

    The ability of WebRT to break away into another process and remove all the browser UI makes the experience for gaming that much sweeter. There’s something about having an icon in the dock on a Mac that launches your game in it’s own “native” window with no mention or feel that this is a browser.

    Rawkets Web app

    As a developer this is slightly magical. It allows you to break free from a game being a glorified website, instead turning it into an application, an experience in its own right. Mark my words, this will be a turning point in the transition from 5-minute puzzle games on the Web to professional-grade games that have a console-like experience.

    The great news is that enabling WebRT on your game is just a case of creating an app manifest and calling the navigator.mozApps.install method through JavaScript.

    This is an example app manifest from the Rawkets game:

    { "name":"Rawkets", "developer": { "name":"Rob Hawkes", "url":"http://rawkes.com" }, "description":"Space shooter game.", "icons": { "16": "/style/img/icon-16.png", "48": "/style/img/icon-48.png", "128": "/style/img/icon-128.png" }, "launch_path": "/index.html", "installs_allowed_from": ["*"] } 

    As you can see it’s a simple JSON structure. You would then need to pass that manifest file in the call to the navigator.mozApps.install method:

    function installCallback(result) { // great - display a message, or redirect to a launch page } function errorCallback(result) { // whoops - result.code and result.message have details } navigator.mozApps.install( "http://rawkets.dev:8000/manifest.webapp", installCallback, errorCallback ) 

    This is the most basic implementation possible and would result in a panel popping up in the player’s browser asking if they’d like to install your game and whether they’d like that installation to be a native app or not (like the screenshot above).

    Making money from your passion with the Mozilla Labs Apps project

    Finally, the last thing about the apps project that I want to cover is that of making money from your game. Let’s be honest, if you can’t make a living from something then you’re going to have a hard time keeping it up for a long period of time.

    For the Mozilla app marketplace we are using PayPal as the payment provider. All you need do is set a price on the store and the rest will happen automatically as people start paying for your game — you’ll start receiving money in your PayPal account.

    In the future you’ll be able to provide your game on your own website or another store and charge for it there also. This means you’ll be able to use your own payment provider.

    Scratching the surface

    These technologies are really just scratching the surface when it comes to creating games on the Web using open technologies. At Mozilla we’re working hard to bring you these APIs and services to help make the Web a better place for games. The year 2012 will be a game-changer, for sure (pun intended).

    Keep an eye on this blog as we’ll be sure to post more updates and guides as our gaming technologies and the apps project mature.

    In the meantime, here are a few of places to go for extra information on the Mozilla Labs Apps project and gaming at Mozilla: