Mozilla

Articles by Robin Hawkes

Sort by:

View:

  1. Lessons learnt building ViziCities

    Just over 2 weeks ago Peter Smart and Robin Hawkes released the first version of ViziCities to the world. It’s free to use and open-sourced under an MIT license.

    In this post I will talk to you about the lessons learnt during the development of ViziCities. From application architecture to fine-detailed WebGL rendering improvements, we learnt a lot in the past year and we hope that by sharing our experiences we can help others avoid the same mistakes.

    What is ViziCities?

    In a rather geeky nutshell, ViziCities is a WebGL application that allows you to visualise anywhere in the world in 3D. It’s primary purpose is to look at urban areas, though it’ll work perfectly fine in other places if they have decent OpenStreetMap coverage.

    Demo

    The best way to explain what ViziCities does is to try it out yourself. You’ll need a WebGL-enabled browser and an awareness that you’re using pre-alpha quality software. Click and drag your way around, zoom in using the mouse wheel, and rotate the camera by clicking the mouse wheel or holding down shift while clicking and dragging.

    You can always take a look at this short video if you’re unable to try the demo right now:

    What’s the point of it?

    We started the project for multiple reasons. One of those reasons is that it’s an exciting technical and design challenge for us – both Peter and I thrive on pushing ourselves to the limits by exploring something unknown to us.

    Another reason is that we were inspired by the latest SimCity game and how they visualise data about the performance of your city – in fact, Maxis, the developers behind SimCity reached out to us to tell us how much they like the project!

    There’s something exciting about creating a way to do that for real-world cities rather than fictional ones. The idea of visualising a city in 3D with up-to-date data about that city overlaid is an appealing one. Imagine if you could see census data about your area, education data, health data, crime data, property information, live transport (trains, buses, traffic, etc), you’d be able to learn and understand so much more about the place you live.

    This is really just the beginning – the possibilities are endless.

    Why 3D?

    A common question we get is “Why 3D?” – the short answer, beyond “because it’s a visually interesting way of looking at a city”, is that 3D allows you to do things and analyse data in ways that you can’t do in a 2D map. For example by using 3D you can take height and depth into consideration, so you can better visualise the sheer volume of stuff that lies above and below you in a city – things like pipes and underground tunnels, or bridges, overpasses, tall buildings, the weather, and planes! On a 2D map, looking at all of this would be a confusing mess, in 3D you get to see it exactly how it would look in the real world – you can easily see how objects within a city relate to each other.

    Core technology

    At the most basic level ViziCities is built using Three.js, a WebGL library that abstracts all the complexity of 3D rendering in the browser. We use a whole range of other technologies too, like Web Workers, each of which serves a specific purpose or solves a specific problem that we’ve encountered along the way.

    Let’s take a look at some of those now.

    Lessons learnt

    Over the past year we’ve come from knowing practically nothing about 3D rendering and geographic data visualisation, to knowing at least enough about each of them to be dangerous. Along the way we’ve hit many roadblocks and have had to spend a significant amount of time working out what’s wrong and coming up with solutions to get around them.

    The process of problem solving is one I absolutely thrive on, but it’s not for everybody and I hope that the lessons I’m about to share will help you avoid these same problems, allowing you to save time and do more important things with your life.

    These lessons are in no particular order.

    Using a modular, decoupled application architecture pays off in the long run

    We originally started out with hacky, prototypal experiments that were dependency heavy and couldn’t easily be pulled apart and used in other experiments. Although it allowed us to learn how everything worked, it was a mess and caused a world of pain when it came to building out a proper application.

    In the end we re-wrote everything based on a simple approach using the Constructor Pattern and the prototype property. Using this allowed us to separate out logic into decoupled modules, making everything a bit more understandable whilst also allowing us to extend and replace functionality without breaking anything else (we use the Underscore _.extend method to extend objects).

    Here’s an example of our use of the Constructor Pattern.

    To communicate amongst modules we use the Mediator Pattern. This allows us to keep things as decoupled as possible as we can publish events without having to know about who is subscribing to them.

    Here’s an example of our use of the Mediator Pattern:

    /* globals window, _, VIZI */
    (function() {
      "use strict";
     
      // Apply to other objects using _.extend(newObj, VIZI.Mediator);
      VIZI.Mediator = (function() {
        // Storage for topics that can be broadcast or listened to
        var topics = {};
     
        // Subscribe to a topic, supply a callback to be executed
        // when that topic is broadcast to
        var subscribe = function( topic, fn ){
          if ( !topics[topic] ){ 
            topics[topic] = [];
          }
     
          topics[topic].push( { context: this, callback: fn } );
     
          return this;
        };
     
        // Publish/broadcast an event to the rest of the application
        var publish = function( topic ){
          var args;
     
          if ( !topics[topic] ){
            return false;
          } 
     
          args = Array.prototype.slice.call( arguments, 1 );
          for ( var i = 0, l = topics[topic].length; i < l; i++ ) {
     
            var subscription = topics[topic][i];
            subscription.callback.apply( subscription.context, args );
          }
          return this;
        };
     
        return {
          publish: publish,
          subscribe: subscribe
        };
      }());
    }());

    I’d argue that these 2 patterns are the most useful aspects of the new ViziCities application architecture – they have allowed us to iterate quickly without fear of breaking everything.

    Using promises instead of wrestling with callbacks

    Early on in the project I was talking to my friend Hannah Wolfe (Ghost’s CTO) about how annoying callbacks are, particularly when you want to load a bunch of stuff in order. It didn’t take Hannah long to point out how stupid I was being (thanks Hannah) and that I should be using promises instead of wrestling with callbacks. At the time I brushed them off as another hipster fad but in the end she was right (as always) and from that point onwards I used promises wherever possible to take back control of application flow.

    For ViziCities we ended up using the Q library, though there are plenty others to choose from (Hannah uses when.js for Ghost).

    The general usage is the same whatever library you choose – you set up promises and you deal with them at a later date. However, the beauty comes when you want to queue up a bunch of tasks and either handle them in order, or do something when they’re all complete. We use this in a variety of places, most noticeably when loading ViziCities for the first time (also allowing us to output a progress bar).

    I won’t lie, promises take a little while to get your head around but once you do you’ll never look back. I promise (sorry, couldn’t resist).

    Using a consistent build process with basic tests

    I’ve never been one to care too much about process, code quality, testing, or even making sure things are Done Right™. I’m a tinkerer and I much prefer learning and seeing results than spending what feels like wasted time on building a solid process. It turns out my tinkering approach doesn’t work too well for a large Web application which requires consistency and robustness. Who knew?

    The first step for code consistency and quality was to enable strict mode and linting. This meant that the most glaring of errors and inconsistencies were flagged up early on. As an aside, due to our use of the Constructor Pattern we wrapped each module in an anonymous function so we could enable strict mode per module without necessarily enabling it globally.

    At this point it was still a faff to use a manual process for creating new builds (generating a single JavaScript file with all the modules and external dependencies) and to serve the examples. The break-through was adopting a proper build system using Grunt, thanks mostly to a chat I had with Jack Franklin about my woes at an event last year (he subsequently gave me his cold which took 8 weeks to get rid of, but it was worth it).

    Grunt allows us to run a simple command in the terminal to do things like automatically test, concatenate and minify files ready for release. We also use it to serve the local build and auto-refresh examples if they’re open in a browser. You can look at our Grunt setup to see how we set everything up.

    For automated testing we use Mocha, Chai, Sinon.js, Sinon-Chai and PhantomJS. Each of which serves a slightly different purpose in the testing process:

    • Mocha is used for the overall testing framework
    • Chai is used as an assertion library to allows you to write readable tests
    • Sinon.js is used to fake application logic and track behaviour through the testing process
    • PhantomJS is used to run client-side tests in a headless browser from the terminal

    We’ve already put together some (admittedly basic) tests and we plan to improve and increase the test coverage before releasing 0.1.0.

    Travis CI is used to make sure we don’t break anything when pushing changes to GitHub. It automatically performs linting and runs our tests via Grunt when changes are pushed, including pull requests from other contributors (a life saver). Plus it lets you have a cool badge to put on your GitHub readme that shows everyone whether the current version is building successfully.

    Together, these solutions have made ViziCities much more reliable than it has ever been. They also mean that we can move rapidly by building automatically, and they allow us to not have to worry so much about accidentally breaking something. The peace of mind is priceless.

    Monitoring performance to measure improvements

    General performance in frames-per-second can be monitored using FPSMeter. It’s useful for debugging parts of the application that are locking up the browser or preventing the rendering loop from running at a fast pace.

    You can also use the Three.js renderer.info property to monitor what you’re rendering and how it changes over time.

    It’s worth keeping an eye on this to make sure objects are not being rendered when they move out of the current viewport. Early on in ViziCities we had a lot of issues with this not happening and the only way to be sure we to monitor these values.

    Turning geographic coordinates into 2D pixel coordinates using D3.js

    One of the very first problems we encountered was how to turn geographic coordinates (latitude and longitude) into pixel-based coordinates. The math involved to achieve this isn’t simple and it gets even more complicated if you want to consider different geographic projections (trust me, it gets confusing fast).

    Fortunately, the D3.js library has already solved these problems for you, specifically within its geo module. Assuming you’ve included D3.js, you can convert coordinates like so:

    var geoCoords = [-0.01924, 51.50358]; // Central point as [lon, lat]
    var tileSize = 256; // Pixel size of a single map tile
    var zoom = 15; // Zoom level
     
    var projection = d3.geo.mercator()
      .center(geoCoords) // Geographic coordinates of map centre
      .translate([0, 0]) // Pixel coordinates of .center()
      .scale(tileSize << zoom); // Scaling value
     
    // Pixel location of Heathrow Airport to relation to central point (geoCoords)
    var pixelValue = projection([-0.465567112, 51.4718071791]); // Returns [x, y]

    The scale() value is the hardest part of the process to understand. It basically changes the pixel value that’s returned based on how zoomed in you want to be (imagine zooming in on Google Maps). It took me a very long time to understand so I detailed how scale works in the ViziCities source code for others to learn from (and so I can remember!). Once you nail the scaling then you will be in full control of the geographic-to-pixel conversion process.

    Extruding 2D building outlines into 3D objects on-the-fly

    While 2D building outlines are easy to find, turning them into 3D objects turned out to be not quite as easy as we imagined. There’s currently no public dataset containing 3D buildings, which is a shame though it makes it more fun to do it yourself.

    What we ended up using was the THREE.ExtrudeGeometry object, passing in a reference to an array of pixel points (as a THREE.Shape object) representing a 2D building footprint.

    The following is a basic example that would extrude a 2D outline into a 3D object:

    var shape = new THREE.Shape();
    shape.moveTo(0, 0);
    shape.lineTo(10, 0);
    shape.lineTo(10, 10);
    shape.lineTo(0, 10);
    shape.lineTo(0, 0); // Remember to close the shape
     
    var height = 10;
    var extrudeSettings = { amount: height, bevelEnabled: false };
     
    var geom = new THREE.ExtrudeGeometry( shape, extrudeSettings );
    var mesh = new THREE.Mesh(geom);

    What turned out interesting was how it actually turned out quicker to generate 3D objects on-the-fly than to pre-render them and load them in. This was mostly due to the fact it would take longer to download a pre-rendered 3D object than downloading the 2D coordinates string and generating it at runtime.

    Using Web Workers to dramatically increase performance and prevent browser lockup

    One thing we did notice with the generation of 3D objects was that it locked up the browser, particularly when processing a large number of shapes at the same time (you know, like an entire city). To work around this we delved into the magical world of Web Workers.

    Web Workers allow you to run parts of your application in a completely separate processor thread to the browser renderer, meaning anything that happens in the Web Worker thread won’t slow down the browser renderer (ie. it won’t lock up). It’s incredibly powerful but it can also be incredibly complicated to get working as you want it to.

    We ended up using the Catiline.js Web Worker library to abstract some of the complexity and allow us to focus on using Web Workers to our advantage, rather than fighting against them. The result is a Web Worker processing script that’s passed 2D coordinate arrays and returns generated 3D objects.

    After getting this working we noticed that while the majority of browser lock-ups were eliminated, there were two new lock-ups introduced. Specifically, there was a lock-up when the 2D coordinates were passed into the Web Worker scripts, and another lock-up when the 3D objects were returned back to the main application.

    The solution to this problem came from the inspiring mind of Vladimir Agafonkin (of LeafletJS fame). He helped me understand that to avoid the latter of the lock-ups (passing the 3D objects back to the application) I needed to use transferrable objects), namely ArrayBuffer objects. Doing this allows you to effectively transfer ownership of objects created within a Web Worker thread to the main application thread, rather than copying them. We implemented this to great effect, eliminating the second lock-up entirely.

    To eliminate the first lock-up (passing 2D coordinates into the Web Worker) we need to take a different approach. The problem lies again with the copying of the data, though in this case you can’t use transferrable objects. The solution instead lies in loading the data into the Web Worker script using the importScripts method. Unfortunately, I’ve not yet worked out a way to do this with dynamic data sourced from XHR requests. Still, this is definitely a solution that would work.

    Using simplify.js to reduce the complexity of 2D shapes before rendering

    Something we found early on was that complex 2D shapes caused a lot of strain when rendered as 3D objects en-masse. To get around this we use Vladimir Agafonkin’s simplify.js library to reduce the quality of 2D shapes before rendering.

    It’s a great little tool that allows you to keep the general shape while dramatically reducing the number of points used, thus reducing its complexity and render cost. By using this method we could render many more objects with little to no change in how the objects look.

    Getting accurate heights for buildings is really difficult

    One problem we never imagined encountering was getting accurate height information for buildings within a city. While the data does exist, it’s usually unfathomably expensive or requires you to be in education to get discounted access.

    The approach we went for uses accurate height data from OpenStreetMap (if available), falling back to a best-guess that uses the building type combined with 2D footprint area. In most cases this will give a far more accurate guess at the height than simply going for a random height (which is how we originally did it).

    Restricting camera movement to control performance

    The original dream with ViziCities was to visualise an entire city in one go, flying around, looking down on the city from the clouds like some kind of God. We fast learnt that this came at a price, a performance price, and a data-size price. Neither of which we were able to afford.

    When we realised this wasn’t going to be possible we looked at how to approach things from a different angle. How can you feel like you’re looking at an entire city without rendering an entire city? The solution was deceptively simple.

    By restricting camera movement to only view a small area at a time (limiting zoom and rotation) you’re able to have much more control over how many objects can possibly be in view at one time. For example, if you prevent someone from being able to tilt a camera to look at the horizon then you’ll never need to render every single object between the camera and the edge of your scene.

    This simple approach means that you can go absolutely anywhere in the world within ViziCities, whether a thriving metropolis or a country-side retreat, and not have to change the way you handle performance. Every situation is predictable and therefore optimisable.

    Tile-based batching of objects to improve loading and rendering performance

    Another approach we took to improve performance was by splitting the entire world into a grid system, exactly like how Google and other map providers do things. This allows you to load data in small chunks that eventually build up to a complete image.

    In the case of ViziCities, we use the tiles to only request JSON data for the geographic area visible to you. This means that you can start outputting 3D objects as each tile loads rather than waiting for everything to load.

    A by-product of this approach is that you get to benefit from frustum culling, which is when objects not within your view are not rendered, dramatically increasing performance.

    Caching of loaded data to save on time and resources when viewing the same location

    Coupled with the tile-based loading is a caching system that means that you don’t request the same data twice, instead pulling the data from a local store. This saves bandwidth but also saves time as it can take a while to download each JSON tile.

    We currently use a dumb local approach that resets the cache on each refresh, but we plan to implement something like localForage to have the cache persist between browser sessions.

    Using the OpenStreetMap Overpass API rather than rolling your own PostGIS database

    Late into the development of ViziCities we realised that it was unfeasible to continue using our own PostGIS database to store and manipulate geographic data. For one, it would require a huge server just to store the entirety of OpenStreetMap in a database, but really it was just a pain to set up and manage and an external approach was required.

    The solution came in the shape of the Overpass API, an external JSON and XML endpoint to OpenStreetMap data. Overpass allows you to send a request for specific OpenStreetMap tags within a bounding box (in our case, a map tile):

    http://overpass-api.de/api/interpreter?data=[out:json];((way(51.50874,-0.02197,51.51558,-0.01099)[%22building%22]);(._;node(w);););out;

    And get back a lovely JSON response:

    {
      "version": 0.6,
      "generator": "Overpass API",
      "osm3s": {
        "timestamp_osm_base": "2014-03-02T22:08:02Z",
        "copyright": "The data included in this document is from www.openstreetmap.org. The data is made available under ODbL."
      },
      "elements": [
     
    {
      "type": "node",
      "id": 262890340,
      "lat": 51.5118466,
      "lon": -0.0205134
    },
    {
      "type": "node",
      "id": 278157418,
      "lat": 51.5143963,
      "lon": -0.0144833
    },
    ...
    {
      "type": "way",
      "id": 50258319,
      "nodes": [
        638736123,
        638736125,
        638736127,
        638736129,
        638736123
      ],
      "tags": {
        "building": "yes",
        "leisure": "sports_centre",
        "name": "The Workhouse"
      }
    },
    {
      "type": "way",
      "id": 50258326,
      "nodes": [
        638736168,
        638736170,
        638736171,
        638736172,
        638736168
      ],
      "tags": {
        "building": "yes",
        "name": "Poplar Playcentre"
      }
    },
    ...
      ]
    }

    The by-product of this was that you get worldwide support out of the box and benefit from minutely OpenSteetMap updates. Seriously, if you edit or add something to OpenStreetMap (please do) it will show up in ViziCities within minutes.

    Limiting the number of concurrent XHR requests

    Something we learnt very recently was that spamming the Overpass API endpoint with a tonne of XHR requests at the same time wasn’t particularly good for us nor for Overpass. It generally caused delays as Overpass rate-limited us so data took a long time to make its way back to the browser. The great thing was that by already using promises to manage the XHR requests we were half-way ready to solve the problem.

    The final piece of the puzzle is to use throat.js to limit the number of concurrent XHR requests so we can take control and load resources without abusing external APIs. It’s beautifully simple and worked perfectly. No more loading delays!

    Using ViziCities in your own project

    I hope that these lessons and tips have helped in some way, and I hope that it encourages you to try out ViziCities for yourself. Getting set up is easy and well documented, just head to the ViziCities GitHub repo and you’ll find everything you need.

    Contributing to ViziCities

    Part of the reason why we opened up ViziCities was to encourage other people to help build it and make it even more awesome than Peter and I could ever make it. Since launch, we’ve had over 1,000 people favourite the project on GitHub, as well as nearly 100 forks. More importantly, we’ve had 9 Pull Requests from members of the community who we didn’t previously know and who we’ve not asked to help. It’s such an amazing feeling to see people help out like that.

    If we were to pick a favourite contribution so far, it would be adding the ability to load anywhere in the world by putting coordinates in the URL. Such a cool feature and one that has made the project much more usable for everyone.

    We’d love to have more people contribute, whether dealing with issues or playing with the visual styling. Read more about how to contribute and give it a go!

    What’s next?

    It’s been a crazy year and an even crazier fortnight since we launched the project. We never imagined it would excite people in the way it has, it’s blown us away.

    The next steps are slowly going through the issues and getting ready for the 0.1.0 release, which will still be alpha quality but will be sort of stable. Aside from that we’ll continue experimenting with exciting new technologies like the Oculus Rift (yes, that’s me with one strapped to my face)…

    Visualising realtime air traffic in 3D…

    And much, much more. Watch this space.

  2. Progress update on WebRTC for Firefox on desktop

    WebRTC for desktop is now in Firefox Nightly and is also in Firefox Aurora, though Nightly has the hottest up-to-date fixes.

    We support mozGetUserMedia, mozRTCPeerConnection and DataChannels. We have a basic UI for mozGetUserMedia which we expect to be updating in the
    coming weeks.

    Enabling WebRTC in Firefox

    The code is behind a pref for now, pending more testing. To enable our WebRTC code in Firefox’s Nightly desktop build, browse to about:config
    and change the media.peerconnection.enabled preference to true.

    Here are the 3 relevant prefs to getUserMedia() and mozRTCPeerConnection():

    • media.navigator.enabled enables calls to mozGetUserMedia() only
    • media.navigator.permission.disabled automatically gives permission to access the camera/microphone and bypasses the permission/selection dialog
    • media.peerconnection.enabled enables use of mozRTCPeerConnection()

    Note: media.peerconnection.enabled implies media.navigator.enabled has been set to true.

    Demos & upcoming changes

    There’s a lot you can do with these APIs, even today. For examples, check out our test landing page on GitHub. We’ll try and put up notices if you’re running an out-of-date browser — as well as news updates about important bug fixes and API changes there!

    Upcoming changes include:

    • Support for constraints (to getUserMedia and createOffer/Answer)
    • Control of bandwidth, resolution, echo cancellation, etc.
    • Statistics
    • TURN support (to allow connections between devices behind symmetric NATs)
    • Fixes for audio drift (progressive loss of A/V sync)
    • Trickle ICE, rtcp-mux and BUNDLE support
    • getUserMedia() UI updates
    • And many bugfixes

    To give you an idea of the power of these APIs, in a couple of days our team whipped up a Social API integration demo that allows you to video +
    text chat with your friends, drag-and-drop files to each other, drop links, tabs, etc, all making simple use of the DataChannel API.

    The DataChannel API is quite simple on the surface, and has an API very similar to WebSockets. A quick example:

    Filing bugs & moving forward

    Progress on WebRTC (and bug-fixing) is rapid, and we encourage you to try it out and submit bugs. (We have plenty! But we’re nailing them as fast as we can, so make sure you’re on nightly and update regularly.)

    Bug reports are highly appreciated. Please file them on
    Bugzilla under “Product:Core”, “Component:WebRTC”.

    The team is both excited by all the progress, and exhausted. The work so far represents tons of hours of work from so many people on the Firefox team (too many people to name — especially because we don’t want to forget
    someone — but you know who you are!). Thank you to everyone who helped us land this “747″ on the flight deck.

    We’ll continue to blog regularly on our progress as we work to make this a great product feature for Firefox and the web.

  3. Creating the future of mobile with Firefox OS – resources, docs and more!

    Just under a month ago I wrote a personal post about my thoughts on Firefox OS and why I think there is something ‘magical’ about what it stands for and the possibilities it brings to the table. This post is a follow-up that aims to cover much of the same ground but with extra detail and more of a technical focus.

    What is Firefox OS?

    In short, Firefox OS is about taking the technologies behind the Web, like JavaScript, and using them to produce an entire mobile operating system. It’s effectively a mobile OS powered by JavaScript!

    Firefox OS screenshots

    This is achieved with a custom version of Gecko, the rendering engine in Firefox, that introduces a variety of new JavaScript APIs needed to create a phone-like experience; things like WebSMS to send text messages, and WebTelephony to make phone calls.

    You might be wondering what’s running Gecko, as a phone can’t naturally boot directly into Gecko. To do that, the phone boots into a very lightweight Linux kernel that, in turn, boots the Gecko process. The process is a little more involved than that and much more detail can be found in the B2G Architecture documentation, including how Gecko accesses the radio hardware and other phone-specific functionality.

    The Firefox OS project also aims to combine many of the other projects at Mozilla into a single vision, what we refer to as the Web as the platform. These projects include the Open Web Apps initiative and Persona, our solution to identity and logins on the Web (formally known as BrowserID). It’s the combination of these various technologies that completes Firefox OS.

    If you want to find out more technical information about the OS then definitely check out the Firefox OS pages on MDN.

    Why Firefox OS?

    A couple of common questions that come up are, “Why Firefox OS?” and more specifically, “Why build a mobile OS using JavaScript?” These are incredibly important questions so let’s take a moment to delve into them in a little detail.

    Why build a mobile OS using JavaScript?

    Answering this question can quite simply be boiled down to one sentence; because it’s possible. It’s not the one and only answer but it succinctly handles most of the arguments against JavaScript being used in this way.

    A longer answer is that a JavaScript-powered OS unlocks a whole range of possibilities that aren’t normally or easily available to developers and users with existing operating systems.

    The most obvious of the possibilities is the ability to build applications using the technologies that we already use to build websites; namely JavaScript, CSS, and HTML. While not a truly unique feature of Firefox OS — projects like PhoneGap have done this for years on ‘native’ platforms — it allows developers everywhere to create mobile applications without having to learn native languages and APIs.

    Another draw of JavaScript is that it’s both extremely well documented and free to develop with. Anyone could sit down for a weekend and put together an application without having to pay for a single thing. Obviously that’s not true in the majority of cases, as people tend to buy their own hosting or tooling, but theoretically there is nothing to stop you building with these technologies for free.

    What’s arguably most interesting about JavaScript being used in this way is that it inherently enables physical devices to communicate using the same APIs that we already use on websites. In effect, instead of accessing the Web through a mobile browser the entire phone is now capable of accessing and communicating with any Web API. For example, there is nothing to stop you building an application for Firefox OS that uses WebRTC (once added) to create Skype-like P2P video communication between phones, desktop computers, or anything else that supports WebRTC.

    This really only scrapes the surface of “Why JavaScript?” but it certainly gives you a feel of how this is both interesting and important, beyond the tired debate of ‘native’ vs. Web. If you’re still not convinced, just think for a moment about how you can now customise an entire mobile OS using nothing by JavaScript. You’d be hard pushed to deny that it’s pretty darn interesting!

    OK, but why Firefox OS?

    Effectively, Firefox OS has been built to put our money where our mouth is (so to speak) and prove that JavaScript is capable of doing what we say it can do. However, there is much more to the project than just proving the the technology is fast enough.

    The first reason ‘Why Firefox OS’ is that the mobile ecosystem is overrun with proprietary platforms, most of which prevent you from easily moving between various platforms. What Firefox OS aims to achieve is a truly ‘open’ platform that doesn’t lock you in and inherently makes it as easy and possible to move between devices as and when you choose.

    Mozilla is effectively replicating its success with Firefox, in which it stormed the browser market and showed users that there is an alternative, one that lets them be in control of how they use the Web. In this case, it’s less about browsers and more about mobile platforms and application portability.

    Another reason is that Firefox OS is an attempt to push the Web forward into the world of physical devices. One direct benefit of this is the addition of brand new Web standards and APIs that allow for things like hardware access using JavaScript.

    Plenty of challenges

    It’s fair to say that the Firefox OS journey will contain a number of technical challenges along the way, however that’s part of the fun and the reasons why we’re working on it.

    One of those challenges is how to manage an apps ecosystem that is open and distributed. This is something that we are tackling with the Open Web Apps initiative and the Mozilla Marketplace. It’s a challenge that we are dealing with as things progress and as we learn more about how things work best, as is the nature with new ways of thinking.

    Another of the challenges is making sure that the phone runs as fast as possible, creating the best experience possible. This also relates to questions raised within the developer community around the performance capabilities of JavaScript, particularly when it is used to do things that are perceived to be complex, or when it is compared against ‘native’ technologies. This is a challenge that we are taking very seriously and one which we feel we can overcome. In fact, it’s a challenge that I believe we have already overcome.

    One prime example of how capable JavaScript has become is seeing beautiful JavaScript games running in Firefox OS at near-enough 60 frames per-second, on a low-end, cheap phone.

    Beyond the mobile phone

    While the phone aspect of Firefox OS is immediately interesting, you should consider the wider implications of a JavaScript OS and what possibilities it unlocks. For example, what other devices could benefit from being powered by JavaScript? And, what would a network of JavaScript-powered devices allow us to do — things like Ubiquitous Computing, perhaps?

    These aren’t things that we are exploring directly at Mozilla, but they are things that are now inherently possible as a result of the work that we’re doing. There is nothing to stop you taking the Firefox OS source code from GitHub and porting it to a device that we’ve never even considered.

    We’re already starting to see this happen with examples like a Firefox OS port for the Raspberry Pi, as well as another for the Pandaboard.

    What about a game console powered by Firefox OS? A TV, or set-top box? What about a fridge? Individually, these are all interesting projects, but together they offer something we don’t really have at the moment, a network of different devices powered by the same, open technologies and able to access and communicate across the Web with the same APIs.

    We are a long way away from that kind of world but it is projects like Firefox OS that may pave the way for it to happen. You could even be a part of it!

    Getting started with Firefox OS

    The hope is that by now you’re sufficiently interested in Firefox OS to begin exploring, experimenting and playing with it. The good news is that there are a whole host of ways that you can do that.

    Documentation

    One of the first places to start is the MDN documentation surrounding Firefox OS and its related technologies. This is where you’ll find everything you need to know about the developer-facing aspects of the platform.

    If you’re more interested with the inner-workings of the platform then you’ll want to cast an eye over the B2G wiki, which outlines much of the internals in plenty of detail.

    Source code

    If you’re keen to get to grips with the source code of Firefox OS then you’ll want to head over to GitHub and check it out. The two main repositories that you want are ‘b2g’ (the underlying Gecko engine) and ‘gaia’ (everything you can see, the OS).

    Getting involved

    There are a few ways to get involved with the project. You could check out some of the issues and get involved in fixing them, or perhaps just hang out in the mailing list for B2G, or the one for Gaia, and take part in the discussions there.

    If you just want to ask a few immediate questions then try out the #b2g and #gaia rooms on irc.mozilla.org. We’re all pretty friendly!

    Development options

    If you just want to dig in and make some applications, or perhaps customise the OS, then you’ll need to know about the various development options available to you. They are covered in some detail on MDN but here is a brief overview.

    The simplest method to get started is running Gaia (the visual side of Firefox OS) within Firefox Nightly. This doesn’t give you a true representation of a phone environment but it will allow you to install applications and use all of the developer tools within the browser that you’re already used to.

    Slightly more involved than Nightly is using the desktop B2G client. This is effectively a chromeless build of Firefox that looks phone-like has some added APIs that aren’t normally available in standard Firefox. This doesn’t replicate phone hardware but it’s the next best thing before starting to develop on an actual device.

    Setting up the desktop B2G client isn’t too hard, but it could be made easier. In the meantime, projects like r2d2b2g aim to make the process super simple. Definitely worth checking out.

    The last method, and arguably the most important one, is developing on an actual Firefox OS device. This is the only method that will give you a true representation of how your application will perform. It is also the only method that will give you access to the all the new APIs that come with Firefox OS.

    Right now, you’ll need to build and install Firefox OS on one of the supported devices. In the future you will be able to skip this step and get access to devices that already run Firefox OS. We don’t have any dates for that just yet.

    Go forth and be part of something big

    My hope is that by now you should now have enough inspiration and information to go forth and begin building for this new platform, powered by the technologies you already use. We hope you do and we’d love to see what you come up with.

    It’s not every day that you get the opportunity to be a part of something that could quite literally change the way we do things.

  4. Mozilla supports the Liberated Pixel Cup for open games

    We’re excited to announce that Mozilla is supporting the Liberated Pixel Cup, a fantastic competition aiming to spark the creation of artwork and code for games that are free and available for others to use.

    The Liberated Pixel Cup is the brainchild of OpenGameArt, a long-standing community of artists that provide graphics and sound effects for others to use in their games. Joining us in supporting the competition are the Creative Commons, Free Software Foundation, as well as the hundreds of everyday supporters who donated to the prize fund.

    Getting involved with the Liberated Pixel Cup was an obvious choice for us. Over the past year we’ve vastly improved our performance for games in Firefox, we’ve added new features for games, we’ve hosted a work week to discuss the future of games, we’ve attended and represented the open Web at important game-related events like GDC, as well as much, much more. To put it simply, we think games are pretty darn important to the future of the Web!

    As part of our involvement, we’ll be:

    • Supporting the HTML5 category of the competition
    • Using our various spaces around the world to host Liberated Pixel Cup events
    • Donating funds to support the running of the competition

    The art side of the competition starts in June, with the code side starting in July; so mark those dates in your diary and get ready to create some game-related goodness!

    Check out the Liberated Pixel Cup website for more information.

  5. Come and make games with us at GameHack

    This coming weekend (28–29th April) we’ll be at GameHack, hosted at Pinewood Studios in the UK.

    GameHack

    The weekend-long event is focussed on creating amazing games on a variety of platforms. We’re working with the highly-experienced guys at Turbulenz to encourage the creation of HTML5 games.

    And it’s not just about having fun making games (thought that’s mostly the point). If your game is good enough then you could win yourself one of our Boot to Gecko phones!

    More information can be found on the GameHack website. Tickets are still available and cost £5, with all proceeds going to charity.

    I’ll be the dude in the black Firefox hoodie, and I’ll give you a demo of Boot to Gecko if you ask nicely. I look forward to seeing you there.

  6. Developing a simple HTML5 space shooter

    Experimenting with modern web technologies is always fun. For my latest project, I came up with the following requirements:

    • Not a complex game, rather a proof-of-concept
    • Space shooter theme
    • Canvas-based rendering, but no WebGL
    • Re-use of existing sprites (I am no artist)
    • Rudimentary audio support
    • AI/Bots
    • Working network multiplayer

    I worked on this project only in my spare time; after approx. two weeks, the work was done: Just Spaceships! This article describes some decisions and approaches I took during the development. If you are interested in the code, you can check the relevant Google Code project.

    Animation & timing

    There are two basic ways to maintain an animation: requestAnimationFrame and setTimeout/setInterval. What are their strengths and weaknesses?

    • requestAnimationFrame instructs the browser to execute a callback function when it is a good time (for animation). In most browsers this means 60fps, but other values are used as well (30fps on certain tablets). Deciding the correct timing is up to the browser; the developer has no control over it. When the page is in background (user switches to another tab), animation can be automatically slowed down or even completely stopped. This approach is well suited for fluent animation tasks.
    • setTimeout instructs the browser to execute your next (animation) step after a given time has passed; the browser will try to fulfill this request as precisely as possible. No slowdowns are performed when the page is in background, which means that this approach is well suited for physics simulation.

    To solve animation-related stuff, I created a HTML5 Animation Framework (HAF), which combines both approaches together. Two independent loops are used: one for simulation, the second one for rendering. Our objects (actors in HAF terminology) must implement two basic methods:

    /* simulation: this is called in a setTimeout-based loop */
    Ship.prototype.tick = function(dt) {
    	var oldPosition = this._position;
    	this._position += dt * this._velocity;
    	return (oldPosition != this._position);
    }
     
    /* animation: this is called in a requestAnimationFrame loop */
    Ship.prototype.draw = function(context) {
    	context.drawImage(this._image, this._position);
    }

    Note that the tick method returns a boolean value: it corresponds to the fact that a (visual) position of an object might (or might not) change during the simulation. HAF takes care of this – when it comes to rendering, it redraws only those actors that returned true in their tick method.

    Rendering

    Just Spaceships makes extensive use of sprites; pre-rendered images which are drawn to canvas using its drawImage method. I created a benchmarking page which you can try online; it turns out that using sprites is way faster than drawing stuff via beginPath or putImageData.

    Most sprites are animated: their source images contain all animation frames (such as in CSS sprites) and only one frame (rectangular part of the full source image) is drawn at a time. Therefore, the core drawing part for an animated sprite looks like this:

    Ship.prototype.draw = function(context) {
    	var fps = 10;          /* frames per second */
    	var totalFrames = 100; /* how many frames the whole animation has? */
    	var currentFrame = Math.floor(this._currentTime * fps) % totalFrames;
     
    	var size = 16; /* size of one sprite frame */
     
    	context.drawImage(
    		/* HTML <img> or another canvas */
    		this._image,
     
    		/* position and size within source sprite */
    		0, currentFrame * size, size, size,
     
    		/* position and size within target canvas */
    		this._x, this._y, size, size
    	);
    }

    By the way: the sprites for Just Spaceships! have been taken from Space Rangers 2, my favourite game.

    The golden rule of rendering is intuitive: redraw only those parts of the screen which actually changed. While it sounds pretty simple, following it might turn out to be rather challenging. In Just Spaceships, I took two different approaches for re-drawing sprites:

    • Ships, lasers and explosions use technique known as “dirty rectangles”; when an object changes (moves, animates, …), we redraw (clearRect + drawImage) only the area covered by its bounding box. Some deeper bounding box analysis must be performed, because bounding boxes of multiple objects can overlap; in this case, all overlapping objects must be redrawn.
    • Starfield background has its own layer (canvas), positioned below other objects. The full background image is first pre-rendered into a large (3000×3000px) hidden canvas; when the viewport changes, a subset of this large canvas is drawn into background’s layer.

    Sound

    Using HTML5 audio is a piece of cake – at least for desktop browsers. There were only two issues that needed to be taken care of:

    1. File format – the format war is far from being over; the most universal approach is to offer both MP3 and OGG versions.
    2. Performance issues on certain slower configurations when playing multiple sounds simultaneously. More particularly, my Linux box exhibited some slowdowns; it would be best to offer an option to turn the whole audio off (not implemented yet).

    At the end of the day, the audio code looks basically like this:

    /* detection */
    var supported = !!window.Audio && !audio_disabled_for_performance_reasons;
     
    /* format */
    var format = new Audio().canPlayType("audio/ogg") ? "ogg" : "mp3";
     
    /* playback */
    if (supported) { new Audio(file + "." + format).play(); }

    Multiplayer & networking model

    Choosing a proper networking model is crucial for user experience. For a realtime game, I decided to go with a client-server architecture. Every client maintains a WebSocket connection to central server, which controls the game flow.

    In an ideal world, clients would send just keystrokes and the server would repeat them to other clients. Unfortunately, this is not possible (due to latency); to maintain consistency and synchronicity between players, it is necessary to run the full simulation at server and periodically notify clients about various physical attributes of ships and other entities. This approach is known as an authoritative server.

    Finally, clients cannot just wait for server packets to update their state; players need the game to work even between periodical server messages. This means that browsers run their version of the simulation as well – and correct their internal state by data sent by server. This is known as a client-side prediction. A sample implementation of these principles looks like this:

    /* physical simulation step - client-side prediction */
    Ship.prototype.tick = function(dt) {
    	/*
    		assume only these physical properties:
    			acceleration, velocity and position
    	*/
    	this._position += dt * this._velocity;
    	this._velocity += dt * this._acceleration;
    }
     
    /* "onmessage" event handler for a WebSocket data connection;
    	used to override our physical attributes by server-sent values */
    Ship.prototype.onMessage = function(e) {
    	var data = JSON.parse(e.data);
     
    	this._position = data.position;
    	this._velocity = data.velocity;
    	this._acceleration = data.acceleration;
    }

    You can find very useful reading about this at Glenn Fiedler’s site.

    Multiplayer & server

    Choosing a server-side solution was very easy: I decided to go with v8cgi, a multi-purpose server-side javascripting environment, based on V8. Not only it is older than Node, but (most importantly) it was created and maintained by myself ;-).

    The advantage of using server-side JavaScript is obvious: game’s server runs the identical code that is executed in browser. Even HAF works server-side; I just turned off its rendering loop and the simulation works as expected. This is a cool demonstration of client-server code sharing; something we will probably see more and more in the upcoming years.

    Modulus

    In order to make the game more interesting and challenging, I decided that the whole playing area should wrap around – think of the game universe as of a large toroidal surface. When a spaceship flies far to the left, it will appear from the right; the same holds for other directions as well. How do we implement this? Let’s have a look at a typical simulation time step:

    /* Variant #1 - no wrapping */
    Ship.prototype.tick = function(dt) {
    	this._position += dt * this._velocity;
    }

    To create an illusion of a wrapped surface, we need the ship to remain in a fixed area of a given size. A modulus operator comes to help:

    /* Variant #2 - wrapping using modulus operator */
    Ship.prototype.tick = function(dt) {
    	var universe_size = 3000; // some large constant number
    	this._position += (dt * this._velocity) % universe_size;
    }

    However, there is a glitch. To see it, we have to solve this (elementary school) formula:

    (-7) % 5 = ?

    JavaScript’s answer is -2; Python says 3. Who is the winner? The correctness of the result depends on the definition, but for my purpose, the positive value is certainly more useful. A simple trick was necessary to correct JavaScript’s behavior:

    /* Returned value is always >= 0 && < n */
    Number.prototype.mod = function(n) {
    	return ((this % n) + n) % n;
    }
     
    /* Variant #3 - wrapping using custom modulus method */
    Ship.prototype.tick = function(dt) {
    	var universe_size = 3000; // some large constant number
    	this._position += (dt * this._velocity).mod(universe_size);
    }

    Lessons learned

    Here is a short summary of general tips & hints I gathered when developing JS games in general, but mostly during Just Spaceships development:

    • Know the language! It is very difficult to create anything without properly understanding the programming language.
    • Lack of art (sprites, music, sfx, …) should not stop you from developing. There are tons of resources for getting these assets; the need for original art is relevant only it later stages of the project.
    • Use the paper, Luke! You know what my favorite development tools are? A pen and a sheet of squared paper.
    • If you are not 100% sure about the whole game architecture, start with smaller (working) parts. Refactoring them later to form a larger project is natural, useful and easy.
    • Collect feedback as soon as possible – at the end of the day, it is the users’ opinion that matters the most.

    TODO

    As stated before, Just Spaceships is not a complete project. There is definitely room for improvements, most notably:

    • Optimize the network protocol by reducing the amount of data sent.
    • Offer more options to optimize canvas performance (decrease simulation FPS, turn off background, turn off audio, …).
    • Improve the AI by implementing different behavior models.

    Even with these unfinished issues, I think the game reached a playable state. We had quite a lot of fun testing its multiplayer component; I hope you will enjoy playing it!

     

  7. Rounding-up the first Mozilla HTML5 games work week

    A few weeks ago I wrote about the first Mozilla HTML5 games work week in Toronto and why it is important. In this post I summarise the week’s events and highlight some key observations.

    DSCF0062

    Before I start I should point out that I won’t be making any dramatic announcements from Mozilla or going into too much detail about some of the decisions that were made during the work week. While some important discussions and decisions did occur, some aren’t quite ready to be made public yet. However, we will be posting specifically about some of these things in the coming weeks so keep an eye open. The purpose of this post is to give an overview of the week’s events and some of the discussions that were made.

    So with that out of the way, let’s crack on.

    Why a HTML5 games work week?

    With the amount of buzz happening around HTML5 and JavaScript, it could be seen as surprising that we hadn’t organised a get-together of all the interested parties within Mozilla sooner. The long and short of it is that it has only been recently that we have begun to construct a consistent vision and commitment to HTML5 games.

    To create a vision we need to find out who within the Mozilla community is actually interested in games and how we can work together to lay the path towards the future. The HTML5 games work week was set up to start the ball rolling. It’s safe to say that Mozilla cares a lot about games and this work week is just the beginning.

    Aside from the internal interest and passion to build games, there are an increasing number of HTML5 games being released in the Web community each and every week. This shows that there is desire for these types of games and for browsers to work out how to support them.

    DSCF0068

    It’s also clear that there are areas of improvement that need to be addressed within all browsers so that HTML5 games can flourish in the future. The point of the work week has been to bring all interested parties together in one place to make stuff happen and actually make decisions about how to address these issues.

    From another perspective, the work week gave Mozilla the chance to listen to external developers and hear about the issues they are experiencing in the real world of HTML5 game development. We don’t develop games every day, so it is of absolute importance for us to fully understand the concerns of the people who do. Without this type of feedback we simply won’t be able to make our improvements as useful as they could be.

    Who was there?

    The work week was split into 2 parts: the first was attended by Mozilla staff and key Mozilla contributors, and the second included some invited external guests.

    DSCF0083

    It was clear from the beginning that we had a whole bunch of people within Mozilla who were passionate about games and wanted to see us commit to them. I already knew a handful of these people, but I had absolutely no idea that so many Mozillians from so many different areas of the organisation were so interested. It great to see all those people in one (big) room working together to make awesome stuff happen.

    The invited guests for the second part of the work week included developers from game studios like EA, as well as individual developers who have proven themselves experts in specific technologies, like WebGL. We worked with these guests to hear about issues from the guys dealing with this stuff every single day. It was absolutely fascinating to hear their concerns and work with them to find ways of improving the situation for games on the Web.

    What was discussed?

    Throughout the week we discussed a whole variety of topics ranging from how to better engage with the game developer community all the way to narrowing down fine-grained performance improvements to be made in JavaScript.

    DSCF0082

    Overall, both the internal Mozilla staff and invited guests covered the following areas and many more (in no particular order):

    • Looking into the issue of mobile and how its requirements differ from those on the desktop
    • Working out what tools are needed for authoring HTML5 games
    • Graphics, specifically Canvas and WebGL
    • Delving into the issues surrounding DRM and the protection of assets
    • NaCl and Emscripten as methods of porting existing games to the Web
    • Being clear about who exactly wants HTML5 games
    • Looking at what dev tools are required to ease the debugging of games
    • Learning the business model behind games from those with years of experience in this area
    • Working out how to better engage with the game development community
    • Producing a roadmap for the Firefox platform specifically related to games
    • Looking at ways JavaScript performance could be increased
    • Discussing the situation with asset management and how it can be solved
    • Uncovering any new WebAPIs that might become necessary
    • Delving deep into common audio issues and looking at its future
    • Hearing how high-precision timers would be used by game developers
    • Learning about WebRTC and how it will provide the functionality for peer-to-peer communication
    • Looking at how games tie into the Mozilla Marketplace and the questions that Mozilla needs to answer in this area

    All in all we covered a huge amount of stuff in the space of just a few days. I won’t list each and every decision that was made but rest assured that you’ll be hearing about them in the near-future as they get finalised and released.

    In the meantime, please check out the Are We Fun Yet site as it’s where you’ll be able to keep tabs on the status of game-related technology within Mozilla. There are also some links to the detailed feedback and notes taken during this work week.

    External demos and code review

    During the final 2 days of the work week we saw demos from our external guests and worked with them to review specific issues that they have been experiencing in Firefox.

    DSCF0080

    These demos proved invaluable for those within Mozilla to get a better idea of how professional game developers are using the technologies on the Web. We were inspired and amazed by the quality of these demos and the kind of games that could be created by pushing JavaScript right to the very limits.

    As a result of these demos and code reviews, issues have been pinned down, test-cases written, and in some cases even fixes produced there and then. It was really a very useful aspect of the work week that we clearly need to explore further in the future.

    What’s next?

    So, what does that future look like for Mozilla and game development? Frankly, very bright indeed. There is so much interest and desire within Mozilla to further the Web as a platform for games. It’s clear that we have the knowledge and resources, we just need to work out how best to apply them.

    While we work on refining our position on games and exactly how we’re going to tackle them, there are a few things that I can say for sure about the future.

    The first is that we’re dedicated to helping developers learn how to use Web technologies, that’s the reason why MDN and this blog exist. We’ve started to work on ways to help developers learn how to use Web technologies to create games. We’re not quite there yet but it’s a priority.

    Stack of gamepads

    Aside from MDN, we are going to be reaching out to developers at game-related events. At some events we’ll be even bringing along hardware for you to play with and hack with JavaScript APIs, like gamepads for the Gamepad API. We’ll also be present at some of the larger events like GDC so we can keep up to speed with the game world and talk to developers about the issues they’re facing.

    You’ll also see games as a focus on this year’s Mozilla Festival. We’re refining the program but we’re working on tying in elements like the successful Game On competition that we ran last year.

    Documenting the game-related decisions and discussions within Mozilla is key, and it is something that we’re working on improving. For now the best place to keep up to date is this blog, the #games channel on irc.mozilla.org, and the HTML5 games page on the Mozilla wiki.

    With recent announcements it’s clear that Boot to Gecko (B2G) and games will be an area that Mozilla and the developer community will explore in the near future. It will be interesting to see what people do with a mobile platform as open and hackable as B2G.

    I could go on but it’s safe to say that there is a whole bunch of stuff happening around games at Mozilla, it’s an exciting time indeed. No doubt we’ll have another work week in the coming months to take stock and take more steps in the right direction.