Mozilla

Found 438 results for “html5”

Sort by:

View:

  1. Time to get hacking – Introducing Rec Room

    It’s no secret that the best frameworks and tools are extracted, not created out of thin air. Since launching Firefox OS, Mozilla has been approached by countless app developers and web developers with a simple question: “How do I make apps for Firefox OS?” The answer: “It’s the web; use existing web technologies.” was—and still is—a good answer.

    But if you don’t already have an existing toolchain as a web developer, I’ve been working on extracting something out of the way I’ve been creating web apps at Mozilla that you can use to write your next web app. From project creation to templating to deployment, Mozilla’s Rec Room will help you create awesome web apps in less time with more ease.

    Rec Room is a Node.js utility belt you can wear to build client side web apps. It includes:

    • Brick to add components like appbars and buttons to your UI.
    • Ember for your app’s controllers, models, and views.
    • Handlebars to write your app’s templates.
    • Grunt to run the tasks for your app, including building for production.
    • I18n.js to localize your app.
    • Mocha to test your app.
    • Stylus to write your CSS.
    • Yeoman to scaffold new code for your app’s models and templates.

    In this post I’ll walk through how to create a simple world clock web app with Rec Room, how to deploy it, and how you can try out Rec Room for yourself.

    Where Does Rec Room Come From?

    Much of Rec Room came from a recent rewrite of the HTML5 podcast app. I started working on this app well over a year ago, but its original version wasn’t as easy to work on; it had a lot of global state and a lot of by-hand data-binding. I liked the look of Ember for app development, but back when I started it didn’t quite feel mature enough. These days it’s much better, and I’ve tweaked it in Rec Room to work perfectly without a server.

    I tried to take the best from that system and extract it into a set of tools and documentation that anyone can use.

    Create your own Rec Room app

    Rec Room has just recently been extracted from my experiences with Podcasts; it hasn’t been tested by more than a handful of developers. That said: we’d love your help trying to build your own app for Firefox OS using these tools. They integrate well with tools you probably already know and use–like Node.js and Firefox’s own Web IDE.

    To get started, install Rec Room using Node.js:

    npm install -g recroom

    Clock App

    We’ll create a simple clock app with (minimal) time zone support for our example. The app will let you have a clock and compare it with a few time zones.

    The recroom binary is your entry point to all of the cool things Rec Room can do for you. First, create your app using recroom new world-clock. This creates the basic app structure. To see the basic app skeleton that Rec Room creates we can now enter that directory and run our app: cd world-clock and then type recroom run. The app will open in your default browser.

    First, we’ll add the current time to the main tab. Rec Room supports Ember’s MVC app structure, but also offers simple “pages” for a controller without a 1:1 relationship to a model. We’ll generate a new page that will show our actual clock:

    recroom generate page Clock

    We can edit its template by opening app/templates/clock.hbs. Let’s change clock.hbs to include the variable that will output our local time:

    <h2>Local Time: {{localTime}}</h2>

    That won’t do much yet, so let’s add that variable to our ClockController, in app/scripts/controllers/clock_controller.js:

    WorldClock.ClockController = Ember.ObjectController.extend({
        localTime: new Date().toLocaleTimeString()
    });

    You can see that any property inside the controller is accessible inside that controller’s template. We define the 1ocalTime property and it gets carried into our template context.

    Now our clock app will show the current local time when we navigate to http://localhost:9000/#clock. Of course, it just shows the time it was when the controller was initialized; there is no live updating of the time. We should update the time every second inside the controller:

    WorldClock.ClockController = Ember.ObjectController.extend({
        init: function() {
            // Update the time.
            this.updateTime();
     
        // Run other controller setup.
            this._super();
        },
     
        updateTime: function() {
            var _this = this;
     
            // Update the time every second.
            Ember.run.later(function() {
                _this.set('localTime', new Date().toLocaleTimeString());
                _this.updateTime();
            }, 1000);
        },
     
        localTime: new Date().toLocaleTimeString()
    });

    Now we can go to our clock URL and see our clock automatically updates every second. This is thanks to Ember’s data-binding between controllers and templates; if we change a value in a controller, model, or view that’s wired up to a template, the template will automatically change that data for us.

    Adding Timezones

    Next, we want to add a few timezones that the user can add to their own collection of timezones to compare against local time. This will help them schedule their meetings with friends in San Francisco, Buenos Aires, and London.

    We can create a timezone model (and accompanying controllers/routes/templates) with the same generate command, but this time we’ll generate a model:

    recroom generate model Timezone

    We want each timezone we’re to include in our app to have a name and an offset value, so we should add them as model attributes. We use Ember Data for this, inside app/scripts/models/timezone_model.js:

    WorldClock.Timezone = DS.Model.extend({
        name: DS.attr('string'),
        offset: DS.attr('number')
    });

    Next we’ll want a list of all timezones to offer the user. For this we’ll grab a copy of Moment Timezone. It’s an awesome JavaScript library for dealing with dates and times in JavaScript. We’ll install it with bower:

    bower install moment-timezone --save

    And then add it to our app inside app/index.html:

    <!-- build:js(app) scripts/components.js -->
    <!-- [Other script tags] -->
    <script src="bower_components/moment/moment.js"></script>
    <script src="bower_components/moment-timezone/builds/moment-timezone-with-data-2010-2020.js"></script>
    <!-- endbuild -->

    Adding that tag will automatically add moment-timezone-with-data-2010-2020.js to our built app. We’ll add a tab to the page that lets us edit our timezones, on a different screen than the clocks. To add a tab, we just need to open app/templates/application.hbs and add a tab. While we’re there, we’ll change the main tab from the useless {{#linkTo 'index'}} and point it to {{#linkTo 'clock'}}. The new application.hbs should look like this:

    <x-layout>
      <header>
        <x-appbar>
          <h1>{{t app.title}}</h1>
        </x-appbar>
      </header>
      <section>
        {{outlet}}
      </section>
      <footer>
        <x-tabbar>
          <x-tabbar-tab>
            {{#link-to 'clock'}}Clock{{/link-to}}
          </x-tabbar-tab>
          <x-tabbar-tab>
            {{#link-to 'timezones'}}Timezones{{/link-to}}
          </x-tabbar-tab>
        </x-tabbar>
      </footer>
    </x-layout>

    Side note: notice the root URL points to a useless welcome page? We probably want the default route to be our ClockController, so we can set the index route to redirect to it. Let’s do that now, in app/scripts/routes/application_route.js:

    WorldClock.ApplicationRoute = Ember.Route.extend({
        redirect: function() {
            this.transitionTo('clock');
        }
    });

    Interacting with Timezone models

    We’ll keep things simple for our example and allow users to select a timezone from a <select> tag and add it with a button. It will show up in their list of timezones, and they can delete it if they want from there. The clock tab will show all times. First, we’ll add our timezone data from Moment.js into our TimezonesController in app/scripts/controllers/timezones_controller.js. We’re also going to implement two actions: “add” and “remove”. These will be used in our template:

    WorldClock.TimezonesController = Ember.ObjectController.extend({
        init: function() {
            var timezones = [];
     
            for (var i in moment.tz._zones) {
              timezones.push({
                  name: moment.tz._zones[i].name,
                  offset: moment.tz._zones[i].offset[0]
              });
          }
     
          this.set('timezones', timezones);
     
          this._super();
      },
     
      selectedTimezone: null,
     
      actions: {
          add: function() {
              var timezone = this.store.createRecord('timezone', {
                  name: this.get('selectedTimezone').name,
                  offset: this.get('selectedTimezone').offset
              });
     
              timezone.save();
          },
     
          remove: function(timezone) {
              timezone.destroyRecord();
          }
      }
    });

    So we create a list of all available timezones with offsets. Then we add methods that allow us to add or remove timezones from our offline data store. Next we modify the timezones template in app/templates/timezones.hbs to use the actions and variables we created. All we need to utilize these variables is the Ember SelectView and the {{action}} helper to call our add and remove methods:

    <h2>Add Timezone</h2>
     
    <p>{{view Ember.Select content=timezones selection=selectedTimezone
           optionValuePath='content.offset' optionLabelPath='content.name'}}</p>
     
    <p><button {{action add}}>Add Timezone</button></p>
     
    <h2>My Timezones</h2>
     
    <ul>
      {{#each model}}
        <li>{{name}} <button {{action remove this}}>Delete</button></li>
      {{/each}}
    </ul>

    Now we have a Timezones tab that allows us to add and remove Timezones we want to track. This data persists between app refreshes. The last thing we need to do is show these times relative to our local time in our clock tab. To do this we need to load all the Timezone models in the ClockRoute. They’re automatically loaded in the TimezonesRoute, but it’s easy to add them in the ClockRoute (in app/scripts/routes/clock_route.js):

    WorldClock.ClockRoute = Ember.Route.extend({
        model: function() {
            return this.get('store').find('timezone');
        }
    });

    Because of the way our Ember app is wired up, we load all our models in the route and they are sent to the controller once the data store has asynchonously loaded all of the models. The request to find('timezone') actually returns a Promise object, but Ember’s router handles the Promise resolving for us automatically so we don’t have to manage callbacks or Promises ourselves.

    Now we have access to all the user’s Timezones in the ClockController, so we can make times in each timezone the user has requested and show them in a list. First we’ll add each Timezone’s current time to our ClockController in app/scripts/controllers/clock_controller.js using Moment.js:

    WorldClock.ClockController = Ember.ObjectController.extend({
        updateTime: function() {
            var _this = this;
     
            // Update the time every second.
            Ember.run.later(function() {
                _this.set('localTime', moment().format('h:mm:ss a'));
     
                _this.get('model').forEach(function(model) {
                    model.set('time',
                              moment().tz(model.get('name')).format('h:mm:ss a'));
                });
     
                _this.updateTime();
            }, 1000);
        }.on('init'),
     
        localTime: moment().format('h:mm:ss a')
    });

    Our final app/templates/clock.hbs should look like this:

    <h2>Local Time: {{localTime}}</h2>
     
    <p>
      {{#each model}}
        <h3>{{name}}: {{time}}</h3>
      {{/each}}
    </p>

    And that’s it! Now we have an offline app that shows us time zones in various places, saves the data offline, and updates every second without us having to do much work!

    Command Line Tools

    The old Podcasts app used a (rather awful) Makefile. It wasn’t very useful, and I don’t think it ran on Windows without some serious effort. The new build system uses Node so it runs comfortably on Windows, Mac, and Linux. Commands are proxied via the recroom binary, also written in Node, so you don’t have to worry about the underlying system if you don’t need to modify build steps. recroom new my-app creates a new app; recroom serve serves up your new app, and recroom generate model Podcast creates a new model for you.

    To build your app, you just need to run recroom build and a version with minified CSS, JS, and even HTML will be created for you in the dist/ folder. This version is ready to be packaged into a packaged app or uploaded to a server as a hosted app. You can even run recroom deploy to deploy directory to your git repository’s GitHub pages branch, if applicable.

    See the app in action!

    This entire sample app is available at worldclock.tofumatt.com and the source code is available on GitHub.

    Try Using Rec Room for Your Next Web App

    You can try out Rec Room on Github. Right now some docs and tools are still being abstracted and built, but you can start building apps today using it and filing bugs for missing features. We’d really love it if you could give it a try and let us know what’s missing. Together we can build a cohesive and polished solution to the all-too-common question: “How do I build a web app?”

  2. Building the Firefox browser for Firefox OS

    As soon as the Boot to Gecko (B2G) project was announced in July 2011 I knew it something I wanted to contribute to. I’d already been working on the idea of a browser based OS for a while but it seemed Mozilla had the people, the technology and the influence to build something truly disruptive.

    At the time Mozilla weren’t actively recruiting people to work on B2G, the team still only consisted of the four co-founders and the project was little more than an empty GitHub repository. But I got in touch the day after the announcement and after conversations with Chris, Andreas and Mike over Skype and a brief visit to Silicon Valley, I somehow managed to convince them to take me on (initially as a contractor) so I could work on the project full time.

    A Web Browser Built from Web Technologies

    On my first day Chris Jones told me “The next, highest-priority project is a very basic web browser, just a URL bar and back button basically.”

    Chris and his bitesize browser, Taipei, December 2011

    The team was creating a prototype smartphone user interface codenamed “Gaia”, built entirely with web technologies. Partly to prove it could be done, but partly to find the holes in the web platform that made it difficult and fill those holes with new Web APIs. I was asked to work on the first prototypes of a browser app, a camera app and a gallery app to help find some of those holes.

    You might wonder why a browser-based OS needs a browser app at all, but the thinking for this prototype was that if other smartphone platforms had a browser app, then B2G would need one too.

    The user interface of the desktop version of Firefox is written in highly privileged “chrome” code using the XUL markup language. On B2G it would need to be written in “content” using nothing but HTML, CSS and JavaScript, just like all the other apps. That would present some interesting challenges.

    In the beginning, there was an <iframe>

    It all started with a humble iframe, a text input for the URL bar and a go button, in fact you can see the first commit here. When you clicked the go button, it set the src attribute of the iframe to the contents of the text input, which caused the iframe to load the web page at that URL.

    First commit, November 2011

    The first problem with trying to build a web browser using an iframe is that the same-origin policy in JavaScript prevents you accessing just about any information about what’s going on inside it if the content comes from a different origin than the browser itself. In particular, it’s not possible to access the contentWindow property and all of the information that gives access to. This policy exists for good reasons so in order to build a fully functional web browser we would have to figure out a way for a privileged web app to safely poke holes in that cross-origin boundary to get just enough information to do its job, but without creating serious security vulnerabilities or compromising the user’s privacy.

    Another problem we came across quite quickly was that many web authors will go to great lengths to prevent their web site being loaded inside an iframe in order to prevent phishing attacks. A web server can send an X-Frame-Options HTTP response header instructing a user agent to simply not render the content, and there are also a variety of techniques for “framebusting” where a web site will actively try to break out of an iframe and load itself in the parent frame instead.

    It was quickly obvious that we weren’t going to get very far building a web browser using web technologies without evolving the web technologies themselves.

    The Browser API

    I met Justin Lebar at the first B2G work week in Taipei in December 2011. He was tasked with modifying Gecko to make the browser app on Boot to Gecko possible. To me Gecko was (and largely still is) a giant black box of magic spells which take the code I write and turn it into dancing images on the screen. I needed a wizard who had a grasp on some of these spells, including a particularly strong spell called Docshell which only the most practised of wizards dare peer into.

    Justin at the first B2G Work Week in Taipei, December 2011

    When I told Justin what I needed he made the kinds of sounds a mechanic makes when you take your car in for what you think is a simple problem but turns out costing the price of a new car. Justin had a better idea than I did as to what was needed, but I don’t think either of us realised the full scale of the task at hand.

    With the adding of a simple boolean “mozbrowser” attribute to the HTML iframe element in Gecko, the Browser API was born. I tried adding features to the browser app and every time I found something that wasn’t possible with current web technologies, I went back to Justin to get him to cast a new magic spell.

    There were easier approaches we could have taken to build the browser app. We could have added a mechanism to allow the browser to inject scripts into the iframe and communicate freely with the content inside, but we wanted to provide a safe API which anyone could use to build their own browser app and this approach would be too risky. So instead we built an explicit privileged API into the DOM to create a new class of iframe which could one day become a new standard HTML tag.

    Keeping the Web Contained

    The first thing we did was to try to trick web pages loaded inside an iframe into thinking they were not in fact inside an iframe. At first we had a crude solution which just ignored X-Frame-Options headers for iframes in whitelisted domains that had the mozbrowser attribute. That’s when we discovered that some web sites are quite clever at busting out of iframes. In the end we had to take other measures like making sure window.top pointed at the iframe rather than its parent so a web site couldn’t detect that it had a parent, and eventually also run every browser tab in its own system process to completely isolate them from each other.

    Once we had the animal that is the web contained, we needed to poke a few air holes to let it breathe. There’s some information we need to let out of the iframe in the form of events: when the location, title or icon of a web page changes (locationchange, titlechange and iconchange); when a page starts and finishes loading (loadstart, loadend) and when the security characteristics of the currently loaded page changes (securitychange). This all allows us to keep the address bar and title bar up to date and show a progress indicator.

    The browser app needs to be able to navigate the iframe by telling it to goBack(), goForward(), stop() and reload(). We also need to be able to explicitly ask for information like characteristics of the session history (getCanGoBack(), getCanGoForward()) to determine which navigation buttons to display.

    With these basics in place it was possible to build a simple functional browser app.

    The Prototype

    The Gaia project’s first UX designer was Josh Carpenter. At an intensive work week in Paris the week before Mobile World Congress in February 2012, Josh created UI mockups for all the basic features of a smartphone, including a simple browser, and we built a prototype to those designs.

    Josh and me plotting over a beer in Paris.

     

    The prototype browser app could navigate web content, keep it contained and display basic information about the content being viewed. This would be the version demonstrated at MWC in Barcelona that year.

    Simple browser demo for Mobile World Congress, February 2012

    Building a Team

    At a work week in Qualcomm’s offices in San Diego in May 2012 I was able to give a demo of a slightly more advanced basic browser web app running inside Firefox on the desktop. But it was still very basic. We needed a team to start building something good enough that we could ship it on real devices.

    “Browser Inception”, San Diego May 2012

    San Diego was also where I first met Dale Harvey, a brave Scotsman who came on board to help with Gaia. His first port of call was to help out with the browser app.

    Dale Getting on Board in San Diego, May 2012

    One of the first things Dale worked on was creating multiple tabs in the browser and even adding a screenshotting spell to the Browser API to show thumbnails of browser tabs (I told you he was brave).

    By this time we had also started to borrow Larissa Co, a brilliant designer from the Firefox team, to work on the interaction design and Patryk Adamczyk, formerly of RIM, to work on the visual design for the browser on B2G. That was when it started to look more like a Firefox browser.

    Early UI Mockup, July 2012

    Things that Pop Up

    Web pages like to make things pop up. For a start they like to alert(), prompt() or confirm() things with you. Sometimes they like to open() a new browser window (and close() them again), open a link in a _blank window, ask you for a password, ask for your permission to do something, ask you to select an option from a menu, open a context menu or confirm re-sending the contents of a form.

    An alert(), version 1.0

    All of this required new events in the Browser API, which meant more spells for Justin to cast.

    Scroll, Pan and Zoom

    Moving around web pages on web devices works a little differently from on the desktop. Rather than scroll bars or a scroll wheel on a mouse it uses touch input and a system called Asynchronous Pan and Zoom to allow the user to pan around a web page by dragging it and scrolling it using “kinetic scrolling” which feels like it has some physics to it.

    The first implementation of kinetic scrolling was written in JavaScript by Frenchman and Gaia leader Vivien Nicolas, specifically for Gaia, but it would later be written in a cross-platform way in Gecko to unify the code used on B2G and Android.

    One of the trickier interactions to get right was that we wanted the address bar to hide as you scrolled down the page in order to make more room for content, then show again when you scroll back to the top of the page.

    This required adding asyncscroll events which tapped directly into the Asynchronous Pan and Zoom code so that the browser knew not only when the user directly manipulated the page, but how much it scrolled based on physics, asynchronously from the user’s interaction.

    Storing Stuff

    One of the most loved features of Firefox is the “Awesomebar”, a combined address bar, search bar (and on mobile, title bar) which lets you quickly get to the content you’re looking for. You type a few characters and immediately start to see matching web pages from your browsing history, ranked by a “frecency” algorithm.

    On the desktop and on Android all of this data is stored in the “Places” database as part of privileged “chrome” code. In order to implement this feature in B2G we would need to use the local storage capabilities of the web, and for that we chose IndexedDB. We built a Places database in IndexedDB which would store all of the “places” a user visits on the web including their URL, title and icon, and store all the times the user visited that page. It would also be used to store the users bookmarks and rank top sites by “frecency”.

    Awesomebar, version 1.0

    Clearing Stuff

    As you browse around the web Gecko also stores a bunch of data about the places you’ve been. That can be cookies, offline pages, localStorage, IndexedDB databases and all sorts of other bits of data. Firefox browsers provide a way for you to clear all of this data, so methods needed to be added to the Browser API to allow this data to be cleared from the browser settings in B2G.

    Browser settings, version 1.0

    Handling Crashes

    Sometimes web pages crash the browser. In B2G every web app and every browser tab runs in its own system process so that should the worst happen, it will only cause that one window/tab to crash. In fact, due to the memory constraints of the low-end smartphones B2G would initially target, sometimes the system will intentionally kill a background app or browser tab in order to conserve memory. The browser app needs to be informed when this happens and needs to be able to recover seamlessly so that in most cases the user doesn’t even realise a process was killed. Events were added to the Browser API for this purpose.

    Crashed tab, version 1.0

    Talking to Other Apps

    Common use cases of a mobile browser are for the user to want to share a URL using another app like a social networking tool, or for another app to want to view a URL using the browser.

    B2G implemented Web Activities for this purpose, to add a capability to the web for apps to interact with each other, but in an app-agnostic way. So for example the user can click on a share button in the browser app and B2G will fire a “share URL” Web Activity which can then be handled by any installed app which has registered to handle that type of Web Activity.

    Share Web Activity, version 1.2

    Working Offline

    Despite the fact that B2G and Gaia are built on the web, it is a requirement that all of the built-in Gaia apps should be able to function offline, when an Internet connection is unavailable or patchy, so that the user can still make phone calls, take photos and listen to music etc.. At first we started to use AppCache for this purpose, which was the web’s first attempt at making web apps work offline. Unfortunately we soon ran into many of the common problems and limitations of that technology and found it didn’t fulfill all of our requirements.

    In order to ship version 1.0 of B2G on time, we were forced to implement “packaged apps” to fulfill all of the offline and security requirements for built-in Gaia apps. Packaged apps solved our problems but they are not truly web apps because they don’t have a real URL on the Internet, and attempts to standardise them didn’t get much traction. Packaged apps were intended very much as a temporary solution and we are working hard at adding new capabilities like ServiceWorkers, standardised hosted packages and manifests to the web so that eventually proprietary packaged apps won’t be necessary for a full offline experience.

    Offline, version 1.4

    Spit and Polish

    Finally we applied a good deal of spit and polish to the browser app UI to make it clean and fluid to use, making full use of hardware-accelerated CSS animations, and a sprinkling of Firefoxy interaction and visual design to make the youngest member of the Firefox browser family feel consistent with its brothers and sisters on other platforms.

    Shipping 1.0

    At an epic work week in Berlin in January 2013 hosted by Deutsche Telekom the whole B2G team, including engineers from multiple competing mobile networks and device manufacturers, got together with the common cause of shipping B2G 1.0, in time to demo at Mobile World Congress in Barcelona in February. The team sprinted towards this goal by fixing an incredible 200 bugs in one week.

    Version 1.0 Team, Berlin Work Week, January 2013

    In the last few minutes of the week Andreas Gal excitedly declared “Zarro Gaia Boogs”, signifying version 1.0 of Gaia was complete, with the rest of B2G to shortly follow over the weekend. Within around 18 months a dedicated team spanning multiple organisations had come together working entirely in the open to turn an empty GitHub repository into a fully functioning mobile operating system which would later ship on real devices as Firefox OS 1.0.1.

    Zarro Gaia Boogs, January 2013

    Browser app v1.0

    So having attended Mobile World Congress 2012 with a prototype and a promise to deliver commercial devices into the market, we were able to return in 2013 having delivered on that promise by fully launching the “Firefox OS” brand with multiple devices on multiple mobile networks with a launch that really stole the show at the biggest mobile conference in the world. Firefox OS had arrived.

    Mobile World Congress, Barcelona, February 2013

    1.x

    Firefox OS 1.1 quickly followed and by the time we started working on version 1.2 the project had grown significantly. We re-organised into autonomous agile teams focused on product areas, the browser app being one. That meant we now had a dedicated team with designers, engineers, a test engineer, a product manager and a project manager.

    The browser team, London work week, July 2013

    Firefox OS moved to a rapid release “train model” of development like Firefox, where a new version is delivered every 12 weeks. We quickly added new features and worked on improving performance to get the best out of the low end hardware we were shipping on in emerging markets.

    Browser app v1.4

    “Haida”

    Version 1.0 of Firefox OS was very much about proving that we could build what already exists on other smartphones, but entirely using open web technologies. That included a browser app.

    Once we’d proved that was possible and put real devices on shelves in the market it was time to figure out what would differentiate Firefox OS as a product going forward. We wanted to build something that doesn’t just imitate what’s already been done, but which plays to the unique strengths of the web to build something that’s true to Mozilla’s DNA, is the best way to experience the web, and is the platform that HTML5 deserves.

    Below is a mockup I created right back towards the start of the project at the end of 2011, before we even had a UX team. I mentioned earlier that the Awesomebar is a core part of the Firefox experience in Firefox browsers. My proposal back then was to build a system-wide Awesomebar which could search the whole device, including your apps and their contents, and be accessible from anywhere in the OS.

    Very early mockup of a system-wide Awesomebar, December 2011

    At the time, this was considered a little too radical for version 1.0 and our focus really needed to be on innovating in the web technology needed to build a mobile OS, not necessarily the UX. We would instead take a more conservative approach to the user interface design and build a browser app a lot like the one we’d built for Android.

    In practice that meant that we in fact built two browsers in Firefox OS. One was the browser app which managed the world of “web sites” and the other was the window manager in the system app which managed the world of “web apps” .

    In reality on the web there isn’t so much of a distinction between web apps and web sites – each exists on a long continuum of user experience with a very blurry boundary in the middle.

    In March 2013, with Firefox OS 1.0 out of the door, Josh Carpenter put me in touch with Gordon Brander, a member of the UX team who had been thinking along the same lines as me. In fact Gordon being as much of an engineer as he is a designer, had gone as far as to write a basic prototype in JavaScript.

    Gordon’s Rocketbar Prototype, March 2013

    Gordon and I started to meet weekly to discuss the concept he had by then codenamed “Rocketbar”, but it was a bit of a side project with a few interested people.

    In April 2013 the UX team had a summit in London where they got together to discuss future directions for the user experience of Firefox OS. I was lucky enough to be invited along to not only observe but participate in this process, Josh being keen to maintain a close collaboration between Design and Engineering.

    We brainstormed around what was unique about the experience of the web and how we might create a unique user experience which played to those strengths. A big focus was on “flow”, the way that we can meander through the web by following hyperlinks. The web isn’t a world of monolithic apps with clear boundaries between them, it is an experience of surfing from one web site to another, flowing through content.

    Brainstorming session, London, April 2013

    In the coming weeks the UX team would create some early designs for a concept (eventually codenamed “Haida”) which would blur the lines between web apps and web sites and create a unique user experience which flows like the web does. This would eventually include not only the “Rocketbar”, which would be accessible across the whole OS and seamlessly adapt to different types of web content, but also “sheets”, which would split single page web apps into multiple pages which you could swipe through with intuitive edge gestures. It would also eventually include a content model based around live apps which you can surf to, use, and then bookmark if you choose to, rather than monolithic apps which you have to install from a central app store before you can use them.

    In June 2013 a small group of designers and engineers met in Paris to develop a throwaway prototype of Haida, to rapidly iterate on some of the more radical concepts and put them through user testing.

    Haida Prototyping, Paris, June 2013

    Josh and Gordon working in a highly co-ordinated fashion, Paris, June 2013

    Wizards at work, Paris, June 2013

    2.x and the Future

    Fast forward to the present and the browser team has been merged into the “Systems Front End” team. The results of the Haida prototyping and user testing are slowly starting to make their way into the main Firefox OS product. It won’t happen all at once, but it will happen in small pieces as we iterate and learn.

    In version 2.0 of Firefox OS the homescreen search feature from 1.x will be replaced with a new search experience developed in conjunction with a new homescreen, implemented by Kevin Grandon, which will lay the foundations for “Rocketbar”. In version 2.1 our intention is to completely merge the browser app into the system app so that browser tabs become “sheets” alongside apps in the task manager and the “Rocketbar” is accessible from anywhere in the OS. The Rocketbar will adapt to different types of web content and shrink down into the status bar when not in use. Edge gestures will allow you to swipe between web apps and browser windows and eventually apps will be able to spawn multiple sheets.

    UI Mockups of Rocketbar in expanded and collapsed state, July 2014

    In parallel we see the evolution of web standards around manifests, packages and webviews and ongoing discussions around what defines the scope of an “app”.

    Rounding up

    Version 1.x of Firefox OS was built with web technologies but still has quite a similar user experience to other mobile platforms when it comes to installing and using apps, and browsing the web. Going forward I think you can expect to see the DNA of the web come through into the user interface with a unified experience which breaks down the barriers between web apps and web sites, allowing you to freely flow between the two.

    Firefox OS is an open source project developed completely in the open. If you’re interested in contributing to Gaia, take a look at the “Developing Gaia” page on MDN. If you’re interested in creating your own HTML5 app to run on Firefox OS take a look at the “App Center“.

  3. Building applications for Firefox OS using AngularJS

    When you start developing for Firefox OS you might be underwhelmed by the tools that are provided. There is no standard UI toolkit, or a JavaScript framework that all apps build on. This is not a situation that’s inherently bad because in essence Firefox OS is the web; and thus gives you complete freedom in the toolchain you use. This gives us the advantage to use any new technology that pops up also on Firefox OS. The downside is that you’ll miss out on things you might be used to on Android or iOS, like built-in templates; view transitions and UI components.

    In Telenor Digital we decided to build a ready-to-go application framework that deals with these shortcomings, built on top of AngularJS, the MVW framework by Google. The template is the result of iterating over our internal applications that we built for Firefox OS, and addresses the following things:

    1. Built on top of AngularJS, that has provides data binding; templating; routes; and code structure
    2. Built in set of UI components
      and transitions, in the style of Firefox OS
    3. Ability to publish apps as a mobile web page (hosted app),
      or as a packaged app for the Firefox OS marketplace
    4. Offline first approach. Every app built on top of the template works offline,
      also when hosted on your own web server.
    5. A build system to create release builds with one command,
      that does minification and template caching for optimal performance

    Let’s look at how the demo application looks like. It’s a standard CRUD app that shows a list-detail pattern: http://janjongboom.com/ffos-list-detail/. You can click on items to go to the detail view, you can edit items, or add new items. The ‘+’ button is an install button (only visible in Firefox) and allows you to add the app to your phone (Android / FxOS).

    Getting the code

    To start building, do this:

    • git clone git@github.com:comoyo/ffos-list-detail.git
    • npm install
    • ./node_modules/bower/bin/bower install
    • Now you can open www/index.html in any browser, or use the app manager and add the
      www folder as a packaged app.

    Structure

    The application lives in the www/ folder, and is made up of the following subfolders:

    • components/, third party libraries, loaded through bower
    • css/, style sheets. List all styles used by your app in
      css/main.css.
      They will be combined into one big stylesheet, for optimal performance.
    • img/, holds the icons for the app in three formats.
    • js/, our code
      • controllers/, the code that binds data to our UI
      • lib/, external libraries that are not in bower
      • services/, data providers, or code that is not bound to UI
      • app.js, starting point of the application, contains global configuration like routes
      • main.js, bootstrap file based on RequireJS.
        Lists all the JavaScript files we use. When you create a new JS file, add it here.
    • views/, view templates
    • index.html, bootstrap file where we load the application. You probably never will touch this.
    • manifest.appcache, AppCache file.
      You’ll need to list all the images & other resources (other than CSS/JS) that your app needs here,
      to enable offline for hosted applications.
    • manifest.webapp, Firefox OS App manifest file.

    You don’t need any build chain set up during development, you can just edit files in www, and refresh index.html at will. That’s the power of the web :-) Of course if you’re developing in the app manager, press UPDATE to refresh the app.

    Now let’s add some new functionality to this application, so we can see how developing new features works in practice.

    Adding a new button

    Let’s say that we want to add a credits screen that shows who built the application. First thing we need to do is add a button somewhere. In our case let’s put it on the home screen of the app. The code of the view is in www/views/list.html

    The components that you see come from the Firefox OS Building Blocks, which are the same blocks that are used to build Firefox OS itself. Let’s add a new button at the bottom of the screen (below the </ul> and the </section>:

    <a class="recommend" role="button" ng-tap="go('/credits', 'popup')">Credits</a>

    Important here is the ng-tap attribute. When we tap this item we go to /credits URL, with animation popup. There are four built in animations: forward, backward, popup and popdown; but you can create your own using simple CSS.

    Now when we look at this it doesn’t look like a button yet, because we didn’t tell that we needed the button building block. Go to css/main.css and add the following line to make it look nice:

    @import url("../components/building-blocks/style/buttons.css");

    All this is always documented on the page on the Building Blocks website.

    Hooking it up

    When we click on the button nothing happens though (well, we get redirected back to the list view), and that’s because we don’t listen on the /credits URL yet. To fix that we need to create a route handler (like in any MV* server side framework as well). Open the list of routes in js/app.js, and add a handler for the credits URL (before the otherwise handler):

    .when('/credits', {
      templateUrl: 'views/credits.html',
      controller: 'CreditsCtrl'
    })

    Here we tell which controller we want to consult (with JS code), and which view (with HTML) belongs to that. Let’s create the view first. Add a new file called credits.html in the views folder.

    <section role="region">
      <!-- Header building block http://buildingfirefoxos.com/building-blocks/headers.html -->
      <header>
        <!-- here we handle the back click and we do a popdown animation -->
        <a ng-tap="go('/', 'popdown')"><span class="icon icon-back">back</span></a>
        <h1>Credits</h1>
      </header>
    </section>
    <!-- The content of the view should have a 'view' class, and add the name of
         the view to easily style the view later -->
    <section class="view credits">
      This application is made by {{ name }}. <!-- AngularJS does data binding for us -->
    </section>

    To style this view we can add some content in css/app.css, f.e. add some padding and make the text bigger:

    .view.credits {
      padding: 1.5rem;
      font-size: 2rem;
    }

    Now write a simple controller to fill the content of {{ name }}, using standard AngularJS data binding. Add a new file called credits.js in www/js/controllers:

    /* We use RequireJS AMD style modules to get a reference to the app object */
    define(['app'], function(app) {
      /* Tell that we're defining a controller with name
        CreditsCtrl, and with dependencies on $scope, we specify this as string
        to not break when minifying
      */
      app.controller('CreditsCtrl', ['$scope',
        /* Implementation. AngularJS does dependency injection to fill the $scope var */
        function CreditsCtrl($scope) {
          /* Data binding to the view */
          $scope.name = 'Your name';
        }
      ]);
    });

    Last thing is to tell RequireJS that we have a new JS file that needs to be included in our builds, by editing js/main.js and adding a line above 'js/controllers/edit.js':

    'js/controllers/credits.js',

    When we now click the button in the app, everything works as expected. The view pops in, we have data, and we can dismiss by clicking the back button. What’s also great is that when you send the URL to someone else (f.e. http://your/url/index.html#/credits) they will go to the same view by default. That’s because we do proper state management through URLs by default.

    Talking to a third party data source

    The app currently only talks static data, so we want to hook it up to a real data source. In our case the project list should come from GitHub’s page with projects by mozilla-b2g. They have an API at: https://api.github.com/users/mozilla-b2g/repos.

    AngularJS has an idea of services, that abstract data away from your controller. For this app we have a database service that currently returns in-mem data. We can modify the service to talk to a web service instead. Clear out www/js/services/database.js and replace the content with:

    /*global define */
    "use strict";
    define(['app'], function(app) {
      /* Add a new factory called database, with a dependency on http */
      app.factory('database', ['http', function(http) {
        var getItems = function() {
          /* getItems makes a HTTP get call to github */
          return http.get('https://api.github.com/users/mozilla-b2g/repos', {
            // this is the cache configuration, we want to always cache requests
            // because it gives better UX. Plus when there is no internet, we can
            // get the data from cache and not break for the user...
            idbCache: {
              cacheKey: 'api.index',
              // expiration time in ms. from now (this is 5 minutes)
              // This is only obeyed if there is an internet connection!
              expiresInMs: 5 * 60 * 1000
            }
          }).then(function(res) {
            // Format it, sort it and map it to have the same format as our previous in mem dataset
            return res.data.sort(function(a, b) {
              return a.stargazers_count < b.stargazers_count;
            }).map(function(item) {
              return {
                title: item.name,
                description: item.description,
                id: item.name
              };
            });
          });
        };
     
        // Similar story but now for just one item
        var getItemById = function(id) {
          return http.get('https://api.github.com/repos/mozilla-b2g/device-flatfish', {
            idbCache: {
              cacheKey: 'api.detail.' + id,
              expiresInMs: 10 * 60 * 1000
            }
          }).then(function(res) {
            var repo = res.data;
            return {
              title: repo.name,
              description: repo.description,
              id: repo.name,
              date: new Date((repo.pushed_at || "").replace(/-/g,"/").replace(/[TZ]/g," "))
            };
          });
        };
     
        return {
          getItems: getItems,
          getItemById: getItemById
        };
      }]);
    });

    This API is now asynchronous though, but that doesn’t matter for Angular. If you data-bind to a promise, Angular will wait until the promise resolves until data binding happens.

    The beauty here is now that even when there is no Internet connection, the data will still load (as long as it was loaded at least once), and the data is auto-cached. No need for the controller to worry about that.

    Publishing the app

    These were two ways we quickly added some functionality to this application. First, adding a new button and a new view; and second, showing data binding and offline caching of server data. Please note that this application template can be used for much more than just list->detail applications, you’ve got the whole power of AngularJS at your hands!

    Now when we want to share this application with the rest of the world, we can go two ways:

    • Create a hosted application. This is an app that lives on your own server, like any mobile website. Hosted apps can still be published on the marketplace, and will work offline, but cannot use all the APIs in Firefox OS due to security limitations.
    • Create a packaged application. This is a ZIP file, similar to APK files on Android, that contain all the assets of your app, and are distributed through the marketplace.

    Both of these applications can be generated using our build script. The script will create a new folder dist/ that lists all the files the app needs. If you want to publish the app to your own server, just copy over the contents of the folder. If you want to publish the app as a packaged app, ZIP up the content and publish to the marketplace.

    To build, run:

    • Packaged: node build.js
    • Hosted: node build.js appcache

    Happy coding!

  4. Resources for HTML5 game developers

    Today we released Firefox 31 and it offers a couple of new features that help HTML5 game developers to code and debug sophisticated games. In addition Mozilla blogged about the first commercial games leveraging asm.js, Dungeon Defenders Eternity and Cloud Raiders both of which were cross-compiled in to JavaScript using the Emscripten compiler. Games like these show that HTML5 is ready as a game platform.

    If you are interested in working with Emscripten you can get more information at the main Emscripten wiki or grab the code on the github page. Another good resource is the getting started with Emscripten tutorial on MDN. If you are wondering about the performance of asm.js, read asm.js performance improvements in the latest version of Firefox make games fly! for details.

    In this post we’ll introduce you to some of the resources built by Mozillians that allow you to code and debug HTML5 based games. This list is not exhaustive and we appreciate feedback on any valuable resources that would help in this arena. Don’t be shy and tell us about them in the comments.

    Where To Start

    When developing an HTML5 based game, you have a lot of choices to make. These range from what editor to use, if the game will use Canvas 2d, WebGL, SVG, or CSS up to which specific rendering frameworks and game engines to use. Most of these decisions will be based on the developer experience and the platforms the game will be published on. No article will answer all these questions but we wanted to put together a post that would help get you started down the path.

    One of the key resources available for game developers on MDN is the Games Zone. This section of MDN contains general game development articles, demos, external resources and examples. It also includes detailed descriptions of some of the APIs that a developer will need to be aware of when implementing an HMTL5 game, including sound management, networking, storage and graphics rendering. We are currently in the process of adding content and upgrading the zone. In the future we hope to have content and examples for most common scenarios, frameworks and tool chains.

    In the meantime here are a few posts and MDN articles that help game developers getting started.

    Tools

    As an HTML5 developer you will have no shortage of tools at your disposal. In the Mozilla community we have been hard at work expanding the features that Firefox Developer Tools provide. These include a full-featured JavaScript Debugger, Style Editor, Page Inspector, Scratchpad, Profiler, Network Monitor and Web Console.

    In addition to these, some notable tools have been updated or introduced recently and offer some great functionality for the game developer.

    Canvas Debugger

    With the current release of Firefox, we added a Canvas Debugger to the browser.
    s_canvasdebugger
    The Canvas Debugger allows you to trace through all canvas context calls that are used to generate a frame. Calls are color coded for specific calls for things like drawing elements or using a specific shader program. The Canvas Debugger is not only useful when developing a WebGL based game but can also be used when debugging a Canvas 2D based game. In the game below you can see in the animation strip as each image is drawn to the canvas. You can click any of these lines to get directly to the part of your JavaScript responsible for this action.
    s_captainrogers
    Two very common issues that have been reported when using the Canvas Debugger are with animations generated using setInterval instead of requestAnimationFrame and inspecting canvas elements in an iFrame.

    To get more information about the Canvas Debugger be sure to read over Introducing the Canvas Debugger in Firefox Developer Tools.

    Shader Editor

    When developing WebGL based games it is very useful to be able to test and alter shader programs while the application is running. Using the Shader Editor within the developer tools makes this possible. Vertex and Fragment Shader programs can be modified without the need to reload the page, or black boxed to see what effect this has on the resulting output.
    s_ShaderEditor

    For more information on the Shader Editor, be sure to see Live editing WebGL shaders with Firefox Developer Tools post and take a look at this MDN article which contains a couple of videos showing live editing.

    Web Audio Editor

    The current version of Firefox Aurora (32) – has a Web Audio Editor. The Editor displays a graphical representation of all the Audio Nodes and their connections in the current AudioContext. You can drill down to specific attributes of each node to inspect them.
    s_webaudioeditor

    The Web Audio API provides more robust and complex sound creation, manipulation and processing than what is available in the HTML5 Audio tag. When using the Web Audio API make sure to read over Writing Web Audio API code that works in every browser as it contains pertinent information about support for the various audio nodes.

    For more information on the Web Audio Editor be sure to read this Hacks article introducing the Web Editor and this MDN article.

    Network Monitor

    When developing an HTML5 based game network impact can be not only cumbersome but also costly if the user is on mobile device. Using the Network Monitor you can visually inspect all network request for location, time spent on the operation, and the type and size of the artifact.
    s_networkmon
    In addition you can use the Network Monitor to get a visual performance analysis of your app when cached versus non-cached.
    s_networkcache

    To get more information on the Network Monitor see the MDN page.

    Web IDE

    When starting your game one of your first choices will be which editor to use. And there are a lot of them (Sublime, Eclipse, Dreamweaver, vi, etc). In most cases a you already have a favorite. If you are interested in doing your development within the Browser you may want to have a look at the Web IDE that was recently released in Firefox Nightly.
    s_webide

    The Web IDE project provides not only a fully functional editor but also acts as a publishing agent to various local and remote platforms, debugger, template framework and application manager. In addition the framework supporting this project provides APIs that will allow other editors to use functionality provided in the tool. To get more details on the work that is being done in this area have a look at this post.

    In order to keep up-to-date with news on the Firefox Developer Tools, follow their article series on the Hacks blog. For more detailed information on new, stable developer tools features, check out their documentation on MDN.

    APIs

    The MDN Games Zone lists various APIs and articles that are useful for beginning game development.
    s_apis
    In addition to these resources you may be interested in looking over some additional posts that can be valuable for development.

    If your game is going to support multiplayer interaction using either WebRTC or WebSockets you may also be interested in looking at Together.js which provides collaborative features for web apps. To get an idea what is possible take a look at Introducing TogetherJS.

    Many games will require storage and IndexedDB can be used to handle these needs. For information on extending the capabilities of IndexedDB read over Breaking the Borders of IndexedDB. You may also be interested in localForage which provides browser agnostic support for simple storage. To get more details about this library read over this Hacks post.

    Game Optimization

    HTML5 games today offer a great deal of power to the game developer. That said many of these games are going to be played on a mobile device, which in comparison to your desktop will pale in performance. So if you plan on your game being a success across platforms it is important that you optimize your code. The Optimizing your JavaScript Game for Firefox OS post has a lot of great techniques to help you build a game that performs well on low-end mobile devices.

    Localization

    In order to reach the most users of your game you may want to consider offering it in different languages. As part of this developers should start with localization built into the game. We are doing a great deal of work around recruiting translators to help you translate your game. To get more information about this initiative see this post.

    Your Voice

    As Mozilla is about the community of developers and users, we want your help and your feedback. If you have recommendations for specific features that you would like to see in the products make sure to either get involved in discussion on irc.mozilla.org or through our mailing lists. You can also log bugs at bugzilla.mozilla.org. In addition we are also provide additional feedback channels for our DevTools and Open Web Apps.

  5. Adding captions and subtitles to HTML5 video

    This article is also available on MDN.

    With the introduction of the <video> and <audio> elements to HTML5, we finally have a native way to add video and audio to our websites. We also have a JavaScript API that allows us to interact with this media content in different ways, be it writing our own controls or simply seeing how long a video file is. As responsible web developers, we should also be constantly thinking about making our content more accessible, and this doesn’t stop with video and audio content. Making our content accessible to all is an important step, be it for someone who is hard of hearing or someone who doesn’t understand the language that the content is delivered in, inclusion can be paramount.

    Thankfully HTML5 also provides us with a native way of making our media content more accessible by adding subtitles and captions via the <track> element. Most major browsers support this natively to varying degrees, which the first part of this article shows, but it also provides a JavaScript API, with which we can access and use the text tracks (e.g. subtitles) that are available. This article also shows how this API can be used to detect what captions/subtitles have been added to a HTML5 video, and how that data can be used to build a selectable menu of available text tracks and ultimately provide a more consistent interface across the various browsers.

    In articles on MDN, we have looked at how to build a cross browser video player using the HTMLMediaElement and Window.fullScreen APIs, and also at how to style the player. This article will take the same player and show how to add captions and subtitles to it, using Web_Video_Text_Tracks_Format and the <track> element.

    Captioned video example

    In this article, we will refer to the Video player with captions example. This example uses an excerpt from the Sintel open movie, created by the Blender Foundation.

    Video player with stand controls such as play, stop, volume, and captions on and off. The video playing shows a scene of a man holding a spear-like weapon, and a caption reads "Esta hoja tiene pasado oscuro."

    Note: You can find the source on Github, and also view the example live.

    HTML5 and Video Captions

    Before diving into how to add captions to the video player, there are a number of things that we will first mention, which you should be aware of before we start.

    Captions versus subtitles

    Captions and subtitles are not the same thing: they have significantly different audiences, and convey different information, and it is recommended that you read up on the differences if you are not sure what they are. They are however implemented in the same way technically, so the material in this article will apply to both.

    For this article we will refer to the text tracks displayed as captions, as their content is aimed at hearing people who have difficulty understanding the language of the film, rather than deaf or hard-of-hearing people.

    The <track> element

    HTML5 allows us to specify captions for a video using the Web Video Text Tracks (WebVTT) format. The WebVTT specification is still being worked on, but major parts of it are stable so we can use it today.

    Video providers (such as the Blender Foundation) provide captions and subtitles in a text format with their videos, but they’re usually in the SubRip Text (SRT) format. These can be easily converted to WebVTT using an online converter such as srt2vtt.

    Modifications to the HTML and CSS

    This section summarises the modifications made to the previous article’s code in order to facilitate the addition of subtitles to the video. If you are not interested in thism and just want to get straight into the JavaScript and more relevant CSS, skip to the Caption implementation section.

    In this example we are using a different video, Sintel, as it actually has some speech in it and therefore is better for illustrating how captions work!

    HTML Markup

    As mentioned above, we need to make use of the new HTML5 <track> element to add our caption files to the HTML5 video. We actually have our captions in three different languages — English, German, and Spanish — so we will reference all three of the relevant VTT files by adding <track> elements inside our HTML5 <video> element:

    <video id="video" controls preload="metadata">
       <source src="video/sintel-short.mp4" type="video/mp4">
       <source src="video/sintel-short.webm" type="video/webm">
       <track label="English" kind="captions" srclang="en" src="captions/vtt/sintel-en.vtt" default>
       <track label="Deutsch" kind="captions" srclang="de" src="captions/vtt/sintel-de.vtt">
       <track label="Español" kind="captions" srclang="es" src="captions/vtt/sintel-es.vtt">
    </video>

    As you can see, each <track> element has the following attributes set:

    • kind is given a value of captions, indicating the type of content the files contain
    • label is given a value indicating which language that caption set is for for example English or Deutsch — these labels will appear in the user interface to allow the user to easily select which caption language they want to see.
    • src is assigned a valid URL pointing to the relevant WebVTT caption file in each case.
    • srclang indicates what language each captions files’ contents are in.
    • The default attribute is set on the English <track> element, indicating to the browser that this is the default caption file definition to use when captions have been turned on and the user has not made a specific selection.

    In addition to adding the <track> elements, we have also added a new button to control the captions menu that we will build. As a consequence, the video controls now look as follows:

    <div id="video-controls" class="controls" data-state="hidden">
       <button id="playpause" type="button" data-state="play">Play/Pause</button>
       <button id="stop" type="button" data-state="stop">Stop</button>
       <div class="progress">
          <progress id="progress" value="0" min="0">
             <span id="progress-bar"></span>
          </progress>
       </div>
       <button id="mute" type="button" data-state="mute">Mute/Unmute</button>
       <button id="volinc" type="button" data-state="volup">Vol+</button>
       <button id="voldec" type="button" data-state="voldown">Vol-</button>
       <button id="fs" type="button" data-state="go-fullscreen">Fullscreen</button>
       <button id="captions" type="button" data-state="captions">CC</button>
    </div>

    CSS Changes

    The video controls have undergone some minor changes in order to make space for the extra button, but these are relatively straightforward.

    No image is used for the captions button, so it is simply styled as:

    .controls button[data-state="captions"] {
        height:85%;
        text-indent:0;
        font-size:16px;
        font-size:1rem;
        font-weight:bold;
        color:#666;
        background:#000;
        border-radius:2px;
    }

    There are also other CSS changes that are specific to some extra JavaScript implementation, but these will be mentioned at the appropriate place below.

    Caption implementation

    A lot of what we do to access the video captions revolves around JavaScript. Similar to the video controls, if a browser supports HTML5 video captions, there will be a button provided within the native control set to access them. However, since we have defined our own video controls, this button is hidden, and we need to define our own.

    Browsers do vary as to what they support, so we will be attempting to bring a more unified UI to each browser where possible. There’s more on browser compatibility issues later on.

    Initial setup

    As with all the other buttons, one of the first things we need to do is store a handle to the captions’ button:

    var captions = document.getElementById('captions');

    We also initially turn off all captions, in case the browser turns any of them on by default:

    for (var i = 0; i &lt; video.textTracks.length; i++) {
       video.textTracks[i].mode = 'hidden';
    }

    The video.textTracks property contains an array of all the text tracks attached to the video. We loop through each one and set its mode to hidden.

    Note: The WebVTT API gives us access to all the text tracks that are defined for an HTML5 video using the <track> element.

    Building a caption menu

    Our aim is to use the captions button we added earlier to display a menu that allows users to choose which language they want the captions displayed in, or to turn them off entirely.

    We have added the button, but before we make it do anything, we need to build the menu that goes with it. This menu is built dynamically, so that languages can be added or removed later by simply editing the <track> elements within the video’s markup.

    All we need to do is to go through the video’s textTracks, reading their properties and building the menu up from there:

    var captionsMenu;
    if (video.textTracks) {
       var df = document.createDocumentFragment();
       var captionsMenu = df.appendChild(document.createElement('ul'));
       captionsMenu.className = 'captions-menu';
       captionsMenu.appendChild(createMenuItem('captions-off', '', 'Off'));
       for (var i = 0; i < video.textTracks.length; i++) {
          captionsMenu.appendChild(createMenuItem('captions-' + video.textTracks[i].language, video.textTracks[i].language,         video.textTracks[i].label));
       }
       videoContainer.appendChild(captionsMenu);
    }

    This code creates a documentFragment, which is used to hold an unordered list containing our captions menu. First of all an option is added to allow the user to switch all captions off, and then buttons are added for each text track, reading the language and label from each one.

    The creation of each list item and button is done by the createMenuItem() function, which is defined as follows:

    var captionMenuButtons = [];
    var createMenuItem = function(id, lang, label) {
       var listItem = document.createElement('li');
       var button = listItem.appendChild(document.createElement('button'));
       button.setAttribute('id', id);
       button.className = 'captions-button';
       if (lang.length > 0) button.setAttribute('lang', lang);
       button.value = label;
       button.setAttribute('data-state', 'inactive');
       button.appendChild(document.createTextNode(label));
       button.addEventListener('click', function(e) {
          // Set all buttons to inactive
          captionMenuButtons.map(function(v, i, a) {
             captionMenuButtons[i].setAttribute('data-state', 'inactive');
          });
          // Find the language to activate
          var lang = this.getAttribute('lang');
          for (var i = 0; i < video.textTracks.length; i++) {
             // For the 'captions-off' button, the first condition will never match so all will captions be turned off
             if (video.textTracks[i].language == lang) {
                video.textTracks[i].mode = 'showing';
                this.setAttribute('data-state', 'active');
             }
             else {
                video.textTracks[i].mode = 'hidden';
             }
          }
          captionsMenu.style.display = 'none';
       });
       captionMenuButtons.push(button);
       return listItem;
    }

    This function builds the required <li> and <button> elements, and returns them so they can be added to the captions menu list. It also sets up the required event listeners on the button to toggle the relevant caption set on or off. This is done by simply setting the required caption’s mode attribute to showing, and setting the others to hidden.

    Once the menu is built, it is then inserted into the DOM at the bottom of the videoContainer.

    Initially the menu is hidden by default, so an event listener needs to be added to our captions button to toggle it:

    captions.addEventListener('click', function(e) {
       if (captionsMenu) {
          captionsMenu.style.display = (captionsMenu.style.display == 'block' ? 'none' : 'block');
       }
    });

    Caption menu CSS

    We also added some rudimentary styling for the newly created captions menu:

    .captions-menu {
        display:none;
        position:absolute;
        bottom:14.8%;
        right:20px;
        background:#666;
        list-style-type:none;
        margin:0;
        padding:0;
        width:100px;
        padding:10px;
    }
     
    .captions-menu li {
        padding:0;
        text-align:center;
    }
     
    .captions-menu li button {
        border:none;
        background:#000;
        color:#fff;
        cursor:pointer;
        width:90%;
        padding:2px 5px;
        border-radius:2px;
    }

    Styling the displayed captions

    One of the less well known about and supported features of WebVTT is the ability to style the individual captions (something called text cues) via CSS Extensions.

    The ::cue pseudo-element is the key to targetting individual text track cues for styling, as it matches any defined cue. There are only a handful of CSS properties that can be applied to a text cue:

    For example, to change the text colour of the text track cues you can write:

    ::cue {
       color:#ccc;
    }

    If the WebVTT file uses voice spans, which allow cues to be defined as having a particular “voice”:

    0
    00:00:00.000 --> 00:00:12.000
    <v Test>[Test]</v>

    Then this specific ‘voice’ will be stylable like so:

    ::cue(v[voice='Test']) {
       color:#fff;
       background:#0095dd;
    }

    Note: Some of the styling of cues with ::cue currently works on Chrome, Opera, and Safari, but not yet on Firefox.

    Browser Compatibility

    Browser support for WebVTT and the <track> element is fairly good, although some browsers differ slightly in their implementation.

    Internet Explorer

    Since Internet Explorer 10+ captions are enabled by default, and the default controls contain a button and a menu that offers the same functionality as the menu we just built. The default attribute is also supported.

    Note: IE will completely ignore WebVTT files unless you setup the MIME type. This can easily be done by adding an .htaccess file to an appropriate directory that contains AddType text/vtt .vtt.

    Safari

    Safari 6.1+ has similar support to Internet Explorer 11, displaying a menu with the different available options, with the addition of an “Auto” option, which allows the browser to choose.

    Chrome and Opera

    These browsers have similar implementations again: captions are enabled by default and the default control set contains a ‘cc’ button that turns captions on and off. Chrome and Opera ignore the default attribute on the <track> element and will instead try to match the browser’s language to the caption’s language.

    Firefox

    Firefox’s implementation was completely broken due to a bug, leading to Mozilla turning off WebVTT support by default (you can turn it on via the media.webvtt.enabled flag.) However, this bug looks to have been fixed and WebVTT support re-enabled as of Gecko 31, so this will not be a problem for Firefox final release users for much longer (on Gecko 29 as of the time of this writing.) this has been fixed as of Firefox 31, and everything works as it should.

    Plugins

    If, after reading through this article you decide that you can’t be bothered to do all of this and want someone else to do it for you, there are plenty of plugins out there that offer caption and subtitle support that you can use.

    playr
    This small plugin implements subtitles, captions, and chapters as well as both WebVTT and SRT file formats.
    jwplayer
    This video player is very extensive and does a lot more than simply support video captions. It supports WebVTT, SRT, and DFXP file formats.
    MediaElement.js
    Another complete video player that also support video captions, albeit only in SRT format.
    LeanBack Player
    Yet another video player that supports WebVTT captions as well as providing other standard player functionality.
    SublimeVideo
    This player also supports captions through WebVTT and SRT files.
    Video.js
    Supports WebVTT video subtitles.

    Note: You can find an excellent list of HTML5 Video Players and their current state at HTML5 Video Player Comparison.

  6. Mozilla at conferences – June edition

    Welcome to a quick round-up of what Mozillians have been talking about at events in and around June.

    Campus Party Guadalajara

  7. Frédéric Harper spoke at Devoxx UK about “Getting the best out of your design with responsive web design
  8. Robert Nyman spoke at JSCamp Romania about “Five stages of development (slidesvideo)”
  9. David Baron spoke at CSS Day about “Efficient CSS Animations (slideshowall slides)
  10. Chris Heilmann visited Campus Party Mexico to deliver the keynote “The Future of the Open Web (video, slides)”. There were also various talks by local Mozillians.
  11. Nick Desaulniers spoke at HTML5DevConf about “Raw WebGL (video)” whilst Chris Heilmann delivered the “Write less, achieve meh (notes)” keynote.
  12. Soledad Penadés covered an inordinate amount of amazing things at Goto Amsterdam in her “Invest in the future: build for the web! (notes and screencast)” talk.
  13. Open Source Bridge in Portland had a whole bunch of Mozillians presenting (the site also has lots of notes of what happened where and when and is worth visiting):
  14. Dale Harvey attended Scotch on the Rocks and talked about “The offline Web (slidesvideo)” and took part in the panel discussion on “When is Enough Enough (video)
  15. Campus Party Guadalajara

    If you want to know where we will present, check the Where is Mozilla? page. If you are interested in getting a Mozilla presenter for your event, why not tell us about it?

  16. Introducing the Web Audio Editor in Firefox Developer Tools

    In Firefox 32, the Web Audio Editor joins the Shader Editor and Canvas Debugger in Firefox Developer Tools for debugging media-rich content on the web. When developing HTML5 games or fun synthesizers using web audio, the Web Audio Editor assists in visualizing and modifying all of the audio nodes within the web audio AudioContext.

    Visualizing the Audio Context

    When working with the Web Audio API‘s modular routing, it can be difficult to translate how all of the audio nodes are connected just by listening to the audio output. Often, it is challenging to debug our AudioContext just by listening to the output and looking at the imperative code that creates audio nodes. With the Web Audio Editor, all of the AudioNodes are rendered in a directed graph, illustrating the hierarchy and connections of all audio nodes. With the rendered graph, a developer can ensure that all of the nodes are connected in a way that they expect. This can be especially useful when the context becomes complex, with a network of nodes dedicated to manipulating audio and another for analyzing the data, and we’ve seen some pretty impressive uses of Web Audio resulting in such graphs!

    To enable the Web Audio Editor, open up the options in the Developer Tools, and check the “Web Audio Editor” option. Once enabled, open up the tool and reload the page so that all web audio activity can be monitored by the tool. When new audio nodes are created, or when nodes are connected and disconnected from one another, the graph will update with the latest representation of the context.

    Modifying AudioNode Properties

    Once the graph is rendered, individual audio nodes can be inspected. Clicking on an AudioNode in the graph opens up the audio node inspector where AudioParam‘s and specific properties on the node can be viewed and modified.

    Future Work

    This is just our first shippable release of the Web Audio Editor, and we are looking forward to making this tool more powerful for all of our audio developers.

    • Visual feedback for nodes that are playing, and time/frequency domain visualizations.
    • Ability to create, connect and disconnect audio nodes from the editor.
    • Tools for debugging onaudioprocess events and audio glitches.
    • Display additional AudioContext information and support multiple contexts.
    • Modify more than just primitives in the node inspector, like adding an AudioBuffer.

    We have many dream features and ideas that we’re excited about, and you can view all open bugs for the Web Audio Editor or submit new bugs. Be sure to check out the MDN documentation on the Web Audio Editor and we would also love feedback and thoughts at our UserVoice feedback channel and on Twitter @firefoxdevtools.

  17. Stack Overflow Dashboard – check engagement, metrics and more

    Recently I put together a little Mozilla tags on Stack Overflow dashboard to check the engagement and numbers for the tags we sponsor. I liked the idea and wanted to create a general purpose dashboard for Stack Overflow, and share the feature, code and thinking with you.

    Features

    Numbers and developer behavior are always interesting areas, especially to spot trends, common questions, related areas and more. As Stack Overflow is one of the de facto channels where developers ask questions to solve problems I find it very interesting to look at those numbers and see possible correlations.

    You can filter the data based on dates and time periods (different data is available through one or the other option), and the areas the dashboard offers information for are:

    • Questions:
      • # with activity
      • # of unanswered
      • Percentage of unanswered questions (Note that a question must have at least one upvoted answer to be considered answered)
      • List of unanswered questions
      • Frequently asked questions
    • Top answerers
    • Top askers
    • Related tags

    The approach

    My thinking was to use the Stack Exchange API and do simple requests for various tags end the engagement around them. I also wanted to make it easy for the user and autocomplete values for the tag criteria field. Given how many tags there are on Stack Overflow, though, to avoid massive overload I only get the 100 most popular tags and put them in a <datalist> element, connected to the <input> element where the user enters the tag to look for data for. This is being done directly on page load.

    General mindset

    The general mindset when building up the dashboard:

    1. Create a simple HTML form with tag, date period input
    2. Use a simple XMLHttpRequest to get the most popular tags for the <datalist> element
    3. For any request, use the basic XHR to get the data directly in JSON
    4. Depending on the requested data – multiple requests are needed to fill the dashboard – parse the returned JSON and present the results

    API methods being used

    The other API methods that are being called are:

    Request throttling

    Note that the limit for each IP number is 300 requests per 24 hours (unless you have an access_token, then the limit is 10,000).

    Code

    The code is available on GitHub and the idea has been to keep it as simple and free of dependencies as possible. It doesn’t use any JavaScript libraries as I see the use cases here, and where we are right now with HTML5, competent enough not to need that.

    Pre-populating the <datalist> element

    As an example, this is (trimmed-down) version of the code to get the most popular tags as JSON through the Stack Exchange API and polulate the <datalist>:

    function getPopularTags () {
        getItems("popularTags", "http://api.stackexchange.com/2.2/tags?pagesize=100&amp;order=desc&amp;sort=popular&amp;site=stackoverflow");
    }
     
    // Run automatically at page load to pre-populate the &lt;datalist&gt; element
    getPopularTags();
     
    function getItems(type, url) {
        var xhr = new XMLHttpRequest();
        xhr.onreadystatechange = function () {
            if (xhr.readyState === 4) {
                var response = xhr.response;
     
                if (response.error_message) {
                    // Show errors
                }
                else {
                    addResults(type);
                }
            }
        };
     
        xhr.open("GET", url, true);
        xhr.responseType = "json";
        xhr.send(null);
    };
     
    function addResults (type) {
        // Popular tags, for filling the &lt;datalist&gt; element
        if (type === "popularTags") {
            var popularTagsList = document.querySelector("#popular-tags"),
                popularTags = questions.popularTags.items,
                popularTagsResults = "";
            for (var i=0,l=popularTags.length, tag; i&lt;l; i++) {
                tag = popularTags[i];
                popularTagsResults += '&lt;option value="' + tag["name"] + '"&gt;';
            }
            popularTagsList.innerHTML = popularTagsResults;
        }
    }

    Give feedback & hack it

    I hope you find this interesting, and a good point to evaluate which areas to focus on and learn more! Also feel more than welcome to use the dashboard, check out the code and issue pull requests, suggest features and more!

  18. PlayCanvas Goes Open Source

    This is a guest post by Will Eastcott of the PlayCanvas engine. As outlined in What Mozilla Hacks is, we constantly cover interesting information about open source and the Open Web, both from external as well as Mozilla authors, so feel free to share with us!

    On March 22nd 2011, Mozilla released Firefox 4.0 which enabled WebGL by default. A month later, we formed PlayCanvas and started building a game engine unlike anything that had gone before. Fast forward three years, and WebGL is everywhere. Only this week, Apple announced support for WebGL in both OS X and iOS 8. So what better time pass on some more exciting news for you:

    The PlayCanvas Engine has been open sourced!

    MozBlog500

    Introducing the PlayCanvas Engine

    The PlayCanvas Engine is a JavaScript library engineered specifically for building video games. It implements all of the major components that you need to write high quality games:

    • Graphics: model loading, per-pixel lighting, shadow mapping, post effects
    • Physics: rigid body simulation, ray casting, joints, trigger volumes, vehicles
    • Animation: keyframing, skeletal blending, skinning
    • Audio engine: 2D and 3D audio sources
    • Input devices: mouse, keyboard, touch and gamepad support
    • Entity-component system: high level game object management

    We had a couple of goals in mind when we originally designed the engine.

    1. It had to be easy to work with.
    2. It had to be blazingly fast.

    Simple Yet Powerful

    As a developer, you want well documented and well architected APIs. But you also want to be able to understand what’s going on under the hood and to debug when things go wrong. For this, there’s no substitute for a carefully hand-crafted, unminified, open source codebase.

    Additionally, you need great graphics, physics and audio engines. But the PlayCanvas Engine takes things a step further. It exposes a game framework that implements an entity-component system, allowing you to build the objects in your games as if they were made of Lego-like blocks of functionality. So what does this look like? Let’s check out a simple example on CodePen: a cannonball smashing a wall:

    playcanvas_codepen

    As you can see from the Pen’s JS panel, in just over 100 lines of code, you can create, light, simulate and view interesting 3D scenes. Try forking the CodePen and change some values for yourself.

    Need For Speed

    To ensure we get great performance, we’ve built PlayCanvas as a hybrid of hand-written JavaScript and machine generated asm.js. The most performance critical portion of the codebase is the physics engine. This is implemented as a thin, hand-written layer that wraps Ammo.js, the Emscripten-generated JavaScript port of the open source physics engine Bullet. If you haven’t heard of Bullet before, it powers amazing AAA games like Red Dead Redemption and GTAV. So thanks to Mozilla’s pioneering work on Emscripten and asm.js, all of this power is also exposed via the PlayCanvas engine. Ammo.js executes at approximately 1.5x native code speed in recent builds of Firefox so if you think that complex physics simulation is just not practical with JavaScript, think again.

    But what about the non-asm.js parts of the codebase? Performance is clearly still super-important, especially for the graphics engine. The renderer is highly optimized to sort draw calls by material and eliminate redundant WebGL calls. It has also been carefully written to avoid making dynamic allocations to head off potential stalls due to garbage collection. So the code performs brilliantly but is also lightweight and human readable.

    Powering Awesome Projects

    The PlayCanvas Engine is already powering some great projects. By far and away, the biggest is the PlayCanvas web site: the world’s first cloud-hosted game development platform.

    For years, we’ve been frustrated with the limitations of current generation game engines. So shortly after starting work on the PlayCanvas Engine, we began designing a new breed of game development environment that would be:

    Accessible
    using any device with a web browser, plug in a URL and instantly access simple, intuitive yet powerful tools.
    Collaborative
    See what you teammates are working on in real-time or just sit back and watch a game as it’s built live before your eyes.
    Social
    Making games is easier with the help of others. Be part of an online community of developers like you.

    PlayCanvas ticks all of these boxes beautifully. But don’t take our word for it – head to https://playcanvas.com and discover a better way to make games.

    In fact, here’s a game we have built using these very tools. It’s called SWOOOP:

    PLAY NOW!

    It’s a great demonstration of what you can achieve with HTML5 and WebGL today. The game runs great in both mobile and desktop browsers, and you are free to deploy your PlayCanvas games to app stores as well. For Google Play and the iOS App Store, there are wrapping technologies available that can generate a native app of your game. Examples of these are Ludei’s CocoonJS and the open source Ejecta project. For Firefox OS, the process is a breeze since the OS treats HTML5 apps as first class citizens. PlayCanvas games will run out of the box.

    Want!

    So if you think this is sounding tasty, where should you go to get started? The entire engine sourcebase is now live on GitHub:

    https://github.com/playcanvas/engine

    Get cloning, starring and forking while it’s fresh!

    Stay in the Loop

    Lastly, I want to give you some useful links that should help you stay informed and find help whenever you need it.

    We’re super excited to see what the open source community will do with the PlayCanvas Engine. So get creative and be sure to let us know about your projects.

    Toodle pip!

  19. Build Your Next App With a Flame

    Earlier this week, we introduced Flame, the Firefox OS reference device for developers, testers and reviewers from T2Mobile, and announced the opening of the everbuying.com pre-order page. The Flame retails at $170 (USD), global shipping included.

    Wanted: Engaging apps for Firefox OS

    Flame2If you are an experienced HTML5 app developer with a published, well-rated app that you’d like to port to Firefox OS, we’d love to hear from you! It’s always exciting to discover topnotch apps (such as PhoneGap app Find the Movie, pictured to the right running on a Flame) and see them ported from native platforms to Firefox OS. We currently have a small inventory of Flame phones for qualified HTML5 app developers with published, well-rated apps.

    How to qualify

    Through our ongoing Phones for Apps program, there’s an opportunity now for a limited number of invited app developers to receive Flame devices in exchange for a commitment to port their qualifying HTML5 apps within a month of receiving the device. Please apply here.

    There are only three ways to qualify:

    1. You’ve built a successful, well-rated HTML5 app on another platform (such as Amazon Web Apps, Blackberry WebWorks, Chrome Web Store, WebOS, Windows Phone or the PhoneGap store) and are ready to port it now to Firefox OS.
    2. You’ve built a successful, well-rated native app for iOS or Android using a cross-platform wrapper like Cordova or PhoneGap and are ready to port it to Firefox OS. Be sure to indicate the cross-platform tool you used.
    3. You’ve already published a well-rated app in Firefox Marketplace, and you have a second app in progress or already built, and are ready to port it now to Firefox OS.

    Resources

    • Learn more about the Flame (Mozilla Developer Network)
    • Get started with Open Web Apps (Mozilla Developer Network)
    • Mark your calendars for Marketplace Day, June 26 – it’s all about Apps and how you can contribute – as app developers, testers, reviewers and localizers. Hope you can join us!