Here is the story of how we did it.

On a technological side, we decided to go vanilla but with a little help from ZeptoJS to handle the DOM manipulation. We also integrated Web Activities to hook into the sharing options in the browser, and to share the selected links via email. We are using also the Readability API to have a nice formatted version of the link. The app is simple and that’s how we wanted it to be. but we have a great plus: it is internationalized in English, Spanish, Portuguese, Polish, German, Italian. Hungarian and Greek are coming soon.

You can install Manana from the Firefox Marketplace.

Building the layout and managing transitions between the views

To provide a branded UI/UX, we use a small framework built in-house. With some clever CSS properties and a few lines of JavaScript (to handle the visibility of the Views) the whole thing was quite easy to set up. Basically, we have each View arranged like cards in a deck. Every time we need one of them, we pull it out while the old one goes in the deck again (becoming hidden). When the animation is ended, we capture the animationend event, so right after that we remove the class that brings the animation properties and at the same time we add a current class, which lets the card stay in place and be visible. The old one is hidden automatically since that’s its default state.

Below you will find a small example of how it works with a moveRight animation.

Optimize CSS3 transitions

Every smartphone comes with at least a small GPU, so that the CPU doesn’t have to do all the graphical calculations. It’s very much in fashion to take advantage of the Hardware Acceleration and so we do. There are some CSS3 properties that are automatically handled by the GPU, but others are not – usually the ones that are related to 3D. This is because the element (tied to the animated property) is promoted to a new layer that doesn’t suffer from repaints imposed by other properties (like left, for example. But this can change in the future).

tl;dr Where you can, avoid expensive CPU work, especially on big or complicated elements. Here is a small extract of how we implemented such properties.

/* moveLeftIn directive */
.moveLeftIn {
    animation: 0.5s moveLeftIn cubic-bezier(…) forwards; /* 1 */
    backface-visibility: hidden; /* 2 */
    transform: translateY(0) translateX(99%); /* 3 */
    /* 1. `forwards` let the animation stop at the last keyframe */
    /* 2. avoid some possible glitches caused by 3D rendering */
    /* 3. take advantage of the GPU with 3D-related properties */
}
 
/* moveLeftIn animation */
@keyframes moveLeftIn {
    0%   { transform: translateY(0) translateX(99%); z-index: 999; }
    100% { transform: translateY(0) translateX(0);   z-index: 999; }
}

Hooking to the Firefox OS specific Web Activities

One of our favourite APIs of the WebAPIs, is Web Activities. An application emits an Activity (for example, save bookmark) then the user is prompted to chose among other applications that handles that activity. At the moment it is possible to register an application as an activity handler through the app manifest (declaration registration), but in the future this could be done inside the app itself (dynamic registration).

Saving a link from the browser to Manana is an example of Web Activity (save-bookmark). Registering an activity is trivial:

manifest.webapp

"activities": {
    "save-bookmark": {
      "disposition": "inline",
        "index.html#/savebookmark",
        "returnValue": true
    }
},

returnValue is of great importance, otherwise the handler application (Manana in our case) will never return to the caller (Browser). The handling in the app is also very easy:

navigator.mozSetMessageHandler("activity", function(activity) {
    ... check if activity is savebookmark
 
    var data = activity.source.data;
 
    ... do stuff with activity data
 
    activity.postResult("saved");
});

The applications included in your phone are part of Gaia, the UI of Firefox OS, and you can actually browse the source. They provide a lot of examples on how to use WebAPIs. If you are interested in the save-bookmark activity, have a look at the Homescreen app source code.

Manana users can also share their resources via email, directly from the app. To achieve that, we’ve implemented the email Web Activity. It’s easy to pre-populate the email subject and body, just follow the mailto URI scheme and you are done.

DB.Get(id, function (rs) {
    var subject       = rs.title,
        body          = rs.url,
        shareActivity = new MozActivity({
            name: "new",
            data: {
                type: "mail",
                url : "mailto:?body=" + encodeURIComponent(body) +
                      "&subject=" + encodeURIComponent(subject)
            }
        });
});

Retrieving and storing data with IndexedDB

IndexedDB is a client side NoSQL DB. It has some nice features which make it easier working with app updates and is easy to integrate in your app, even if you’re using a framework like AngularJS. Setting up models is fairly easy and extensively documented on the web and in Gaia Apps.

You don’t have a quota limit, although the Firefox OS Simulator seems to have a limit when dealing with datasets larger than 5mb. Each database is tied to an app and a version. Only one version can run at a time, which makes it easier not to break compatibility, especially if you rely on some external API.

Since it is asynchronous, dealing with IndexedDB operations requires having callback. Below is an example on how we deal with the action of adding a link:

1. Call Link.Add function, passing a link object and a success callback
2. Link.Add get a DB connection by calling getDB
   • In the case a DB does not exist, the onupgradeneeded callback is executed
     • Create DB
     • Create collections
     • Create indexes
   • getDB check if the connection is already open, otherwise it opens a new one
3. Link.Add fills the missing property of links with default values
4. We call the objectStore put method to save/update new data into the collection
   • If the Link object we are going to save already contains a primary key attribute, IndexedDB does an update
   • Otherwise, an insert
5. If the put method is successful it calls a success callback, returning the primary key in event.target.result

Below is an example of DB action defined in our models: (for the sake of brevity we omitted error checking code)

Add: function(data, onsuccess) {
    getDB(function(db) {
            var transaction = db.transaction(["links"], "readwrite"),
                objectStore = transaction.objectStore("links"),
                request = objectStore.put(data);
 
            request.onsucces = function(e) {
                onsuccess(e.target.result) // e.target.result is the id of the new Link
            }
    });
},

This is how you call it from the other part of the app:

var newLink = {url: "http://zombo.com",
               title: "Welcome to zombo com"};
Link.Add(newLink, function(linkId) {
    window.location.hash = "#/read/:" + linkId;
});

Enhance localization with Markdown support

Localization is a key feature in our app. Manana currently supports 6 languages (soon it will be 8). There is no official recommendation, but after some research we decided to use the same technique that Gaia is using. This is a great feature of working with Firefox OS: the whole source code is there, and you can get inspiration by reading it.

L10n has been used in two different cases: to translate the text content of DOM elements and to translate internal help pages. To make the Localization process as simple as possible, we avoided putting HTML tags in the strings and in the internal help pages; we used markdown instead. By design, we avoided having strings in the Javascript code, all messages and alerts are contained in the HTML.

To support markdown we added some glue between the l10n.js library and marked.js.

To load the help resources, Manana first does a lookup in the current installation to find the file named <resourceName>.<locale>.md. If the lookup fails, the fallback is the English version of the resource. Once the file is retrieved succesfully, the content is applied to the marked function to generate the HTML, and then added to the DOM.

function resourceInit($oldView, $newView, resource) {
    var fallback  = "/locales/resources/" + resource + ".en.md",
        localized = "/locales/resources/" + resource + "." + getLanguage() + ".md",
        renderMd  = function (txt) {
            $(".js-subpage", $newView).html(marked(txt));
        },
        renderError  = function (txt) {
            $(".js-subpage", $newView).text("ouch, cannot find: " + resource);
        };
 
    getFile(localized,
            renderMd,
            function () { getFile(fallback, renderMd, renderError); });
}

A note on retrieving local files with a XMLHttpRequest

Retrieving files asynchronously from your local app is as easy as doing a XMLHttpRequest, but there is a case that took us some time to sort out. If the local file you are retrieving is missing, the request won’t fire the success callback (of course), but neither the error callback. It will just hang there forever.

After some research, we noticed a try { ... } catch block in the l10n.js library. Basically, “in Firefox OS with the app:// protocol, trying to XHR a non-existing URL will raise an exception.”

function getFile (url, success, error) {
    var xhr = new XMLHttpRequest();
 
    xhr.open("get", url, true);
    xhr.setRequestHeader("Content-Type", "text/plain;charset=UTF-8");
 
    xhr.onreadystatechange = function() {
        if (xhr.readyState == 4) {
            if (xhr.status == 200 || xhr.status === 0)
                success(xhr.responseText);
            else
                error && error();
        }
    };
 
    xhr.onerror = error;
    xhr.ontimeout = error;
 
    // Here it is!
    try {
        xhr.send(null);
    } catch (e) {
        error && error();
    }
}

Keeping the UI synchronized with the Database

One of the most challenging aspect of an HTML app is synchronization of state between a data source (db, file, api, etc) and the UI. Frameworks like Backbone and Angular make your life easier but they are not easy beasts to tame. Going vanilla is fun, mostly because JavaScript gives you a lot of flexibility when deciding how to do things.

We really like OO programming by message passing, so we came up with a message based architecture. The body element acts as our global dispatcher while view root element is the local dispatcher. Global events get registered when the application starts and live as long as the app. Local events get registered when entering a view and live until we leave that view.

An example of global events are the CRUD operations we want to perform on our model.

Each action on the UI which changes the state of models is tied to a global event, registered on document.body. On entering a new view, we call a constructor function, which executes some initialization code.

var events = {
        // global events are never unbinded
        global: {
            "save-link": function() { ... },
            "remove-link": function() { ... },
            ...
 
            "update-link-collecetion": function { ... },
 
            ...
        },
 
        // view specific events are unbinded everytime we leave that view
        home: {
            "update-link-collection-complete": function(e) {
                var $caller = e.detail.$caller;
 
                ...
            }  
        },
 
        ...
}
 
function homeInit($oldView, $newView, dispatch) {
        // $oldView is the previous view
        // $newView is the current view
 
        // dispatch in an helper which wrap 
        // document.body.dispatchEvent(new CustomEvent(...))
        dispatch("update-link-collection")
}

Leaving a view calls a destructor on that view, and unbinds all the local events. At the end of the day we can open the app source code without spending hours trying to figure out how things work because of framework related black magic.

Integration with git

When it comes to Git, our hearts melt. We love git. Seriously. Git and Vim are the best pieces of software ever written. Love.

Ok, lets get serious again.

Our application is composed from two components, the app itself and an API Server. The API server is a Golang API proxy to Readability API. So our app directory layout looks like this:

manana/
 |- distrib.sh
 |- srv/
 |- app/

distrib.sh is a simple bash script which perform three simple tasks:

• Install a post-{commit, merge, rebase} hook, if not already installed
• Update app/js/revision.js writing the current git revision hash.
• Create a clean zip containing the app, ready for the app store or your device.

This makes our lives easier.

Conclusion. What Next?

Manana relies on the Readability API to get the stripped down version of the web page. Readability is doing a really good job, but we’d like to experiment with other solutions, like parsing the page directly in the device. It’s a hard job, but it’d be cool to do it on the client side, and some work has already been done.

We had a great time developing for Firefox OS. The most important thing for us, techwise, is that we developed a webapp, not an OS-specific app. We can use all the know-how we gained in the past years, all the great open source libraries we want, and we can even use the browser debug tools we use every day.

To help newcomers in building a Firefox OS app, we have created a list of resources we found indispensable.

Our solution was to convert Firefox into a Platform for Ethical Hacking.

How did we do it?

We searched the Web, looking for a solution that satisfied the requirements and obligations to not use automated tools. We found many, a lot of them being add-ons and extensions. Many of them were unknown, unnoticed and unofficial.

Having compiled some useful findings, our next challenge was to create something that maintained the essence of Mozilla Firefox: open, free and easy for everyone. The project was named FireCAT (Firefox Catalog of Auditing exTensions)

What is FireCAT?

FireCAT is a mind map organized into different categories and subcategories with a variety of add-ons available. These are themed around application security auditing and assessment, adding news functionality, customizing your browser and providing the ability to adapt it to different user’s need.

Current Status

The latest version is 2.0, and was presented at Black Hat Arsenal during 2011. From the web site it is possible to list the content and browse the categories, directly accessing the add-ons on Mozilla Add-ons; you can also download the entire mind map with the categories:

What’s New?

• Current Version: 2.0
• Add-ons: > 90
• Categories: 7
• Subcategories: 19

FireCAT v2.0 – Firefox Catalog of Auditing exTensions

• Information Gathering
  • Whois
  • Location Info
  • Enumeration & Fingerprint
  • Data Mining
  • Googling & Spidering
• Proxies & Web Utilities
• Editors
• Network Utilities
  • Intrusion Detection System
  • Sniffers
  • Wireless
  • Passwords
  • Protocols & Applications
• Misc
  • Tweaks & Hacks
  • Encryption / Hashing
  • Antivirus & Malware Scanner
  • Anti Spoof
  • Anti-phishing / Pharming / Jacking
  • Automation
  • Logs / History
  • Backup & Synchronization
  • Protection
• IT Security Related
• Application Auditing

What can we find in the categories?

The 7 main categories are as follows:

Information Gathering

An important part of the process of conducting vulnerability assessment or penetration testing is related to obtaining information in a passive or active way about our objective, to be analyzed later.

From the Firefox platform, we could identify IP addresses, query DNS, verify HTTP headers, generate reports of domains via Whois, use different search engines, and identify applications and versions from Web Server, modules, etc.

Example

“Show IP” shows the IP address(es) of the current page in the status bar. It also allows querying of custom services by IP (right mouse button) and hostname (left mouse button), like whois, netcraft, etc.

Proxies & Web Utilities

Sometimes it’s necessary to use proxies to connect with different hosts or through networks for exaple using TOR to hide our real IP, analyze time responses, detect load balancer, etc.

Example

The User Agent Switcher extension adds a menu and a toolbar button to switch the user agent of the browser:

Editors

From the ubiquitous FireBug to different JavaScript or XML editors, the available editors can help us to visualize and modify content as required.

Example

The JavaScript Deobfuscator will show you what JavaScript gets to run on a web page, even if it is obfuscated and generated on the fly.

Network Utilities

In this section you can use and connect with applications and services such as FTP, DNS and databases in real time.

Miscellaneous

This section contains a lot of useful add-ons, for example tools to convert your ASCII into md5, sha1, base64, hex, etc., and methods to identify the WOT (Web of Trust) in a web site, guage reputation and ratings, modify your HTTP Referer, automate actions through Macros, and make backups of your configuration and installed addons.

Example

The Certificate Patrol add-on reveals when certificates are updated, so you can ensure it was a legitimate change.

IT Security Related

Sometimes it is necessary to use search engines to find information within some Web sites, content related to errors, configurations, and news vulnerabilities by CVE.

Application Auditing

The HackBar is probably the best known addon in this group, but I recommend you take a deeper look because many of them are useful. For example, Exploit-Me’s suite to perform vulnerability analysis of cross-site scripting and SQL Injection.

Example

The SQLite Manager allows you to manage any SQLite database on your computer.

Recommended add-ons

The top ten recommended add-ons would be:

• Firebug
• Web Developer
• HackBar
• FoxyProxy
• Exploit Me
• Tamper Data
• iMacros for Firefox
• SQLite Manager
• Live HTTP Headers
• PassiveRecon

Final Words

I invite you to explore the different add-ons and extensions available, identifying which ones are most interesting and helpful for your work, and their relative performance and compatibilities. Firefox was a great help during our work!

If you decide to install a large amount of add-ons, memory usage might be a problem: consider this carefully.

Feel free to send us your comments and recommendations for new options to add to the mind map!

We’ve talked about in-app payments before in navigator.mozPay() For Web Payments, but this is an overview from an app developer’s standpoint. This also includes a few updates, such as real payments being live in the Firefox Marketplace. Payments are different than simply being a paid app, which charges the user once on the initial purchase. In-app payments is a different workflow, and allows you to charge the customer for specific items within your app.

Architecture Overview

Payments are only enabled on Firefox OS devices for now. We are working on enabling them across other platforms, even desktop Firefox, but that is a ways off. For the rest of this article, we will assume the Firefox OS environment.

The MDN page for in-app payments has a good overview of the process, but here’s an even quicker overview. You need a server to generate the payments requests, because otherwise a hacker could easily change the prices to 0. The server creates JWT tokens, passes them to your app, and your app calls navigator.mozPay with the token. The user will be taken to a built-in payments workflow, and when finished, your server will be notified. The previous Hacks article on payments goes into the lower-level mechanics and reasons for payments more thoroughly.

It’s up to you to handle the rest of the workflow: updating the app when an item is successfully purchased, linking purchases to accounts, etc.

Getting a Simulation Key

In order to use payments, you’ll need use the app key and secret generated for you by the Marketplace. If you just want to simulate payments, you can easily grab a testing key and secret. The only thing you need to do is create an account if you haven’t done so already.

You will only be able to simulate payments with that key. When you are ready to use real money, you need to submit your app to the Marketplace, setup payments, and obtain a real key and secret. All of this is explained in more detail below.

Setting up the Server-Side Code

The first thing you need to do is set up the server-side code. A “mozpay” module exists in a few different languages which implements the payment workflow for you. If it does not exist in your programming language, you’ll need to use a JWT module and do some more legwork. Check the rather simple code from mozpay-js for an example of what you need to do.

I wrote a game for Firefox OS called Webfighter which uses node.js as the backend, so I used mozpay-js on the server-side. Webfighter is a shooter with a store where you can buy more weapons and ships, and it’s a good reference for developers to see actual code for working with payments. Examples below are taken from Webfighter. You can check out Webfighter’s source here.

First, require and configure the mozpay module:

var pay = require('mozpay');
var settings = require('settings');
 
pay.configure({
    mozPayKey: settings.payKey,
    mozPaySecret: settings.paySecret,
    mozPayAudience: 'marketplace.firefox.com',
    mozPayType: 'mozilla/payments/pay/v1',
    mozPayRoutePrefix: '/mozpay'
});

The settings module is just a file with JSON that specifies configuration of the app. It’s nice to separate the configuration so that you can easily swap it out in different environments, like development and production.

Use the key and secret that you got from the Marketplace. You should only change the audience and type options if you are using a different payment server than Mozilla’s. The routePrefix is optional and specifies the URL prefix for postbacks from the payment server.

Next you need to define a route that creates the data used for the payment transaction. The client is the one making the actual payments request (so that you don’t have to handle credit card details, and it’s possible to even be charged through carrier billing). You just need to create an object that represents a purchase. This example uses express.js as the http server.

var purchaseQueue = {};
 
app.post('/sign-jwt', function(req, res) {
    var token = 'o' + Date.now();
    var item = getItem(req.body.name);
 
    var jwt = pay.request({
        id: item.name,
        name: item.name,
        description: item.description,
        icons: { '64': settings.url + item.icon },
        pricePoint: item.price,
        productData: token,
        postbackURL: settings.url + 'mozpay/postback',
        chargebackURL: settings.url + 'mozpay/chargeback',
        simulate: { result: 'postback' }
    });
 
    // Keep track of which JWT objects we are waiting on
    purchaseQueue[token] = 'processing';
 
    res.send(JSON.stringify({
        jwt: jwt,
        token: token
    });
});

Notice the settings object again; it’s just a dict that has some configuration properties about our app. settings.url is the base URL where our app is hosted, so that all of the passed URLs are absolute.

We call getItem to get the details about an item (which we don’t define here). I found it easiest to define a JSON file containing info about all the available items. You can see the JSON for available items in Webfighter on GitHub.

There are several fields for a purchase request. id is just a unique identifer, and we simply use the product name. Other fields describe the product, such as icons which lists icons of various sizes. The pricePoint is a number that maps to a specific price for each available region, and you should look at the price point table to find the one you need. For example, a price point of 10 indicates a price of $0.99 in the US.

The simulate parameter tells the payments system to just simulate the request. If it’s passed, no real charges are made and you will be taken through a simulated workflow. This is useful for development. Just remember to remove this flag in production.

You can store anything you want to in productData, which you can later use when handling events. Here I generated an arbitrary token to track requests, and I store it in the global object purchaseQueue. We use this when we handle events.

Next, subscribe to the postback event from the mozpay module:

pay.on('postback', function(data) {
    // the payment was successful
    var req = data.request;
    purchaseQueue[req.productData] = 'success';
});

You can also subscribe to the chargeback event. Chargebacks happen when a customer disputes a charge and gets the money back. This could happen a month later, and is difficult to process. Webfighter keeps track of chargebacks, but the app does not check for them and revoke items. You would need to check periodically or on startup if the user still has access to all purchased items.

pay.on('chargeback', function(data) {
    var req = data.request;
    purchaseQueue[req.productData] = 'chargeback';
});

That’s it for the server-side.

Using mozPay On the Client Side

On the client-side, you need to post to /sign-swt and use mozPay to initiate the purchase. In the following code, we check if mozPay is available and if it’s not we just give the user the item. Remember, it’s just a test app! Once a purchase is initiated, we start polling the server until we get a successful purchase.

function buy(name) {
    // Purchase an item by requesting a JWT object from the
    // server, and posting it to the mozPay API
    $.post('/sign-jwt', { name: name }, function(res) {
        if(navigator.mozPay) {
            var req = navigator.mozPay([res.jwt]);
 
            req.onerror = function() {
                console.log('mozPay error: ' + this.error.name);
                clearPolling();
            };
 
            // Poll to see when the payment is complete
            startPolling();
        }
        else {
            alert('in-app payments unavailable, so giving it to you for free');
            onPurchase(name);
        }
    });
}

Here are the polling functions which wait until the server responds with a successful purchase. Note that we only check for a successful purchase. The user will be notified of any errors within the native payment screen, and when they close the screen an error event is fired on the mozPay request object, which we handle and stop the polling. See the above code with req.onerror.

function pollQueue(token, name) {
    // Poll the server every second to see the status of our
    // payment request
 
    $.get('/purchaseQueue?token=' + token, function(res) {
        if(res == 'success') {
            onPurchase(name); 
            clearPolling();
        }
    });
}
 
var pollTimer;
function startPolling() {
    pollTimer = setInterval(function() { pollQueue(res.token, name); }, 1000);
}
 
function clearPolling() {
    if(pollTimer) {
        clearInterval(pollTimer);
    }
    pollTimer = null;
}

We call onPurchase when an item is successfully purchased and should be enabled for the user. You should create that function and write your app-specific code.

Real payments: submitting app to the Firefox Marketplace

When you are ready to run real payments, remove the simulate field when the server creates the payment request, and submit your app to the Firefox Marketplace. It needs to be submitted so that you can hook up real payments.

Get paid: configuring bank account details and payment options

Once your app is submitted, edit your app and click on “Compatibility & Payments” on the left side and select “Paid / In-App”. Under “Payment Accounts”, and add your bank account. One thing that tripped me up was needing the SWIFT code for my bank. In the future, Marketplace will hopefully autofill it, but for now you can look it up on the theswiftcodes.com.

Under “Price & Countries” you can set the price for your app. You may just want “Free with in-app payments”, which is what Webfighter is. Additionally, you can selectively choose which region your app is available in. Apps with payments are only available in regions with them enabled.

Use a real key & secret

Now click the “In-App Payments” link in the left sidebar. You’ll see a screen where you can grab a new key & secret for real payments. Use these in the configure call on the server. Never share your secret.

At this time of writing, carrier billing is only available in specific countries. Check out the Payments Status page for where it’s available. When it is not available, the user will be presented with a credit card screen.

Go make some money with your app!

We want you to be successful in participating in the app ecosystem and economy. Let us know what your experience is like! We are happy to answer any questions over on the marketplace mailing list.

You can watch the video on YouTube.

Firefox OS was built on top of the technologies that power the Web. Following Mozilla’s security practices and knowledge from over 10 years of securing Firefox, Firefox OS is engineered as a multi-tiered system that protects users while delivering the power of the mobile web. The design ensures users are in control of their data and developers have APIs and technologies at their disposal to unlock the power of the Web.

Additional links for more information:

• Firefox OS security architecture
• Firefox OS app permission model
• Application security
• Data sandboxing
• Firefox OS release schedule

Using a script to determine if a plugin is installed

To detect if a plugin is actually installed, we can query navigator.mimeTypes for the plugin MIME type we intend to use, to differentiate between plugins that are not installed and those that are click-to-activate. For example:

function isJavaAvailable() {
    return 'application/x-java-applet' in navigator.mimeTypes;
}

Note: Do not iterate through navigator.mimeTypes or navigator.plugins, as enumeration may well be removed as a privacy measure in a future version of Firefox.

Using a script callback to determine when a plugin is activated

The next thing to be careful of is scripting plugins immediately after instances are created on the page, to avoid breakage due to the plugin not being properly loaded. The plugin should make a call into JavaScript after it is created, using NPRuntime scripting:

function pluginCreated() {
    document.getElementById('myPlugin').callPluginMethod();
}

<object type="application/x-my-plugin" data="somedata.mytype" id="myPlugin">
  <param name="callback" value="pluginCreated()">
</object>

Note that the “callback” parameter (or something equivalent) must be implemented by your plugin. This can be done in Flash using the flash.external.ExternalInterface API, or in Java using the netscape.javascript package.

Using properties on the plugin to determine when it activated

When using a plugin that doesn’t allow us to specify callbacks and we can’t modify it, an alternative technique is to test for properties that the plugin should have, using code constructs like so:

<p id="myNotification">Waiting for the plugin to activate!</p>
<object id="myPlugin" type="application/x-my-plugin"></object>

window.onload = function () {
    if (document.getElementById('myPlugin').myProperty !== undefined) {
        document.getElementById('myNotification').style.display = 'none';
        document.getElementById('myPlugin').callPluginMethod();  
    } else {
        console.log("Plugin not activated yet.");
        setTimeout(checkPlugin, 500);
    }
}

Making plugins visible on the page

When a site wants the user to enable a plugin, the primary indicator is that the plugin is visible on the page, for example:

If a page creates a plugin that is very small or completely hidden, the only visual indication to the user is the small icon in the Firefox location bar. Even if the plugin element will eventually be hidden, pages should create the plugin element visible on the page, and then resize or hide it only after the user has activated the plugin. This can be done in a similar fashion to the callback technique we showed above:

function pluginCreated() {
  // We don't need to see the plugin, so hide it by resizing
  var plugin = document.getElementById('myPlugin');
  plugin.height = 0;
  plugin.width = 0;
  plugin.callPluginMethod();
}

<!-- Give the plugin an initial size so it is visible -->
<object type="application/x-my-plugin" data="somedata.mytype" id="myPlugin" width="300" height="300">
  <param name="callback" value="pluginCreated()">
</object>

Note: For more basic information on how plugins operate in Firefox, read Why do I have to click to activate plugins? on support.mozilla.org.

Introducing The Pond

The Pond is a multi-platform HTML5 game (source code) that explores minimalistic design and resolution independent gameplay. The Pond isn’t about reaching a high score, or about buying weapon upgrades. It’s about relaxing and exploring a beautiful world.

It is available on all these platforms/in all these stores:

In making The Pond I came across many performance obstacles which I will explore in detail (especially when optimizing the codebase for mobile).

Tools

Before I begin, I would like to mention the two tools that made coding The Pond both efficient and highly enjoyable: Light Table and CocoonJS.

Light Table is an IDE (still in alpha) which provides an integrated development environment for real-time javascript code injection. This means that javascript edited within the editor can be previewed without reloading the page. If we look at the shape of the fish in the game we notice that it is comprised of Bézier curves. Instead of trying to find an editor for creating Bézier curves, I simply estimated a basic shape and modified the variables in real-time until I was satisfied with it’s look and feel.

CocoonJS on the otherhand provides a canvas optimized compatibility layer for improved performance on mobile devices. Not only does it optimize, it also provides an interface for exporting our application to many devices (Android, iOS, Amazon (android), Pokki, and Chrome Web Store).

Physics

The Pond may seem simple on the outside, but on the inside it’s full of performance optimizations and responsive features. As we resize the game, it updates and re-optimizes itself to render less objects and spawn less fish, and if that’s not enough the framerate degrades smoothly to keep physics in check. This is thanks to the use of a fixed interval physics time step. Gameprogrammingpatterns.com provides a good explanation for how to do this and why it matters, but honestly the code makes the most sense:

var MS_PER_UPDATE = 18; // Time between physics calculations
var lag = 0.0; // accumulate lag over frames
var previousTime = 0.0; // used for calculating the time delta
 
// main game loop
function draw(time) {
  requestAnimFrame(draw); // immidiately queue another frame
  lag += time - previousTime; // add time delta
  previousTime = time;
 
  var MAX_CYCLES = 18; // prevent infinite looping/hanging on slow machines
 
  // physics calculations
  while(lag >= MS_PER_UPDATE && MAX_CYCLES) {
 
    // user input, movement, and animation calculations
    physics();
    lag -= MS_PER_UPDATE;
    MAX_CYCLES--;
  }
 
  // if we exhausted our cycles, the client must be lagging
  if(MAX_CYCLES === 0) {
 
    // adaptive quality
    lowerQuality();
  }
 
  // if 5 frames behind after update, jump
  // this prevents an infinite input lag from ocurring
  if(lag/MS_PER_UPDATE > 75) {
    lag = 0.0;
  }
 
  // draw to canvas
  paint();
}

What’s important to notice here is that physics is not calculated based on the time delta, instead it’s calculated at a fixed 18ms interval. This is important because it means that any client lag will not be reflected in physics calculations, and that slower machines will simply lose framerate.

Dynamic Quality

The next optimization we notice is the lowerQuality() function, which adaptively decreases the render quality of the game. The way this works is simply by re-sizing the drawing canvas (it’s still full screen, it simply gets streched out), which in-turn leads to reduced spawns and collisions.

function resizeWindow() {
 
  // quality is a global variable, updated by lowerQuality()
  $canv.width = window.innerWidth * quality/10
  $canv.height = window.innerHeight * quality/10
  ctx = $canv.getContext('2d')
  ctx.lineJoin = 'round'
 
  // resize HUD elements, and reduce spawning
  if(GAME.state === 'playing') {
    GAME.spawner.resize($canv.width, $canv.height)
    GAME.levelBar.resize($canv.width, $canv.height)
    GAME.levelBalls.resize($canv.width, $canv.height)
  } else {
    if(ASSETS.loaded) drawMenu()
  }
}

Spawning

Now, we’ve been talking about reducing spawning to improve performance so let me explain how that happens. The spawning algorithm works by creating a virtual grid sized based on the window size. As the player travels from one grid zone to another, the adjacent zones are populated with enemies:

Spawner.prototype.spawn = function(zone) {
  // spawn 1-3  fish per 500sqpx, maybe larger maybe smaller than player
  // 0.5 chance that it will be bigger/smaller
  var mult = this.width*this.height/(500*500)
  for(var i=0, l=(Math.floor(Math.random()*3) + 1) * mult; i < l; i++) {
 
    // spawn coordinates random within a zone
    var x = zone[0]+Math.floor(this.width*Math.random()) - this.width/2
    var y = zone[1]+Math.floor(this.height*Math.random()) - this.height/2
    var size = Math.random() > 0.5 ? this.player.size + Math.floor(Math.random() * 10) : this.player.size - Math.floor(Math.random() * 10)
 
    // spawn a new fish
    var fish = new Fish(true, x, y, size, Math.random()*Math.PI*2-Math.PI, Math.random()*Math.PI)
 
    this.fishes.push(fish)
  }
  return zone
}

The last piece of the puzzle is removing enemies as they move far enough away:

// if far enough away from player, remove
if(distance(fish, player) > Math.max($canv.width, $canv.height) * 2) {
  fish.dead = true
}

Collisions

The next performance optimization lies with the collision code. Colliding irregularly shaped objects can be extremely difficult and resource intensive. One option is to do color based collision (scan for overlapping colors), but that is much too slow. Another option might be to mathematically calculate Bézier curve collisions, however this is not only CPU intensive, it is also quite difficult to code. I finally opted for an approximation approach using circles. Basically I calculate the position of circles within each fish and detect circle collision among the fish. Boolean circle collision is extremely efficient, as it simply requires measuring the distance between objects. This ends up looking like this (debug mode):

Fish.prototype.collide = function (fish) {
 
  // the fish has been killed and is being removed or it is far away
  if (this.dying || fish.dying || distance(this, fish) > this.size * 5 + fish.size*5) {
    return false
  }
 
  // there are 6 circles that make up the collision box of each fish
  var c1, c2
  for (var i=-1, l = this.circles.length; ++i < l;) {
    c1 = this.circles[i]
    for (var j=-1, n = fish.circles.length; ++j < n;) {
      c2 = fish.circles[j]
 
      // check if they touch
      if(distance(c1, c2) <= c2.r + c1.r) {
        return true
      }
    }
  }
 
  return false
}

We also avoid unnecessary collision calculations by only checking the fish that are visible (or near-visible):

if(Math.abs(fish2.x - player.x) < $canv.width && Math.abs(fish2.y - player.y) < $canv.height) {
    // check
}

Drawing

After getting the physics+ out of the way, it’s time to optimize drawing operations. Many games use sprite maps for animation (Senshi for example) which can be highly optimized. Unfortunately our fish are dynamically generated so we must find other ways to optimizing drawing. First lets use Chrome’s javascript profiler to identify bottlenecks:

What we see here is that stroke is using a lot of resources. Truth be told, fill used to be there too. This is because both were called heavily when drawing fish. The game looked a bit like this:

After removing fill I saw a huge performance increase, and the game looked much better. The reason the drawImage function is up there as well is because I take advantage of offscreen canvas rendering. Each fish is drawn on its own offscreen canvas which is then rendered onto the larger visible canvas. This is also what allowed me to easily explode the fish into particles by reading pixel data:

Fish.prototype.toParticles = function(target) {
  var particles = []
 
  // read canvas pixel data
  var pixels = this.ctx.getImageData(0,0,this.canv.width, this.canv.height).data
  for(var i = 0; i < pixels.length; i += 36 * Math.ceil(this.size/20) * (isMobile ? 6 : 1)) {
    var r = pixels[i]
    var g = pixels[i + 1]
    var b = pixels[i + 2]
 
    // black pixel - no data
    if(!r && !g && !b){
      continue
    }
 
    // Math to calculate position
    var x = i/4 % this.canv.width - (this.canv.width/2 + this.size)
    var y = Math.floor(i/4 / this.canv.width) - (this.canv.height/2)
    var relativePos = rot(x, y, this.dir)
    x=this.x + relativePos[0]
    y=this.y + relativePos[1]
 
    var col = new Color(r, g, b)
    var dir = directionTowards({x: x, y: y}, this)
    particles.push(new Particle(x, y, col, target, Math.PI*Math.random()*2 - Math.PI, this.size/20))
  }
  return particles
}

The End

In the end the performance optimizations paid off and made the game feel more polished and playable even on lower-end mobile devices.

If you enjoyed this post, I regularly blog about my development projects over at http://zolmeister.com.

The Pond awaits exploring…

Stack Overflow is one of – if not the – most well-known question and answer sites among developers, and we believe it’s very important to go where developers are and choose to have their discussions while developing.

Initially our main focus is on Firefox OS app development and Open Web Apps in general. There are three tags that we are monitoring:

• firefox
• firefox-os
• html5-apps

All tags and posts are available at our landing page on Stack Overflow: http://stackoverflow.com/r/mozilla

Take part!

We’d love for you to participate with both questions and answers, so we together can collaborate and make the Open Web and open technologies the first choice for developers!

A little extra motivation might be that if you answer a lot of questions, you’ll show up in the Top Users page for that tag, e.g. http://stackoverflow.com/tags/firefox-os/topusers

Resources that might be good for you about HTML5, Apps, Developer Tools and more are:

• HTML5 on the Mozilla Developer Network (MDN)
• App Center on MDN
• Firefox OS on MDN
• Firefox Developer Tools on MDN

See you on Stack Overflow!

Firefox OS was built on top of the technologies that power the Web. Following Mozilla’s security practices and knowledge from over 10 years of securing Firefox, Firefox OS is engineered as a multi-tiered system that protects users while delivering the power of the mobile web. The design ensures users are in control of their data and developers have APIs and technologies at their disposal to unlock the power of the Web.

Watch the following video where we talk more about the security design and controls present in Firefox OS. In this, the first of two videos on Firefox OS security, we’ll cover items such as the multi-tiered architecture, the permission model, run time decision making, protection of users data and the update model. You can watch the video on YouTube.

Additional links for more information:

• Firefox OS security architecture
• Firefox OS app permission model
• Application security
• Data sandboxing
• Firefox OS release schedule

Not only that, but it is also very interesting when combined with other APIs. One of these APIs is getUserMedia(), which allows us to capture audio and/or video from the local PC’s microphone / camera devices. Whiteboard Drum (code on GitHub) is a music application, and a great example of what can be achieved using Web Audio API and getUserMedia().

I demonstrated Whiteboard Drum at the Web Music Hackathon Tokyo in October. It was a very exciting event on the subject of Web Audio API and Web MIDI API. Many instruments can collaborate with the browser, and it can also create new interfaces to the real world.

I believe this suggests further possibilities of Web-based music applications, especially using Web Audio API in conjunction with other APIs. Let’s explain how the key features of Whiteboard Drum work, showing relevant code fragments as we go.

Overview

First of all, let me show you a picture from the Hackathon:

And a easy movie demo:

As you can see, Whiteboard Drum plays a rhythm according to the matrix pattern on the whiteboard. The whiteboard has no magic; it just needs to be pointed at by a WebCam. Though I used magnets in the demo, you can draw the markers using pen if you wish. Each row represents the corresponding instruments of Cymbal, Hi-Hat, Snare-Drum and Bass-Drum, and each column represents timing steps. In this implementation, the sequence has 8 steps. The color blue will activate the grid normally, and the red will activate with accent.

The processing flow is:

1. The whiteboard image is captured by the WebCam
2. The matrix pattern is analysed
3. This pattern is fed to the drum sound generators to create the corresponding sound patterns

Although it uses nascent browser technologies, each process itself is not so complicated. Some key points are described below.

Image capture by getUserMedia()

getUserMedia() is a function for capturing video/audio from webcam/microphone devices. It is a part of WebRTC and a fairly new feature in web browsers. Note that the user’s permission is required to get the image from the WebCam. If we were just displaying the WebCam image on the screen, it would be trivially easy. However, we want to access the image’s raw pixel data in JavaScript for more processing, so we need to use canvas and the createImageData() function.

Because pixel-by-pixel processing is needed later in this application, the captured image’s resolution is reduced to 400 x 200px; that means one rhythm grid is 50 x 50 px in the rhythm pattern matrix.

Note: Though most recent laptops/notebooks have embedded WebCams, you will get the best results on Whiteboard Drum from an external camera, because the camera needs to be precisely aimed at the picture on the whiteboard. Also, the selection of input from multiple available devices/cameras is not standardized currently, and cannot be controlled in JavaScript. In Firefox, it is selectable in the permission dialog when connecting, and it can be set up from “contents setup” option of the setup screen in Google Chrome.

Get the WebCam video

We don’t want to show these parts of the processing on the screen, so first we hide the video:

<video id="video" style="display:none"></video>

Now to grab the video:

video = document.getElementById("video");
navigator.getUserMedia=navigator.getUserMedia||navigator.webkitGetUserMedia||navigator.mozGetUserMedia;
navigator.getUserMedia({"video":true},
    function(stream) {
        video.src= window.URL.createObjectURL(stream);
        video.play();
    },
    function(err) {
        alert("Camera Error");
    });

Capture it and get pixel values

We also hide the canvas:

<canvas id="capture" width=400 height=200 style="display:none"></canvas>

Then capture our video data on the canvas:

function Capture() {
    ctxcapture.drawImage(video,0,0,400,200);
    imgdatcapture=ctxcapture.getImageData(0,0,400,200);
}

The video from the WebCam will be drawn onto the canvas at periodic intervals.

Image analyzing

Next, we need to get the 400 x 200 pixel values with getImageData(). The analyzing phase analyses the 400 x 200 image data in an 8 x 4 matrix rhythm pattern, where a single matrix grid is 50 x 50 px. All necessary input data is stored in the imgdatcapture.data array in RGBA format, 4 elements per pixel.

var pixarray = imgdatcapture.data;
var step;
for(var x = 0; x < 8; ++x) {
    var px = x * 50;
    if(invert)
        step=7-x;
    else
        step=x;
    for(var y = 0; y < 4; ++y) {
        var py = y * 50;
        var lum = 0;
        var red = 0;
        for(var dx = 0; dx < 50; ++dx) {
            for(var dy = 0; dy < 50; ++dy) {
                var offset = ((py + dy) * 400 + px + dx)*4;
                lum += pixarray[offset] * 3 + pixarray[offset+1] * 6 + pixarray[offset+2];
                red += (pixarray[offset]-pixarray[offset+2]);
            }
        }
        if(lum < lumthresh) {
            if(red > redthresh)
                rhythmpat[step][y]=2;
            else
                rhythmpat[step][y]=1;
        }
        else
            rhythmpat[step][y]=0;
    }
}

This is honest pixel-by-pixel analysis of grid-by-grid loops. In this implementation, the analysis is done for the luminance and redness. If the grid is “dark”, the grid is activated; if it is red, it should be accented.

The luminance calculation uses a simplified matrix — R * 3 + G * 6 + B — that will get ten times the value – meaning – which means getting the value of range 0 to 2550 for each pixel. And the redness R – B is a experimental value because all that is required is a decision of Red or Blue. The result is stored in the rhythmpat array, with a value of 0 for nothing, 1 for blue or 2 for red.

Sound generation through the Web Audio API

Because the Web Audio API is a very cutting edge technology, it is not yet supported by every web browser. Currently, Google Chrome/Safari/Webkit-based Opera and Firefox (25 or later) support this API. Note: Firefox 25 is the latest version released at the end of October.

For other web browsers, I have developed a polyfill that falls back to Flash: WAAPISim, available on GitHub. It provides almost all functions of the Web Audio API to unsupported browsers, for example Internet Explorer.

Web Audio API is a large scale specification, but in our case, the sound generation part requires just a very simple use of the Web Audio API: load one sound for each instrument and trigger them at the right times. First we create an audio context, taking care of vendor prefixes in the process. Prefixes currently used are webkit or no prefix.

audioctx = new (window.AudioContext||window.webkitAudioContext)();

Next we load sounds to buffers via XMLHttpRequest. In this case, different sounds for each instrument (bd.wav / sd.wav / hh.wav / cy.wav) are loaded into the buffers array:

var buffers = [];
var req = new XMLHttpRequest();
var loadidx = 0;
var files = [
    "samples/bd.wav",
    "samples/sd.wav",
    "samples/hh.wav",
    "samples/cy.wav"
];
function LoadBuffers() {
    req.open("GET", files[loadidx], true);
    req.responseType = "arraybuffer";
    req.onload = function() {
        if(req.response) {
            audioctx.decodeAudioData(req.response,function(b){
                buffers[loadidx]=b;
                if(++loadidx < files.length)
                    LoadBuffers();
            },function(){});
        }
    };
    req.send();
}

The Web Audio API generates sounds by routing graphs of nodes. Whiteboard Drum uses a simple graph that is accessed via AudioBufferSourceNode and GainNode. AudioBufferSourceNode play back the AudioBuffer and route to destination(output) directly (for normal *blue* sound), or route to destination via the GainNode (for accented *red* sound). Because the AudioBufferSourceNode can be used just once, it will be newly created for each trigger.

Preparing the GainNode as the output point for accented sounds is done like this.

gain=audioctx.createGain();
    gain.gain.value=2;
    gain.connect(audioctx.destination);

And the trigger function looks like so:

function Trigger(instrument,accent,when) {
    var src=audioctx.createBufferSource();
    src.buffer=buffers[instrument];
    if(accent)
        src.connect(gain);
    else
        src.connect(audioctx.destination);
    src.start(when);
}

All that is left to discuss is the accuracy of the playback timing, according to the rhythm pattern. Though it would be simple to keep creating the triggers with a setInterval() timer, it is not recommended. The timing can be easily messed up by any CPU load.

To get accurate timing, using the time management system embedded in the Web Audio API is recommended. It calculates the when argument of the Trigger() function above.

// console.log(nexttick-audioctx.currentTime);
while(nexttick - audioctx.currentTime < 0.3) {
    var p = rhythmpat[step];
    for(var i = 0; i < 4; ++i)
        Trigger(i, p[i], nexttick);
    if(++step >= 8)
        step = 0;
    nexttick += deltatick;
}

In Whiteboard Drum, this code controls the core of the functionality. nexttick contains the accurate time (in seconds) of the next step, while audioctx.currentTime is the accurate current time (again, in seconds). Thus, this routine is getting triggered every 300ms – meaning look ahead to 300ms in the future (triggered in advance while nextticktime – currenttime < 0.3). The commented console.log will print the timing margin. While this routine is called periodically, the timing is collapsed if this value is negative.

For more detail, here is a helpful document: A Tale of Two Clocks – Scheduling Web Audio with Precision

About the UI

Especially in music production software like DAW or VST plugins, the UI is important. Web applications do not have to emulate this exactly, but something similar would be a good idea. Fortunately, the very handy WebComponent library webaudio-controls is available, allowing us to define knobs or sliders with just a single HTML tag.

NOTE: webaudio-controls uses Polymer.js, which sometimes has stability issues, causing unexpected behavior once in a while, especially when combining it with complex APIs.

Future work

This is already an interesting application, but it can be improved further. Obviously the camera position adjustment is an issue. Analysis can be more smarter if it has an auto adjustment of the position (using some kind of marker?), and adaptive color detection. Sound generation could also be improved, with more instruments, more steps and more sound effects.

How about a challenge?

Whiteboard Drum is available at http://www.g200kg.com/whiteboarddrum/, and the code is on GitHub.

Have a play with it and see what rhythms you can create!

Writing Vertex and Fragment shaders are an essential part of developing 3D on the web even if you are using a library, in fact the Epic Citadel demo includes over 200 shader programs. This is because most rendering is customised and optimised to fit a game’s needs. Shader development is currently awkward for a few reasons:

• Seeing your changes requires a refresh
• Some shaders are applied under very specific conditions

Here is a screencast that shows a how to manipulate shader code using a relatively simple WebGL demo:

Starting in Firefox 27 we’ve introduced a new tool called the ‘Shader Editor’ that makes working with shader programs much simpler: the editor lists all shader programs running in the WebGL context, and you can live-edit shaders and see immediate results without interrupting any animations or state. Additionally editing shaders should not impact WebGL performance.

Enabling the Shader Editor

The Shader Editor is not shown by default, because not all the web pages out there contain WebGL, but you can easily enable it:

1. Open the Toolbox by pressing either F12 or Ctrl/Cmd + Shift + I.
2. Click on the ‘gear’ icon near the top edge of the Toolbox to open the ‘Toolbox Options’.
3. On the left-hand side under ‘Default Firefox Developer Tools’ make sure ‘Shader Editor’ is checked. You should immediately see a new ‘Shader Editor’ Tool tab.

Using the Shader Editor

To see the Shader Editor in action, just go to a WebGL demo such as this one and open the toolbox. When you click on the shader editor tab, you’ll see a reload button you will need to click in order to get the editor attached to the WebGL context. Once you’ve done this you’ll see the Shader Editor UI:

• On the left you have a list of programs, a vertex and fragment shader corresponds to each program and their source is displayed and syntax highlighted in the editors on the right.
• The shader type is displayed underneath each editor.
• Hovering a program highlights the geometry drawn by its corresponding shaders in red – this is useful for finding the right program to work on.
• Clicking on the eyeball right next to each program hides the rendered geometry (useful in the likely case an author wants to focus solely on some geometry but not other, or to hide overlapping geometry).
• The tool is responsive when docked to the side.

Editing Shader Programs

The first thing you’ll notice about Shader program code is that it is not JavaScript. For more information on how Shader programs work, I highly recommend you start with the WebGL demo on the Khronos wiki and/or Paul Lewis’ excellent HTML5 Rocks post. There also some great long standing tutorials on the Learning WebGL blog. The Shader Editor gives you direct access to the programs so you can play around with how they work:

• Editing code in any of the editors will compile the source and apply it as soon as the user stops typing;
• If an error was made in the code, the rendering won’t be affected, but an error will be displayed in the editor, highlighting the faulty line of code; hovering the icon gutter will display a tooltip describing the error.

Learn more about the Shader Editor on the Mozilla Developer Network.

Here is a second screencast showing how you could directly edit the shader programs in the Epic Citadel demo:

Audio and video HTML tags

Since we are talking about HTML, it makes total sense to think about using the <audio>, and <video> tag to play those media in your Firefox OS app. If you want to add a video in your application, just use this code.

<video src="http://v2v.cc/~j/theora_testsuite/320x240.ogg" controls>
  Your browser does not support the video element.
</video>

In this code example, the user will see a video player with controls, and will have the opportunity to start the video. If your application is running in a browser not supporting the video tag, the user will see the text between the tag. It’s still a good practice to do so, even if your primary target is a Firefox OS app, because since it uses HTML5, someone may access it from another browser if it’s a hosted app. Note that you can use other attributes for this element.

As for the audio tag, it’s basically the same.

<audio id="demo" src="/music/audio.mp3" autoplay loop></audio>

In this example, the audio will start automatically, and will play the audio file, in a loop, from the relative path: it’s perfect for background music if you are building a game. Note that you can add other attributes to this element too.

Of course, using those elements without JavaScript give you basic features, but no worries, you can programmatically control them with code. Once you have your HTML element, like the audio example you just saw, you can use JavaScript to play, pause, change the volume, and more.

document.querySelector("#demo").play(); //Play the Audio
document.querySelector("#demo").pause(); //Pause the Audio
document.querySelector("#demo").volume+=0.1; //Increase Volume
document.querySelector("#demo").volume-=0.1; //Decrease Volume

You can read more on what you can do with those two elements in the Mozilla Developer Network documentation. You also want to give a closer look to the supported format list.

Use audio while the screen is locked

Maybe you are building a podcast app, or at least you need to be able to play audio while the screen is locked? There is a way to do it by using the audio tag. You simply need to add the mozaudiochannel attribute with the value of content to your actual tag.

<audio mozaudiochannel="content" preload="none" 
  src="http://upload.wikimedia.org/wikipedia/en/4/45/ACDC_-_Back_In_Black-sample.ogg" 
  autoplay></audio>

Actually, it’s not quite true as this code won’t work as is. You also need to add a permission to the manifest file.

"permissions": {
  "audio-channel-content":{
    "description":"Use the audio channel for the music player"
  }
}

Having the manifest line above will authorize your application to use the audio channel to play music, even when the screen is locked. Having said that, you probably realize that this code is specific to Firefox OS for now. I intentionally put the end of the last sentence in bold as it’s one thing you need to understand about Firefox OS: we had to create some APIs, features or elements to give the power HTML deserve for developers, but we are working with the W3C to make those standards. In the case that the standards won’t be the same as what we created, we’ll change it to reflect it.

Firefox OS Web activities

Finally, something very handy for Firefox OS developers: the Web Activities. They define a way for applications to delegate an activity to another (usually user-chosen) application. They aren’t standardized, at the time of writing. In the case that will be interesting to us, we’ll use the Web Activity call open, to open music or video files. Note that for video, you can also use the view activity that basically does the same. Let’s say I want to open a remote video when someone clicks on a button with the id open-video: I’ll use the following code in my JavaScript to make it happen.

var openVideo = document.querySelector("#open-video");
if (openVideo) {
    openVideo.onclick = function () {
        var openingVideo = new MozActivity({
            name: "open",
            data: {
                type: [
                  "video/webm",
                  "video/mp4",
                  "video/3gpp",
                  "video/youtube"
                ],
                url: "http://v2v.cc/~j/theora_testsuite/320x240.ogg"
            }
        });
    }
}

In that situation, the video player of Firefox OS will open, and play the video: it’s that easy!

In the end…

You may or may not need to use those tricks in your app, but adding videos or audio can enhance the quality of your application, and make it feel alive. At the end, you have to give a strong experience to your users, and it’s what will make the difference between a good and a great app!

JS Debugger: Break on DOM Events

You can now automatically break on a variety of DOM events, without needing to manually set a breakpoint. To do this, click on the “Expand Panes” button on the top right of the debugger panel (right next to the search box). Then flip over to the events tab. Click on an event name to automatically pause the next time it happens. This will only show events that currently have listeners bound from your code. If you click on one of the headings, like “Mouse” or “Keyboard”, then all of the relevant events will be selected.

Inspector improvements

We’ve listened to feedback from web developers and made a number of enhancements to the Inspector:

Edit as HTML

Now in the inspector, you can right click on an element and open up an editor that allows you to set the outerHTML on an element.

Select default color format

You now have an option to select the default color format in the option panel:

Color swatch previews

The Developer Tools now offer color swatch previews that show up in the rule view:

Image previews for background image urls

Highly requested, we now offer image previews for background image URLs:

In addition to above improvements, Mutated DOM elements are now highlighted in the Inspector.

Keep an eye out for more tooltips coming soon, and feel free to chime in if you have any others you’d like to see!

Codemirror

Codemirror is a popular HTML5-based code editor component used on web sites. It is customizable and theme-able. The Firefox Devtools now use CodeMirror in various places: Style editor, Debugger, Inspector (Edit as HTML) and Scratchpad.

From the Option panel, the user can select which theme to use (dark or light).

WebConsole: Reflow Logging

When the layout is invalidated (CSS or DOM changes), gecko needs to re-compute the position of some nodes. This computation doesn’t happen immediatly. It’s triggered for various reasons. For example, if you do “node.clientTop”, gecko needs to do this computation. This computation is called a “reflow”. Reflows are expensive. Avoiding reflows is important for responsiveness.

To enable reflow logging, check the “Log” option under the “CSS” menu in the Console tab. Now, everytime a reflow happens, a log will be printed with the name of the JS function that triggered this reflow (if caused by JS).

That’s all for this time. Hope you like the new improvements!

Designing a webapp now means designing for 2014 and beyond, where users buy giant phones, phablets and tablets in many sizes. 4G connections may be faster than DSL or the throttled Wi-Fi at a coffee-shop. A number of laptop and “all-in-one” computers have touchscreens. Users may spend an evening hour on the couch with a tablet. And it was never true that all desktop users wanted apps to take up their whole screen – writing an e-mail or report often requires consulting another app, which ought to be content with half the screen.

Segmenting users by device today typically leads to bad assumptions for large minorities. Users use multiple devices throughout the day and dislike learning different interaction patterns for different devices. Apple’s “Back to the Mac” initiative sought to bring smartphone features to OS X, and Windows 8 and Ubuntu Unity strive to have a single experience from phones to desktops.

App examples

It is difficult to achieve a good user experience for all devices and screen sizes, as seen by the criticism of Windows 8 and “Back to the Mac”. It can be done for a range of devices, especially by new apps unburdened by legacy expectations. I present here two web apps optimized for smartphones through desktops (but not “smart” TVs, feature phones nor smartwatches): a photo gallery and a memo app.

Photo gallery

The photo gallery is half of a photo sharing app. A sample album of the gallery can be seen online. (The application described as a whole is worth seeing.)

The photo gallery follows the paradigm used by a number of prior photo browsing apps. Thumbnails are presented in a grid which scrolls vertically. The number and width of columns depends on the screen width. Grid cells could be small or large; I chose the minimum cell size to be the smallest common screen width for smartphones: 320 CSS pixels. Thus, the number of columns is floor(windowWidth / 320) and the cell size is windowWidth / number of columns. Columns normally are from 320 to 639 pixels wide.

The number of columns is capped at four, so scrolling doesn’t outpace data transfer.

One approach would use fixed size thumbnails of 300×300 and increase the padding as cells get larger. However, the image library available (libjpeg) only supports half-, quarter- and eighth-size reductions, so padding is fixed and images are resized to fill. The web app requests a photo of appropriate size and the server (written in node.js, and normally running on a phone) returns the smallest version of the photo that fills the requested size cell.

Tapping on a thumbnail switches to a carousel which fills the window. The user can zoom in further by double-tapping the image or using the mousewheel. Originally, full-screen mode was engaged when the carousel was displayed; however, changing to full-screen takes several seconds on many devices, so now full-screen mode must be manually set. Touchscreen users can swipe to navigate forward and backward, while desktop users can use the keyboard. A weakness is that a mouse user can click on a picture to navigate forward, but must swipe (or use the keyboard) to navigate backward. Mouse users could benefit from visible navigation buttons, but mice aren’t easy for a web app to detect.

Switching modes from the thumbnail gallery to the carousel and back is easy on any device. This supports the typical scenario of: user skims thumbnails to photo of interest, taps to enlarge, uses carousel to go through related photos, and returns to thumbnails. Users can navigate a substantial number of photos the same way on any device. A larger screen is shows more thumbnails at once and more detail on individual photos, but interaction is the same.

Development was eased by creative use of the List widget of Enyo 2 (though any list widget could have been used). Application code sizes the cells and fills each row with 1-4 thumbnails, while the List widget handles creating and re-using DOM elements. The carousel is Enyo’s ImageCarousel, which is designed from the ground up to run well on multiple types of devices.

Memo app

The second example is a deceptively simple memo app, which can be used on-line or installed on Firefox OS from the Marketplace

The fundamental layout is two panels side-by-side. In windows narrower than 640 CSS pixels, the second panel slides in on top of the first. Collapse is based solely on window width, not device class, so phablet users will have one column in portrait mode and two columns in landscape mode.

Tablet and desktop users don’t need to muck with dialogs or layers, while phone users have fast transitions. Most importantly, the shift in mental models is easy when going from a phone to a larger device. While this app does not currently sync user data between devices, adding that would give a seamless experience across devices.

Presently, the Help pane displaces the Details pane. In windows wider than 960px, Help should really be a third panel.

The collapsing panels use the Enyo Panels widget. Another instance of Panels is used to swap the Details and Help panes in the second panel. Using a JavaScript framework designed to support multiple devices really speeds development.

To maximize display of user data the list foregoes controls on items in favor of direct manipulation: items can be swiped (or double-clicked) for operations such as Delete. Reordering is done by hold-then-drag. Discoverability was traded off for power – many apps will need to strike a different balance. A possible future enhancement on larger screens would be to add visible controls or at least tool tips.

Fixed groupings such as folders or notebooks would require the user to navigate an extra level to access data. Search allows a memo to be in multiple groupings, and requires less UI. (However, it is not appropriate for users who rely on spatial memory.)

While typing on a touchscreen is laborious, the app minimizes typing by using prefix searches. Searching for “s fra” will find all memos that contain “San Francisco” and few false positives among hundreds of memos. The on-screen keyboard on most phones obscures all but a few list items, but live search and the counter of hits tells the user when she has narrowed the search to a handful of memos. Skimming the list then allows the user to select the desired record. Live full-text search is implemented using IndexedDB.

Typing is also minimized by localizing the search to the preferred language of the browser or OS. In Spanish ñ is a different letter than n (and Spanish keyboards have a key devoted to it), just as w is a different letter than u in English. For Spanish users, peña never matches pena, reducing false matches. However, users are not required to know languages other than their own, so for a Spanish user “cote” matches the French cote, coté, côte and côté.

Hardware keyboards allow users to type more quickly and accurately, larger screens minimize scrolling, and faster processors and disks make complicated searches nearly instantaneous, but the interface works the same for all devices.

Mobile first

Both apps were designed “mobile-first”, which tends to result in lean apps, with fewer features than standard desktop apps. Given that most features in desktop apps are rarely used, that’s not necessarily a problem for desktop users. Carefully-chosen defaults eliminate the need for preference settings.

Working out a user interface design that works well across a range of devices often requires more effort initially, but can result in a more user friendly experience with the added benefit of being easier to maintain. A few powerful user-interface elements can eliminate the need for many panes, dialogs, dropdowns and buttons.

This is part two, out of five, focusing on the built-in Developer Tools in Firefox, their features and where we are now with them. The intention is to show you all the possibilities available, the progress and what we are aiming for.

In the first post in the series we discussed the Web Console and the JavaScript Debugger. While these two tools are powerful and provide capabilities to interrogate and alter your web applications, additional tools are available to further enhance the developer experience while building and debugging your apps. In this post we briefly cover the Scratchpad and the Style editor.

As with the first post, we present each tool with a quick screencast demonstrating some of their capabilities.

The Style Editor

The Style Editor is primarily used to edit, debug or create new stylesheets within the context of the current app. Changes made in the style editor are automatically reflected in the loaded page. If you are not familiar with Cascading Style Sheets (CSS), please be sure to take a look at the CSS MDN documentation.

The Style Editor allows saving the changes made while using the editor. In addition you can also import existing stylesheets and apply them to the current page or individually disable specific stylesheets. The Style Editor is also linked to the Inspector allowing developers quick access to the stylesheet for the inspected element. The following screencast presents an overview of the Style Editor’s features.

For more detailed information on the Style Editor take a look at the MDN documentation.

Scratchpad

The Scratchpad has many uses and is essentially a live JavaScript editor and prototyping tool. Using the Scratchpad, a developer can access the current page’s objects, variables and script. In addition, complete functions can be written and tested in the editor within the scope of the live page. These changes can then be attached and saved with the current application.

External JavaScript files can also be loaded and tested. Several run options are available to allow a developer to just execute the code, execute the code and inspect the returned object, or execute the code and print out the results as a comment within the Scratchpad. The following screencast illustrates some of the features of the Scratchpad. Note that Scratchpad script runs in the same
context as a script loaded into the page. In the screencast, the example uses the jQuery library and some custom script to illustrate this feature.

For more information on the Scratchpad, see the MDN Debugger documentation.

If you are not very familiar with JavaScript, make sure to check out the MDN documentation for a comprehensive list of resources on learning the language and mechanics.

Learn More

These screencasts give a quick introduction to the main features of these tools. For the full details on all of the Developer Tools, check out the full MDN Tools documentation.

Coming Up

In the next post, we will focus on some Mobile design features including the Responsive Design View and Remote debugging using the App Manager. Please provide your suggestions on what features you would like to see explained in more detail in this upcoming post, by commenting below.

This article will go through the process of integrating Inneractive ads in your Firefox OS app.

Getting Started

1. Download the library from the Github page, specifically inneractive.js.
2. Include the inneractive.js script in your HTML (or through any other script loader):

<script src="inneractive.js"></script>

3. Create an account at Inneractive. Once your account is approved you can access the console and create an App.

   

   This will generate a unique ‘App ID’ and can be found at the bottom of the Dashboard.

   

Creating the Ad

Your app should have access to the global Inneractive object. Create an ad with the function createAd.

var myAd = Inneractive.createAd(options)

options is an object where you customize the ad. The available options
are:

• APP_ID – This can be found in the previous section when creating an App through the Inneractive Console.
• TYPE – Can be one of three types of ads:
  • Banner: Small ad that is usually constant at the bottom of the screen.
  • Rectangle: Medium sized ad that is usually centered in the middle of
    the screen.
  • Interstitial: Fullscreen ad to display usually during levels or
    screens.
• REFRESH_RATE – Time in seconds between rotating ads. Minimum is 15 seconds, default is 30.

Example

var options = {
    TYPE: "Banner",
    REFRESH_RATE: 18,
    APP_ID: "Test_App_ID"
};
 
var myAd = Inneractive.createAd(options);

Placing the Ad

Once the ad has been created with your options, you need to place it on the screen.

The function addTo will place the ad into the DOM tree under a parent node. You can usually just use document.body for this:

myAd.addTo(document.body);

This will place the ad under the <body> element of your webpage.

Then you need to position the ad with the function placement. This function takes two arguments, vertical position and horizontal position where the options are top, bottom, center and left, right, center.

For a banner ad to sit at the bottom of the screen you would do the following:

myAd.placement("bottom", "center");

A rectangle that is exactly in the center of the screen:

myAd.placement("center", "center");

Removing the Ad

If you need to remove the ad from the screen for whatever reason, use the function remove.

myAd.remove();

Once the ad is removed you cannot bring it back and will need to create a new ad with Inneractive.createAd().

If you have any issues or need support using the library you can file issues on the Github Issue Tracker.

More Monetization options

With Firefox OS you may use any of your favorite ad networks that would work in a browser such as Google Adsense using their integration code.

You also have the option of integrating In-app payments for selling digital goods through your app or game.

Since then, web browsers have gown up a lot, and nowadays Mozilla is executing the vision of a browser-based operating system with Firefox OS. In any case, I’ve been digging back and dusting off some of my old apps. When you get your app running on FirefoxOS, you don’t just port it to yet another device – you port it to the web. So now, Halloween Artist runs on near anything, including those awesome (and affordable!) Firefox phones that are starting to spring up everywhere.

The platform-agnostic web app: http://halloweenartist.penduin.net
On the Firefox Marketplace: https://marketplace.firefox.com/app/halloween-artist

Play around with it before reading on, if Halloween Artist is new to you.

But enough history; the point of this post is to dive into the jack-o-lantern guts and talk about how the program actually works!

References

Before we begin, some links!

• To this day, I start here whenever I tinker with canvas
• Back in the day I only used mouse events, and just recently added actual touch support. Nowadays, I’d suggest going the other way ’round, or at least keeping both in mind.
• There are many guides for doing simple drawing programs using canvas. This one is very detailed.

Let’s go already!

The first step is to get ourselves a pumpkin image in the background, and a canvas layered on top. This canvas will track mouse and touch events and let the user trace out shapes.

Next, we need an “inside” image that will show through the carved shapes. Over that, we’ll draw the pumpkin but with the user-drawn parts cut out (made transparent). As luck would have it, the canvas API has some handy compositing modes that are perfect for these tasks. The main operation we need is “source-out”. Keep the destination image, except where it intersects with the source shape. Then it’s just a matter of doing a normal, source-over composite.

var face = document.getElementById("draw");
var faceCtx = face.getContext("2d");
var glow = document.getElementById("glow");
var dest = document.getElementById("bg");
var destCtx = dest.getContext("2d");
var img = document.getElementById("pumpkin");
 
dest.width = dest.width;  // reset the destination canvas
faceCtx.globalCompositeOperation = "source-out";
faceCtx.drawImage(img,
                  0, 0, img.width, img.height,
                  0, 0, size, size);
 
// draw glowing background (to dest)
destCtx.drawImage(glow, 0, 0, glow.width, glow.height,
                  0, 0, size, size);
 
// apply the face
destCtx.globalAlpha = 1;
destCtx.globalCompositeOperation = "source-over";
destCtx.drawImage(face, 0, 0);

It’s a start! But to look like an actual carved pumpkin, we need to add some 3D magic to draw the inside edges. Actually, scratch that – we’re going to cheat! :^) We’ll start with a lower-resolution, slightly-blown-up pumpkin image:

…then we’ll lighten it up using canvas’s “lighter” globalCompositeOperation and a globalAlpha value of, oh let’s say 0.5:

…then we’ll “source-out” the face, same as we did with the foreground:

…then shrink it a bit, center it, and draw it between the inside background and the outer face:

We’re getting there! But depending on the shape, our corners might not look very convincing.

Fortunately, all this cheating we’ve been doing – these composite operations and scaling – it’s all very fast. Even on mobile browsers. Let’s turn up the cheating to maximum and draw that middle layer in a loop, shrinking it less each time.

…While we’re at it, each step could lighten up the current layer to a lesser degree than the previous (more “inner”) one. And while we’re at that, let’s lighten it using a more yellow color; our first pass looks a little pinkish.

// build pumpkin by layer and draw it shrunk, inner to outer (to dest)
var i;
var darken = 0.4;
 
for(i = 56; i > 0; i -= 4) {
    // mix a color for the layer
    scratchCtx.globalCompositeOperation = "source-over";
    scratchCtx.globalAlpha = 1;
    scratchCtx.drawImage(glow, 0, 0, glow.width, glow.height,
                         0, 0, 768, 768);       // bright glow...
    scratchCtx.globalAlpha = 0.3;
    scratchCtx.drawImage(flick,                 // light...
                         0, 0, 16, 16, 0, 0, 768, 768);
    scratchCtx.globalAlpha = darken;
    scratchCtx.drawImage(flesh,                 // darken with flesh
                         0, 0, 256, 256, -8, 0, 768 + 16, 768);
    darken += 0.02;                             // ...more each time
    // cut out the face
    // NOTE: "face" is already the outer layer.
    //       we want to copy its alpha mask, so "-in" instead of "-out".
    scratchCtx.globalCompositeOperation = "destination-in";
    scratchCtx.globalAlpha = 1;
    scratchCtx.drawImage(face, 0, 0);
 
    // draw layer
    destCtx.drawImage(scratch, 0, 0, 768, 768,
                      i, i, 768 - (i * 2), 768 - (i * 2));
}

By putting that middle-layer step inside a loop and making a few tweaks, we get much more realistic edges:

So, we draw a whole bunch of these middle layers, only to draw right over most of it during the next pass. It’s a bit wasteful if you think about it that way, but consider how much simpler this is than trying to simulate all these arbitrary cut-out surfaces “for real”. Instead, this code builds little bits of pumpkin shell, one layer at a time, from the inside out. It’s a fairly elegant illusion if I do say so myself; we can make a passably-realistic image with a sense of depth, without actually doing any complex calculations or intensive processing.

Adding polish

Warning: more history ahead. I’ll try to keep it brief and on-point. :^)

Figuring out how “deep” to start (how much to shrink), and how many steps to draw in between, all while keeping the “carve” function reasonably fast, was one of those fun bits of experimentation and compromise. On a high-resolution display, there can still be artifacts if you draw very steep, jagged shapes. But as far as bang per buck, compatibility with low-end devices, and the 99%-of-shapes use cases, I’m pretty pleased.

I wanted this to be pick-up-and-play friendly, so every day I’d load the latest version onto my TouchPad and hand it to my coworkers, giving them no instructions.

Early on, it was suggested that the carved image should flicker as if the candle inside were burning unevenly. Easy! The carve function now produces two images, the normal one and a slightly brighter version, which is positioned (using CSS) right over the main one. It fades in and out using a randomized timeout and CSS animation on the “opacity” property. A few minutes of polish, and the illusion was even better.

.fade {
    transition: opacity 0.5s linear;
    -moz-transition: opacity 0.5s linear;
    -webkit-transition: opacity 0.5s linear;
}
.fade.out {
    opacity: 0;
}

function animateFlicker() {
    var flicker = document.getElementById("flicker");
 
    if(HA.flickerTimer) {
        clearTimeout(HA.flickerTimer);
        HA.flickerTimer = null;
    }
    if(!HA.settings.flicker) {
        return;
    }
 
    if(flicker.className === "fade out") {
        flicker.className = "fade ";
    } else {
        flicker.className = "fade out";
    }
    HA.flickerTimer = setTimeout(animateFlicker, (Math.random() * 1000));
}

I forget who, but one coworker went right for the “Carve” button before drawing anything. Natural enough instinct. So, added some logic to see if any shapes had been drawn yet, and if not, give the user some quick instructions. I’m really glad I caught that, because in showing the app to more people later, about a quarter did the same thing. Much better to show a hint popup than have users wonder why nothing’s happening.

What if the user drew outside the pumpkin? All kinds of goofy artifacts, that’s what. So, I used that handy canvas compositing and masked the user’s input before carving. As a nice side effect, you can now carve all the way out to the edge of a pumpkin, as though you’d chopped it in half.

This solution led to another problem. When people realized they could recklessly carve giant holes, they’d see the empty, glowing inside surface of the pumpkin. Where was the light coming from?

So, between drawing the background and drawing the scaled flesh layers, I dropped in a candle. I actually took some photos of a nice white candle and GIMPed it into the shape of my blocky “penduin” avatar. May as well include some kind of signature in this program, eh? I made two different versions, one for each of the flicker-fading images, and made the flame a bit bigger on the brighter version. It gives a nice little touch of animation, and adds a bit more to the illusion.

Then of course there was the less exciting stuff. Take out those nasty hard-coded values and make it work at any screen size. Make sure touch and mouse support both work as expected. Rearrange the buttons if the screen is small and they’d be in the way. All that jazz.

Code on GitHub – Happy Halloween!

Well, that about covers it. You’re free and encouraged to poke around in the source if you’d like to learn more or add your own tweaks. (Mind the mess; I left some experimental tweaks and previous-attempts in there, commented out.) Halloween Artist is GPLv3, and since you might not feel like scraping down the source from the web app itself (and since my lousy DSL might be down at any given moment) I’ve made it available on GitHub.

Have fun, and Happy Halloween! :^)

We started discussing the project with the game developers behind “Legend of Diridum” (see below) and came up with the idea of a small market place and a small jazz band on a stage. The feeling we wanted to capture was that of a market place coming to life. The band is playing a song to warm up and the crowd has not gathered around yet. The evening is warm and the party is about to start.

We call the resulting demo “Songs of Diridum”.

What the Web Audio API can do for you

We will take a brief look at the web audio system from three perspectives. These are game design, audio engineering and programming.

From a game designer perspective we can use the functionality of the Web Audio API to tune the soundscape of our game. We can run a whole lot of separate sounds simultaneously while also adjusting their character to fit an environment or a game mechanic. You can have muffled sounds coming through a closed door and open the filters for these sounds to unmuffle them gradually as the door opens. In real time. We can add reflecting sounds of the environment to the footsteps of my character as we walk from a sidewalk into a church. The ambient sounds of the street will be echoing around in the church together with my footsteps. We can attach the sounds of a roaring fire to my magicians fireball, hurl it away and hear the fireball moving towards its target. We can hear the siren of a police car approaching and hear how it passes by from the pitch shift known as doppler effect. And we know we can use these features without needing to manage the production of an audio engine. Its already there and it works.

From an audio engineering perspective we view the Web Audio API as a big patch bay with a slew of outboard gear tape stations and mixers. On a low level we feel reasonably comfortable with the fundamental aspects of the system. We can work comfortably with changing the volume of a sound while it is playing without running the risk of inducing digital distortion from the volume level changing from one sample to another. The system will make the interpolation needed for this type of adjustment. We can also build the type of effects we want and hook them up however we want. As long as we keep my system reasonably small we can make a nice studio with the Web Audio API.

From a programmer perspective we can write the code needed for our project with ease. If we run into a problem we will usually find a good solution to it on the web. We don’t have to spend our time with learning how to work with some poorly documented proprietary audio engine. The type of problem we will be working with the most is probably related to how the code is structured. We will be figuring out how to handle the loading of the sounds and which sounds to load when. How to provide these sounds to the game designer through some suitable data structure or other design pipelines. We will also work with the team to figure out how to handle the budgeting of the sound system. How much data can we use? How many sounds can we play at the same time? How many effects can we use on the target platform? It is likely that the hardest problem, the biggest technical risk, is related to handling the diversity of hardware and browsers running on the web.

About Legend of Diridum

This sound demo called “Songs of Diridum” is actually a special demo based on graphics and setting from the upcoming game “LOD: Legend of Diridum”. The LOD team is led by the game design veteran Michael Stenmark.

LOD is an easy to learn, user-friendly fantasy role playing game set on top of a so called sandbox world. It is a mix of Japanese fantasy and roleplaying games drawing inspiration from Grandia, Final Fantasy, Zelda and games like Animal Crossing and Minecraft.

The game is set in a huge fantasy world, in the aftermath of a terrible magic war. The world is haunted by the monsters, ghosts and undead that was part of the warlocks armies and the player starts the game as the Empire’s official ghost hunter to cleanse the lands of the evil and keep the people of Diridum safe. LOD is built in the Goo Engine and can be played in almost any web browser without the need download anything.

About the music

The song was important to set the mood and to embrace the warm feeling of a hot summer night turning into a party. Adam Hagstrand, the composer, nailed it right away. We love the way he got it very laid back, jazzy. Just that kind of tune a band would warm up with before the crowd arrives.

Quickly building a 3D game with Goo Engine

We love the web, and we love HTML5. HTML5 runs in the browser on multiple devices and does not need any special client software to be downloaded and installed. This allows for games to be published on nearly every conceivable web site, and since it runs directly in the browser, it opens up unprecedented viral opportunities and social media integration.

We wanted to build Songs of Diridum as a HTML5 browser game, but how to do that in 3D? The answer was WebGL. WebGL is a new standard in HTML5 that allows games to gain access to hardware acceleration, just like native games. The introduction of WebGL in HTML5 is a massive leap in what the browser can deliver and it allows for web games to be built with previously unseen graphics quality. WebGL powered HTML5 does not require that much bandwidth during gameplay. Since the game assets are downloaded (pre-cached) before and during gameplay, even modest speed internet connections suffice.

But building a WebGL game from scratch is a massive undertaking. The Goo Platform from Goo Technologies is the solution for making it much easier to build WebGL games and applications. In November, Goo Create is released making it even more accessible and easy to use.

Goo is a HTML5 and WebGL based graphics development platform capable of powering the next generation of web games and apps. From the ground up, it’s been built for amazing graphics smoothness and performance while at the same time making things easy for graphics creators. Since it’s HTML5, it enables creators to build and distribute advanced interactive graphics on the web without the need for special browser plugins or software downloads. With Goo you have the power to publish hardware accelerated games & apps on desktops, laptops, smart TVs, tablets or mobile devices. It gives instant access to smooth rich graphics and previously unimagined browser game play.

UPDATE: Goo Technologies has just launched their interactive 3D editor, Goo Create, which radically simplifies the process of creating interactive WebGL graphics.

Building Songs of Diridum

We built this demo project with a rather small team working for a relatively short time. In total we have had about seven or so people involved with the production. Most team members have done sporadic updates to our data and code. Roughly speaking we have not been following any conventional development process but rather tried to get as good a result as we can without going into any bigger scale production.

The programming of the demo has two distinct sides. One for building the world and one for wiring up the sound system. Since the user interface primarily is used for controlling the state of the sound system we let the sound system drive the user interface. It’s a simple approach but we also have a relatively simple problem to solve. Building a small 3D world like this one is mostly a matter of loading model data to various positions in 3D space. All the low level programming needed to get the scene to render with the proper colors and shapes is handled by the Goo Engine so we have not had to write any code for those parts.

We defined a simple data format for adding model data to the scene, we also included the ability to add sounds to the world and some slightly more complex systems such as the animated models and the bubbly water sound effect.

The little mixer panel in which you can play around with to change the mix of the jazz band is dynamically generated by the sound system:

Since we expected this app to be used on touch screen devices we also decided to only use clickable buttons for interface. We would not have enough time to test the usability of any other type of controls when aiming at a diffuse collection of mobile devices.

Using Web Audio in a 3D world

To build the soundscape of a 3D world we have access to spatialization of sound sources and the ears of the player. The spatial aspects of sounds and listeners are boiled down to position, direction and velocity. Each sound can also be made to emit its sound in a directional cone, in short this emulates the difference between the front and back of loudspeaker. A piano would not really need any directionality as it sounds quite similar in all directions. A megaphone on the other hand is comparably directional and should be louder at the front than at the back.

if (is3dSource) {
    // 3D sound source
    this.panNode = this.context.createPanner();
    this.gainNode.connect(this.panNode);
    this.lastPos = [0,0,0];
    this.panNode.rolloffFactor = 0.5;
} else {
    // Stereo sound source “hack”
    this.panNode = mixNodeFactory.buildStereoChannelSplitter(this.gainNode, context);
    this.panNode.setPosition(0, this.context.currentTime);
}

The position of the sound is used to determine panning or which speaker the sound is louder in and how loud the sound should be.

The velocity of the sound and the listener together with their positions provide the information needed to doppler shift all sound sources in the world accordingly. For other worldly effects such as muffling sounds behind a door or changing the sound of the room with reverbs we’ll have to write some code and configure processing nodes to meet with our desired results.

For adding sounds to the user interface and such direct effects, we can hook the sounds up without going through the spatialization, which makes them a bit simpler. You can still process the effects and be creative if you like. Perhaps pan the sound source based on where the mouse pointer clicked.

Initializing Web Audio

Setting up for using Web Audio is quite simple. The tricky parts are related to figuring out how much sound you need to preload and how much you feel comfortable with loading later. You also need to take into account that loading a sound contains two asynchronous and potentially slow operations. The first is the download, the second is the decompression from some small format such as OGG or MP3 to arraybuffer. When developing against a local machine you’ll find that the decompression is a lot slower than the download and as with download speeds in general we can expect to not know how much time this will require for any given user.

Playing sound streams

Once you have a sound decompressed and ready it can be used to create a sound source. A sound source is a relatively low level object which streams its sound data at some speed to its selected target node. For the simplest system this target node is the speaker output. Even with this primitive system, you can already manipulate the playback rate of the sound, this changes its pitch and duration. There is a nice feature in the Web Audio API which allows you to adjust the behaviour of interpolating a change like this to fit your desires.

Adding sound effects: reverb, delay, and distortion

To add an effect to our simple system you put a processor node between the source and the speakers. us audio engineers wants to split the source to have a “dry” and a “wet” component at this point. The dry component is the non-processed sound and the wet is processed. Then you’ll send both the wet and the dry to the speakers and adjust the mix between them by adding a gain node on each of these tracks. Gain is an engineery way of saying “volume”. You can keep on like this and add nodes between the source and the speakers as you please. Sometimes you’ll want effects in parallel and sometimes you want them serially. When coding your system its probably a good idea to make it easy to change how this wiring is hooked up for any given node.

Conclusion

We’re quite happy with how “Songs of Diridum” turned out, and we’re impressed with the audio and 3D performance available in today’s web browsers. Hopefully the mobile devices will catch up soon performance-wise, and HTML5 will become the most versatile cross-platform game environment available.
Now go play “Songs of Diridum”!

I’ve created a short video that shows how Open Web Apps from Firefox OS will work on any platform where Gecko is available.

Firefox OS is our benchmark platform for Open Web Apps. On Firefox OS, users can discover apps in the Firefox Marketplace and install them directly onto the phone’s home screen. As an example I’m using my app Shotclock, an open web app for computing sun angles for outdoor photographers. Let’s find out what happens when we install this app on other platforms.

Android

Android users discover apps in Firefox Marketplace using the Firefox for Android browser. Firefox Marketplace has approved Shotclock for Android, so we just click the install button as we did on Firefox OS. We will automatically repackage the Open Web App as a native Android app to give our users a native app experience for Open Web Apps.

Because we installed it from an android APK, we can manage it from the recent app list and we find it in the app drawer like every other app.

Windows

Windows users discover apps in the Firefox Marketplace using desktop Firefox. Firefox Marketplace has approved Shotclock for Windows laptops too, so we just click the Marketplace install button. We will automatically repackage the open web app as a native Windows app.

Here’s Shotclock running on Windows, just like a real app. Our repackaging will mean that users can launch their open web apps from the Windows Start menu and quit them from the File menu. Users will also uninstall them from the Programs control panel.

Mac OS X

Mac OS X users also discover apps in the Firefox Marketplace using desktop Firefox. We will automatically repackage the open web app as a native Mac OS X app. When the user clicks the install button, we install Shotclock in the Mac OS X Applications folder.

From there, it launches and runs just like a real app. The native packaging means users can switch between open web apps by pressing Control-Tab, and quit them from the File menu. How much code did the app developer rewrite? Zero.

Privileged Apps

So far we’ve looked at unprivileged Apps. We will also support privileged apps on all these platforms. Here is Kitchen Sink, our app for testing the Firefox OS privileged APIs. What happens when we install it on Android?

The experience of discovering and installing privileged apps will follow the Android convention of presenting a list of permissions to the user at install time. These permissions are copied from the open web app manifest. After the user completes the installation process, the App is ready to use, and is able to access the phone hardware.

Linux Desktop

The email application that comes with Firefox OS is basically a privileged App that uses the Socket API for networking. Marco Castelluccio, our open web apps intern, got it running on it on his Linux laptop.

He copied over the app package from Gaia and made one tweak to the app manifest. So, if you like the apps that come with your firefox os phone and want to run them on your other devices, cross-platform open web apps can make that happen.

iOS

We’d love to support Open Web Apps on iOS devices, but iOS does not, at this time, include the option to install a Gecko-based web browser, which is currently needed to support Open Web Apps.

Edit: We’re working with the Cordova community, both to allow Cordova apps to run unmodified on Firefox OS and to allow Open Web Apps packaged by Cordova to run on iOS. For more details see the Cordova Firefox OS project page and the Cordova Firefox OS GitHub repository.

Timetable

Desktop — You can install hosted, unprivileged apps on your desktops and laptops using Firefox 16 or newer. Privileged app support should land in Firefox Nightly in the next two months.

Android — You can install apps using Mobile Firefox Aurora, but you won’t get a native app experience yet. The native app experience should land in Mobile Firefox Nightly in December.

The audio tag

It started by implementing the main functionality of the app, playing an ogg stream I got from the Internet radio station, using the HTML5 audio element

<audio src="http://stream-sd.radioparadise.com/rp_192m.ogg" controls preload></audio>

That was easy! At this point our app is completely functional. If you don’t believe me, checkout this jsfiddle. But please continue reading, since there will be a few more sweet features added. In fact, checkout the short video below to see how it will turn out.

Because this content belongs to radio paradise, before implementing the app, I contacted them to ask for their permission to make a Firefox OS app for their radio station; they responded:

Thanks. We’d be happy to have you do that. Our existing web player is html5-based. That might be a place to start. Firefox should have native support for our Ogg Vorbis streams.

I couldn’t have asked for a more encouraging response, and that was enough to set things in motion.

Features of the app

I wanted the app to be very minimal and simple — both in terms of user experience and the code backing it. Here is a list of the features I decided to include:

• A single, easy to access, button to play and pause the music
• Artist name, song title and album cover for the current song playing should fill up the interface
• Setting option to select song quality (for situation when bandwidth is not enough to handle highest quality)
• Setting option to start app with music playing or paused
• Continue playing even when the app is sent to the background
• Keep the screen on when the app is running in the forground

Instead of using the HTML tag, I decided to create the audio element and configure it in JavaScript. Then I hooked up an event listener for a button to play or stop music.

  var btn = document.getElementById('play-btn');
  var state = 'stop';
  btn.addEventListener('click', stop_play);
 
  // create an audio element that can be played in the background
  var audio = new Audio();
  audio.preload = 'auto';
  audio.mozAudioChannelType = 'content';
 
  function play() {
    audio.play();
    state = 'playing';
    btn.classList.remove('stop');
    btn.classList.add('playing');
  }
 
  function stop() {
    audio.pause();
    state = 'stop';
    btn.classList.add('stop');
    btn.classList.remove('playing');
  }
 
  // toggle between play and stop state
  function stop_play() {
    (state == 'stop') ? play() : stop();
  }

Accessing current song information

The first challenge I faced was accessing the current song information. Normally we should not need any special privilege to access third party API’s as long as they provide correct header information. However, the link radio paradise provided me for getting the current song information did not allow for cross origin access. Luckily FirefoxOS has a special power reserved for this kind of situation — systemXHR comes to the rescue.

function get_current_songinfo() {
  var cache_killer = Math.floor(Math.random() * 10000);
  var playlist_url =
    'http://www.radioparadise.com/ajax_rp2_playlist.php?' +
    cache_killer;
  var song_info = document.getElementById('song-info-holder');
  var crossxhr = new XMLHttpRequest({mozSystem: true});
  crossxhr.onload = function() {
    var infoArray = crossxhr.responseText.split('|');
    song_info.innerHTML = infoArray[1];
    next_song = setInterval(get_current_songinfo, infoArray[0]);
    update_info();
  };
  crossxhr.onerror = function() {
    console.log('Error getting current song info', crossxhr);
    nex_song = setInterval(get_current_singinfo, 200000);
  };
  crossxhr.open('GET', playlist_url);
  crossxhr.send();
  clearInterval(next_song);
}

This meant that the app would have to be privileged and thus packaged. I normally would try to keep my apps hosted, because that is very natural for a web app and has several benefits including the added bonus of being accessible to search engines. However, in cases such as this we have no other option but to package the app and give it the special privileges it needs.

{
  "version": "1.1",
  "name": "Radio Paradise",
  "launch_path": "/index.html",
  "description": "An unofficial app for radio paradise",
  "type": "privileged",
  "icons": {
    "32": "/img/rp_logo32.png",
    "60": "/img/rp_logo60.png",
    "64": "/img/rp_logo64.png",
    "128": "/img/rp_logo128.png"
  },
  "developer": {
    "name": "Aras Balali Moghaddam",
    "url": "http://arasbm.com"
  },
  "permissions": {
    "systemXHR": {
      "description" : "Access current song info on radioparadise.com"
    },
    "audio-channel-content": {
      "description" : "Play music when app goes into background"
    }
  },
  "installs_allowed_from": ["*"],
  "default_locale": "en"
}

Updating song info and album cover

That XHR call to radio paradise proides me with three important pieces of information:

• Name of the current song playing and it’s artist
• An image tag containing the album cover
• Time left to the end of current song in miliseconds

Time left to the end of current song is very nice to have. It means that I can execute the XHR call and update the song information only once for every song. I first tried using the setTimeout function like this:

//NOT working example. Can you spot the error?
crossxhr.onload = function() {
  var infoArray = crossxhr.responseText.split('|');
  song_info.innerHTML = infoArray[1];
  setTimeout('get_current_songinfo()', infoArray[0]);
  update_info();
};

To my surprise, that did not work, and I got a nice error in logcat about a CSP restriction. It turns out that any attempt at dynamically executing code is banned for security reasons. All we have to do in this scenario to avoid the CSP issue is to pass a callable object, instead of a string.

  // instead of passing a string to setTimout we pass
  // a callable object to it
  setTimeout(get_current_songinfo, infoArray[0]);

Update: Mindaugas pointed out in the comments below that using innerHTML to parse unknown content in this way, introduces some security risks. Because of these security implications, we should retrieve the remote content as text instead of HTML. One way to do this is to use song_info.textContent which does not interpret the passed content as HTML. Another option, as Frederik Braun pointed out is to use a text node which can not render HTML.

With a bit of CSS magic, things started to fall into place pretty quickly

Adding a unique touch

One of the great advantages of developing mobile applications for the web is that you are completely free to design your app in any way you want. There is no enforcement of style or restriction on interaction design innovation. Knowing that, it was hard to hold myself back from trying to explore new ideas and have some fun with the app. I decided to hide the settings behind the main content and then add a feature so user can literally cut open the app in the middle to get to setting. That way they are tucked away, but still can be discovered in an intuitive way. For UI elements in the setting page to toggle options I decided to give Brick a try., with a bit of custom styling added.

User can slide open the cover image to access app settings behind it

Using the swipe gesture

As you saw in the video above, to open and close the cover image I use pan and swipe gestures. To implement that, I took gesture detector from Gaia. It was very easy to integrated the gesture code as a module into my app and hook it up to the cover image.

Organizing the code

For an app this small, we do not have to use modular code. However, since I have recently started to learn about AMD practices, I decided to use a module system. I asked James Burke about implications of using requirejs in an app like this. He suggested I use Alameda instead, since it is geared toward modern browsers.

Saving app settings

I wanted to let users choose stream quality as well as whether they want the app to start playing music as soon as it opens. Both of these options need to be persisted somewhere and retrieved when the app starts. I just needed to save a couple of key/value pairs. I went to #openwebapps irc channel and asked for advice. Fabrice pointed me to a nice piece of code in Gaia (again!) that is used for asynchronous storing of key/value pairs and even whole objects. That was perfect for my use case, so I took it as well. Gaia appears to be a goldmine. Here is the module I created for settings.

define(['helper/async_storage'], function(asyncStorage) {
  var setting = {
    values: {
      quality: 'high',
      play_on_start: false
    },
    get_quality: function() {
      return setting.values.quality;
    },
    set_quality: function(q) {
      setting.values.quality = q;
      setting.save();
    },
    get_play_on_start: function() {
      return setting.values.play_on_start;
    },
    set_play_on_start: function(p) {
      setting.values.play_on_start = p;
      setting.save();
    },
    save: function() {
      asyncStorage.setItem('setting', setting.values);
    },
    load: function(callback) {
      asyncStorage.getItem('setting', function(values_obj) {
        if (values_obj) setting.values = values_obj;
        callback();
      });
    }
  };
  return setting;
});

Splitting the cover image

Now we get to the really fun part that is splitting the cover image in half. To achieve this effect, I made two identical overlapping canvas element both of which are sized to fit the device width. One canvas clips the image and keeps the left portion of it while the other keeps the right side.

Each canvas clips and renders half of the image

Here is the code for draw function where most of the action is happening. Note that this function runs only once for each song, or when user changes the orientation of the device from portrait to landscape and vice versa.

function draw(img_src) {
  width = cover.clientWidth;
  height = cover.clientHeight;
  draw_half(left_canvas, 'left');
  draw_half(right_canvas, 'right');
  function draw_half(canvas, side) {
    canvas.setAttribute('width', width);
    canvas.setAttribute('height', height);
    var ctx = canvas.getContext('2d');
    var img = new Image();
    var clip_img = new Image();
    // opacity 0.01 is used to make any glitch in clip invisible
    ctx.fillStyle = 'rgba(255,255,255,0.01)';
 
    ctx.beginPath();
    if (side == 'left') {
      ctx.moveTo(0, 0);
      // add one pixel to ensure there is no gap
      var center = (width / 2) + 1;
    } else {
      ctx.moveTo(width, 0);
      var center = (width / 2) - 1;
    }
 
    ctx.lineTo(width / 2, 0);
 
    // Draw a wavy pattern down the center
    var step = 40;
    var count = parseInt(height / step);
    for (var i = 0; i < count; i++) {
      ctx.lineTo(center, i * step);
 
      // alternate curve control point 20 pixels, every other time
      ctx.quadraticCurveTo((i % 2) ? center - 20 :
        center + 20, i * step + step * 0.5, center, (i + 1) * step);
    }
    ctx.lineTo(center, height);
    if (side == 'left') {
      ctx.lineTo(0, height);
      ctx.lineTo(0, 0);
    } else {
      ctx.lineTo(width, height);
      ctx.lineTo(width, 0);
    }
 
    ctx.closePath();
    ctx.fill();
    ctx.clip();
 
    img.onload = function() {
      var h = width * img.height / img.width;
      ctx.drawImage(img, 0, 0, width, h);
    };
    img.src = img_src;
  }
}

Keeping the screen on

The last feature I needed to add was keeping the screen on when the app is running in foreground and that turned out to be very easy to implement as well. We need to request a screen wake lock

  var lock = window.navigator.requestWakeLock(resourceName);

The screen wake lock is actually pretty smart. It will be automatically released when app is sent to the background, and then will given back to your app when it comes to the foreground. Currently in this app I have not provided an option to release the lock. If in future I get requests to add that option, all I have to do is release the lock that has been obtained before setting the option to false

  lock.unlock();

Getting the app

If you have a FirefoxOS device and like great music, you can now install this app on your device. Search for “radio paradise” in the marketplace, or install it directly from this link. You can also checkout the full source code from github. Feel free to fork and modify the app as you wish, to create your own Internet Radio apps! I would love it if you report issues, ask for features or send pull requests.

Conclusion

I am more and more impressed by how quickly we can create very functional and unique mobile apps using web technologies. If you have not build a mobile web app for Firefox OS yet, you should definitely give it a try. The future of open web apps is very exciting, and Firefox OS provides a great platform to get a taste of that excitement.

Now it is your turn to leave a comment. What is your favourite feature of this app? What things would you have done differently if you developed this app? How could we make this app better (both code and UX)?

Receipt verification is how we ensure that your app has been paid for in the case of both hosted and packaged apps. It’s important to point out that we don’t limit the installation or distribution of apps from the Marketplace. They can be installed by anyone without limitation, they just won’t have the receipt.

That means as developers, it is essential to check the receipt in your app. This is the only way to ensure that the payment has been processed.

Receipt format

Receipts are based on the Web Application Receipt specification. A receipt is a JSON object that contains information about the purchase. That JSON is then signed using the JSON Web Token specification.

When the App is installed after purchase, the receipt is installed along with the app. To check that a receipt is installed you can access the receipt through the mozApps API, for example:

var request = window.navigator.mozApps.getSelf();
request.onsuccess = function(){
    for (let k = 0; k < request.result.receipts.length; k++) {
        let receipt = request.result.receipts[k];
        console.log('Got receipt: ' + receipt);
    }
}

Once you’ve decoded and expanded a receipt it would look something like the following:

{
  "product": {
    "url": "http://example.com",
    "storedata": "id=111111"
  },
  "iss": "https://marketplace.firefox.com",
  "verify": "https://receiptcheck.marketplace.firefox.com/verify/111111", // The verify URL
  "detail": "https://marketplace.firefox.com/en-US/purchases/111111",
  "reissue": "https://marketplace.firefox.com/en-US/app/example/purchase/reissue",
  "user": {
    "type": "directed-identifier",
    "value": "1234-abcd99ef-a123-456b-bbbb-cccc11112222"
  },
  "exp": 1353028900,
  "iat": 1337304100,
  "typ": "purchase-receipt",
  "nbf": 1337304100
}

Receipt verifier library

Just checking for the presence of the receipt is not enough, there are a few checks that should be performed as well:

• That the receipt is correctly signed and has not be tampered with
• That the receipt is from a marketplace you approve of
• That the receipt is for your app

There are two optional steps you can perform:

• That the crypto signing on the receipt is correct
• That the receipt is still valid with the store

These last steps require the app to have internet access to be able to call servers.

An easy way to perform most of these checks is to use the receipt verifier library. After including it:

var verifier = new mozmarket.receipts.Verifier({
  // checks that the receipt is for your app.
  productURL: 'http://example.com',
  // only allow apps that came from the firefox marketplace.
  installs_allowed_from: ['https://marketplace.firefox.com']
});
verifier.verify(function (verifier) {
  if (verifier.state instanceof verifier.states.OK) {
    // everything is good.
    console.log('good');
  } else {
    // something went wrong.
    console.log('wrong');
  }
});

See the docs for a full list of the options.

The receipt verifier returns a state object that tells you the exact error. We don’t prescribe what the app should do under those circumstances, that is left completely to the developer. For example a NetworkError indicates that we couldn’t connect to the verification server. That may, or may not be, a fatal condition for your app.

The receipt verifier library also includes a basic user interface for showing errors to the user. It is great for testing, but since the user interface in your app is going to be different, the chances are you’ll want to display messages back to the user in your own style. If you include receiptverifier-ui.js, then you can use the prompter like this:

mozmarket.receipts.Prompter({
  storeURL: "https://marketplace.mozilla.org/app/myapp",
  supportHTML: '<a href="mailto:me@example.com">email me@example.com</a>',
  verify: true
});

If you ran the app without the receipt installed you would see a message like this:

One last thing that the receipt verifier library won’t do is verify that a receipt has not been shared between users. That is left as something the app developer will need to implement: an example might be using the django-receipts library.

Getting help

If you need more help then there’s a mailing list or a helpful IRC channel at #marketplace.