Device APIs Articles

Last Friday I had the pleasure of attending and speaking at the Web Developer Conference in Bristol. This was the fifth conference in the event’s history and was attended by well over 200 Web designers and developers from across the UK.

In my talk I covered some Web technologies that are on the horizon and coming your way in the near future. Some topics of particular interest are the WebAPIs that we’re working on, with the WebVibrator API easily being the most popular amongst the attendees (no idea why).

But in all seriousness, it was a great event and I’m glad to see some of the attendees exploring these new technologies as a result of the talk.

You can check out the slides below:

As we’ve hoped, there has been a lot of interest in the newly announced WebAPI effort. So I figured that I should explain in more detail some of my thinking around what we’re hoping to do and the challenges that are ahead of us.

Goal

The goal of this effort is to create APIs to expand what the Web can do. We don’t want people to end up choosing to develop for a proprietary platform just because the Web is lacking some capability.

The main effort, at least initially, is to enable access to hardware connected to the device, and data which is stored or available to the device. As for hardware, we want to make the full range of hardware that people use available to the web platform. From common hardware like cameras, to more rarely used (but no less awesome) hardware like USB-driven Nerf cannons. We also want to enable communication hardware like Bluetooth and NFC.

For data stored on the device, the most commonly discussed data today is contacts and calendar. This includes the ability to both read and write data. That is, we both want the Web platform to be able to enumerate contacts stored on the device, and read their details, as well as add and remove contacts. In short, we want it to be possible to create a Web page or Web app which lets the user manage his contact list. Same thing for calendar events and other types of data stored on devices.

Security and Privacy

One big reason these types of APIs haven’t been developed for the Web platform yet is because of their security and privacy implications. I would obviously not want every single Web page out there to be able to mess around with my contact list or my calendar. And being able to issue any commands to any USB device that I happen to have plugged in would likely result in everyone’s computer immediately being zombified.

So as we are developing these APIs, we always have to develop a security model to go along with them. In some cases simply asking the user, which is how we currently do Geolocation, might work. In others, where security implications are scarier or where describing the risk to the user is harder, we’ll have to come up with better solutions.

I really want to emphasize that we don’t yet know what the security story is going to be, but that we’re absolutely planning on having a solid security solution before we ship an API to millions of users.

Robert O’Callahan has a really great post about permissions for Web applications.

Standards

Mozilla has always had a strong commitment to Web standards. This is of course not something we’re changing! All of the APIs that we are developing will be developed with the goal of being standardized and implemented across both browsers and devices.

But it’s important to ensure that standards are good standards. This takes experimenting. Especially in areas which are as new to the Web, and as security sensitive, as these are.

Standards organizations aren’t a good place to do research. This is why we want to experiment and do research outside the standards organizations first. But always in the open, and always listening to feedback. We’re also going to clearly prefix any APIs as to indicate that they are experiments and might change once they get standardized.

Once we have a better understanding of what we think makes a good API we will create a proposal and bring to working groups like the Device API group at W3C, WAC and WHATWG.

Throughout this process we will of course be in contact with other interested parties, such as other browser vendors and web developers. This is part of the normal research and making sure that an API is a good API.

Mozilla always has and always will be a good steward of the open Web. We are not interested in creating a Mozilla-specific Web platform. We are interested in moving the open Web platform forward.

High Level vs. Low Level

One thing that often comes up with API design is whether we should do high level or low level APIs. For example, do we provide a low-level USB API, or a high-level API for cameras?

There are pros and cons with both. High level means that we can create more developer-friendly APIs. We can also provide a better security model since we can ensure that the page won’t issue any unexpected USB commands, and we can ensure that no privacy-sensitive access is made without user approval. But high level also means that developers can’t access a type of device until we’ve added support for it. So until we’ve added an API for Nerf cannons, there will be no way to talk to them.

Exposing a low-level USB API on the other hand, means that web pages can talk to any USB device in existence, with no need for us to add an explicit API for them. However it also requires developers to get their hands dirty with the details of the USB protocol and differences between devices.

The approach we’re planning on taking is to do both high-level and low-level APIs, as well as give people the proper incentives to use the one that is best for the user. But a very important point is to provide low-level APIs early to ensure that Mozilla isn’t on the critical path for innovation. Over time, we can add high-level APIs where that makes sense.

How you can join

As with all things Mozilla, we’re planning on doing all our work in the open. In fact, we’ll be relying on your help to make this successful! As to keep discussions focused, we’re going to use the a new mozilla.dev.webapi discussion forum for all communication. This means that you can participate through email, newsgroups, or the web-based google group UI.

You can subscribe to the mailing list at https://lists.mozilla.org/listinfo/dev-webapi

For other methods go to: http://www.mozilla.org/about/forums/#dev-webapi

We also use the #webapi IRC channel on irc.mozilla.org.

We’ll also be tracking progress on the wiki page https://wiki.mozilla.org/WebAPI

Looking forward to hearing from you to help build the next stage for the web platform!

Hiring developers

Edit: Forgot to mention, we are hiring several full time engineers for working on the WebAPI team! Read the job description and apply.

Mozilla would like to introduce WebAPI with the goal to provide a basic HTML5 phone experience within 3 to 6 months.

Continued…

Mozilla recently announced the Boot to Gecko (B2G) Project which is a project towards the goal of building a complete, standalone operating system for the open web.

Continued…

This is a guest post from Tomas Barreto, a developer who works at box.net. They recently adopted HTML5 drag and drop as a way to share files with other people using new features in Firefox. The included video is a pitch for the feature and service, but shows how easy it is to do simple HTML5-based upload progress even with multiple files. Tomas gives an overview of the relatively simple JavaScript required to do this, and how improvements in Firefox 4 will make things even easier. Also have a quick look at the bottom of the post for links to additional docs and resources.

At Box.net, we’re always exploring new ways to help users get content quickly and securely onto our cloud content management platform. So when asked, “What feature would make you use Box more?” during the Box Hack Olympics in April, my colleague CJ and I decided to tackle the most intuitive way to upload files: simply dragging them from the desktop into Box.

We considered technologies ranging from Gears to Firefox plugins, but only HTML5 had sufficient adoption. By using some of the JavaScript APIs defined in the HTML5 standard, CJ and I could create a seamless drag and drop experience for our users on supporting browsers. Furthermore, using an HTML5-based upload feature would allow us to enable users to select multiple files at once, and also display progress on the client without polling. And with HTML5 adoption across the latest versions of three of the top four browsers, we felt confident about building an upload method based on this new technology without the trade-offs of using a third-party plug-in.

We rolled out the first rev of our drag and drop feature a few weeks ago, and we’re impressed with how quickly it has been adopted. It’s already one of the most popular ways to get files onto Box, and in its first week it surpassed our old AJAX upload method. You can check out our demo video to get a feel for the feature:

To build this feature, we referenced a handful of online examples that explained how to use Firefox 3 FileReader object and the drag and drop file event support. Our first implementation used this object to load the file into memory and then took advantage of the latest XMLHttpRequest events to track progress on the client.

var files = event.originalEvent.dataTransfer.files; // drop event
var reader = new FileReader();
 
reader.onload = function(event) {
  var file_contents = event.target.result;
  var request = new XMLHttpRequest();
 
  ... // attach event listeners to monitor progress and detect errors
 
  var post_body = '';
 
  .. // build post body
 
  post_body += file_contents;
 
  .. // finish post body
 
  var url = 'http://example.com/file_upload';
 
  var request = new XMLHttpRequest();
 
  request.open("POST",  url, true); // open asynchronous post request
  request.setRequestHeader('content-type', 'multipart/form-data; boundary=""'); // make sure to set a boundary
  request.send(post_body);
}
 
reader.readAsBinaryString(files[0]);

This approach worked well because we could use the same server processing code that we previously used for uploads. The main disadvantage here is that the FileReader object reads the entire file into memory, which is not optimal for a general upload use case. Our current HTML5 implementation uses this logic and has forced us to restrict drag and drop uploads to just 25mb. However, thanks to recommendations from the Mozilla team, we’ll be taking an alternative approach for V2 of drag and drop, where the file is read chunks as needed by the request. Here’s how we’re going to do it:

var files = event.originalEvent.dataTransfer.files; // drop event
var url = 'http://example.com/file_upload';
 
var request = new XMLHttpRequest();
request.open("POST",  url, true); // open asynchronous post request
request.send(files[0]);

Since this approach is not formatted as a multipart form-data, it will require some adjustments on our back-end to support receiving file uploads in this way. However, it’s definitely worth the trade-off since we’ll get all the benefits of the previous method and we don’t need special file size restrictions. In the future, we’ll consider using yet another way to efficiently upload files that is supported in Firefox 4 and uses the traditional multi-part form:

var files = event.originalEvent.dataTransfer.files; // drop event
var url = 'http://example.com/file_upload';
 
var request = new XMLHttpRequest();
 
var fd = new FormData;
fd.append("myFile", files[0]);
 
request.open("POST",  url, true); // open asynchronous post request
request.send(fd);

We’re already exploring more ways to enrich the Box experience using HTML5. With HTML5, we can build faster, richer and more interactive features with native browser support, and bridge the traditional gap between desktop software and web applications. Here are just a few cool new upload-related features on our roadmap:

• Pause/Resume uploads using the Blob slice API to split files into chunks (this will be a huge robustness boost, especially for large uploads)
• Allowing uploads to resume even after the browser closes by caching the file using IndexedDB support (possibly in Firefox 4)

We’d also like to begin a discussion about supporting the reverse drag and drop use case: dragging files from the browser to the desktop. Based on our users’ enthusiasm around the drag and drop upload feature, we think the reverse functionality would well received. If you are interested in contributing to a specification for this feature, please let us know (html5 [-at$] box.net)!

Resources:

• Using File APIs from web applications
• Using XMLHttpRequest
• Using HTML5 Drag and Drop
  

Firefox 3.6 supports multiple file input. This new capability allows you to get several files as input at once, using standard technologies. This is a big improvement, since you used to be constrained to one file at a time, or needed to use a third party (proprietary) application. This will be particularly useful, for example, for photo uploads.

The input tag

To let your user select a local file, use the input tag on your Web page. This will show the file picker to the user:

<input type="file"/>

In Firefox 3.6, the input tag has been expanded to support multiple files:

<input type="file" multiple=""/>

The user will still see the same file picker, but will be able to select more than one file.

The form tag

You can still use the classic form mechanism:

<form method="post" action="upload.php" enctype="multipart/form-data">
  <input name='uploads[]' type="file" multiple=""/>
  <input type="submit" value="Send">
</form>

If the server side code is in PHP, don’t forget to make sure that the value of the name attribute has brackets. The brackets are not from the HTML specification, but are required to manipulate the result of the request as an array (see PHP documentation).

Here’s an example, which goes through the file list and prints each file name:

foreach ($_FILES['uploads']['name'] as $filename) {
    echo '<li>' . $filename . '</li>';
}

Using File API

Firefox 3.6 also supports FileAPI. This allows you to do extra processing on the client slide before sending the files to the server. You can access the selected files with the files property of the input DOM element and then manipulate the files using the FileAPI.

For example, here’s how to get the name of each file selected by the user. This is done on the client side, unlike the previous PHP example.

  var input = document.querySelector("input[type='file']");
  // You've selected input.files.length files
  for (var i = 0; i < input.files.length; i++) {
    // input.files[i] is a file object
    var li = document.createElement("li");
    li.innerHTML = input.files[i].name;
    ul.appendChild(li);
  }

Demo

See this mechanism in action in our multiple file input demo. You’ll need Firefox 3.6 (beta).

Documentation

To learn more about multiple file input, check out the documentation on MDC.