Mozilla

Why no FileSystem API in Firefox?

A question that I get asked a lot is why Firefox doesn’t support the FileSystem API. Usually, but not always, they are referring specifically to the FileSystem and FileWriter specifications which Google is implementing in Chrome, and which they have proposed for standardization in W3C.

The answer is somewhat complex, and depends greatly on what exact capabilities of the above two specifications the person is actually wanting to use. The specifications are quite big and feature full, so it’s no surprise that people are wanting to do very different things with it. This blog post is an attempt at giving my answer to this question and explain why we haven’t implemented the above two specifications. But note that this post represents my personal opinion, intended to spur more conversation on this topic.

As stated above, people asking for “FileSystem API support” in Firefox are actually often interested in solving many different problems. In my opinion most, but so far not all, of these problems have better solutions than the FileSystem API. So let me walk through them below.

Storing resources locally

Probably the most common thing that people want to do is to simply store a set of resources so that they are available without having to use the network. This is useful if you need quick access to the resources, or if you want to be able to access them even if the user is offline. Games are a very common type of application where this is needed. For example an enemy space ship might have a few associated images, as well as a couple of associated sounds, used when the enemy is moving around the screen and shooting. Today, people generally solve this by storing the images and sound files in a file system, and then store the file names of those files along with things like speed and firepower of the enemy.

However it seems a bit non-optimal to me to have to store some data separated from the rest. Especially when there is a solution which can store both structured data as well as file data. IndexedDB treats file data just like any other type of data. You can write a File or a Blob into IndexedDB just like you can store strings, numbers and JavaScript objects. This is specified by the IndexedDB spec and so far implemented in both the Firefox and IE implementations of IndexedDB. Using this, you can store all information that you need in one place, and a single query to IndexedDB can return all the data you need. So for example, if you were building a web based email client, you could store an object like:

{
  subject: "Hi there",
  body: "Hi Sven,\nHow are you doing...",
  attachments: [blob1, blob2, blob3]
}

Another advantage here is that there’s no need to make up file names for resources. Just store the File or Blob object. No name needed.

In Firefox’s IndexedDB implementation (and I believe IE’s too) the files are transparently stored outside of the actual database. This means that performance of storing a file in IndexedDB is just as good as storing the file in a filesystem. It does not bloat the database itself slowing down other operations, and reading from the file means that the implementation just reads from an OS file, so it’s just as fast as a filesystem.

Firefox IndexedDB implementation is even smart enough that if you store the same Blob multiple files in a IndexedDB database it just creates one copy of the file. Writing further references to the same Blob just adds to an internal reference counter. This is completely transparent to the web page, the only thing it will notice is faster writes and less resource use. However I’m not sure if IE does the same, so check there first before relying on it.

Access pictures and music folders

The second most common thing that people ask for related to a file system APIs is to be able to access things like the user’s picture or music libraries. This is something that the FileSystem API submitted to W3C doesn’t actually provide, though many people seems to think it does. To satisfy that use-case we have the DeviceStorage API. This API allows full file system capabilities for “user files”. I.e. files that aren’t specific to a website, but rather resources that are managed and owned by the user and that the user might want to access through several apps. Such as photos and music. The DeviceStorage API is basically a simple file system API mostly optimized for these types of files.

We’re still in the process of specifying and implementing this API. It’s available to test with in recent nightly builds, but so far isn’t enabled by default. The main problem with exposing this functionality to the web is security. You wouldn’t want just any website to read or modify your images. We could put up a prompt like we do with the GeoLocation API, given that this API potentially can delete all your pictures from the last 10 years, we probably want something more. This is something we are actively working on. But it’s definitely the case here that security is the hard part here, not implementing the low-level file operations.

Low-level file manipulation

A less common request is the ability to do low-level create, read, update and delete (CRUD) file operations. For example being able to write 10 bytes in the middle of a 10MB file. This is not something IndexedDB supports right now, it only allows adding and removing whole files. This is supported by the FileWriter specification draft. However I think this part of this API has some pretty fundamental problems. Specifically there are no locking capabilities, so there is no way to do multiple file operations and be sure that another tab didn’t modify or read the file in between those operations. There is also no way to do fsync which means that you can’t implement ACID type applications on top of FileWriter, such as a database.

We have instead created an API with the same goal, but which has capabilities for locking a file and doing multiple operations. This is done in a way to ensure that there is no risk that pages can forget to unlock a file, or that deadlocks can occur. The API also allows fsync operations which should enable doing things like databases on top of FileHandle. However most importantly, the API is done in such a way that you shouldn’t need to nest asynchronous callbacks as much as with FileWriter. In other words it should easier to use for authors. You can read more about FileHandle at

https://wiki.mozilla.org/WebAPI/FileHandleAPI

The filesystem URL scheme

There is one more capability that exist in the FileSystem API not covered above. The specification introduces a new filesystem: URL scheme. When loading URLs from filesystem: it returns the contents of files in stored using the FileSystem API. This is a very cool feature for a couple of reasons. First of all these URLs are predictable. Once you’ve stored a file in the file system, you always know which URL can be used to load from it. And the URL will continue to work as long as the file is stored in the file system, even if the web page is reloaded. Second, relative URLs work with the filesystem: scheme. So you can create links from one resource stored in the filesystem to another resource stored in the filesystem.

Firefox does support the blob: URL scheme, which does allow loading data from a Blob anywhere where URLs can be used. However it doesn’t have the above mentioned capabilities. This is something that I’d really like to find a solution for. If we can’t find a better solution, implementing the Google specifications is definitely an option.

Conclusions

As always when talking about features to be added to the web platform it’s important to talk about use cases and capabilities, and not jump directly to a particular solution. Most of the use cases that the FileSystem API aims to solve can be solved in other ways. In my opinion many times in better ways.

This is why we haven’t prioritized implementing the FileSystem API, but instead focused on things like making our IndexedDB implementation awesome, and coming up with a good API for low-level file manipulation.

Focusing on IndexedDB has also meant that we very soon have a good API for basic file storage available in 3 browsers: IE10, Firefox and Chrome.

On a related note, we just fixed the last known spec compliance issues in our IndexedDB implementation, so Firefox 16 will ship with IndexedDB unprefixed!

As always, we’re very interested in getting feedback from other people, especially from web developers. Do you think that FileSystem API is something we should prioritize? If so, why?

117 comments

Comments are now closed.

  1. Rob wrote on March 17th, 2013 at 10:43:

    I’ll just reiterate my final thought and step out of the conversation, as I think I’ve already said everything I can say. I hope my thoughts will inspire people to do their own research and come up with an informed decision.

    My main point is that you are developing for yourself. You want local code running in a local environment with full privileges. Fine. Give them that. Hell, maybe someone should create a browser for only that purpose (hta?). It’s no skin off my back until you make that code accessible (and privileged) from a server. At that point I believe you are putting people at risk of malicious attacks that surly will happen. If you noticed, one of my first posts I stated that I WILL NOT INSTALL ANY FIREFOX PLUGINS. I understand the risk and avoid it. Possibly I’m being paranoid, but I also cannot deny the fact that people much smarter than me are actively trying to exploit browsers weaknesses as we speak. It’s not that I believe someone is actually trying to spy on me (really there is no good reason, anything of interest is already on facebook), just that there is the potential that I have no control over my privacy.

    All I want is the most stable, secure, and compatible development platform, and that is what many of us dream the internet could be. It will take multiple technologies to accomplish (html, css, javascript, php), but they are all open source and should be considered the same platform (LAMP stack). I’ve managed to avoid adobe flash this long and concentrate on open source technologies, but if you want it all in one spot maybe you should look at flash? Anyway, thank you for the intelligent conversation and the forum to speak my mind.

    Rob

    1. Brett Zamir wrote on March 18th, 2013 at 18:50:

      Thank you also Rob for your intelligent and civilized engagement, as well as willingness to express frankly your concerns.

      I also intend to leave this as the last post, but anyone wishing to continue the conversation is welcome to open an issue: https://github.com/brettz9/asyouwish/issues

      I am developing partly for myself, and partly for others. Think of addon development. It is local with full privileges, but people want to share their creations. The only difference is that AYW helps you avoid packaging files, could theoretically become cross-browser, and also lets you choose whether the addon runs on each browser restart or is just something to run when the user needs it (e.g., you could bookmark it). Yes, the code can change, but so can add-on code (as with web apps, though, perhaps some local installation mechanism could be developed to import other sites and prevent this until an upgrade is approved). If you don’t trust, don’t approve (or keep the default to not allow sites to even ask you to approve unless you manually wish to go in to add the ones you trust).

      I can appreciate your reason for avoiding installing add-ons. I think the SDK restartless add-ons (with all requiring of privileged modules being through static analysis) ought to be easier and safer to review (and AYW goes a step further to inform the user what those modules are). But despite the best efforts of even the most competent people, and automatic detection tools at AMO, I don’t think anything potentially powerful can be 100% fail-safe with human oversight.

      But if the likes of AsYouWish were added into Firefox (as I hope it could be), as long as the AsYouWish code were as rigorously studied and tested as the rest of Firefox (and if the length of the add-on approval process for AYW is any indication, it seems at least as an add-on that it will be!), this should not affect you personally, as the default behavior is not to allow any site to even ask for privileges let alone grant them.

      Your concern for others is a separate issue though because there is some potential for social engineering. But there is also a potential for social engineering if someone writes on their blog:

      How to make an egg salad sandwich:
      1. Obtain a knife.
      2. Stab your finger.

      If someone is dumb enough to follow this, should we add scanners to all websites that prevent people from reading the words “stab”?

      Or for a less extreme example, what about social engineering like:

      1. Download this exe to “upgrade” your Firefox.

      There is a spectrum for both stupidity and for protecting people from themselves. If we banned cars, we would probably protect people, but at what cost? Firefox has got along fine with some dangerous preferences in about:config that could similarly be subject to social engineering. Even if some sorry soul gets exploited for this, as regrettable and unpopular as such frankness is to say, the whole web should not be held up, imo, because of this (and I think if people are honest with themselves, they will also agree that some risk of social engineering is inevitable).

      I believe the solution is three-fold:

      1) Provide powerful and easy-to-develop/distribute functionality out-of-the-box (but disabled) so that developers and their users who wish to enable these features can quickly benefit from them.
      2) Make the default behavior paranoid to protect users from themselves and help cautious people avoid worrying about needing to make configuration changes.
      3) Make overriding of the default behavior not too trivial (but not impossible to share the steps with others who do want to pass the hurdles) and adequately warn users to minimize social engineering.

      Flash is maybe the closest, well-used analogue to what I’d like to see. As per the AYW readme though “I’m hoping this can really turn into an API agreed on by the browsers so available by default. The add-on itself is written in JavaScript, so hopefully accessible enough to easy introspection. AsYouWish has no need for special file formats, HTML object tags, etc.” Moreover, this can allow development beyond what Flash allows (e.g., to develop a browser which imports your existing tabs).

1 2

Comments are closed for this article.