Why no FileSystem API in Firefox?

A question that I get asked a lot is why Firefox doesn’t support the FileSystem API. Usually, but not always, they are referring specifically to the FileSystem and FileWriter specifications which Google is implementing in Chrome, and which they have proposed for standardization in W3C.

The answer is somewhat complex, and depends greatly on what exact capabilities of the above two specifications the person is actually wanting to use. The specifications are quite big and feature full, so it’s no surprise that people are wanting to do very different things with it. This blog post is an attempt at giving my answer to this question and explain why we haven’t implemented the above two specifications. But note that this post represents my personal opinion, intended to spur more conversation on this topic.

As stated above, people asking for “FileSystem API support” in Firefox are actually often interested in solving many different problems. In my opinion most, but so far not all, of these problems have better solutions than the FileSystem API. So let me walk through them below.

Storing resources locally

Probably the most common thing that people want to do is to simply store a set of resources so that they are available without having to use the network. This is useful if you need quick access to the resources, or if you want to be able to access them even if the user is offline. Games are a very common type of application where this is needed. For example an enemy space ship might have a few associated images, as well as a couple of associated sounds, used when the enemy is moving around the screen and shooting. Today, people generally solve this by storing the images and sound files in a file system, and then store the file names of those files along with things like speed and firepower of the enemy.

However it seems a bit non-optimal to me to have to store some data separated from the rest. Especially when there is a solution which can store both structured data as well as file data. IndexedDB treats file data just like any other type of data. You can write a File or a Blob into IndexedDB just like you can store strings, numbers and JavaScript objects. This is specified by the IndexedDB spec and so far implemented in both the Firefox and IE implementations of IndexedDB. Using this, you can store all information that you need in one place, and a single query to IndexedDB can return all the data you need. So for example, if you were building a web based email client, you could store an object like:

{
  subject: "Hi there",
  body: "Hi Sven,\nHow are you doing...",
  attachments: [blob1, blob2, blob3]
}

Another advantage here is that there’s no need to make up file names for resources. Just store the File or Blob object. No name needed.

In Firefox’s IndexedDB implementation (and I believe IE’s too) the files are transparently stored outside of the actual database. This means that performance of storing a file in IndexedDB is just as good as storing the file in a filesystem. It does not bloat the database itself slowing down other operations, and reading from the file means that the implementation just reads from an OS file, so it’s just as fast as a filesystem.

Firefox IndexedDB implementation is even smart enough that if you store the same Blob multiple files in a IndexedDB database it just creates one copy of the file. Writing further references to the same Blob just adds to an internal reference counter. This is completely transparent to the web page, the only thing it will notice is faster writes and less resource use. However I’m not sure if IE does the same, so check there first before relying on it.

Access pictures and music folders

The second most common thing that people ask for related to a file system APIs is to be able to access things like the user’s picture or music libraries. This is something that the FileSystem API submitted to W3C doesn’t actually provide, though many people seems to think it does. To satisfy that use-case we have the DeviceStorage API. This API allows full file system capabilities for “user files”. I.e. files that aren’t specific to a website, but rather resources that are managed and owned by the user and that the user might want to access through several apps. Such as photos and music. The DeviceStorage API is basically a simple file system API mostly optimized for these types of files.

We’re still in the process of specifying and implementing this API. It’s available to test with in recent nightly builds, but so far isn’t enabled by default. The main problem with exposing this functionality to the web is security. You wouldn’t want just any website to read or modify your images. We could put up a prompt like we do with the GeoLocation API, given that this API potentially can delete all your pictures from the last 10 years, we probably want something more. This is something we are actively working on. But it’s definitely the case here that security is the hard part here, not implementing the low-level file operations.

Low-level file manipulation

A less common request is the ability to do low-level create, read, update and delete (CRUD) file operations. For example being able to write 10 bytes in the middle of a 10MB file. This is not something IndexedDB supports right now, it only allows adding and removing whole files. This is supported by the FileWriter specification draft. However I think this part of this API has some pretty fundamental problems. Specifically there are no locking capabilities, so there is no way to do multiple file operations and be sure that another tab didn’t modify or read the file in between those operations. There is also no way to do fsync which means that you can’t implement ACID type applications on top of FileWriter, such as a database.

We have instead created an API with the same goal, but which has capabilities for locking a file and doing multiple operations. This is done in a way to ensure that there is no risk that pages can forget to unlock a file, or that deadlocks can occur. The API also allows fsync operations which should enable doing things like databases on top of FileHandle. However most importantly, the API is done in such a way that you shouldn’t need to nest asynchronous callbacks as much as with FileWriter. In other words it should easier to use for authors. You can read more about FileHandle at

https://wiki.mozilla.org/WebAPI/FileHandleAPI

The filesystem URL scheme

There is one more capability that exist in the FileSystem API not covered above. The specification introduces a new filesystem: URL scheme. When loading URLs from filesystem: it returns the contents of files in stored using the FileSystem API. This is a very cool feature for a couple of reasons. First of all these URLs are predictable. Once you’ve stored a file in the file system, you always know which URL can be used to load from it. And the URL will continue to work as long as the file is stored in the file system, even if the web page is reloaded. Second, relative URLs work with the filesystem: scheme. So you can create links from one resource stored in the filesystem to another resource stored in the filesystem.

Firefox does support the blob: URL scheme, which does allow loading data from a Blob anywhere where URLs can be used. However it doesn’t have the above mentioned capabilities. This is something that I’d really like to find a solution for. If we can’t find a better solution, implementing the Google specifications is definitely an option.

Conclusions

As always when talking about features to be added to the web platform it’s important to talk about use cases and capabilities, and not jump directly to a particular solution. Most of the use cases that the FileSystem API aims to solve can be solved in other ways. In my opinion many times in better ways.

This is why we haven’t prioritized implementing the FileSystem API, but instead focused on things like making our IndexedDB implementation awesome, and coming up with a good API for low-level file manipulation.

Focusing on IndexedDB has also meant that we very soon have a good API for basic file storage available in 3 browsers: IE10, Firefox and Chrome.

On a related note, we just fixed the last known spec compliance issues in our IndexedDB implementation, so Firefox 16 will ship with IndexedDB unprefixed!

As always, we’re very interested in getting feedback from other people, especially from web developers. Do you think that FileSystem API is something we should prioritize? If so, why?

About Jonas Sicking

Jonas has been hacking on web browsers for over a decade. He started as a open source contributor in 2000 contributing to the newly open sourced mozilla project. In 2005 he joined mozilla full time and has since been working on the DOM and other parts of the web platform. He is now the Tech Lead of the Web API project at mozilla as well as an editor for the IndexedDB and File API specifications at W3C.

More articles by Jonas Sicking…


117 comments

  1. Neel Mehta

    Great take, Robert. I agree that, for security’s sake, the FileSystem API should not be added – at least right now. There’s always the case of an app saying it’ll find any photos of kittens you have on your computer that actually deletes all your photos from the past 10 years once you grant access.

    July 5th, 2012 at 13:41

    1. Devon Govett

      You’re misunderstanding the FileSystem API. It does not, in its current form, access any files on the user’s system. It is a sandboxed filesystem, meaning it can only access the files it originally wrote, within the sandbox, not any file. There aren’t any security concerns with that. Make sense?

      July 5th, 2012 at 13:45

      1. Jonas Sicking

        Indeed. The FileSystem API proposed by Google, and implemented in Chrome, does not expose access to user files.

        Like I said above, the reason I’ve included it in this blog post is because many people are asking for that functionality, and think that implementing the FileSystem API would give them that.

        This post was intended to make things more clear, though apparently I didn’t manage to make that clear to everyone :-)

        July 5th, 2012 at 13:53

  2. Marat Denenberg

    What I really want is the ability to generate data and provide the user with a prompt to save it to a file, as if they are downloading it from a server. Like a PDF or a CSV file, generated completely client-side using javascript.

    You can do this using data uris, but it is really awkward and the suggested filename is random, without any proper extension.

    Seems like something that can be easily fixed.

    July 5th, 2012 at 13:50

    1. Devon Govett

      I have a proposal for such an API at https://github.com/devongovett/standards/issues/3 and I’d love feedback. I’ll hopefully be submitting a request for such an API to the standards bodies and mailing lists soon. Definitely a problem I want solved as well, although the current FileSystem API has a different usecase in mind.

      July 5th, 2012 at 14:01

    2. Jonas Sicking

      The FileSaver API is intended for exactly this use case. I absolutely agree that it’s something we should implement.

      July 5th, 2012 at 14:23

      1. Robert Kaiser

        Jonas, if there’s a Bugzilla bug for an API that lets us do this, please point me to it, I’d like to track that.
        From my POV, ideally it would be letting the web app propose a file name and prompting the user for where to save the file and what filename (s)he actually wants (but that can be dependent on the pref to silently save downloads or not). Then it would tell the web app the actually used file name (not the full local path for privacy reasons), and let it write data directly there in some way. Going through assembling a data URI as a long string like we have to do now is really ugly.

        July 5th, 2012 at 15:47

      2. Devon Govett

        Jonas,

        FileSaver has some serious limitations that I address in my proposal. For example, it appears that you have to have the blob entirely in memory before writing the file – you can’t do streaming writing of chunks asynchronously like FileWriter from the filesystem spec allows which would be key for large files.

        There is also no notion of write permissions for existing files. Users shouldn’t get a save as dialog box every time they need to save a file. They should be able to open a file, make changes to it, and save back to that same file without having to select it again (of course, they’d have to grant write permissions at least once). See my proposal for details on that, and let me know what you think.

        Thanks!

        July 5th, 2012 at 16:16

  3. Devon Govett

    Robert, I’m glad you took the time to write this post and to get feedback from developers! Here’s my thoughts.

    The filesystem API is really nice because, for example, it makes it easy to reproduce the file structure on your server, locally. You could, for example, make a request to the server, have the server zip up some resources, send the zip file down, and expand it into the same directory structure locally. This make it really easy to then use filesystem links in place of normal URLs to the server when your app is offline.

    Files also have metadata included in them such as last modified date, file size, etc. The FileSystem API gives you access to this information, which would make two way syncing with the server easier as well. Imagine a word processing application that stores its files in the FileSystem API and syncs them with the server. The user could have folders and the like, and the FileSystem API would make this much easier to sync two ways with the metadata about each file, as well as an exact copy of the server’s file structure.

    You could fake this with IndexedDB by storing the metadata and file paths as additional keys, but that’s kinda lame when we could have a much nicer API for it.

    Additionally, in regards to performance, the FileSystem API is nice because you can do streaming writes and reads to and from files, rather than having to build the whole thing up in memory before writing it into an IndexedDB key. For large files, this is really important as you don’t want to hog memory.

    Anyway, I’m sure I can think of more examples of where the FileSystem API is useful. IndexedDB is cool, and it’s awesome that it can store files because it makes some of this stuff shimmable, but eventually I think Firefox and other browsers should implement the FileSystem API. It has a different usecase in mind, and a very valid one at that. Thanks!

    July 5th, 2012 at 13:56

    1. Devon Govett

      Oops, realized afterward that it was actually Jonas who wrote this post, not Robert. I saw his link on Twitter. Sorry!

      July 5th, 2012 at 13:57

      1. Robert Nyman

        No worries. :-)
        Jonas wrote this great blog post and I’m glad people want to discuss the topic!

        July 5th, 2012 at 14:21

    2. Jonas Sicking

      Why do say that storing metadata and files in IndexedDB is lame?

      Looking at the resulting code, I think you’ll need a lot less code to interact with IndexedDB than you will with the FileSystem API submitted to W3C.

      July 5th, 2012 at 16:26

      1. Devon Govett

        Well for the use cases I mentioned above, you have to implement the filesystem abstraction on top of IndexedDB yourself rather than just using the FileSystem API. Stuff like directory listings would have to be written, and of course storing the metadata like file modification times would have to be written manually as well. It is entirely possible and I have even done it as a shim, but the FileSystem API is a nice abstraction that browsers could just build in. It also allows streaming writing, but it looks like you’re already working on a solution to that with FileHandle. The filesystem URLs are also nice, as I mentioned, and they rely on having a specific file path, which the FileSystem abstraction provides given the fact that it has the concept of directories baked right in.

        July 5th, 2012 at 16:42

        1. Jonas Sicking

          Good feedback. I agree that a FileSystem API does remove some abstraction logic compared to using IndexedDB if what you want is exactly a file system. I’m just not sure how common that is compared to other ways of accessing data, such the two examples given in the article (game data or web mail client).

          Definitely interested in hearing more feedback like this!

          July 5th, 2012 at 16:56

  4. Samuel Erdtman

    Interesting, A while back I tried to get the file writer API to save files sent form one user to another through websockets. At that time it was not implemented anywhere, so I ended up with another solution that worked better in the end and with less requirements on the receiving browser (http://code.google.com/p/sendnow/). This was an interesting learning experience and one of the thing that I wanted the most was to be able to save files from the browser without the round trip to the server as mentioned by Marat. So this might be the case that I find useful.
    However I hope that we will se less of files in the future. The operating system is trying to give the user a higher abstraction level of the files to the end-user. Instead of having music files we have songs, that could be one or many files and we do not need to know where hay are stored. So I don’t like that we create the file abstraction in the browser too, just so that we can abstract it away from the end users whom only wants to see her images, songs, places, friends, texts etc. Why do we so badly want to create files again why not just resources.
    Further why should I as a web developer need to care about where a resource comes from if the user is offline the browser should feed me the resource from a cache and when the user comes back online it can be fetched from its original location (of course the resource has to be marked for offline availability). With this in mind IndexedDB sounded like a really nice beginning. Regarding the worry about having to load large files into memory with the IndexedDB solution, I think that resource should not be that big, better to handle something that big as smaller resources which in turn would solve the problem with wanting to access the middle of large files.

    July 5th, 2012 at 15:12

  5. Daniel

    Let’s not minimize or sidestep a significant use-case of all File APIs on any system: CRUD of files. To pretend that powerful web applications (under the assumption that the “Web is the platform”) will not want to write to CRUD real McCoy files in the user’s folder on Windows/OSX/etc would require you to ignore reality.

    Consider the following:

    Want to make a photo editing app that saves a modded copy of a JPEG next to original? –> Can’t do it.

    Want to develop a code editor that writes or modifies code files that live in standard places on a user’s machine? –> Not happening.

    Want to create a slideshow/presentation tool that saves content to well-known formats for consumption by natives apps? –> Don’t even try.

    This is reality – refusal to acknowledge it does not make it go away.

    July 5th, 2012 at 16:03

    1. Jonas Sicking

      I completely agree. That’s why we developed the DeviceStorage API. Once fully implemented it will support CRUD in user directories. The security model is the hard part here.

      As noted in the article, the FileSystem API spec submitted to W3C doesn’t support any type of file handling in user directories, only in a browser sandbox.

      July 5th, 2012 at 16:19

      1. Angelo Borsotti

        Applications normally access the local filesystem. If we want to
        implement applications with web technologies (instead of, e.g.
        in C++ or Java), we need to have a means to access the local
        filesystem as well (beside other parts of the underlying OS).
        Security had never been much an issue for traditional applications.
        E.g. when installing Libreoffice (knowing that it does not contain
        viruses) I did not think that it would disrupt my data or tamper my
        OS. I.e. I trusted it when I installed it.
        If web apps is just a technology for implementing apps, it should
        do the same: ask me if I trust them, and if so, run with full privileges,
        as normal apps.

        September 15th, 2012 at 01:17

  6. Daniel

    Jonas – I’m not suggesting that Chrome’s current File API implementation is what we should copy, or that it does or doesn’t support real-deal file CRUDing. I’m simply saying that real file CRUDing is a platform capability we need to support in order to seriously make the claim “the Web is the platform”. Do you disagree?

    July 5th, 2012 at 16:25

    1. Jonas Sicking

      Not at all. As mentioned in the article, that’s why we have implemented FileHandle, specifically to support CRUD. It will ship in Firefox 15.

      CRUD in user directories is trickier for security reasons. But it’s definitely something that we need and that’s why we designed the DeviceStorage API.

      July 5th, 2012 at 16:31

      1. Daniel

        Well then, I’m sold!

        *** dbuc goes to cross item off bucket list

        July 5th, 2012 at 16:36

  7. Style Thing

    just for dumbification of the whole wall of text above here..
    no sandbox = no fileAPI

    July 5th, 2012 at 23:47

  8. Tane Piper

    Having written an API wrapper around this google implementation (http://github.com/tanepiper/webfs) I’ve found that is is a total pain to work with – but at the same time very handy.

    Personally I would like to see it implemented – it would provide x-browser compatibility without having to wrap more sugar around IndexDB – but I’d also like to see the draft spec be improved a lot by vendor working together on it.

    July 6th, 2012 at 01:22

    1. Jonas Sicking

      This is the feedback I hear everyone giving. That the FileSystem API is a pain to work with. It’s one of the reasons I think IndexedDB seems like a simpler solution. IndexedDB is also a solution which is likely to become x-browser sooner, no matter what we at Mozilla do.

      July 9th, 2012 at 20:50

  9. Jussi Kalliokoski

    For the game use case where you have a lot of resources like images and sounds you’d want downloaded and used locally in the future and possibly offline as well, I don’t think a file system API should be the choice, and especially not IndexedDB. Instead, I hope that the mess that is right now called Application Cache gets fixed and for a use case like this one, that would be used.

    Great post, Jonas! File system APIs are a hard nut to crack, and the safety requirements that come with the web platform do not make it easier at all.

    July 6th, 2012 at 04:18

    1. Jonas Sicking

      Yes, for cases when AppCache can solve the problem I agree it’s generally a better solution. At least once we’ve improved it to the point when people actually *like* using it :-)

      July 9th, 2012 at 20:51

  10. Oleg

    “Access pictures and music folders”, “Low-level file manipulation”, “The filesystem URL scheme”

    So instead of one solution (FileWriter and Filesystem), you present two new, different ones, which not only you not yet fully support, but no other vendors support, whereas, for third feature, you do not have any solution whatsoever.

    No API is ideal, if you can recognize that in Filesystem and in Filewriter, does not make them unusable, i would prefer, if you try to implement Filesystem first and then try yours ideas, and let us, developers, choose which one we want to use.

    IndexedDB is implemented in two other browsers, but many people thinks its very hard to use, i intend to agree with that point of view.
    And for some reason (oh, yeah, security and all that) only Firefox ask user permission to use it. For me (and many other devs and apps) is a big red flag, which is discourageable.

    I’m not saying you should mimic Chromes designs, i’m saying i don’t want write those API wrappers that i did for IE and i don’t want learn
    two (three, four, how many vendors want to create something cool?) different API’s to do pretty match the same thing.

    Big plus for Filesystem API, that it works, and works okey, for use-cases it was designed.

    July 6th, 2012 at 08:43

    1. Jonas Sicking

      I’m not sure I understand what you mean by “So instead of one solution (FileWriter and Filesystem), you present two new, different ones”.

      Surely IndexedDB + FileHandle is a smaller set of solutions than IndexedDB + FileSystem + FileWriter.

      I agree that IndexedDB is hard to use. But not harder than FileSystem. And if all you need is to store resources rather than doing low-level file manipulation then IndexedDB seems outright simpler to use. But I concede to being biased :-)

      By the way, starting with Firefox 16, we will no longer prompt for IndexedDB usage.

      July 9th, 2012 at 22:04

      1. Eric Uhrhane

        I think a more apples-to-apples comparison is
        IDB + FileWriter + FileSystem + some very small API to get you access to “My Photos”
        vs.
        IDB + FileHandle + DeviceStorage API

        At that point, it’s clear that they’re actually fairly similar in scope. I’m assuming that you’re trying to process user media files in some way; if you’re just storing transactional data, or you want to edit large files in a sandbox, the answer would change one way or the other.

        I include IDB on both because 3 major browsers are already implementing it, so it’s there if you want it for transactional data.

        July 10th, 2012 at 16:55

    2. Jonas Sicking

      Ugh, I forgot the most important question.

      You mention that you prefer to use a FileSystem API to using IndexedDB. Could you elaborate on why that is?

      July 9th, 2012 at 22:05

  11. Eric Bidelman

    In working with devs using the Filesystem API, the “storing metadata alongside File/Blob data” isn’t a common requirement that comes up. Most people just want a more versatile AppCache. What folks also really want is an incarnation of the FileSaver API. +1 to that!

    As you point out though, one major benefit of the W3C Filesystem API are its filesystem: URLs. Not having to write ANY code to use a locally cached resource is extremely attractive. Combine this with folder hierarchies, and you’ve got yourself a complete offline solution to replicate the server’s file structure. Pretty nice.

    From my personal experience tinkering around with a Filesystem API polyfill (https://github.com/ebidel/idb.filesystem.js), it wasn’t trivial to use IDB to organize data in logical folders. Also, later on, nuking an entire “folder” involved setting up tricky key ranges and getting ordering right. The whole thing felt icky.

    July 6th, 2012 at 09:23

  12. Eric Uhrhane

    Re: the filesystem API only granting access to a sandbox, not e.g. “My Photos”.

    Sort of–99% of the API doesn’t care what’s underneath (whether you’re talking to a sandbox or the real filesystem). For obvious security reasons, the only way for a web page to obtain a FileSystem object at present is to ask for access to the sandbox. We’ve implemented extension APIs that grant access to the full filesystem–that’s what ChromeOS uses for its file browser–but we don’t expose them to the web platform yet. However, the rest of the API is exactly the same, as it was designed with the intention that we’d eventually figure out how to expose real directories to the web safely.

    We think it makes sense to use this same API to solve multiple problems, rather than invent a narrow DeviceStorage API for just this use case. A filesystem is a very general, familiar API; it’s not necessarily the best thing for all use cases, but it’s pretty decent for quite a few. As you point out, we just need to solve the security problem first.

    Re: FileHandle for more efficient read-modify-write with locking. It’s a pity the discussion of this problem petered out on public-webapps; I thought it was a pretty interesting proposal, and we’re certainly open to talking about how to merge it in to our FileSystem implementation. I don’t know of any apps having problems with the speed of the current API, so I don’t know how much the potential efficiency gains would buy us, but I really like the locking model.

    July 6th, 2012 at 17:10

    1. Jonas Sicking

      Hi Eric!

      Yes, we should definitely continue the FileHandle discussion!

      I agree with you that if we should have an API for exposing a sandboxed filesystem to web pages then we should use the same API for exposing user files as for exposing the sandboxed filesystem.

      July 9th, 2012 at 22:18

      1. Brian Stell

        Doesn’t a persistent url (eg, the filesystem url) offer much lower latency to the data? For example, using a persistent url couldn’t one load a css, javascript, or other resources during parsing? My understanding is that once one has a blob url it is just as fast as a persistent url but … doesn’t the blob url arrive in a callback? What is the earliest a time callback made during parsing will reliably return?

        March 21st, 2013 at 16:17

        1. Jonas Sicking

          Getting a blob URL is really fast once you have a Blob. Just do

          url = URL.createObjectURL(blob);

          No async callbacks needed or anything.

          However if the Blob is stored in a database, then it’ll be an async action to get the Blob yes.

          So yes, the filesystem URL scheme definitely has advantages. As discussed in the “The filesystem URL scheme” part of the article above.

          March 22nd, 2013 at 11:38

          1. Brian Stell

            Ah right, a callback is needed to get the blob not the URL. Thx :-)

            Agreed that the filesystem URL has advantages. Any plans to implement it (or something similar) in IDB?

            March 22nd, 2013 at 13:26

          2. Jonas Sicking

            Yes, it’s something that I’d love to solve. It’s not trivial though. Filesystems have an obvious mapping between a URL and a storage location, since a filesystem is basically a database keyed on a path, and URLs have a clear path.

            And the value stored at a filesystem is always a File which makes it easy to map to a network response.

            However IDB keys things off of a much more complex concept of keys, which makes mapping a URL to an IDB location is much more difficult. And the values stored in IDB are arbitrary structured clones, which makes them harder to map to a network response.

            Not saying that it’s impossible, just saying that is less obvious how to do it.

            I’d love to have a conversation about it, but I don’t think this is the location. The W3C webapps mailing list would be the most obvious choice. Or feel free to contact me directly.

            March 22nd, 2013 at 14:10

  13. pd

    Off the top of my head, you might be missing the one simple reason why people want the filesystem API: because they see Chrome as the most appropriate browser to target, perhaps with the easiest APIs to use, so they target that and want to do cross-browser easier than re-writing their work.

    Or to put it more succinctly: Firefox doesn’t rule the roost in innovation terms anymore. Mozilla just might find itself having to cave in to the standardisation push of a more dominant innovative browser for the first time in it’s life.

    July 7th, 2012 at 00:44

    1. Jonas Sicking

      Most web developers are interested in a solution that works across multiple browsers. The FileSystem API is pretty far from this as last I heard only Google had expressed interest in implementing it. Meanwhile IndexedDB is implemented in three browsers with a forth one on its way.

      But I agree that if you want is exactly a file system, then the FileSystem API has a nicer ring to it than a database API.

      Mind elaborating on why you are looking for a file system API though?

      July 9th, 2012 at 22:29

      1. Eric Uhrhane

        I have to agree with Jonas on this one. If you have to pick a single browser to target, you’re not writing a web page, you’re writing an app. That’s not necessarily bad, but we’d really like there to be a powerful web platform to target, that would run on any browser, for stuff that doesn’t require the priveleges granted to an app.

        Obviously we’ll be happier if other browsers adopt the FileSystem as well, but we’re willing to put it out for experimentation to try to prove its utility, to work toward convincing them. Likewise, other browsers are putting forward their own experiments [such as FileHandle]; no one browser team “rules the roost in innovation terms”.

        July 10th, 2012 at 17:01

  14. John Thomas

    I think part of the appeal of a file system api (without commenting on the Google spec) is it is very intuitive to work with if you are using files or interact with the file system. There are a wide range of web apps that do so or that attempt to do so from document editors to games that want to save state. If you are encouraging developers to work in offline mode, then they are going to be thinking about a file system api.

    And if I could bring up an essential limitation of IndexedDB in this regard – it is not cross-browser shareable (nor should it be for users who say use one browser for work and one browser for play or want one browser on their computer to be light and nimble while allowing other browsers to become memory heavy). If I am loading and saving a file, I will not want to always use it in that browser, or even necessarily in that application. If you consider how many applications are frustrating because you want to export/import between applications and they do not let you, I think you can see the advantage of having a storage medium independent of the browser.

    Of course, if you wanted to go with something really intuitive, browsers could implement some variation of the POSIX file api (admittedly adapted to JS constraints). Yes, it is not Webby but a large number of programming languages have realized that even if the POSIX file api does not necessarily fit their dominant paradigm, it is still useful to support it because most developers do not work with only one programming language and most will be familiar with this api.

    This is the same essential argument behind DOM. Given that the spec authorities seem to have rejected that approach in favor of JS-oriented apis, I doubt my advice will be followed.

    July 10th, 2012 at 05:58

  15. Eric Uhrhane

    One other thing, regarding fsync/flush. I agree completely that we need to add that to FileWriter [and possibly FileSystem–I need to check the implementation details]. That’s a small and easy change, that just needs a bit of thought to get the semantics exactly right.

    July 10th, 2012 at 18:41

  16. Joran Greef

    The heart of the matter is that Mozilla’s approach to innovation in the browser today is very much top down. Browser vendors get together, decide what high level features they want to provide (UndoManager, IndexedDB etc.) and then spec it out via committee. Developers get to chime in on the Public Web Apps list. We end up with hundreds of sub-par APIs in the browser, and bugs that get filed for these APIs take years to get fixed.

    For example, it may be a few more years before IndexedDB supports binary keys, or reading ranges from a binary value, or concurrency semantics beyond the notion of “readers-block-readers” or “readers-block-writers”.

    Much more innovative would be for browser vendors to open up and move towards bottom-up innovation. To provide just 3 powerful low-level APIs and trust the developer community do the rest.

    For example, if the user would be allowed by the browser to empower a web app with raw POSIX, then that would unleash an explosion of databases running in the browser, orders of magnitude better and faster than IndexedDB.

    The crux of the matter is that browsers need to “allow” me as a user to empower web apps that I trust. If I buy a powerful machine with UDP, TCP and POSIX, then I want to be able to delegate that power to web apps that I trust. And if a web app abuses my trust, then I want to be able to revoke that trust.

    If the browser won’t trust me as the user and enable me to empower web apps in this way, then the browser is ultimately not doing what I want it to do, it’s not serving me, and web apps will not be able to compete.

    Tim Berners-Lee first raised this point on Public Web Apps (http://lists.w3.org/Archives/Public/public-webapps/2012JanMar/0464.html).

    July 19th, 2012 at 00:48

    1. Jonas Sicking

      The high-level vs. low-level is something where different developers strongly feel different.

      On one hand you are saying that IndexedDB is too high-level and if you could only get a low-level API like POSIX you could do something much better.

      On the other hand the main complaint about IndexedDB that I hear over and over is that it’s too low-level and that a simpler API like asynchronous localStorage would have been better.

      The fact that it’s low level enough that you can implement a FileSystem API on top of it, and a asychronous-localStorage API on top of it, yet high level enough that you can actually use it directly in many cases, tells me that we probably weren’t too far off.

      Regarding POSIX: With FileHandle you have a very low-level API for doing file handling. It should have all the primitives needed in order to implement a database on top of it. So I’m awaiting your better and faster database API on top of it :-)

      Regarding TCP/UDP: I’d love to support that! The main problem is security. Exposing a raw TCP API would both allow reading data behind corporate firewalls, as well as allow using website visitors as clients for doing DDoS attacks or distributed brute force password hacking.

      July 19th, 2012 at 01:54

      1. Joran Greef

        You skirt the original point that innovation in the browser is currently top-down, whether it be FileHandle, WebSocket, IndexedDB, WebRTC you name it. Each of these is just a limited abstraction over an OS API. If Mozilla were truly open, it would expose these.

        Sadly, web apps are locked up tight inside the various spec committees. Innovation should not be made to go through that kind of process at all in the first place.

        Alan Kay says it best:

        http://www.drdobbs.com/architecture-and-design/interview-with-alan-kay/240003442?pgno=2

        Alan Kay: “…what you definitely don’t want in a Web browser is any features.”

        Interviewer: “Any features?”

        Alan Kay: “Yeah. You want to get those from the objects. You want it to be a mini-operating system, and the people who did the browser mistook it as an application. They flunked Operating Systems 101.”

        Interviewer: “How so?”

        Alan Kay: “I mean, look at it: The job of an operating system is to run arbitrary code safely. It’s not there to tell you what kind of code you can run. Most operating systems have way too many features. The nice thing about UNIX when it was first done is not just that there were only 20 system commands, but the kernel was only about 1,000 lines of code. This is true of Linux also.”

        The web platform needs progress to be decentralized as much as possible, not centralized. Put the minimum OS APIs necessary out there (TCP, UDP, POSIX) along with a way for a user to say they trust the web app, and then get out of the way as quickly as possible and let the community write the databases and the P2P code and the “FileHandle” and the “IndexedDB” and all the rest. Don’t try and bundle that as part of the browser. Let the next-generation of library authors take care of it.

        Regarding your point that the web should be denied UDP because “the main problem is security”, there is nothing inherently insecure about UDP being used by trusted apps, and the world has worked fine so far like that.

        It must be 1984 that one can use a browser, but not be allowed to give raw TCP or UDP or POSIX annointing to web apps that one trusts.

        July 19th, 2012 at 03:19

  17. Benz

    For our use case the filesystem api works really well and i don`t really see how your recommended alternatives would fit. We have a website which offers video learning material. People can decide to download videos and watch them offline. These video can be 500mb big. This would be impossible with the IndexedDB cause it`s really not an option to have the 500mb files in memory to read it from a blob url or?
    The second part of our application is a video editor that you can use on and offline. The filesystem api allows us to let the user choose video files from his harddisk and then once in the sandbox he can do all kinds of manipulations before uploading the final piece. Again those files can be of several GBs. I don`t think blob urls would work here. Also i think for those types of apps where you deal with files anything else than a filesystem api is another abstraction that`s not needed. Ideally a file should be treated as a file and nothing more or? I would love to show you the app (it`s currently in beta and not yet accessible) cause i think it shows really well why a filesystem api is really powerful and also to get some inspiration of you how it could maybe be done without it. So if you like i could give you a rundown of the app through a hangout screen sharing session…

    October 20th, 2012 at 13:07

    1. Jonas Sicking

      IndexedDB stores files on the disk just like the filesystem API does. Nothing is kept in memory. And they are stored outside of the database so the size doesn’t affect the performance of the database at all.

      A blob-url is just a 30 byte long string, so you can definitely load GBs of data through it. And the video code has enough smarts that it’s able to skip to any part of the file just as if it was reading from a file directly.

      The file parts of the files that are stored in IndexedDB act just like a file which is stored in a filesystem. The only thing that is different is navigating the directory structure. So most things should feel exactly the same.

      October 21st, 2012 at 10:13

      1. Benz

        Hey Jonas,

        thx for clarifying this. I always assumed that you`ll need to load the complete file in memory in order to access it through a blob url. So now that i know better i`ll give it a try. If it works that might change my view towards this topic completely :-) So then it seems the only missing feature for me is the filesystem url scheme as you mentioned in your article. It would be really nice to be able to predict the urls of a certain blob somehow.

        October 21st, 2012 at 11:04

  18. Daniel O’Callaghan

    Any thoughts regarding folder uploads?
    The filesystem api supports recursive iteration through folders/subfolders where you can get a reference to each object, optionally copying it into persistent storage in order to support resumable uploads etc.
    Is this something that can be implemented without the filesystem api?
    Thanks

    January 3rd, 2013 at 21:13

  19. Michaela Merz

    You know what? I personally couldn’t care less about the details on how to achieve the perfect way of doing things. I find it ridiculous to be able to upload data via xhr, but not to download and store data in the real world. Everybody is talking to get rid of flash, java. But if the shit hits the fan, we still need it to be able to export data.

    There are plenty of reasons why we need xhr downloads to get data into the javascript environment, modify it and finally allow the user to save it into a location of his choosing.

    The people @ Chrome seem to understand that.

    January 13th, 2013 at 18:26

    1. Jonas Sicking

      It would be helpful if you actually read the article. You seem to be missing several points:

      * The FileSystem API available in Chrome only permits storing files in a sandboxed filesystem which is not accessible to the user.
      * The FileSystem API does not permit saving files to a location of the user’s choosing.
      * There are solutions for storing xhr downloads client side. These solutions work in both Firefox and IE10. Opera is also working on implementing this, and the Chrome team has said they are planning on adding support too.

      January 14th, 2013 at 18:23

      1. Michaela Merz

        Jonas: This is not quite correct. Any file in the sandbox is easily converted in to a download with the filesystem:// URL.

        Michaela

        January 17th, 2013 at 15:45

        1. Jonas Sicking

          Sure. That is true for files stored in IndexedDB too. Except that you use a blob: URL.

          January 17th, 2013 at 16:49

      2. Michaela Merz

        All I am aware of are blob wirlh cryptic filenames.

        January 18th, 2013 at 01:37

  20. Michaela Merz

    Jonas: Don’t ge me wrong. I do appricate the work. But please keep in mind, that browsers are becoming universal tools. And we must be able to use it as such. As I wrote earlier: This is not a competition of which way is better. I don’t care if it is filesystem or IIndexeddb + filewriter + saveAs function. But, as of now, there is no ‘saveas’ function, just cryptic blogurls that won’t give me the chance to save the data under a specific name.

    January 18th, 2013 at 01:51

    1. Jonas Sicking

      Ah, what you are missing with blob URLs is the ability to specify a suggested filename?

      If so, have a look at recent nightlies which support the new attribute. This attribute is generally a better way of triggering download attributes since it allows you to trigger a save-as dialog for files of all types.

      A plain will only trigger a save-as dialog if the URL returns a file type that it doesn’t recognize. So you can’t use it for images unless you are sending a content-disposition header, which doesn’t work with neither filesystem URLs or blob URLs.

      January 18th, 2013 at 02:13

      1. Jonas Sicking

        Arg, that should say:

        Ah, what you are missing with blob URLs is the ability to specify a suggested filename?

        If so, have a look at recent nightlies which support the new <a download> attribute. This attribute is generally a better way of triggering download attributes since it allows you to trigger a save-as dialog for files of all types.

        A plain <a href=”…”> will only trigger a save-as dialog if the URL returns a file type that it doesn’t recognize. So you can’t use it for images unless you are sending a content-disposition header, which doesn’t work with neither filesystem URLs or blob URLs.

        January 18th, 2013 at 02:16

  21. Michaela Merz

    Jonas: One can of course trigger a forced download with Chrome’s filesystem call. As a matter of fact, I believe it to be imperative to be able able to download data into a file – and not to display it’s content within the browser.

    So – I think the “a download” attribute might be a good way to approach the issue at hand. Even if it means, that we will have to re-code the complete download part again.

    Hope to see it in the Moziila ‘official’ version soon. Any idea, how long this is going to take?

    Michaela

    January 18th, 2013 at 02:50

    1. Jonas Sicking

      I don’t know what you mean by “One can of course trigger a forced download with Chrome’s filesystem call”? Links to documentation would be very interesting.

      the <a download> attribute will be in Firefox 20 which should be released around April 2nd. It’s already implemented in Chrome I believe.

      January 18th, 2013 at 03:00

  22. Benz

    After getting more familiar with the IndexedDB i start to really like it and don`t miss much functionality from chrome`s filesystem api. The only thing that`s really missing is a predictable link to the file (blob) like the filesystem url scheme. As a start it would be nice if the blob: url could stay valid for the live of the blob once created so one does not have to use createObjectURL for each browser session.
    Benjamin

    January 18th, 2013 at 04:03

  23. Michaela Merz

    Well – after a day of heavy fiddling, I can confirm that uploading, downloading and storing (sort of, bloburl) arbitrary files with small and large sizes is working fine – though for whatever reason the browser completely croaks sometimes on filesizes > 50Mb. Haven’t found the reason yet.

    Chrome seems not to have a filehandle API, so I use the filesystem there. I don’t have an Internet Explorer (who has? ;) so – no testing.

    Now – if we please could get the download attribute ;)

    Thank you Jonas for your work.

    January 18th, 2013 at 14:25

    1. Jonas Sicking

      Note that you don’t need to use the FileHandle API if all you are doing is downloading files and saving them. Simply stick the Blob in the database just like any other type of value, like a string or a boolean.

      You only need to use FileHandle if you want to modify the contents of the after it has been saved to disk.

      January 18th, 2013 at 14:57

  24. Michaela Merz

    Thanks Jonas. I am aware of that. However, I am modifying the data (filtering), so I do need the file handle API.

    Another question: Is there anything that could run out of bounds with multiple ( > 40) xhr requests or so each posting 3 Megs of data? I am getting all kinds of funny behavior, from disappearing variable contents to complete crashes.

    Reminds me on my old ‘C’ days .. ;)

    Michaela

    January 18th, 2013 at 15:06

  25. Benz

    @Michaela: I did some tests with IE10 and to my surprise it handles this stuff pretty well. I use it to store 300mb video files and it works like charm. Firefox also seems to do well but unfortunately can`t play my h264 files. We use a flash fallback in firefox but flash can`t access blob: urls :-) Funny times…

    January 18th, 2013 at 15:29

  26. Michaela Merz

    @Benz Just Up- and downloaded 121886616 bytes with Mozilla and lots of low level file modifications in between. I am quite happy, almost make it to native up- and download speed, even with the low-level modifications.

    Oh .. and the crash is a result of firebug being open on upload.

    m.

    January 18th, 2013 at 15:43

  27. Geoff Flarity

    Meanwhile, back on earth, the iOS and Android API architects are laughing… Open web standards development has become a joke. Too much bike shedding, not enough progress.

    January 19th, 2013 at 06:54

  28. maxw3st

    I would definitely stick with the indexdb solutions you have already, and add to them where possible. The whole URL/file system discussion smacks of something best handled by a database for URL’s with those URL’s indexed to their associated files.

    Multiple URL’s associated with the same file could be indexed to allow the storage of one instance of that file rather than 1 for each URL.

    February 1st, 2013 at 21:46

  29. Michaela Merz

    Weird .. after a long day of playing around with indexddb, it just gave up on me. Blocked on open .. no way to make it come back. No errors. just no results or callbacks either. Restarted the browser and all was fine.

    February 4th, 2013 at 13:02

  30. John McLane

    So you’re saying Mozilla knows better than the W3C who are working on the FileAPI standard and other browser vendors who are implementing it?

    February 9th, 2013 at 09:48

  31. Anthony Caudill

    I smell blood in the water. The emperor wears no clothes.

    I just learned last weekend that my game creation system webapp which relies on nsiFile privileges to load files is now nonfunctional because enablePrivilege was disabled in Firefox 17. Since when did Mozilla become Big Brother? I thought that was M$’s prerogative. I winced when I heard Bing was available as a … really that’s beyond the pale. Do these people have any psychological awareness at all? I guess I should have realized then that Mozilla was in a downward spiral from within. Kinda makes me mad… I supported you all these years and made a bet that you’d beat MS, and make the browser the end-all be-all of computing, and now you turn your back on portability and compatibility. If this is what I can expect from Firefox OS, count me out.

    I’m expecting a shakeup at Mozilla is in the offing, as I see the seeds of a rebellion by coders. Here’s some advice for those emerge from the ashes: focus on what users want. We want HTML 5, as WHATWG envisions it. Nothing more, nothing less. We want an end to browser-specific prefixes. We want game pad support put on the front burner… really I’ve been waiting for that for like a year now. I bothered to try programming a demo and all I got for my trouble was was a failed read of my Logitec Dual-Action, which I regularly use to play games outside the browser. If you can’t read the stock Walmart controller what can you read? Where are Mozilla’s priorities?

    Not with me, I guess. I will be watching the fireworks from afar… sometimes progressive orgs get taken over by idealistic, detached idiots, and have to wander in the wilderness for a while in self-inflicted penance. I guess for Mozilla this is one of those times.

    Oh, and I’m goin’ back in Firefox 16. Supposed security flaws be damned.

    February 14th, 2013 at 08:52

  32. Jonas Sicking

    It appears you were not at the last TPAC where all browser vendors, except google, said that they were not planning on implementing the current FileSystem API proposal.

    You might also not have noticed that the FileAPI is completely independent from file systems, is implemented by Firefox, and is edited by me :-)

    February 14th, 2013 at 20:40

    1. Jonas Sicking

      To clarify, Arun is doing all of the heavy lifting on the editing of the FileAPI spec. But I’m heavily involved in that spec still, so don’t worry, we’re keeping up-to-date on it.

      February 14th, 2013 at 20:45

    2. Anthony Caudill

      But FireWriter API is not, and that’s where my complaint lies. You broke my app. It’s my understanding that enablePrivilege support will return in FF 19 (according to Bugzilla).

      Developer requirements matter. You have no authority to take out essential user interface features. If you want to assert it, fine. People will react as they will. I’ll enjoy reading on CNET about how you got kicked out and replaced with someone who was ready to get right down to buisness.

      I am quite perturbed that I needed to interlocute with you at all. I thought FF was the browser that “had my back”. Instead it’s trying to protect me from myself. You have no right.

      This spec has been in limbo for years. Whatever your station others had the idea before you. Get off your asses and get something done.

      February 14th, 2013 at 23:40

  33. Michaela Merz

    @Anthony Caudill I have been supporting Mozilla from the very first moment on when they promised to send me a big poster with all supporters. Even contributed cash in the name of my dog and my parrot to have all the family together on that poster (which I never got). But – in the defense of the guys and gals working to get this browser going: It’s probably a tough job to get everybody happy. I couldn’t care less about a game controller or the support of Apple’s display, but I desperately need the ‘download’ attribute for links to be able to store blobs from indexeddb under the correct name.

    But they’ve got only two arms (I suppose) – so – unfortunately, I guess I have to wait (and have to suggest Chrome to our customers for the time being). I hate that .. but what you’re gonna do?

    @Jonas: Any change that feature is going to get bumped up a bit? Would love to show our new system on the CeBIT trade fair with Firefox ;)

    Michaela

    February 15th, 2013 at 05:39

    1. Jonas Sicking

      The download attribute was implemented in bug 676619 and will be shipping in Firefox 20 in early April.

      February 15th, 2013 at 06:13

  34. Michaela Merz

    @Jonas This is great news. Too late for the CeBit though :) Thanks.

    February 15th, 2013 at 06:28

    1. Jonas Sicking

      It’s already available in nightlies and on the aurora channel, so you can demo it already. Just not push it to end users yet.

      February 15th, 2013 at 07:12

  35. Rob Higgins

    Sorry to add noise when you’ve basically already said it all, but come on, please give me a cross-browser filesystem api. I’m most comfortable working in PHP and communicating with my client json/ajax. I see the client filesystem as a cloud endpoint that should sync with the server.
    The sandbox is our biggest advantage, and the only security scheme I’ll support. If ANY website EVER asked me for access to my local file system (including chrome/fox extensions) I will instantly navigate away from that page and tell my family/friends to stay away. Seriously? Open source code running on a 3rd party server that is passing through countless computers is going to have access to my user files? Uh no. Put your time into writing code that will benefit the server/client situation, not complicate it.

    February 21st, 2013 at 18:18

    1. Anthony Caudill

      Well Rob I wouldn’t. Firefox has had apps that access the file system for years. So you’re not exactly in the majority judging by the success of those apps.

      I’d really prefer that the author put aside their pissing match with their counter-part at Google, because nothing else appears to be working. We need a standard. Maybe not a perfect standard, but a working standard. I’m highly tempted to start advocating to Firefox customizers that they go rogue, because this standards process seems increasingly co-opted by moneyed interests. Where in the past academics were overseeing everything, I see no indication that is still the case. It’s what now, Moz, Google, and MS doing all the actual development standardization? If a fourth, “I’ll take my ball and go home if you don’t play fair” player is what the people need, it’s what they need, and no one is more prepared to make that make than the custom builders.

      March 4th, 2013 at 10:27

  36. Angelo Borsotti

    The browser is the means that I use more frequently to access data and in general computing services. It has a reach and programmable user interface (HTML+javascript). When I need to implement a new application I can use traditional technologies such as C and Java plus some user-interface libraries, or I can do it with web pages. The latter is more appealing because I can then access applications as web pages, just tabs in the browser window. It is also more appealing because it relieves me from learning other languages and libraries. The only real obstacle is that web pages cannot access the local computer (e.g. the local filesystem) as traditional apps can do.
    However, with Firefox that can be done with the AsYouWish add-on.
    This means that from now on I can easily implement portable applications, that I can run on any computer that has Firefox.

    March 4th, 2013 at 11:49

  37. Rob

    I realize that I am in the minority here, but personally I do not install ANY firefox plugins (http://www.lifehacker.com.au/2011/01/how-writing-firefox-extensions-can-scare-you-away-from-them/). Every single chrome extension I install I first check the privacy policy, and if it asks for anything out of the ordinary I do not install it. There is a very good chance that web development could become the be-all standard of programming, but in order for that to happen leading browsers like ff have to get their privacy controls sorted. Someday I will not be in the minority, people will realize how insecure the web is, and demand comprehensive controls over their data flow. Why wait until something bad happens? Or worse, why kill the ultimate IDE that the browser could become, before it even has a real chance?

    March 4th, 2013 at 22:34

    1. Brett Zamir

      That is one reason in AsYouWish I inform the user of the specific privilege or privileges being requested.

      March 13th, 2013 at 02:51

  38. Michaela Merz

    Angelo Borsotti: You can do that with Firefox – sorta. I am shuffling GBs of data in and out the file system using Firefox and indexddb. No need for add-ons or plugins. In other words: Firefox can access local files just fine.

    I wish Javascript would support directories too – but that is another story.

    With the upcoming Release 20, Firefox’s file system API will be complete. But it works ok (sans “Save as” capability).

    Unfortunately, I also had to add Chrome file system to support both universes. Sounds like the early days of web programming :)

    Kind regards
    Michaela

    March 5th, 2013 at 08:21

    1. Angelo Borsotti

      Michaela: I understood that this API allows to access files in a sort of sanboxed filesystem or a dedicated database, not to access plain local files that can be seen also by other applications. Is this not longer the case?
      -Angelo

      March 5th, 2013 at 11:37

      1. Jonas Sicking

        Both IndexedDB and the Google FileSystem API only provides access to a sandbox yes.

        We are working on exposing other parts of the filesystem using the DeviceStorage API. But the security and privacy issues are really thorny. See the “Access pictures and music folders” section of the article.

        March 5th, 2013 at 11:45

  39. Michaela Merz

    Angelo: Well, if you are asking whether you are able to access, transfer, download and store arbitrary data – yes you can. You are able to select a file, access and transfer it via xhr, download and store it via indexddb and finally export the blob into the real world.The only thing currently missing (but will be included in Mozilla 20) is the “Save as” functionality – in other words: the final step currently gives you fancy blob ‘file names’ such as ‘g670xixN.part’

    Michaela

    March 5th, 2013 at 14:17

    1. Angelo Borsotti

      Michaela, I am asking if a script in a page can read a file without having to display a file chooser to the user and letting her/him select the file name, and can write a file without asking the user permission to do it (e.g. displaying a Save As dialog box).
      This is what a traditional app can do, and also a web page using the AsYouWish addon.

      March 5th, 2013 at 16:42

  40. Michaela Merz

    Angelo: No. AFAIK this is not possible (and it is not possible in Chrome either). As a security specialist, I would consider this HIGHLY dangerous and would always advise against such technology.

    Michaela

    March 5th, 2013 at 18:14

    1. Angelo Borsotti

      Michaela: my computer is mine, and the data in it are mine to manage.
      No one else can advocate the right to protect them. Firefox is taking the
      position to use the “fatherly care” on users’ behalf (see, e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=546848).
      I think this is not correct. Firefox should provide only tools, and tell users what might be dangerous and what not. Then it is up to users to handle them, dangerous as they could be.
      When I download and run a native app I accept the risk of tampering my
      data, and cope with it using only apps from some trusted provider.
      When I use a web app I want to do plainly the same.

      Angelo

      March 6th, 2013 at 00:54

  41. Michaela Merz

    Angelo: This is why there are addons. I appreciate Firefox’ efforts to protect the common users data.

    Michaela

    March 6th, 2013 at 01:30

    1. Angelo Borsotti

      Michaela: I appreciate that effort too. However, protection must not overgrow to blocking the user.

      March 6th, 2013 at 02:25

  42. Rob Higgins

    I completely agree with Michaela Merz. Addons are for those that want to extend privileges beyond the security model of the browser. A fresh install of FF should not allow any website access to my computer under the assumption that I trust it to do so, for the same reasons that it should not allow cross-domain access. Modern browsers after a fresh install SHOULD have file system support, but each domain isolated in their own sandbox.

    People that do not see the issue with this are either ignorant or advocating evil behavior. Everything that you are trying to do with user files could be accomplished in a much more secure way (from the server).

    The main difference between Chrome and FF is that Chrome specifically tells you what privileges you are authorizing, I really hope FF goes this route in the future.

    March 6th, 2013 at 13:21

    1. Angelo Borsotti

      Rob: Any install (be it fresh or not) of ff must not access my computer.
      It is only the pages containing apps that I explicitly state as trusted
      that can do that.
      Whether the functionality of accessing the computer is provided
      natively by ff or by an addon is another issue.

      March 6th, 2013 at 16:59

      1. Rob Higgins

        Angelo: That is where it gets messy, native or plugin. My vote is for keeping the browsers security model intact, and leaving high risk operations to add-ons. I don’t believe my grandmother knows what rights she’s giving up when she clicks a random “accept” button, and neither does the majority of internet users. Does my grandmother know how to install a firefox plug-in? Probably not.

        I guess my stance is based on the concepts I’m used to. If you think of the browser as an operating system, and plug-ins as applications, you can see where the line should be drawn. Out of the box the operating system does not allow 3rd party code privileged access. “Installing” the plug-in would grant privileges, and deleting or disabling the plug-in would revoke them.

        The Chromebook is a fantastic product and a firebook could be too. Browsers in the future can be fast and efficient operating systems, but in order for that to happen we need comprehensive security controls that can be turned on and off at will. Sure, you can do that with a whitelist, but what it really comes down to is encouraging better programming that doesn’t put it’s users at risk.

        March 6th, 2013 at 19:04

        1. Angelo Borsotti

          Rob: I do not object to using addons too. This is exactly what AsYouWish does: when you install it you are able to open pages that perform privileged operations (after asking you grant…).
          Another solution would be to have this integrated into the browser, to be enabled explicitly by the user. This would prevent inadvertent granting of rights.

          March 7th, 2013 at 00:50

  43. Anthony Caudill

    The anti-sandbox argument seems stupid to me. I mean really, how hard is it to limit a local page’s file access to its own folder and subfolders? I’ve seen the source for Firefox and it’s not that hard… someone with experience working with the source could implement this in a single evening. I might have already managed it except it’s not my call, is it?

    Back before the invention of the registry, it was standard practice for 16-bit apps to place all the files they accessed in one folder. Hell Moz itself mostly still does this. It would be trivial to do a test and say, “ok this file’s path does not contain the path of the calling document, therefore it is unavailable for reading”. This would work fine for local file access by local files.

    Regarding access to local files by websites, you could provide the option to make a folder just for that site. The issue is creating a recognizable name… for this you’d have a command in Firefox that would open the site’s designated folder upon request by the user. The user doesn’t have to know the real name of the folder… they only need the ability to see the files in it and to export/import them.

    That, as I see it, is the solution to your problem. Devs shouldn’t be given access to My Documents… that’s for user eyes only. The solution I just gave you is, I argue, the only tenable answer.

    (…I’m at my wits end here. Chrome is getting worse and Firefox is getting paranoid. Please fix.)

    March 12th, 2013 at 23:18

    1. Anthony Caudill

      The method of creating the file dialogue popup must be considered. If Moz is going to offer a standard popup for this purpose, that’s fine. Its appearance could be customized via CSS. The user could change between folder views a la Windows… it’d work just like Windows, ideally, but depending on device constraints could just be a vertical list of file names and their sizes and dates of creation.

      Don’t force this on the web designer… users don’t really care to have animated fairies talking to them and flying about as they search their folders for the files they need to get their work done. (yes a small vocal group does but their pithy concerns should not delay the receipt of functionality.)

      March 12th, 2013 at 23:35

  44. Brett Zamir

    At least one use case not mentioned:

    Creating a generic file desktop interface. There is a pathetic lack of extensibility of the desktop interface, and even on OS’ where more variety exists, one cannot necessarily use the likes of JavaScript to do it, and where one can, it is probably not standardized nor portable to other OS’. On the Mac a while back, there was a cool 3d OS interface, for example. Using enablePrivilege, in some early programming experimenting of mine, I got a Mac-style column browser working (at http://subversion.assembla.com/svn/brettz9/colbrowser/ — not yet adapted for AsYouWish). Lots of possibilities for innovation and even extensibility.

    Until we can do EVERYTHING in the browser (including designing one’s own browser UI with desktop interface itself)–with carefully designed strongly disabled-by-default options, it is not really free, imo. I think real freedom would be the ability to hack everything using the same language and syntax (i.e. JS). Always aim for the best, imv, and then worry about how to do it safely, rather than throwing hands up and saying it can’t be done. It can be done and has been though–though right now that is all controlled by those who designed Windows, *nix, and the Mac (or have the skills to hack them). Firefox OS sounded like it might be a positive way to bring that control to the average developer at least on mobile, but neither its browser nor desktop appears to be add-on extensible (even though it is hackable).

    By the way, thank you to Jonas for addressing this issue of concern to many of us, and sharing your plans in (high level) detail.

    March 13th, 2013 at 03:25

    1. Brett Zamir

      Another use case apparently not covered by the filesystem: feature (though maybe by the DeviceStorage API if the types are extensible) is the ability to read from or store data in a predictable namespace/location BEFORE the file is created. For example, if I were to create and publish a standard for storing file format data (let’s say game data), but wanted other developers to be free to innovate independently on their UIs, I’d have to enable my own site to work with postMessage, and developers would have to depend on me to get access to that data.

      It’d be like if LibreOffice had to go through Microsoft’s website each time it wanted to access the user’s OOXML files.

      It seems like the DeviceStorageAPI might address this. I would think it’s security objections could be assisted in part by granular control. For example, as a user, if I were explicitly asked by a site to grant the ability to delete photos (or if I had the option to confirm any time certain types of privileges were about to be employed–e.g., if I wanted confirmation for each delete usage but not read usage), I may be able to make a more informed decision based on my trust level with the requesting site. In addition, it could help sites provide different levels, or start with lesser levels of, access, e.g., asking for permission only to create new files would let a site not have to overcome as high of a trust barrier to be useful.

      March 13th, 2013 at 03:46

    2. Rob Higgins

      Brett Zamir, respectfully, I COMPLETELY disagree with everything your trying to accomplish. If any browser gave access to my system I’ll delete that browser. Who are you developing for? Yourself. You are trying to make a desktop that only you would use, because only you would trust yourself with that kind of power (or your one of the many that doesn’t know any better).

      A FAR smarter idea is to extend your home server to your client netbook/tablet, and create a desktop there (I’ve done it, works beautifully). Any access to the file system is done on the server. PERIOD. You have to understand that any desktop “service” code is hosted on a different computer than your own. If the code is hosted on a different server than your own, it can change at anytime. If code can change at anytime, it can easily be used for malicious purposes. Ever heard of a ‘man in the middle’ attack?

      It seems like you want to write code that runs locally. Then why use a browser? Is it because that is the language you already know? Not a good answer, you will confuse your users, and put them at risk.

      March 14th, 2013 at 11:08

      1. Rob Higgins

        Also, I do not want ‘freedom’ from my browser, I want security. The implementation your asking for will never be installed on any corporate network. Then guess what, we’re stuck with IE 9 for another 10 years (at least they’re not dumb enough to alienate their target audience).

        March 14th, 2013 at 12:09

        1. Anthony Caudill

          Brett you used Firefox for years and it gave your browser more access to your file system than any other browser (except Opera).

          March 14th, 2013 at 12:21

          1. Brett Zamir

            I think Tony meant to address you, Rob, here, and it is true. Firefox (as with IE) gave the browser access to the file system. Firefox also gave access to any other API (password manager, etc.) as long as there was user approval. And again, even today without AsYouWish there still is full privileged access available to any remotely hosted extension that asks you to install (as I just mentioned do Chrome/Safari/Opera as well). It may be wise to require the extra step that Chrome now does — dragging the downloaded file to the extensions tab (making it harder for social engineering), but they didn’t stop the extensibility of their own browser by trying to be the exclusive nanny of what extensions can be installed using their browser, and I hope they never will.

            March 14th, 2013 at 16:49

        2. Brett Zamir

          To address this point first, IE became part of corporate networks for years with a technology called ActiveX which does exactly what I want I am re-enabling Firefox to do (though in what I hope might become a more potentially cross-browser manner, as well as more granular in informing you what kind of privs are being requested). The likes of ActiveX should not be turned on by default (perhaps even to ask permission), but it is part of the IE browser and has been for years (as have hta applications).

          Google Chrome allowed remote extension installation (though it requires an extra step now), Opera and Safari do, and so does Firefox. How are these any different, except for the fact that the developer has to manage a file packaging process? Failing to make this available from the browser is only doing one thing: restricting developer and user convenience and choice. That’s all. There’s no real net decrease in security here.

          And actually, from the sound of things and without surprise to me, it would appear some of the users frustrated by the loss of enablePrivilege are small companies who the benefit in hiring low cost web programmers to create privileged applications for their specific needs in a familiar language.

          March 14th, 2013 at 16:37

          1. Rob Higgins

            Our systems are locked down pretty tight, our users can not use activex controls on websites unless first enabled from an administrative account. Can firefox offer this kind of administrative control? (serious question, I don’t know) In the case of sharing a computer, I would think the plugin architecture (if it were to list explicitly what rights are given like chrome) would be the most transparent way to accomplish this, lift the burden from the developer, and give people like me a piece of mind (it’s not that I trust Moz, I just trust Moz more than I trust you). But I guess it all comes down to personal preference.

            March 14th, 2013 at 16:50

  45. Angelo Borsotti

    Rob: Brett is not developing for himself. There is at least one other person
    for which he is developing: ME.
    I have use what Brett accomplished, it does not decrease the security
    of the firefox I am using, and allowed me to implement apps using web
    technologies, without which I had instead to revert to proprietary, less
    portable technologies.

    March 14th, 2013 at 13:17

    1. Rob Higgins

      So Angelo, where is this code hosted? On your computer, or on Brett’s? Or even worse, on a go-daddy type 3rd party server?

      March 14th, 2013 at 13:50

      1. Angelo Borsotti

        Rob: the app is hosted on my computer, but I could host it on any
        web-serverd computer that I trust. E.g. could be hosted on Brett’s
        because I trust him.
        P.S. I downloaded MS Office from A Microsoft server. I know that
        Office can access my files, and potentially wreck any of them.
        I use it because I trust Microsoft Office.

        March 15th, 2013 at 00:08

  46. Brett Zamir

    1. Who I am developing for? I am developing for myself and for any others who appreciate what I am creating. Even if you were correct that no intelligent person would trust me personally to design their desktop interface (though thank you for reminding me to adapt my old column browser code as another demo), what is so wrong about providing the opportunity to others to do so and let their users choose? People install privileged executable files or remotely-hosted extensions precisely because they ARE willing to make a calculated risk or placement of trust in a source they have hopefully found some evidence of trustworthiness for and don’t want to be confined in their experience. I’m just making it one step easier for developers (and one step can make a lot of difference).

    I don’t want to be personally limited, nor do I think any developers should be limited (including those developers for whom I create tools) to have to install a server. Besides, Firefox itself has some server capabilities as it is.

    2. Yes, I am aware that remotely hosted code can change at any time and can be used maliciously. This is why AsYouWish by default does not allow any website to request access of users and gives strong warnings inside door hanger notifications when the user does allow sites to request access. And while it is true that bona fide add-ons have the advantage of allowing perpetual installation of a single version of an add-on which can still be limited from automatic updates (unlike AsYouWish sites or “addons” currently which give perpetual live access until you uninstall), a first time installation of a Firefox extension could similarly install malicious code independent of the add-on (or otherwise do its damage). The only real benefit of preventing updates is for those likely select few who fully review the source code of each version of a remotely-installed extension personally. I recommend this, but there are sources (like one’s friends or companies that have established themselves with respectability) where the non-paranoid may wish to give a little trust (as do all executable file users).

    Yes, it is certainly convenient for others we trust to do some vetting work for us (especially for code hosted at sites where we cannot easily make an informed decision about trustworthiness but which may offer useful programs), which is why I hope Firefox will allow such capabilities by default, though making similar restrictions on sites that can request access, such as allowing only AMO by default (but without the hassle of needing to package extensions in a browser-specific format).

    3. As far as man-in-the-middle attack, yes I am familiar with the potential, which is why AsYouWish only enables https or file protocols by default and puts big red warnings next to the options to change the default.

    4. As far as why use a browser to do this, there are a number of reasons:

    a) Yes, it allows me to use one of the languages I know. I don’t see any inherent advantage in having yet another syntax–even an old MIT-full-scholarship friend remarked to me at one time on the annoying interference he faced between different syntaxes.

    b) It allows use by developers who only know one language, yes (which again, I do not think is a bad thing). If there is going to be any snobbery about development, it should be about knowing a language well, not knowing multiple languages (as peripherally nice as the latter can be for getting ideas for adapting concepts into one’s own language and granting one access to other environments). It’s like human language; yes, learning more languages will let me understand others’ ideas but that doesn’t mean the world couldn’t spare all of us the need (though should not prevent us the option) to struggle with languages after our prime, in adopting a convenient world language for use alongside our native languages (just as individual nations have done to unite their people while allowing continued local language use). The AsYouWish wiki moreoever seeks to warn developers that such creations can be dangerous for their users (again, as with add-ons which can already do the same thing)–something I think which needs to be given greater attention in web extension documentation.

    c) JavaScript is a great language (albeit with a few big warts) and it would be nice if the likes of the SDK APIs could become standard across browser too. As with web standards, there is no INHERENT reason why we should have to say “apple” (as in the fruit) in 100 ways. Unfortunately, pressure on the browsers for unifying extension APIs is probably less than on web APIs because of a smaller developer base and because some seem to be resigned to this situation (and even browser standardization took some time). I don’t know if that is because some of the browsers are dead set against this, or whether it is just because of a lack of trying or perceived developer interest.

    d) Besides running locally, I like having the option of offering (or benefiting from others’) trustworthy creations hosted remotely.

    e) Perhaps the most compelling reason–using a browser also offers the benefit of providing potential access to APIs (not just the file API) which integrate with the browsing experience. If I want my desktop browsing app to let me launch the same bookmarks manager shared with the browser, I can do so. Or if I want to implement my own browser interface, I should be able to do so.

    And off topic: Another use case for file access beyond a web sandbox: a Git server implemented in JavaScript for use by online IDEs.

    March 14th, 2013 at 18:21

    1. Brett Zamir

      A couple more points I neglected to mention:

      1) SSJS offers some of the benefits I mentioned, but not all (and besides, the SDK I am using is CommonJS-friendly anyways).

      2) Unlike browser addons, AsYouWish at least exposes granular awareness of requested permissions to the user (Safari at least shows access types under a “Privacy” button AFTER an install, but AYW does so before approval).

      March 14th, 2013 at 18:31

  47. Michaela Merz

    Time again, to throw my two cents into the round. We are data security professionals and, believe me, the situation is already bad. Even with all protection mechanisms in place and explicit “do not use the Internet for private purposes” regulations, we find all sorts of malware, viruses, trojans, explots and other bad stuff on plenty of computers in corporate environments. Obviously, it’s just too tempting for some folks.

    We are currently suggesting to NOT allow people access the web at all (in their workplace) – Instead installing dedicated computers in lobbies, cafeterias etc. that are wiped every day.

    From my professional point of view, we have to create WAY more security, we must find ways to completely disallow access to the file system – even for downloads, caches and plug-ins. We just can’t keep up with modern, highly professional methods of exploits. If this continues, we might see more and more companies withdrawing from casual Internet browsing.

    So – my suggestion would be: Firefox should have a mode that allows us to completely block all up- and download, harddisk-caching and module (add-on) installation – protected my an administration password.

    Again – just the opinion of somebody who tries to clean up the mess, users leave on their workplace computers every day.

    Michaela

    March 15th, 2013 at 02:25

  48. Rob

    Brett, thank you for your comment, I’m glad we can debate our sides in an intelligent and civil mannor. I can tell you feel passionately about your opinion, as do I, and totally understand where your coming from.

    The best advice I can give to a developer that wants unrestricted access to a file system is to learn PHP. It is my opinion that doing that kind of work on the server is the only viable option.

    I admit I know nothing about Node.js, but if you do not feel comfortable learning a new syntax maybe that is where you should be looking. Node.js is all javascript server side code. I’d still recommend PHP over node just because it is a well established open source standard, and there is a lot more documentation online.

    As for git repositories, again, use the server. IMO, it’s the easiest, most cross-browser, most profitable, and most secure way to do all of things you are trying to do. The hardest part about servers is setting up your LAMP stack and Apache settings, but in the long run well worth it for coding real apps on the internet. I believe the server should be responsible for the security of your app, not passing off that task to the front end developer.

    March 16th, 2013 at 13:22

    1. Brett Zamir

      Yes, I like and use PHP. I like it for:
      1) what it lets one do
      2) being, as you say, cross-browser
      3) its non-deeply-namespace nested global API (friendly for beginners and sometimes lighter on memory load, though there are some annoying inconsistencies in the API (e.g., “str” vs. “string”)).

      But none of these reasons justify it indefinitely over JavaScript, including Mozilla as a platform. To the extent I want to use its API (#3), we have http://phpjs.org/ . Mozilla’s platform is already very powerful (#1), and as far as #2, I do agree that can be remedied by encouraging and inviting others to standardize on a cross-browser extension API (or if not, building a bridge to do so, though that would require some active maintenance).

      Besides, as Angelo may have been getting at in his response re: performance (not to mention convenience for oneself or others one wishes to train), why add another layer?

      As far as your argument on security, PHP can clearly also be exploited. I do agree with you that wherever possible, the task of security should not be passed off to the front end developer, but to the extent it is possible, it should be just as doable to enhance the SDK APIs (e.g., having the default behavior for any inclusion of text escape for XSS) as it would be to do the same for PHP (e.g., using a framework which overcomes PHP’s own weaknesses).

      Free your mind, and realize the oneness of all things. :)

      March 16th, 2013 at 18:51

  49. Angelo Borsotti

    Rob: Let’s compare the solution you are suggesting vs the one Brett (and me) are suggesting, as far as I can understand your proposal.
    Yours: implement apps with a client-server paradigm with a browser hosting the client side (html+javascript, and I guess, ajax) and a web server hosting the server-side part (php, nodes.js, etc.).
    Ours: implement apps using a browser (html+javascript) that has an addon installed in it.
    Clearly the second is a lot cheaper, both in terms of the languages that one has to learn, and the amount of code one has to write, not to consider the performance and the need to install and configure a web server.

    March 16th, 2013 at 15:34

  50. Brett Zamir

    Actually, come to think of it, the “standardize on a cross-browser extension API” item should probably include join in with CommonJS.

    March 16th, 2013 at 18:53

Comments are closed for this article.