Mozilla

Beyond HTML5: Database APIs and the Road to IndexedDB

IndexedDB is an evolving web standard for the storage of significant amounts of structured data in the browser and for high performance searches on this data using indexes. Mozilla has submitted substantial technical feedback on the specification, and we plan to implement it in Firefox 4. We spoke to prominent web developers about evolving an elegant structured storage API for the web. While versions of Safari, Chrome, and Opera support a technology called Web SQL Database, which uses SQL statements as string arguments passed to a JavaScript API, we think developer aesthetics are an important consideration, and that this is a particularly inelegant solution for client-side web applications. We brought developer feedback to the editor of the IndexedDB specification, and also spoke with Microsoft, who agree with us that IndexedDB is a good option for the web. With additional implementations from the Chrome team in the offing, we think it is worth explaining our design choices, and why we think IndexedDB is a better solution for the web than Web SQL Database.

Web applications can already take advantage of localStorage and sessionStorage in IE 8+, Safari 4+, Chrome 4+, Opera 10.5+ and Firefox 2+ to store key-value pairs with a simple JavaScript API. The Web Storage standard (encompassing localStorage and sessionStorage), now widely implemented, is useful for storing smaller amounts of data, but less useful for storing larger amounts of structured data. While many server-side databases use SQL to programmatically store structured data and to meaningfully query it, on the client-side, the use of SQL in a JavaScript API has been contentious.

SQL? Which SQL?

Many web developers certainly are familiar with SQL, since many developers touch just as much server-side code (e.g. PHP and database operations) as client-side code (e.g. JavaScript, CSS, and markup). However, despite the ubiquity that SQL enjoys, there isn’t a single normative SQL standard that defines the technology. In particular, SQLite supports most of SQL-92, with some notable omissions, and is what the WebDatabase API is based on. But SQLite itself isn’t a specification — it’s a release-ready technology! And the best definition of what constitutes the supported subset of SQL that SQLite uses is the SQLite manual. In order to really get Web SQL Database right, we’d have to first start with defining a meaningful subset of SQL for web applications. Why define a whole other language, when more elegant solutions exist within JavaScript itself?

The Benefits and Pitfalls of SQLite

We think SQLite is an extremely useful technology for applications, and make it available for Firefox extensions and trusted code. We don’t think it is the right basis for an API exposed to general web content, not least of all because there isn’t a credible, widely accepted standard that subsets SQL in a useful way. Additionally, we don’t want changes to SQLite to affect the web later, and don’t think harnessing major browser releases (and a web standard) to SQLite is prudent. IndexedDB does not have this problem; even though our underlying implementation of IndexedDB may be based on SQLite, we keep developers insulated from changes to SQLite by exposing an API that isn’t based on SQLite’s supported syntax.

Aesthetics and Web Developers

Last year, we held a summit at the Mozilla campus to discuss storage on the web. We invited web developers to speak to us about a desirable structured storage API on the web. Many did express resigned acceptance of a SQLite-based API, since they had already experimented with releases of Web SQL Database in some browsers, and claimed that something in circulation was better than a collection of ideas. Yet, all voiced enthusiasm for better design choices, and how a simpler model would make life easier for them. We watched as developers whiteboarded a simple BTree API that addressed their application storage needs, and this galvanized us to consider other options. We were resolved that using strings representing SQL commands lacked the elegance of a “web native” JavaScript API, and started looking at alternatives. Along with Microsoft, we sent feedback about the IndexedDB proposal and actively became involved in the standardization effort.

In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter. IndexedDB leaves room for a third-party JavaScript library to straddle the underlying primitives with a BTree API, and we look forward to seeing initiatives like BrowserCouch built on top of IndexedDB. Intrepid web developers can even build a SQL API on top of IndexedDB. We’d particularly welcome an implementation of the Web SQL Database API on top of IndexedDB, since we think that this is technically feasible. Starting with a SQL-based API for use with browser primitives wasn’t the right first step, but certainly there’s room for SQL-based APIs on top of IndexedDB.

We want to continue the discussion with web developers about storage on the web, since that helps us structure our thoughts about product features and future web standards. We look forward to seeing the next generation of web applications with support for high performance searches on indexed data, and to seeing web applications work even more robustly in “airplane mode.”

Links

117 comments

Comments are now closed.

  1. Mats wrote on May 14th, 2011 at 13:39:

    Hi – a swede is calling.

    I have been into webdev for some years and recently stumbled over Web SQL Database. And it make it possible for me to write apps that I have been dreaming of for mobile devices. As a fan of FireFox I was really dissapointed that there was no support for this in my Firefox 4.

    So, now I am FORCED to use Chrome in my developing.
    Please, re-think and start support Web SQL as soon as possible.

    Cheers from Mats

  2. Christian R Simms wrote on May 17th, 2011 at 12:35:

    I don’t have much to contribute, just that I’ve used sqlite for several years in a server-side app, and it’s really solid. Solid enough that if all the browser vendors adopted its use, it would be a good thing. Dr. Hipp did a great job developing it. And no client-side layering on top of IndexedDB will be as fast, plus will take at least a year to get solid.

  3. david karapetyan wrote on June 25th, 2011 at 00:54:

    None of what you said is holding up in the current implementation. The stuff that is documented in MDC is anything but elegant compared to SQL. All I see is a more imperative and clunky implementation of a small subset of SQL with none of the elegance of the declarative SQL syntax. Plus, who was the genius that thought of separating requests and transactions?

  4. Guido Leenders wrote on July 10th, 2011 at 04:52:

    I am happy to see that parties are working together to find a standard solution across all browser for midsized storage. That finally allows html apps to work when offline. Not all regions in the world currently have 100% coverage.

    I can imagine that the current status of the SQL standard is insufficient and semantics should further be cleared. But as far as I can recall from my university days, SQL is based upon solid semantics from set theory.

    The alternative (Indexed DB API) is very low level and I strongly encourage our own developers to work on specifications that map directly into code instead of many low level operations and with of course too many forgotten exception handlers. Sure developers can be trained, but with training needs comes a lower bang for the development buck.

    I would hoera it when you Mozilla would reconsider to add Web SQL to Firefox and parties would continue working on the Web SQL standard.

    1. Bob H wrote on August 16th, 2011 at 19:32:

      I have to agree, this seems to be an attempt to make data storage in web applications as difficult as possible. I just read the specification, I am surprised someone didn’t just go a step further and ask web developers to code data functions within ADA+ASM.

      Web SQL seems like a far better solution and the arguments presented here by Mozilla really don’t stand up when you think about this from a web development perspective. Perhaps from the perspective of someone who uses C++ every day IndexedDB looks good, but most web developers aren’t doing that and this is overly complex.

      I agree with the previous posters, step away from this effort.

  5. Pingback from HTML5本地存储不完全指南 on August 20th, 2011 at 22:18:

    [...] 最后我们要介绍的就是IndexedDB了,相比其它两个规范,目前只有Firefox实现了IndexedDB(顺便提一下,Mozilla表示它们永远不会去实现Web SQL Database),不过Google已经表示正在考虑在Chrome中加入IndexDB支持。 [...]

  6. Pingback from HTML5本地存储不完全指南 | 网络大学 on August 24th, 2011 at 02:25:

    [...] 最后我们要介绍的就是IndexedDB了,相比其他两个规范,目前只有Firefox实现了IndexedDB(顺便提一下,Mozilla表示它们永远不会去实现Web SQL Database),不过Google已经表示正在考虑在Chrome中加入IndexDB支持。 [...]

  7. klkl wrote on August 26th, 2011 at 16:14:

    Good decisoin!

    I appreciate that you stand for IndexedDB, even though most developers unfamiliar with web standards process are unable to understand depth of the issue.

  8. Alek Paunov wrote on September 5th, 2011 at 14:19:

    Gentleman,

    Complex applications (like evolving IDEs as instance) need a real DB (sqlite or other), not just key-value API without query language.

    It is obvious which solution is better:
    * it is easy for everyone to emulate IndexedDB using WebSQL (with exactly same performance)
    * it is impossible for anyone to emulate WebSQL on top of IndexedDB – (as “Aesthetics” argument absolutely irresponsible suggests)

    Current official Moziilla position on this decision (which is very, very influential for the shape of the Web in the next few years) is just a horrible mistake, which need to be corrected.

  9. Leigh Harrison wrote on September 20th, 2011 at 01:23:

    This really is an unfortunate decision.

    Writing complex software for off-line use finally appeared possible with the Web SQL DB. Having carefully considered IndexedDB it doesn’t even start to be an viable alternative, both in terms of its feature set and its interface. Writing a second abstraction layer on top of IndexedDB to emulate the underlying SQLite interface is crazy, and the performance hit would be unacceptable.

    It will be sad not to include Firefox among the target browsers for future projects. I sincerely hope Mozilla will rethink this decision.

  10. Greg wrote on September 21st, 2011 at 11:25:

    I would encourage folks to contact the Mozilla advisory board and/or individuals within Mozilla (even Brendan Eich) to complain about the decision to exclude WebSQL; from looking at these posts at least, they seem to be out-of-sync with the development community.

  11. ed wrote on October 24th, 2011 at 04:13:

    thanks for not giving developers what they need. You could give us web sql while you think of and develop better solutions. Standards are nice but we have work to do.

  12. Gary Schaecher wrote on October 30th, 2011 at 19:07:

    Sounds like a last ditch effort by two companies late to the party who want to spoil the hard work and productivity of all who making a goog living with WebSQL and SQLite! Booooooo

  13. Alon Raskin wrote on November 8th, 2011 at 12:44:

    Definitely a horrible decision. We have apps written for Android, BB and iOS using WebSQL. We are now faced with the decision of support Windows Phone and it turns out that like Mozilla they too have gone down the IndexDB. What do we do now? Do we create another version of our app just to support Windows Phone?

    We are even toying with writing a JS layer which will take our WebSQL calls and convert them to IndexDB Calls! The effect of this is to shield our developers from IndexDB!! I dont see what other choice we have…

    1. Leigh Harrison wrote on November 8th, 2011 at 13:37:

      The abstraction layer is a sensible solution.

      I’m adopting a slightly different abstraction approach, as my apps do background synchronisation to a MySQL DB on the server anyway. If the user is running a compatible browser we use WebSQL, if the user’s running Firefox or Internet Exploder we use a RESTful service to the MySQL DB on the server. I looked at an IndexDB abstraction, but for the complex databases I’m using it was slower than getting and setting via the web.

      To make it clear where the speed issue lies we display a message advising the user that they should switch to Chrome or Safari for superior performance.

      I don’t like this solution much, and I’m hoping Firefox will soon see sense and support both IndexDB and WebSQL. If they do, Microsoft will eventually get with the programme too, although on past performance that may not be until IE13 or thereabouts.

  14. Eric Kizaki wrote on November 25th, 2011 at 14:08:

    This is why M$ does not support SQLite: http://www.microsoft.com/download/en/details.aspx?id=3613

    They have SQL Server Mobile.

  15. Eric Kizaki wrote on November 25th, 2011 at 14:23:

    Here is a more updated version of LINQ to SQL for MS.
    http://msdn.microsoft.com/en-us/library/hh202860(v=vs.92).aspx

    So saying that MS does not support Web db is a lame argument because MS is doing their own proprietary thing. There was no mention of IndexedDB. If I was developing a MS app I would use this stuff not IndexedDB.

  16. Marcelo Cantos wrote on November 26th, 2011 at 02:40:

    You are talking about mobile apps. This discussion is about the web.

  17. Ramin wrote on November 26th, 2011 at 12:08:

    Marcelo, this is the same thing.
    every mobile device has a browser, and devs can use HTML, CSS, Javascript and something like phonegap to develop software for not one mobile os but for all of them, just changing the CSS to match the native OS look. Or you can go and use something like JQueryMobile.
    So you can use web technology to develop standalone local software for all kinds of mobile OSs that works offline (so they have no access to a web server with a database). Still you want to use a database for your software, and of course it is nice to be able to use the same SQL queries that you would send to a sql serverto send them to your local sqlite DB.
    as a dev you do not want to learn a new thing like indexed db that you can use no where else you want to use WebSQL, to use your SQL knowledge, or you want to use LINQ what you use everyday when you develop with C#.

    1. Marcelo Cantos wrote on November 26th, 2011 at 20:04:

      @Ramin: If you and/or Eric are trying to make a case that Microsoft is being hypocritical, I won’t dispute that. Microsoft is a very large company, and to some extent, all large companies struggle to publicly convey a coherent view of the world. You wouldn’t try to argue that Microsoft doesn’t think tablets are the way to go just because the Office org has always had it in for the technology and refused to make Office usable on tablet devices.

      1. Ramin wrote on November 27th, 2011 at 10:13:

        No, every company can and should do what is best for it. And actually I am very impressed by what MS offers with LINQ. I am just saying that accessing DB on mobiles is really important, and that I think that no dev wants to learn about indexed db.
        WebSQL is a nice thing and should be supported by mozilla and by MS.
        Like most other devs I really cannot understand the argument against WebSQL.

  18. Marcelo Cantos wrote on November 27th, 2011 at 15:27:

    I guess we’re in violent agreement then. :-)

  19. Waseem wrote on March 22nd, 2012 at 07:17:

    Disappointing to say the least. It would be nice to use a real relational DB engine on the client side, or at least *have the option*. Mozilla completely screwed the pooch on this one, and I am disgusted by this decision. WebSQL was implemented almost completely across the board, and Mozilla’s petulance has set the web 1-3 years back in terms of offline and local development. It is for reasons like this (see also Firefox’s html5 audio/video supported formats) that I no longer recommend Firefox to other users, but Chrome.

  20. Pingback from HTML5本地存储不完全指南 | 松鼠博客 on April 1st, 2012 at 05:10:

    [...] 最后我们要介绍的就是IndexedDB了,相比其他两个规范,目前只有Firefox实现了IndexedDB(顺便提一下,Mozilla表示它们永远不会去实现Web SQL Database),不过Google已经表示正在考虑在Chrome中加入IndexDB支持。 [...]

  21. Pingback from HTML5 Offline Storage for Web Applications on May 20th, 2012 at 07:16:

    [...] http://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the-road-to-indexeddb/ [...]

  22. Freddy May wrote on July 16th, 2012 at 04:16:

    I agree with the vast majority of the commenters here. Sure, Mozilla are right to add support for IndexedDB – having NoSQL support is a great thing. But to put the frighteners on the development community by deprecating a hugely practical RDBMS that works beautifully in desktop and especially mobile applications is a bloody disgrace.

  23. Eric wrote on August 14th, 2012 at 05:08:

    “In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter”

    Uh? In what world? The article demonstrates the very opposite of that statement!

  24. John Willis wrote on September 21st, 2012 at 12:32:

    Yet another asinine screwup by Mozilla… I really hope that Chrome completely destroys FF’s market share so that developers like me won’t have to pander to Firefox, which is to web browsers in this decade what IE was to web browsers in the previous one.

  25. Rene Schickbauer wrote on October 1st, 2012 at 12:59:

    I’m the developer of the Maplat Webserver (and Framework). I’m currently in the process of designing Client-Side storing/caching of data for a large project.

    After evaluating the documentation, IndexedDB has no chance whatsoever of getting used in my projects. It does not fullfill requirement, it’s neither readable, fast nor remotely ACID compliant. IT#s API is just a junk of “when i grow up, someday i’ll be a database”.

    WebSQL on the other hand fits the bill nicely for most parts. This is what i’m probably going to use.

    So, the plan is as follows: Either completly drop support for browsers that don’t support WebSQL (Firefox is currently at less than 1% of “market share” on my business systems anyway) or don’t cache/store data on the client side for those browsers (which would make them much slower). Either way, i’m probably going to have to present affected users with a link to the latest Chrome build and suggest they “upgrade”.

    This isn’t an ideal solution. But in my opinion, IndexedDB is just too broken and convoluted for any serious use…

  26. Nick Vaidyanathan wrote on November 8th, 2012 at 17:23:

    This is probably a correct and important architectural decision. The mature software engineering community has long acknowledged that it is better to Program to an Interface, Not An Implementation.

    Most of the web developers responding to this with disgust are likely to be “coders”, not engineers or programmers. The common “web developer”, sadly, can’t architect their way out of a paper bag and needs 4 Drupal/Wordpress modules to display JSON content on a page. Don’t worry about the harshness of their criticism, it’s simply the crying of children being told “No, don’t stick your fork in the electric socket.” (“But it looks really fun! >:(“)

    Passing around hard coded SQL queries as strings is wrong. The last thing we need on the web is a slew of client side SQL injection attacks because dumbo.IcopypastaJqueryPlugin().dev didn’t sanitize his inputs, potentially knocking out tables/harvesting user personally identifiable information.

    Additionally, the idea of the web being built around open standards as opposed to “whatever does the job right now” is the reason we’re all using different programs that implement HTTP 1.x and TCP/IP, and not MSIE (or, poignantly for Mozilla, Netscape Navigator). Philosophically, this is an excellent decision.

    But the commenters do raise a good point. Come on, guys. It’s 2012. Hibernate has been around for over a decade. GORM, LINQ to SQL..there are a lot of implementations of ORM/Abstract Data Mapper layers out there.

    If you want to roll your own NoSQL hash map based data store, you should be providing an entity based model with nice iterators and convenience magic methods like GORM, not some low level cursor transaction thing like we’re programming with ADO.NET 2.0.

    1. Waseem S wrote on December 3rd, 2012 at 21:14:

      Please, the argument that there are too many bad developers to allow a standard like WebSQL is not relevant at all. We already have AJAX, client side scripting and IndexedDB – if a developer is out to cause havoc, havoc will be caused. A SQL statement pointing to client side data is just as “dangerous” as a JavaScript command that modifies the local data store.

      It’s almost 2013; web apps have come a long way and matured. To treat web developers with kiddie gloves only hinders progress.

    2. ArchE wrote on December 4th, 2012 at 13:36:

      Nick, it’s so much architectural discussion that for trees you don’t see the forest.

      Something like this:
      scholar: “we grow a red sweet apples dear Master. Everyone likes it.”
      master: “dear scholar, that is bad. The apples must be blue”
      scholar: “but blue apples are toxic”
      master: “nevermind dear scholar, no one will eat them then. Finally we will have blue apples.”

  27. ArchE wrote on November 13th, 2012 at 23:57:

    Generally I strongly disagree. By this decision you limit the freedom of choice.

    Philosophically BAD!
    Give us FREEDOM OF CHOICE!

    1. SQL relational engine is philosophocally on the same level as NoSQL. While NoSQL uses JavaScript to access objects, RDBMS uses SQL.
    Difference? No…there’s no real ORM in NoSQL.
    2. SQLLite is ANSI SQL and ANSI SQL IS STANDARD
    3. Aesthetics – code unification? Okay, not really the argue, but I understand
    4. SQLLite changes? common, this is only supplemental argument, you’re always enable to hold stable builds

    You should take a care of developer, instead you say how developer would work. Thi is not the good way, as I said, you limit the freedom of choice.

    Moreover, just now, you kicked off FF from applications which are using WebSQL and are available for mobile devices also. Because, Chrome, Safari and IE supports WebSQL.

    1. Robert Nyman wrote on November 14th, 2012 at 01:30:

      When it comes to Internet Explorer, it has never supported WebSQL. Current support is listed at caniuse.

      Additionally, the WebSQL specification is deprecated.

      1. ArchE wrote on November 14th, 2012 at 02:56:

        Deprecated, right…according to W3C:
        The specification reached an impasse: all interested implementors have used the same SQL backend (Sqlite), but we need multiple independent implementations to proceed along a standardisation path.

        From this POV I really don not understand. Multiple implementations?
        Many projects already use WebSQL or persistence layers.
        Why not to support such an easy implementation like SQLLite?

        It seems to me there is strong dislike of SQLLite :)

  28. ArchE wrote on November 14th, 2012 at 01:07:

    Other arguments why SQL Lite is not philosophically bad idea can be found here http://www.html5rocks.com/en/tutorials/webdatabase/websql-indexeddb/

  29. Richard Collins wrote on December 3rd, 2012 at 12:59:

    I got really excited about having a decent database available for use in an offline web app I am starting to develop. Then I discovered this crazy decision by Mozilla to not support WebSQL. I am now faced with the decision of increasing development time significantly by using IndexedDB, or creating a Chrome/Safari only app.

    Whilst I understand the reasoning, it seems quite narrow. Let’s back a good technology. Sqlite is public domain. In the unlikely event that incompatibilities are introduced in the future the community will fork it.

    1. ArchE wrote on December 3rd, 2012 at 23:48:

      Thumbs up.
      It seems there’s strong disappointment with this descicion in the community.

      While I’m in the phase I already have WebSQL application my decision is to not support FF.

      I strongly believe guys from FF will change the decision. Hope it will not be too late.

  30. Waseem S wrote on December 3rd, 2012 at 21:09:

    Firefox (Mozilla) with decisions like these come across as puritanical sticklers who can’t be bothered to listen to other people because they know what’s best for everyone. Aside from sabotaging client side data storage, Firefox is the only mainstream browser that can’t play files encoded in the ubiquitous MP3 format via the HTML5 tag. Any attempts to create a cross platform media solution has to include re-encoding all media to ogg or some other format, which doubles encoding time, media storage requirements and costs. Instead of making the web more accessible to all, they are putting hurdles, especially to smaller organizations.
    Their argument that MP3 is a proprietary format that shouldn’t be utilized is laughable when you see their GIF support. .

  31. Sebastian Baltes wrote on January 17th, 2013 at 10:06:

    Mozilla is so wrong about this.

    We are building a medium sized html5 app based on websql and phonegap, and it’s wonderfull to have sql in javascript! We have it in all mobile plattforms (in win8 via a sqlite plugin). Even in a medium sized app the relational model has a lot of advantages: a powerfull query language, transactions, speed and solidness. And we have some large sql queries – doing that by hand with IndexDB would be a nightmare.

    More than ten years ago, Object-Oriented DBs were the hype. What survived? SQL. Then NoSQL was the next thing… it’s nice, and they have their niches, but still the majority uses relational databases. Guess what will be dominant in 10 years?

    Your IndexDB may be good for the easiest use cases, but for this we allready have localStorage. The complex use cases are not covered by your immature design-baby.

    And why the heck must you guys at mozilla design your own peristent standard at all? Please explain this to me: Why didn’t you took MongoDB for example? Because it’s not a standard??? Smells like some kind of hybris, kind of ‘oh, we are sooo smart, and we have all this money and don’t know how to spend it’ attitude. You guess what? Follow this road, and in ten years firefox will be irrelevant, but we will still use SQL in the browser.

  32. Sergei wrote on February 5th, 2013 at 07:19:

    I’d agree that structured indexed storage is more useful as a basis for Web platform. And it’s better to have Web SQL Storage atop of primitive structured storage. I’d agree with Mozilla’s arguments about depending on SQLite’s sysntax is a bad thing. But. IndexedDB’s API is terrible. Just look at example 4 in https://hacks.mozilla.org/2010/06/comparing-indexeddb-and-webdatabase/.
    It’s simply not a way to go.

  33. Vikash Madhow wrote on March 8th, 2013 at 23:14:

    Anyone who says that IndexedDB is better than WebSQL does not know what he/she is talking about.

    Except for the very simple use-cases, IndexedDB is NEVER going to enable the web-browser to become a platform where we have a useful offline mode for web apps when connectivity is down. I recommend that we move collectively to Chrome and recommend to all to do same. I, for one, need something like WebSQL for the type of apps I’m building and therefore firefox is off the supported-browsers list.

1 2

Comments are closed for this article.