Beyond HTML5: Database APIs and the Road to IndexedDB

on June 1, 2010 by Arun Ranganathan

100 comments

IndexedDB is an evolving web standard for the storage of significant amounts of structured data in the browser and for high performance searches on this data using indexes. Mozilla has submitted substantial technical feedback on the specification, and we plan to implement it in Firefox 4. We spoke to prominent web developers about evolving an elegant structured storage API for the web. While versions of Safari, Chrome, and Opera support a technology called Web SQL Database, which uses SQL statements as string arguments passed to a JavaScript API, we think developer aesthetics are an important consideration, and that this is a particularly inelegant solution for client-side web applications. We brought developer feedback to the editor of the IndexedDB specification, and also spoke with Microsoft, who agree with us that IndexedDB is a good option for the web. With additional implementations from the Chrome team in the offing, we think it is worth explaining our design choices, and why we think IndexedDB is a better solution for the web than Web SQL Database.

Web applications can already take advantage of localStorage and sessionStorage in IE 8+, Safari 4+, Chrome 4+, Opera 10.5+ and Firefox 2+ to store key-value pairs with a simple JavaScript API. The Web Storage standard (encompassing localStorage and sessionStorage), now widely implemented, is useful for storing smaller amounts of data, but less useful for storing larger amounts of structured data. While many server-side databases use SQL to programmatically store structured data and to meaningfully query it, on the client-side, the use of SQL in a JavaScript API has been contentious.

SQL? Which SQL?

Many web developers certainly are familiar with SQL, since many developers touch just as much server-side code (e.g. PHP and database operations) as client-side code (e.g. JavaScript, CSS, and markup). However, despite the ubiquity that SQL enjoys, there isn’t a single normative SQL standard that defines the technology. In particular, SQLite supports most of SQL-92, with some notable omissions, and is what the WebDatabase API is based on. But SQLite itself isn’t a specification — it’s a release-ready technology! And the best definition of what constitutes the supported subset of SQL that SQLite uses is the SQLite manual. In order to really get Web SQL Database right, we’d have to first start with defining a meaningful subset of SQL for web applications. Why define a whole other language, when more elegant solutions exist within JavaScript itself?

The Benefits and Pitfalls of SQLite

We think SQLite is an extremely useful technology for applications, and make it available for Firefox extensions and trusted code. We don’t think it is the right basis for an API exposed to general web content, not least of all because there isn’t a credible, widely accepted standard that subsets SQL in a useful way. Additionally, we don’t want changes to SQLite to affect the web later, and don’t think harnessing major browser releases (and a web standard) to SQLite is prudent. IndexedDB does not have this problem; even though our underlying implementation of IndexedDB may be based on SQLite, we keep developers insulated from changes to SQLite by exposing an API that isn’t based on SQLite’s supported syntax.

Aesthetics and Web Developers

Last year, we held a summit at the Mozilla campus to discuss storage on the web. We invited web developers to speak to us about a desirable structured storage API on the web. Many did express resigned acceptance of a SQLite-based API, since they had already experimented with releases of Web SQL Database in some browsers, and claimed that something in circulation was better than a collection of ideas. Yet, all voiced enthusiasm for better design choices, and how a simpler model would make life easier for them. We watched as developers whiteboarded a simple BTree API that addressed their application storage needs, and this galvanized us to consider other options. We were resolved that using strings representing SQL commands lacked the elegance of a “web native” JavaScript API, and started looking at alternatives. Along with Microsoft, we sent feedback about the IndexedDB proposal and actively became involved in the standardization effort.

In another article, we compare IndexedDB with Web SQL Database, and note that the former provides much syntactic simplicity over the latter. IndexedDB leaves room for a third-party JavaScript library to straddle the underlying primitives with a BTree API, and we look forward to seeing initiatives like BrowserCouch built on top of IndexedDB. Intrepid web developers can even build a SQL API on top of IndexedDB. We’d particularly welcome an implementation of the Web SQL Database API on top of IndexedDB, since we think that this is technically feasible. Starting with a SQL-based API for use with browser primitives wasn’t the right first step, but certainly there’s room for SQL-based APIs on top of IndexedDB.

We want to continue the discussion with web developers about storage on the web, since that helps us structure our thoughts about product features and future web standards. We look forward to seeing the next generation of web applications with support for high performance searches on indexed data, and to seeing web applications work even more robustly in “airplane mode.”