Mozilla

Articles

Sort by:

View:

  1. Event listeners popup, @media sidebar, Cubic bezier editor + more – Firefox Developer Tools Episode 33

    A new set of Firefox Developer Tools features has just been uplifted to the Aurora channel. These features are available right now in Aurora, and will be in the Firefox 33 release in October. This release brings many new additions, especially to the Inspector tool:

    Event listeners popup

    Any node with a JavaScript event listener attached to it will now have an “ev” icon next to it in the Inspector. Clicking the icon will open a list of all the event listeners attached to that element. Click the pause icon to get taken to that function in the Debugger, where you can set breakpoints and debug it further. (development notes & UserVoice request)

    Event listener popup screenshot

    Keep in mind the events pane in the Debugger as well, which lists all the event listeners on the page

    @media sidebar

    There’s a new sidebar in the Style Editor which displays a list of shortcuts to every @media rule in the stylesheet (or Sass source) you’re editing. Click an item to jump to that rule. The condition text of the rule is greyed-out if the media query doesn’t currently apply. This works well in conjunction with the Responsive Design View (Opt+Cmd+M / Ctrl+Shift+M) for creating and debugging mobile layouts. (development notes)

    @media rules sidebar

    Add new rule

    Right-click anywhere in the Rules section of the Inspector to get an “Add Rule” option. Selecting this will add a new CSS rule, pre-populated with a selector that matches the currently selected node. (development notes & UserVoice request)

    Add rule screenshot

    Edit keyframes

    Any @keyframes rules associated with the currently selected element are now displayed in the Rules section of the Inspector, and are editable. This is the first step on the way to better debugging of CSS animations. (development notes)

    Editing @keyframe rules

    Cubic bezier editor

    To aid in editing easing animations, there’s now a cubic bezier editor that appears when you click the icon next to an animation timing function in a CSS rule. This feature used open source code from Lea Verou’s cubic-bezier.com. (development notes)

    Transform highlighter

    There’s a new awesome way to visualize how an element has been transformed from its original position and shape. Hovering over a CSS transform property in the Inspector will show the original position of the element on the page and draw lines mapping the original points to their new positions. (development notes)

    Transform highlighter screenshot

    Persistent disable cache

    You can disable the browser cache while you’re developing by checking Advanced Settings > Disable Cache in the Settings tab. Now this setting will persist the next time you open the devtools. As usual, caching is re-enabled for the tab when you close the tools. (development notes & UserVoice request)

    New Commands

    New commands have been added to the Developer Toolbar (Shift+F2):

    inject
    The new inject command lets you easily inject jQuery or other JavaScript libraries into your page. Use inject jQuery, inject underscore, or provide your own url with inject <url>. (development notes)
    highlight
    The highlight command takes a selector and highlights all the nodes on that page that match that selector. (video)(development notes)
    folder
    The folder command opens a directory in your system’s file explorer. Use folder openprofile to open your Firefox profile directory. (development notes)

    Editor preferences

    A host of editor preferences are now available in the Settings panel. From here you can change your indentation settings and change editor keybindings to Sublime Text, Vim, or Emacs. (development notes)

    WebIDE

    A big feature called WebIDE has landed, but is behind a preference for this release while it gets more testing. WebIDE is a tool for in-browser app development. See WebIDE lands in Nightly for more details.

    Other features

    Edit selectors
    Click the selector of any CSS rule in the Inspector to edit it. (development notes)
    Black boxed minified sources
    JavaScript sources with “min.js” extensions are now automatically black boxed. You can turn this option off in the Debugger settings menu. (development notes)
    Custom viewport dimensions
    The dimensions in the Responsive Design View are now editable so you can input the exact size you want the content to be. (development notes)

    Special thanks to the 33 contributors that added all the features and fixes in this release.

    Three of the features from this release came from feedback on the Developer Tools feedback channel, so that’s a great way to suggest features. You can also comment here or shoot feedback to @FirefoxDevTools on Twitter. If you’d like to help out, check out the guide to getting involved.

  2. Building Firefox Hub Add-ons for Firefox for Android

    The Firefox Hub APIs allow add-ons to add new panels to the Firefox for Android home page, where users normally find their top sites, bookmarks and history. These APIs were introduced in Firefox 30, but there are more features and bug fixes in Firefox 31 and 32. You can already find some of these add-ons on addons.mozilla.org, and there is some boilerplate code on github to help you get started.

    image01
    image00

    Overview

    There are two main parts to building a Firefox Hub add-on: creating a home panel, and storing data to show in that panel. Home panels consist of different views, each of which displays data from a given dataset.

    Creating a new home panel

    To create a home panel, first use the Home.panels API to register a panel. The register API takes a panel id and an options callback function as parameters. This options callback is called to dynamically generate an options object whenever a panel is installed or updated, which allows for dynamic locale changes.

    function optionsCallback() {
      return {
        title: "My Panel",
        views: [{
          type: Home.panels.View.LIST,
          dataset: "my.dataset@mydomain.org"
        }]
      };
    }
     
    Home.panels.register("my.panel@mydomain.org", optionsCallback);

    You must always register any existing panels on startup, but the first time you want the panel to actually appear on the user’s home page (e.g. when your add-on is installed), you also need to explicitly install the panel.

    Home.panels.install("my.panel@mydomain.org");

    You can modify the options callback function to customize the way data is displayed in your panel. For example, you can choose to display your data in a grid or a list, customize the view that is displayed when no data is available, or choose to launch an intent when the user taps on one of the items.

    Storing data for the panel

    To actually show something in your new home panel, use the HomeProvider API to store data. This API allows you to asynchronously save and delete data, as well as register a callback to allow the browser to periodically sync your data for you.

    The HomeProvider API gives you access to HomeStorage objects, which you can interact with to save and delete data from a given dataset. These methods are designed to be used with Task.jsm to execute asynchronous transactions within a task.

    let storage = HomeProvider.getStorage("my.dataset@mydomain.org");
    Task.spawn(function() {
      yield storage.save(items);
    }).then(null, Cu.reportError);

    In Firefox 31, we expanded the save API to support replacing existing data for you, which is convenient for periodically refreshing your dataset.

    function refreshDataset() {
      let items = fetchItems();
      Task.spawn(function() {
            yield storage.save(items, { replace: true });
      }).then(null, Cu.reportError);
    }
     
    HomeProvider.addPeriodicSync("my.dataset@mydomain.org", 3600, 
    refreshDataset);

    This code snippet will ensure that our dataset is refreshed once every 3600 seconds (1 hour).

    What’s new in Firefox 32 Beta

    In addition to bug fixes, Firefox 32 also adds a few more features to the set of Firefox Hub APIs.

    Refresh handler

    In addition to support for periodically updating data, we also added support for “pull to refresh”, which gives users the power to manually refresh panel data. To take advantage of this feature, you can add an onrefresh property to your view declaration.

    function optionsCallback() {
      return {
        title: "My Panel",
        views: [{
          type: Home.panels.View.LIST,
          dataset: "my.dataset@mydomain.org",
          onrefresh: refreshDataset
        }]
      };
    }

    With this new line added, swiping down on your panel will trigger a refresh indicator and call the refreshDataset function. The refresh indicator will disappear after a save call is made for that dataset.

    Authentication view

    We added support for an authentication view, to make it easier for your add-on to use data that requires authentication. This view includes space for text and an image, as well as a button that triggers an authentication flow. To use this feature, you can add an auth property to your panel declaration.

    function optionsCallback() {
      return {
        title: "My Panel",
        views: [{
          type: Home.panels.View.LIST,
          dataset: "my.dataset@mydomain.org"
        }],
        auth: {
         authenticate: function authenticate() {
            // … do some stuff to authenticate the user …
           Home.panels.setAuthenticated("my.panel@mydomain.org", true);
         },
         messageText: "Please log in to see your data",
         buttonText: "Log in"
       }
      };
    }

    By default, the authentication view will appear when your panel is first installed, and the authenticate function will be called when the user taps the button in the view. It is up to you to call setAuthenticated(true) when the user successfully completes an authentication flow, and you can also call setAuthenticated(false) when a user becomes unauthenticated. This authentication state will persist between app runs, so it is up to you to reset it if you need to.

    Future work

    We have ideas about ways to expand these APIs, but please let us know if there is anything you would like to see! We’re also always looking for new contributors to Firefox for Android, and we’d love to help you get started writing patches.

  3. Resources for HTML5 game developers

    Today we released Firefox 31 and it offers a couple of new features that help HTML5 game developers to code and debug sophisticated games. In addition Mozilla blogged about the first commercial games leveraging asm.js, Dungeon Defenders Eternity and Cloud Raiders both of which were cross-compiled in to JavaScript using the Emscripten compiler. Games like these show that HTML5 is ready as a game platform.

    If you are interested in working with Emscripten you can get more information at the main Emscripten wiki or grab the code on the github page. Another good resource is the getting started with Emscripten tutorial on MDN. If you are wondering about the performance of asm.js, read asm.js performance improvements in the latest version of Firefox make games fly! for details.

    In this post we’ll introduce you to some of the resources built by Mozillians that allow you to code and debug HTML5 based games. This list is not exhaustive and we appreciate feedback on any valuable resources that would help in this arena. Don’t be shy and tell us about them in the comments.

    Where To Start

    When developing an HTML5 based game, you have a lot of choices to make. These range from what editor to use, if the game will use Canvas 2d, WebGL, SVG, or CSS up to which specific rendering frameworks and game engines to use. Most of these decisions will be based on the developer experience and the platforms the game will be published on. No article will answer all these questions but we wanted to put together a post that would help get you started down the path.

    One of the key resources available for game developers on MDN is the Games Zone. This section of MDN contains general game development articles, demos, external resources and examples. It also includes detailed descriptions of some of the APIs that a developer will need to be aware of when implementing an HMTL5 game, including sound management, networking, storage and graphics rendering. We are currently in the process of adding content and upgrading the zone. In the future we hope to have content and examples for most common scenarios, frameworks and tool chains.

    In the meantime here are a few posts and MDN articles that help game developers getting started.

    Tools

    As an HTML5 developer you will have no shortage of tools at your disposal. In the Mozilla community we have been hard at work expanding the features that Firefox Developer Tools provide. These include a full-featured JavaScript Debugger, Style Editor, Page Inspector, Scratchpad, Profiler, Network Monitor and Web Console.

    In addition to these, some notable tools have been updated or introduced recently and offer some great functionality for the game developer.

    Canvas Debugger

    With the current release of Firefox, we added a Canvas Debugger to the browser.
    s_canvasdebugger
    The Canvas Debugger allows you to trace through all canvas context calls that are used to generate a frame. Calls are color coded for specific calls for things like drawing elements or using a specific shader program. The Canvas Debugger is not only useful when developing a WebGL based game but can also be used when debugging a Canvas 2D based game. In the game below you can see in the animation strip as each image is drawn to the canvas. You can click any of these lines to get directly to the part of your JavaScript responsible for this action.
    s_captainrogers
    Two very common issues that have been reported when using the Canvas Debugger are with animations generated using setInterval instead of requestAnimationFrame and inspecting canvas elements in an iFrame.

    To get more information about the Canvas Debugger be sure to read over Introducing the Canvas Debugger in Firefox Developer Tools.

    Shader Editor

    When developing WebGL based games it is very useful to be able to test and alter shader programs while the application is running. Using the Shader Editor within the developer tools makes this possible. Vertex and Fragment Shader programs can be modified without the need to reload the page, or black boxed to see what effect this has on the resulting output.
    s_ShaderEditor

    For more information on the Shader Editor, be sure to see Live editing WebGL shaders with Firefox Developer Tools post and take a look at this MDN article which contains a couple of videos showing live editing.

    Web Audio Editor

    The current version of Firefox Aurora (32) – has a Web Audio Editor. The Editor displays a graphical representation of all the Audio Nodes and their connections in the current AudioContext. You can drill down to specific attributes of each node to inspect them.
    s_webaudioeditor

    The Web Audio API provides more robust and complex sound creation, manipulation and processing than what is available in the HTML5 Audio tag. When using the Web Audio API make sure to read over Writing Web Audio API code that works in every browser as it contains pertinent information about support for the various audio nodes.

    For more information on the Web Audio Editor be sure to read this Hacks article introducing the Web Editor and this MDN article.

    Network Monitor

    When developing an HTML5 based game network impact can be not only cumbersome but also costly if the user is on mobile device. Using the Network Monitor you can visually inspect all network request for location, time spent on the operation, and the type and size of the artifact.
    s_networkmon
    In addition you can use the Network Monitor to get a visual performance analysis of your app when cached versus non-cached.
    s_networkcache

    To get more information on the Network Monitor see the MDN page.

    Web IDE

    When starting your game one of your first choices will be which editor to use. And there are a lot of them (Sublime, Eclipse, Dreamweaver, vi, etc). In most cases a you already have a favorite. If you are interested in doing your development within the Browser you may want to have a look at the Web IDE that was recently released in Firefox Nightly.
    s_webide

    The Web IDE project provides not only a fully functional editor but also acts as a publishing agent to various local and remote platforms, debugger, template framework and application manager. In addition the framework supporting this project provides APIs that will allow other editors to use functionality provided in the tool. To get more details on the work that is being done in this area have a look at this post.

    In order to keep up-to-date with news on the Firefox Developer Tools, follow their article series on the Hacks blog. For more detailed information on new, stable developer tools features, check out their documentation on MDN.

    APIs

    The MDN Games Zone lists various APIs and articles that are useful for beginning game development.
    s_apis
    In addition to these resources you may be interested in looking over some additional posts that can be valuable for development.

    If your game is going to support multiplayer interaction using either WebRTC or WebSockets you may also be interested in looking at Together.js which provides collaborative features for web apps. To get an idea what is possible take a look at Introducing TogetherJS.

    Many games will require storage and IndexedDB can be used to handle these needs. For information on extending the capabilities of IndexedDB read over Breaking the Borders of IndexedDB. You may also be interested in localForage which provides browser agnostic support for simple storage. To get more details about this library read over this Hacks post.

    Game Optimization

    HTML5 games today offer a great deal of power to the game developer. That said many of these games are going to be played on a mobile device, which in comparison to your desktop will pale in performance. So if you plan on your game being a success across platforms it is important that you optimize your code. The Optimizing your JavaScript Game for Firefox OS post has a lot of great techniques to help you build a game that performs well on low-end mobile devices.

    Localization

    In order to reach the most users of your game you may want to consider offering it in different languages. As part of this developers should start with localization built into the game. We are doing a great deal of work around recruiting translators to help you translate your game. To get more information about this initiative see this post.

    Your Voice

    As Mozilla is about the community of developers and users, we want your help and your feedback. If you have recommendations for specific features that you would like to see in the products make sure to either get involved in discussion on irc.mozilla.org or through our mailing lists. You can also log bugs at bugzilla.mozilla.org. In addition we are also provide additional feedback channels for our DevTools and Open Web Apps.

  4. How can we write better software? – Interview series, part 1 with Fernando Jimenez Moreno

    Do you ever look code and murmur a string of “WTFs?” Yeah, me too. As often as not, the code is my own.

    I have spent my entire professional career trying to write software that I can be proud of. Writing software that “works” is difficult. Writing software that works while also being bug-free, readable, extensible, maintainable and secure is a Herculean task.

    Luckily, I am part of a community that is made up of some of the best development, QA and security folks in the industry. Mozillians have proven themselves time and time again with projects like Webmaker, MDN, Firefox and Firefox OS. These projects are complex, huge in scope, developed over many years, and require the effort of hundreds.

    Our community is amazing, flush with skill, and has a lot to teach.

    Interviews and feedback

    This is the first of a series of interviews where I take on the role of an apprentice and ask some of Mozilla’s finest

    “How can we, as developers, write more superb software?”

    Since shipping software to millions of people involves more than writing code, I approach the question from many viewpoints: QA, Security and development have all taken part.

    The target audience is anybody who writes software, regardless of the preferred language or level of experience. If you are reading this, you are part of the part of the target audience! Most questions are high level and can be applied to any language. A minority are about tooling or language specific features.

    Questions

    Each interview has a distinct set of questions based around finding answers to:

    • How do other developers approach writing high quality, maintainable software?
    • What does “high quality, maintainable software” even mean?
    • What processes/standards/tools are being used?
    • How do others approach code review?
    • How can development/Security/QA work together to support each-other’s efforts?
    • What matters to Security? What do they look for when doing a security review/audit?
    • What matters to QA? What do they look for before signing off on a release?
    • What can I do, as a developer, to write great software and facilitate the entire process?

    I present the highlights of one or two interviews per article. Each interview contains a short introduction to the person being interviewed followed by a series of questions and answers.

    Where an interview’s audio was recorded, I will provide a link to the full transcript. If the interview was done over email, I will link to the contents of the original email.

    Now, on to the first interview!

    Introducing Fernando Jimenez Moreno

    Fernando Jimenez MorenoThe first interview is with Fernando Jimenez Moreno, a Firefox OS developer from Telefonica. I had the opportunity to work with Fernando last autumn when we integrated Firefox Accounts into Firefox OS. I was impressed not only with Fernando’s technical prowess, but also his ability to bring together the employees of three companies in six countries on two continents to work towards a common goal.

    Fernando talks about how Telefonica became involved in Firefox OS, how to bring a disparate group together, common standards, code reviews, and above all, being pragmatic.

    What do you and your team at Telefonica do?

    I’m part of what we call the platform team. We have different teams at Telefonica, one is focused on front end development in Gaia, and the other one is focused on the platform itself, like Gecko, Gonk and external services. We work in several parts of Firefox OS, from Gecko to Gaia, to services like the SimplePush server. I’ve personally worked on things like the Radio Interface Layer (RIL), payments, applications API and other Web APIs, and almost always jump from Gecko to Gaia and back. Most recently, I started working on a WebRTC service for Firefox OS.

    How did Telefonica get involved working with Mozilla?

    Well, that’s a longer story. We started working on a similar project to Firefox OS, but instead of being based on Gecko, we were working with WebKit. So we were creating this open web device platform based on WebKit. When we heard about Mozilla doing the same with Gecko, we decided to contact you and started working on the same thing. Our previous implementation was based on a closed source port of WebKit and it was really hard to work that way. Since then, my day to day work is just like any other member of Telefonica’s Firefox OS team, which I believe is pretty much the same as any other Mozilla engineer working on B2G.

    You are known as a great architect, developer, and inter-company coordinator. For Firefox Accounts on Firefox OS, you brought together people from Telefonica, Telenor, and Mozilla. What challenges are present when you have to work across three different companies?

    It was quite a challenge, especially during the first days of Firefox OS. We started working with Mozilla back in 2011, and it took some time for both companies to find a common work flow that fit well for both parts. I mean, we were coming from a telco culture where many things were closed and confidential by default, as opposed to the openness and transparency of Mozilla. For some of us coming from other open source projects, it wasn’t that hard to start working in the open and to be ready to discuss and defend our work on public forums. But, for other members of the team it took some time to get used to that new way of working, and new way of presenting our work.

    Also, because we were following agile methodologies in Telefonica while Mozilla wasn’t still doing it, we had to find this common workflow that suits both parts. It took some time to do it, a lot of management meetings, a lot of discussions about it. Regarding working with other telco companies, the experience has also been quite good so far, especially with Telenor. We still have to be careful about the information that we share with them, because at the end of the day, we are still competitors. But that doesn’t mean we cannot work with them in a common target like what happened with Firefox Accounts.

    When Mozilla and Telefonica started out on this process, both sides had to change. How did you decide what common practices to use and how did you establish a common culture?

    I think for this agile methodology, we focused more on the front end parts because Gecko already had a very known process and a very known way of developing. It has it’s own train mechanism of 6 weeks. The ones doing the most, the biggest effort of finding that common workflow were the front end team because we started working on Gaia and Gaia was a new project with no fixed methodologies.

    I don’t know if we really found the workflow, the perfect workflow, but I think we are doing good. I mean we participate in agile methodologies, but when it turns out that we need to do Gecko development and we need to focus on that, we just do it their way.

    In a multi-disciplinary, multi-company project, how important are common standards like style guides, tools, and processes?

    Well, I believe when talking about software engineering, standards are very important in general. But, I don’t care if you call it SCRUM or KANBAN or SCRUMBAN or whatever, or if you use Git workflow or Mercurial workflow, or if you use Google or Mozilla’s Javascript style guide. But you totally need some common processes and standards, especially in large engineering groups like open source, or Mozilla development in general. When talking about this, the lines are very thin. It’s quite easy to fail spending too much time defining and defending the usage of these standards and common processes and losing the focus on the real target. So, I think we shouldn’t forget these are only tools, they are important, but they are only tools to help us, and help our managers. We should be smart enough to be flexible about them when needed.

    We do a lot of code reviews about code style, but in the end what you want is to land the patch and to fix the issue. If you have code style issues, I want you to fix them, but if you need to land the patch to make a train, land it and file a follow on bug to fix the issues, or maybe the reviewer can do it if they have the chance.

    Firefox OS is made up of Gonk, Gecko and Gaia. Each system is large, complex, and intimidating to a newcomer. You regularly submit patches to Gecko and Gaia. Whenever you dive into an existing project, how do you learn about the system?

    I’m afraid there is no magic technique. What works for me might not work for others for sure. What I try to do is to read as much documentation as possible inside and outside of the code, if it’s possible. I try to ask the owners of that code when needed, and also if that’s possible, because sometimes they just don’t work in the same code or they are not available. I try to avoid reading the code line by line at first and I always try to understand the big picture before digging into the specifics of the code. I think that along the years, you somehow develop this ability to identify patterns in the code and to identify common architectures that help you understand the software problems that you are facing.

    When you start coding in unfamiliar territory, how do you ensure your changes don’t cause unintended side effects? Is testing a large part of this?

    Yeah, basically tests, tests and more tests. You need tests, smoke tests, black box tests, tests in general. Also at first, you depend a lot on what the reviewer said, and you trust the reviewer, or you can ask QA or the reviewer to add tests to the patch.

    Let’s flip this on its head and you are the reviewer and you are reviewing somebody’s code. Again, do you rely on the tests whenever you say “OK, this code adds this functionality. How do we make sure it doesn’t break something over there?”

    I usually test the patches that I have review if I think the patch can cause any regression. I also try and run the tests on the “try” server, or ask the developer to trigger a “try” run.

    OK, so tests.. A lot of tests.

    Yeah, now that we are starting to have good tests in Firefox OS, we have to make use of them.

    What do you look for when you are doing a review?

    In general where I look first is correctness. I mean, the patch should actually fix the issue it was written for. And of course it shouldn’t have collateral effects. It shouldn’t introduce any regressions. And as I said, I try to test the patches myself if I have the time or if the patch is critical enough, to see how it works and to see if it introduces a regression. And also I look that the code is performant and is secure, and also if, I always try to ask for tests if I think they are possible to write for the patch. And I finally look for things like quality of the code in general, and documentation, coding style, contribution, process correctness.

    You reviewed one of my large patches to integrate Firefox Accounts into Firefox OS. You placed much more of an emphasis on consistency than any review I have had. By far.

    Well it certainly helps with overall code quality. When I do reviews, I mark these kinds of comments as “nit:” which is quite common in Mozilla, meaning that “I would like to see that changed, but you still get my positive review if you don’t change it, but I would really like to see them changed.”

    Two part question. As a reviewer, how can you ensure that your comments are not taken too personally by the developer? The second part is, as a developer, how can you be sure that you don’t take it too personally?

    For the record, I have received quite a few hard revisions myself. I never take them personally. I mean, I always try to take it, the reviews, as a positive learning experience. I know reviewers usually don’t have a lot of time to, in their life, to do reviews. They also have to write code. So, they just quickly write “needs to be fixed” without spending too much time thinking about the nicest ways to say it. Reviewers only say things about negative things in your code, not negative, but things that they consider are not correct. But they don’t usually say that the things that are correct in your code and I know that can be hard at first.

    But once you start doing it, you understand why they don’t do that. I mean, you have your work to do. This is actually especially hard for me, being a non-native English speaker, because sometimes I try to express things in the nicest way possible but the lack of words make the review comments sound stronger than it was supposed to be. And, what I try to do is use a lot of smileys if possible. And always, I try to avoid the “r-” flag if I mean, the “r-” is really bad. I just clear it, use use the “feedback +” or whatever.

    You already mentioned that you try to take it as a learning experience whenever you are developer. Do you use review as a potential teaching moment if you are the reviewer?

    Yeah, for sure. I mean just the simple fact of reviewing a patch is a teaching experience. You are telling the coder what you think is more correct. Sometimes there is a lack of theory and reasons behind the comments, but we should all do that, we should explain the reasons and try to make the process as good as possible.

    Do you have a snippet of code, from you or anybody else, that you think is particularly elegant that others could learn from?

    I am pretty critical with my own code so I can’t really think about a snippet of code of my own that I am particularly proud enough to show :). But if I have to choose a quick example I was quite happy with the result of the last big refactor of the call log database for the Gaia Dialer app or the recent Mobile Identity API implementation.

    What open source projects would you like to encourage people to participate in, and where can they go to get involved?

    Firefox OS of course! No, seriously, I believe Firefox OS gives to software engineers the chance to get involved in an amazing open source community with tons of technical challenges from low level to front end code. Having the chance to dig into the guts of a web browser and a mobile operative system in such an open environment is quite a privilege. It may seem hard at first to get involved and jump into the process and the code, but there are already some very nice Firefox OS docs on the MDN and a lot of nice people willing to help on IRC (#b2g and #gaia), the mailing lists (dev-b2g and dev-gaia) or ask.mozilla.org.

    How can people keep up to date about what you are working on?

    I don’t have a blog, but I have my public GitHub account and my Twitter account.

    Transcript

    A huge thanks to Fernando for doing this interview.

    The full transcript is available on GitHub.

    Next article

    In the next article, I interview Brian Warner from the Cloud Services team. Brian is a security expert who shares his thoughts on architecting for security, analyzing threats, “belts and suspenders”, and writing code that can be audited.

    As a parting note, I have had a lot of fun doing these interviews and I would like your input on how to make this series useful. I am also looking for Mozillians to interview. If you would like to nominate someone, even yourself, please let me know! Email me at stomlinson@mozilla.com.

  5. Adding captions and subtitles to HTML5 video

    This article is also available on MDN.

    With the introduction of the <video> and <audio> elements to HTML5, we finally have a native way to add video and audio to our websites. We also have a JavaScript API that allows us to interact with this media content in different ways, be it writing our own controls or simply seeing how long a video file is. As responsible web developers, we should also be constantly thinking about making our content more accessible, and this doesn’t stop with video and audio content. Making our content accessible to all is an important step, be it for someone who is hard of hearing or someone who doesn’t understand the language that the content is delivered in, inclusion can be paramount.

    Thankfully HTML5 also provides us with a native way of making our media content more accessible by adding subtitles and captions via the <track> element. Most major browsers support this natively to varying degrees, which the first part of this article shows, but it also provides a JavaScript API, with which we can access and use the text tracks (e.g. subtitles) that are available. This article also shows how this API can be used to detect what captions/subtitles have been added to a HTML5 video, and how that data can be used to build a selectable menu of available text tracks and ultimately provide a more consistent interface across the various browsers.

    In articles on MDN, we have looked at how to build a cross browser video player using the HTMLMediaElement and Window.fullScreen APIs, and also at how to style the player. This article will take the same player and show how to add captions and subtitles to it, using Web_Video_Text_Tracks_Format and the <track> element.

    Captioned video example

    In this article, we will refer to the Video player with captions example. This example uses an excerpt from the Sintel open movie, created by the Blender Foundation.

    Video player with stand controls such as play, stop, volume, and captions on and off. The video playing shows a scene of a man holding a spear-like weapon, and a caption reads "Esta hoja tiene pasado oscuro."

    Note: You can find the source on Github, and also view the example live.

    HTML5 and Video Captions

    Before diving into how to add captions to the video player, there are a number of things that we will first mention, which you should be aware of before we start.

    Captions versus subtitles

    Captions and subtitles are not the same thing: they have significantly different audiences, and convey different information, and it is recommended that you read up on the differences if you are not sure what they are. They are however implemented in the same way technically, so the material in this article will apply to both.

    For this article we will refer to the text tracks displayed as captions, as their content is aimed at hearing people who have difficulty understanding the language of the film, rather than deaf or hard-of-hearing people.

    The <track> element

    HTML5 allows us to specify captions for a video using the Web Video Text Tracks (WebVTT) format. The WebVTT specification is still being worked on, but major parts of it are stable so we can use it today.

    Video providers (such as the Blender Foundation) provide captions and subtitles in a text format with their videos, but they’re usually in the SubRip Text (SRT) format. These can be easily converted to WebVTT using an online converter such as srt2vtt.

    Modifications to the HTML and CSS

    This section summarises the modifications made to the previous article’s code in order to facilitate the addition of subtitles to the video. If you are not interested in thism and just want to get straight into the JavaScript and more relevant CSS, skip to the Caption implementation section.

    In this example we are using a different video, Sintel, as it actually has some speech in it and therefore is better for illustrating how captions work!

    HTML Markup

    As mentioned above, we need to make use of the new HTML5 <track> element to add our caption files to the HTML5 video. We actually have our captions in three different languages — English, German, and Spanish — so we will reference all three of the relevant VTT files by adding <track> elements inside our HTML5 <video> element:

    <video id="video" controls preload="metadata">
       <source src="video/sintel-short.mp4" type="video/mp4">
       <source src="video/sintel-short.webm" type="video/webm">
       <track label="English" kind="captions" srclang="en" src="captions/vtt/sintel-en.vtt" default>
       <track label="Deutsch" kind="captions" srclang="de" src="captions/vtt/sintel-de.vtt">
       <track label="Español" kind="captions" srclang="es" src="captions/vtt/sintel-es.vtt">
    </video>

    As you can see, each <track> element has the following attributes set:

    • kind is given a value of captions, indicating the type of content the files contain
    • label is given a value indicating which language that caption set is for for example English or Deutsch — these labels will appear in the user interface to allow the user to easily select which caption language they want to see.
    • src is assigned a valid URL pointing to the relevant WebVTT caption file in each case.
    • srclang indicates what language each captions files’ contents are in.
    • The default attribute is set on the English <track> element, indicating to the browser that this is the default caption file definition to use when captions have been turned on and the user has not made a specific selection.

    In addition to adding the <track> elements, we have also added a new button to control the captions menu that we will build. As a consequence, the video controls now look as follows:

    <div id="video-controls" class="controls" data-state="hidden">
       <button id="playpause" type="button" data-state="play">Play/Pause</button>
       <button id="stop" type="button" data-state="stop">Stop</button>
       <div class="progress">
          <progress id="progress" value="0" min="0">
             <span id="progress-bar"></span>
          </progress>
       </div>
       <button id="mute" type="button" data-state="mute">Mute/Unmute</button>
       <button id="volinc" type="button" data-state="volup">Vol+</button>
       <button id="voldec" type="button" data-state="voldown">Vol-</button>
       <button id="fs" type="button" data-state="go-fullscreen">Fullscreen</button>
       <button id="captions" type="button" data-state="captions">CC</button>
    </div>

    CSS Changes

    The video controls have undergone some minor changes in order to make space for the extra button, but these are relatively straightforward.

    No image is used for the captions button, so it is simply styled as:

    .controls button[data-state="captions"] {
        height:85%;
        text-indent:0;
        font-size:16px;
        font-size:1rem;
        font-weight:bold;
        color:#666;
        background:#000;
        border-radius:2px;
    }

    There are also other CSS changes that are specific to some extra JavaScript implementation, but these will be mentioned at the appropriate place below.

    Caption implementation

    A lot of what we do to access the video captions revolves around JavaScript. Similar to the video controls, if a browser supports HTML5 video captions, there will be a button provided within the native control set to access them. However, since we have defined our own video controls, this button is hidden, and we need to define our own.

    Browsers do vary as to what they support, so we will be attempting to bring a more unified UI to each browser where possible. There’s more on browser compatibility issues later on.

    Initial setup

    As with all the other buttons, one of the first things we need to do is store a handle to the captions’ button:

    var captions = document.getElementById('captions');

    We also initially turn off all captions, in case the browser turns any of them on by default:

    for (var i = 0; i &lt; video.textTracks.length; i++) {
       video.textTracks[i].mode = 'hidden';
    }

    The video.textTracks property contains an array of all the text tracks attached to the video. We loop through each one and set its mode to hidden.

    Note: The WebVTT API gives us access to all the text tracks that are defined for an HTML5 video using the <track> element.

    Building a caption menu

    Our aim is to use the captions button we added earlier to display a menu that allows users to choose which language they want the captions displayed in, or to turn them off entirely.

    We have added the button, but before we make it do anything, we need to build the menu that goes with it. This menu is built dynamically, so that languages can be added or removed later by simply editing the <track> elements within the video’s markup.

    All we need to do is to go through the video’s textTracks, reading their properties and building the menu up from there:

    var captionsMenu;
    if (video.textTracks) {
       var df = document.createDocumentFragment();
       var captionsMenu = df.appendChild(document.createElement('ul'));
       captionsMenu.className = 'captions-menu';
       captionsMenu.appendChild(createMenuItem('captions-off', '', 'Off'));
       for (var i = 0; i < video.textTracks.length; i++) {
          captionsMenu.appendChild(createMenuItem('captions-' + video.textTracks[i].language, video.textTracks[i].language,         video.textTracks[i].label));
       }
       videoContainer.appendChild(captionsMenu);
    }

    This code creates a documentFragment, which is used to hold an unordered list containing our captions menu. First of all an option is added to allow the user to switch all captions off, and then buttons are added for each text track, reading the language and label from each one.

    The creation of each list item and button is done by the createMenuItem() function, which is defined as follows:

    var captionMenuButtons = [];
    var createMenuItem = function(id, lang, label) {
       var listItem = document.createElement('li');
       var button = listItem.appendChild(document.createElement('button'));
       button.setAttribute('id', id);
       button.className = 'captions-button';
       if (lang.length > 0) button.setAttribute('lang', lang);
       button.value = label;
       button.setAttribute('data-state', 'inactive');
       button.appendChild(document.createTextNode(label));
       button.addEventListener('click', function(e) {
          // Set all buttons to inactive
          captionMenuButtons.map(function(v, i, a) {
             captionMenuButtons[i].setAttribute('data-state', 'inactive');
          });
          // Find the language to activate
          var lang = this.getAttribute('lang');
          for (var i = 0; i < video.textTracks.length; i++) {
             // For the 'captions-off' button, the first condition will never match so all will captions be turned off
             if (video.textTracks[i].language == lang) {
                video.textTracks[i].mode = 'showing';
                this.setAttribute('data-state', 'active');
             }
             else {
                video.textTracks[i].mode = 'hidden';
             }
          }
          captionsMenu.style.display = 'none';
       });
       captionMenuButtons.push(button);
       return listItem;
    }

    This function builds the required <li> and <button> elements, and returns them so they can be added to the captions menu list. It also sets up the required event listeners on the button to toggle the relevant caption set on or off. This is done by simply setting the required caption’s mode attribute to showing, and setting the others to hidden.

    Once the menu is built, it is then inserted into the DOM at the bottom of the videoContainer.

    Initially the menu is hidden by default, so an event listener needs to be added to our captions button to toggle it:

    captions.addEventListener('click', function(e) {
       if (captionsMenu) {
          captionsMenu.style.display = (captionsMenu.style.display == 'block' ? 'none' : 'block');
       }
    });

    Caption menu CSS

    We also added some rudimentary styling for the newly created captions menu:

    .captions-menu {
        display:none;
        position:absolute;
        bottom:14.8%;
        right:20px;
        background:#666;
        list-style-type:none;
        margin:0;
        padding:0;
        width:100px;
        padding:10px;
    }
     
    .captions-menu li {
        padding:0;
        text-align:center;
    }
     
    .captions-menu li button {
        border:none;
        background:#000;
        color:#fff;
        cursor:pointer;
        width:90%;
        padding:2px 5px;
        border-radius:2px;
    }

    Styling the displayed captions

    One of the less well known about and supported features of WebVTT is the ability to style the individual captions (something called text cues) via CSS Extensions.

    The ::cue pseudo-element is the key to targetting individual text track cues for styling, as it matches any defined cue. There are only a handful of CSS properties that can be applied to a text cue:

    For example, to change the text colour of the text track cues you can write:

    ::cue {
       color:#ccc;
    }

    If the WebVTT file uses voice spans, which allow cues to be defined as having a particular “voice”:

    0
    00:00:00.000 --> 00:00:12.000
    <v Test>[Test]</v>

    Then this specific ‘voice’ will be stylable like so:

    ::cue(v[voice='Test']) {
       color:#fff;
       background:#0095dd;
    }

    Note: Some of the styling of cues with ::cue currently works on Chrome, Opera, and Safari, but not yet on Firefox.

    Browser Compatibility

    Browser support for WebVTT and the <track> element is fairly good, although some browsers differ slightly in their implementation.

    Internet Explorer

    Since Internet Explorer 10+ captions are enabled by default, and the default controls contain a button and a menu that offers the same functionality as the menu we just built. The default attribute is also supported.

    Note: IE will completely ignore WebVTT files unless you setup the MIME type. This can easily be done by adding an .htaccess file to an appropriate directory that contains AddType text/vtt .vtt.

    Safari

    Safari 6.1+ has similar support to Internet Explorer 11, displaying a menu with the different available options, with the addition of an “Auto” option, which allows the browser to choose.

    Chrome and Opera

    These browsers have similar implementations again: captions are enabled by default and the default control set contains a ‘cc’ button that turns captions on and off. Chrome and Opera ignore the default attribute on the <track> element and will instead try to match the browser’s language to the caption’s language.

    Firefox

    Firefox’s implementation was completely broken due to a bug, leading to Mozilla turning off WebVTT support by default (you can turn it on via the media.webvtt.enabled flag.) However, this bug looks to have been fixed and WebVTT support re-enabled as of Gecko 31, so this will not be a problem for Firefox final release users for much longer (on Gecko 29 as of the time of this writing.) this has been fixed as of Firefox 31, and everything works as it should.

    Plugins

    If, after reading through this article you decide that you can’t be bothered to do all of this and want someone else to do it for you, there are plenty of plugins out there that offer caption and subtitle support that you can use.

    playr
    This small plugin implements subtitles, captions, and chapters as well as both WebVTT and SRT file formats.
    jwplayer
    This video player is very extensive and does a lot more than simply support video captions. It supports WebVTT, SRT, and DFXP file formats.
    MediaElement.js
    Another complete video player that also support video captions, albeit only in SRT format.
    LeanBack Player
    Yet another video player that supports WebVTT captions as well as providing other standard player functionality.
    SublimeVideo
    This player also supports captions through WebVTT and SRT files.
    Video.js
    Supports WebVTT video subtitles.

    Note: You can find an excellent list of HTML5 Video Players and their current state at HTML5 Video Player Comparison.

  6. Mozilla at conferences – June edition

    Welcome to a quick round-up of what Mozillians have been talking about at events in and around June.

    Campus Party Guadalajara

  7. Frédéric Harper spoke at Devoxx UK about “Getting the best out of your design with responsive web design
  8. Robert Nyman spoke at JSCamp Romania about “Five stages of development (slidesvideo)”
  9. David Baron spoke at CSS Day about “Efficient CSS Animations (slideshowall slides)
  10. Chris Heilmann visited Campus Party Mexico to deliver the keynote “The Future of the Open Web (video, slides)”. There were also various talks by local Mozillians.
  11. Nick Desaulniers spoke at HTML5DevConf about “Raw WebGL (video)” whilst Chris Heilmann delivered the “Write less, achieve meh (notes)” keynote.
  12. Soledad Penadés covered an inordinate amount of amazing things at Goto Amsterdam in her “Invest in the future: build for the web! (notes and screencast)” talk.
  13. Open Source Bridge in Portland had a whole bunch of Mozillians presenting (the site also has lots of notes of what happened where and when and is worth visiting):
  14. Dale Harvey attended Scotch on the Rocks and talked about “The offline Web (slidesvideo)” and took part in the panel discussion on “When is Enough Enough (video)
  15. Campus Party Guadalajara

    If you want to know where we will present, check the Where is Mozilla? page. If you are interested in getting a Mozilla presenter for your event, why not tell us about it?

  16. ServiceWorkers and Firefox

    Since early 2013, Mozillians have been involved with the design of the Service Worker. Thanks to work by Google, Samsung, Mozilla, and others, this exciting new feature of the web platform has evolved to the point that it is being implemented in various web browser engines.

    What are Service Workers?

    At their simplest, Service Workers are scripts that act as client-side proxies for web pages. JavaScript code can intercept network requests, deliver manufactured responses and perform granular caching based on the unique needs of the application, a feature that the web platform has lacked before now. This powerful capability being made available to web developers enables, among other things, the creation of fully-functioning offline experiences. Jake Archibald has summarized some of these features in his blog post.

    Since Service Workers run in the “background”, they open up several possibilities for the Web that were previously only available on native platforms. Apart from the networking capabilities provided by the base specification, Service Workers are intended to be used by the Push API and the Background Sync API to deliver messages from the user-agent to web applications.

    Service Workers in Firefox

    A number of Mozillians have been hard at work implementing Service Workers in Gecko while Anne van Kesteren and Jonas Sicking help with the design and specification. Members of the Necko team and others have provided input from networking and related perspectives. Nikhil Marathe recently published a blog post about the status of Service Workers in Gecko.

    The Service Worker implementation in Gecko is landing in pieces as soon as they are finished and reviewed. For the time being, as the specification continues toward stability and other implementations — notably Blink’s — progress, all functionality in Gecko is behind the dom.serviceWorkers.enabled preference which is set to false by default but can be toggled in about:config.

    Our plan is that web developers will soon be able to exercise most Service Worker functionality in Firefox Nightly with the above preference flipped to true. The best plans can always be waylaid but we hope for this to happen by the end of September 2014 at the latest.

    Status of Service Worker implementations

    The inimitable Jake Archibald has written a tool to easily see the status of Service Worker implementations. You can follow along with the gecko implementation via the meta bug.

  17. WebIDE Lands in Nightly

    Editor’s note: if you want to help test it on a recent nightly you can toggle the devtools.webide.enabled preference in about:config. The WebIDE is available today under Tools>Web Developer>App Manager and will be renamed in tomorrow’s Nightly into WebIDE.

    If you’ve been following our Developer Tools series on the Mozilla Hacks blog, you’ve seen the developer tools evolve from pure inspection to a debugging environment both for web sites and apps on desktop and mobile. Today we want to introduce you to the next step of evolution: adding in-browser editing features across devices.

    WebIDE

    Developers tell us that they are not sure how to start app development on the Web, with so many different tools and templates that they need to download from a variety of different sources. We’re solving that problem with WebIDE, built directly into Firefox. Instead of starting from zero we provide you with a functioning blueprint app with the click of a button. You then have all the tools you need to start creating your own app based on a solid foundation. WebIDE helps you create, edit, and test a new Web application right from your browser. It lets you install and test apps on Firefox OS devices and simulators and integrates the Firefox Developer Tools for seamless debugging and inspection across those devices. This is a first step towards debugging across various platforms and devices over WiFi using open remote debugging APIs.

    Readability app open in WebIDE

    Getting Started

    After opening WebIDE you start creating your new application with a few clicks. You choose from a set of example starter templates – and we’re working with the community to build a wide variety of additional examples. You can help creating templates by visiting https://github.com/mozilla/mortar.

    Editing

    Once you’ve started your project, you can edit the source files in WebIDE. Its integrated editor – based on the open source CodeMirror editor with the tern.js code analysis framework – gives you a simple but powerful editor for HTML, JavaScript, and CSS files.

    Editing in WebIDE

    While working with your web application manifest, the application validator will help you find common problems before submitting to the marketplace.

    While this is enough for your basic editing needs, you might want to use your own preferred editor instead. To do this you can use a simple API which allows external editors to access all the advanced functionality of the tool – its runtime management, pushing applications to different devices and connecting Firefox Developer Tools. You can turn off our internal editor and leave WebIDE with a simple, clean interface for managing runtimes and validating applications. We want to make it easy for users of any code editor to sling their code around to various devices.

    Packaged App Demo in WebIDE

    Runtimes and Testing

    When you’re ready to test your application, choose a runtime – we’ll install Firefox OS simulators for you or help you connect to your Firefox OS device. Once connected to a runtime, you can use the tools that you’re used to in desktop Firefox to try your application. You have the same sort of rapid iteration that makes developing for the desktop web simple – just hit Ctrl/Command+R to reload the application on your device or simulator. You can get information about the device and runtime and take screenshots.

    Take a Look

    You can see the WebIDE in action in this screencast

    While we put on some finishing touches, WebIDE is checked into Firefox Nightly but is hiding behind a pref – if you want to help test it on a recent nightly you can toggle the devtools.webide.enabled preference in about:config:

    Enabling WebIDE in Firefox Nightly

    The Future: Developing for the Whole Web

    At the heart of the WebIDE is the Firefox Remote Debugging Protocol. The Firefox Developer Tools team has put a lot of work into making sure all of our tools work on remote devices running over USB, but that’s really only the beginning. With WebIDE managing these connections we can use the Firefox Developer Tools – and any other tools that want to use our protocol – anywhere.

    Right now this protocol is useful for Firefox Desktop, Firefox Android, and Firefox OS. But we aren’t stopping there. We’re working on a protocol adapter that will allow clients using the Firefox Remote Debugging Protocol – including the Developer Tools and WebIDE – talk to all mobile browsers, regardless of rendering engine or runtime. Our first targets are Chrome for Android and Safari on iOS.

    This web-everywhere project isn’t yet ready for testing, but if you’re excited to get involved or see how it is being implemented, it currently lives at https://github.com/campd/fxdt-adapters.

    If you’re building a tool and want to start playing with the protocol you can start now, developing against Firefox Desktop, Firefox Android, and Firefox OS. For information about the Remote Debugging Protocol, take a look at the documentation.

    Thanks for reading. We look forward to hearing your feedback in the comments, on our UserVoice feedback channel and on Twitter @firefoxdevtools.

  18. What’s new in Cordova 3.5.0 for Firefox OS

    The Cordova community recently released version 3.5.0 of the tools. This version includes some exciting improvements to the Firefox OS development workflow. Before we dive into the new features, make sure you have the latest version by running:

    $ sudo npm install -g cordova
    $ sudo npm install -g plugman

    Now that we’re all set up, let’s dive into the new features.

    Improved manifest management

    In previous versions of Cordova, developers had to manually edit the manifest.webapp file to add permissions and other app information. This file has crucial information that Firefox OS needs to interact with your app.

    Cordova has a configuration file called config.xml that already contains the same information needed for the manifest file. Cordova will create and update the manifest based on your config.xml file. In the new version, plugins can add configuration specifying which permissions are necessary. Whenever you run a cordova prepare, the manifest is updated based on your configuration. Now you can have all your app’s information in one place.

    Building packages with Cordova

    Firefox OS uses web technologies that do not require a compilation step to generate binaries. The related Cordova commands build and compile were left unimplemented and would throw an exception when called. That behavior was confusing and left some people wondering what went wrong.

    Now cordova build or Cordova compile will create a zip of your packaged app in the build folder inside the platform/firefoxos folder. A big thank you to the contributor Gert-Jan Braas for implementing this!

    Plugins

    A fresh batch of core plugins were released too. We added Firefox OS support to a few more plugins:

    To update to the latest version of the plugins, you need to remove and add them again. For example, to use the latest version of the file plugin run:

    $ cordova plugin rm org.apache.cordova.file
    $ cordova plugin add org.apache.cordova.file

    Replace the plugin name for the plugin you want to update. The geolocation and contacts plugins have been updated to support auto managing permissions, make sure you update them too.

    Check our status page for updated information on plugin status.

    What’s next

    A highly requested feature is support for emulate and run Cordova commands. We are working with the Dev Tools team to create an awesome experience for debugging Cordova applications using Firefox’s App Manager. Here is a sneak preview of what’s coming!

    Meanwhile you can debug your app by adding the platforms/firefoxos/www folder to the app manager in Firefox. For more information, check out Cordova for Firefox OS on MDN.

    We are working on creating default icons for a newly created app. They will serve as placeholders that can be easily replaced with your app’s brand.

    We also have a development status page where you can see up to the minute information on what is being worked on.

    We’d love to hear your feedback and feature requests. You can reach us in the #cordova channel on IRC, or through email at mozilla-cordova@mozilla.org or log your issues and requests on the Apache Cordova issue site. Also if you are interested in helping out with the project let us know.

  19. Breaking the Borders of IndexedDB

    In this article I want to share with you how to do some cool IndexedDB queries that aren’t ‘possible’ out of the box unless you add some ‘tricks’.

    The algorithms I’m going to show, except the ‘full-text-search’ one, were invented by me while I was writing on the open source javascript library Dexie.js. Some of them are not unique inventions since other people have discovered them too, but I believe that the case insensitive search algorithm, at least, is unique for my Dexie.js library.

    All code snippets in this article are conceptual. In case you need to look at bullet proof code, I encourage you to dive into the source of Dexie.js where all of these algorithms are implemented and thoroughly unit tested.

    The Four IndexedDB Queries

    IndexedDB API is only capable of doing four type of queries:

    1. IDBKeyRange.only (Exact match)
    2. IDBKeyRange.upperBound () – Find objects where property X is below certain value
    3. IDBKeyRange.lowerBound () – Find objects where property X is above certain value
    4. IDBKeyRange.bound() – Find objects where property X is between certain values.

    What I Will Show

    I am going to show you how to extend IndexedDB to support the following:

    anyOf()
    Find objects where property X equals any of a given set of possible values
    equalsIgnoreCase()
    Case insensitive search
    startsWith()
    Find strings starting with a certain prefix
    startsWithIgnoreCase()
    Find strings starting with case insensitive prefix
    or()
    Combine two queries and get the union of these two.
    full-text-search
    Find objects by searching for words contained in a large text

    The list could be much longer than that, but let’s start with these for now.

    None of these extended types of queries, except full-text-search require you to store any meta-data. They will work with any IndexedDB database already being used. For example, you do not have to store lowercase versions of strings that you need to do case insensitive matching on.

    So lets start showing the ‘tricks’ that breaks the borders of IndexedDB.

    anyOf()

    In SQL, this is equal to the IN keyword:

    SELECT * FROM TABLE WHERE X IN (a,b,c)

    The reason I begin with this one, is that it pretty straight forward and simple to understand. Understanding this one will make it easier to understand the algorithms for doing case insensitive search.

    Some Background Knowledge

    Let’s start by diving into some IndexedDB basics:
    To iterate ALL records in a table with IndexedDB, you call indexOrStore.openCursor() on the IDBObjectStore or IDBIndex instance. For each record, you’ll get called back on your onsuccess callback:

    // Start a transaction to operate on your 'friends' table
    var trans = db.transaction(["friends"], "readonly");
     
    // Get the 'name' index from your 'friends' table.
    var index = trans.objectStore("friends").index("name");
     
    // Start iteration by opening a cursor
    var cursorReq = index.openCursor();
     
    // When any record is found, you get notified in onsuccess
    cursorReq.onsuccess = function (e) {
     
        var cursor = e.target.result;
        if (cursor) {
            // We have a record in cursor.value
            console.log(JSON.stringify(cursor.value));
            cursor.continue();
        } else {
            // Iteration complete
        }
    };

    Overloaded Version of IDBCursor.continue()

    In the sample above, we call cursor.continue() without any argument. This will make the cursor advance to next key. But if we provide an argument to cursor.continue(nextKey), we tell the cursor to fast-forward to given key. If we were iterating the “name” index and wrote:

    cursor.continue("David");

    …the next onsuccess would have a cursor positioned at the first record where name equals “David” if that key exists. The specification requires that the cursor must be positioned at the first record that is equal to or greater than the specified key in the same sort order as the index.

    The Algorithm

    So lets say we have a huge database of friends and friends of friends with various names. Now we want to find all friends that have any of the following names: “David”, “Daniel” or “Zlatan”.

    We are going to do a cursor based iteration on the ‘name’ index as shown above, and use cursor.continue(nextName) to fast forward to the names we are looking for. When opening a cursor on an index, the order of found items will be in the sort order of that index. So if we do it on the ‘name’ index, we will get all friends in the sort order of their ‘name’.

    What we want to do first, is to sort() the array of names we are looking for, so we get the following JavaScript array:

    ["Daniel", "David", "Zlatan"]

    (since n comes before v). Then we do the following:

    1. call cursor.continue("Daniel") (first item in sorted set)
    2. onsuccess: Check if cursor.key == "Daniel". If so, include cursor.value in result and call cursor.continue() without arguments to check for more Daniels.
    3. When no more Daniels found, call cursor.continue("David") (next item…)
    4. onsuccess: Check if cursor.key == "David". If so, include cursor.value in result and call cursor.continue() without arguments to check for more Davids.
    5. When no more Davids found, call cursor.continue("Zlatan")
    6. onsuccess: Check if cursor.key == "Zlatan". If so, include cursor.value in result and call cursor.continue() without arguments to check for more Zlatans. Else, we could stop iterating by just don’t call cursor.continue() anymore because we know we wont find any more results (The set was sorted!)

    Sample Implementation of the Algorithm

    function comparer (a,b) {
        return a < b ? -1 : a > b ? 1 : 0;
    }
     
    function equalsAnyOf(index, keysToFind, onfound, onfinish) {
     
        var set = keysToFind.sort(comparer);
        var i = 0;
        var cursorReq = index.openCursor();
     
        cursorReq.onsuccess = function (event) {
            var cursor = event.target.result;
     
            if (!cursor) { onfinish(); return; }
     
            var key = cursor.key;
     
            while (key > set[i]) {
     
                // The cursor has passed beyond this key. Check next.
                ++i;
     
                if (i === set.length) {
                    // There is no next. Stop searching.
                    onfinish();
                    return;
                }
            }
     
            if (key === set[i]) {
                // The current cursor value should be included and we should continue
                // a single step in case next item has the same key or possibly our
                // next key in set.
                onfound(cursor.value);
                cursor.continue();
            } else {
                // cursor.key not yet at set[i]. Forward cursor to the next key to hunt for.
                cursor.continue(set[i]);
            }
        };
    }

    Case Insensitive Search

    Dexie.js implements case insensitive searching using a similar IDBCursor.continue(key) method as with the anyOf() algorithm above, however a little more complexity in the algorithm is needed.

    Let’s say we need to find “david” in table “friends” no matter its casing. “David” or “david” should both be found. Obviously, we could create an array containing all possible case combinations of “david” and then use the anyOf() algorithm above. However, the number of combinations would increase exponentially with the length of the key we’re looking for. But there is a trick we can use; since cursor.continue() will land on the next record in sort order, that will reveal what combinations we can skip when landing on the key that is not a match.

    What we do is to start searching for the lowest possible value of “David”, which would be “DAVID” since uppercase unicode characters have a lesser value than their corresponding lowercase version. If there is no “DAVID” in the DB, we will land on the least possible key yet greater than “DAVID”. Now, instead of testing the next combination of davids (which would be “DAVId”), we first inspect the key we landed on and from that key, we derive what would be the next “david” variant to search for. Look at the function nextCasing() in the code snippet below to see how we derive next possible version of a key based on the key that the IDBCursor landed on. I will not go through it line-by line here, but instead, I refer to the code comments in function nextCasing(key, lowerKey) {…} below.

    function findIgnoreCaseAlgorithm(index, needle, onfound, onfinish) {
     
        // index: An instance of IDBIndex
        // needle: The string to search for
        // onfound: A function to call for each found item
        // onfinish: A function to call when we're finshed searching.
     
        var upperNeedle = needle.toUpperCase();
        var lowerNeedle = needle.toLowerCase();
        var cursorReq = index.openCursor();
     
        cursorReq.onsuccess = function (event) {
            var cursor = event.target.result;
            if (!cursor) {
                // No more data to iterate over - call onfinish()
                onfinish();
                return;
            }
     
            var key = cursor.key;
            if (typeof key !== 'string') {
                // Just in case we stumble on data that isnt what we expect -
                // toLowerCase() wont work on this object. Check next.
                cursor.continue();
                return;
            }
     
            var lowerKey = key.toLowerCase();
            if (lowerKey === lowerNeedle) {
                onfound(cursor.value); // Notify caller we found somethin
                cursor.continue(); // Check next record, it might match too!
            } else {
                // Derive least possible casing to appear after key in sort order
                var nextNeedle = nextCasing(key, lowerKey, upperNeedle, lowerNeedle);
                if (nextNeedle) {
                    cursor.continue(nextNeedle);
                } else {
                    // No more possible case combinations to look for.
                    // Call onfinish() and dont call cursor.continue() anymore.
                    onfinish();
                }
            }
        };
     
        function nextCasing(key, lowerKey) {
            var length = Math.min(key.length, lowerNeedle.length); // lowerNeedle is from outer scope
            var llp = -1; // "llp = least lowerable position"
     
            // Iterate through the most common first chars for cursor.key and needle.
            for (var i = 0; i < length; ++i) {
                var lwrKeyChar = lowerKey[i];
     
                if (lwrKeyChar !== lowerNeedle[i]) {
                    // The char at position i differs between the found key and needle being
                    // looked for when just doing case insensitive match.
                    // Now check how they differ and how to trace next casing from this:
                    if (key[i] < upperNeedle[i]) {
                        // We could just append the UPPER version of the key we're looking for
                        // since found key is less than that.
                        return key.substr(0, i) + upperNeedle[i] + upperNeedle.substr(i + 1);
                    }
     
                    if (key[i] < lowerNeedle[i]) {
                        // Found key is between lower and upper version. Lets first append a
                        // lowercase char and the rest as uppercase.
                        return key.substr(0, i) + lowerNeedle[i] + upperNeedle.substr(i + 1);
                    }
     
                    if (llp >= 0) {
                        // Found key is beyond this key. Need to rewind to last lowerable
                        // position and return key + 1 lowercase char + uppercase rest.
                        return key.substr(0, llp) + lowerKey[llp] + upperNeedle.substr(llp + 1)
                    }
     
                    // There are no lowerable positions - all chars are already lowercase
                    // (or non-lowerable chars such as space, periods etc)
     
                    return null;
                }
     
                if (key[i] < lwrKeyChar) {
                    // Making lowercase of this char would make it appear after key.
                    // Therefore set llp = i.
                    llp = i; 
            }
     
            // All first common chars of found key and the key we're looking for are equal
            // when ignoring case.
            if (length < lowerNeedle.length) {
                // key was shorter than needle, meaning that we may look for key + UPPERCASE
                // version of the rest of needle.
                return key + upperNeedle.substr(key.length);
            }
     
            // Found key was longer than the key we're looking for
            if (llp < 0) {
                // ...and there is no way to make key we're looking for appear after found key.
                return null;
            } else {
                // There is a position of a char, that if we make that char lowercase,
                // needle will become greater than found key.
                return key.substr(0, llp) + lowerNeedle[llp] + upperNeedle.substr(llp + 1);
            }
        }
    }

    Performance

    In a performance test I created 10,000 objects with random strings and let the equalsIgnoreCase() algorithm try to find items in it. For Opera, it took between 4 and 5 milliseconds to find 6 matching items among 10,000. For Mozilla latest version, it took 7 ms. If instead iterating through all 10,000 items and comparing manually, it took 1514 ms for Mozilla and 346 ms for Opera.

    startsWith(str)

    Matching prefix of string keys is the most straight-forward trick you can do with IndexedDB. It’s not unique for Dexie.js as other libraries support it as well. However, for the sake of completeness, here is how it is done: Just do an IDBKeyRange.bound() where lowerBound is the prefix and upperBound is a string that would include all possible continuations of the prefix. The simplest way to do this is just to append a character with highest possible value; ‘\uffff’.

    IDBKeyRange.bound(prefix, prefix + '\uffff', false, false)

    startsWithIgnoreCase(str)

    When the matched string should be compared without case sensitivity, we need to do a cursor search, as for equalsIgnoreCase(). We can actually take the same algorithm but change how we compare each entry. Instead of comparing using the ‘==’ operator, we use String.indexOf(). This will give us the value 0 if the string starts with given value. So, just copy the code samples from equalsIgnoreCase() above and change the onsuccess part to the following:

    cursorReq.onsuccess = function (event) {
        var cursor = event.target.result;
        if (!cursor) {
            // No more data to iterate over - call onfinish()
            onfinish();
            return;
        }
     
        var key = cursor.key;
        if (typeof key !== 'string') {
            // Just in case we stumble on data that isnt what we expect -
            // toLowerCase() wont work on this object. Check next.
            cursor.continue();
            return;
        }
     
        var lowerKey = key.toLowerCase();
        if (lowerKey.indexOf(lowerNeedle) === 0) {
            onfound(cursor.value); // Notify caller we found somethin
            cursor.continue(); // Check next record, it might match too!
        } else {
            // Derive least possible casing to appear after key in sort order
            var nextNeedle = nextCasing(key, lowerKey, upperNeedle, lowerNeedle);
            if (nextNeedle) {
                cursor.continue(nextNeedle);
     
            } else {
                // No more possible case combinations to look for.
                // Call onfinish() and dont call cursor.continue() anymore.
                onfinish();
            }
        }
    };

    Logical OR

    IndexedDB has no support for logical or. You can only specify one keyrange at a time. However, what it does have support for, is to run several queries in parallel – even when using same transaction (as long as the queries are readonly queries). Even if the queries wouldn’t run in parallel, it would still work but a little less performant. The OR operation in Dexie.js has been unit tested with Chrome, IE, Firefox and Opera.

    The only thing we need to do except executing both queries in parallel, is to remove any duplicates. To do that, we can use a set of found primary keys. Whenever a new record match is found on any of the parallel queries, it first checks the set if it’s already included. If not, it calls onfound for the entry and sets set[primKey] = true so that if the same entry would be found on the other query, it would silently ignore to call onfound().

    Here’s how it’s done. The code snipped below is a simplified version only supporting the logical OR of two standard IDBKeyRange queries. With Dexie, you can do arbritary number of OR with any standard or extended operation such as equalsIgnoreCase().

    function logical_or(index1, keyRange1, index2, keyRange2, onfound, onfinish) {
        var openCursorRequest1 = index1.openCursor(keyRange1);
        var openCursorRequest2 = index2.openCursor(keyRange2);
        assert(index1.objectStore === index2.objectStore); // OR can only be done on same store
        var primKey = index1.objectStore.keyPath;
        var set = {};
        var resolved = 0;
     
        function complete() {
            if (++resolved === 2) onfinish();
        }
     
        function union(item) {
            var key = JSON.stringify(item[primKey]);
            if (!set.hasOwnProperty(key)) {
                set[key] = true;
                onfound(item);
            }
        }
     
        openCursorRequest1.onsuccess = function (event) {
            var cursor = event.target.result;
            if (cursor) {
                union(cursor.value);
            } else {
                complete();
            }
        }
     
        openCursorRequest2.onsuccess = function (event) {
            var cursor = event.target.result;
            if (cursor) {
                union(cursor.value);
            } else {
                complete();
            }
        }
    }

    To access this algorithm with Dexie.js, you type something like the following:

    db.friends.where('name').equalsIgnoreCase('david').or('shoeSize').above(40).toArray(fn)

    … which would make given callback function (fn) be called with an array of the results.

    When using parallel OR execution, the sort order will not be valid. Partly because the two different queries execute on different indexes with different sort order, but mainly because the two queries run in parallel by the browser background threads and we cannot know which onsuccess is called before the other. However, this can be resolved by implementing onfound to push the items to an array, and onfinish to sort it using any requested sort order:

    var index1 = transaction.objectStore("friends").index("name");
    var index2 = transaction.objectStire("friends").index("shoeSize");
    var keyRange1 = IDBKeyRange.bound ("Da", "Da\uffff");
    var keyRange2 = IDBKeyRange.lowerBound (40, true);
    //
    // SELECT * FROM friends WHERE name LIKE 'Da%' OR shoeSize &gt; 40 ORDERBY shoeSize;
    //
     
    var result = [];
    logical_or (index1, keyRange1, index2, keyRange2, function (friend) {
        result.push(friend);
    }, function () {
        result.sort (function (a,b) { return a.shoeSize - b.shoeSize; });
     
    });

    Full Text Search

    Searching for certain words in a large string (text field) is another common use case that is not supported out-of-the-box by IndexedDB. However, it can be easily implemented by splitting the text into words and store the resulting set in a ‘multiEntry’ index. With all my other algorithms in this article, no meta-data is required to make them work, but with Full Text Search, it must be prepared by adding a specific multiEntry index and each time a text is changed, the multiEntry index must be updated accordingly with the resulting set of words in the text.

    1. When defining your schema, create an index “words” with multiEntry set to true.
      • In bare-bone IndexedDB: store.createIndex("words", "words", { multiEntry: true });
      • In Dexie.js: Prefix with asterisk (*):db.version(1).stores({blogs: '++id,title,text,*words'});
    2. Whenever you update your text, split the text into words and store all unique words in the “words” property of the object.

      var blogEntry = new BlogEntry(
       
          "Blogging about IndexedDB",
       
          "Much to say about IndexedDB there is... bla bla bla - hope I'm not boring you...");
       
      db.put(blogEntry);
       
      blogEntry.setText("...changing my blog post to another IndexedDB text...");
       
      db.put(blogEntry);
       
      function BlogEntry(title, words) {
       
          ...
       
          this.setText = function (value) {
              this.text = value;
              this.words = getAllWords(value);
          }
       
          function getAllWords(text) {
              var allWordsIncludingDups = text.split(' ');
              var wordSet = allWordsIncludingDups.reduce(function (prev, current) {
                  prev[current] = true;
                  return prev;
              }, {});
              return Object.keys(wordSet);
          }
      }
    3. To find an item by searching for a word, use the “words” index to launch the query on.
      • bare-bone IndexedDB: index.openCursor(IDBKeyRange.only('IndexedDB'));
      • Dexie.js: db.blogs.where('words').equalsIgnoreCase('indexeddb')

    An example of how to do this with Dexie follows in the FullTextSearch.js file in Dexie

    Live Samples

    If you want to play with the samples in your browser or phone, go to Samples on the Dexie.js wiki where you have these samples and some more to experiment with.

    A Last Word

    Whether you are just curious about the IndexedDB API, or you’re implementing your own IndexedDB library, I hope the techniques shown in this article could be useful for you when you need to maximize the posibilities with IndexedDB.

    I would also like to encourage you to have a look at Dexie.js to discover how it may help you achieve your goals. A lot of effort has been put to make this library well documented and tested. The library is still young but the few users that has discovered it so far, have expressed a great deal of enthusiasm in the Dexie.js forum.