Mozilla

Firefox Articles

Sort by:

View:

  1. Handling click-to-activate plugins using JavaScript

    From Firefox 26 onwards — and in the case of insecure Flash/Java in older Firefox versions — most plugins will not be automatically activated. We therefore can no longer plugins starting immediately after they have been inserted into the page. This article covers JavaScript techniques we can employ to handle plugins, making it less likely that affected sites will break.

    Using a script to determine if a plugin is installed

    To detect if a plugin is actually installed, we can query navigator.mimeTypes for the plugin MIME type we intend to use, to differentiate between plugins that are not installed and those that are click-to-activate. For example:

    function isJavaAvailable() {
        return 'application/x-java-applet' in navigator.mimeTypes;
    }

    Note: Do not iterate through navigator.mimeTypes or navigator.plugins, as enumeration may well be removed as a privacy measure in a future version of Firefox.

    Using a script callback to determine when a plugin is activated

    The next thing to be careful of is scripting plugins immediately after instances are created on the page, to avoid breakage due to the plugin not being properly loaded. The plugin should make a call into JavaScript after it is created, using NPRuntime scripting:

    function pluginCreated() {
        document.getElementById('myPlugin').callPluginMethod();
    }
    <object type="application/x-my-plugin" data="somedata.mytype" id="myPlugin">
      <param name="callback" value="pluginCreated()">
    </object>

    Note that the “callback” parameter (or something equivalent) must be implemented by your plugin. This can be done in Flash using the flash.external.ExternalInterface API, or in Java using the netscape.javascript package.

    Using properties on the plugin to determine when it activated

    When using a plugin that doesn’t allow us to specify callbacks and we can’t modify it, an alternative technique is to test for properties that the plugin should have, using code constructs like so:

    <p id="myNotification">Waiting for the plugin to activate!</p>
    <object id="myPlugin" type="application/x-my-plugin"></object>
    window.onload = function () {
        if (document.getElementById('myPlugin').myProperty !== undefined) {
            document.getElementById('myNotification').style.display = 'none';
            document.getElementById('myPlugin').callPluginMethod();
        } else {
            console.log("Plugin not activated yet.");
            setTimeout(checkPlugin, 500);
        }
    }

    Making plugins visible on the page

    When a site wants the user to enable a plugin, the primary indicator is that the plugin is visible on the page, for example:

    Screenshot of the silverlight plugin activation on the Netflix website.

    If a page creates a plugin that is very small or completely hidden, the only visual indication to the user is the small icon in the Firefox location bar. Even if the plugin element will eventually be hidden, pages should create the plugin element visible on the page, and then resize or hide it only after the user has activated the plugin. This can be done in a similar fashion to the callback technique we showed above:

    function pluginCreated() {
      // We don't need to see the plugin, so hide it by resizing
      var plugin = document.getElementById('myPlugin');
      plugin.height = 0;
      plugin.width = 0;
      plugin.callPluginMethod();
    }
    <!-- Give the plugin an initial size so it is visible -->
    <object type="application/x-my-plugin" data="somedata.mytype" id="myPlugin" width="300" height="300">
      <param name="callback" value="pluginCreated()">
    </object>

    Note: For more basic information on how plugins operate in Firefox, read Why do I have to click to activate plugins? on support.mozilla.org.

  2. Firefox OS Security: Part 1 – The Web Security Model

    When presenting Firefox OS to people, security is a big topic. Can an operating system built on web technologies be secure? What has Mozilla built in to avoid drive-by downloads and malware? In this two part video series Christian Heilmann (@codepo8), principal evangelist of Mozilla, talks to Michael Coates (@_mwc), chair of @OWASP Board about all things security in Firefox OS.

    Firefox OS was built on top of the technologies that power the Web. Following Mozilla’s security practices and knowledge from over 10 years of securing Firefox, Firefox OS is engineered as a multi-tiered system that protects users while delivering the power of the mobile web. The design ensures users are in control of their data and developers have APIs and technologies at their disposal to unlock the power of the Web.

    Watch the following video where we talk more about the security design and controls present in Firefox OS. In this, the first of two videos on Firefox OS security, we’ll cover items such as the multi-tiered architecture, the permission model, run time decision making, protection of users data and the update model. You can watch the video on YouTube.

    Additional links for more information:

  3. Firefox Developer Tools: Episode 27 – Edit as HTML, Codemirror & more

    Firefox 27 was just uplifted to the Aurora release channel which means we are back to report on new features in Firefox Developer Tools. Below are just some of the new features, you can also take a look at all bugs resolved in DevTools for this release).

    JS Debugger: Break on DOM Events

    You can now automatically break on a variety of DOM events, without needing to manually set a breakpoint. To do this, click on the “Expand Panes” button on the top right of the debugger panel (right next to the search box). Then flip over to the events tab. Click on an event name to automatically pause the next time it happens. This will only show events that currently have listeners bound from your code. If you click on one of the headings, like “Mouse” or “Keyboard”, then all of the relevant events will be selected.

    Inspector improvements

    We’ve listened to feedback from web developers and made a number of enhancements to the Inspector:

    Edit as HTML

    Now in the inspector, you can right click on an element and open up an editor that allows you to set the outerHTML on an element.

    Select default color format

    You now have an option to select the default color format in the option panel:

    Color swatch previews

    The Developer Tools now offer color swatch previews that show up in the rule view:

    Image previews for background image urls

    Highly requested, we now offer image previews for background image URLs:

    In addition to above improvements, Mutated DOM elements are now highlighted in the Inspector.

    Keep an eye out for more tooltips coming soon, and feel free to chime in if you have any others you’d like to see!

    Codemirror

    Codemirror is a popular HTML5-based code editor component used on web sites. It is customizable and theme-able. The Firefox Devtools now use CodeMirror in various places: Style editor, Debugger, Inspector (Edit as HTML) and Scratchpad.

    From the Option panel, the user can select which theme to use (dark or light).

    WebConsole: Reflow Logging

    When the layout is invalidated (CSS or DOM changes), gecko needs to re-compute the position of some nodes. This computation doesn’t happen immediatly. It’s triggered for various reasons. For example, if you do “node.clientTop”, gecko needs to do this computation. This computation is called a “reflow”. Reflows are expensive. Avoiding reflows is important for responsiveness.

    To enable reflow logging, check the “Log” option under the “CSS” menu in the Console tab. Now, everytime a reflow happens, a log will be printed with the name of the JS function that triggered this reflow (if caused by JS).

    That’s all for this time. Hope you like the new improvements!

  4. Reintroducing the Firefox Developer Tools, part 1: the Web Console and the JavaScript Debugger

    This is part one, out of 5, focusing on the built-in Developer Tools in Firefox, their features and where we are now with them. The intention is to show you all the possibilities available, the progress and what we are aiming for.

    Firefox 4 saw the launch of the Web Console, the first of the new developer tools built into Firefox. Since then we’ve been adding more capabilities to the developer tools, which now perform a wide range of functions and can be used to debug and analyze web applications on desktop Firefox, Firefox OS, and Firefox for Android.

    cola1

    This is the first in a series of posts in which we’ll look at where the developer tools have got to since Firefox 4. We’ll present each tool in a short screencast, then wrap things up with a couple of screencasts illustrating specific workflow patterns that should help you get the most of the developer tools. These will include scenarios such as mobile development, and altering and debugging CSS based applications, etc.

    In this first post we present the latest Web Console and JavaScript Debugger.

    Web Console

    The Web Console is primarily used to display information associated with the currently loaded web page. This includes HTML, CSS, JavaScript, and Security warnings and errors. In addition network requests are displayed and the console indicates whether they succeeded or failed. When warnings and errors are detected the Web Console also offers a link to the line of code which caused the problem. Often the Web Console is the first stop in debugging a Web Application that is not performing correctly.

    webconsole

    The Web Console also lets you execute JavaScript within the context of the page. This means you can inspect objects defined by the page, execute functions within the scope of the page, and access specific elements using CSS selectors. The following screencast presents an overview of the Web Console’s features.

    Take a look at the MDN Web Console documentation for more information.

    JavaScript Debugger

    The JavaScript Debugger is used for debugging and refining JavaScript that your Web Application is currently using. The Debugger can be used to debug code running in Firefox OS and Firefox for Android, as well as Firefox Desktop. It’s a full-featured debugger providing watch expressions, scoped variables, breakpoints, conditional expressions, step-through, step-over and run to end functionality. In addition you can change the values of variables at run time while the debugger has paused your application.

    debugger

    The following screencast illustrates some of the features of the JavaScript Debugger.

    For more information on the JavaScript Debugger, see the MDN Debugger documentation.

    Learn More

    These screencasts give a quick introduction to the main features of these tools. For the full details on all of the developer tools, check out the full MDN Tools documentation.

    Coming Up

    In the next post in this series we will be delving into the Style Editor and the Scratchpad. Please give us your feedback on what features you would like to see explained in more detail within the comments.

  5. User-Agent detection, history and checklist

    History

    User-Agent: <something> is a string of characters sent by HTTP clients (browsers, bots, calendar applications, etc.) for each individual HTTP request to a server. The HTTP Protocol as defined in 1991 didn’t have this field, but the next version defined in 1992 added User-Agent in the HTTP requests headers. Its syntax was defined as “the software product name, with an optional slash and version designator“. The prose already invited people to use it for analytics and identify the products with implementation issues.

    This line if present gives the software program used by the original client. This is for statistical purposes and the tracing of protocol violations. It should be included.

    Fast forward to August 2013, the HTTP/1.1 specification is being revised and also defines User-Agent.

    A user agent SHOULD NOT generate a User-Agent field containing
    needlessly fine-grained detail and SHOULD limit the addition of
    subproducts by third parties. Overly long and detailed User-Agent
    field values increase request latency and the risk of a user being
    identified against their wishes (“fingerprinting”).

    Likewise, implementations are encouraged not to use the product
    tokens of other implementations in order to declare compatibility
    with them
    , as this circumvents the purpose of the field. If a user
    agent masquerades as a different user agent, recipients can assume
    that the user intentionally desires to see responses tailored for
    that identified user agent, even if they might not work as well for
    the actual user agent being used.

    Basically, the HTTP specification discouraged since its inception the detection of the User-Agent string for tailoring the user experience. Currently, the user agent strings have become overly long. They are abused in every possible way. They include detailed information. They lie about what they really are and they are used for branding and advertising the devices they run on.

    User-Agent Detection

    User agent detection (or sniffing) is the mechanism used for parsing the User-Agent string and inferring physical and applicative properties about the device and its browser. But let get the record straight. User-Agent sniffing is a future fail strategy. By design, you will detect only what is known, not what will come. The space of small devices (smartphones, feature phones, tablets, watches, arduino, etc.) is a very fast-paced evolving space. The diversity in terms of physical characteristics will only increase. Updating databases and algorithms for identifying correctly is a very high maintenance task which is doomed to fail at a point in the future. Sites get abandoned, libraries are not maintained and Web sites will break just because they were not planned for the future coming devices. All of these have costs in resources and branding.

    New solutions are being developed for helping people to adjust the user experience depending on the capabilities of the products, not its name. Responsive design helps to create Web sites that are adjusting for different screen sizes. Each time you detect a product or a feature, it is important to thoroughly understand why you are trying to detect this feature. You could fall in the same traps as the ones existing with user agent detection algorithms.

    We have to deal on a daily basis with abusive user agent detection blocking Firefox OS and/or Firefox on Android. It is not only Mozilla products, every product and brand has to deal at a point with the fact to be excluded because they didn’t have the right token to pass an ill-coded algorithm. User agent detection leads to situation where a new player can hardly enter the market even if it has the right set of technologies. Remember that there are huge benefits to create a system which is resilient to many situations.

    Some companies will be using the User-Agent string as an identifier for bypassing a pay-wall or offering specific content for a group of users during a marketing campaign. It seems to be an easy solution at first but it creates an environment easy to by-pass in spoofing the user agent.

    Firefox and Mobile

    Firefox OS and Firefox on Android have very simple documented User-Agent strings.

    Firefox OS

    Mozilla/5.0 (Mobile; rv:18.0) Gecko/18.0 Firefox/18.0

    Firefox on Android

    Mozilla/5.0 (Android; Mobile; rv:18.0) Gecko/18.0 Firefox/18.0

    The most current case of user agent detection is to know if the device is a mobile to redirect the browser to a dedicated Web site tailored with mobile content. We recommend you to limit your detection to the simplest possible string by matching the substring mobi in lowercase.

    /mobi/i

    If you are detecting on the client side with JavaScript, one possibility among many would be to do:

    // Put the User Agent string in lowercase
    var ua = navigator.userAgent.toLowerCase();
    // Better to test on mobi than mobile (Firefox, Opera, IE)
     
    if (/mobi/i.test(ua)) {
     
        // do something here
     
    } else {
     
        // if not identified, still do something useful
     
    }

    You might want to add more than one token in the if statement.

    /mobi|android|touch|mini/i.test(ua)

    Remember that whatever the number of tokens you put there, you will fail at a point in the future. Some devices will not have JavaScript, will not have the right token. The pattern or the length of the token was not as you had initially planned. The stones on the path are plenty, choose the way of the simplicity.

    Summary: UA detection Checklist Zen

    1. Do not detect user agent strings
    2. Use responsive design for your new mobile sites (media queries)
    3. If you are using a specific feature, use feature detections to enhance, not block
    4. And if finally you are using UA detection, just go with the most simple and generic strings.
    5. Always provide a working fallback whatever the solutions you chose are.

    Practice. Learn. Imagine. Modify. And start again. There will be many road blocks on the way depending on the context, the business requirements, the social infrastructure of your own company. Keep this checklist close to you and give the Web to more people.

  6. New Features of Firefox Developer Tools: Episode 25

    Firefox 25 was just uplifted to the Aurora release channel which means we are back to report about new features in Firefox Developer Tools.

    Here’s a summary of some of the most exciting new features, and to get the whole picture you can check the complete list of resolved bugzilla tickets.

    Black box libraries in the Debugger

    In modern web development, we often rely on libraries like JQuery, Ember, or Angular, and 99% of the time we can safely assume that they “just work”. We don’t care about the internal implementation of these libraries: we treat them like a black box. However, a library’s abstraction leaks during debugging sessions when you are forced to step through its stack frames in order to reach your own code. To alleviate this problem, we introduced black boxing: a feature where you can tell the debugger to ignore the details of selected sources.

    To black box a source, you can either mark them one at a time by disabling the little eyeball next to it in the sources list:
    eyeball

    Or you can black box many sources at once by bringing up the developer toolbar with Shift+F2 and using the dbg blackbox command:

    dbg blackbox --glob *-min.js[source]

    When a source is black boxed:

    • Any breakpoints it may have are disabled.
    • When “pause on exceptions” is enabled, the debugger won’t pause when an exception is thrown in the black boxed source; instead it will wait until (and if) the stack unwinds to a frame in a source that isn’t black boxed.
    • The debugger will skip through black boxed sources when stepping.

    To see this in action and learn more about the details, check out the black boxing screencast on YouTube.

    Replay and edit requests in the Network Monitor

    You can now debug a network request by modifying headers before resending it. Right-click on an existing request and select the “resend” context menu item:

    resend request

    Now you can tweak the HTTP method, URL, headers, and request body before sending the request off again:

    tweak

    CSS Autocompletion in the inspector

    Writing CSS in the inspector is now much easier as we enabled autocompletion of CSS properties and values.

    autocomplete

    What’s more, it even works on inline style attributes

    inline

    Aside: this feature was implemented by contributors Girish Sharma and Mina Almasry. If you want to take your tools into your own hands too, check out our wiki page on how to get involved with developer tools.

    Execute JS in the current paused frame

    One request we’ve heard repeatedly is the ability to execute JS from the webconsole in the scope of the current paused frame in the debugger rather than the global scope. This is now possible. Using the webconsole to execute JS in the current frame can make it much easier to debug your apps.

    Edit: The webconsole has actually been executing in the current frame since Firefox 23, in Firefox 25 the scratchpad will execute in the current frame as well.

    Import and export profiled data in the Profiler

    Hacking on a shared project and think you found a performance regression in some bit of code owned by one of your friends? Don’t just file a github issue with steps to reproduce the slowness, export and attach a profile of the code that shows exactly how much slowness there is, and where it occurs. Your friend will thank you when he or she is trying to reproduce and debug the regression. Click the “import” button next to the start profiling button to load a profile from disk, and hit “save” on an existing profile to export it.

    profileimport

    When can I use these features?

    All of these features and more are available in the Aurora release channel. In another 12 weeks, these features will roll over into Firefox stable.

    Have some feedback about devtools? Ping @FirefoxDevTools on Twitter, or swing by #devtools on irc.mozilla.org.

  7. New Features in Firefox Developer Tools: Episode 24

    Releases have recently rolled, and there are a slew of new features coming to the developer tools in Firefox 24. Firefox 24 is currently in our alpha channel Firefox Aurora, and will hit the main release channel on September 17th, 2013.

    A little alitteration leads to lots of cool features coming this release. I’m only touching on a few highlights here; to see a comprehensive list of all the new changes to developer tools, check out the full list of bugs fixed since last release.

    Console Clean Ups

    An oft-requested feature, the console will now clear all of the existing logs when you refresh the page. If you need logs to persist across reloads, don’t worry: this behavior is toggle-able in the options panel. Here’s to less clutter!

    Users were also reporting that the “clear” button was confusing. It wasn’t obvious whether it was clearing the logs or the the filter text because the button as adjacent to the filtering search box. We moved it to so that this won’t happen anymore!

    New position of the clear button

    Source Map Sourcery

    Source maps are now enabled by default. If you need to debug the JavaScript generated by your tool or compiler rather than the original source, you can still toggle them on and off in the debugger options.

    Furthermore, because of issues with Internet Explorer, the source map specification was updated so that you link a source map to its script with //#
    sourceMappingURL=...
    rather than the deprecated //@ sourceMappingURL=.... We will still respect the old style, but will warn you that it is deprecated. Here is a summary of the why this change was made.

    Finally, we will load the text of a source mapped source from its source map’s sourcesContent property, if it exists. This property was added to source maps to enable dynamic source map creation, and to bundle your sources into the source map so that you don’t need to juggle extra files on your server.

    Debugger Developments

    Frozen, sealed, and non-extensible objects are now visually distinguished in the debugger’s variables view.

    Frozen, sealed, and non-extensible objects

    Also, when stepping out of a stack frame in the debugger, you will now see either the return value or the exception thrown (depending on why you are exiting the frame) in the variables view labeled <return> and <exception> respectively.

    A stack frame's return value

    A stack frame's exception

    JavaScript-Free Fun

    Need to see how your web page gracefully degrades when JavaScript is disabled? Head on over to the options panel and you can quickly disable JavaScript for the current tab. This persists until either you close the tab or the developer tools.

    Network Monitor Magic

    You can now filter by types of requests in the network monitor. Want to see only requests for stylesheets? Or maybe only XHRs? We’ve got your back. On top of that, there are now total number of requests, file sizes, and response times listed in the bottom that update with you filters, so you can see just how many KBs (or even MBs) of images you are requesting on every page load.

    Filtering only images

    Profiler Pleasantries

    You can now control the profiler from your web app’s code base via console.profile() to start a new profile, and console.profileEnd() to end it. No more trimming your sample range, start and stop profiling in the exact places you want to from your code!

    That’s it for this episode. Keep on hacking the free web!

  8. Network Monitor, now in Firefox Beta

    The Firefox Developer Tools team is particularly proud announce that Firefox 23 (in Firefox Beta, to be released today) ships with an initial but very functional Network Monitor tool that not only provides similar functionality to other tool sets, but in many improves on them. This important step is the result of lots of hard work from the entire team, but in particular Victor Porof and Mihai Sucan.

    Let’s get started

    To give you the best idea of how the network monitor works, here’s a series of steps to follow:

    1. Download, install and run Firefox 23 beta if you haven’t already (or Firefox Aurora or Firefox Nightly – more on Firefox and the release channels).
    2. Open the Network Monitor tool:
      • from the menu: Tools => Web Developer => Network
      • from the keyboard: Ctrl + Alt + Q (Windows/Linux) or Cmd + Alt + Q (Mac OS X)
    3. Load an interesting site.

    As Firefox loads the page you’ll see each individual request get added as a row, much as you would expect from other tools. In particular it is now very easy to visualize not just how quickly parts of the page load and in what order, but also where problems are: missing assets, slow web servers, buggy apis.

    As you get used to using the Network Monitor, you’ll also notice that you don’t need to necessarily hit refresh on a page that is already loaded, once you open the network tool any subsequent requests that happen will be logged. For example, if you’re interested in xhr requests made by an app, just open the monitor and start clicking buttons – you should see any api calls logged as normal without needing to reload.

    Status at a glance

    One thing you should notice right away about the way the Network tool displays a page load is that we’ve taken care to provide both information and design cues to show you what is happening at a glance as the page loads:

    We’ve tried to create a clean UI that delivers key aspects of how an entire page loads without creating too much visual clutter:

    • An error such as a 404 are colored dark red to make them stand out visually from the green successful or yellow cached requests.
    • The url is parsed out and only the file name and domain are shown.
    • By default the load is sorted in chronological order, but you can also sort by any of the top columns, for example by domain or filename.
    • You can easily see the type of request, whether it be html content, an image or an API request.
    • You can click on the column headers to sort requests by method, file name, domain, type or size.

    It’s all in the details

    Once all the page’s assets have been loaded, clicking on a single request opens up the sidebar view which provides detailed information about that request, broken down into panels that display different aspects of a given request: Headers, Cookies, Parameters, Response and Timings. Similar to how the Inspector tool works, this panel is responsive and will neatly pop down below the request list if you’ve docked the tools to the right side:

    A quick tour

    The Headers panel is particularly useful for debugging asset caching problems, and this panel provides parsed lists for both the request and response headers that also supports filtering based on either name or value:

    The Timings panel gives you a detailed, visual breakdown of where time was spent for the selected request. Is it DNS? Is it the server itself?

    If you’re debugging API requests or form posts, you’ll really like how we’ve parsed out the url parameters in the Params panel:

    Are you using a lot of API calls to improve the responsiveness of your app? JSON responses are parsed and displayed as a navigable object in the Response panel to make it easy to check on the data coming back from the server. You can filter the data based on JSON values as well.

    Next Steps

    It’s great to deliver the Network Monitor to a much wider set of developers in Firefox 23 Beta but we haven’t stopped adding new features in the meantime. Firefox 24 ( now in the Aurora channel ) features several notable improvements that have landed in the last 6 weeks or so:

    • You can now filter requests based on the request ‘type’, including XHR, images, html.
    • POST request parameters are supported in the Params panel.
    • You can browse the list of requests via the Up/Down arrow keys.
    • Many more bug fixes and refinements!
  9. Content Security Policy 1.0 lands in Firefox Aurora

    The information in this article is based on work together with Ian Melven, Kailas Patil and Tanvi Vyas.

    We have just landed support for the Content Security Policy (CSP) 1.0
    specification
    in Firefox Aurora (Firefox 23), available as of tomorrow (May 30th). CSP is a security mechanism that aims to protect a website against content injection attacks by providing a whitelist of known-good domain names to accept JavaScript (and other content) from. CSP does this by sending a Content-Security-Policy header with the document it protects (yes, we lost the X prefix with the 1.0 version of the spec).

    To effectively protect against XSS, a few JavaScript features have to be
    disabled:

    • All inline JavaScript is disallowed. This means, that all the JavaScript code must be placed in a separate file that is linked via <script src=... >
    • All calls to functions which allow JavaScript code being executed from strings (e.g., eval) are disabled

    CSP now more intuitive and consistent

    While Firefox has had support for CSP since its invention here at Mozilla, things have been changing a lot. The streamlined development of a specification within the W3C has made the concept more intuitive and consistent. Most directives in a CSP header are now of a unified form which explicitly specifies the type of content you want to restrict:

    • img-src
    • object-src
    • script-src
    • style-src and so on.

    Oh and if you feel like you must allow less secure JavaScript coding styles, you can add the values unsafe-inline or unsafe-eval to your list of script sources. (This used to be inline-script and eval-script before).

    Start protecting your website by implementing CSP now!

    But wait – isn’t that a bit tedious… Writing a complex policy and making sure that you remembered all the resources that your website requires? Don’t fret! Here comes UserCSP again!

    Generate your Content Security Policies with UserCSP!

    During the last few months, Kailas Patil, a student in our Security Mentorship Program has continued his GSoC work from last year to update UserCSP.

    UserCSP is a Firefox add-on that helps web developers and security-minded users use CSP. Web developers can create a Content Security Policy (CSP) for their site by using UserCSP’s infer CSP feature. This feature can list required resource URLs and turn them into a policy ready to plug into a CSP header.

    In addition, UserCSP is the first step to expose a policy enforcement mechanism directly to web users. Furthermore, users can enforce a stricter policy than a page supplies through the add-on or apply a policy to certain websites that don’t currently support CSP.

    While earlier versions of UserCSP were more aligned to content security policies as originally invented at Mozilla, this version is updated to be in compliance with the CSP 1.0 specification. This means that policies derived with this add-on may work in all browsers as soon as they support the specification. Hooray!

    As this evolves and ships, our MDN documentation on Content Security Policy (CSP) will keep on evolving, and we also plan to write more about this in the Mozilla Security Blog in the next few weeks, so stay tuned!

  10. Compiling to JavaScript, and Debugging with Source Maps

    Update 2013/05/29: I have updated the article to reflect recent changes in the source map specification where the //@ syntax for linking a source map to a script has been deprecated in favor of //# due to problems with Internet Explorer.

    This is a tutorial on how to write a compiler which generates JavaScript as its target language, and maintains line and column meta-data in source maps for debugging. Storing line and column coordinates in a source map allows the end-user of the compiler to debug the source code that they wrote, rather than the ugly, generated JavaScript they are not familiar with.

    In this tutorial, we will be compiling a small Reverse Polish Notation, or RPN, language to JavaScript. The language is super simple, and is nothing more than simple arithmetic with variable storage and output capabilities. We are keeping the language simple so that we can focus on integrating source maps with the compiler, rather than language implementation details.

    Availability

    Initial support for source maps in the debugger is available in Firefox 23 (Aurora at time of writing) with more improvements coming in Firefox 24 (Nightly at time of writing). Chrome DevTools also have support for source maps.

    Overview of the Source Language

    RPN uses postfix notation, meaning that the operator follows its two operands. One of the benefits of RPN is that as long as we limit ourselves to binary operators, we do not need any parentheses, and do not need to worry about operator precedence.

    Here is an example program in our source language:

    a 5 =;
    b 3 =;
    c a b + 4 * =;

    This is an equivalent program written in a language which uses infix notation for its arithmetic operators:

    a = 5;
    b = 3;
    c = (a + b) * 4;

    Our language will support addition, subtraction, multiplication, division, assignment, and printing. The print operator’s first operand is the value to print, the second operand is how many times to print the value and must be greater than or equal to one:

    5 1 print;
    # Output:
    # 5
     
    3 4 print;
    # Output:
    # 3
    # 3
    # 3
    # 3
     
    4 print;
    # Syntax error
     
    n -1 =;
    4 n print;
    # Runtime error

    Lastly, division by zero should throw an error:

    5 0 /;
    # Runtime error

    Getting Setup

    We will be writing our compiler on Node.js, using Jison to generate the parser for our language from a grammar, and using the source-map library to help generate source maps.

    The first step is to download and install Node.js if you don’t already have it on your system.

    After you have installed Node.js, use its package manager npm to create a new project for the compiler:

    $ mkdir rpn
    $ cd rpn/
    $ npm init .

    After the last command, npm will prompt you with a bunch of questions. Enter your name and email, answer ./lib/rpn.js for the main module/entry point, and just let npm use the defaults that it supplies for the rest of the questions.

    Once you have finished answering the prompts, create the directory layout for the project:

    $ mkdir lib
    $ touch lib/rpn.js
    $ mkdir -p lib/rpn

    The public API for the compiler will reside within lib/rpn.js, while the submodules we use to implement various things such as the lexer and abstract syntax tree will live in lib/rpn/*.js.

    Next, open up the package.json file and add jison and source-map to the project’s dependencies:

    ...
    "dependencies": {
      "jison": ">=0.4.4",
      "source-map": ">=0.1.22"
    },
    ...

    Now we will install a link to our package in Node.js’s globally installed packages directory. This allows us to import our package from the Node.js shell:

    $ npm link .

    Make sure that everything works by opening the Node.js shell and importing our package:

    $ node
    > require("rpn")
    {}

    Writing the Lexer

    A lexer (also known as a scanner or tokenizer) breaks the inputted raw source code into a stream of semantic tokens. For example in our case, we would want to break the raw input string "5 3 +;" into something like ["5", "3", "+", ";"].

    Because we are using Jison, rather than writing the lexer and parser by hand, our job is much easier. All that is required is to supply a list of rules that describe the types of tokens we are expecting. The left hand side of the rules are regular expressions to match individual tokens, the right hand side are the snippets of code to execute when an instance of the corresponding token type is found. These tokens will be passed on to the parser in the next phase of the compiler.

    Create the rules for lexical analysis in lib/rpn/lex.js:

    exports.lex = {
      rules: [
        ["\s+",                   "/* Skip whitespace! */"],
        ["#.*\n",                 "/* Skip comments! */"],
        [";",                      "return 'SEMICOLON'"],
        ["\-?[0-9]+(\.[0-9]+)?", "return 'NUMBER';"],
        ["print",                  "return 'PRINT';"],
        ["[a-zA-Z][a-zA-Z0-9_]*",  "return 'VARIABLE';"],
        ["=",                      "return '=';"],
        ["\+",                    "return '+';"],
        ["\-",                    "return '-';"],
        ["\*",                    "return '*';"],
        ["\/",                    "return '/';"],
        ["$",                      "return 'EOF';"]
      ]
    };

    Writing the Parser

    The parser takes the tokens from the lexer one at a time and confirms that the input is a valid program in our source language.

    Once again, the task of writing the parser is much easier than it would otherwise be thanks to Jison. Rather than writing the parser ourselves, Jison will programmatically create one for us if we provide a grammar for the language.

    If all we cared about was whether the input was a valid program, we would stop here. However, we are also going to compile the input to JavaScript, and to do that we need to create an abstract syntax tree. We build the AST in the code snippets next to each rule.

    A typical grammar contains productions with the form:

    LeftHandSide → RightHandSide1
                 | RightHandSide2
                 ...

    However, in Jison we are a) writing in JavaScript, and b) also providing code to execute for each rule so that we can create the AST. Therefore, we use the following format:

    LeftHandSide: [
      [RightHandSide1, CodeToExecute1],
      [RightHandSide2, CodeToExecute2],
      ...
    ]

    Inside the code snippets, there are a handful of magic variables we have access to:

    • $$: The value of the left hand side of the production.
    • $1/$2/$3/etc: The value of the the nth form in the right hand side of the production.
    • @1/@2/@3/etc: An object containing the line and column coordinates where the nth form in the right hand side of the production was parsed.
    • yytext: The full text of currently matched rule.

    Using this information, we can create the grammar in lib/rpn/bnf.js:

    exports.bnf = {
      start: [
        ["input EOF", "return $$;"]
      ],
      input: [
        ["",           "$$ = [];"],
        ["line input", "$$ = [$1].concat($2);"]
      ],
      line: [
        ["exp SEMICOLON", "$$ = $1;"]
      ],
      exp: [
        ["NUMBER",           "$$ = new yy.Number(@1.first_line, @1.first_column, yytext);"],
        ["VARIABLE",         "$$ = new yy.Variable(@1.first_line, @1.first_column, yytext);"],
        ["exp exp operator", "$$ = new yy.Expression(@3.first_line, @3.first_column, $1, $2, $3);"]
      ],
      operator: [
        ["PRINT", "$$ = new yy.Operator(@1.first_line, @1.first_column, yytext);"],
        ["=",     "$$ = new yy.Operator(@1.first_line, @1.first_column, yytext);"],
        ["+",     "$$ = new yy.Operator(@1.first_line, @1.first_column, yytext);"],
        ["-",     "$$ = new yy.Operator(@1.first_line, @1.first_column, yytext);"],
        ["*",     "$$ = new yy.Operator(@1.first_line, @1.first_column, yytext);"],
        ["/",     "$$ = new yy.Operator(@1.first_line, @1.first_column, yytext);"]
      ]
    };

    Implementing the Abstract Syntax Tree

    Create the definitions for the abstract syntax tree nodes in lib/rpn/ast.js.

    Since we will be maintaining line and column information in all of the AST nodes, we can reuse some code by making a base prototype:

    var AstNode = function (line, column) {
      this._line = line;
      this._column = column;
    };

    The definitions for the rest of the AST nodes are pretty straight forward. Link up the prototype chain, assign relevant attributes, and don’t forget to call AstNode‘s constructor:

    exports.Number = function (line, column, numberText) {
      AstNode.call(this, line, column);
      this._value = Number(numberText);
    };
    exports.Number.prototype = Object.create(AstNode.prototype);
     
    exports.Variable = function (line, column, variableText) {
      AstNode.call(this, line, column);
      this._name = variableText;
    };
    exports.Variable.prototype = Object.create(AstNode.prototype);
     
    exports.Expression = function (line, column, operand1, operand2, operator) {
      AstNode.call(this, line, column);
      this._left = operand1;
      this._right = operand2;
      this._operator = operator;
    };
    exports.Expression.prototype = Object.create(AstNode.prototype);
     
    exports.Operator = function (line, column, operatorText) {
      AstNode.call(this, line, column);
      this.symbol = operatorText;
    };
    exports.Operator.prototype = Object.create(AstNode.prototype);

    Compilation

    Generated JavaScript

    Before we generate JavaScript, we need a plan. There are a couple ways we can structure the outputted JavaScript.

    One strategy is to translate the RPN expressions to the equivalent human readable JavaScript expression we would create if we had been writing JavaScript all along. For example, if we were to port this RPN example:

    a 8 =;
    b 2 =;
    c a b 1 - / =;

    We might write the following JavaScript:

    var a = 8;
    var b = 3;
    var c = a / (b - 1);

    However, this means that we are completely adopting the nuances of JavaScript’s arithmetic. In an earlier example, we saw that a helpful runtime error was thrown when any number was divided by zero. Most languages throw an error when this occurs, however JavaScript does not; instead, the result is Infinity. Therefore, we can’t completely embrace JavaScript’s arithmetic system, and we must generate some code to check for divide-by-zero errors ourselves. Adding this code gets a little tricky if we want to maintain the strategy of generating human readable code.

    Another option is treating the JavaScript interpreter as a stack machine of sorts and generating code that pushes and pops values to and from a stack. Furthermore, stack machines are a natural fit for evaluating RPN. In fact, it is such a good fit that RPN “was independently reinvented by F. L. Bauer and E. W. Dijkstra in the early 1960s to reduce computer memory access and utilize the stack to evaluate expressions.”

    Generating JavaScript code for the same example above, but utilizing the JavaScript interpreter as a stack machine, might look something like this:

    push(8);
    push('a');
    env[pop()] = pop();
    push(2);
    push('b');
    env[pop()] = pop();
    push('a');
    push('b');
    push(1);
    temp = pop();
    push(pop() - temp);
    temp = pop();
    if (temp === 0) throw new Error("Divide by zero");
    push(pop() / temp);
    push('c');
    env[pop()] = pop();

    This is the strategy we will follow. The generated code is a bit larger, and we will require a preamble to define push, pop, etc, but compilation becomes much easier. Furthermore, the fact that the generated code isn’t as human readable only highlights the benefits of using source maps!

    Creating Source Maps

    If we weren’t generating source maps along with our generated JavaScript, we could build the generated code via concatenating strings of code:

    code += "push(" + operand1.compile() + " "
      + operator.compile() + " "
      + operand2.compile() + ");n";

    However, this doesn’t work when we are creating source maps because we need to maintain line and column information. When we concatenate strings of code, we lose that information.

    The source-map library contains SourceNode for exactly this reason. If we add a new method on our base AstNode prototype, we can rewrite our example like this:

    var SourceNode = require("source-map").SourceNode;
    AstNode.prototype._sn = function (originalFilename, chunk) {
      return new SourceNode(this._line, this._column, originalFilename, chunk);
    };
     
    ...
     
    code = this._sn("foo.rpn", [code,
                                "push(",
                                operand1.compile(), " ",
                                operator.compile(), " ",
                                operand2.compile(), ");n"]);

    Once we have completed building the SourceNode structure for the whole input program, we can generate the compiled source and the source map by calling the SourceNode.prototype.toStringWithSourceMap method. This method returns an object with two properties: code, which is a string containing the generated JavaScript source code; and map, which is the source map.

    Implementing Compilation

    Now that we have a strategy for generating code, and understand how to maintain line and column information so that we can generate source maps easily, we can add the methods to compile our AST nodes to lib/rpn/ast.js.

    To play nice with the global JavaScript environment, we will namespace push, pop, etc, under __rpn.

    function push(val) {
      return ["__rpn.push(", val, ");n"];
    }
     
    AstNode.prototype.compile = function (data) {
      throw new Error("Not Yet Implemented");
    };
    AstNode.prototype.compileReference = function (data) {
      return this.compile(data);
    };
    AstNode.prototype._sn = function (originalFilename, chunk) {
      return new SourceNode(this._line, this._column, originalFilename, chunk);
    };
     
    exports.Number.prototype.compile = function (data) {
      return this._sn(data.originalFilename,
                      push(this._value.toString()));
    };
     
    exports.Variable.prototype.compileReference = function (data) {
      return this._sn(data.originalFilename,
                      push(["'", this._name, "'"]));
    };
    exports.Variable.prototype.compile = function (data) {
      return this._sn(data.originalFilename,
                      push(["window.", this._name]));
    };
     
    exports.Expression.prototype.compile = function (data) {
      var temp = "__rpn.temp";
      var output = this._sn(data.originalFilename, "");
     
      switch (this._operator.symbol) {
      case 'print':
        return output
          .add(this._left.compile(data))
          .add(this._right.compile(data))
          .add([temp, " = __rpn.pop();n"])
          .add(["if (", temp, " <= 0) throw new Error('argument must be greater than 0');n"])
          .add(["if (Math.floor(", temp, ") != ", temp,
                ") throw new Error('argument must be an integer');n"])
          .add([this._operator.compile(data), "(__rpn.pop(), ", temp, ");n"]);
      case '=':
        return output
          .add(this._right.compile(data))
          .add(this._left.compileReference(data))
          .add(["window[__rpn.pop()] ", this._operator.compile(data), " __rpn.pop();n"]);
      case '/':
        return output
          .add(this._left.compile(data))
          .add(this._right.compile(data))
          .add([temp, " = __rpn.pop();n"])
          .add(["if (", temp, " === 0) throw new Error('divide by zero error');n"])
          .add(push(["__rpn.pop() ", this._operator.compile(data), " ", temp]));
      default:
        return output
          .add(this._left.compile(data))
          .add(this._right.compile(data))
          .add([temp, " = __rpn.pop();n"])
          .add(push(["__rpn.pop() ", this._operator.compile(data), " ", temp]));
      }
    };
     
    exports.Operator.prototype.compile = function (data) {
      if (this.symbol === "print") {
        return this._sn(data.originalFilename,
                        "__rpn.print");
      }
      else {
        return this._sn(data.originalFilename,
                        this.symbol);
      }
    };

    Gluing it Together

    From here we have done all the difficult work, and we can run a victory lap by connecting the modules together with a public API, and by creating a command line script to call the compiler.

    The public API resides in lib/rpn.js. It also contains the preamble, to initialize __rpn:

    var jison = require("jison");
    var sourceMap = require("source-map");
    var lex = require("./rpn/lex").lex;
    var bnf = require("./rpn/bnf").bnf;
     
    var parser = new jison.Parser({
      lex: lex,
      bnf: bnf
    });
     
    parser.yy = require("./rpn/ast");
     
    function getPreamble () {
      return new sourceMap.SourceNode(null, null, null, "")
        .add("var __rpn = {};n")
        .add("__rpn._stack = [];n")
        .add("__rpn.temp = 0;n")
     
        .add("__rpn.push = function (val) {n")
        .add("  __rpn._stack.push(val);n")
        .add("};n")
     
        .add("__rpn.pop = function () {n")
        .add("  if (__rpn._stack.length > 0) {n")
        .add("    return __rpn._stack.pop();n")
        .add("  }n")
        .add("  else {n")
        .add("    throw new Error('can\'t pop from empty stack');n")
        .add("  }n")
        .add("};n")
     
        .add("__rpn.print = function (val, repeat) {n")
        .add("  while (repeat-- > 0) {n")
        .add("    var el = document.createElement('div');n")
        .add("    var txt = document.createTextNode(val);n")
        .add("    el.appendChild(txt);n")
        .add("    document.body.appendChild(el);n")
        .add("  }n")
        .add("};n");
    }
     
    exports.compile = function (input, data) {
      var expressions = parser.parse(input.toString());
      var preamble = getPreamble();
     
      var result = new sourceMap.SourceNode(null, null, null, preamble);
      result.add(expressions.map(function (exp) {
        return exp.compile(data);
      }));
     
      return result;
    };

    Create the command line script in bin/rpn.js:

    #!/usr/bin/env node
    var fs = require("fs");
    var rpn = require("rpn");
     
    process.argv.slice(2).forEach(function (file) {
      var input = fs.readFileSync(file);
      var output = rpn.compile(input, {
        originalFilename: file
      }).toStringWithSourceMap({
        file: file.replace(/.[w]+$/, ".js.map")
      });
      var sourceMapFile = file.replace(/.[w]+$/, ".js.map");
      fs.writeFileSync(file.replace(/.[w]+$/, ".js"),
                       output.code + "n//# sourceMappingURL=" + sourceMapFile);
      fs.writeFileSync(sourceMapFile, output.map);
    });

    Note that our script will automatically add the //# sourceMappingURL comment directive so that the browser’s debugger knows where to find the source map.

    After you create the script, update your package.json:

    ...
    "bin": {
      "rpn.js": "./bin/rpn.js"
    },
    ...

    And link the package again so that the script is installed on your system:

    $ npm link .

    Seeing Results

    Here is an RPN program that we can use to test our compiler. I have saved it in examples/simple-example.rpn:

    a 8 =;
    b 3 =;
    c a b 1 - / =;
    c 1 print;

    Next, compile the script:

    $ cd examples/
    $ rpn.js simple-example.rpn

    This generates simple-example.js and simple-example.js.map. When we include the JavaScript file in a web page we should see the result of the computation printed on the page:

    Screenshot of simple-example.rpn's result

    Great success!

    However, we aren’t always so lucky, and our arithmetic might have some errors. Consider the following example, examples/with-error.rpn:

    a 9 =;
    b 3 =;
    c a b / =;
    c a b c - / =;
    c 1 print;

    We can compile this script and include the resulting JavaScript in a web page, but this time we won’t see any output on the page.

    By opening the debugger, setting the pause on exceptions option, and reloading, we can see how daunting debugging without source maps can be:

    Screenshot of enabling pause on exceptions.

    Screenshot of debugging with-error.rpn without source maps.

    The generated JavaScript is difficult to read, and unfamiliar to anyone who authored the original RPN script. By enabling source maps in the debugger, we can refresh and the exact line where the error ocurred in our original source will be highlighted:

    Screenshot of enabling source maps.


    Screenshot of debugging with-error.rpn with source maps.

    The debugging experience with source maps is orders of magnitude improved, and makes compiling languages to JavaScript a serious possibility.

    At the end of the day though, the debugging experience is only as good as the information encoded in the source maps by your compiler. It can be hard to judge the quality of your source maps simply by looking at the set of source location coordinates that they are mapping between, so Tobias Koppers created a tool to let you easily visualize your source maps.

    Here is the visualization of one of our source maps:


    Screenshot of the source map visualization tool.

    Good luck writing your own compiler that targets JavaScript!

    References