Mozilla

Faster Canvas Pixel Manipulation with Typed Arrays

Edit: See the section about Endiannes.

Typed Arrays can significantly increase the pixel manipulation performance of your HTML5 2D canvas Web apps. This is of particular importance to developers looking to use HTML5 for making browser-based games.

This is a guest post by Andrew J. Baker. Andrew is a professional software engineer currently working for Ibuildings UK where his time is divided equally between front- and back-end enterprise Web development. He is a principal member of the browser-based games channel #bbg on Freenode, spoke at the first HTML5 games conference in September 2011, and is a scout for Mozilla’s WebFWD innovation accelerator.


Eschewing the higher-level methods available for drawing images and primitives to a canvas, we’re going to get down and dirty, manipulating pixels using ImageData.

Conventional 8-bit Pixel Manipulation

The following example demonstrates pixel manipulation using image data to generate a greyscale moire pattern on the canvas.

JSFiddle demo.

Let’s break it down.

First, we obtain a reference to the canvas element that has an id attribute of canvas from the DOM.

var canvas = document.getElementById('canvas');

The next two lines might appear to be a micro-optimisation and in truth they are. But given the number of times the canvas width and height is accessed within the main loop, copying the values of canvas.width and canvas.height to the variables canvasWidth and canvasHeight respectively, can have a noticeable effect on performance.

var canvasWidth  = canvas.width;
var canvasHeight = canvas.height;

We now need to get a reference to the 2D context of the canvas.

var ctx = canvas.getContext('2d');

Armed with a reference to the 2D context of the canvas, we can now obtain a reference to the canvas’ image data. Note that here we get the image data for the entire canvas, though this isn’t always necessary.

var imageData = ctx.getImageData(0, 0, canvasWidth, canvasHeight);

Again, another seemingly innocuous micro-optimisation to get a reference to the raw pixel data that can also have a noticeable effect on performance.

var data = imageData.data;

Now comes the main body of code. There are two loops, one nested inside the other. The outer loop iterates over the y axis and the inner loop iterates over the x axis.

for (var y = 0; y < canvasHeight; ++y) {
    for (var x = 0; x < canvasWidth; ++x) {

We draw pixels to image data in a top-to-bottom, left-to-right sequence. Remember, the y axis is inverted, so the origin (0,0) refers to the top, left-hand corner of the canvas.

The ImageData.data property referenced by the variable data is a one-dimensional array of integers, where each element is in the range 0..255. ImageData.data is arranged in a repeating sequence so that each element refers to an individual channel. That repeating sequence is as follows:

data[0]  = red channel of first pixel on first row
data[1]  = green channel of first pixel on first row
data[2]  = blue channel of first pixel on first row
data[3]  = alpha channel of first pixel on first row
 
data[4]  = red channel of second pixel on first row
data[5]  = green channel of second pixel on first row
data[6]  = blue channel of second pixel on first row
data[7]  = alpha channel of second pixel on first row
 
data[8]  = red channel of third pixel on first row
data[9]  = green channel of third pixel on first row
data[10] = blue channel of third pixel on first row
data[11] = alpha channel of third pixel on first row
 
 
...

Before we can plot a pixel, we must translate the x and y coordinates into an index representing the offset of the first channel within the one-dimensional array.

        var index = (y * canvasWidth + x) * 4;

We multiply the y coordinate by the width of the canvas, add the x coordinate, then multiply by four. We must multiply by four because there are four elements per pixel, one for each channel.

Now we calculate the colour of the pixel.

To generate the moire pattern, we multiply the x coordinate by the y coordinate then bitwise AND the result with hexadecimal 0xff (decimal 255) to ensure that the value is in the range 0..255.

        var value = x * y & 0xff;

Greyscale colours have red, green and blue channels with identical values. So we assign the same value to each of the red, green and blue channels. The sequence of the one-dimensional array requires us to assign a value for the red channel at index, the green channel at index + 1, and the blue channel at index + 2.

        data[index]   = value;	// red
        data[++index] = value;	// green
        data[++index] = value;	// blue

Here we’re incrementing index, as we recalculate it with each iteration, at the start of the inner loop.

The last channel we need to take into account is the alpha channel at index + 3. To ensure that the plotted pixel is 100% opaque, we set the alpha channel to a value of 255 and terminate both loops.

        data[++index] = 255;	// alpha
    }
}

For the altered image data to appear in the canvas, we must put the image data at the origin (0,0).

ctx.putImageData(imageData, 0, 0);

Note that because data is a reference to imageData.data, we don’t need to explicitly reassign it.

The ImageData Object

At time of writing this article, the HTML5 specification is still in a state of flux.

Earlier revisions of the HTML5 specification declared the ImageData object like this:

interface ImageData {
    readonly attribute unsigned long width;
    readonly attribute unsigned long height;
    readonly attribute CanvasPixelArray data;
}

With the introduction of typed arrays, the type of the data attribute has altered from CanvasPixelArray to Uint8ClampedArray and now looks like this:

interface ImageData {
    readonly attribute unsigned long width;
    readonly attribute unsigned long height;
    readonly attribute Uint8ClampedArray data;
}

At first glance, this doesn’t appear to offer us any great improvement, aside from using a type that is also used elsewhere within the HTML5 specification.

But, we’re now going to show you how you can leverage the increased flexibility introduced by deprecating CanvasPixelArray in favour of Uint8ClampedArray.

Previously, we were forced to write colour values to the image data one-dimensional array a single channel at a time.

Taking advantage of typed arrays and the ArrayBuffer and ArrayBufferView objects, we can write colour values to the image data array an entire pixel at a time!

Faster 32-bit Pixel Manipulation

Here’s an example that replicates the functionality of the previous example, but uses unsigned 32-bit writes instead.

NOTE: If your browser doesn’t use Uint8ClampedArray as the type of the data property of the ImageData object, this example won’t work!

JSFiddle demo.

The first deviation from the original example begins with the instantiation of an ArrayBuffer called buf.

var buf = new ArrayBuffer(imageData.data.length);

This ArrayBuffer will be used to temporarily hold the contents of the image data.

Next we create two ArrayBuffer views. One that allows us to view buf as a one-dimensional array of unsigned 8-bit values and another that allows us to view buf as a one-dimensional array of unsigned 32-bit values.

var buf8 = new Uint8ClampedArray(buf);
var data = new Uint32Array(buf);

Don’t be misled by the term ‘view’. Both buf8 and data can be read from and written to. More information about ArrayBufferView is available on MDN.

The next alteration is to the body of the inner loop. We no longer need to calculate the index in a local variable so we jump straight into calculating the value used to populate the red, green, and blue channels as we did before.

Once calculated, we can proceed to plot the pixel using only one assignment. The values of the red, green, and blue channels, along with the alpha channel are packed into a single integer using bitwise left-shifts and bitwise ORs.

        data[y * canvasWidth + x] =
            (255   << 24) |	// alpha
            (value << 16) |	// blue
            (value <<  8) |	// green
             value;		// red
    }
}

Because we’re dealing with unsigned 32-bit values now, there’s no need to multiply the offset by four.

Having terminated both loops, we must now assign the contents of the ArrayBuffer buf to imageData.data. We use the Uint8ClampedArray.set() method to set the data property to the Uint8ClampedArray view of our ArrayBuffer by specifying buf8 as the parameter.

imageData.data.set(buf8);

Finally, we use putImageData() to copy the image data back to the canvas.

Testing Performance

We’ve told you that using typed arrays for pixel manipulation is faster. We really should test it though, and that’s what this jsperf test does.

At time of writing, 32-bit pixel manipulation is indeed faster.

Wrapping Up

There won’t always be occasions where you need to resort to manipulating canvas at the pixel level, but when you do, be sure to check out typed arrays for a potential performance increase.

EDIT: Endianness

As has quite rightly been highlighted in the comments, the code originally presented does not correctly account for the endianness of the processor on which the JavaScript is being executed.

The code below, however, rectifies this oversight by testing the endianness of the target processor and then executing a different version of the main loop dependent on whether the processor is big- or little-endian.

JSFiddle demo.

A corresponding jsperf test for this amended code has also been written and shows near-identical results to the original jsperf test. Therefore, our final conclusion remains the same.

Many thanks to all commenters and testers.

21 comments

Comments are now closed.

  1. Boris wrote on December 1st, 2011 at 14:38:

    This trick will only work on little-endian hardware. On big-endian hardware, the high byte of a 32-bit integer will be the red channel, not alpha. So you’re getting a performance improvement for some users at the cost of completely incorrect behavior for other users.

    Now you can detect endianness and work around it by running slightly different code but that either increases code complexity or reduces performance or both.

    1. Andrew J. Baker wrote on December 1st, 2011 at 15:00:

      Very true re: endianness. There is an additional parameter that can be used to specify whether little-endianness is desired for setUint32().

      http://www.khronos.org/registry/typedarray/specs/latest/#8

      1. Boris wrote on December 1st, 2011 at 19:12:

        Yes, there is. How does it perform?

    2. Joe wrote on December 1st, 2011 at 15:26:

      You sure? The spec makes it clear that the image data is in RGBA order.

      1. Boris wrote on December 1st, 2011 at 15:31:

        That’s precisely the problem. The image data is always RGBA. But the order of bytes generated by byte shifts depends on endianness, since it’s not shifting on physical bytes but on numeric values. So the expression 255 << 24 is always the 32-bit integer 0xff000000 which on a little-endian system is represented by the byte sequence [0, 0, 0, 0xff] in memory and on a big-endian system is represented by the byte sequence [0xff, 0, 0, 0]. So on a little-endian system that will give you an opaque black pixel while on a big-endian system it's a transparent red pixel.

        1. Ryan Badour wrote on December 1st, 2011 at 16:53:

          Javascript is an interpreted language, this isn’t low level C or C++ who’s to say that endianness is abstracted away? Are you sure it isn’t?

          1. Boris wrote on December 2nd, 2011 at 05:50:

            Yep. The typed array spec is very clear about endianness not being abstracted away in typed arrays. I suggest just reading the spec instead of guessing and hoping.

        2. barryvan wrote on December 1st, 2011 at 17:29:

          So does this then effectively rule out the shift+or optimisation when manipulating pixel data? I read the article with high hopes, because I’m struggling with pixel-manipulation performance on a demo I wrote [1], which meant that I had to reduce the canvas size down from fullscreen to just 400×400.

          A potential solution, as you say, is to check the endianness, and use different logic for each. I’d be curious to see if the performance penalty of calling function A or function B to assign data to a pixel would be higher than just managing each channel separately.

          I’m actually starting to wonder whether the endianness should even be exposed to a JS developer. An Intel paper [2] suggests that the JVM is big-endian; perhaps JS engines should also present a common endianness irrespective of the underlying hardware. Is that feasible?

          [1] http://barryvan.github.com/trackPerformer/joy.html
          [2] http://www.intel.com/design/intarch/papers/endian.pdf

          1. Boris wrote on December 2nd, 2011 at 05:55:

            You can also use typed array setters that will do explicit endianness conversions. They may well be slower than using assignment to native-endian array entries, of course.

            And yes, exposing the endianness to JS is not obviously a good idea….

        3. Gordon wrote on December 1st, 2011 at 17:59:

          I’m not so sure about this. Many interpreted languages simulate little endian bits in their operators so that shifts left and right are instead shift bigger/smaller. JavaScript is probably the same.

        4. Joe wrote on December 2nd, 2011 at 12:55:

          After googling around, iPhone/iPad both seem to be little endian like x86, and many Android devices too.

          1. Boris wrote on December 2nd, 2011 at 12:57:

            ARM processors can be either little or big endian; in many cases you can actually pick at runtime (certainly at boot time) with OS support.

            In practice, many shipping ARM devices only support little-endian operation.

  2. David Wilhelm wrote on December 1st, 2011 at 19:03:

    Is Firefox the only browser which supports Uint8ClampedArray ?

    1. Andrew J. Baker wrote on December 2nd, 2011 at 14:15:

      Firefox is the only browser I know of at present that has switched out CanvasPixelArray for Uint8ClampedArray as the type of the data property for the ImageData object, in accordance w/ the latest HTML5 spec.

      http://www.whatwg.org/specs/web-apps/current-work/#imagedata

  3. Jon wrote on December 1st, 2011 at 19:08:

    Supported by Firefox but not Chrome (nor Android)

  4. 4esn0k wrote on December 1st, 2011 at 20:17:

    really strange test – http://jsperf.com/canvas-pixel-manipulation – only Firefox, only TypedArrays

  5. Joe wrote on December 4th, 2011 at 00:02:

    Here is a slight alternative: http://jsperf.com/canvas-pixel-manipulation/8

    I added a third test which uses an Int32Array, and works directly on the buffer used by the canvas image data. It no longer needs to manually produce a ‘Uint8ClampedArray’, but is still Firefox only, because the image data that the canvas context returns still needs to be one for any of this to work.

    I built it after finding that in some circumstances, Uint32Array was significantly slower then Int32Array, when setting a value. Only Firefox 11 shows a difference in that test, but in my own code I have found a difference in FF 8 too (but am unable to replicate this on JSPerf).

    By working directly on the image data buffer directly it allows you to avoid creating an intermediate buffer, which is useful when working with very large images, which could take up several megabytes in size. It also means no ‘set’ is required at the end (which I see as a disadvantage).

  6. dhaber wrote on April 16th, 2012 at 09:38:

    I may be wrong about this, but shouldn’t the endian tests read bytes out of buf8 instead of buf? I tried this on chromium and was unable to read the bytes out of buf, but buf8 worked as expected.

  7. Paul Neave wrote on April 25th, 2012 at 02:05:

    Uint8ClampedArray just landed in WebKit, should be available in Google Chrome from 20.0.1116 (Canary): https://bugs.webkit.org/show_bug.cgi?id=73011

    1. Andrew J. Baker wrote on April 25th, 2012 at 08:39:

      Wahoo! I’ve been tracking this too. Great stuff. I can’t wait to see if there’s a considerable performance increase, like there was with Firefox.

    2. Andrew J. Baker wrote on May 9th, 2012 at 09:13:

      Beginning to see performance details coming in for Chrome.

      http://jsperf.com/canvas-pixel-manipulation/6

      Similar performance increases to Firefox. Yay!

Comments are closed for this article.