Mozilla Hacks - the Web developer blog » Canvas

Creating thumbnails with drag and drop and HTML5 canvas

Janet Swisher — Wed, 01 Feb 2012 14:00:28 +0000

HTML5 Canvas is a very cool feature. Seemingly just an opportunity to paint inside the browser with a very low-level API you can use it to heavily convert and change image and video content in the document. Today, let’s take a quick look at how you can use Canvas and the FileReader API to create thumbnails from images dropped into a browser document.

The final code is available on GitHub and you can see an online demo here. There is also a screencast available on YouTube:

Step 1: Getting the files into the browser

The first step to resize images in the browser is to somehow get them. For this, we can just add an element in the page and assign drag and drop event handlers to it:

s.addEventListener( 'dragover', function ( evt ) {
  evt.preventDefault();
}, false );
s.addEventListener( 'drop', getfiles, false );

Notice that we only prevent the default behaviour when we drag things over the element. This is to prevent the browser from just showing the images when we drag them in.

The getfiles() function then does the hard work of reading all the files in and sending them on to the functions that do the resizing and image generation:

function getfiles( ev ) {
  var files = ev.dataTransfer.files;
  if ( files.length > 0 ) {
    var i = files.length;
    while ( i-- ) {
      var file = files[ i ];
      if ( file.type.indexOf( 'image' ) === -1 ) { continue; }
      var reader = new FileReader();
      reader.readAsDataURL( file );
      reader.onload = function ( ev ) {
        var img = new Image();
        img.src = ev.target.result;
        img.onload = function() {
        imagetocanvas( this, thumbwidth, thumbheight, crop, background );
        };
      };
    }
  }
  ev.preventDefault();
};

The drop event gives us a property called dataTransfer which contains a list of all the files that have been dropped. We make sure that there was at least one file in the drop and then iterate over them.

If the file type was not an image (or in other words the type property of the file does not contain the string “image”) we don’t do anything with the file and continue the loop.

If the file is an image we instantiate a new FileReader and tell it to read the file as a Data URL. When the reader successfully loaded the file it fires its onload handler.

In this handler we create a new image and set its src attribute to the result of the file transfer. We then send this image to the imagetocanvas() function with the parameters to resize (in the demo these come from the form):

function imagetocanvas( img, thumbwidth, thumbheight, crop, background ) {
  c.width = thumbwidth;
  c.height = thumbheight;
  var dimensions = resize( img.width, img.height, thumbwidth, thumbheight );
  if ( crop ) {
    c.width = dimensions.w;
    c.height = dimensions.h;
    dimensions.x = 0;
    dimensions.y = 0;
  }
  if ( background !== 'transparent' ) {
    cx.fillStyle = background;
    cx.fillRect ( 0, 0, thumbwidth, thumbheight );
  }
  cx.drawImage( 
    img, dimensions.x, dimensions.y, dimensions.w, dimensions.h 
  );
  addtothumbslist( jpeg, quality );
};

This function gets the desired thumbnail size and resizes the canvas to these dimensions. This has the added benefit of wiping the canvas so that no old image data would be added to our thumbnail. We then resize the image to fit into the thumbnail using a resize() function. You can see for yourself what this one does in the source code, it just means the image gets resized to fit. The function returns an object with the width and the height of the new image and the x and y position where it should be positioned onto the canvas.

If we don’t want the full-size thumbnail but instead crop it we resize the canvas accordingly and reset x and y to 0.

If the user requested a background we fill the canvas with the colour. After that we put the image on the canvas with the x and y coordinates and the new width and height.

This takes care of creating a new thumbnail on the canvas, but we haven’t got it as an image in the document yet. To this end, we call addtothumbslist():

function addtothumbslist( jpeg, quality ) {
  var thumb = new Image(),
      url = jpeg ? c.toDataURL( 'image/jpeg' , quality ) : c.toDataURL();
  thumb.src = url;
  thumb.title = Math.round( url.length / 1000 * 100 ) / 100 + ' KB';
  o.appendChild( thumb );
};

This one creates a new image and checks if the users wanted a JPG or PNG image to be created. PNG images tend to be better quality but also bigger in file size. If a JPG was requested we call the canvas’ toDataURL() method with two parameters: the requested JPEG mime type and the quality of the image (ranging between 0 and 1 with 1 being best quality). If a PNG is wanted, we can just call toDataURL() without any parameters as this is the default.

We set the src of the image to the generated url string and add a title showing the size of the image in KB (rounded to two decimals). All that is left then is to add the thumb to the output element on the page.

That’s it, you can now drag and drop images into the browser to generate thumbnails. Right now, we can only save them one at a time (or if you have some download add-ons all at once). Would be fun to add Zip.js to the mix to offer them as a zip. I dare you! :)

Interview: Nikhil Suresh on Building His Winning Canvas Demo

Janet Swisher — Wed, 01 Feb 2012 05:36:48 +0000

Editor’s Note: Back in November, Nikhil Suresh (@nklsrh2) from Sydney, Australia, won the MDN Developer Derby with his distinctive, non-violent 2-person shooter game Bouncy and the Apple. We thought it’d be fun to learn a little more about Nikhil and what inspires him.
Congratulations Nikhil, and thanks for your thoughtful words about Mozilla. We’re honored to showcase your demo.

Tell us about developing Bouncy and the Apple and where your idea came from?

I really wanted my demo to stand out from the crowd, so it had to possess a singular feature that would impress. Since my passion is game development, I decided to implement game controller support in whatever game I would build for the Derby. I built Bouncy to be a twin-stick shooter which would stand out as a demo for the Mozilla Gamepad API as well as Canvas.

Usually, I tend to stay away from violence in my games, so it was a bit difficult at first (a ‘shooter’ without any violence!)

Incidentally, it was the Mozilla mascots that inspired me to draw Bouncy (the spikes in the hair were directly from the brown dude with the hat!):

How did you get started coding and hacking?

Funnily enough, it started in Microsoft PowerPoint, where I realized I could ‘hyperlink’ from a slide to any other one. I then created a quiz game, not unlike the TV show ‘Who wants to be a Millionaire?.’ From that, I moved onto Visual Basic (in which I became Microsoft-Certified in 2008) C# and XNA, and eventually began learning JavaScript (after playing around with the Processing language).

What was your first computer?

I don’t really remember much about my first computer, except the fact it had a Pentium II, which was not powerful enough to run some of the new games I wanted to play.

What games do you play the most these days?

On the console, I prefer racing games like F1 2011 and Gran Turismo. On the PC, I like to play indie games like Super Crate Box and Jamestown. I also love Valve games like Portal.

Are you working on another demo or any other cool projects you want to share?

I’m currently participating in the Pokki 1UP game development competition, with a friend. We are building a 2D/3D arcade game called BLASTR:ARENA. It uses WebGL for 3D graphics and Canvas for the 2D fallback.

You mentioned that you recently graduated from high school? What’s next for you?

I have just enrolled in the Bachelor of Engineering (Software) course at the University of New South Wales. During my degree, I will also be developing games part-time, hopefully for a chance at breaking into the gamedev industry.

When you think about HTML5 and new Web technologies what are you most excited about?

As my game clearly demonstrates, HTML5 and other web technologies provide the browser the ability to render other traditional applications obsolete, replacing them with a unified system which can serve as the basis for anything from games to word processors.

As I’m most interested in game development, the introduction of the Fullscreen and Gamepad APIs, as well as performance increases in WebGL and Canvas give the Web a greater chance of overthrowing consoles as an effective game platform.

Anybody else who helped you with Bouncy and the Apple whom you’d like to give a shout out to?

My aunt, who had come over for a holiday, was a fantastic source of inspiration and ideas, so it was a great feeling to finally play the finished game with her on the big screen. I’d also like to thank my parents for letting me stick to my plans, even if I skipped dinner and stayed up until 1AM on some nights. I’m also grateful to my brother for putting up with my endless repeating game music.

You mentioned that you’re a big fan of Mozilla’s work on the Open Web. Can you say something more about why you think that’s important?

I think Mozilla is a fantastic, one-of-a-kind organisation since it works and innovates solely with the well-being of the Web and its users in mind. With such an open and direct goal, it means everyone, from the users to the developers, benefit from the development of Mozilla products and technologies.
Also, Mozilla’s work on HTML, CSS, Javascript and other web technology is for the mutual benefit of all, fostering the advancement of the open Web as a mainstream platform of the present and future generations.

Congratulations November Dev Derby Winners

Havi Hoffman — Sat, 14 Jan 2012 01:09:50 +0000

Canvas is a new HTML5 element which creates a digital “drawing board.” A web developer can use one of these drawing boards along with some JavaScript to create simple shapes, graphs, animations, interactive games, and more. Recently, eighteen creative minds showed us just how powerful and important Canvas is by sharing their work in the November Dev Derby.

Once again, this was a fairly close race, so join us in congratulating the winners:

1st Place: Bouncy and the Apple – Canvas by nklsrh
2nd Place: Rob in Soundland by michal.b
3rd Place: Vandalism by tuxie

Runners-up:
Jump the Wall by avnishkgaur
Cirplosion by Awesome

Thanks to everyone who participated in the November and December Dev Derbies! We love your demos. In the days ahead, we’ll be rounding up the votes on December’s IndexedDB submissions. Now’s the time to get your January Orientation demos built and submitted. Can’t wait to give them a spin (or a tilt).

REMINDER: We recently updated our Dev Derby rules to allow developers to participate in multiple contests until they win 1st place. That means if you’ve submitted awesome demos and come up short in the past, you still have a chance to win that top spot in future Dev Derbies… so keep those demos coming!

People of HTML5: Joe Lambert unshredding images in Canvas

Chris Heilmann — Mon, 28 Nov 2011 16:22:19 +0000

Today we have a quickie for you: Joe Lambert, a web developer from Southampton, England working for Rareloop took on the Instagram engineering challenge of un-shredding a shredded image but instead of using server-side code, he used HTML5 canvas. Here’s a screencast of his solution to the problem:

And here we are in a quick interview, chatting about how he approached the problem and why he used web technologies:

1) The specs of the competition said you can use Python, Ruby or C++ and you went instead for JavaScript and Canvas. Why?

JavaScript is my language of choice and what I use most of the time. I also find it the most fun as its easy to post your creations online and show people immediately, no downloads or dependencies to worry about. I wasn’t really looking to get hired by Instagram so adhering strictly to the rules wasn’t a big deal. Essentially it was an appealing problem that I thought could be solved in a browser, so wanted to prove that was the case.

I actually knew from the start that creating the algorithm to unshred the image was only half of what I wanted to do. That in itself was going to be a challenge and quite satisfying to solve but I really wanted to get to a point where I could make the slices move into the right place. I wouldn’t have been able to create this effect as quickly in any other environment.

2) How did you approach the problem of unshredding the images? Did you do similar things before? What is the logic you use?

I wasn’t too worried about the most efficient algorithm so I just started by thinking about how a human might solve this type of problem. I’ve played around with image data a little before but nothing that would have been directly useful to this particular problem.

If I were solving this in the physical world I’d start by picking up a slice at random then pick up each other shredded slice and see if it looked like it fitted to the left or right of the piece in my hand. I’d then repeat this till the image was back to its original state.

This gave an indication of the kind of structure my code should have, I just needed to work out how to computationally measure whether two strips ought to be next to one another. For this I just compared the pixels closest to each edge and measured the euclidean distance between the colour values. It seemed to work pretty well!

3) I see you are not using typed arrays in your solution yet. It seems to perform well the way it is now, do you think they’d make a difference?

I’m not sure, efficiency wasn’t high on my priorities for this challenge so I didn’t really look to optimise it but as you mention it seems to perform quite well. I haven’t tried the algorithm with larger images but I suspect it would perform not so well.

4) What was the feedback so far? Has Instagram contacted you?

The feedback has been really positive, especially via Twitter. I think generally the competition has had quite a high profile which certainly helped.

Instagram haven’t been in contact yet but it would be good to hear from them. I suspect they’re focusing on all the people who used the recommended languages

5) Seeing how easy this was can you see other challenges with images and canvas? Could you think of ways how to make it easier for developers, for example by extending the API?

The way you access pixel data in Canvas is a little complicated, it might be useful to have some helper functions to let you get access to a single pixel via X/Y coordinates. I ended up writing a little helper function to do this for this challenge, I’m going to try get the code up on GitHub shortly so this might be of use for others.

It would also have been useful to have been able to call toDataURL() on a CanvasPixelArray object. My implementation used the DOM and CSS3 transitions to move the images into their correct places after the algorithm had solved the order, being able to access a Data URI for each slice would have be handy.

If you want to chat with Joe, he is available on Twitter as @joelambert.

Mozilla hacks Weekly, November 24th 2011

Robert Nyman — Thu, 24 Nov 2011 12:44:29 +0000

It’s Thursday, folks, and it means our Developer Engagement Team at Mozilla has some reading tips for you!

Weekly links November 24th 2011

If there is anything you think we should read or know about, don’t hesitate to post a comment, contact us on Twitter or through any other mean.
The picks this week are:

Christian Heilmann

HTML5rocks has a simple CORS tutorial.

If you want to read more tips or discuss the web with Christian, he’s available on Twitter as @codepo8.

Havi Hoffman

Read it and weep: Home for the Holidays Tech Support – Will you be called upon to “fix the Internet” or help your dad root his new tablet?

Havi can be found on Twitter as @freshelectrons

Janet Swisher

Kameleoon is a web application that lets non-coding designers tweak an existing website’s design, and then generates the modified HTML and CSS for them. You can also use it for A/B testing of different designs.

If you want to read more tips or discuss the web with Janet, she’s available on Twitter as @jmswisher.

Jeff Griffiths

Brilliant post from Rob Campbell detailing some excellent Firefox dev tools power tips currently available in Firefox Aurora.

If you want to read more tips or discuss the web with Jeff, he’s available on Twitter as @canuckistani.

Robert Nyman

Going Fullscreen with HTML5 Canvas – Nice and simple example how you can utilize the Fullscreen API to create a complete full-screen canvas experience.

If you want to read more tips or discuss the web with Robert, he’s available on Twitter as @robertnyman.

Stormy Peters

Should you really have to wait for that email to send? Asynchronous UIs – the future of web user interfaces.

If you want to read more tips or discuss the web with Stormy, she’s available on Twitter as @storming

Webinar: Canvas with Rob Hawkes

Janet Swisher — Tue, 01 Nov 2011 19:38:17 +0000

Update 2011-11-10: Video of this webinar is now available:

type="video/mp4">
type="video/webm">
type="video/ogg">
This video is also available on Vimeo.

You can download the code that Rob demos in the webinar.

You might also check out the recording of the webinar created by BigBlueButton. It syncs the audio with the slides and chat window using Popcorn.js. On the downside, BBB starts recording as soon as anyone joins the conference audio bridge, which in this case happened an hour before the webinar started. Fast-forward to time 58:20 for the start of the webinar.

Please join us on Tuesday, November 8th at 17:00 UTC (convert to your local time) for a webinar on using the Canvas API, with our own Rob Hawkes, author of Foundation HTML5 Canvas.

Canvas is the topic for the November Dev Derby. Rob is planning to show how to make an image editor for avatars using Canvas. Spark your imagination with the webinar, create some new and amazing use of Canvas, and submit it to the Dev Derby.

We will use BigBlueButton for web-conferencing, and will avoid using the feature that led to the server problems with the September webinar. We will be recording the session, and I promise to double- and triple-check the recording set-up in advance :-)

Add this event to your calendar:

We’d like to get a rough estimate of how many people will be attending. If you happen to use Plancast, and you plan to attend the seminar, please join the event on Plancast.

To join the webinar:

Go to the BigBlueButton server for Mozilla.
For Full Name, enter your name.
Select “Mozilla Developer Network” in the Room list.
For Password, enter MDNHacks.

Note: Big Blue Button uses Flash, so make sure you have the latest version installed, especially if you are running Mac OS X Lion.

Direct2D Azure hits Firefox 7

Mozilla Hacks — Tue, 27 Sep 2011 14:11:25 +0000

Based on a blog post originally posted here by Bas Schouten, Firefox Developer.

Hrm, Azure, what’s that again?

You can find out all about Azure other blog posts, there’s an introduction from Joe Drew and there’s several more in detailed posts discussing the Direct2D Azure backend and the performance implications to be found on my blog. The bottom line is that we’re working on a new graphics API that will be used for rendering in Gecko.

What does that mean for Firefox 7?

Well, we’re currently only using it with Direct2D and when using canvas. This allows us to stress test it, although a wide array of tests has been run, and it has been in use by our Aurora and Beta testers for a while now, there might still be issues we might have missed. If these issues show in the final release we’ll only have caused a regression in Canvas and for a limited subset of our users, rather than in all browser rendering. The bottom line is you should generally see a speed improvement using 2D Canvas in Firefox 7 when using Windows 7 or Vista with a sufficiently powerful graphics card.

So what’s next, what’s the status?

We’re currently working hard on both a Cairo and a Skia backend for the Azure API, this means we’ll be able to use the Azure API on all platforms. Possibly getting some quick performance benefits on platforms where Skia outperforms the cairo backends we’re currently using. At the same time we’re working on creating a layer that will allow controlled migration of all our content drawing code from the current ‘Thebes’ API’s to the new Azure API. Once that is done webpage rendering in general can start taking advantage of all the latest work!

That’s about all I have for you right now, enjoy!

Calculated drop shadows in HTML5 canvas

Chris Heilmann — Mon, 29 Aug 2011 10:09:48 +0000

One of the best new features of HTML5 when it comes to visual effects is the canvas element and its API. On the surface, it doesn’t look like much – just a rectangle in the page you can paint on and wipe. Much like an etch-a-sketch. However, the ability to transform, rotate and scale its coordinate system is very powerful indeed once you master it.

Today I want to quickly show how you can do (well simulate) something rather complex with it, like a calculated drop shadow on an element. To see what I mean with this, check out the following demo which is also available on the Demo Studio:

See animated version on JSFiddle

(This is using JSFiddle to show you the demos, so you can click the different tabs to see the JavaScript and CSS needed for the effects. All of the demos are also available on GitHub.)

As you can see, the shadow becomes more blurred and less pronounced the further away the “sun” is from it. You can use the mouse to see the effect in the following demo:

See mouse enabled demo on JSFiddle

Let’s take a look how that is done. The first step is to have a canvas we can paint on – you do this simply by having a mouse detection script (which we used for years and years) and a canvas with access to its API:

See step one on JSFiddle

Click the play button of the above example and you see that you can paint on the canvas. However, the issue is that you keep painting on the canvas instead of only having the orb follow the cursor. To do this, we need to wipe the canvas every time the mouse moves. You do this with clearRect()

See step two on JSFiddle

Running the above example now shows that the orb moves with the mouse. Cool, so this is going to be our “sun”. Now we need to place an object on the canvas to cast a shadow. We could just plot it somewhere, but what we really want is it to be in the middle of the canvas and the shadow to go left and right from that. You can move the origin of the canvas’ coordinate system using translate(). Which means though that our orb is now offset from the mouse:

See step three on JSFiddle

If you check the “fix mouse position” checkbox you see that this is fixed. As we move the coordinate system to the half of the width of the canvas and half of its height, we also need to substract these values from the mouse x and y position.

Now we can draw a line from the centre of the canvas to the mouse position to see the distance using c.moveTo( 0, 0 );c.lineTo( distx, disty ); where distx and disty are the mouse position values after the shifting:

See step four on JSFiddle

In order to find out the distance of the shadow, all we need to do is multiply the mouse coordinates with -1 – in this demo shown as a red line:

See step five on JSFiddle

This gives us a shadow distance from the centre opposite of the mouse position, but we don’t want the full length. Therefore we can apply a factor to the length, in our case 0.6 or 60%:

See step six on JSFiddle

Now we are ready for some drop shadow action. You can apply shadows to canvas objects using shadowColor and its distance is shadowOffsetX and shadowOffsetY. In our case, this is the end of the red line, the inversed and factored distance from the mouse position to the centre of the canvas:

See step seven on JSFiddle

Now, let’s blur the shadow. Blurring is done using the shadowBlur property and it is a number starting from 0 to the strength of the blur. We now need to find a way to calculate the blur strength from the distance of the mouse to the centre of the canvas. Luckily enough, Pythagoras found out for us years ago how to do it. As the x and y coordinate of the mouse are the catheti of a right-angled triangle, we can calculate the length of the hypothenuse (the distance of the point from the centre of the canvas) using the Square root of the squares of the coordinates or Math.sqrt( ( distx * distx ) + ( disty * disty ) ).

This gives us the distance in pixels, but what the really want is a number much lower. Therefore we can again calculate a factor for the blur strength – here we use an array for the weakest and strongest blur blur = [ 2, 9 ]. As the canvas itself also has a right-angled triangle from the centre to the top edge points we can calculate the longest possible distance from the center using longest = Math.sqrt( ( hw * hw ) + ( hh * hh ) ) where hw is half the width of the canvas and hh half the height. Now all we need to do is to calculate the factor to multiply the distance with as blurfactor = blur[1] / longest. The blur during the drawing of the canvas is the distance of the mouse position multiplied with the factor or currentblur = parseInt( blurfactor * realdistance, 10 );. We disregard blur values below the range we defined earlier and we have our blurred shadow:

See step eight on JSFiddle

In order to make the shadow weaker the further away the mouse is we can use the alpha value of its rgba() colour. The same principle applies as with the blur, we set our edge values as shadowalpha = [ 3, 8 ] and after calculating them from the distance we apply their inverse as the alpha value with c.shadowColor = 'rgba(0,0,0,' + (1 - currentalpha / 10) + ')';. This blurs and weakens the shadow:

See step nine on JSFiddle

You can do a lot more with this, for example we could also scale the sun orb the further out it gets or use a second shape to resize and blur it. You can also go completely overboard.

Found a way to optimise this? Tell us about it!

Living on the Edge – new Adobe animation tool sparks necessary conversations

Chris Heilmann — Wed, 03 Aug 2011 09:14:10 +0000

Adobe made quite some splash in the last days by releasing Edge, a Flash-like tool to create HTML5/CSS3/JS driven animations. There is a need for a tool like that and I for one am very happy to see that Adobe are recognising this. Other tools that try to tackle the same task are already around, with the yet to be released Animatable being shown at quite a few conferences and Radi leading the way.

I am even happier that this sparked quite a conversation amongst the developer community – there are far too many truisms thrown around about HTML5 and CSS3, so anything that makes us re-evaluate the current state is a great idea.

Edge, according to Adobe themselves has been downloaded 50,000 times in the first 24 hours and looks much like Flash used to a long time ago:

The initial feedback was very polarised, whereas fans of Adobe products heralded this as a new dawn of standards-based animations, others were less impressed as the UK based .net magazine reports. This is to be expected, as everything with the HTML5 stamp these days gets that kind of treatment.

So what is going on under the hood of Edge?

Anna Debenham was one of the first to have a go at reviewing Edge and having a play with it:

As you can see in the frame above, the animations generated with it are fixed in size (either pixels or ems) and all consist of a mix of JavaScript and CSS3 transitions. The engine behind the animations is jQuery. Looking it up in Firebug, we are faced with a lot of DIV elements with positioning and CSS transition information:

This is in stark contrast to tools that try to turn Flash into open technology solutions like the conversions done by Google’s Swiffy. Swiffy uses SVG to simulate the path-based nature of Flash; Edge doesn’t and relies on DOM animation of DIV elements instead.

A total lack of HTML5 in the output

This caused the main confusion amongst developers and sparked a longer discussion on the Adobe forums initiated by Rob Hawkes. The original problem Rob found with Edge is that it doesn’t use HTML5 technologies like Canvas or other W3C animation and painting technologies like SVG:

Why has Edge gone with div-based animation over other technologies like canvas and SVG? I was deeply saddened to see that not only were divs used in the example files that you released, but that divs are the default option for the stage and any other element that is added to it.

Adobe’s answer was that animating DOM elements simply is faster and allows for more browser compatibility, especially on mobiles:

We seriously considered Canvas, but current performance on mobile browsers (especially iOS) is very bad. We didn’t want to have the first experience produce content that couldn’t run acceptably. Note that this may be changing in iOS 5, so that’s good. Finally, SVG can be a little slow, though we do support bringing in SVG elements and animating them – just not reach into them.

The performance argument is something that keeps cropping up in discussions. A lot of the animations by Google also moved DIV elements around rather than using Canvas and/or SVG (remember the exploding balls logo?) whilst sporting the HTML5 label.

HTML5 Games developer and co-organiser of the onGameStart conference Kamil Trebunia, questioned the concerns over Canvas being too slow on mobile devices. The great news is that in a backchannel discussion on Twitter, Paul Bakaus, CTO of Zynga Germany pointed out that they are doing intensive research into the subject at the moment and that they are happy to share the outcome at a later stage. So this is something very cool to look forward to and something to complement the HTML5 games performance research by Facebook a few months ago.

Lack of semantics in the output

The other issue with Edge that puzzled developers is the HTML that gets generated. As Morena Fiore asked with her Edge demo it seems weird that Adobe doesn’t use SVG and Canvas for animating when the output of the tool doesn’t feature any semantic HTML whatsoever:

The text content, the tweens and positioning and all the other information of the animation is in JSON objects instead:

Adobe’s original answer to these concerns was that they don’t want to touch the original HTML as developers don’t like that:

Our approach is that we separate out the stuff you add from the underlying HTML *AND* we don’t try to edit the HTML in the way old-style tools did – in general a lot of people don’t like when tools muck with HTML.

That said, we will be looking at ways of trimming down the HTML that gets added to a page in the future if that’s what people want.

That sounds like a slippery slope to me. As Tobias Leingruber showed in his firmly tongue-in-cheek demo done with Edge a lack of semantics and enhancing existing markup just means that we now create Flash tunnel pages in a different technology and makes you wonder if the main use case of Edge would be interactive ads that are not blocked by Flashblockers.

This is a challenge that any tool faces: how do I not only allow people to create but also make them aware that web content is not only pretty moving pictures? The Madmaninmation demo of animatable is a great example — it is a pure animation but when you check the source code it falls back to a list with the script of the animation. This allows everybody to know what is going on and helps your product to be understood by search engines. Will every user of animatable go through that level of support for the web or simply create animations?

With Edge, Adobe has a chance to do a very cool thing for open web technologies. It will be interesting to see where it goes.

As Bruce Lawson of Opera put it:

Designers will require authoring tools that can help them turn their creativity into code, and Adobe have a great track record on making IDEs that designers find user-friendly.

So the ball is in Adobe’s court to do the right thing for the web and move Edge further along (with help from the research the discussion sparked). And they are well on the way and declared that they are actively looking for feedback:

Edge will be evolving rapidly – the product is by no means feature complete. We expect to add support for more and more of the HTML universe over time. Edge does currently support SVG graphics (you can File > Import an SVG image), but you can’t yet reach into those graphics and animate their components. The good news is that our JS runtime is capable of animations that include all sorts of HTML Elements, SVG Elements and even Canvas graphics, so we have not boxed ourselves in.

Which of these enhancements would you like to see Edge tackle first? Canvas? Embedded SVG? Ability to select HTML tags other than DIV?

I, for one am excited to see Adobe start to wave the flag of open web technologies higher than before. Their new The Expressive Web showcase site is a proof of that. It is up to us now to cry foul at what they do or work side by side to bring the best experiences to the web for our users out there.

The HTML5 canvas clip method

Robert Nyman — Wed, 27 Jul 2011 08:26:31 +0000

HTML5 canvas offers a lot of interesting ways to create effects and experiences on the web. One of the methods to do that, which seem lesser known, is canvas clip.

I wrote about it in Using the HTML5 canvas clip method to hide certain content, showing how to stage content and create a looking glass effect. You could also see it in action in the canvas clip demo.

You could also read up more on it in the Clipping paths in the Compositing article on MDN and learn how to create some interesting combinations.

HTML5 APIs – Where No Man Has Gone Before! – Presentation at Gotham JS

Robert Nyman — Wed, 20 Jul 2011 13:21:04 +0000

Last weekend I was in New York City to speak at the GothamJS conference and Mozilla also sponsored it. It was a nice event with about 200 attendees, taking place in the NYIT Auditorium on Broadway.

The event was one-track with 8 speakers, and personally I always prefer when it’s just one track for follow-up discussions and that everyone have seen and heard the same thing. The topics were ranging broadly between script loaders and HTML5 in one end, and voice-controlled telephony applications in the other.

My presentation

My talk was about HTML5 APIs in general, to give an introduction to them but also to inspire people to try things out and also give feedback to both working groups and web browser vendors about current implementations.

Slides can also be downloaded at SlideShare.

Additionally to the APIs covered in my London Ajax Mobile Event presentation, I went through Web Sockets, File API, HTML5 video, canvas and WebGL. Also, if you are more interested in the

What I especially liked talking about is services like vid.ly which helps you take control over the problem of different video codec support in different web browsers, by storing various formats and then deliver the most suitable one depending on the web browser/device accessing it.

Another favorite is Universal Subtitles, which is an excellent tool for everyone to be able to add subtitles to a video clip, empowering users with varying language skills to take part of a video and its content and sharing it with the world.

An option to make the content of a web site richer, there is Popcorn.js to sync key events in the video playing to what kind of text or other information you want to present to go with that. To complement that, the Butterapp is an editor to create that kind of content syncing, currently in alpha.

I also mentioned videograbber for taking easy video screenshots in the web browser.

Dev Derby challenge

I also want to take the opportunity to remind you that Mozilla Dev Derby has a challenge for what you can accomplish with the that goes till the end of July, so please submit anything if you have a good idea!

Doom on the Web

Janet Swisher — Fri, 03 Jun 2011 14:00:15 +0000

Update: We had a doubt whether this port of the Open Source Doom respected its term of use. We decided to remove it from our Website before taking an informed and definitive decision.

This is a guest post written by Alon Zakai. Alon is one of the Firefox Mobile developers, and in his spare time does experiments with JavaScript and new web technologies. One of those experiments is Emscripten, an LLVM-to-JavaScript compiler, and below Alon explains how it uses typed arrays to run the classic first-person shooter ~~Doom on the web~~.

As a longtime fan of first-person shooters, I’ve wanted to bring them to the web. Writing one from scratch is very hard, though, so instead I took the original Doom, which is open source, and compiled it from C to JavaScript using Emscripten. The result is a version of Doom that can be ~~played on the web~~, using standard web technologies.

Doom renders by writing out pixel data to memory, then copying that pixel data to the screen, after converting colors and so forth. For this demo, the compiled code has memory that is simulated using a large JavaScript array (so element N in that array represents the contents of memory address N in normal native code). That means that rendering, color conversion, and copying to the screen are all operations done on that large JavaScript array. Basically the code has large loops that copy or modify elements of that array. For that to be as fast as possible, the demo optionally uses JavaScript typed arrays, which look like normal JavaScript arrays but are guaranteed to be flat arrays of a particular data type.

// Create an array which contains only 32-bit Integers
var buffer = new Int32Array(1000);
for ( var i = 0 ; i < 1000 ; i++ ) {
    buffer[i] = i;
}

When using a typed array, the main difference from a normal JavaScript array is that the elements of the array all have the type that you set. That means that working on that array can be much faster than a normal array, because it corresponds very closely to a normal low-level C or C++ array. In comparison, a normal JavaScript array can also be sparse, which means that it isn't a single contiguous section of memory. In that case, each access of the array has a cost, that of calculating the proper memory address. Finding the memory address is much faster with a typed array because it is simple and direct. As a consequence, in the Doom demo the frame rate is almost twice as fast with typed arrays than without them.

Typed arrays are very important in WebGL and in the Audio Data API, as well as in Canvas elements (the pixel data received from getImageData() is, in fact, a typed array). However, typed arrays can also be used independently if you are working on large amounts of array-like data, which is exactly the case with the Doom demo. Just be careful that your code also works if the user's browser does not support typed arrays. This is fairly easy to do because typed arrays look and behave, for the most part, like normal ones — you access their elements using square brackets, and so forth. The main potential pitfalls are:

Typed arrays do not have the slice(). Instead they have the subarray(), which does not create a copy of the array — instead it's a view into the same data.
Don't forget that the type of the typed array is silently enforced. If you write 5.25 to an element of an integer-typed array and then read back that exact same element, you get 5 and not 5.25.

MarbleRun, an HTML5 Game

Paul Rouget — Wed, 27 Apr 2011 16:28:18 +0000

Take the HTML5 Canvas element, a Javascript library to do the physics (box2d), add a nice design, a social touch, and you have an awesome HTML5 Game called MarbleRun!

MarbleRun was the winner of the last Mozilla Game On Challenge, among other exciting HTML5 games (check them out), and had been added to Web O’ Wonder.

The authors of MarbleRun visited us a couple of weeks ago and gave the following talk about the story of their game:

Firefox 4 Demos: Runfield – a Canvas Game

Paul Rouget — Wed, 16 Mar 2011 16:54:41 +0000

Yeah! Another awesome demo in Web’o Wonder!

With Hardware Acceleration and a fast JavaScript engine, the web platform is ready for Games. Runfield is an example. This HTML5 Game has been developed by Ilmari Heikkinen. It’s based on Canvas 2D, and this game works on all the recent browsers.

(small definition version here – Oh! and check-out the little built-in level editor)

Want to see more HTML5 Games? Take a look at Mozilla Labs Gaming:

People of HTML5 – Rob Hawkes

Chris Heilmann — Thu, 27 Jan 2011 09:08:00 +0000

HTML5 needs spokespeople to work. There are a lot of people out there who took on this role, and here at Mozilla we thought it is a good idea to introduce some of them to you with a series of interviews and short videos. The format is simple – we send the experts 10 questions to answer and then do a quick video interview to let them introduce themselves and ask for more detail on some of their answers.

Leggi la traduzione in italiano

Today we are featuring Rob Hawkes author of Rawkets, a multiplayer game using canvas and websockets.

Rob was one of the presenters at the Game On 2010 event in London, England where he showed off his game. The video and his slides are available on the Mozilla Games Blog. He is currently busy working on a book about Canvas. You can find Rob on Twitter as @robhawkes.

The message I really loved getting from Rob was that it is OK to use any technology you want and that there is a benefit to people messing around with your code – even if it is to game the system.

The video interview

You can see the video on any HTML5 enabled device here (courtesy of vid.ly).

Your browser does not support the video tag, Check the video on vid.ly.

Ten questions about HTML5 for Rob Hawkes

1) So, you’ve been fiddling around with Canvas for gaming a lot – why the fascination? Did you write games in other environments before?

I sure have. My life has practically revolved around making games of some sort for the past year or two now. I just love the concept of using games to experiment with a programming languages and really learn everything I can about them. It’s a good way to try out most of the features involved in visual programming as well, particularly with harder concepts like physics, which I’m still no expert in. My fascination with games programming started out of necessity really; I had to create a game in my second year of university, so I co-created an augmented reality game using Papervision3D and ActionScript. From there I co-created another game of sorts in Adobe Flex that allowed people to play against each other using a webcam, which was fun. And now I’m making games in canvas; partly because it’s fun, and partly to learn everything I can about it. I get a kick out of the interactivity and playability factor that games bring, which I think hooks into my addiction for visual programming in general.

2) The cool thing about open technologies is that it is easy to see what is going on under the hood. View source and you get the picture. For gaming, isn’t that a problem? To “game” a highscore for a Flash game at least you need to do some HTTP sniffing – with Canvas it must be even easier. How do you protect yourself from that?

The fact that canvas is so open is something that makes me interested in it. I’m usually pretty open about programming and I’d love it if people can learn from my code, so being able to view the source code directly within the browser is amazing for that. However, the major drawback is that your code is open for everyone to see, so security becomes a pretty big issue.

You’re right though, this is particularly troublesome with gaming, and I’m inclined to say that there isn’t much that you can do about it. In fact, is wasn’t long after I released the first version of Rawkets that people started to poke around the code and manipulate the game to their advantage.

In all honesty, I actually see it as a good thing, as it showed me where my code was weak, and highlighted the areas that I needed to secure. You can obfuscate and minify your code as much as you want, but people can still get access to it because it’s just the way JavaScript works. However, there are a few ways to mitigate the problem, like passing core logic for the game to a remote server and communicating to it via restrictive and secure API calls. This is a good method for multiplayer games, where data received from the server is seen as trustworthy, and where as data on the client side is not. Or, you can wrap sensitive code in closures and take advantage of local variables to hide away data that you don’t want people to easily access. Another option is to just not worry about it. So long as you don’t let the hackers ruin the game for other players, does it really matter if they screw it up in their own browser?

In reality, none of these methods will 100% protect you, but they’ll certainly make things harder for would-be hackers.

3) Talking about open. Isn’t Canvas just a Java applet with a predefined API and no need for a slow JVM? You still only have a rectangle somewhere in the page you can manipulate…

If you mean that canvas is like a sandbox that you can only access via a specific set of API calls, then yes, pretty much. Then again, the API calls allow you to do a hell of a lot, and couldn’t you say the same about Flash and all the closed methods of creating graphics?

I suppose when you say canvas is open; you’re mainly talking about the development behind its creation (ie. the W3C), and the fact that the code to draw elements on it isn’t hidden away from prying eyes. Canvas is definitely not open in the sense that you are free to extend it and do whatever you want with it. So long as you abide by the API calls, which is fine for 99% of the time, then you’ve not got a problem.

As you say, canvas is effectively just a rectangular area that the browser allows you to draw into. Once you’ve drawn a few shapes into it, you can’t access them individually, but then again, that’s exactly how canvas is meant to work. It’s not like SVG, where every element that you’ve drawn can be accessed from the DOM and edited. Canvas is a bitmap system that has no concept of individual elements. It’s a destructive system that basically just changes the colour of pixels within its rectangle, based on the API calls that you give it. In that way, it’s just like Microsoft Paint from back in the day.

4) What about accessibility? Can assistive technology access canvas content? How about keyboard access?

To put it bluntly, not really. At least, not yet.

Canvas does have fallback content, which is placed between the canvas tags and shows up when a browser doesn’t support canvas, but it doesn’t have much else to aid with accessibility. The cool thing about the fallback content is that it’s possible to place content in there for keyboard access. I believe IE9 already has this functionality, which lets you show the canvas, but still give keyboard access so you can describe what’s going in within the canvas in text, or something like that. The problem with methods like this is that you effectively duplicate all of your content; you draw everything on the canvas, and then you have to write it all up within the canvas tags to explain what you’ve drawn.

What you can’t do yet is give assistive access to individual elements within the canvas, simply because canvas is just a big rectangle of pixels with no concept about what’s bee drawn. This is the major drawback with using a bitmap system that has a pretty closed API. There is talk about using something called the shadow DOM, to open up the canvas and allow some sort of access into the elements that have been drawn, but I don’t believe much has been done about this yet.

It’s definitely an area that I’m interested to see progress, but in the meantime, Bruce Lawson highlights a couple of the proposals that are floating around to sort out the accessibility issue with canvas.

5) What about performance? What are the hot tips to keep a canvas game run smoothly?

This is a massive sticking point with canvas at the moment; it can be so god damn slow! Actually, this isn’t really a problem with canvas, but rather the fact that canvas uses the browser process to draw everything, which in turn uses the processor (CPU).

Some browsers are introducing something called hardware acceleration, which passes off canvas and graphical processes to the graphics card (GPU). This simple feature really speeds things up, and allows you to do some pretty intensive stuff. Still, hardware acceleration isn’t a magic bullet; you still need to program defensively and be aware of potential bottlenecks and performance issues with your code.

The most common issues arise around unnecessarily extravagant timers and loops. For example, animating in canvas usually requires the use of a JavaScript timeout, which runs a function that draws everything onto the canvas 30 times a second; fast enough to look like things are moving. This is great, but what about if you’re not animating things any more? I’ve made the mistake of not turning off the animation timer, and inherently sucking precious CPU juice for no good reason.

Another example is that of collision detection, or any other check that has to be performed on a large amount of objects. The bad way of doing this is to loop through every object, and then loop through every object a second time on each iteration of the original loop. The problem with this is that you end up checking some objects twice, or more, and basically wasting resources. Instead, you can optimise your loops to ignore objects that you’ve already checked, which can cut the amount of loops down considerably.

There are countless ways to optimise your code, and most won’t achieve much on their own, but together they can add up to save a pretty significant amount of resources.

6) Your game uses Web sockets for communication. How did you find that to work with? Does it scale well? Are there any things you don’t like about the technology?

You mean that it used to use WebSockets? It’s a shame FireFox and others decided it would be a great idea to drop support for them for the time being. Jesting aside, the game is built atop WebSockets, and it’s been an interesting journey. I love WebSockets, and I think that having a full-duplex method of communicating within a browser is fantastic. The fact that the data is streamed live opens up a whole range of possibilities for games and Web apps.

I didn’t find too many issues with WebSockets regarding stability and scalability, as in reality most of the problems lay with the code on the server that deals with sending information back and forth. I found that one thing you need to be aware of is that the amount of data being sent back and forth can increase at an exponential rate if you’re not used to optimising communication for a large amount of users.

I have to admit though, my biggest bug bear with WebSockets is that it sends data as text, and it doesn’t support the UDP protocol. Right now, WebSockets sending everything as text data is great because it’s dead simple, but it sucks because it can potentially take a lot of precious bandwidth, especially when you’re sending data many times a second to hundreds of different players in a game.

The UDP problems relates to how canvas currently only supports the TCP protocol. What this means is that every piece of data that is sent back and forth between a server and a client has to be received in order. This is not good for gaming, where a bit of lag can end up causing a backlog of messages that have to be sent in order before any new messages can be sent. UDP on the other hand would allow you to send messages as and when they occur, without worrying about order. This is very good for games that rely on precise data that is time-sensitive, like the movement of a player in an FPS game.

You have to remember that WebSockets is something new, and it’s likely to have some teething troubles at the beginning; hence it being disabled temporarily. Still, it’s got a bright future, and I’m looking forward to seeing how it progresses.

7) Have you played with SVG, too? If so, when would you use what and is there a chance to use the strong parts of both technologies together?

Admittedly, I’ve not played with SVG as much as I’d like to. I’ve just become so caught up in the world of canvas in recent months. However, it’s something that I strongly believe should be used instead of canvas in particular circumstances, like when dealing with the conversion of existing HTML data into a graphical format (eg. creating graphs from a table).

SVG is also great because it’s accessible via the DOM, but it falls down with animation and gaming. It was never built with this kind of use in mind, and it’s exactly why canvas exists. Canvas is perfect for fast and dynamic animations, and it’s particularly perfect for creating games.

Fortunately, it isn’t always a case of being either or with SVG and canvas. It’s completely possible to draw SVG shapes within canvas, which has the added benefit of letting you have all the vector goodness of SVG within a bitmap system. I’ve heard of people using techniques like this for sprites within games, and for drawing images onto canvas that need to be resized a lot.

8) What’s the next barrier you’d love to see opened up? What features of closed technology are there that are beyond your reach but you are itching to try out?

I’m really interested in seeing the Device APIs (DAP) being supported in the major browsers. One of the things that is lacking in browsers at the moment is support for external devices like webcams and microphones. I can just see canvas and DAP being used together for some pretty awesome Web apps and visualisations. Imagine being able to create an avatar for a website by using a bit of canvas with an attached webcam. Or, what about live video chat through the browser with WebSockets and canvas/HTML5 video? That would be JavaScript awesomeness right there!

9) Currently you’re writing a book on Canvas. As not everybody has time to wait for this, where could people go in the meantime to get the real info about starting with canvas?

It’s not a massive wait (it’ll be out in May), but in the meantime I’d suggest going to the Mozilla Developer Network. It’s where I learnt how to use canvas, so it must be good! Apart from that, there really is a lack of decent resources for learning canvas at the moment, but I predict that changing for the better over the next year.

10) Where could people go to get inspiration? What other uses of Canvas have you seen that made you go “woooo I want that!”?

There are some fantastic projects being created with canvas at the moment. I could create a massive list, but here are some of my favourite:

Some particularly impressive uses of canvas that have got my juices flowing are projects that really push the pixel manipulation features of canvas, like using it to detect faces in images, or nudity. Weird, but pretty impressive stuff!

Do you know anyone I should interview for “People of HTML5″? Tell me on Twitter: @codepo8

HTML5 Guitar Tab Player with the Firefox 4 Audio Data API

Paul Rouget — Fri, 07 Jan 2011 16:55:19 +0000

Greg Jopa, an Illinois State University grad student studying web development, built a web-based guitar tab player using Firefox’s Audio Data API and Vexflow (HTML5 music notation rendering API). Here is some details from Greg. You can also read more about this experiment on his blog.

I created a mashup using the Firefox 4 Audio Data API, MusicXML, and Vexflow to create a HTML 5 Guitar Tab Player.

This is a Youtube video. It uses the HTML5 tag if you activate it here.

View demo.
Source code.

Using JavaScript, this guitar tab player converts the music note data from a MusicXML file to frequencies playable by the Audio Data API. This note data is also converted to a format that can be read by Vexflow to display the guitar tab.

I have broken down this functionality into 4 steps. Here’s how the HTML5 Guitar Tab Player works:

MusicXML

When a tab (a tablature) is selected, the jQuery ajax() function loads the MusicXML file, parses through the contents, and appends all the formatted note data to a