Audio Articles

Text-to-Speech (TTS) can make content more accessible, but there is so far no simple and universal way to do that on the web. One possible approach is shown in this demo, which is powered by speak.js, a new 100% pure JavaScript/HTML5 TTS implementation. speak.js is a port of eSpeak, an open source speech synthesizer, from C++ to JavaScript using Emscripten.

Compiling an existing speech synthesis engine to JavaScript is a good way to avoid writing a complicated project like eSpeak from scratch. Once compiled, the eSpeak code in speak.js doesn’t know it’s running on the web: speak.js uses the Emscripten emulated filesystem to ‘fake’ the normal file reading and writing calls that the eSpeak C++ code has (fopen, fread, etc.). This allows the normal eSpeak datafiles to be used (either through an xhr, or by converting them to JSON and bundling them with the script file). The result of running the compiled eSpeak code is that it ‘writes’ a .wav file with the generated audio to the emulated filesystem. speak.js then takes that data, encodes it using base64, and creates a data URL. That URL is then loaded in an HTML5 audio element, letting the browser handle playback. (Note that while that is a very simple way to do things, it isn’t the most efficient. speak.js has not yet focused on speed, but with some additional work it could be much faster, if that turns out to be an issue.)

Why would you want TTS in JavaScript? Well, with speak.js you can bundle a single .js file in your website, and then generating speech is about as simple as writing

speak("hello world")

(see the speak.js website for instructions). The generated speech will be exactly the same on all platforms, unlike if your users each did TTS in their own way (using an OS capability, or a separate program). speak.js can also be used to build browser addons in a straightforward way, since it’s pure JavaScript – no need for platform dependent binaries, and the addon will work the same on all OSes.

A few more comments:

• JavaScript is getting more and more capable all the time. The development versions of the top JavaScript engines today can run code compiled from C++ only 3-5X slower than a fast C++ compiler, and getting even better. As a consequence, expanding the capabilities of the web platform can in many cases be done in JavaScript or by compiling to JavaScript, instead of adding new code to the browsers themselves, which inevitably takes longer – especially if you wait for all browsers to implement a particular feature.
• While speak.js uses only standards-based APIs, due to browser limitations it can’t work everywhere yet. It won’t work in IE, Safari or Opera since they don’t support typed arrays, nor in Chrome since it doesn’t support WAV data URLs. So currently speak.js only works properly in Firefox. However, the missing features just mentioned are not huge and hopefully those browser makers will implement them soon. It is also possible to implement workarounds in speak.js for these issues (see next comment).
• Help with improving speak.js is very welcome! One important thing we need is to implement workarounds for the issues that prevent speak.js from running on the browsers it currently can’t run on. Another goal is to build browser addons using speak.js. Please get in touch on github if you want to help out.
• eSpeak supports multiple languages so speak.js can too. You do need to include the additional language files though. Here is an experimental build where you can switch between English and French support (note that it is an unoptimized build, so it will run slower).

The audio team is made up of a group Mozilla volunteers who developed the Audio API and, most recently,  a new generation of WebGL demos. This is the story of the development of the No Comply demo.

In the fall, after finishing Flight of the Navigator, our team of audio and WebGL hackers was looking for a new challenge. We’d finished the new Audio API in time for Firefox 4, and were each maintaining various open web libraries, exploiting the new features of HTML5, Audio, JavaScript, and WebGL. We wanted to take another shot at testing the limits of Firefox 4 – then, still in beta.

Seth Bindernagel had the answer. He’d been in contact with a DJ and producer friend named Kraddy, who had just finished an amazing new album. “What if we tried to do something with his sound?” The idea was too good to pass up, and with Kraddy’s support, we dove into the tracks and started imagining what these songs might look like, when interpreted through the medium of the web.

«The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike»

Kraddy’s music was easy to demo because of its complex nature, with plenty of emphatic transitions and cue points–this music wants to be visualized! The music for No Comply also provided a dark and introspective sound on which to build a narrative. On his blog, Kraddy had already written about how he understood the album’s meaning:

This EP is about Theseus’ decision to be a hero and his decent into the Labyrinth to kill the Minotaur. In a broader sense the EP is about the battle we all face when we challenge ourselves as people. We must enter the Labyrinth of our minds and at the center we find our greatest fears. To defeat those fears we must kill a part of ourselves. And in killing a part of ourselves we create the potential to grow into a more developed person.

Kraddy’s vision informed our early outlines and storyboards. We knew that we wanted to play on the story of the Minotaur and the Maze, and the idea of facing down ones’ own fears. Together we came up with the idea of re-telling the story using a mixture of real-life video and 8-bit video game styling. Because the album was deeply personal to Kraddy, we decided to feature him in the demo. Kraddy agreed to be filmed, and Brett Gaylor used the footage to create the opening and closing video sequences. We also used Kraddy as the inspiration for the demo’s main video game character.

The launch of Firefox 4 brings a lot to the web, not least WebGL. As the web shifts from a 2D-only to a 2D and 3D space, we wanted to explore the intersection of these two familiar graphical paradigms. Rather than picking just one, we chose to create a hybrid, dream world, composed of 3D and 2D elements. Many people will recognize in our 2D characters and graphics an homage to much earlier video games, like Double Dragon. We wanted to celebrate the fact that these two paradigms can now exist together in a simple web page–everything we do in the demo is one web page, whether audio, video, 2D, 3D, or text.

Like the Flight of the Navigator(FOTN) demo before it, we chose the CubicVR.js engine to drive all the 3D graphics. Over the months leading up to the demo, Charles J. Cliffe had begun the painstaking process of porting features from his C++ engine over to JavaScript. The simple environment of WebGL and JavaScript allowed for features that even his C++ version did not yet posses to be quickly prototyped. Many bottlenecks had to be overcome during iterations of the demo, as we wanted to push the limits further than before. The biggest hurdle was visibility and lighting. Luckily, Bobby Richter came to the rescue. Using his experience with Octrees, he was able to work with Charles to produce a visibility and lighting pipeline which provides impressive performance for the task. In contrast, FOTN has no visibility system and was shaded by a single global directional light and ambient surface textures (for window lights, etc.) to simulate the rest. In No Comply we were able to push the limits with high poly counts and many overlapping point lights and were still able to reach the framerate cap.

Creating a 3D world like the one in this demo requires a lot of original content creation, which in turn requires some sophisticated tools. Instead of developing our own, and in the open-nature of our group, we decided to use existing technology like Blender. The community that develops Blender and creates content with it is rich and diverse, and because it’s an open tool, we could add the features we needed when they weren’t already present.

Our preference for open technologies also meant that the COLLADA scene format was an obvious choice. Unfortunately, as of version 2.49, Blender exports an Autodesk-inspired format of COLLADA, which isn’t quite up to the official standard, missing many important bits of information. Fixing this directly in Blender (with a little bit of Python hacking) let CubicVR stay standards-compliant, and let us milk Blender for all of the scene information we could think of using.

The demo’s 3D modelling, while important, comprises perhaps only half of No Comply’s original content. An incredible undertaking on the part of Omar Noory provided the textures for the rich environment through which Kraddy rumbles and tumbles. Frequently, spontaneous requests for “an 8 bit trash can,” “a cool sign with our names on it,” or, “some beefy bad lookin’ dudes” were answered almost instantly by Omar’s gracious and masterful digital pen. You may have recognized Omar’s name from his claim to meme-fame with “Haters Gonna Hate”.

Adding the perfect amount of flare to the graphics pipeline is Al MacDonald’s Burst animation engine. Al not only wrote our sprite animation engine, but also the web-based toolset we used to create the animations. The 8-bit Kraddy and all of No Comply’s 8-bit baddies are driven by animation paths prepared with Burst, and engineered with a set of tools that work right inside the browser.

In addition to cutting edge graphics with WebGL and <canvas>, we also wanted to explore how far we could push the new Firefox 4 Audio API we’d developed. The Audio Data API allows us to do many new things with the HTML5 <audio> and <video> tags, such as outputting generated audio and revealing realtime audio data to JavaScript. Libraries like Corban Brook’s DSP.js and and Charles’ BeatDetektor.js were used to analyze the audio in realtime and trigger various effects and animation sequences. Tracks of audio triggers were also recorded for tighter sequencing of key elements in the song we wanted to emphasize. One of the really new techniques we played with a lot in the demo was controlling GLSL shaders and lighting directly with audio, punching in and out with every beat and clap. Unlike most treatments of audio on the web, in this demo the song isn’t a background element, but is woven into the fabric of all the visuals and effects.

Getting a demo of this scale to work in the browser means figuring out how to make every bit of it work fast, and keep framerates high. Everything we do in the demo, from loading and parsing massive COLLADA models, to controlling 3D scene graphs, to analyzing real-time audio data, is done with JavaScript. We think it’s important to point this out because so many people begin with the assumption that JavaScript isn’t fast enough for the kind of work we’re presenting. The truth is that modern JavaScript, like that in Firefox 4, has been so heavily optimized that we all need to rethink what is and isn’t possible on the web.

We’ve taken advantage of a bunch of Firefox 4′s new performance features, as well as new HTML5 goodies, in order to make this all possible. For example Web Workers let us move heavy resource parsing off the main thread, freeing it for audio analysis and 3D effects. While a large portion of each second is consumed by simply pushing information to the video card, it isn’t necessary for the browser to wait for that to happen. In the background, we can use other threads to load and parse data, so that it’s ready to draw when the main thread needs it. Of course, a host of problems arise immediately whenever concurrency is involved, but we managed to draw a large performance and overall stability increase by utilizing Web Workers.

Another performance trick was using JavaScript Typed Arrays, which give us a tremendous speed boost when working with audio and pixel data. When you’re analyzing slices of audio data hundreds of bytes wide as fast as possible, your Fourier Transform code needs to be blazingly quick. Thanks to Corban’s highly optimized dsp.js library, this was hardly on our minds.

Next, we spent a lot of time optimizing our JavaScript so that it could take advantage of Firefox’s Tracing and Method JIT. Writing code that can be easily byte-compiled by the browser makes sure that anything we write runs as fast as possible. This is a fairly new and surprising concept, especially to those who remember the JavaScript of yesterday.

Part of what appealed to us about writing this demo was that it let those of us who are browser developers, and those of us who are web developers, work together on a single project. Most of the technology showcased in this demo was made on bleeding edge Firefox nightlies and our development process involved lots of feedback about performance or stability issues in the browser. Dave Humphrey focused on the internals of the Audio API, instrumenting and profiling our JavaScript, and helped us work closely with Mozilla’s JavaScript, graphics, and WebGL engineers. People like Benoit Jacob and Boris Zbarsky, among others, were indispensable as we worked to fix various bottlenecks. Part of what makes Mozilla such a successful project is that their engineers are not locked away, unable to work with web developers. Having engineers at our beck and call was essential to our success with such a demanding schedule, and we were proud to be able to help Mozilla test and improve Firefox 4 along the way.

Beyond the technical aspects of the demo, it also points to the spirit of how these technologies are meant to be used. We worked as a distributed team during evenings and on weekends, to plan and code and create everything, from the tools we needed to the graphical resources to the demo’s final code. Some of our team are browser developers, some web and audio hackers, others are graphic designers or filmmakers, still others storytellers and writers–everyone had a place around the table, and a role to play. We think this is part of what makes the web such a powerful platform for creative and collaborative work: there isn’t one right way to be, no single technology you need to know, and the techniques and tools are democratized and open to anyone willing to pick them up. The web that Firefox 4 makes possible is a web ready for artists, developers, filmmakers, and musicians alike.

Greg Jopa, an Illinois State University grad student studying web development, built a web-based guitar tab player using Firefox’s Audio Data API and Vexflow (HTML5 music notation rendering API). Here is some details from Greg. You can also read more about this experiment on his blog.

I created a mashup using the Firefox 4 Audio Data API, MusicXML, and Vexflow to create a HTML 5 Guitar Tab Player.

This is a Youtube video. It uses the HTML5 <video> tag if you activate it here.

• View demo.

• Source code.

Using JavaScript, this guitar tab player converts the music note data from a MusicXML file to frequencies playable by the Audio Data API. This note data is also converted to a format that can be read by Vexflow to display the guitar tab.

I have broken down this functionality into 4 steps. Here’s how the HTML5 Guitar Tab Player works:

MusicXML

When a tab (a tablature) is selected, the jQuery ajax() function loads the MusicXML file, parses through the contents, and appends all the formatted note data to a <script> tag. The format for each note is: playNote(note, octave, duration).

playNote(notes.C, 4, 1);

The tablature notation information in the MusicXML file is also processed and appended to a <div> tag for Vexflow.

Here is the structure of the nodes I am using from the MusicXML file:

<measure number="1">
    <attributes>
        <divisions>1024</divisions>
    </attributes>
    <note>
        <pitch>
            <step>C</step>
            <octave>4</octave>
        </pitch>
        <duration>1024</duration>
        <notations>
            <technical>
                <fret>1</fret>
                <string>2</string>
            </technical>
        </notations>
    </note>
    the note node repeats for the rest of the notes in the measure…
</measure>

And here is the generated source that is produced by the ajax() function.

For the audio:

<script id="audio" type="text/javascript">
	function loadAudio(){
	  playNote(notes.C, 4, 1);
	  playNote(notes.C, 4, 1);
	  playNote(notes.G, 4, 1);
	  playNote(notes.G, 4, 1);
	  …
	}
</script>

For the tab:

<div id="tab" class="vex-tabdiv" width=780 scale=1.0 editor="false">
tabstave notation=true
	notes :q 1/2 :q 1/2 :q 3/1 :q 3/1 | :q 5/1
   …
</div>

Frequency Calculation

When the Play button is clicked the music data <script id=”audio”> tag is evaluated and the note information is converted to frequencies using the following formula (assumes equal temperament):

frequency = 440 * 2^(n/12)

440 is the frequency for the note A at the 4th octave.

n = distance away from A4 (440)

I am using an array to store the distance between a certain note and the base note “A”

notes = {C:-9, Cs:-8, D:-7, Ds:-6, E:-5, F:-4, Fs:-3, G:-2, Gs:-1, A:0, As:1, B:2};

And to incorporate that a note can be played at different octaves I have adapted the above formula to the following:

frequency = 440 * 2^((octave-4) * 12 + distance)/12);

Audiodata.js

I am using the Audiodata.js library in the guitar tab player which makes it easy to play continuous audio using a single array. This library encapsulates the Audio Data API methods. Audiodata.js is available on GitHub here: https://github.com/notmasteryet/audiodata. The Audiodata.js project contains a score example with “Twinkle, Twinkle, Little Star” which inspired me to build this guitar tab player.

Vexflow and the Animated Cursor

Vexflow is an open-source web-based music notation rendering API based on the HTML5 canvas element (http://www.vexflow.com/). I am using Vexflow to display the tab for each MusicXML file. I have added an additional canvas element on top of the Vexflow canvas to control the red cursor that links the audio to the tablature. The speed of the audio is controlled by the tempo which is measured in beats per minute. By converting this tempo to milliseconds I am able to use it for the redraw speed of the second canvas. Every time this canvas is redrawn the red cursor is moved 5 pixels to the right to highlight which note is currently being played.
Keep in mind that this HTML5 Guitar Tab Player is just a proof of concept and can only play single notes.

If you have recommendations on how to make this tab player better or would like to contribute to this project check out the following post: http://www.gregjopa.com/2010/12/html5-guitar-tab-player-with-firefox-audio-data-api/

The new Firefox 4 Beta is here, and comes with WebGL activated by default. You can download this new beta here: http://firefox.com/beta.

Flight of the Navigator is a WebGL + Audio API demo developed by a team of Mozilla volunteers.

You can see the demo online here (you need a WebGL compatible browser).

More information about this demo on David Humphrey’s blog (one of the developer of this demo) and more about WebGL in Firefox Beta8 on Vlad’s blog.

The screencast:

I’ve always been a big fan of the travel/flight sequences in the Indiana Jones movies and judging by the amount of copy attempts on YouTube I am not alone in this. As I don’t own any video editing software I thought it should be possible to create the same effect with web technologies and Google Maps and lo and behold it is:

See the demo online

You can download the animation demo for yourself and try it out locally – all you need is a browser that supports HTML5 video. I know – the music is not quite the same as in the movies, but at least this one is not copyright infringing and it came from the heart (5 minutes in a meeting room in the Mozilla office).

So how was this done and what are problems that needed solving? Here’s how and what.

Step 1: Find the movie and get it to the right format

That was the easy part. Archive.org has a lot of awesome public domain movies available for you and they are already in the formats needed to use in an HTML5 video element. In this case, I took the short movie of Charles Lindbergh taking off for his record breaking flight from New York to fly to Paris in 1927.

Step 2: Displaying the video

Using the video is pretty simple:

<div id="stage">
  <video>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff_512kb.mp4" 
type='video/mp4; codecs="avc1.42E01E, mp4a.40.2"'>
    <source src="http://www.archive.org/download/
CharlesLindbergTakesOff/CharlesLindbergTakesOff.ogv"
 type='video/ogg; codecs="theora, vorbis"'> 
  </video>
</div>

The MP4 format will be used by Webkit based browsers and the Ogg version by Firefox and others. As we want to control the video we omit the controls attribute on the video element – instead we create a button to play the video with JavaScript:

window.addEventListener('load', 
  function() {
    var stage = document.getElementById('stage');
    var v = document.getElementsByTagName('video')[0];
    but = document.createElement('button');
    but.innerHTML = 'Click to see Lindbergh\'s flight';
    stage.appendChild(but);
    but.addEventListener('click',function(e) {
      v.play();
      e.preventDefault();
    },false);
  }, 
false);

As the video is markup we can do whatever we please with it – the power of open technologies. For example as we will do here we can set its opacity in CSS and put in on top of a map.

Step 3: Create the map path animation

Talking of which, let’s get that moving path done. Google Earth has an API to do that, but it needs a special plugin. Google Maps allows you to paint paths on maps (which actually are SVG, another open standard). Put that in a recursive function and you get the desired effect:

In essence, what I did was take the latitude and longitude of the start and end points and calculate as many points in between the two as I need for the duration of the animation. I store the points in an array called pos and then paint a path from the start to the current point and move the map centre to this point on every iteration.

spirit.draw = function(){
  var path = new google.maps.Polyline({
        path: [startpos,pos[now]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
  });
  path.setMap(map);
  map.panTo(pos[now])
  now = now + 1;
  if(now < animationend-1){
    setTimeout(spirit.draw,200);
  }
}

Check the highly commented source of the map example for the details. Now, we could use this kind of animation and play the video over it – the problem though is that they may get out of sync. When the movie stalls (as it frequently does on this hotel wireless connection) we don’t want the animation to keep moving, right?

Step 4: Syncing video and the map movement

Instead of having two sources of timing information we have to limit ourselves to one source of truth. This is the time stamp of the currently playing movie.

By the way – you might have noticed that I wrapped the map code in a tilesloaded event handler. This is another safeguard for keeping things in sync. I found that on slow connections the tile loading can delay the whole interface immensely (because of all the subdomain lookups), so I make the whole interface dependent on the loading of the map and only proceed when the tiles have finished loading. As the tilesloaded event also fires when the map pans we need to use a boolean to stop it from starting the effect several times:

google.maps.event.addListener(map,'tilesloaded',function(){
  if(played === false){
    // [...other code...]
    played = true;
  }
});

You can read the current timestamp of a video with video.currentTime and whilst the movie is playing it constantly fires an event called timeupdate. As the event fires a lot we need to throttle it somehow. The trick here is to only take the full seconds and increase a counter when a new second is reached. You can see the timestamp and the second interval firing in the video syncing demo:

var now = 0;
v.addEventListener('timeupdate',function(o){
  log.innerHTML = v.currentTime; /* logging the real timestamp */
  var full = parseInt(v.currentTime);
  if(full >= now) {
    seqlog.innerHTML = now;  /* logging the seconds firing */
    now = now + 1;
  }
},false);

That way the movie can lag in between and the sequence still stays in sync. Check the source of this demo on Github.

Putting it all together

And that was about it – all I had to do is to set the movie’s opacity at a certain time stamp, start the sound at another and show and hide the copyright complaint at another. As we rely on the timestamp for the other effects we needed a boolean switch to avoid repeated firing:

v.addEventListener('timeupdate',function(o){
  full = parseInt(v.currentTime);
  if(full === now-1){
    mapelm.style.opacity = .8;
    v.style.opacity = .4;
  }
  if(full === animationstart+1 && audioplay === false){
    a.play();
    audioplay = true;
  }
  if(full === animationstart+2 && hidden === true){
    drmbedamned.style.display = 'block';
    hidden = false;
  }
  if(full === animationstart+3 && hidden === false){
    drmbedamned.style.display = 'none';
    hidden = true;
  }
  if(full >= now) {
    path = new google.maps.Polyline({
        path: [startpos,pos[full]],
        strokeColor: "#c00",
        strokeOpacity: .7,
        strokeWeight: 10
    });
    path.setMap(map);
    map.panTo(pos[full])
    now = now + 1;
  }
},false);

Another event we needed to subscribe to was the movie ending so we can stop the music and start to roll the credits:

v.addEventListener('ended',function(o){
  a.pause();
  spirit.credslist.parentNode.style.display = 'block';
  spirit.creds();
},false)

As the theme is too short for the whole animation we need to loop it. This can be done by testing for the ended event and rolling back the time to 0:

a.addEventListener('ended', function(o) {
  a.currentTime = 0;
},false);

Summary

And there you have it – Indiana Jones style maps using open services and open technologies. A workaround for the copyrighted audio (recorded, edited and converted with the free Audacity sound editor) and using Google’s Web Fonts as graphics.

You can now take this and change it for even more awesome:

• Replace Google Maps with Openstreetmap to avoid going over the limit
• Add a slight curve to the path from NYC to Paris to make it more accurate (but then again the time is not accurate either – it took charles a tad longer)
• Use a static map and paint the path with Canvas to speed up and smoothen the animation

Why not have a go – it is free and fun to play.

At BarCamp Bankgkok 4, 16 year-old programmer, Thai Pangsakulyanont, did an awesome presentation (that I now know he hacked together in 2 hours before the presentation) on the Audio Data API in Firefox 4: audio generation demo.

Simply, the demo uses the mouse button for a crash cymbal, mouse actions as an arpeggiator, and the accelerometer (Orientation events) to apply a low-pass filter. I remember Thai from BarCamp Bangkok 2 where he made a Wiimote demo with Linux (and he was 14 then.)

If you also have a demo and some code to share, please let us know: @mozhacks (twitter account) or leave us a comment.