By
Editor’s Note: An earlier version of this post appeared on Mason Chang’s personal blog.
For the past few months, I’ve been working on Project Silk which improves smoothness across the browser. Very much like Project Butter for Android, part of it is finally live on Firefox OS. Silk does three things:
- Align Painting with hardware vsync
- Resample touch input events based on hardware vsync
- Align composites with hardware vsync
What is vsync, why vsync, and why does it matter at all?
Vertical synchronization (vsync) occurs when the hardware display shows a new frame on the screen. This rate is set by specific hardware, but major displays in the US occur at a rate of 60 times a second, or every 16.6 ms (milliseconds). This is where you hear about 60 frames per second, one frame each time the hardware display refreshes. What this means in reality is that no matter how many frames are produced in software, the hardware display will still only show at most 60 unique frames per second.
Currently in Firefox, we mimic 60 frames per second and therefore vsync with a software timer that schedules rendering every 16.6 ms. However, the software scheduler has two problems: (a) it’s noisy and (b) it can be scheduled at bad times relative to vsync.
In regards to noise, software timers are much noisier than hardware timers. This creates micro-jank for a number of reasons. First, many animations are keyed off timestamps that are generated by the software scheduler to update the position of the animation. If you’ve ever used requestAnimationFrame, you get a timestamp from a software timer. If you want smooth animations, the timestamp provided to requestAnimationFrame should be uniform. Non-uniform timestamps will create non-uniform and janky animations. Here is a graph showing software versus hardware vsync timer uniformity:
Wow! Big improvement with a hardware timer. We get a much more uniform, and therefore smoother, timestamp to key animations off of. So that addresses problem (a), noisy timers in software versus hardware.
With part (b), software timers can be scheduled at bad times relative to vsync. Regardless of what the software does, the hardware display will refresh on its own clock. If our rendering pipeline finishes producing a frame before the next vsync, the display is updated with new content. If we fail to finish producing a frame before the next vsync, the previous frame will be displayed, causing jankiness. Some rendering functions can occur close to vsync and overflow until the next interval. Thus, we actually introduce more potential latency since the frame won’t be displayed on the screen anyway until the next vsync. Let’s look at this in graphic form:
At time 0, we start producing frames. For example, let’s say all frames take a constant time of 10 ms. Our frame budget is 16.6 ms because we only have to finish producing a frame before the next hardware vsync occurs. Since frame 1 is finished 6 ms before the next vsync (time t=16 ms), everything is successful and life is good. The frame is produced in time and the hardware display will be refreshed with the updated content.
Now let’s look at Frame 2. Since software timers are noisy, we start producing a frame 9 ms from the next vsync (time t=32). Since our frame takes 10 ms to produce, we actually finish producing this frame at 1 ms AFTER the next vsync. That means at vsync number 2 (t=32), there is no new frame to display, so the display still shows the previous frame. In addition, the frame just produced won’t be shown until vsync 3 (t=48), because that’s when the hardware updates itself. This creates jank since now the display will have skipped one frame and will try to catch up in the upcoming frames. This also produces one extra frame of latency, which is terrible for games.
Vsync addresses both of these problems since we get a much more uniform timer and the maximum amount of frame budget time to produce a new frame. Now that we know what vsync is, we can finally go on to what Project Silk is and how it helps create smooth experiences in Firefox.
The Rendering Pipeline
In super simplified terms, Gecko’s rendering pipeline does three things:
- Paint / draw the new frame on the main thread.
- Send the updated content to the Compositor via a LayerTransaction.
- Composite the new content.
In an ideal world, we’d be able to do all three steps within 16.6 ms, but that’s not the case most of the time. Both steps (1) and (3) occur on independent software timers. Thus, there is no real synchronizing clock between the three steps, they are all ad hoc. They also have no relation to vsync, so the timing of the pipeline isn’t related to when the display actually updates the screen with content. With Silk, we replace both independent software timers with the hardware vsync timer. For our purposes, (2) doesn’t really affect the outcome, but is presented here for completeness.
Align Painting with Hardware Vsync
Aligning the timer used to tick the refresh driver with vsync creates smoothness in a couple of ways. First, many animations are still done on the main thread, which means any animation using timestamps to set the position of an animation should be smoother. This includes requestAnimationFrame animations! The other nice thing is that we now have a very strict ordering of when rendering is kicked off. Instead of (1) and (3), which occur at separate synched offsets, we start rendering at a specific time.
Resample Touch Input Events Based on Vsync
With Silk, we can enable touch resampling, which improves smoothness while tracking your finger. Since I’ve already blogged about touch resampling quite a bit, I’ll keep this short. With Silk, we can finally enable it!
Align Composites with Hardware Vsync
Finally, the last part of Silk is about aligning composites with hardware vsync. Compositing takes all the painted content and merges it together to create the single image you see on the display. With Silk, all composites start right after a hardware vsync occurs. This has actually produced a rather nice side benefit — the reduced composite times seen here:
Within the device driver on a Flame device, there’s a global lock that’s grabbed when close to vsync intervals. This lock can take 5-6 ms to get, greatly increasing the composite times. However, when we start a composite right after a vsync, there is little contention to grab the lock. Thus we can shave off the wait, therefore reducing composite times quite a bit. Not only do we get smoother animations, but also reduced composite times, and therefore better battery life. What a nice win!
With all three pieces, we now have a nice strict ordering of the rendering pipeline. We paint and send the updated content to the Compositor within 16.6 ms. At the next vsync, we composite the updated content. At the vsync after that, the frame should have gone through the rendering pipeline and will be displayed on the screen. Keeping this order reduces jank because we reduce the chance that the timers will schedule each step at a bad time. In a best-case scenario in the current implementation without Silk, a frame could be painted and composited within a single 16.6 ms frame. This is great. However, if the next frame takes 2 frames instead, we’ve just created extra jank, even though no stage in the pipeline was really slow. Aligning the whole pipeline to create a strict sequence of events reduces the chance that we mis-schedule a frame.
Here’s a picture of the rendering pipeline without Silk. We have Composites (3) at the bottom of this profile. We have painting (1) in the middle, where you see Styles, Reflow, Displaylist, and Rasterize. We have Vsync, represented by those small orange boxes at the top. Finally we have Layer Transactions (2) at the bottom. At first, when we start, compositing and painting are not aligned, so animations are at different positions depending on whether they are on the main thread or the compositor thread. Second, we see long composites because the compositor is waiting on a global lock in the device driver. Lastly, it’s difficult to read any ordering or see if there is a problem without deep knowledge of why / when things should be happening.
Here is a picture of the same pipeline with Silk. Composites are a little shorter, and the whole pipeline only starts at vsync intervals. Composite times are reduced because we start composites exactly at vsync intervals. There is a clear ordering of when things should happen. Both composites and painting are keyed off the same timestamp, ensuring smoother animations. Finally, there is a clear indicator that as long as everything finishes before the next Vsync, things will be smooth.
Ultimately, Silk aims to create a smoother experience across Firefox and the Web. Numerous people contributed to the project. Thanks to Jerry Shih, Boris Chou, Jeff Hwang, Mike Lee, Kartikaya Gupta, Benoit Girard, Michael Wu, Ben Turner, and Milan Sreckovic for their help in making Silk happen.
23 comments