Thanks for sharing the slides, Nathan. Looks really complex! I wish there was a simple benchmark that would be used so everyone can understand the comparisons, e.g. how long does it take to encode a 4k video using the different video codecs? The other benchmark we can all understand is how do the file sizes compare for each codec when encoding an original 1 hour 4k video. I think it would be most helpful to include these benchmarks for the larger public. Thanks again for your contribution.

1. Nathan Egge

   Indeed, you are right. Testing video and image coding is complicated, and probably a good topic for another talk.

   Let me attempt to answer your very reasonable question of why there isn’t “a simple benchmark that would be used so everyone can understand the comparisons”. The short answer is that nobody can agree on what that one benchmark would be, and for good reason.

   You mention “file sizes”, i.e. rate, but the implicit assumption is “rate at the same quality”. How do we measure quality? There are at least a dozen different objective (deterministic) quality metrics, but even the experts agree that they are often biased, disagree with each other, and don’t correlate well with human perception. For example a favorite objective metric, mean squared error a.k.a. PSNR, is generally useful except that you can “improve” PSNR just by blurring an image. Then there is the question of how you come to a single objective “quality” value for an individual test clip. The way you combine objective metrics matters, e.g., based on: region of interest (across the frame), human perception (across color planes), coding unit (across frame types), average stream size (across rate controlled scenes), etc.

   You mention “how long … to encode”, i.e., complexity, and here the issue is that modern video codecs are designed to be used at a wide range of operating points. The algorithms used for video-on-demand (high latency), live streaming (some latency) and interactive video conferencing (low latency) are often different and so what you really want to compare is “quality for a given rate and complexity budget”. You can do this, but it takes time and is only fair if you really ensure it is an apples-to-apples comparison, i.e., you need to define the test conditions well, but also ensure that the encoders are configured properly.

   This points to the other big problem which is that you wanted to compare video formats, e.g., “AV1, HEVC, H.264 and VP9”, but what you are really comparing are encoders, e.g., libaom, HM, OpenH264 and libvpx. These are a mix of reference (verification model) and production (enterprise quality) software and have different levels of maturity, usability and feature completeness. Add to it the implicit biases and level of effort it takes to configure software you are familiar with and that of the alternative, and you can see why it is hard to get consistent evaluations of video and image formats.

   November 8th, 2018 at 15:00