Mozilla Hacks

@media, MathML, and Django 1.11: MDN Changelog for May 2018

By John Whitlock

Posted on June 8, 2018 in MDN

Editor’s note: A changelog is “a log or record of all notable changes made to a project. [It] usually includes records of changes such as bug fixes, new features, etc.” Publishing a changelog is kind of a tradition in open source, and a long-time practice on the web. We thought readers of Hacks and folks who use and contribute to MDN Web Docs would be interested in learning more about the work of the MDN engineering team, and the impact they have in a given month. We’ll also introduce code contribution opportunities, interesting projects, and new ways to participate.

Done in May

Here’s what happened in May to the code, data, and tools that support MDN Web Docs:

We’ll continue this work in June.

Migrated CSS @media and MathML compat data

The browser compatibility migration continued, jumping from 72% to 80% complete. Daniel D. Beck completed the CSS @media rule features, by converting data (PR 2087 and a half-dozen others) and by reviewing PRs like 1977 from Nathan Cook. Mark Boas finished up MathML, submitting 26 pull requests.

The first few entries of the CSS @media browser compatiblity table

The @media table has 32 features. It’s a big one.

There are over 8500 features in the BCD dataset, and half of the remaining work is already submitted as pull requests. We’re making steady progress toward completing the migration effort.

Prepared for Django 1.11

MDN runs on Django 1.8, the second long-term support (LTS) release. In May, we updated the code so that MDN’s test suite and core processes work on Django 1.8, 1.9, 1.10, and 1.11. We didn’t make it by the end of May, but by the time this report is published, we’ll be on Django 1.11 in production.

Many Mozilla web projects use Django, and most stick to the LTS releases. In 2015, the MDN team upgraded from Django 1.4 LTS to 1.7 in PR 3073, which weighed in at 223 commits, 12,000 files, 1.2 million lines of code, and at least six months of effort. Django 1.8 followed six months later in PR 3525, a lighter change of 30 commits, 2,600 files, and 250,000 lines of code. After this effort, the team was happy to stick with 1.8 for a while.

Most of the upgrade pain was specific to Kuma. The massive file counts were due to importing libraries as git submodules, a common practice at the time for security, reliability, and customization. It was also harder to update libraries, which meant updates were often delayed until necessary. The 1.7 upgrade included an update from Python 2.6 to 2.7, and it is challenging to support dual-Python installs on a single web server.

The MDN team worked within some of these constraints, and tackled others. They re-implemented the development VM setup scripts in Ansible, but kept Puppet for production. They updated submodules for the 1.7 effort, and switched to hashed requirements files for 1.8. A lot of codebase and process improvements were gained during the update effort. Other improvements, such as Dockerized environments, were added later to avoid issues in the next big update.

Django 1.4 wasn’t originally planned as an LTS release, but instead was scheduled for retirement after the 1.6 release. The end of support was repeatedly pushed further into the future, finally extending to six months after 1.8 was published.

On the other hand, the Django team knew that 1.8 was an LTS release when they shipped it, and would get security updates for three years. Many of the Django team’s decisions in 2015 made MDN’s update easier. Django 1.11 retained Python 2.7 support, avoiding the pain of a simultaneous Python upgrade. Django now has a predictable release process, which is great for website and library maintainers.

The Django supported versions table shows what releases are supported, and when they are supported. There's a new release each 8 months, regular releases are supported for 16, and LTS release for 3 years.

Django’s release roadmap makes scheduling update efforts easier.

Django also maintains a guide on updating Django, and the suggested process worked well for us.

The first step was to upgrade third-party libraries, which we’ve been working on for the past two years. Our goal was to get a library update into most production pushes, and we updated dozens of libraries while shipping unrelated features. Some libraries, such as django-pipeline, didn’t support Django 1.11, so we had to update them ourselves. In other cases, like django-tidings, we also had to take over project maintenance for a while.

The next step was to turn on Django’s deprecation warnings, to find the known update issues. This highlighted a few more libraries to update, and also the big changes needed in our code. One problem area was our language-prefixed URLs. MDN uses Mozilla-standard language codes, such as /en-US/, where the region is in capital letters. Django ships a locale-prefixed URL framework, but uses lowercase language codes, such as /en-us/. Mozilla’s custom framework broke in 1.9, and we tried a few approaches before copying Django’s framework and making adjustments for Mozilla-style language codes (PR 4790).

Next we ran our automated tests, and started fixing the many failures. Our goal was to support multiple Django versions in a single codebase, and we added optional builds to TravisCI to run tests on 1.9, 1.10, and 1.11 (PR 4806). We split the fixes into over 50 pull requests, and tested small batches of changes by deploying to production. In some cases, the same code works across all four versions. In other cases, we switched between code paths based on the Django version. Updates for Django 1.9 were about 90% of the upgrade effort.

This incremental approach is safer than a massive PR, and avoids issues with keeping a long-running branch up to date. It does make it harder to estimate the scope of a change. Looking at the commits that mention the update bugs, we changed 2500 to 3000 lines, which represents 10% of the project code.

Started tracking work in ZenHub

For the past two years, the MDN team had been using Taiga to plan and track our work in 3-week sprints. The team was unhappy with performance issues, and had been experimenting with GitHub’s project management features. Janet Swisher and Eric Shepherd led an effort to explore alternatives. In May, we started using ZenHub, which provides a sprint planning layer on top of GitHub issues and milestones.

The ZenHub board collects task cards into columns, that mive to the right as they approach completion.

See how the ‘sausage’ gets made with our ZenHub board.

If you have the ZenHub plugin installed, you can view the sprint board in the new mdn/sprints repository, which collects tasks across our 10 active repositories. It adds additional data to GitHub issues and pull requests, linking them into project planning. If you don’t have the plugin, you can view the sprint board on the ZenHub website. If you don’t want to sign in to ZenHub with your GitHub account, you can view the milestones in the individual projects, like the Sprint 4 milestone in the Interactive Examples project.

Continued HTML Interactive Examples

We’re still working through the technical challenges of shipping HTML interactive examples. One of the blocking issues was restricting style changes to the demo output on the right, but not the example code on the left. This is a use case for Shadow DOM, which can restrict styles to a subset of elements. Schalk Neethling shipped a solution based on Shadow DOM in PR 873 and PR 927, and it works natively in Chrome. Not all current browsers support Shadow DOM, so Schalk added a shim in PR 894. There are some other issues that we’ll continue to fix before including tabbed HTML examples on MDN.

Meanwhile, contributors from the global MDN community have written some interesting demos for <track> (Leonard Lee, PR 940), <map> (Adilson Sandoval, PR 931), and <blockquote> (Florian Scholz, PR 906). We’re excited to get these and the other examples on MDN.

On the left, HTML defines a <map> of polygon targets, and on the right the resulting image with link targets is displayed.

A demo of the polygon targets for <map>.

Shipped Tweaks and Fixes

There were 397 PRs merged in May:

60 of these were from first-time contributors:

Other significant PRs:

Planned for June

We plan to ship Django 1.11 to production in early June, even before this post is published. We’ll spend some additional time in June fixing any issues that MDN’s millions of visitors discover, and upgrading the code to take advantage of some of the new features.

We plan to keep working on the compatibility data migration, HTML interactive examples, and other in-progress projects. June is the end of the second quarter, and is a deadline for many quarterly goals.

We are gathering in San Francisco for Mozilla’s All-Hands. It will be a chance for the remote team to be together in the same room, to celebrate the year’s accomplishments to date, and to make plans for the rest of the year.

About John Whitlock

John is a web developer working on the engine of MDN Web Docs

More articles by John Whitlock…

One comment

  1. John Whitlock

    The original post said that we had problems getting the Shadow DOM shim working. That was based on old information, and Schalk shipped the working shim early in May. I’ve updated the post.

    June 8th, 2018 at 11:16

Comments are closed for this article.