Build docs from pylock.toml

[hugovk]

Background

As discussed in the last Docs WG meeting, maintaining the docs builds in older branches can be difficult, because of differing dependencies and version support. And especially for the oldest branches, which are rarely updated, and rarely rebuilt, so we don't see failures.

A docs build is essentially an application: we want to install a given set of deps each time. We have some limited pinning in Doc/requirements.txt and Doc/constraints.txt, but usually just upper or lower bounds, and this doesn't include all transitive dependencies.

A lockfile would help towards reproducible builds.

pip and pylock.toml

The newest pip 26.1 (released yesterday) now comes with experimental support for installing from pylock.toml lockfiles 🎉

pip has had experimental support for generating lockfiles for a year, since 25.1, however:

The produced lockfile is specific to the Python version and platform pip is invoked under. In other words, the lockfile is NOT a universal lockfile.

So if I create a lockfile on macOS, you can't use it on Linux because it only lists macOS wheels. So we need another tool, such as uv or pyenv, to create a universal lockfile.

I created it with:

uv pip compile Doc/requirements.txt \
  --exclude-newer P14D \
  --no-cache \
  --output-file Doc/pylock.toml \
  --python-version 3.12 \
  --universal

And updated the Makefile to consume the lockfile by default instead of requirements.txt.

Testing: the docs build succesfully on macOS with make -C Doc clean venv htmllive, using both pip and uv, and also on Read the Docs (Linux/uv) (build, docs).

Experimental

The pip team make it clear that their pylock.toml support is experimental, and can be modified/deprecated/removed without notice, but also that they need feedback in order to get this production ready.

I think we are well placed to help try this feature out and give feedback. We might not want to backport this all the way back just yet. We could also/instead use this in our other docs sites, such as the devguide, although we don't have as strong a need for a lockfile there.

---

📚 Documentation preview 📚: https://cpython-previews--149058.org.readthedocs.build/

[hugovk]

On hold pending: pypa/packaging#1178

[sbidoul]

It might work if you can afford to remove requires-python from the lock file.

[hugovk]

Thanks! That's working, we can use this in the short term, as I expect tooling will re-add it when relocking.

Trying to install with 3.11 gives this error, which is clear enough:

ERROR: Package 'linklint' requires a different Python: 3.11.9 not in '>=3.12'

[StanFromIreland]

Hmm, I'd prefer to keep our bug fixes, can we exclude certain packages (i.e., first party packages) from the 14 day rule?

[hugovk]

Yes we can with something like:

--exclude-newer-package python-docs-theme=PT0S

Are there any others we'd want to exclude? Perhaps linklint?

[StanFromIreland]

Perhaps linklint?

That seems fine!

[StanFromIreland]

What about make.bat?

And the docs build workflow:

cpython/.github/workflows/reusable-docs.yml

Line 59 in fc829e8

cache-dependency-path: 'Doc/requirements.txt'

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #32456804 | 📁 Comparing be2ff41 against main (276f474)

  🔍 Preview build  

1 file changed

± download.html

[StanFromIreland]

LGTM

[hugovk]

Thanks both!