Commit generated PDFs, rebuild on dependency changes, and support Markdown documents - #21
Open
CSchank wants to merge 3 commits into
Open
Commit generated PDFs, rebuild on dependency changes, and support Markdown documents#21CSchank wants to merge 3 commits into
CSchank wants to merge 3 commits into
Conversation
Three changes to how documents are built, plus a reduction in build time. Generated PDFs are written to pdfs/ and committed back to main, so they can be read directly in the repository without downloading build artifacts or installing LaTeX. build-pdf/ is seeded from that directory before compiling, which makes the repository rather than the Actions cache the source of truth: previously a cache miss combined with the rsync --delete would have removed every generated PDF. Documents are rebuilt when the files they read change. Only documents whose own file changed were rebuilt before, and because the shared includes are named .text rather than .tex, editing docs/Common.text, which fourteen documents read, rebuilt nothing and still reported success. refs/References.bib, read by eight documents, could not even start the workflow. Rather than maintain a list of dependencies by hand, pdflatex now runs with -recorder and record_deps.py reads the resulting .fls, which reports every file the document opened, including figures and transitively included files. .bib files come from the \bibdata entries in the .aux, since pdflatex reads the generated .bbl and never the bibliography source. select_docs.py reverses that graph to choose what to rebuild, and falls back to a full rebuild for any changed file it cannot account for, so a missing record makes a build slower rather than wrong. Markdown documents under docs/ are compiled with pandoc, using the pdflatex already installed for the .tex rule. README.md and Expectations*.md are excluded as instructor-owned. Where a folder holds both Foo.md and Foo.tex the Makefile lists the Markdown rule first, so Markdown wins. TeX Live and pandoc are cached rather than reinstalled each run, which takes that step from about 126 seconds to about 30. Almost none of that time was downloading, so the installed files are cached rather than the .deb archives. Restoring files skips their postinst scripts, so mktexlsr and updmap-sys are run afterwards to rebuild the filename and font map databases, and a short verification step compiles a small Markdown file before any real document is built. The actions GitHub reported as running on the deprecated Node 20 runtime are updated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A reader had no way to tell whether a PDF reflected the current source. Each entry now shows when its PDF was last committed, and the page records when it was generated. The times come from git rather than file mtimes, because the PDFs are restored from a cache or checked out fresh and their mtimes are the time of the run. They are written as UTC in a <time> element and rewritten by a small script into the reader's own timezone, falling back to the UTC text when JavaScript is unavailable. Generating the list moves from awk to a script, which also fixes the markup: the placeholder sat inside a <ul>, so the generated headings and lists were nested inside a list item. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A pull request runs with a read-only token, so the deploy job failed on every pull request, including ones from forks that cannot deploy by definition. Restrict it to pushes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Hi @smiths, what are your thoughts on this? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This started as a fix for one problem and grew into four related changes to
the PDF build. Happy to split it up, drop parts of it, or rework anything —
feedback very welcome, particularly on the first and third items, which are
the most opinionated.
What this changes
Generated PDFs are committed to the repository.
Every compiled PDF is written to a
pdfs/directory and committed back tomain, so the documents can be read directly in the repository withoutdownloading build artifacts or installing LaTeX. The published site is
unchanged; this adds an in-repository copy alongside it.
Documents are rebuilt when the files they depend on change.
Previously only a document whose own file changed was rebuilt. Because the
shared includes are named
.textrather than.tex, editingdocs/Common.text— which fourteen documents read — rebuilt nothing at alland still reported success, and
refs/References.bib, read by eightdocuments, could not even start the workflow. Rather than maintain a list of
dependencies by hand,
pdflatexnow runs with-recorderand the buildreads the resulting
.fls, which reports every file the document actuallyopened. That picks up figures and transitively included files for free.
Deliverables can be written in Markdown.
Any
.mdfile underdocs/is compiled to PDF with pandoc, apart fromREADME.mdandExpectations*.md, which are instructor-owned. Where afolder contains both
Foo.mdandFoo.tex, the Markdown version is used.Builds are faster, and the index page shows timestamps.
TeX Live and pandoc are cached rather than reinstalled on every run, taking
that step from about 126 seconds to about 30. The actions GitHub flagged as
running on the deprecated Node 20 runtime are updated. Each PDF on the index
page now shows when it was last updated, rendered in the reader's own
timezone. The deploy job is also restricted to pushes, since a pull request
has a read-only token and could never deploy.
Please treat the Markdown support as experimental
It has compiled exactly one test document. Pandoc's LaTeX template requires
packages the
.texdocuments never use —lmodernwas missing on the firstattempt — so a team using Markdown features that have not been exercised,
images in particular, may hit a missing package. That surfaces as a failed
build with a clear LaTeX error rather than a quietly wrong PDF.
Markdown dependency tracking is also weaker than the LaTeX equivalent, since
pandoc discards the
.flsand those dependencies come from scanning thesource instead. Anything missed falls back to rebuilding everything, so the
result stays correct, just less precise. There is one known gap on the LaTeX
side too:
.flsrecords what was read on a given run, so a conditional\inputonly registers the branch actually taken.More generally, the dependency rule is deliberately fail-safe. A changed file
that is neither a document, nor a recorded dependency, nor explicitly
ignorable triggers a full rebuild, so a missing record makes a build slower
rather than wrong.
What is not in this pull request
The generated
pdfs/directory and the dependency manifest are left out, tokeep the diff reviewable. Both regenerate on the first run: with no manifest
recorded, every document is rebuilt.
How it was tested
Everything here has been running on
CSchank/capTemplate, where the site ispublished at https://cschank.github.io/capTemplate/ with all 29 documents.
This branch was also run as a pull request first, to confirm the build passes
and the deploy job is correctly skipped.
Worth noting that the Markdown support failed twice before it worked, once on
a missing
lmodernand once on font maps that the package cache had notrebuilt, so the verification step now compiles a small Markdown file before
any real document is built.
🤖 Generated with Claude Code