Fix stale doc_sourced reference and enforce provenance consistency

[leggetter]

Two small follow-ups after #26 and #25 merged.

The README still referenced doc_sourced

It told contributors that a provider whose samples are all doc-sourced carries doc_sourced: true in its index.json. Nothing has done that since provenance replaced the flag — no index.json sets it, and no code reads it. Replaced with what actually marks those versions:

"provenance": {
  "latest": { "sourced_via": "docs", "sourced_on": "2026-07-29" }
}

The two provenance records could drift

After both PRs, the same fact is recorded twice: the per-file source key that doc-sourced samples carry (1,260 files), and the version-level provenance block (95 providers). They're complementary rather than redundant — per-file travels with the payload and records exactly where each example was read, version-level is what a consumer can filter on without downloading every version file — but nothing kept them in step.

compile.ts now fails the build when a version marked capture contains a file with a source key:

Error: scrapfly/latest is marked sourced_via "capture" but crawler_started.json
carries a "source" key, which only doc-sourced samples have

That's the drift worth catching: doc-sourced samples published under a claim they were captured live. The reverse isn't checked — a docs version legitimately needn't mark every file.

Testing

yarn compile passes on the full tree: 112 providers, 1,260 doc-sourced files, publishing docs: 93, unknown: 21, capture: 2. Planting a source key into a captured version fails as intended, and removing it passes again.

One thing worth a second opinion

This is stricter than the tree it validates, so it's a new way for a build to break. If someone captures fresh samples over a version that was previously docs and flips sourced_via to capture without clearing the old source keys, compile stops rather than warns. That's deliberate — publishing a doc example as captured is the failure mode the whole provenance idea exists to prevent — but @garethx it's your samples-doc/ workflow it would interrupt most, so say if you'd rather it warned.

🤖 Generated with Claude Code

https://claude.ai/code/session_019pce9oHWGjdwHsJNya1ovP

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
webhook-samples	Ready	Preview	Aug 7, 2026 10:57am