chore: repository cleanup — line endings, dangling doc references, module split, hygiene

[snchimata]

Repository cleanup: no behaviour change, no public API change, no shipped capability removed. All ten workspace crates still build and test.

Each commit is self-contained and reviewable on its own; they are ordered so the tree is green at every point.

Why this starts with .gitattributes

The test suite was red in any fresh Windows clone before this branch — and had been silently. Git for Windows sets core.autocrlf=true system-wide, the repo had no .gitattributes, and the golden fixtures are stored as LF blobs. A clean checkout smudged them to CRLF while the transforms emit LF, and tests/golden.rs compares raw bytes with no normalization, so 3 of 7 golden tests failed. The working tree passed only by accident of how those files happened to be materialized.

18d1869 pins * text=auto eol=lf. Verified by cloning the resulting commit: 7/7 golden, 0 CR bytes across all tracked files. git add --renormalize stages zero content changes, because all 224 tracked text blobs were already LF in the index.

Nothing downstream in this branch was verifiable until that landed.

Dangling documentation references

docs/ has no tracked files — the whole docs/solution-design/ tree is gitignored — yet tracked source cited it 288 times across 50 files, along with opaque F-/D-/R- identifiers defined only inside those documents. Every one of those was a dead pointer for anyone but the author.

Two escaped the repository entirely:

• crates/tokenfold-cli/src/main.rs — shipped verbatim in tokenfold compress --help
• crates/tokenfold-core/src/budget.rs — published rustdoc on docs.rs for tokenfold-core 0.4.0

Each citation is replaced with the behaviour or contract it pointed at, read off the code rather than the spec. Where a tracked artifact enforces the same thing, the comment now points there instead — golden fixtures, the test that pins exit codes, README sections.

2ae4d5c adds a CI gate so the class cannot come back. It is tested in both directions: it passes on this tree, and it fires on an introduced ROADMAP.md F-999 probe. CHANGELOG.md is exempt as released history, and eval/tasks/*.json are exempt because those byte-exact fixtures contain synthetic id-shaped strings as test data.

Scope note: an initial pass matching only docs/solution-design and [FDR]-[0-9]{3} missed ~48 further references written as INTERFACES.md, ENGINEERING.md, ROADMAP.md, PLAN.md. The gate matches both spellings plus every lowercase bare form.

Splitting pipeline.rs

2301 lines becomes 1011 of implementation and 1290 of tests, as two commits on purpose.

A single-commit split silently reassigns the file's history to the test file: at 1293/2301 = 56% similarity git pairs the rename with tests.rs, while mod.rs at 43.8% falls below the threshold. The implementation would lose its blame lineage to the tests. Doing the git mv alone first (rename ... (100%), zero content hunks) and extracting second avoids that.

Result: git log --follow pipeline/mod.rs returns 9 commits, tests.rs returns 1.

The test module is declared mod tests; — not pub mod — inside pipeline/mod.rs. Both details are load-bearing. Declaring it in lib.rs instead fails with 52 name-resolution errors, because the tests reach parent-private imports through use super::*. pub mod would have added tokenfold_core::pipeline::tests to a published crate's API.

lib.rs is byte-identical. pipeline/mod.rs still exposes exactly compress and compress_with_estimator.

Three test lines are rejoined by rustfmt: dedenting by one level frees four columns, so expressions that had to wrap now fit. Formatting only.

Hygiene

.gitignore covered only part of what this tree generates. node_modules/ and dist/ were ignored solely by packages/tokenfold/.gitignore; .pytest_cache/ only by pytest's own generated file; and .env was a bare literal, so .env.local and .env.production were not ignored at all. Adds those plus .venv/, venv/, *.egg-info/, /data/, and moves the .codex/ and CLAUDE.local.md rules out of machine-local .git/info/exclude so they are portable. Verified that no currently-tracked file becomes ignored.

Secret scanning did not exist, in a repo that publishes to four registries — a leaked token is a four-registry problem. Adds gitleaks to the security job, scanning full history (fetch-depth: 0), since a tip-only scan would miss exactly the case that matters. .gitleaks.toml allowlists the secret-shaped strings this repo contains deliberately — the redaction transform's tests must feed it realistic credentials — scoped to specific literals and paths so a real credential in those files is still caught.

Fixture provenance. eval/tasks/v04/README.md described its corpus as tier A | B | C with Training + evaluation use. These are held-out evaluation fixtures; nothing is trained from them, so the tier table read as a training-data governance record for something that is not training data. Replaced with a provenance statement — project-owned synthetic material only — written as a precondition for adding fixtures rather than an observation about the current contents.

Also corrected

Two stale comments found along the way, both asserting the opposite of the code:

• transforms/diff.rs claimed the module was not wired into the crate. lib.rs:13 has pub mod transforms;.
• tokenfold-admin claimed a tokenfold update subcommand wires its primitives. No such subcommand exists in the shipped binary.

Verification

Run against a fresh clone of this branch, not the working tree:

• cargo test --workspace --locked — 409 passed, 0 failed
• cargo fmt --all --check — clean
• cargo clippy --workspace --all-targets -- -D warnings — clean
• python eval/run_fidelity.py --gate --profile smoke-first-consumer — pass
• 0 CR bytes across all tracked files; zero dangling references
• ci.yml and release.yml parse with all 8 and 9 jobs intact

Reviewer notes

• The MCP tools/list store_originals description text changed. Nothing in the workspace asserts on it (verified repo-wide, and by running the server and parsing the response), but an external MCP client string-matching that description would notice.
• tokenfold mcp --help and tokenfold filters verify --help print different text. No help-text assertions exist anywhere in the workspace.
• Beyond comments, exactly five string literals changed — the MCP description, two test assertion messages, and the runtime Err() messages from vector::embed and ocr_summarize, all of which carried dead identifiers into user-facing output. Nothing asserts on any of them; no should_panic exists in the repo.
• CHANGELOG.md is untouched, as is every eval/tasks/**/*.json fixture.

[snchimata]

Added 5ca5633 to unblock CI: node-api was failing on both Node versions, and it is not a regression from this branch.

The five @tokenfold/cli-* optionalDependencies were recorded in package-lock.json with no version while package.json requires 0.4.0 — a leftover from the v0.4.0 bump, when those platform packages were not yet published so npm could not resolve them. npm tolerated it until Node 24's bundled npm tightened npm ci's sync check.

Confirmed pre-existing rather than assumed: this branch does not touch packages/ at all (empty diff against main), and re-running the node-api (Node 24) job from main's own last green run fails today on unmodified main code.

Regenerated with npm install --package-lock-only; all five now resolve with integrity hashes. npm ci succeeds and npm test passes 12/12 locally.