Pin to dspy 3.3.0, and give the port a CI that is not this laptop - #30
Draft
tosinamuda wants to merge 416 commits into
Draft
Pin to dspy 3.3.0, and give the port a CI that is not this laptop#30tosinamuda wants to merge 416 commits into
tosinamuda wants to merge 416 commits into
Conversation
The doc said a candidate reads its own demo set 'then the ones after and before it', which reads like a heuristic someone chose. It is one upstream expression — [sets[i]] + sets[i+1:] + sets[:i], a left rotation starting at the candidate's own index — and the comment now says so, with dspy's own recorded output as the evidence that it rotates: candidate 4 is shown France, Spain, Germany where candidate 1 is shown France, Germany, Spain.
…hes to see it Mutation baseline for dsrust-gepa: 51 of 369, the largest gap measured. 38 survivors and 13 non-terminating — both counted so the number cannot fall silently as tests get slower, but only survivors are work. Closed the one that mattered most. PyIntSet::intersection decides which operand CPython iterates, and its `>` could be `<`, `==` or `>=` with every test still green. That comparison sits upstream of GEPA's whole draw order, since the set order is what rng.sample(list(set(...)), 2) reads. Two searches, and both say why the gap existed. The size tie is unobservable on most pairs — including the equal-size case already in the fixture — because the result is re-inserted into a fresh table that usually lands on the same order either way; a discriminating pair took a search over half a million. And the tie case alone still left `<` alive, because `<` only differs on unequal sizes: that needed a second search for a pair where iterating the larger operand gives a different order. Both recorded with their reasons, and all three mutants now die. Also: the doc for the demo rotation now names the operation rather than describing it. "Its own set then the ones after and before" reads like a heuristic; it is [sets[i]] + sets[i+1:] + sets[:i], a left rotation, and dspy's own recorded output is the evidence.
Mutation testing the byte-critical adapter path — chat, prompt, exchange, demos,
history, parse — leaves 43 survivors of 143 viable, 35 of them in parse.rs and 20
in next_tag alone. next_tag can have its arithmetic changed almost arbitrarily, or
return Some(("xyzzy","xyzzy","xyzzy")) outright, with 1078 tests green.
The cause is structural, not a scatter of missing cases: nineteen adapter fixtures
record expected_system and expected_turns, and none record a parsed reply. The
claim is that the bytes match dspy's in both directions and only one direction has
an oracle. The parser is exercised only incidentally, by end-to-end tests whose
scripted replies are always well formed — never a marker inside a value, an
unterminated block, a field out of order, or a repeated one. Filed as
parse-side-goldens.
No ratchet entry for dsrust yet: 3619 mutants is about five hours, and a floor
nobody runs is not a gate.
One operational trap found the hard way. This machine points build.build-dir at a
cache shared across projects, which defeats cargo-mutants' isolation: it copies
the tree, but every copy compiles into the same build dir as the real one, and a
mutated rlib gets linked into an ordinary cargo test afterwards. It left xyzzy
inside a BAML prompt assertion with forty-one tests failing and git status clean,
and had spread past dsrust into the shared dependencies. run_mutants.sh now
exports its own CARGO_BUILD_BUILD_DIR.
The machine shares build.build-dir across projects so agent worktrees do not each hold their own copy of the same dependency object code. That is the right default, and it is exactly why cargo-mutants cannot use it: mutants isolates by copying the source tree, so every copy compiles into the same build dir as the real one and leaves a mutated rlib where an ordinary cargo test will link it. run_mutants.sh now exports its own build dir and removes it on exit, and the comment says why the override is affordable rather than just that it exists. build-dir dedupes object files on disk; sccache dedupes the compilation itself and is content-addressed, so a mutated source hashes to its own entry rather than overwriting anyone's. Measured on the tpe run: no net disk, and three minutes against four. Recorded in memory too, since the failure mode is a green git status with forty-one failing tests and nothing pointing at the cause.
Eighteen replies through dspy's ChatAdapter.parse, chosen from the parser's own branches rather than from what a well-behaved model emits, recording what it returns and which five it refuses. Nothing pinned this before: 35 mutation survivors sat in adapter/parse.rs. Two divergences on the first run. An indented marker: dspy matches the header against line.strip() and then slices the *unstripped* line at that match's end, so four spaces of indent cut four characters off what follows and `answer` comes back as "# ]]\nParis". Upstream's off-by-the-indent, reproduced rather than corrected, because a model that indents a marker must get the same bytes from both. And strict JSON in a structured field: the crate kept `["a", "b"]` as the text spelling it, reasoning that a structured field is for the caller's typing to judge. True of text that only might be JSON; not of text that is. A third is a design fork and got its own story. dspy casts scalars while parsing and raises when one will not fit, and ChatAdapter.__call__ answers any exception by re-asking through JSONAdapter — so a bad number switches adapters upstream. Here the cast happens in validation, which Predict::feedback_retry is built on and names in its test. Calling coerce_scalars in parse_markers matches dspy and turns that test red; both cannot stand. Filed as parse-time-casting, with the three affected cases asserted the other way round and counted, so resolving it turns the golden red and says which flag to drop.
…ther than hang Sixteen more cases, built from the properties of <(?P<name>\w+)>(.*?)</\1> under DOTALL rather than from what a model emits: a non-greedy body, a same-name nest, a hyphenated name, an attribute, an unclosed tag, a mismatched close, tags buried in prose, an empty body. Fourteen passed on the first run. The one real gap was the strict-JSON reading the marker path had just been given: parse_tags inserted a tag's body as a raw string rather than through section_value, so a list[str] written as ["a", "b"] came back as the text that spells it. Both paths go through section_value now. Worth separating from the survivor count that motivated this. Twenty of the thirty-five parse.rs survivors were in next_tag, but most were non-terminating — flipping a + to a - in a cursor makes the scan loop forever, which the suite detects by hanging rather than failing. The scanner was largely right. Measured after the 34-case golden: 35 survivors down to 6 missed, 24 non-terminating.
Seventeen more cases, from json_repair.loads followed by the recursive brace
regex: an object in prose or a fence, a nested object, an array holding one, a
trailing comma, single quotes, unquoted keys, a missing brace, Python's literals.
Three genuine differences, all fixed.
A str field handed an object came back as JSON. dspy's parse_value(v, str) is
Python's str(), so it is {'why': 'Because.'} — single quotes, None, True. The
python::repr written last week for the dataset summary is exactly what it needed.
A top-level array was accepted as an answer. dspy asks isinstance(fields, dict)
and falls through to the brace search when it is not, so [{"answer": "Paris"}]
reaches the object inside; the crate took the array and then failed on it.
A structured field handed the text that spells it stayed text. The JSON path uses
the full coerce now rather than coerce_scalars — which the marker and tag paths
cannot, since they receive every field as text and cannot tell JSON from a type's
own spelling, where a JSON reply has already told them apart.
One recorded rather than closed: json_repair recovers an unescaped quote inside a
string and this crate's repair does not. Matching json_repair in full is a
library, not a fix. It is asserted as a disagreement so it cannot quietly start
passing.
The 51 golden cases were each chosen by reading a branch, which is the method that leaves the branches nobody thought of. Both sides are pure functions from a string to a value-or-error with dspy on hand as the reference, so: scripts/fuzz_parse.py generates seeded random replies — mangled markers, malformed tags, broken JSON — runs all three dspy parsers over them and records the answers, and parse_fuzz.rs replays them through the crate and groups every disagreement by adapter and direction. The corpus lives under target/ and is not committed; random strings are evidence, not documentation. Two thousand replies: 226 disagreements, every one of them [json] dspy parsed, we refused. Zero for the marker parser, zero for the tag parser, and zero in the direction that hands a caller a wrong value instead of an error. That measurement is what turns json_repair from a question into json-repair-port, a scoped story — it is an external PyPI library rather than dspy's own code, so there is nothing to copy, and the Rust crates of that name are not that library. The test allows exactly that category and nothing else rather than asserting no disagreements, because a blanket assertion against a known gap either blocks every run or gets turned off. And the control found a hole in the fuzzer itself: breaking first-occurrence-wins on purpose left it green, because rng.sample draws without replacement and no generated reply ever named a field twice. With repeats allowed the same break produces four disagreements outside the known gap.
restrict filters the range the loop already walks, so it narrows and never extends: a seed the search would not have reached is dropped rather than added or refused, and the three baselines get no special treatment. Skipping moves nothing either, since each shuffled attempt seeds its own generator with its own seed — which is what makes it usable for resuming a search rather than a way to get a different answer, and there is a test for that property. labeled_sample reaches one attempt of n+3, seed -2, where it becomes LabeledFewShot's sample. The search had no conformance golden at all, only unit tests. It has one now, recording which seeds dspy actually attempted — read off score_data, so the search is asked to compile and then says what it did rather than having its loop read — plus the demos the labels-only attempt kept under both arms of labeled_sample. The generator refuses to write a case set where no restrict drops an attempt, or where drawing and taking-in-order happen to agree on the trainset. Writing the controls is what made this more than a pass-through. The first labeled_sample control was wrong: flipping LabeledFewShot's sample default left everything green, because the test sets sample explicitly and the default never reads. Replacing the sampling itself with take-in-order is the control that bites. Method-parameter coverage 244/247 to 246/247.
`dspy.JSONAdapter.parse` opens with `json_repair.loads(completion)` — not dspy's code, a separate PyPI package, and the Rust crates of that name are different implementations. So this reproduces the pinned 0.61.7 the way `dsrust-tpe` reproduces optuna's sampler: the whole parser, the entry points, and CPython's own `json` grammar for the fast paths it takes. Three representation decisions are load-bearing. The cursor counts **code points**, because that is what Python indexes and every heuristic here is written in offsets — `get_char_at(-1)` wraps to the end of the input the way a Python index does, and `_should_split_duplicate_object` reaches it with a lookback well below zero. A byte cursor agrees on ASCII and diverges on the first smart quote, which is the input this library exists for. The character predicates are **CPython's**, generated from the interpreter dspy runs on. Rust's own answer differently in both directions: `char::is_whitespace` refuses `\x1c`, which CPython calls a space; `char::is_alphabetic` accepts combining marks, which `str.isalpha()` refuses; and unicode-general-category is Unicode 16.0 where CPython 3.13 is 15.1. An integer keeps every digit. Python's are unbounded, `parse_number` reaches them by reading a run of digits, and narrowing to a machine word would answer a different number. The one thing not reproduced is JSON Schema *validation* — upstream imports `jsonschema`, another separate package, and raises when it is absent. That is a `SchemaValidator` seam here, and with none plugged in the crate answers exactly as an environment without `jsonschema` does. 111 generated cases, and the generator refuses to write a corpus that does not reach the modules it claims to cover — traced through `json_repair` itself, 1074 lines over 15 modules. Five mutations were confirmed to turn it red, including two the first corpus missed.
… first trusted-lm-state said there was nothing to gate because the crate never saves the model. That held — but only after the middle of it was wrong, and the middle is the useful part. Checking turned up what looked like the opposite: a ProgramState deserialised and reserialised does carry an lm block, api_base and all, where dspy's load-then-save strips it. That reads as laundering, and had a sanitising load written for it. Different path. A program load-then-save goes load_state, which restores a signature and its demos, then dump_state, which rebuilds from the predictor — and Predict has nowhere to hold a saved LM dict. The block is dropped whole. Measured after the sanitising change failed its own tests because its premise was false, and reverted. So allow_unsafe_lm_state is a divergence with nothing to select: a redirect can neither be acted on here nor passed onward, which is stricter than upstream rather than looser. tests/lm_state.rs holds both halves against dspy's own round-trip. What it costs is the model pin, which dspy keeps and this loses. Filed as saved-lm-round-trip, with a test asserting today's behaviour that names the story when it changes — it is a decision about carrying inert data on a core type, not a patch. The API surface is 247/247 across all four tables; this was the last todo. Writing the tests also reproduced the fixed-temp-path race one process wide: two tests in the same binary sharing a path and running in parallel, one removing the file as the other read it. The process id does not separate them; the test name does.
… the adapter's could not
`JSONAdapter.parse` opens with `json_repair.loads(completion)`; `parse_json` now does, and falls
back the way upstream does — to the *first balanced* `{…}` run, which is what dspy's
`\{(?:[^{}]|(?R))*\}` finds. Not the span from the first `{` to the last `}`: for
`{"a": 1} and {"b": 2}` the recursive pattern takes the first object alone where the outermost span
took both objects and the prose between them.
`section_value` follows `parse_value`'s order for a non-`str` annotation: json-repair first, and
Python's literal syntax only where json-repair answered with the empty string, which is how it
reports having found nothing. `'a'` is the case that separates them.
`chat_parse` said so itself. `json_unescaped_quote_inside_a_string` was recorded as a divergence
because only json_repair's heuristics recover an unescaped quote inside a string; the assertion
written *the other way round* turned red the moment it stopped diverging, and named the flag to
drop. Three divergences left, all `parse-time-casting`.
The allowance in `parse_fuzz.rs` is gone: the rule is now every disagreement, no category excused.
Zero over seeds 0, 1 and 7 at five thousand replies each, and the control — stubbing out the
balanced-brace search — turns it red with 65.
**And the adapter's fuzzer was not enough.** It reaches this library through one reply shape: no
nesting, no comments, no escapes, no tuples. A grammar that builds malformed JSON directly found
two real disagreements in its first five thousand inputs, both the same defect: when
`_handle_right_delimiter_candidate` breaks, Python breaks with the character it stepped *back*
onto, and this translation broke with the old one — so `finalize_string_result` read the wrong
character, advanced past the quote, and the next key came out as `s` rather than `Paris`. Both are
now named cases in the corpus. Zero over five seeds since, 60,000 inputs.
The other 116 the first run reported were all the declared lone-surrogate divergence, which is
pinned by a named case with a test asserting it. Rediscovering it a hundred times a run buries the
report, so the grammar no longer generates it.
check_api_surface.py walks dspy to Rust and is at 247/247 across four tables.
Nothing walked the other way, so whatever this crate invented was invisible to
every gate here — which is how LM::with_capabilities survived with no callers at
all, not in src, not in tests, not in the docs.
rust_surface.py enumerates the public surface by reachability from each crate
root: an item counts only when every module between it and the root is public
too, so a pub fn in a private module is not API and a pub use of one is. A raw
grep finds about sixteen hundred pub tokens; the surface is 767. It cross-checks
against the names tests/public_api.rs imports and finds all of them.
check_rust_surface.py gates the count as a ratchet rather than a target of zero,
for the reason run_mutants.sh is one: a gate that starts four hundred red either
blocks every run or gets switched off. Verified by adding a public struct and
watching it fail at 446.
The extractor's first version said 138, and that would have been the reported
finding. A guard meant to skip pub(crate) mod tested pub({name} against a ^\s*pub\(
pattern, which matches every module, so the walk never left the crate root. The
total looked plausible; the per-module breakdown showing everything at depth zero
did not.
Seven items classified so far — Steering, both Feedback re-exports, Scoring and
the three interpreter seams. The other 445 are the work: each is a claim about why
this crate has something dspy does not, and none of those has been made.
…chema validator
`schema_repair.py` is the half of the library dspy never reaches, and the half that delegates:
`is_valid` and `validate` are `jsonschema`, a separate package again. So the fixture records every
question the repairer asked it and what it was told, and the Rust test plugs in a validator that
replays that table and **fails on a question that was never asked**. A port that validates a
different value, validates one time fewer, or skips validation is caught; `jsonschema` itself is not
under test, which is the point.
Fifty cases, sixty-four replayed calls, and writing them found three things.
**`_repair_union` catches plain `ValueError`, and `SchemaDefinitionError` is one.** So a subschema
the repairer cannot read is a branch that failed, not an error — unlike inside an array or a list
mapping, which re-raise it explicitly. This had it the other way round.
**The seam needs two kinds of refusal.** `jsonschema` raising `ValidationError` becomes a
`ValueError` and is caught in six places; raising `UnknownType` for `{"type": "date"}` is not a
`ValueError` and is caught nowhere, reaching the caller through `repair_json`'s own `except`. The
trait now says which, and `Error` carries the distinction.
**Two cases were named for what they did not test.** `default_inserted` passed `{}`, which is valid
JSON *and* valid against its schema, so the fast path handed it back without inserting anything;
`one_of_with_a_broken_branch` passed `7`, which the fast path asked about before the union was ever
reached. Both are now shaped to reach the branch they claim, and both go red when it is mutated.
The log is asserted alongside the value in both fixtures, and it is the stronger oracle: the value
says where the parse arrived, the log says which rule got it there. 191 entries on the parser side
and 40 on the schema side, matching exactly — including the ten-code-point context window, which
pins where the cursor was when it decided.
…rial The story's outcome, plus the two documents that describe how this repo is tested. `run_mutants.sh` gains the warning that cost an hour to learn. Overriding `build.build-dir` to one fixed path is what keeps a mutated rlib out of the shared build dir — and it is also exactly what cargo-mutants gives each parallel job its own of. With `-j` the jobs all write the same `libjson_repair-<hash>.rlib`, because the metadata hash is the crate and not the mutation, so test binaries link someone else's mutant. Measured: 24 survivors against a serial 0 for the same crate, including all six escape arms of a `json.dumps` reimplementation, each of which fails on its own in under a second. The script has always run serially; nothing said why, so nothing stopped an `-j` being added. HANDOFF gains the differential fuzzers as a third layer, and the thing they got wrong: a fuzzer only covers the grammar it generates. `fuzz_parse.py` reported zero while two real defects sat in the string parser, because its replies have no nesting, no comments, no escapes and no tuples. When a fuzz run is green, ask what its generator cannot produce.
…search
Each was a line no assertion reached: the two free functions (`loads` and `repair_json`, which
every other test reaches through `Repair` instead), `Display for Error` (read off `message()`
everywhere), and `SchemaDefinitionError` being told apart from an ordinary refusal — visible only
where a salvaged array re-raises one instead of dropping the item.
And `first_balanced_braces` against dspy's own `\{(?:[^{}]|(?R))*\}`, case by case on the pinned
version. Four of the nine are why it is not `find('{')..rfind('}')`: the pattern backtracks past a
brace that never balances, stops at the first complete object rather than the last brace in the
reply, and is blind to quoting.
`.venv/` matches a directory. A worktree borrows the main checkout's environment with a *symlink* of that name, which the trailing slash does not match, so `git add -A` committed a link to an absolute path nobody else has.
Neither had one. `section_value` hands a structured field to json-repair before the annotation sees it, and `parse_json` falls back to the first *balanced* brace run; both changes passed the whole suite when reverted. Three shapes only json-repair recovers — bare words, asymmetric quotes, smart quotes — and a reply holding two objects with prose between them, where a search from the first brace to the last takes both. Reverting either change now names the case that noticed. The list cases sit on a new `Tagged` signature rather than `Typed`: `Typed`'s `score: int` drags in `parse-time-casting`, so every case using it is already recorded as a divergence for a reason that has nothing to do with the field under test.
`check_api_surface.py` walks dspy to Rust and is at 251/251. Nothing walked back, so anything this crate invented was invisible to every gate here — which is how `LM::with_capabilities` kept zero callers. The extractor keyed an inherent method by its module, so `pub fn new` in `impl LmBuilder` became `dsrust::lm::new`: not a path a caller can write, and one key for every `new` in the module. It now tracks the enclosing `impl`/`trait` at column zero, where rustfmt puts them, rather than counting braces through string literals. `impl Trait for Type` is skipped — those methods belong to the trait, which is recorded where it is declared. That found 14 items the module keying had merged. 98 items classified: the 49 a caller reaches as `dsrust::X`, and the LM wire, which is the largest cluster precisely because litellm owns it for dspy and there is no upstream counterpart to map to. The stale check earned itself immediately — 12 entries pointed at the pre-`impl` keys, and two of those names turned out to be two live methods each, `LM::x` beside `LmBuilder::x`. `make_signature` was filed as a divergence reading "Rust uses the derive/builder". The macro exists; the walk put it in front of the entry. Now mapped, with its four parameters classified. The count moved 395 -> 399 when the extractor sharpened. A ratchet you raise is worthless, so the baseline stayed and the slice was classified past it: 356.
A mutation pass named six survivors in the reimplementation. Four are real gaps: `\b`, `\f` and `\r` have short names in JSON and no model writes one, so each arm was indistinguishable from the `\u00XX` fallback; and U+FFFF against U+10000 is the only pair that tells one escape from a surrogate pair. The other two are equivalent. `number > 0.0` against `>= 0.0` decides the sign of an infinity, and an infinity is never zero. `exponent < 0` against `<= 0` is only reached when the decimal point sits at or below -4 or above 16, so the exponent is never zero either.
dspy records the model each predictor was pinned to, sanitises that block on load, and rebuilds a live LM from it. This crate wrote `null` and dropped whatever it read, so a dspy-compiled program opened here lost the model its author chose and answered from whatever the process happened to configure. The plan left three alternatives open. Faithfulness settles it: dspy's behaviour is reconstructible here, so carrying the block sanitised is the only one that is not a divergence somebody has to be told about. `ChatModel::dump_state` is defaulted, as `BaseLM.dump_state` is overridable, so a scripted double still states dspy's `null`. `NamedPredictor` carries the model for the reason it carries the other three: reaching every predictor is one problem. `Trust` is the shape `allow_unsafe_lm_state` takes here. Key order is upstream's, pinned byte for byte against a generated corpus, including two places nobody would guess: `temperature: null` is present for a model that set none, and a reasoning model's `max_tokens` lands *after* the three finetuning keys, because dspy pops `max_completion_tokens` and re-sets it. Three controls confirmed the comparison fails when any of that is wrong. Narrower than dspy in two places, both asserted rather than left to a diff: `base_url` and `model_list` survive a trusted load and are not honoured, and no `allow_custom_lm_class` path can exist because a Rust binary has no importer. Writing the trusted-load test is what found `api_base` kept but never applied. -- Two upstream tests were red at HEAD, and the gate did not run the suite that says so. `dspy.Code` accepts any bare string — its validator takes `isinstance(data, str)` and filters a markdown fence — and the crate's list of annotations with a string form named only the four temporal types, so a `Code` field was cast as JSON and rejected a reply dspy accepts. The list is now measured against pydantic rather than written twice; `Reasoning` belongs to the same set and is already covered, since its annotation name is `str`. The suite is in the gate now. Two of the five gates were a checklist once before; this is the third time something sat red because passing was a command someone had to remember.
…umbers were rubbish `CARGO_BUILD_BUILD_DIR` was a constant. This repo is worked in many worktrees over one machine, so two sessions mutating two crates wrote the same `dsrs-mutants-build`: each linked the other's mutated rlibs, and whichever finished first deleted the directory out from under the other — the cleanup trap removes the path it names. It is the same collision the override exists to prevent, one directory further out, and neither run said anything was wrong. `mktemp -d` keeps the isolation and takes the collision away. The cost is one crate's object code per concurrent run, which is what isolation costs. Also here: a committed sweep of the fuzz grammar, and the grammar widened to reach what it did not. `tests/fuzz.rs` skips when `target/json_repair_fuzz.json` is absent, and cargo-mutants copies a tree that never has it — so the crate's strongest oracle was missing from every mutation run. `lookahead.rs` lost 131 of 139 viable mutants that way. `json_repair_sweep.json` is the same grammar at a fixed seed, 500 drawn inputs with the repairs each one logged, recorded as a golden like every other fixture here; the campaign fuzzer keeps running at twenty thousand where a committed file cannot. The grammar itself was too narrow in three places the survivors named: no key beginning with `_` or carrying a digit, though the bare-key scan accepts both; no fenced snippet after a closing brace; no container opened straight after a separator. Thirty thousand inputs over three seeds on the widened grammar, still no disagreement.
`parse_fuzz` reads its corpus from `target/parse_fuzz.json` and returns early when there is none. cargo-mutants copies the source tree, which never has a `target/`, so the differential comparison against dspy — tens of thousands of cases, the strongest thing this crate has on the parse side — contributed nothing to any survivor count. So did a fresh clone. An early return reports the same green as a run. A 1500-case sweep at a fixed seed is committed now, and the test reads it always and the campaign corpus as well when one is lying around. Neither path can skip. The sweep only measures what its grammar can produce, and the grammar was too narrow in one place the survivors named: every generated field name was lowercase letters, so no reply could tell a correct word check from a broken one. It now draws underscored, leading-underscore, digit-carrying, hyphenated, dotted and empty names — with the declared names weighted up, because giving every spelling equal odds took the accepted share from 180 in 1500 to 38, and a corpus that is 97% refusals says little about the path that hands a caller a value. 1500 cases, still no disagreement. Four named cases from the same survivors: a marker whose name is not a word and one whose name is empty; an XML tag with an underscore, which is the shape most signatures use; and a declared field inside a non-word tag, which shows the scan resumes one character past the `<` instead of swallowing what the tag wraps. `next_tag` is a `match_indices` walk rather than a self-advanced cursor. Its termination rested on one `cursor = open + 1` at the bottom of a loop, and mutating that line hung the suite instead of failing it — no assertion catches a function that never returns. `parse_tags` gets the same treatment: it breaks unless the remainder actually shrank. A parser reading model output is reading input nobody wrote. `Adapter::name` was unasserted, and three of five reported the Rust type's name where dspy's class is `JSONAdapter`, `XMLAdapter`, `BAMLAdapter`. That string is what a callback watcher reads, since upstream dispatches by type and hands the handler the instance for `type(instance).__name__`. Recorded off the classes, not transcribed. `kind_of`'s output lookup had no test with two differently-kinded outputs, so finding the wrong field went unnoticed. Given a list, a `str` field renders dspy's enumerated form and a structured one renders JSON. `run_mutants.sh` builds into `mktemp -d` instead of one fixed path. Two worktree sessions on this machine shared it, and `dsrust` depends on `dsrust-json-repair`, so one run's mutated rlib landed where the other's test binaries link from — the poisoning the override exists to prevent, one directory out. The cleanup is `rm -rf`, so the first to finish also deleted the other's build dir mid-run. Every survivor count taken before this is void. The corpus was `chat_parse.json` holding three adapters with the result under a `chat` key. It is `adapter_parse.json`, keyed `dspy`.
It reproduces a Python library and nothing about that library knows what calls it, so neither should its documentation. Every mention of the project this lives in is gone from `src/`, along with the internal voice — backlog stories, mutation counts, which fuzzer found what. What is left says what the crate does, why a port rather than another repairing parser, and where it cannot follow Python. `missing_docs` is denied and every public item answers for itself, with runnable examples on the entry points and a README that is the crate's own. The `serde` feature, off by default, because `Value` being its own type is unavoidable inside the parser and a nuisance outside it: Python distinguishes `7` from `7.0` and holds an integer of any width, and `serde_json::Value` does neither. So it converts both ways, serializes, deserializes, and `loads_as` goes straight to a caller's own type. It does not replace the writer. `serde_json` writes JSON its own way and Python's `json.dumps` does not — a space after every comma and colon, everything outside `\x20`-`\x7e` escaped, and a float through `float.__repr__`. Those bytes are the point of the crate. Two shapes have no `serde_json` spelling and are named rather than left to chance: an integer past `u64::MAX` becomes a float, which is what `serde_json` already does reading one out of ordinary JSON, and a non-finite float becomes `null`, since JSON has no literal for one.
Three hundred and forty-nine distinct calls the project's fourteen test files make — the input, the
keyword arguments, and what came back — written down by a `conftest.py` that wraps the entry point
before pytest imports a single test module. Those cases were written by the people who wrote the
heuristics and grown one bug report at a time, which makes them better than any list assembled by
reading the library once. All 349 pass.
Recorded rather than transcribed, and the recorder had to be made honest twice before it was.
**It filtered arguments it did not understand.** `repair_json` forwards `**json_dumps_args` to
`json.dumps`, so `repair_json(text, ensure_ascii=False)` produces different bytes — and keeping the
result while dropping the argument writes a case that looks comparable and is not. It now drops the
whole call and says which argument did it. That surfaced a missing piece of the API, so
`Repair::ensure_ascii` exists and the case is back in.
**It guessed the return shape from its type.** `return_objects=True` answering with a *string* is
indistinguishable by type from the repaired text the default returns, and guessing put
`repair_json('("x")', return_objects=True)` — whose answer is the value `x` — into the fixture as
text that `json.dumps` would have written `"x"`. Only the call site knows which it was, so it reads
the kwargs now.
The checkout is verified byte-identical to the installed package before anything is recorded: a
suite from one version held against a port of another is a comparison that looks like conformance
and is not.
149 functions and classes in the pinned package, each mapped to what answers for it here — checked by a script that walks the pin with `ast` and fails when a symbol is unmapped, when a mapping names a Rust item that does not exist, when a signature moves under the pin, or when an absence has no reason written against it. A port is read once and then trusted forever; this is what stops a function quietly going missing. 124 ported, 25 absent with a reason each. Writing it found `load` and `from_file`, which were not ported at all — and they are not aliases for `loads`. Upstream ties the suffix fast path to where the input came from (`try_valid_json_suffix = json_fd is None`), so a valid JSON value after a prefix goes through the repair parser for a file and through CPython's scanner for a string. Measured over twenty thousand generated inputs, the two disagree on 37. `from_file` and `from_reader` now exist and turn the path off, with four cases pinning it — asserted as a *difference*, because an equality would pass just as well if one were an alias for the other. Both controls fire: aliasing them, or forcing the fast path on, turns the test red. The reader is drained into a string rather than chunked through a `StringFileWrapper`. That wrapper implements indexing and length and nothing else, so the parse sees the same characters either way and the difference is memory — which is also why `chunk_length` is not carried. The *behavioural* difference between file and string input is the suffix path, and that is reproduced. The other absences are Python being Python: a context-manager object where Rust has no `with`, a `MissingValueType` sentinel that is `Option<Value>` here, a validator cache for an object this crate does not have, and `cli`, which is a program rather than a library.
**The byte fixture records both sides in hex.** Every other fixture holds its cases as JSON strings, so the comparison passes through two JSON encoders and a Rust `&str` — fine until something normalises, and then the strings match while the bytes do not. Sixteen cases chosen for where an encoder is tempted to interfere: a byte-order mark, CRLF, a NUL, an astral character that is one code point and two UTF-16 units, a combining sequence *and* its composed form, and both settings of `ensure_ascii`. Writing it found a bug in the `ensure_ascii` added an hour ago. Turning it off stops the `\uXXXX` fallback for ordinary characters and **not** for control ones — JSON requires those escaped either way, which is why `py_encode_basestring`'s pattern is `[\x00-\x1f\\"]`. The first control case used only characters with short escapes, so it did not notice; one with no short name does. **`dsrust`'s `python_json::json_dumps` now delegates here**, which is the same concern in two places and was the same concern getting a different answer. Writing scalars with `serde_json`'s `Display` agreed with Python on everything except a float outside the fixed-notation window: `1e16` where `float.__repr__` gives `1e+16`. A schema or a demo carrying such a number rendered a prompt off dspy's bytes. Reverting the delegation turns the new test red. `lib.rs` had grown past four hundred lines, most of it documentation, so the refusal and the builder each have their own module and the crate root is the documentation and the exports. Gates: exit 0, 1144 tests.
Measured clean — nothing else running, a private build directory, and the
first run where `parse_fuzz` had a corpus to read. 132 caught, 17
unviable, 0 timeouts, 1 missed. The story opened at 43 of 143 viable.
The one survivor is equivalent: `parse_json`'s `start < end` against
`start <= end`, where `find('{')` and `rfind('}')` cannot return the same
index because a byte is not both braces. It is recorded in the source and
in the baseline note, so a later run does not offer it again as work.
`run_mutants.sh` now runs the slice and holds it to that number.
`check_ratchet` is one function both it and the package baselines call,
so a file-scoped run and a package run cannot drift apart on what counts
as a survivor — TIMEOUT counts as one either way.
`dspy_loads_and_runs_what_this_crate_saved` carries `#[ignore]` because it needs `.dspy-venv`. Nothing ran ignored tests, so it had never executed — and its paths were written as though `CARGO_MANIFEST_DIR` were the workspace root, when it is the crate. Both the interpreter and the checker script were looked for two directories too deep, so the test could only ever have failed. The README's interop claim — that `dspy.load` opens what this crate saves — rested on it. Fixed and in the gate, which needs that venv anyway for the upstream suite, so the reason for the `#[ignore]` does not apply there. It passes, which also says dspy reads the `lm` block this crate started writing earlier today. Controlled: changing the saved instructions turns it red on what dspy actually returned, not on the file we wrote. `parse_fuzz`'s header still claimed the test skips without a corpus. It does not, since the sweep is committed — and a stale claim about what a check does is the same defect as the check itself, one level up. Found by sweeping for the shape behind the mutation finding: not "a check that cannot discriminate" but "a check that does not run". Nineteen ignored tests, five env-gated defaults (all inside ignored live tests, fine), and this one, which was the only one whose stated reason the gate already satisfied.
…ad twice The last four. `llm_query_tools` hands the sandbox a *pair* sharing one budget, so spending it on either leaves less for the other — which is what stops model-authored code looping a sub-LLM. `Ends` is implemented for both things an asynchronous watched point answers with, so `watching` is one function rather than a `describe` argument and an `ended` argument every call site keeps in step. `LmStream` is one type over an iterator, covering dspy's synchronous and asynchronous classes both. And `injection` gets a full trait implementation, because there was no public implementor to point at and what it costs to implement is the documentation a caller needs — it pins the property that matters, that the bytes travel *beside* the code rather than inside it, so a value carrying quotes or newlines cannot break the program it is injected into. 80 -> 0. The floor stops being a ratchet and becomes a rule: a public item this crate invented has a caller in this repo, or it does not ship. The size gate then forced `observe/evaluating.rs`, on a real distinction — a module call is one event with a beginning and an end, an evaluation is a container, and getting that wrong reports five hundred separate runs instead of one. That split failed twice the same way, and it is the way the last one failed. The removal was written before the destination directory existed, so the write threw *after* the file had already lost the block — recovered from HEAD with the post-HEAD example reapplied. Then the `mod` declaration was placed by searching backwards for `///`, which finds the *closing* fence of a doc comment rather than the block's start, so it landed inside `TARGET`'s example and turned the new file's header into a doctest. Both are one root cause: computing a position from a marker instead of from an item boundary. The repair walks up to the first non-doc line, which is the boundary itself. `cargo doc` caught two cross-module links afterwards, which build and test both passed — the sixth and seventh time today.
dspy's `EvaluationResult.score` is `round(100 * ncorrect / ntotal, 2)`, a percentage. The ledger mapped it to `Evaluation::score`, which carried the 0..1 mean — so a caller comparing against a number dspy printed was off by a factor of a hundred, and every optimizer scaled it back by hand through two byte-identical functions under two names. `Evaluate::run` now puts the percentage there, `copro::dspy_score` is gone, and `BootstrapRandomSearch` stops comparing `stop_at_score` against the wrong scale: upstream reads that bar against `result.score`, so `0.9` never met a bar it should have. `on_evaluate_end` had no error arm, on the stated grounds that neither port has one. Running dspy says otherwise — past `max_errors` it raises out of `Evaluate.__call__`, and the decorator reports that to the handler with `outputs=None` and the exception set. The crate returned from inside its own open point, so a handler told an evaluation began was never told it ended. Both arms close it now, and `evaluate_abandoned` records dspy's sequence for the give-up path. The row fails at the model call in both ports: this crate's metric returns `f64` and cannot raise, so a metric failure would compare a shape only Python can produce. Measured while pinning that fixture down: `Adapter.__init_subclass__` runs `cls.format = with_callbacks(cls.format)` for every subclass, so a subclass that inherits an already-wrapped method wraps it twice. `JSONAdapter.format` and `XMLAdapter.format` fire the format point twice, nested, where their `parse` fires once — both classes define their own `parse` and neither defines `format`. Recorded as a deliberate divergence. Five ledger reasons were wrong. Two pasted the `instance` sentence about handing over a name onto handlers that take no name; two said this handler has no error arm; one described `percent` as "not the score". Nineteen doc blocks had a code fence sitting in the middle of a sentence, its tail orphaned below the example, and in four cases a second summary line wedged at the cut. Rustdoc took the first line as the summary, so items were listed under a usage note or a fragment. `exact_match` had lost its summary to `percent` and had none of its own.
Twenty-one adapter methods carried one pasted reason: "internal render step; folded into the adapter's own file, which writes the whole message". Nearly every one has a named counterpart the sentence denies — `chat.rs::output_requirements`, `json.rs::json_output_requirements`, `xml.rs::output_requirements`, `prompt.rs::task_description`, `adapter/demos.rs::demo_turns`, `xml.rs::wrap`, and the three user-message builders. The divergence is real, since none of them is an overridable hook, but the reason described an absence rather than the shape. The paste was already known wrong here: a 2026-08-24 pass corrected the base and chat `format_field_structure` rows and named `json.rs` and `xml.rs` as writing that section, while leaving those two rows on the paste that says they do not. Writing twenty-one reasons that name private helpers should have been checked by `check_ledger_claims.py`, which reports "every substitution points at something that exists". Fabricating one to see it fail proved it does not: `BARE_PATH` matches `file.rs::item` from the `rs`, finds no such crate, and skips the path as foreign. Every reason written that way was unchecked, mine and the four that predate them. `RS_PATH` now checks that spelling, and checks the stronger claim the reason makes: the item must be defined in the file named, not merely somewhere. It caught four of these reasons on its first run — `demos.rs` names two files in this tree and the checker had picked the other one, so it now asks every file the suffix matches, and the reasons say which.
Eight custom types shared one ledger reason: "reproduced as the type's
`Deserialize`, which is where Rust validates an incoming value". Their
`validate_input` is a `mode="before"` validator, so pydantic runs it when a
reply is parsed, not only when a caller constructs one — which makes the
shapes it accepts part of the wire contract. Asking upstream for each shape
found two the crate did not read:
- `Audio` refused `data:audio/wav;base64,…`, the one string form
`encode_audio` takes. The coercion existed, in `Audio::parse`, and
nothing routed to it.
- `ToolCallResults` refused a bare list, a lone `{name, value}`, and an
empty list. The list shape is the one `ToolCalls` itself accepts, so a
provider returning it symmetrically was dropped.
Six refusal messages differed. dspy interpolates the offending value or its
type into an f-string, and `Display` on a `serde_json::Value` is not
Python's `str`: a string kept its quotes, a map had no `{'k': 1}` spelling,
and `type(x)` printed `5` where Python prints `<class 'int'>`. `Reasoning`
also named itself `Reasoning` where upstream says `dspy.Reasoning`. That
vocabulary is one module now, since a copy per type is how three of them
came to render a value three different ways.
`adapter/type_coercion.json` records, per type, every branch of its
`validate_input` and dspy's own words when it refuses — recorded by running
the pinned dspy. A null refusal is pydantic complaining structurally, which
describes a Python type system and has no counterpart worth matching.
`tool.rs` went over the file-size gate on the way, and is split where the
two halves already differed: what a model asks for, and what came back.
`Adapter.acall` and `Tool.acall` carried the reason written for the module and model ports: "Rust is async throughout, so the one `forward`/`call` is the async method". Neither trait has an async method. `Adapter::format` and `Adapter::parse` are synchronous because this adapter never makes the model call — the caller does, which is the `__call__` divergence recorded beside them. `Tool::call` is synchronous on purpose: a tool here is a Rust closure, not a network call. The eight rows that remain on that sentence are the ones it was written about. The claims gate knew dspy's names and not `gepa`'s, though eight reasons cite gepa symbols and all eight resolve. It reads that package too now, and says so when it cannot find it rather than quietly knowing less — which is the failure this same gate had in its `file.rs::item` blind spot. Its summary counted the rows phrased as a substitution, 34 of 2638, and called them "substitutions checked". The number that matters is references resolved: 1780. Rewriting twenty-one reasons moved the first count down and the second up, which is the wrong way round for a line anyone reads to know how much was checked.
The ledger justified this crate's `DummyLM` erroring on an unscripted call
with "Upstream raises". It does not. Measured: an exhausted queue answers
`[[ ## answer ## ]]\nNo more responses`, and a keyed miss answers the bare
`No more responses` — upstream renders the first through
`_format_answer_fields` and returns the second as it stands, so only one of
the two parses. That asymmetry is observable: a keyed miss sends the caller
down the JSON-fallback retry, which is exactly what the `evaluate_abandoned`
fixture ran into earlier today.
Both are reproduced, which needs the mode remembered rather than inferred —
an empty script does not say which door it came through. `fallback` stays,
now as what it is: additive, in place of upstream's words rather than in
place of an error.
Its replies also carried a trailing `[[ ## completed ## ]]`, which dspy's
dummy never writes — measured, `DummyLM([{"answer": "x"}])` answers
`[[ ## answer ## ]]\nx`. The marker is something the adapter asks a real
model for, and both parsers accept a reply without one, so every test
reading `Prediction::raw` was comparing against a string upstream does not
produce.
The one test that wanted a failing model now says so with a failing model,
which is what it meant: an exhausted script is a parse failure and a retry,
not the single clean model error that case is about.
…as never told the set
`question -> colour: Literal['red', 'blue']` parsed to a `Json` field with
no `values`. Two things followed, and the first is prompt bytes: dspy
renders `{colour} # note: the value you produce must exactly match
(no extra characters) one of: red; blue` and this rendered `{colour}`. The
crate writes that note correctly when a field carries `values` — only the
string form never built one, under either spelling dspy accepts.
The second is what a reply may say. Upstream's `parse_value` decides a
`Literal` before every generic branch: it takes the member as it stands,
and unwraps a trimmed `Literal[…]`/`str[…]` spelling and one matched pair
of quotes before refusing. Measured end to end, `"red"` and `Literal[red]`
are answers dspy reads and this refused, and `green` is one dspy refuses
*inside* parse — so its JSON fallback re-asks and the call count is two.
All four now agree, call counts included.
The closed-set check lived in `ensure`, which runs after parse, so it could
not reach the fallback and answered in words dspy never writes. It is a
cast now, in dspy's, and the branch left in `ensure` could no longer fire.
An enum is decided before a closed set, as upstream decides it —
`find_enum_member` takes a member's name as well as its value, and the
description this crate is handed carries only the values. Routing an enum
through the closed-set check refused `IN_PROGRESS` for an auto-valued enum
whose values are `1; 2; 3`; upstream's own
`test_auto_valued_enum_inputs_and_outputs` caught it.
`FieldMismatch` grew `reports_parsed` because `parsed` was answering two
questions — whether to print the trailing line, and what to hand a feedback
retry. Emptying it for the cast failure got the line right and dropped the
retry's previous output.
`python.rs` says one home rather than one per caller; there were four
again, one of them mine from this morning, quoting strings a way Python
does not. `str` and `type` live there now, and `field_type.rs` is split
where it held two jobs: what a field's type is, and what a value has to
become to be one.
CI has installed deno since the sandbox landed, and nothing invoked the tests that need it — the same shape as the saved-program test the gate script already runs, whose comment says a test with `#[ignore]` and no gate has never been red. These take about seven seconds together and all ten pass, so the only thing between them and the gate was the wiring. `run_rust_gates.sh` runs them now and requires deno the way it already requires the venv, with the install line in the message. Breaking one on purpose exits the script 101 rather than carrying on, which is the part worth checking before believing a new gate. The recorded test count was one high: 472e54e deleted a scratch file that `git add -A` had swept into the commit before it, and did not re-record. CI's `git diff --exit-code` on those three files is what would have caught it.
The prompt harness that reads every top-level fixture builds its signature from the fixture's *own* JSON, so `code_proposal.json` proved the adapter renders dspy's signature into dspy's bytes and said nothing about the Rust `code_proposal()`. Corrupting its instructions left every test green. A doc comment named `every_signature_matches_its_dspy_fixture` as holding it; that test covers three MIPROv2 signatures and not this one. `the_code_proposal_signature_is_dspys_own` compares the function to the fixture — instructions, input names and descriptions, output names and descriptions. Both halves were checked by breaking them. The instructions were a three-thousand-character Rust literal holding an escape per newline and quote, which is a transcription hazard on a value nobody can read at a glance. They are a vendored `.txt` now, for the reason `PRIMITIVES_CATALOG` beside them already is, and `generate_fixtures.py` writes it from the same pinned dspy that writes the fixture — regenerating produced no diff, so the extraction is byte-clean.
…an item "Every narrowing is exhausted" stopped being true twice today. Grouping reasons by their text found three bulk pastes hiding two behavioural bugs; matching claims about what upstream *does* — raises, returns, accepts — found two more. Reading the shortest unreached reasons found a third narrowing and a gap in the gate that was supposed to check them. The gate's substitution list had `reproduced via|as|by` and not `reproduced in`, so "reproduced in lm/api" was unchecked — as were `folded into` and every reason using it. Both are in the pattern now, and the count of substitution rows checked went 34 → 52. Nine of those named a module or a phrase rather than an item, so widening the pattern alone would not have checked them: `LMCacheConfig` read "reproduced in lm/api". They name `LmCacheConfig`, `LmPart::Refusal`, `Module::load_state` and the rest now — all verified by hand first, all mechanically checked from here. `PY_FILE` mirrors `RS_FILE` for the other tree. A reason naming `clients/embedding.py` claims where upstream keeps something, and that claim goes stale the day the pin moves and a file is renamed — which is the one thing nothing here re-derived. 100 such paths are cited and all 100 resolve; renaming one on purpose turns the gate red. References resolved against a tree: 1780 → 1910. dspy-facing divergences that no mechanical check reaches: 214 → 152.
Reading `evaluate/evaluate.py`'s unreached rows found `Outcome` in five of them — `Outcome::results`, `Outcome::score` — where the type is `Evaluation`. Two were qualified paths, which the claims gate is supposed to check, and it passed them: `refine.rs` declares an unrelated `enum Outcome`, something somewhere has a `results` field, and the gate compares each half against one flat set of names. Its own comment says a *bare* name resolves against any type that has one. A qualified path was doing the same, which is worse, because `Type::member` is how you say which type. It now checks that the member belongs to that type — for `struct`s and `enum`s with no `impl` block, declared exactly once in the workspace, and nothing else. That restriction is the whole of what a regex can be sure of: anything with an `impl` also carries associated consts and trait methods it never overrides, and asking about those produced fourteen false positives (`Predict::dump_state` is defaulted on `Module` and appears in no block belonging to `Predict`). The gap that leaves is written down beside it. Three bugs on the way in, each caught by a control rather than by reading: a local named `owners` shadowed the type index so the check never fired at all; a multi-line `impl … where` header opened its brace three lines below the name, cutting every such block to nothing; and `<[^>]*>` cannot skip `impl<E: Iterator<Item = LmStreamEvent>> LmStream<E>`, since angle brackets nest. In `evaluate/metrics.py`, `answer_exact_match.trace` said the metric returns a bool rather than a float *when trace is set*. Upstream documents that argument as `Unused; reserved for compatibility` and returns `True`/`False` either way — measured — and the row beside it said so.
`BaseLM.copy` read "reproduced as `Clone` plus the per-call `LmConfig`". `LM` had no derive and no impl — a substitution asserted and never built, which is the shape that reads as resolved forever. Every field was already cloneable, callbacks included, since those travel by `Arc`; there was no constraint, only nobody having checked. `LM` derives `Clone` now, with a doctest that copies a model and changes one setting on the copy. The other half of upstream's `copy(**kwargs)` — the overrides — is a caller naming the change on the clone, so a misspelled attribute is a compile error rather than a key quietly set on the instance; its own row says so, which the API gate demanded the moment the method became mapped. Seven more reasons in `clients/`, `signatures/signature.py` and `evaluate/metrics.py` named something real without naming it precisely enough for any check to hold — "the provider stack (lm/api, lm/anthropic, lm/ollama)" is three modules, `lm/openai/mod.rs`, `lm/anthropic/mod.rs` and `lm/ollama/chat.rs` are three files a gate can open. Verified by hand first, mechanically checked from here.
`GEPA.track_best_outputs` read "GepaOutcome carries the best candidate's outputs". It carried the best *candidate* — a map of component instructions — and nothing kept a prediction past the score it earned. The flag exists for GEPA as a batch inference-time search, where the answers are the result rather than the program that produced them, so the gap was the feature. Ported from the pinned gepa, whose update it rides: `best_outputs_valset` moves with the Pareto front, replaced on a strictly better score and appended to on an exact tie, so each list names exactly the programs `fronts` names. The seed's own answers start the lists, as gepa's initialisation does — an example nothing beats the seed on still reports what it answered. `GepaAdapter` grew an `Output` associated type rather than one concrete value, because the gepa crate knows nothing of predictions and has exactly one dependency: dsrs binds it to `Prediction`, the scoring-only test adapter binds it to `()`. Off by default on both sides, since carrying the outputs means cloning every prediction on every valset evaluation. `Adapter::new` was already nine positional arguments, so the flag is a builder rather than a tenth. Held by `outputs_follow_the_front_they_belong_to`, which asserts the replace-on-better/append-on-tie rule and that every list matches its front, and by `tracking_best_outputs_reports_what_each_front_answered` end to end. Both go red when the outputs stop being carried. `track_stats` beside it also overstated: the run's record is on `GepaOutcome` unconditionally, so there is no flag to map — the one part that is optional is the outputs this adds.
A row said the result was "surfaced through the gepa crate's own result type". Comparing the two field by field found two it did not carry: `val_subscores`, every candidate's per-example scores, and `per_val_instance_best_candidates`, the Pareto front the search itself selects from. Both were already in the state and simply never read out. `the_outcome_reports_what_dspys_result_reports` holds them against each other rather than against their own shapes: every candidate is scored on every row, `val_aggregate_scores` is the mean of its subscores, and no candidate beats the front it is not on. Nineteen reasons across `evaluate/metrics.py`, `evaluate/evaluate.py` and `teleprompt/gepa/gepa.py` were labels rather than justifications — "internal helper", "Python display", "experiment tracking, which is the caller's to wire". Each now says what the thing does and what stands in its place, and the ones naming a Rust counterpart name it precisely enough to be checked. `merge_dicts` prefixes a shared key with `example_`/`pred_`, `truncate_cell` cuts to 25 words, `prediction_is_dictlike` decides whether a prediction spreads across the table's columns — all read from the pinned source rather than inferred from the name.
`litellm_text_completion` read "litellm text-completion wire call; native provider stack", which reads as a wire this crate speaks. Nothing here spoke it: `OpenAiWire` was Chat and Responses, and `LM.model_type` said so in passing — "decides chat vs responses" — without saying the third value had no counterpart. `model_type="text"` is the one place upstream turns a rendered message list back into a single string, and both rules are its own: the prompt is every message's content joined by a blank line with `BEGIN RESPONSE:` appended, and every role flattens into it, so a model on this wire is never told which paragraph was the system prompt. Recorded by replacing litellm with a recorder and reading what dspy passed, rather than by reading its source — six cases, including the empty list, which still sends the marker because upstream appends it to the list rather than joining onto it. Breaking either rule turns the conformance test red. `text.rs` speaks `/completions` directly: request, reply and streaming, whose frames carry `choices[].text` and none of the reasoning or tool-call vocabulary the chat wire reassembles. litellm's `text-completion-openai/` prefix does not travel — it is a routing token for a router this crate does not use — and the golden asserts that difference rather than hiding it. `LM::openai_text_api` selects it, beside `openai_responses_api`.
`signatures/field.py::move_kwargs` read "internal kwargs plumbing for the legacy field". It is called by `InputField` and `OutputField` — the current API — and by nothing else: pydantic refuses arbitrary keywords, so dspy splits them, and the names it knows go under `json_schema_extra`. That split is why a dspy field keeps its metadata in a schema extra at all. Twenty-three more reasons across `signatures/field.py`, `primitives/module.py`, `clients/base_lm.py`, `clients/cache.py`, `core/types.py` and `utils/dummies.py` were labels or bare cross-references — "Python metaclass", "internal attribute plumbing", "as `get`". Each now says what the thing does and what stands in its place, read from the pinned source: `ProgramMeta` runs the base initialisation before `__init__` so a subclass forgetting `super().__init__()` still gets one; `set_attribute_by_name` is `magicattr.set` writing through a dotted path, which `named_predictors` handing back `&mut` makes unnecessary; `normalize_parts` turns a bare string into a one-element text part. `deny_unknown_fields` on `LmResponse`, `LmRequest::configured`, and `dummy_rm` being a retriever and nothing else were each checked before the reason claiming them was left standing.
`to_openai_text_request` read "this crate speaks the chat and Responses
wires" — stale within the hour, since the text wire had just landed. Reading
it to fix the sentence showed it is the *typed* builder, and the one a port
grounded on `openai_format` follows. I had built against
`litellm_text_completion`, which forwards the config untouched and so shows
none of what the typed path decides.
Three divergences that path could not have revealed:
- `text_config_kwargs` writes `temperature, max_tokens, top_p`. Mine had
`temperature, top_p, max_tokens`, copied from the chat wire's order.
- It writes `logprobs`, which mine dropped entirely.
- `messages_to_text_prompt` *raises* on a part that is not text, naming
the Python class. `LmMessage::text` filters silently, so an image
would have gone out as a prompt the caller did not write.
`lm_api/openai_text.json` compares the whole body now, nine cases, against
`to_openai_text_request` — the same shape as the chat wire's fixture. The
litellm capture stays as a second oracle for the prompt rule they share and
is the only place the routing prefix they disagree about is visible.
`LmPart::kind` gives each variant dspy's class name, because the refusal
interpolates `type(part).__name__` and a caller matching that sentence is
matching dspy's.
Lesson for the ledger: a stale row is not only a wrong sentence. This one
was hiding a better oracle for code written twenty minutes earlier.
…dent `COPRO.track_stats` read "the outcome carries them here". `COPRO::compile` returned `Result<()>` — there was no outcome. dspy records three things under that flag and this crate recorded none of them. `compile_traced` returns them, the shape `MIPROv2::compile_traced` already uses: `results_best` (per predictor, per depth, over the top ten scores seen), `results_latest` (over the newest `breadth` candidates), and `total_calls`. Upstream keys its dicts by `id(predictor)` on a `student.deepcopy()` that is never returned, so nothing reachable from outside can match them — the fixture re-keys by insertion order, which is the predictor order they were built in and the only thing that crosses a process boundary. The run comparison alone would have proved almost nothing. The keyed model answers by question, so every candidate at a depth scores the same: every recorded `std` is `0.0` with `min == max`, and a fixture built only from runs cannot tell `pstdev` from the sample deviation, `average` from `max`, or a top-ten slice from taking everything. That is incidental agreement, not verification. `summaries` puts spread in on purpose — fifteen scores, ties across the cut, negatives — against Python's own `statistics`. All four choices go red when broken; the run comparison stays green through three of them. A field named `std` — dspy's name — put `std` in the tree's names and turned every `std::thread` in a reason into a claim about this crate. The claims gate now skips foreign roots by name rather than inferring them from what the workspace happens not to define.
`check_ledger_claims.py` checks that every name a *reason* points at
exists. Doc comments make the same claim — "held by
`the_vendored_shim_is_upstreams_own`", "see
`crates/dsrs-bridge/python/reflect.py`" — and no gate looked at them. Six
were wrong, every one written while fixing something else:
- a test cited under a name it never had
- a conformance test cited as `tests/openai_text_conformance.rs`, which
does not exist — the test is in the module it tests
- three paths written `bridge/python/…` for files under
`crates/dsrs-bridge/python/…`
- dspy's `test_baml_adapter_formats_pydantic_inputs_as_clean_json` cited
singular, and the comment's whole point is that upstream's own test does
not pin what these literals pin
- gepa's `sample_and_attempt_merge_programs_by_common_predictors` cited by
a substring of its name
`check_doc_citations.py` resolves 323 references against both trees, since
naming a dspy function is legitimate and common — the port is written in
terms of them. What belongs to CPython, numpy or litellm is listed by name
rather than inferred, because a checker guessing at that would either miss
real errors or cry wolf.
It found its own bug on the way: dspy's streaming tests are `async def`, so
leaving the keyword out of one regex reported a test missing while a grep
found it immediately.
`Refine`'s feedback ask carries a `modules_defn` field and this is what fills it — every predictor's fields and original instructions, laid out with tabs and an eighty-column rule between blocks. A model reads that text. The ledger called it an internal module walk and marked it a divergence; `refine/describe.rs::modules` reproduces it, and the `offer_feedback` golden holds only the field's *declaration*, not its rendered value. The Rust rendering was pinned by a hand-written expected string. It was right — checked by running dspy — but an expectation written beside the code it tests agrees by construction and would keep agreeing after a pin bump moved the format. `predict/inspect_modules.json` is dspy's own output for five program shapes, including the two the old test never reached: a field *with* a description, and instructions split across lines. Narrowing the rule by one column or turning a tab into spaces now turns it red. Three of those cases were recorded and skipped on the first pass — a golden nothing reads, which is the shape this audit keeps finding. The test panics by name on a case it cannot build rather than continuing past it, which is how the two-predictor case came to be built rather than waved at. `recursive_mask` and `serialize_object` were labels too: both make a value JSON-serialisable for that same ask, and everything crossing that boundary here is a `serde_json::Value` already.
…e by hand `Image.from_file` and `Audio.from_file` shared one reason: "a Rust caller supplies the already-encoded value". Both types have `from_path`, `from_url` and `from_url_unverified` — reading, encoding and fetching — and those constructors are mapped three rows above the ones denying them. `File::from_path` too. What is actually true of `from_file` is narrower and more useful: upstream deprecated it, warns it goes in 3.4, and forwards to `from_path`. So the divergence stands and the sentence changes. `COPRO.track_stats` still read "the outcome carries them here" from before `compile_traced` existed to carry them. `Ensemble.deterministic` now quotes the assertion upstream raises rather than asserting it does. The three `teacher_settings` rows shared a sentence naming `set_lm` without saying which type has it.
…th exist `PythonInterpreter.sync_files` read "this crate mounts them, so a write lands where the caller named rather than in a copy that has to be returned". It does not mount: `DenoInterpreter::sync` sends a `sync_file` notification per writable path after each run, which is upstream's `_sync_files`, and the field's own doc comment says so three lines from the code. Both directions were already tested; only the ledger disagreed. `CodeInterpreter.tools` read "a Rust caller wires those into its own interpreter, since the crate ships none". `CodeInterpreter::define_tools` is that seam, and the row for `PythonInterpreter.tools` already named it. Checked and left standing: `normalize_text` decomposes an accent without dropping it — `café` becomes five codepoints on both sides, ending U+0301 — which the row claimed and which reading either implementation alone would not settle, since both `repr` as `café`.
Three rows — CodeAct, ProgramOfThought, RLM — read "the other half of the same change: `forward` now takes the interpreter the factory built". That is neither what upstream passes nor what this crate does. dspy's `forward(interpreter=None, /)` takes a *caller-owned* interpreter used instead of the factory's, which the caller then shuts down; the crate's `forward` takes only its inputs. `CodeAct::ask_in`, `ProgramOfThought::ask_in` and `Rlm::ask_in` are that call, and were all along — a method rather than a first positional argument because Rust has no positional-optional one, taking `Lease::borrowed` so the pass never shuts the interpreter down. Tools and output fields are still injected, as upstream injects them even for a caller's own.
… had none `dsrust::interpreter::CodeInterpreter` read "dspy has one concrete class and no seam". True of 3.2.1. The pin is 3.3.0b1, which ships `@runtime_checkable class CodeInterpreter(Protocol)` — and the dspy-facing row two hundred lines away already mapped it to this trait, so the ledger disagreed with itself across the bump. This is the one staleness a pin bump creates. Every other wrong reason found in this audit was a claim about *this* crate, which only this crate's changes can falsify; a claim about what dspy *lacks* is falsified by upstream moving, and nothing re-reads it when the submodule does. Eleven rows make one — all readable in a pass, and the other ten are about work dspy genuinely delegates to litellm, checked and left standing. `CodeInterpreter::define_tools` was a divergence beside it for the same reason; it is dspy's `tools` property, as the entry mapping it the other way already said.
…riting them The remaining reasons had been read and their claims checked; what they lacked was a name a gate can resolve. "spelled `raw`" becomes "spelled `raw` on `Adapter::parse`"; "via the `Type` seam's `serialized`" becomes `types::serialized`; the five `validate_input` rows now name the fixture that holds them. A reason naming `Predict::set_lm` is checked forever; one saying "set on the module" is not. Writing them produced an error of exactly the kind this audit keeps finding. `load.allow_pickle` gained "what a caller does decide is `module::Trust`" — but `Trust` is dspy's `allow_unsafe_lm_state`, a different load-time flag, deciding whether a saved `lm` block's redirect keys survive. Reading the type rather than trusting the sentence I had just written caught it; both flags have their own rows and neither was wrong until I nearly made one so. Checked and left standing on the way: `Adapter::parse` does spell it `raw`, `Prediction::set_lm_usage` does spell it `usage`, `Rlm::forward` is the async one, and `Predict::demos` carries what `Parameter` marks.
Ten reasons cite a fixture by name — "held by evaluate/max_errors.json" — and neither checker resolved a `.json`, so a renamed golden would have left the sentence pointing nowhere. All ten resolved when the rule was added; the rule is what keeps that true, and renaming one now turns the gate red. Extending the doc-comment checker the same way found a stale path in `mimetypes.rs`: the golden it names moved under `constants/` and the comment did not follow. Two things the extension got wrong first, both caught by running it rather than by reading it. `[A-Za-z_]*\.json` matches the `self.json` inside `self.json_str`, so the pattern needed a boundary after the extension. And `target/parse_fuzz.json` is written by a fuzz campaign and deliberately not committed — twenty thousand random strings are evidence, not a golden — so those two are named in the external list beside CPython's and numpy's.
…re wrong The pattern of the last stretch, continued: a reason whose claim had been checked but which named nothing a gate could resolve. "set it on the module's config" becomes `Predict::set_lm` and `api::LmConfig`; "a closure over a `FieldEdit`" becomes `side::FieldEdit`; "`LmRequest`'s own fields" becomes `ChatModel::forward`, matching the three sibling `__call__.kwargs` rows that already named theirs. Writing them produced two errors of the kind this audit exists to find. `BootstrapFewShot::max_labeled_demos` was called a builder — it is a public field, and only `BootstrapFewShotWithRandomSearch` and `MIPROv2` have the method. The name resolved; the characterisation did not, which no gate here can catch. And `MultiChainComparison.temperature` said dspy "raises the temperature" without saying from or to what; it defaults that ask to 0.7. `Type.extract_custom_type_from_annotation` was "Python annotation walking", which says what it is made of rather than what it does: it searches an annotation for a `Type` nested inside, because a Python field's type is a value to inspect.
…rrors landed I wrote the certain half of this check a few hours ago and documented the gap it left: a wrong member on a type that has an `impl` passes, because associated consts, associated types and inherited trait methods are not things a regex sees cleanly. Since then I put three wrong names through that exact gap while correcting other rows. The wider index reads inherent methods, trait-impl methods, consts, associated types, and the methods a type gets from every trait it implements. Turning it on found two real errors that had been sitting there: `Tool::new` — `Tool` is a trait and that constructor belongs to `FnTool` — and `ChainOfThought::new`, which does not exist at all; the constructors are `from_signature`, `rationale` and `task`. Zero false positives across 2108 references, so it fails rather than warns. It also caught me correcting something that was never wrong. `Refine::code` is a real builder setting `program_code` and `reward_code`; I had "fixed" it after a grep whose output I truncated at `head -3`, one line above the answer. The row now names the setter and what it sets. Two remaining reasons were labels: `old_getfile` is the real `inspect.getfile`, kept as the fallback while dspy replaces the module attribute so a class defined in `__main__` can still be located — the point being to make `inspect.getsource` work on an interactively-defined program. `from_langchain` wraps `tool.ainvoke` and reads its arguments from `args_schema.model_json_schema()`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, to kick off CI. Not for merge.
The point of opening this now is the workflow in
.github/workflows/ci.yml, which has never run anywhere. Everything in this repo has only ever been verified by./scripts/run_rust_gates.shon one Mac.Two incidents in a single day made that a problem worth fixing rather than noting:
package.jsonin a home directory — awranglerdevDependency, nothing to do with this project — put Deno into node_modules resolution mode and turned 27 upstream RLM tests red with nothing in the repo changed;git statusshowed a clean tree.Both were machine state deciding a result, and there was no second machine to disagree.
What CI does
testcargo build --all-targetsandcargo test,fail-fast: falseso every platform reportslintcargo fmt --check,cargo docunder-D warnings, the cursor-arithmetic lintconformancedsrs-bridgeis excluded from the portable matrix at no coverage cost — the workspace suite reports the same 1311 either way, because the bridge is a PyO3 shim with no tests of its own,publish = false, and the one place this repo is platform-shaped.What to expect on the first run
The Linux and Windows legs have never executed. The YAML parses and every command passes locally on macOS, but
dsrustitself could not be cross-checked from a Mac:ringvendors BoringSSL C and assembly, so cross-compiling it needs a target C toolchain. Each runner builds natively, which is exactly why CI answers what the laptop could not.A red leg here is the measurement working, not a surprise.
Also in this branch
324 commits behind the CI files, the substantive ones being the dspy 3.3.0 pin and the conformance work on top of it. Highlights rather than a list:
.gitattributes, which is not housekeeping here. Two goldens are compared as whole text withinclude_str!+assert_eq!; Git for Windows defaults tocore.autocrlf=true, so a Windows checkout would fail them for a reason with nothing to do with the port. For a project whose claim is byte fidelity the line ending is the thing under test. JSON goldens survive either way..python-version, becauserequires-python = ">=3.12"letuv syncpick any interpreter and CPython is part of the oracle here —pysetreproduces its set table,pyrngitsrandom.DspyAdapteras its oracle. Nothing had ever pinned that function.Status
GATE_EXIT=0locally: 1311 Rust tests, 1035 of DSPy's own across 52 of its 94 files, surface gate OK, file sizes OK.