WIP: Support validation for buffa view types - #26
Draft
advait-specter wants to merge 7 commits into
Draft
Conversation
Rewrites the emitted rule checks so every expression reads the field through `&x[..]` instead of `.as_str()` / `.as_slice()`, and compares string constants as `&v[..] != "lit"`. The generated code currently assumes the owned message shape: `String`, `Vec<u8>`, `Vec<T>`. A slice reborrow is valid on all of those and on the borrowed equivalents (`&str`, `&[u8]`, `RepeatedView<T>`), so the same emitters can later serve both. Two shapes specifically do not work today: `str::as_str` is unstable, and a map-key binding that lands on `&&str` fails `&&str == &str`. No behaviour change — `x.as_str()` and `&x[..]` are the same borrow on every type the emitters see today. Verified with the full upstream conformance suite (2872/2872) plus `cargo test --workspace`. The `optional_string_pattern` regression tests assert on the emitted source, so they move from `is_match(v.as_str())` to `is_match(&v[..])`; what they pin is unchanged, that the pattern check borrows rather than moves.
added 3 commits
August 10, 2026 16:49
Adds `emit::Shape { Owned, View }` and passes it down through
`render_message`, `field::emit`, `oneof::emit`, and `repeated::emit_map`.
Every call site still asks for `Shape::Owned`, so the generated output is
unchanged; this only establishes where the two representations diverge.
Only three decisions read the shape:
* the `impl Validate` target — `Foo` versus
`__buffa::view::FooView<'_>`, with nested messages keeping buffa's
snake_case parent modules (`outer::Inner` -> `view::outer::InnerView`)
* the oneof enum module — `__buffa::oneof::` versus
`__buffa::view::oneof::`. The two enums share their variant names, so
the module prefix is now built once per oneof and the match arms take
a single path rather than a module + enum ident pair
* how a map field is read, via the new `MapAccess`
That last one is the only semantic difference between the shapes.
Protobuf specifies that "if there are duplicate map keys the last key
seen is used", and an owned `HashMap` gets that from the decoder. A
`MapView` is the raw wire entry list with no such step applied, so a
view body rebuilds the canonical map: one linear pass indexing each key
to its last occurrence, then pair counts and entry loops driven off that
index. Without it a payload padded with duplicate keys clears
`map.min_pairs` on the view and then fails on the owned message the
handler converts to, and per-key rules fire against values that were
overwritten and are not part of the message.
buffa ships `len_unique`/`iter_unique` for this, but both are documented
O(n^2) — fine for the serialization path they were written for, wrong
here, where the map is attacker-sized input to a validator. The index is
O(n) and costs one allocation on map fields that carry rules.
Iteration order is deliberately left alone. Protobuf leaves map ordering
undefined and the two shapes take it up differently, so normalising it
would assert something the spec does not.
Notably the CEL transpiler needs no shape awareness at all: it already
models strings as borrowed and reaches fields through `SchemaFieldKind`,
so the preceding slice-borrow change covered it.
Conformance still 2872/2872.
Every message now gets a second `impl Validate`, targeting its borrowed
view:
impl ::protovalidate_buffa::Validate for MapMin { ... }
impl ::protovalidate_buffa::Validate for __buffa::view::MapMinView<'_> { ... }
A handler that only needs to reject bad input can validate a decoded
`FooView<'_>` in place, instead of paying `to_owned_message()` for a deep
copy it is about to discard. The win is skipping the owned decode, not
faster checking — on a string-heavy message, decode+validate measures
~0.69x the owned path, while validate alone is a wash. A view still
allocates for repeated, map, and nested-message fields.
Compiling the view impls across the whole conformance cases corpus
surfaced three emitters that still assumed the owned shape, all fixed at
source:
* the optional-scalar inner binding cloned into an explicit `String` /
`Vec<u8>`; it now reborrows as a slice, which also drops a
per-validation allocation on the owned path
* map-key subscripts handed `Cow::Owned` a `&str` when the key came from
a `MapView`
* the well-known-regex predicates took `&String` closures
The wrapper inner binding likewise splits by kind — string/bytes
reborrow as a slice, other inner types are `Copy` and bind by value —
rather than cloning, which was a no-op method call on a view.
The new `views` option tracks the two buffa-build knobs that decide
whether `__buffa::view` exists at all:
views=true # default; buffa's own defaults, ungated
views=feature:<name> # crate built with gate_impls_on_crate_features,
# which wraps the view module in
# #[cfg(feature = "<name>")]
views=false # crate built with generate_views(false)
`views=feature:...` stamps the matching `cfg` on each emitted view impl
so the validators exist exactly when the types they name do; a mismatched
name fails open and the impls compile away, matching how buffa's own
gates behave.
Owned output is unchanged and conformance stays 2872/2872. `cargo test`
covers view emission via `tests/view_impls.rs`, and the duplicate-map-key
case that motivates the view-side dedup gets a behavioural test against
real generated types in the conformance crate — the upstream corpus has
no such case, so nothing else would catch a regression there.
`build.rs` now emits a second dispatch table, `dispatch_known_view`, resolving each message to its `__buffa::view::FooView` and running it through `decode_view` + `validate`. The owned and view runners share the `Result -> CaseOutcome` mapping. `registry::dispatch` runs both for every case. The owned verdict is what gets reported to the harness, but the view verdict must match it; a disagreement fails the case as a `RuntimeError` naming the message type and both outcomes. A green run therefore asserts owned/view parity across all 2872 cases on top of spec conformance. Violations are compared as a set. They carry no `PartialEq`, so their debug form stands in — it covers field path, rule path, rule id, and the key/value flags that separate otherwise-identical violations — but the comparison sorts first. Protobuf leaves map iteration order undefined and the two shapes take it up differently: an owned map follows its hasher, which is not even stable between decodes, while a view follows wire order. Comparing positions would report a divergence where the verdicts are identical. Confirmed the check actually bites: gutting the view emitter's field blocks takes the suite to `FAIL (failed: 758)` with the divergence reported per case, and reverting restores 2872/2872.
advait-specter
force-pushed
the
feat/view-validators
branch
from
August 10, 2026 23:54
09e71ed to
0e7e093
Compare
advait-specter
marked this pull request as ready for review
August 11, 2026 00:32
advait-specter
marked this pull request as draft
August 11, 2026 00:34
added 2 commits
August 10, 2026 18:57
The view body's per-map-field index now builds on `::buffa::__private::HashMap` rather than `std::collections::HashMap`, so it hashes with the same `foldhash` state buffa uses for the owned maps it decodes from the same untrusted bytes. Construction moves to `with_capacity_and_hasher`, which both the std and no_std aliases expose. Cuts view-side map validation by roughly a third. On the generated validators for `MapValues` (map<string, string>, values.string.min_len) at 4096 entries: 73.79us -> 48.22us, 5.66x owned -> 3.37x. Where a heavier per-entry rule amortizes the index build the gap is smaller still — `MapKeysPattern` goes 2.49x -> 1.82x. Both shapes stay linear in map size. `examples/map_bench.rs` times both shapes through the real generated validators and produced those numbers; run it with `cargo run --release -p protovalidate-buffa-conformance --example map_bench`.
The owned/view parity check ran only when the view dispatch table had an entry for the message, so a message present in the owned table and absent from the view one passed on the owned verdict alone. That is the one condition under which the check stops covering anything, and it was the condition that silenced it. Both tables are generated from the same validator slice, so they agree today; this keeps a future divergence loud instead of shrinking the run's coverage without saying so. Full suite and the upstream harness pass unchanged: 2872/2872 in both the default and --strict_error modes.
Replaces the throwaway `examples/map_bench.rs` with `benches/map_bench.rs`, so the numbers behind the view-vs-owned claims come from criterion's sampling rather than a hand-rolled `Instant` loop. The earlier timings were noisy enough to read as flat in `n` where the real curve narrows as the fixed index build amortizes. The crate grows a `[lib]` target. It was bin-only, so a bench could not reach the generated types without re-`include!`-ing `_include.rs` and duplicating the module scaffolding from `main.rs`. `generated`, `registry` and the harness-violation conversions move to `src/lib.rs`; `main.rs` keeps the stdin/stdout executor and `run_case`. `registry::dispatch` and `CaseOutcome` become `pub` for the binary to reach. criterion is a dev-dependency with default features off, so the plotting and gnuplot trees stay out; `cargo_bench_support` is the only feature needed. `harness = false` means `cargo test --all-targets` runs each benchmark once, which keeps it from rotting without lengthening CI noticeably — the full workspace test run is 30s. First benchmark in the repo, so there was no existing bench convention to follow. Verified with `cargo bench -p protovalidate-buffa-conformance`; fmt, clippy, the workspace tests and the upstream harness (2872/2872 in both default and --strict_error) all still pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
buffa generates a borrowed
FooView<'a>alongside every ownedFoo, but the plugin only emitted validators for the owned side. Now it emits both:Handlers can validate a decoded view in place and skip
to_owned_message().#[connect_impl]still converts to owned; switching it is a follow-up.The win is skipping the owned decode, not faster checking. Decode + validate on a string-heavy message (
repeated stringunder a per-item regex) runs at 0.71–0.75x the owned path. Validate alone is a wash — both shapes read&strat the point of comparison — and on map fields the view is slower, because it has work to do that the owned decoder already did. A view still allocates for repeated, map, and nested-message fields; only strings and bytes borrow.How it works
Both bodies come from the same emitters. Rule checks read through
&x[..], valid onString/&str,Vec<u8>/&[u8], andVec<T>/RepeatedView<T>, so only three things branch on the target shape:Foovs__buffa::view::FooView<'_>__buffa::oneof::vs__buffa::view::oneof::At
-OLLVM folds(&s[..]).len()ands.as_str().len()to the same symbol, so the rewrite costs the owned path nothing. Two bindings that used to clone — the optional-scalar inner and the wrappervalue— now borrow, so the owned path drops an allocation each.Map fields are the one semantic difference
Protobuf specifies that "if there are duplicate map keys the last key seen is used", and an owned
HashMapgets that from the decoder. AMapViewis the raw wire entry list with no such step applied, so the view body rebuilds the canonical map: one linear pass indexing each key to its last occurrence, then pair counts and entry loops driven off that index.Without it, a payload padded with duplicate keys clears
map.min_pairson the view and then fails on the owned message the handler converts to. The index hashes with::buffa::__private::HashMap, the samefoldhashstate buffa builds its owned maps with, so both shapes make the same tradeoff over the same untrusted keys.buffa ships
len_unique/iter_uniquefor this, but both are documented O(n²) — fine for the serialization path they were written for, wrong here, where the map is attacker-sized input to a validator. At 16k entries that is ~320 ms per call versus ~289 µs for the index. Filed upstream.The conformance crate grows a
[lib]target so the bench can reach the same generated types the executor validates, andbenches/map_bench.rsis the first benchmark in the repo.One index per map field, built only where the field carries rules. The owned body keeps reading the
HashMapdirectly — it shares the loop scaffolding, so its map loops now carry an.enumerate()index it ignores, which optimises out (measured below).Cost of the map path
Real generated validators, timed on
validate()alone. Two field shapes differing only in per-entry rule weight —MapValuesismap<string, string>withmap.values.string.min_len,MapKeysPatternthe same map with amap.keys.string.patternregex:Both sides are linear, so the ratio narrows as
ngrows and the fixed index build is amortized: 4.30x down to 2.47x under a cheapmin_lencheck, 1.75x down to 1.44x under a regex. This is per-map-field in isolation; end to end the decode saving dominates, hence the 0.71–0.75x above.Reproduce with
cargo bench -p protovalidate-buffa-conformance.New
viewsoptionMust match how the message crate was generated:
opt:valueviews=true(default)generate_views(true), ungatedviews=feature:NAMEgate_impls_on_crate_features(true); stamps the matching#[cfg(feature = "NAME")]on view implsviews=falsegenerate_views(false)Testing
Conformance validates every case twice — owned and view — and a disagreement fails the case on its own. 2872/2872, so a green run asserts spec conformance and owned/view parity. Verified the check bites by gutting the view emitter:
FAIL (failed: 758). A message with an owned validator but no view one is also a failure, so the check cannot go quiet by covering less.Violations compare as a set, not a sequence: map iteration order is undefined and the two shapes differ (owned follows its hasher, which isn't stable between decodes; the view follows wire order), so positional comparison would flag divergence where verdicts match.
The upstream corpus has no duplicate-key case, so that path gets its own behavioural tests against real generated types in the conformance crate.
tests/view_impls.rscovers emission in CI, and the workspace build type-checks every emitted view validator across the cases corpus.Error messages are unchanged:
--strict_messagefails the same 157 lines before and after, and--strict_errorpasses 2872/2872 on both.Generated validator code roughly doubles (70,167 → 141,267 lines over the corpus);
views=falseopts out.Owned validators are unchanged in behaviour but not byte-identical: 125 of 865 owned impls in the corpus differ textually. The changes are the slice-borrow forms (
as_str()/as_slice()→&x[..], which LLVM folds to the same symbol), an extra block scope around map bodies,.enumerate()on map loops binding an index the owned side ignores, and twoclone()calls dropped from the optional-scalar and wrapper bindings. Owned map validation measures within noise of base across 16–4096 entries — the dead counter is optimised out — and no owned impl gained or lost a rule.Lazy views (
lazy_views: true) remain unsupported — different types, fallible field access, needs its own design.Review order
6 commits, each independently green. The first two are pure refactors, so it bisects cleanly:
refactor:emit slice-borrow expressions in validator codegenrefactor:thread a targetShapethrough the emit layerfeat:emitValidateimpls for buffa view typestest:assert owned/view parity in the conformance runperf:hash the view map index with buffa's foldhash statetest:fail a case when its view validator is missing