Skip to content

WIP: Support validation for buffa view types - #26

Draft
advait-specter wants to merge 7 commits into
mathematic-inc:mainfrom
advait-specter:feat/view-validators
Draft

WIP: Support validation for buffa view types#26
advait-specter wants to merge 7 commits into
mathematic-inc:mainfrom
advait-specter:feat/view-validators

Conversation

@advait-specter

@advait-specter advait-specter commented Aug 10, 2026

Copy link
Copy Markdown

buffa generates a borrowed FooView<'a> alongside every owned Foo, but the plugin only emitted validators for the owned side. Now it emits both:

impl Validate for MapMin { ... }
impl Validate for __buffa::view::MapMinView<'_> { ... }

Handlers can validate a decoded view in place and skip to_owned_message(). #[connect_impl] still converts to owned; switching it is a follow-up.

The win is skipping the owned decode, not faster checking. Decode + validate on a string-heavy message (repeated string under a per-item regex) runs at 0.71–0.75x the owned path. Validate alone is a wash — both shapes read &str at the point of comparison — and on map fields the view is slower, because it has work to do that the owned decoder already did. A view still allocates for repeated, map, and nested-message fields; only strings and bytes borrow.

items owned decode+validate view decode+validate ratio
8 242.73 ns 172.98 ns 0.71x
64 1.7289 µs 1.2967 µs 0.75x
512 12.923 µs 9.6724 µs 0.75x

How it works

Both bodies come from the same emitters. Rule checks read through &x[..], valid on String/&str, Vec<u8>/&[u8], and Vec<T>/RepeatedView<T>, so only three things branch on the target shape:

  • impl targetFoo vs __buffa::view::FooView<'_>
  • oneof enum module__buffa::oneof:: vs __buffa::view::oneof::
  • how a map field is read

At -O LLVM folds (&s[..]).len() and s.as_str().len() to the same symbol, so the rewrite costs the owned path nothing. Two bindings that used to clone — the optional-scalar inner and the wrapper value — now borrow, so the owned path drops an allocation each.

Map fields are the one semantic difference

Protobuf specifies that "if there are duplicate map keys the last key seen is used", and an owned HashMap gets that from the decoder. A MapView is the raw wire entry list with no such step applied, so the view body rebuilds the canonical map: one linear pass indexing each key to its last occurrence, then pair counts and entry loops driven off that index.

let mut __pv_last: ::buffa::__private::HashMap<_, usize> =
    ::buffa::__private::HashMap::with_capacity_and_hasher(self.val.len(), Default::default());
for (__pv_i, (__pv_k, _)) in self.val.iter().enumerate() {
    __pv_last.insert(__pv_k, __pv_i);
}
// map.min_pairs reads __pv_last.len()
for (__pv_i, (key, value)) in self.val.iter().enumerate() {
    if __pv_last.get(key) != Some(&__pv_i) {
        continue; // stale entry: a later one wins
    }

Without it, a payload padded with duplicate keys clears map.min_pairs on the view and then fails on the owned message the handler converts to. The index hashes with ::buffa::__private::HashMap, the same foldhash state buffa builds its owned maps with, so both shapes make the same tradeoff over the same untrusted keys.

buffa ships len_unique/iter_unique for this, but both are documented O(n²) — fine for the serialization path they were written for, wrong here, where the map is attacker-sized input to a validator. At 16k entries that is ~320 ms per call versus ~289 µs for the index. Filed upstream.

The conformance crate grows a [lib] target so the bench can reach the same generated types the executor validates, and benches/map_bench.rs is the first benchmark in the repo.

One index per map field, built only where the field carries rules. The owned body keeps reading the HashMap directly — it shares the loop scaffolding, so its map loops now carry an .enumerate() index it ignores, which optimises out (measured below).

Cost of the map path

Real generated validators, timed on validate() alone. Two field shapes differing only in per-entry rule weight — MapValues is map<string, string> with map.values.string.min_len, MapKeysPattern the same map with a map.keys.string.pattern regex:

entries owned view ratio owned (regex) view (regex) ratio
16 45.9 ns 197.1 ns 4.30x 182.9 ns 319.3 ns 1.75x
64 173.7 ns 656.8 ns 3.78x 733.3 ns 1.1338 µs 1.55x
256 766.3 ns 2.3940 µs 3.12x 2.9182 µs 4.2839 µs 1.47x
1024 3.5246 µs 9.3408 µs 2.65x 11.855 µs 16.969 µs 1.43x
4096 15.367 µs 38.030 µs 2.47x 48.482 µs 69.692 µs 1.44x

Both sides are linear, so the ratio narrows as n grows and the fixed index build is amortized: 4.30x down to 2.47x under a cheap min_len check, 1.75x down to 1.44x under a regex. This is per-map-field in isolation; end to end the decode saving dominates, hence the 0.71–0.75x above.

Reproduce with cargo bench -p protovalidate-buffa-conformance.

New views option

Must match how the message crate was generated:

opt: value Message crate built with
views=true (default) buffa defaults — generate_views(true), ungated
views=feature:NAME gate_impls_on_crate_features(true); stamps the matching #[cfg(feature = "NAME")] on view impls
views=false generate_views(false)

Testing

Conformance validates every case twice — owned and view — and a disagreement fails the case on its own. 2872/2872, so a green run asserts spec conformance and owned/view parity. Verified the check bites by gutting the view emitter: FAIL (failed: 758). A message with an owned validator but no view one is also a failure, so the check cannot go quiet by covering less.

Violations compare as a set, not a sequence: map iteration order is undefined and the two shapes differ (owned follows its hasher, which isn't stable between decodes; the view follows wire order), so positional comparison would flag divergence where verdicts match.

The upstream corpus has no duplicate-key case, so that path gets its own behavioural tests against real generated types in the conformance crate. tests/view_impls.rs covers emission in CI, and the workspace build type-checks every emitted view validator across the cases corpus.

Error messages are unchanged: --strict_message fails the same 157 lines before and after, and --strict_error passes 2872/2872 on both.

Generated validator code roughly doubles (70,167 → 141,267 lines over the corpus); views=false opts out.

Owned validators are unchanged in behaviour but not byte-identical: 125 of 865 owned impls in the corpus differ textually. The changes are the slice-borrow forms (as_str()/as_slice()&x[..], which LLVM folds to the same symbol), an extra block scope around map bodies, .enumerate() on map loops binding an index the owned side ignores, and two clone() calls dropped from the optional-scalar and wrapper bindings. Owned map validation measures within noise of base across 16–4096 entries — the dead counter is optimised out — and no owned impl gained or lost a rule.

Lazy views (lazy_views: true) remain unsupported — different types, fallible field access, needs its own design.

Review order

6 commits, each independently green. The first two are pure refactors, so it bisects cleanly:

  1. refactor: emit slice-borrow expressions in validator codegen
  2. refactor: thread a target Shape through the emit layer
  3. feat: emit Validate impls for buffa view types
  4. test: assert owned/view parity in the conformance run
  5. perf: hash the view map index with buffa's foldhash state
  6. test: fail a case when its view validator is missing

Rewrites the emitted rule checks so every expression reads the field
through `&x[..]` instead of `.as_str()` / `.as_slice()`, and compares
string constants as `&v[..] != "lit"`.

The generated code currently assumes the owned message shape: `String`,
`Vec<u8>`, `Vec<T>`. A slice reborrow is valid on all of those and on
the borrowed equivalents (`&str`, `&[u8]`, `RepeatedView<T>`), so the
same emitters can later serve both. Two shapes specifically do not work
today: `str::as_str` is unstable, and a map-key binding that lands on
`&&str` fails `&&str == &str`.

No behaviour change — `x.as_str()` and `&x[..]` are the same borrow on
every type the emitters see today. Verified with the full upstream
conformance suite (2872/2872) plus `cargo test --workspace`.

The `optional_string_pattern` regression tests assert on the emitted
source, so they move from `is_match(v.as_str())` to `is_match(&v[..])`;
what they pin is unchanged, that the pattern check borrows rather than
moves.
@advait-specter advait-specter changed the title Generate Validate Impl for Views Support validation for buffa view types Aug 10, 2026
Advait Iyer added 3 commits August 10, 2026 16:49
Adds `emit::Shape { Owned, View }` and passes it down through
`render_message`, `field::emit`, `oneof::emit`, and `repeated::emit_map`.
Every call site still asks for `Shape::Owned`, so the generated output is
unchanged; this only establishes where the two representations diverge.

Only three decisions read the shape:

* the `impl Validate` target — `Foo` versus
  `__buffa::view::FooView<'_>`, with nested messages keeping buffa's
  snake_case parent modules (`outer::Inner` -> `view::outer::InnerView`)
* the oneof enum module — `__buffa::oneof::` versus
  `__buffa::view::oneof::`. The two enums share their variant names, so
  the module prefix is now built once per oneof and the match arms take
  a single path rather than a module + enum ident pair
* how a map field is read, via the new `MapAccess`

That last one is the only semantic difference between the shapes.
Protobuf specifies that "if there are duplicate map keys the last key
seen is used", and an owned `HashMap` gets that from the decoder. A
`MapView` is the raw wire entry list with no such step applied, so a
view body rebuilds the canonical map: one linear pass indexing each key
to its last occurrence, then pair counts and entry loops driven off that
index. Without it a payload padded with duplicate keys clears
`map.min_pairs` on the view and then fails on the owned message the
handler converts to, and per-key rules fire against values that were
overwritten and are not part of the message.

buffa ships `len_unique`/`iter_unique` for this, but both are documented
O(n^2) — fine for the serialization path they were written for, wrong
here, where the map is attacker-sized input to a validator. The index is
O(n) and costs one allocation on map fields that carry rules.

Iteration order is deliberately left alone. Protobuf leaves map ordering
undefined and the two shapes take it up differently, so normalising it
would assert something the spec does not.

Notably the CEL transpiler needs no shape awareness at all: it already
models strings as borrowed and reaches fields through `SchemaFieldKind`,
so the preceding slice-borrow change covered it.

Conformance still 2872/2872.
Every message now gets a second `impl Validate`, targeting its borrowed
view:

    impl ::protovalidate_buffa::Validate for MapMin { ... }
    impl ::protovalidate_buffa::Validate for __buffa::view::MapMinView<'_> { ... }

A handler that only needs to reject bad input can validate a decoded
`FooView<'_>` in place, instead of paying `to_owned_message()` for a deep
copy it is about to discard. The win is skipping the owned decode, not
faster checking — on a string-heavy message, decode+validate measures
~0.69x the owned path, while validate alone is a wash. A view still
allocates for repeated, map, and nested-message fields.

Compiling the view impls across the whole conformance cases corpus
surfaced three emitters that still assumed the owned shape, all fixed at
source:

* the optional-scalar inner binding cloned into an explicit `String` /
  `Vec<u8>`; it now reborrows as a slice, which also drops a
  per-validation allocation on the owned path
* map-key subscripts handed `Cow::Owned` a `&str` when the key came from
  a `MapView`
* the well-known-regex predicates took `&String` closures

The wrapper inner binding likewise splits by kind — string/bytes
reborrow as a slice, other inner types are `Copy` and bind by value —
rather than cloning, which was a no-op method call on a view.

The new `views` option tracks the two buffa-build knobs that decide
whether `__buffa::view` exists at all:

    views=true             # default; buffa's own defaults, ungated
    views=feature:<name>   # crate built with gate_impls_on_crate_features,
                           # which wraps the view module in
                           # #[cfg(feature = "<name>")]
    views=false            # crate built with generate_views(false)

`views=feature:...` stamps the matching `cfg` on each emitted view impl
so the validators exist exactly when the types they name do; a mismatched
name fails open and the impls compile away, matching how buffa's own
gates behave.

Owned output is unchanged and conformance stays 2872/2872. `cargo test`
covers view emission via `tests/view_impls.rs`, and the duplicate-map-key
case that motivates the view-side dedup gets a behavioural test against
real generated types in the conformance crate — the upstream corpus has
no such case, so nothing else would catch a regression there.
`build.rs` now emits a second dispatch table, `dispatch_known_view`,
resolving each message to its `__buffa::view::FooView` and running it
through `decode_view` + `validate`. The owned and view runners share the
`Result -> CaseOutcome` mapping.

`registry::dispatch` runs both for every case. The owned verdict is what
gets reported to the harness, but the view verdict must match it; a
disagreement fails the case as a `RuntimeError` naming the message type
and both outcomes. A green run therefore asserts owned/view parity across
all 2872 cases on top of spec conformance.

Violations are compared as a set. They carry no `PartialEq`, so their
debug form stands in — it covers field path, rule path, rule id, and the
key/value flags that separate otherwise-identical violations — but the
comparison sorts first. Protobuf leaves map iteration order undefined and
the two shapes take it up differently: an owned map follows its hasher,
which is not even stable between decodes, while a view follows wire
order. Comparing positions would report a divergence where the verdicts
are identical.

Confirmed the check actually bites: gutting the view emitter's field
blocks takes the suite to `FAIL (failed: 758)` with the divergence
reported per case, and reverting restores 2872/2872.
@advait-specter
advait-specter marked this pull request as ready for review August 11, 2026 00:32
@advait-specter
advait-specter marked this pull request as draft August 11, 2026 00:34
Advait Iyer added 2 commits August 10, 2026 18:57
The view body's per-map-field index now builds on
`::buffa::__private::HashMap` rather than `std::collections::HashMap`,
so it hashes with the same `foldhash` state buffa uses for the owned maps
it decodes from the same untrusted bytes. Construction moves to
`with_capacity_and_hasher`, which both the std and no_std aliases expose.

Cuts view-side map validation by roughly a third. On the generated
validators for `MapValues` (map<string, string>, values.string.min_len) at
4096 entries: 73.79us -> 48.22us, 5.66x owned -> 3.37x. Where a heavier
per-entry rule amortizes the index build the gap is smaller still —
`MapKeysPattern` goes 2.49x -> 1.82x. Both shapes stay linear in map size.

`examples/map_bench.rs` times both shapes through the real generated
validators and produced those numbers; run it with
`cargo run --release -p protovalidate-buffa-conformance --example map_bench`.
The owned/view parity check ran only when the view dispatch table had an
entry for the message, so a message present in the owned table and absent
from the view one passed on the owned verdict alone. That is the one
condition under which the check stops covering anything, and it was the
condition that silenced it.

Both tables are generated from the same validator slice, so they agree
today; this keeps a future divergence loud instead of shrinking the run's
coverage without saying so.

Full suite and the upstream harness pass unchanged: 2872/2872 in both the
default and --strict_error modes.
@advait-specter advait-specter changed the title Support validation for buffa view types WIP: Support validation for buffa view types Aug 11, 2026
Replaces the throwaway `examples/map_bench.rs` with
`benches/map_bench.rs`, so the numbers behind the view-vs-owned claims come
from criterion's sampling rather than a hand-rolled `Instant` loop. The
earlier timings were noisy enough to read as flat in `n` where the real
curve narrows as the fixed index build amortizes.

The crate grows a `[lib]` target. It was bin-only, so a bench could not
reach the generated types without re-`include!`-ing `_include.rs` and
duplicating the module scaffolding from `main.rs`. `generated`, `registry`
and the harness-violation conversions move to `src/lib.rs`; `main.rs` keeps
the stdin/stdout executor and `run_case`. `registry::dispatch` and
`CaseOutcome` become `pub` for the binary to reach.

criterion is a dev-dependency with default features off, so the plotting
and gnuplot trees stay out; `cargo_bench_support` is the only feature
needed. `harness = false` means `cargo test --all-targets` runs each
benchmark once, which keeps it from rotting without lengthening CI
noticeably — the full workspace test run is 30s.

First benchmark in the repo, so there was no existing bench convention to
follow. Verified with `cargo bench -p protovalidate-buffa-conformance`;
fmt, clippy, the workspace tests and the upstream harness (2872/2872 in
both default and --strict_error) all still pass.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant