[default values] Implement ReadBridge.apply without an Iceberg fork by cbb330 · Pull Request #679 · linkedin/openhouse

cbb330 · 2026-08-12T22:59:47Z

Summary

Depends on #678 (server handshake + drop), stacked on #674 → #668 → #645.

Finish the client half of the column-default read bridge: turn server-stamped config into Iceberg NestedField.initialDefault at metadata load — without forking Iceberg.

#678 lands first so apply is never in tree without the drop. Overlays sit on metadata.schema(); the first write after a bridged load would persist them unless the server already strips them.

Step	Where	What
Decode	`ReadBridge.from(config)` in `doRefresh` / before IO	Parse `openhouse.read-bridge.column-default.<fieldId>` (kept from #668, off the retry path)
Apply	`ReadBridge.apply(raw)` in `loadMetadata`	Overlay defaults onto every schema-id, then rebuild metadata

How apply works

Patch initial-default onto schema field objects by field-id in metadata JSON. No Iceberg fork.

Failure policy (mirrors the server encoder):

Case	Behavior
No read-bridge keys / inert bridge	Return `raw` unchanged
Field-id absent from a schema	Capability gap → leave that field NULL
Default cannot bind to column type	Fail loud (`IllegalStateException`)
Malformed known config entry	Fail loud (decode; already unrecoverable for retries via #668)
Unknown `openhouse.read-bridge.*` key	Ignore (forward compatible)

Semantics notes:

Overlay hits all schema-ids (time-travel / tags / branches), not only current.
Decode keeps validated JSON strings (shade-safe bind at apply time).
Current stamp wins — no default history; missing ids on older schemas are gaps, not errors.

Changes

New Features — ReadBridge.apply overlays initialDefault
Tests — ReadBridgeTest: single/multi-schema overlay, nested fields, gaps, fail-loud bind, inert path; OpenHouseTableOperationsTest: PUT still sends stamped initial-default

Testing Done

Added new tests for the changes made.
Updated existing tests to reflect the changes made.
Local: ./gradlew :integrations:java:iceberg-1.2:openhouse-java-itest:test --tests '*ReadBridgeTest' --tests '*OpenHouseTableOperationsTest'

Additional Information

Large PR broken into smaller PRs, and PR plan linked in the description.

Stack

#645 — substrate (seam + encode + decode hook)
#668 — decode off Iceberg’s retry path
#674 — policy / ramp in OpenHouse
#678 — handshake + drop on the server
This PR — ReadBridge.apply
#681 — Spark catalog itest (overlay)

#678) ## Summary Depends on [#674](#674) (policy), stacked on [#668](#668) → [#645](#645). [#679](#679) (`ReadBridge.apply`) restacks on this PR. Read-bridge overlays must not persist. This PR puts the drop on the **server** *before* apply lands, so there is no deployable cut where overlays are applied but not stripped. Default-aware clients send `initial-default` on stamped field-ids (the handshake). The server gates Type 1 / Type 2, then strips those keys before Iceberg commit. Unstamped ids keep the writer's defaults. Until #679, apply is still a no-op; PUTs only carry `initial-default` if the writer set it. The server is already safe: drop is a no-op when the handshake is absent, and Type 2 fail-closes unaware rewrites on ramped tables. ## How strip protection works | Gate | When | Result | |---|---|---| | Type 1 | Ramped table; a previously stamped field-id is still in the schema but missing from incoming stamps | 400 `COLUMN_DEFAULT_REMOVED` | | Type 2 | Ramped rewrite (`replace` / `overwrite` on main) while previous stamps are nonempty | 400 `COLUMN_DEFAULT_REWRITE` unless each remaining id's `initial-default` JSON equals the stamp | | Unusable | Ramped write; source throw / unreadable schema / unreadable snapshots | 400 `COLUMN_DEFAULT_UNUSABLE` | | Drop | After gates pass | Remove `initial-default` on stamped ids (same `findParents("id")` walk as the client) | Unramped tables are a no-op. GET `resolve()` still fail-opens; only the write path is fail-closed. Awareness is `JsonNode.equals` against the stamped config, not key presence. OSS never parses ASL. `ColumnDefaultsSource` is the seam (`NONE` in OSS); LinkedIn fills it. Type 1 compares stamped field-id sets from the resolver. ## Changes - [x] New Features — `ReadBridgeStripProtection.prepare` on PUT table / snapshots - [x] Internal API Changes — resolver write APIs `stampedColumnDefaults` / `isRampedForCommit` - [x] Tests — unit Type 1/2/unusable, mock bean for `MockTablesApplication`, HTTP e2e that GET `initialDefault()` is null after overlay PUT ## Testing Done - [x] Added new tests for the changes made. - [x] Updated existing tests to reflect the changes made. - Local (Java 17): `./gradlew :services:tables:test --tests '*ReadBridgeStripProtectionTest' --tests '*ReadBridgeConfigResolverTest' --tests '*ReadBridgeColumnDefaultE2ETest'` ## Additional Information - [x] Large PR broken into smaller PRs, and PR plan linked in the description. ### Stack 1. [#645](#645) — substrate (seam + encode + decode hook) 2. [#668](#668) — decode off Iceberg’s retry path 3. [#674](#674) — policy / ramp in OpenHouse 4. **This PR** — handshake on PUT, Type 1/2, drop before persist 5. [#679](#679) — `ReadBridge.apply` (merge after this PR) 6. [#681](#681) — Spark catalog itest (overlay) ### Rollback Plan Revert this PR. After #679, clients send overlays; without the server drop they persist in Iceberg metadata.

Overlay stamped column defaults onto schema field objects by field-id using the sanitize JSON helpers.

Apply puts overlays on the in-memory schema; constructMetadataRequestBody must not strip them. Skip-reload still sends the bound overlay. Unstamped writer defaults stay on the wire.

…Parents.

cbb330 mentioned this pull request Aug 12, 2026

[default values] Handshake and drop read-bridge overlays on the server #678

Merged

6 tasks

cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 27edc5d to 4f25ef4 Compare August 12, 2026 23:44

cbb330 force-pushed the chbush/read-bridge-apply branch from 4f25ef4 to e68a724 Compare August 12, 2026 23:57

cbb330 force-pushed the chbush/read-bridge-apply branch from e68a724 to 171134f Compare August 13, 2026 00:19

cbb330 force-pushed the chbush/read-bridge-apply branch from 171134f to 7829367 Compare August 13, 2026 00:37

cbb330 force-pushed the chbush/read-bridge-apply branch from 7829367 to a79c316 Compare August 13, 2026 02:36

cbb330 force-pushed the chbush/read-bridge-apply branch 3 times, most recently from 01a1468 to 3793274 Compare August 13, 2026 05:29

cbb330 mentioned this pull request Aug 13, 2026

[default values] Spark catalog itest for ReadBridge initial-default overlay #681

Open

17 tasks

cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 4c6f354 to a64ea0f Compare August 13, 2026 05:52

cbb330 force-pushed the chbush/read-bridge-apply branch from a64ea0f to 4be6d59 Compare August 13, 2026 06:10

cbb330 changed the base branch from chbush/read-bridge-sanitize-on-write to chbush/read-bridge-policy-ramp August 13, 2026 22:27

cbb330 force-pushed the chbush/read-bridge-apply branch 2 times, most recently from 08551a4 to 145297f Compare August 13, 2026 22:36

cbb330 changed the base branch from chbush/read-bridge-policy-ramp to chbush/read-bridge-sanitize-on-write August 13, 2026 22:36

mkuchenbecker reviewed Aug 13, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java Outdated

cbb330 force-pushed the chbush/read-bridge-apply branch from 145297f to 17b3d21 Compare August 13, 2026 23:56

cbb330 mentioned this pull request Aug 14, 2026

[default values] Tables: apply server-supplied per-column defaults at read time #645

Merged

2 tasks

cbb330 force-pushed the chbush/read-bridge-apply branch from 17b3d21 to d7edc72 Compare August 14, 2026 04:43

cbb330 force-pushed the chbush/read-bridge-apply branch from d7edc72 to 950f3f4 Compare August 14, 2026 20:41

cbb330 force-pushed the chbush/read-bridge-apply branch from 950f3f4 to 78dd9dd Compare August 19, 2026 20:44

cbb330 force-pushed the chbush/read-bridge-apply branch from 78dd9dd to a922b96 Compare August 20, 2026 15:08

cbb330 force-pushed the chbush/read-bridge-apply branch 5 times, most recently from 0aa2f19 to 1fd3fe2 Compare August 22, 2026 00:30

mkuchenbecker reviewed Aug 25, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java Outdated

mkuchenbecker reviewed Aug 25, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java Outdated

mkuchenbecker reviewed Aug 25, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java Outdated

cbb330 force-pushed the chbush/read-bridge-apply branch from 1fd3fe2 to 8f3f655 Compare August 25, 2026 22:59

abhisheknath2011 reviewed Aug 26, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java Outdated

cbb330 force-pushed the chbush/read-bridge-apply branch from 8f3f655 to 50ad16f Compare August 26, 2026 00:41

abhisheknath2011 reviewed Aug 26, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java Outdated

mkuchenbecker reviewed Aug 26, 2026

View reviewed changes

Comment thread ...g-1.2/openhouse-java-runtime/src/main/java/com/linkedin/openhouse/javaclient/ReadBridge.java

Base automatically changed from chbush/read-bridge-sanitize-on-write to main August 27, 2026 00:03

cbb330 added 4 commits August 26, 2026 17:03

Implement ReadBridge.apply without an Iceberg fork

0e63a85

Overlay stamped column defaults onto schema field objects by field-id using the sanitize JSON helpers.

Cover PUT still sending stamped initial-default

0408f84

Apply puts overlays on the in-memory schema; constructMetadataRequestBody must not strip them. Skip-reload still sends the bound overlay. Unstamped writer defaults stay on the wire.

Store column defaults as JsonNode and throw checked ReadBridgeException.

f93a412

Type ReadBridgeException kinds and walk schema fields instead of find…

12b56df

…Parents.

cbb330 force-pushed the chbush/read-bridge-apply branch from d0bedcb to 12b56df Compare August 27, 2026 00:03

mkuchenbecker approved these changes Aug 27, 2026

View reviewed changes

Navigation Menu

Uh oh!

[default values] Implement ReadBridge.apply without an Iceberg fork - #679

[default values] Implement ReadBridge.apply without an Iceberg fork#679
cbb330 wants to merge 4 commits into
mainfrom
chbush/read-bridge-apply

cbb330 commented Aug 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

cbb330 commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How apply works

Changes

Testing Done

Additional Information

Stack

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

cbb330 commented Aug 12, 2026 •

edited

Loading