Skip to content

test(core): pin that append-only reads do not apply deletes - #656

Closed
linliu-code wants to merge 1 commit into
apache:mainfrom
linliu-code:test/append-only-ignores-deletes
Closed

test(core): pin that append-only reads do not apply deletes#656
linliu-code wants to merge 1 commit into
apache:mainfrom
linliu-code:test/append-only-ignores-deletes

Conversation

@linliu-code

Copy link
Copy Markdown
Contributor

Independent of the merge-on-read reader port — this is about the existing read path.

What I found

RecordMerger's append-only arm is:

RecordMergeStrategyValue::AppendOnly => record_batches.concat_data_batches(self.schema.clone())

concat_data_batches concatenates data_batches and returns. delete_batches is never read. A deleted record is still in the result.

Why that is reachable

Append-only is defensible for a table that genuinely only appends. But it is not opt-inconfig/table.rs derives it:

  • hoodie.populate.meta.fields = falseappend_only, or
  • no hoodie.table.ordering.fields / precombine.fieldappend_only

A merge-on-read table meeting either condition has log files that can carry delete blocks. Those deletes are dropped, and updates are returned as duplicates rather than merged (the existing test_merge_records_append_only already documents the duplication — 4 rows out of 2+2 with a repeated key).

Why it went unnoticed

I checked every merge-on-read fixture in the repo: all 14 set an ordering field and leave meta fields on, so none derives append-only. The path has no fixture coverage at all.

What this PR does

Adds a test pinning the current behavior, so a change to it is deliberate rather than accidental. It does not change behavior — I did not want to alter existing read results in a test-only PR, and the right fix is a design question:

  • treat a merge-on-read table with delete blocks as never append-only, or
  • apply deletes on the append-only path too, or
  • reject the combination loudly

Worth a maintainer's call. This came up because the ported merge-on-read reader has no append-only equivalent — it always merges by key — so the same table would return different rows under it.

🤖 Generated with Claude Code

The append-only merge strategy concatenates the data batches and returns, so
the delete batches are never consulted — a deleted record is still present in
the result.

That is defensible for a table that is genuinely append-only, but the strategy
is not opt-in: the table config derives it whenever meta fields are disabled or
no ordering field is set. A merge-on-read table meeting either condition has
log files that can carry delete blocks, and those deletes are dropped.

No fixture in the repo reaches this — every merge-on-read fixture sets an
ordering field and leaves meta fields on — which is why it had gone unnoticed.
The test pins the behavior so a change to it is deliberate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@linliu-code

Copy link
Copy Markdown
Contributor Author

Closing: already contained in the surviving line.

test_append_only_does_not_apply_deletes is the one difference between the old
#639-#662 tip and the squashed commit the current line starts from, so it is
present in #673 today (crates/core/src/merge/record_merger.rs). Keeping this
open would put the same test in two open PRs.

Review it in #673, or in #660 for the whole line squashed.

@linliu-code linliu-code closed this Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant