Skip to content

[Rust][Arrow] Move explicit close into supervisor - #680

Open
teodordelibasic-db wants to merge 10 commits into
mainfrom
effort/zerobus-sdk-supervisor-owned-close
Open

[Rust][Arrow] Move explicit close into supervisor#680
teodordelibasic-db wants to merge 10 commits into
mainfrom
effort/zerobus-sdk-supervisor-owned-close

Conversation

@teodordelibasic-db

@teodordelibasic-db teodordelibasic-db commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

What changes are proposed in this pull request?

Arrow explicit close previously depended on caller-side flush completion before supervisor teardown could begin. During recovery, that allowed backoff, reconnect setup, ready-signal waiting, credential invalidation, or replay to continue until a separate timeout resolved the phase.

This PR moves explicit-close coordination and finalization into the Arrow stream supervisor. The close target and original deadline are published together while holding the ingestion mutex, and the supervisor owns the single teardown.

  • Interrupt recovery backoff, reconnect setup, ready-signal waiting, and partial replay when close is requested. Explicit close does not start or continue recovery.
  • Keep active acknowledgment processing alive through the close flush, then half-close the request and use a fresh bounded drain window even if the flush deadline has expired. A concrete peer error found during that drain takes precedence over a synthetic flush timeout.
  • Skip the remaining flush wait after the active transport has already failed, while still half-closing and draining the response for a concrete peer error.
  • Linearize close publication and terminal outcome selection under the ingestion mutex so an acknowledged close target wins over a peer error observed in the same response cycle.
  • Preserve reconnect timeout and recovery-trigger error precedence while allowing authentication rejection to remain the concrete terminal outcome.
  • Run reconnect credential invalidation in an independently owned, bounded task, including when close was already published before the authentication rejection became visible.
  • Schedule bounded terminal credential invalidation before finalization and detach it from the supervisor lifecycle so close or stream drop cannot cancel it.
  • Publish a replacement sender only after reconnect replay bookkeeping is ready, so cancellation cannot expose a sender whose transport has already been dropped.
  • Preserve a peer error already received by a server-rotation drain when close begins concurrently.
  • Keep close inside an in-progress server-rotation ACK wait, transfer ownership of the published close request into that rotation's drain, and preserve its submitted snapshot and deadlines. Latch a close deadline throughout the ACK wait and drain, including when the biased response path wins a deadline tie or ACK application yields, so a late ACK cannot retroactively turn the timed-out close into success.
  • When server rotation begins during the explicit-close flush wait, let that rotation own the single half-close and response drain, then finalize directly from its outcome.
  • Retain only the exact unacknowledged batch suffixes when close interrupts partial replay.
  • Keep repeated close calls idempotent with the original deadline and stored outcome. Polling close() publishes an irreversible close request; if that future is cancelled, ingestion remains closed and callers must invoke close() again to await completion.
  • Finalize locally with a terminal error if the supervisor exits abnormally after close is requested, so close() cannot hang and unacknowledged batches remain retrievable.
  • Store the selected outcome when close enters finalization so abnormal-exit fallback returns that exact result rather than recomputing it from the wall clock, ACK watermark, or an older server error. Keep the published server error synchronized with that selected outcome.
  • Refuse to report a clean close if teardown reaches final outcome selection with an unacknowledged target and no recorded error.
  • Serialize finalization with ingestion so accepted batches are included in the final failed-batch snapshot.

This is an internal Rust SDK lifecycle change. It does not change public APIs, FFI signatures, ABI, or semver compatibility.

Remaining intentional limitations:

  • Cleanup of a reconnect transport that does not complete setup or replay remains best-effort.
  • Concurrent ingest and close across an FFI boundary remains unsupported.
  • When a prior transport's durable outcome is unknowable, replay on a replacement stream retains unavoidable cross-stream ambiguity even though the client preserves the exact locally unacknowledged suffix.

Fixes #657.

How is this tested?

  • close_outcome_and_publication_matrix table-drives close-outcome precedence and publication/finalization transitions.
  • close_during_rotation_ack_wait_preserves_rotation_state table-drives close publication during rotation ACK waiting, including acknowledged, expired, late-ACK, and deadline-tie cases.
  • test_close_finalizes_after_supervisor_abort covers fallback both before close-state publication and after timeout selection, including late ACK accounting and exactly one half-close.
  • test_empty_close_drains_post_eof_error and test_active_error_survives_post_deadline_ack_with_exact_suffix cover bounded empty-stream draining and the active-error plus post-deadline-ACK race.
  • test_explicit_close_during_rotation_retains_buffered_suffix covers close during rotation wait/drain, exact buffered-suffix retention, and a late drain ACK that must still return DeadlineExceeded.
  • test_close_interrupts_every_recovery_phase covers setup, READY wait, replay, sender commit, and reconnect timeout; close_interrupts_backoff_and_prevents_sender_commit covers deterministic backoff interruption.
  • test_close_preserves_reconnect_auth_rejection_while_invalidation_completes covers detached authentication invalidation through stream drop.
  • test_cancelled_close_rejects_ingest_and_resumes_teardown covers cancellation-safe repeated close with the original deadline and result.
  • The deadline-validation tests reject unrepresentable ACK, flush, and recovery deadlines; pause_and_detach_waits_for_in_flight_ingest and finalize_closed_waits_for_in_flight_ingest cover ingestion/finalization serialization.

@teodordelibasic-db
teodordelibasic-db force-pushed the effort/zerobus-sdk-supervisor-owned-close branch 2 times, most recently from a4094a4 to f54f167 Compare August 6, 2026 14:33
@teodordelibasic-db teodordelibasic-db self-assigned this Aug 6, 2026
@teodordelibasic-db
teodordelibasic-db force-pushed the effort/zerobus-sdk-supervisor-owned-close branch 4 times, most recently from 210d399 to 6a51383 Compare August 7, 2026 12:16
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
@teodordelibasic-db
teodordelibasic-db force-pushed the effort/zerobus-sdk-supervisor-owned-close branch 2 times, most recently from 8426691 to fee72c8 Compare August 7, 2026 13:48
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
@teodordelibasic-db
teodordelibasic-db force-pushed the effort/zerobus-sdk-supervisor-owned-close branch from fee72c8 to 0e87af3 Compare August 7, 2026 13:54
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Signed-off-by: teodordelibasic-db <teodor.delibasic@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Rust][Arrow] Coordinate explicit close with recovery

1 participant