Skip to content

[codex] Fix driver error monitoring - #999

Open
Alb3e3 wants to merge 1 commit into
txpipe:mainfrom
Alb3e3:fix-stale-ouroboros-socket
Open

[codex] Fix driver error monitoring#999
Alb3e3 wants to merge 1 commit into
txpipe:mainfrom
Alb3e3:fix-stale-ouroboros-socket

Conversation

@Alb3e3

@Alb3e3 Alb3e3 commented May 21, 2026

Copy link
Copy Markdown

Summary

  • add a shared driver monitor that polls driver tasks by completion order
  • use it from both serve and daemon so driver failures are observed promptly
  • add a regression test for a later failing driver sitting behind an earlier long-running driver

Root cause

serve and daemon collected driver join handles in a FuturesUnordered, but then awaited each join handle from the collection iterator. That made failures depend on insertion order rather than completion order. If an earlier driver kept running and a later Ouroboros Unix socket driver failed to bind, the bind error could remain unread until shutdown.

Fixes #293.

Bounty note

I linked Githoney bounty 64 from #293 with /githoney link-bounty --bountyId 64. The issue bot comment shows a 2025-01-26 deadline, so please confirm whether it is still claimable.

Verification

  • cargo test monitor_drivers_observes_later_failures_before_waiting_for_earlier_drivers --no-default-features
  • cargo test --no-default-features
  • git diff --check

I also tried cargo fmt --check, but this checkout has pre-existing rustfmt drift outside this PR on stable rustfmt, so I left those unrelated files untouched.

Summary by CodeRabbit

  • Bug Fixes

    • Improved driver failure detection so errors trigger prompt shutdown without waiting for unrelated tasks to complete.
    • Continued monitoring remaining drivers after a failure to ensure additional errors are detected and reported.
  • Refactor

    • Unified driver monitoring and error handling across service configurations, providing more consistent shutdown behavior.

Alb3e3 commented May 21, 2026

Copy link
Copy Markdown
Author

/githoney link-bounty --bountyId 64

@coderabbitai

coderabbitai Bot commented May 21, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: da858b46-8b18-4a9a-93e2-be3b38a10560

📥 Commits

Reviewing files that changed from the base of the PR and between ba00b6c and 3c982c8.

📒 Files selected for processing (1)
  • src/bin/dolos/common.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • src/bin/dolos/common.rs

📝 Walkthrough

Walkthrough

Centralizes driver task monitoring for daemon and serve entry points. The shared monitor_drivers function detects driver and join failures, logs them, cancels the exit token, and continues monitoring remaining tasks. A Tokio test validates prompt failure detection.

Changes

Driver monitoring refactor

Layer / File(s) Summary
monitor_drivers implementation and test
src/bin/dolos/common.rs
Adds monitor_drivers with completion-order task monitoring, failure logging, exit-token cancellation, and a test for later-driver failure detection.
daemon.rs integration
src/bin/dolos/daemon.rs
Replaces local driver handling with a call to monitor_drivers.
serve.rs integration
src/bin/dolos/serve.rs
Replaces local driver handling with a call to monitor_drivers.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: scarmuega

Sequence Diagram(s)

sequenceDiagram
  participant DaemonOrServe
  participant monitor_drivers
  participant DriverTasks
  participant ExitToken
  DaemonOrServe->>monitor_drivers: pass driver tasks and exit token
  monitor_drivers->>DriverTasks: await completed task
  DriverTasks-->>monitor_drivers: return success or failure
  monitor_drivers->>ExitToken: cancel when a failure occurs
  monitor_drivers->>DriverTasks: monitor remaining tasks
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: fixing driver error monitoring.
Linked Issues check ✅ Passed The monitor observes drivers by completion order and surfaces early socket bind failures, addressing issue #293.
Out of Scope Changes check ✅ Passed The changes remain focused on shared driver monitoring, error reporting, cancellation, and regression coverage.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Alb3e3
Alb3e3 marked this pull request as ready for review May 21, 2026 15:02
@Alb3e3
Alb3e3 requested a review from scarmuega as a code owner May 21, 2026 15:02
@Alb3e3
Alb3e3 force-pushed the fix-stale-ouroboros-socket branch from ba00b6c to 3c982c8 Compare August 12, 2026 11:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unable to recover after forceful shutdown

1 participant