Skip to content

fix(ai): deterministic stream recovery for global assistant on mobile - #2065

Merged
2witstudios merged 21 commits into
masterfrom
fix/global-assistant-stream-recovery
Jul 14, 2026
Merged

fix(ai): deterministic stream recovery for global assistant on mobile#2065
2witstudios merged 21 commits into
masterfrom
fix/global-assistant-stream-recovery

Conversation

@2witstudios

@2witstudios 2witstudios commented Jul 14, 2026

Copy link
Copy Markdown
Owner

Summary

On mobile, when an AI stream is in progress and the app backgrounds, the stream keeps generating server-side (disconnect-immune by design). But the client couldn't rejoin it on foreground — both GlobalAssistantView.tsx and SidebarChatTab.tsx had defects in their useAppStateRecovery wiring that orphaned the stream.

This is the bug behind the "invisible AI stream on mobile" issue. PR #2061 fixed the flashing/clobbering on send/reload/switch (client-side state machine). This PR fixes the recovery path that never fired.

Root cause

Defect 1: enabled was a render-time boolean

enabled: !isStreaming && currentConversationId !== null && !useEditingStore.getState().isAnyEditing(),

iOS freezes JS the moment the app backgrounds. The boolean is captured at render — so when the app went away mid-stream, !isStreaming was false. The recovery hook was gated off in exactly the case it was written for.

The useAppStateRecovery docstring documents this exact failure:

A boolean is captured at render, and iOS freezes JS the moment the app backgrounds — so the value that ends up gating the resume is whatever was true when the app went away.

That comment was written when AiChatView.tsx was fixed. GlobalAssistantView and SidebarChatTab were never updated.

Defect 2: onResume was a dumb DB fetch

onResume: handlePullUpRefresh, // or handleAppResume

Both handlers just fetch messages from the DB and setMessages. But the reply isn't persisted until the run completes — so mid-stream the fetch returns a snapshot with no assistant message, and writing it clobbers the in-progress bubble. No local stop, no server rejoin.

The fix

  1. enabled → callback form — evaluated at fire time, gating on user-editing only, never on streaming.
  2. onResume → stop, then tryRecover() — and critically, no blind DB read.
const action = resolveResumeAction({ native: isCapacitorApp(), isStreaming: effectiveIsStreaming });
if (action === 'noop') return;
if (action === 'refresh') { await handlePullUpRefresh(); return; }  // web, idle: safe

// A stream of OUR OWN, for the conversation on screen, was running when iOS froze us.
const hadTurnInFlight = selectedAgent
  ? isOwnAgentStreamForCurrentConversation
  : isOwnGlobalStreamForCurrentConversation;
const conversationAtResume = currentConversationId;

rawStop();                              // local-only; releases the channel's `consuming` mark
let attempt = await tryRecover();       // /active-streams first; no DB read while a run is live
if (attempt.recovered) return;

// Silence is not an answer. Re-ask until BOTH questions come back, bounded.
for (let i = 1; !(attempt.probeAnswered && attempt.dbAnswered) && i <= 2; i++) {
  await delay(i * 1000);
  attempt = await tryRecover();
  if (attempt.recovered) return;
}

if (hadTurnInFlight
    && currentConversationIdRef.current === conversationAtResume
    && canConcludeTurnIsLost(attempt)) {
  await handleRetry();
}

Why tryRecover, and not a refresh

Both components already have tryRecover — the rejoin-first probe useStreamRecovery runs on a network error. A background/foreground cycle is a network error on iOS; it's just one we get told about. It asks /active-streams first — the server's authoritative answer — and only touches the DB when nothing is live:

/active-streams says action
stream still live rejoin it — no DB read at all
nothing live, reply persisted refetch the completed reply (it finished while backgrounded)
neither fall through to the plain refresh

Earlier iterations of this PR tried to make a mid-stream DB refresh safe (stale-guards, mergeServerAndPending, careful ordering). That was the wrong tree to bark up: an own stream is deliberately never in the pending-streams store while its POST body is being consumed (markChannelConsumingshouldAttachStream returns false → chat:stream_start is skipped). So on the resume path the store is empty for the very stream being recovered, the merge had nothing to merge, and the "guarded" refresh still wrote a pre-reply snapshot over the half-streamed bubble. The right answer is to not read the DB while a run is generating.

Why the rejoin must evict the local partial

The server mints one id and uses it for both the assistant UI message (generateId: () => serverAssistantMessageId) and the stream registry row. So the id useChat holds for the half-streamed bubble is the live stream's messageId. Chat.stop() is explicit that it "keeps the generated tokens", so that bubble survives the stop — and the rejoin then re-adds the same stream to the pending store under the same id. Both surfaces drop a pending stream whose messageId already appears in messages (dedupRemoteStreams, ChatMessagesArea.visibleRemoteStreams), so the rejoined stream would be filtered straight back out and not one token of it would render.

tryRecover's rejoin branch therefore reads the live messageId off /active-streams and evicts the matching local message before rejoining. (This also fixes the same latent bug on the network-error rejoin path.)

But only when the server has something to put in its place. /active-streams also returns parts — the registry's debounced checkpoint, persisted every 20 parts, so it is empty for a stream only a few parts old. (Counted with isValidPartFrame, the same predicate the bootstrap seeds with, since it is that post-filter count which becomes skipReplayCount.) Evicting against an empty checkpoint and then failing the SSE join (the documented multi-instance case — the multicast registry is per-process) makes the bootstrap remove the stream, leaving the user with nothing: strictly worse than the frozen partial. So eviction is gated on parts.length > 0; otherwise we keep the partial and still attempt the rejoin.

Why handlePullUpRefresh must NOT reconcile with the pending store

On these two surfaces the pending store is the bubble — an in-flight stream renders from remoteStreams, not from messages. mergeServerAndPending synthesizes a message under the pending stream's messageId, i.e. exactly the id the renderers dedup on. Merging here would freeze the live bubble at a static snapshot and stop it updating until completion. So this loader deliberately writes the DB snapshot as-is; the id stays absent from messages and the stream keeps rendering live.

Why silence is not an answer

"We recovered nothing" is not the same claim as "there was nothing to recover". tryRecover asks two questions, and either can come back unanswered — the first request after a foreground is the one most likely to fail, radio still coming up:

unanswered what might still exist what regenerating would do
/active-streams silent a run may still be live every generation start calls takeOverConversationStreams, so the regenerate does not race it — it aborts it: re-running write tools it already executed (a turn that created a page creates it twice), billing the discarded tokens, stranding its partial
messages GET silent the reply may already be saved handleRetry deletes the trailing assistant by id — the same id the server persisted the reply under — so it deletes the finished reply and pays for it again

So RecoveryAttempt carries probeAnswered and dbAnswered alongside recovered, each set only after a body we actually parsed, and canConcludeTurnIsLost requires both. The resume loop re-asks (bounded) until both come back. On persistent silence it does nothing — which is safe: the stop() released the consuming mark, so a live run is picked up by the socket-reconnect bootstrap once the network returns, a persisted reply is picked up by the next load, and a genuinely dead turn leaves the user their prompt and its retry action.

Was the user's turn persisted? Asked by identity, not by counting

The old guard compared user-message counts (dbUserCount >= localUserCount). That silently breaks past one page: this GET is unpaginated, so the route applies its default limit: 50 and returns only the newest 50 rows, while local messages is the initial 50 plus every turn since. Beyond that boundary the guard is permanently false — step 2 could never recover on a long conversation, and every interrupted turn there fell through to a regenerate that deleted the reply it should have refetched.

It now checks whether the DB contains the id of the last local user message. That message is by definition the newest, so it is always inside the returned window; and both routes persist the user turn under the client's id (userMessage.id || createId()), so the identity round-trips. Correct and pagination-proof.

Why the fallback is a regenerate, not a DB refresh

tryRecover returning false means one of exactly two things, and a DB write is unsafe in both:

  • the probe failed — the first request after a foreground is the likeliest to, radio still coming up. A stream may well be live, and a DB snapshot cannot contain an unpersisted reply, so a refresh would erase the in-progress bubble.
  • the DB is behind local state — step 2's dbUserCount >= localUserCount guard rejected it, e.g. a send whose POST never reached the server. A refresh would erase the user's own prompt.

But doing nothing is wrong too, and this is subtle: Chat.stop() aborts the fetch, so useChat settles at ready with no error — and useStreamRecovery only fires on status === 'error'. The stop we added therefore destroys the very signal that used to drive recovery. A turn whose POST died on the background transition would find no stream, no reply, and no error, and the user's prompt would sit unanswered forever. (Master recovered it: the dead fetch raised an error, useStreamRecovery probed, came up empty, regenerated.)

So we regenerate — the same fallback useStreamRecovery applies — but gated four ways:

  1. a turn of our own, for the conversation on screen, really was in flight (the broad effectiveIsStreaming stays true for a stream still running against a conversation the user has since left);
  2. both of tryRecover's questions came back (above);
  3. we are still on that conversation — the recovery spans seconds of network, and handleRetry always acts on the live conversation, so a switch mid-recovery would otherwise fire a generation for the turn the user moved to;
  4. nothing has already restarted the turn. useStreamRecovery watches the same failure from the other side and regenerates too — and it calls clearError() before its own probes, so throughout its decision window the status reads ready and looks idle. A single regenerationInFlightRef mutex wraps handleRetry (regenerateTurnOnce), and both paths go through it, so the same turn can never be regenerated twice; a !isStreamingRef.current belt then stops us regenerating on top of a turn that has already restarted and moved on (the mutex releases when handleRetry's DELETEs finish, but the generation it kicked off is still running).

Why the stop must come first

rawStop() / stop() is the local-only useChat stop. It does not signal the server (that's abortActiveStreamByMessageId), so the run keeps generating and stays rejoinable. But it also ends the dead response body, which releases the channel's consuming mark — and without that, the rejoin's bootstrap classifies the stream as one this tab is already reading off its POST body and skips attaching it. The rejoin would silently do nothing.

Ordering is safe because both tryRecover and the bootstrap await their /active-streams fetch before consulting isChannelConsuming, so the abort-triggered unmark (a microtask) always lands first.

Changes

File Change
GlobalAssistantView.tsx enabled → callback; onResume → stop + tryRecover (no DB fallback); tryRecover evicts the stale partial on rejoin (gated on the server checkpoint being non-empty, and written through the store-syncing setter); handlePullUpRefresh gains a live-conversation stale-guard
SidebarChatTab.tsx Same resume wiring (hook relocated below tryRecover, which it now closes over); same eviction; handleAppResume funnels through loadGlobalMessages (the documented single writer) instead of its own raw fetch
lib/ai/streams/canResumeRecovery.ts new — the resume gate, extracted. Takes no streaming argument by construction
lib/ai/streams/evictStalePartial.ts new — the eviction rule, extracted (including the checkpoint guard, counted with the bootstrap's own isValidPartFrame)
lib/ai/streams/recoveryAttempt.ts newRecoveryAttempt + canConcludeTurnIsLost
all __tests__ +50 tests. The gate, the eviction and the regenerate predicate are asserted against the real implementations, not mirrors

Tests

The resume wiring previously had zero coverage on either surface — which is how the render-time enabled boolean survived. Following each file's established pattern (pure mirrors of the hook-heavy component's logic), the decision is not mirrored: planResume calls the real resolveResumeAction, so the tests pin the wiring against the real policy.

  • the gate takes no streaming argument — the original regression is not expressible in it
  • native + live stream → stoptryRecover, and the DB is never read
  • native → never falls back to a DB refresh (the two cases where tryRecover returns false are exactly the two where a DB write is unsafe)
  • stop always precedes the probe (it's what releases the consuming mark)
  • native + idle → still probes (deterministic, not flag-gated)
  • web + live fetch → noop (a live fetch survives a tab switch)
  • web + idle → refresh only
  • the rejoin evicts the local partial carrying the live messageId — and only that one message, never the user's turn
  • eviction is skipped entirely when the server's checkpoint is empty (keep what the user has rather than risk an empty screen)
  • regenerate fires only when both questions were answered, for a turn of ours, on the conversation still on screen — never on silence, never on a foreign conversation's stream, never after a mid-recovery switch

Validation

  • bun run typecheck — clean
  • bun run build (apps/web) — exits 0
  • bunx eslint on all touched files — clean
  • vitest — 815 passing across components/layout + lib/ai/streams

Behaviour change worth calling out

Replacing the native DB refresh with a regenerate means two narrow cases no longer refresh on resume: a conversation whose last DB message is a user message where no turn was in flight, and another device having deleted the trailing assistant message. Both are healed by the socket-reconnect refreshSignal that fires on resume anyway.

On the tests

The resume wiring previously had zero coverage on either surface — which is how the render-time enabled boolean survived in the first place.

The three load-bearing rules are now shared pure modules, so the tests assert against the shipped implementation rather than a copy of it: canResumeRecovery, evictStalePartial, and canConcludeTurnIsLost. planResume remains a mirror of the sequencing only (the onResume body is genuinely imperative — stop → probe → bounded re-probe → regenerate), but its two decisions call the real resolveResumeAction and the real canConcludeTurnIsLost.

Known follow-ups (deliberately not in this PR)

  • resolveResumeAction treats all Capacitor platforms as "the fetch is always dead"isCapacitorApp() is true on Android too, so a healthy Android stream gets stopped and rejoined on every foreground ≥5s. It renders correctly now that the eviction is in place, but it is needless churn. Pre-existing shared policy (AiChatView already does this); worth gating on platform === 'ios' separately.
  • AiChatView still does stop → rejoin → refresh, and carries the same dedup collision on its rejoin path. The same fix applies. Left alone rather than widening this PR's blast radius.

Refs: #2061, #2018

@coderabbitai

coderabbitai Bot commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

App resume recovery now dynamically gates execution, resolves a resume action, protects asynchronous message loading, conditionally evicts stale stream messages, and rejoins agent or global streams across both assistant views.

Changes

App resume recovery

Layer / File(s) Summary
Global assistant resume and refresh flow
apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx
Global recovery uses live conversation checks, deterministic native/web resume actions, guarded message refreshes, and conditional stale-message eviction before stream reattachment.
Sidebar resume and rejoin flow
apps/web/src/components/layout/right-sidebar/ai-assistant/SidebarChatTab.tsx
Sidebar recovery awaits the shared global message loader, dynamically gates resume handling, and rejoin-probes agent or global streams after conditional stale-message eviction.
Resume recovery regression coverage
apps/web/src/components/layout/middle-content/page-views/dashboard/__tests__/GlobalAssistantView.test.tsx, apps/web/src/components/layout/right-sidebar/ai-assistant/__tests__/SidebarChatTab.test.tsx
Tests cover resume gating, action ordering, native/web behavior, and preservation or eviction of stale assistant partials based on server stream parts.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AppStateRecovery
  participant ResumeResolver
  participant TryRecover
  participant MessageStore
  AppStateRecovery->>ResumeResolver: Resolve native and streaming state
  ResumeResolver-->>AppStateRecovery: Return resume action
  AppStateRecovery->>TryRecover: Stop locally and recover
  TryRecover->>MessageStore: Probe live stream and conditionally evict stale message
  TryRecover-->>AppStateRecovery: Rejoin stream or load persisted messages
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: deterministic mobile stream recovery for the global assistant.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/global-assistant-stream-recovery

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bb8b15ac02

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

// Block recovery if streaming OR pending send OR any editing active
enabled: !isStreaming && currentConversationId !== null && !useEditingStore.getState().isAnyEditing(),
onResume: useCallback(async () => {
const action = resolveResumeAction({ native: isCapacitorApp(), isStreaming: displayIsStreaming });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Import the resume helpers in SidebarChatTab

This new resume handler calls resolveResumeAction and isCapacitorApp, but SidebarChatTab.tsx does not import either helper. Any build or render path that reaches this component now fails on undefined identifiers instead of mounting the sidebar/recovery hook; add the same imports used by the other chat views before using them here.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in c2cd36d — both imports were genuinely missing. Added to SidebarChatTab.tsx:

import { isCapacitorApp } from '@/hooks/useCapacitor';
import { resolveResumeAction } from '@/lib/ai/streams/resolveResumeAction';

Same two imports AiChatView.tsx uses. This was the other half of the web#build failure.

While confirming the fix I also checked the stop this handler calls: it comes from usePageAgentSidebarChat, which resolves to the raw mode-selected useChat stop (selectedAgent ? agentStop : globalStop) — i.e. local-only, matching the chatStop semantics in AiChatView. It clears local useChat state so the rejoin attaches cleanly and does not signal the server, so the server-owned run keeps generating and is rejoined rather than killed. Documented that in a comment at the call site, since it is the non-obvious part.

Verified locally: tsc --noEmit clean, bun run build exits 0, lint clean.

effectiveIsStreaming,
selectedAgent,
agentStop,
rejoinAgentStream,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Declare the rejoin callback before the resume hook

This dependency array is evaluated during render, but rejoinAgentStream is only initialized later by useAgentChannelMultiplayer in the same component. Rendering GlobalAssistantView therefore hits the temporal dead zone before the later hook runs; move the multiplayer hook above this useAppStateRecovery call or use the existing ref when wiring the resume callback.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed and fixed in c2cd36d — this was a real build break, not a nit: CI failed web#build with Type error: Block-scoped variable 'rejoinAgentStream' used before its declaration.

Fixed by routing the agent rejoin through the existing rejoinAgentStreamRef — declared near the top of the component and assigned immediately after useAgentChannelMultiplayer runs. That ref exists for exactly this hook-ordering problem and is already the in-file idiom: tryRecover calls rejoinAgentStreamRef.current() the same way. I preferred it over reordering the hooks, since hoisting useAgentChannelMultiplayer would drag its dependencies (setAgentMessages, loadConversation, …) up with it for no benefit.

Also collapsed the agent/global stop branch to the already-mode-selected rawStop.

Verified locally: tsc --noEmit clean and bun run build exits 0 (both previously failed).

@2witstudios
2witstudios force-pushed the fix/global-assistant-stream-recovery branch from 82fd84b to 710eb7f Compare July 14, 2026 17:53

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx (1)

710-751: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Resume-recovery logic is duplicated with SidebarChatTab.tsx.

This entire block (comments, resumeEnabled, onResume action resolution and rejoin/refresh sequencing) is near-identical to SidebarChatTab.tsx Lines 622-662. The two already diverged once (this file needed a ref to fix a TDZ that the sidebar never had), which is exactly the kind of drift risk duplicated logic creates. Consider extracting a shared useResumeStreamRecovery hook (native check, action resolution, stop/rejoin/refresh sequencing) parameterized by stop, rejoinAgent, rejoinGlobal, refresh, isStreaming, selectedAgent.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx`
around lines 710 - 751, The resume-recovery implementation around resumeEnabled
and the useAppStateRecovery onResume callback duplicates SidebarChatTab.tsx and
should be centralized. Extract a shared useResumeStreamRecovery hook that owns
native action resolution, enabled gating, stop, agent/global rejoin, and refresh
sequencing, parameterized by stop, rejoinAgent, rejoinGlobal, refresh,
isStreaming, and selectedAgent; then replace both component-local blocks with
the hook while preserving the ref-based agent rejoin behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In
`@apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx`:
- Around line 710-751: The resume-recovery implementation around resumeEnabled
and the useAppStateRecovery onResume callback duplicates SidebarChatTab.tsx and
should be centralized. Extract a shared useResumeStreamRecovery hook that owns
native action resolution, enabled gating, stop, agent/global rejoin, and refresh
sequencing, parameterized by stop, rejoinAgent, rejoinGlobal, refresh,
isStreaming, and selectedAgent; then replace both component-local blocks with
the hook while preserving the ref-based agent rejoin behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6926b1ce-08db-4c3f-b511-f486e6395731

📥 Commits

Reviewing files that changed from the base of the PR and between 2e48bc4 and c2cd36d.

📒 Files selected for processing (2)
  • apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx
  • apps/web/src/components/layout/right-sidebar/ai-assistant/SidebarChatTab.tsx

@2witstudios

Copy link
Copy Markdown
Owner Author

Self-review found a blocker the PR itself introduced — fixed in 496e0a7

Running an adversarial pass over my own diff surfaced that this PR was fixing only half of its own stated "Defect 2". It added the stop+rejoin, but left the DB refresh as the same dumb, unguarded fetch — and then wired that fetch to fire on the rejoin path, where it is at its most dangerous.

The reply is not persisted until the run completes, so a refresh issued while a stream is live returns a snapshot with no assistant message. Two distinct problems fell out of that:

1. The write was unguarded.
GlobalAssistantView.handlePullUpRefresh was a raw fetchWithAuth(...)setMessages(data.messages). SidebarChatTab.handleAppResume did its own raw fetch, bypassing loadGlobalMessages — the documented "single writer for the global-mode server→view path" sitting directly above it. Neither carried the two guards that make a mid-stream DB read safe, and which every other loader in this codebase has (AiChatView.loadMessagesForConversation is explicit that it exists so "there is never a competing write from a stale in-flight fetch"):

  • shouldApplyLoadedMessages — drop a response that arrives after a conversation switch
  • mergeServerAndPending — re-attach the in-flight assistant bubble, which is by definition absent from the DB snapshot mid-run

Both paths now funnel through guarded, reconciled loaders.

2. The order was racy.
The refresh was fired after the rejoin, so its request could still be in flight when the rejoined stream completed — landing last and overwriting the just-completed reply with the pre-reply snapshot.

Concretely (agent mode, the worst variant): resume at T issues the GET; the rejoined stream completes at T+150ms and renders the reply; the GET from T resolves at T+800ms (first request after a mobile resume is routinely slow) and wipes it. Global mode self-heals via the completion→refreshSignal refetch. Agent mode does not — there is no completion-triggered refetch there, so the reply stays invisible until the user reselects the conversation.

The order is now stop → await refresh → rejoin, so no DB request is ever outstanding when the stream completes: the refresh catches a run that finished while backgrounded, and the rejoin attaches to one that is still live.

Also

  • The gate's regression test was vacuous (it asserted the same thing as the test above it, against the test's own copy of the logic). It now asserts on resumeEnabled.length, so an isStreaming parameter cannot be reintroduced silently.
  • Added ordering tests pinning refresh strictly before rejoin.

Known follow-up (not this PR)

AiChatView has the same stop → rejoin → refresh ordering and therefore the same latent completion race, though it self-heals because its loader is the authoritative single writer. Worth aligning it to this order separately rather than expanding this PR's blast radius.

Validation

typecheck clean · bun run build exits 0 · eslint clean on all touched files · 291 tests green across the 34 affected suites · CI green.

@2witstudios

Copy link
Copy Markdown
Owner Author

Rewrote the fix: resume now goes through tryRecover, not a DB refresh (2fe46d1)

A second self-review round found my previous approach was still wrong — and in a way that made the primary symptom worse. Recording it here because the reasoning matters for review.

What was wrong

I had been trying to make a mid-stream DB refresh safe: stale-guards, mergeServerAndPending, careful stop→refresh→rejoin ordering. All of that was barking up the wrong tree.

An own stream is deliberately never in the pending-streams store while its POST body is being consumed — markChannelConsuming runs synchronously before the POST leaves, and shouldAttachStream({isOwn, isConsuming}) then returns false, so chat:stream_start is skipped and no store entry is ever created.

So on the resume path the store is empty for the very stream we are recovering. mergeServerAndPending had nothing to merge and was inert. The "guarded" refresh still wrote a pre-reply DB snapshot straight over the half-streamed assistant bubble — the partial reply would vanish for a full round-trip until the bootstrap re-seeded it. I had moved the bug, not fixed it.

What it does now

The mistake was reaching for a DB read at all. Both components already have tryRecover — the rejoin-first probe useStreamRecovery uses on a network error. A background/foreground cycle is a network error on iOS; it's just one we get told about.

if (action === 'rejoin-and-refresh') {
  rawStop();
  if (await tryRecover()) return;   // rejoined a live stream, or refetched a persisted reply
}
await handlePullUpRefresh();        // only when nothing was live and nothing persisted

tryRecover asks /active-streams first — the server's authoritative answer — and only touches the DB when nothing is live. The DB is never read while a run is still generating. No bespoke ordering, no inert merge, and it reuses logic that already exists and is already exercised on the network-error path.

The subtle part: why stop() must come first

stop() is the local-only useChat stop — it does not signal the server, so the run keeps generating and stays rejoinable. But it also ends the dead response body, which is what releases the channel's consuming mark. Without that release, the rejoin's bootstrap classifies the stream as one this tab is already reading off its POST body and skips attaching it — the rejoin would silently do nothing and the whole recovery would be a no-op.

That's safe here because both tryRecover and runBootstrap await their /active-streams fetch before consulting isChannelConsuming, so the abort-triggered unmark (a microtask) always lands first. Worth knowing if anyone reorders this later.

Also

  • Dropped a vacuous test assertion I'd added (it asserted the arity of the test file's own helper — it could never fail).
  • handlePullUpRefresh keeps its stale-guard + reconciliation: it's still reachable from the pull-up and refreshSignal paths, where a late-landing load can clobber a conversation the user has since switched away from. The stale-guard there now compares against the live conversation, because the "id the load was requested for" ref pattern I first copied from AiChatView is vacuous in this component — nothing else advances that ref here, since this surface doesn't load-on-select through handlePullUpRefresh.

Validation

typecheck clean · bun run build exits 0 · eslint clean · 289 tests green across the 34 affected suites.

@2witstudios

Copy link
Copy Markdown
Owner Author

Round 3 found the PR didn't actually work — fixed in 5fefc3c

Third adversarial pass. The rejoin was attaching correctly and then rendering nothing. Two findings, both real.

1. Dedup collision — the blocker

The server mints one id and uses it for both the assistant UI message and the stream registry row:

const serverAssistantMessageId = createId();      // app/api/ai/global/[id]/messages/route.ts:1027

messageId: serverAssistantMessageId,               // :1089  → aiStreamSessions row
generateId: () => serverAssistantMessageId,        // :1119  → the UI message

So the id useChat holds for the half-streamed bubble is the live stream's messageId. And Chat.stop() is explicit — "Abort the current request immediately, keep the generated tokens if any." The bubble survives the stop.

The rejoin then re-adds that same stream to the pending store under the same id — and both surfaces drop a pending stream whose messageId already appears in messages (dedupRemoteStreams, ChatMessagesArea.visibleRemoteStreams). The rejoined stream was filtered straight back out.

Net effect: we stopped the fetch, rejoined the server stream, and then displayed zero of its tokens. The user would sit in front of a frozen partial reply until completion. The PR was a no-op on its own headline path.

tryRecover's rejoin branch now reads the live stream's messageId off /active-streams and evicts the matching local message before rejoining. Nothing is lost — the bootstrap seeds the stream from the server's registry buffer, which holds every part pushed so far, i.e. strictly more than the partial we froze with. This fixes the same latent bug on the network-error rejoin path too.

2. The DB fallback fired in exactly the cases where a DB write is unsafe

onResume fell through to a blind refresh whenever tryRecover returned false. But false means one of:

  • the /active-streams probe failed — and the first request after a foreground is the likeliest one to fail, radio still coming up. A stream may well be live, and a DB snapshot cannot contain an unpersisted reply, so the refresh erased the in-progress bubble.
  • the DB is behind local state — step 2's dbUserCount >= localUserCount guard rejected it, e.g. a send whose POST never reached the server. The refresh erased the user's own prompt.

The native path now ends at tryRecover, which already refetches when the run finished while we were away. In the other two cases doing nothing is correct: local state is newer than anything we could fetch, and the socket reconnect / refreshSignal path heals it.

Tests

Added coverage pinning the eviction (the dedup collision) and asserting the native path never contains a refresh step. 116 tests green across the two component suites; 770 across all of components/layout + lib/ai/streams.

Validation

typecheck clean · bun run build exits 0 · eslint clean.

Known follow-ups (deliberately not in this PR)

  • resolveResumeAction treats all Capacitor platforms as "fetch is always dead"isCapacitorApp() is true on Android too, so a healthy Android stream gets stopped and rejoined on every foreground ≥5s. It still renders correctly now that the eviction is in place, but it's needless churn. This is pre-existing shared policy (AiChatView already does it); worth gating on platform === 'ios' separately.
  • AiChatView still does stop → rejoin → refresh and carries the same dedup collision on its rejoin path. Same fix applies; left alone rather than widening this PR's blast radius.

@2witstudios

Copy link
Copy Markdown
Owner Author

Round 4: the eviction was being undone, and could leave the screen empty — fixed in 1339151

1. mergeServerAndPending re-inserted the very id the eviction removes

This one was a regression I introduced myself two commits ago, while trying to make a mid-stream DB read safe.

mergeServerAndPending synthesizes an assistant message under the pending stream's messageId when that id is absent from the DB snapshot — which is precisely the id both renderers dedup on. So:

  1. Resume → stop → tryRecover evicts the partial → rejoin. The pending stream renders live. useChat is idle, so isOwnGlobalStreamForCurrentConversation is now false.
  2. The socket reconnects (near-certain right after an iOS foreground) → refreshSignal bumps.
  3. The refreshSignal effect's only guard is !isOwnGlobalStreamForCurrentConversation → passes → handlePullUpRefresh() runs.
  4. My merge appends a frozen snapshot of ownStream.parts under the live id into messages.
  5. visibleRemoteStreams drops the live stream. The bubble stops updating — the exact symptom the eviction exists to prevent.

The insight I'd missed: on these two surfaces the pending store is the bubble — an in-flight stream renders from remoteStreams, not from messages. Merging a static copy into messages isn't needed for visibility and actively kills the live render. The merge is removed, which also restores master's behaviour here (master never had it).

2. Evicting against an empty checkpoint could leave the user with nothing

/active-streams returns parts — but that's the registry's debounced checkpoint, persisted every PERSIST_EVERY_N_PARTS = 20 parts. It is empty for a stream only a few parts old.

If we evicted the local partial and the SSE join then failed — the documented multi-instance case, since the multicast registry is per-process and prod runs several web instances — the bootstrap calls removeStream(messageId) when skipReplayCount === 0. Partial evicted, stream removed: the assistant bubble disappears entirely. Strictly worse than the frozen partial we started with.

Eviction is now gated on the server actually having parts to put in its place. Otherwise we keep the partial and still attempt the rejoin — worst case the user keeps exactly what they had.

I'd also written a comment claiming the bootstrap "seeds the stream from the server's registry buffer, which holds every part pushed so far — strictly more than the partial we froze with." That conflated the in-memory multicast buffer with the DB parts column. It was wrong and is corrected.

3. Eviction now goes through the store-syncing setter (fixed in 4278114)

Agent mode must drop the message from the dashboard store too, not just useChat — the store is the handoff payload to the sidebar on navigation, so a stale partial there re-creates the dedup collision on the other surface. Now uses the existing agentSetMessages wrapper, matching how tryRecover's step 2 already writes both.

Behaviour change worth calling out

Removing the native DB fallback means two narrow cases no longer refresh on resume: a conversation whose last DB message is a user message (an aborted run, or another device's turn whose reply hasn't started), and another device having deleted the trailing assistant message. Both are healed by the socket-reconnect refreshSignal that fires on resume anyway. I've corrected the comment that overclaimed the only skipped cases were unsafe ones.

Validation

typecheck clean · bun run build exits 0 · eslint clean · 770 tests green across components/layout + lib/ai/streams.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx`:
- Around line 780-833: Extract shared resumeEnabled and stale-partial eviction
helpers, then replace all local mirrors with imports so production code and
tests exercise the same implementations. In
apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx:780-833
and
apps/web/src/components/layout/right-sidebar/ai-assistant/SidebarChatTab.tsx:1009-1067,
import and use the shared resumeEnabled while preserving the callback’s
conversation/editing checks. In
apps/web/src/components/layout/middle-content/page-views/dashboard/__tests__/GlobalAssistantView.test.tsx:115-162
and 334-360, and
apps/web/src/components/layout/right-sidebar/ai-assistant/__tests__/SidebarChatTab.test.tsx:133-180
and 613-639, remove the resumeEnabled/ResumeEffect/planResume and
evictStalePartial mirrors and import the shared implementations used by the
views and tryRecover.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 3756dac7-a648-4a24-b4c2-14990119204c

📥 Commits

Reviewing files that changed from the base of the PR and between c2cd36d and 1339151.

📒 Files selected for processing (4)
  • apps/web/src/components/layout/middle-content/page-views/dashboard/GlobalAssistantView.tsx
  • apps/web/src/components/layout/middle-content/page-views/dashboard/__tests__/GlobalAssistantView.test.tsx
  • apps/web/src/components/layout/right-sidebar/ai-assistant/SidebarChatTab.tsx
  • apps/web/src/components/layout/right-sidebar/ai-assistant/__tests__/SidebarChatTab.test.tsx

@2witstudios

Copy link
Copy Markdown
Owner Author

Round 5 — happy path verified clean; two fixes in e6ac419

Fifth adversarial pass traced the full native path end-to-end (background mid-stream → foreground → stop → probe → evict → rejoin → bootstrap → SSE replay → render → completion) and verified it correct, including:

  • the consuming mark is genuinely released before the rejoin (stop() → tracked body cancel → finishOnce()unmarkChannelConsuming, and tryRecover awaits a real network hop before rejoining, so the ordering is load-bearing and holds);
  • skipReplayCount / persistedParts reconcile exactly — seeded checkpoint + un-skipped remainder = the full buffer, no gaps, no dupes;
  • the completed reply lands exactly once in both modes (agent: onStreamComplete replaces-by-id, and since we evicted, appends; global: refreshSignal → DB refetch, store entry already removed);
  • the eviction is safe on useStreamRecovery's original network-error path too.

Two things still needed fixing.

1. The stop suppressed the fallback — a regression against master

Chat.stop() aborts the fetch, so useChat settles at ready with no error. But useStreamRecovery only fires on status === 'error':

if (!error || status !== 'error' || isRetryingRef.current) return;

So the stop() this PR added destroys the very signal that used to drive recovery — and tryRecover()'s false return was being discarded on top of that.

Failure: iOS backgrounds during the TTFB window and the POST never reaches the server (a radio drop right on the background transition is common). On resume: stop aborts the dead POST → /active-streams reports nothing → the DB has neither the reply nor the user's turn → tryRecover() returns false → nothing happens. The user's prompt sits unanswered forever, no error, no retry.

Master handled this: the dead fetch raised an error → useStreamRecovery → probe → empty → handleRetry() → recovered.

onResume now falls through to handleRetry() when tryRecover comes up empty — the same fallback the network-error path uses. Still not a DB refresh (that remains unsafe here, for the reasons documented at the call site). Gated on a turn actually having been in flight when we backgrounded, so an ordinary resume on an idle conversation can never fire a spurious generation — the frozen render-time streaming flag is a faithful record of exactly that, which is all it's used for.

I'd also written that the reconnect/refreshSignal path "heals" every skipped case. It doesn't heal a run that never started server-side. Comment corrected.

2. The eviction gate counted parts differently from the seeder

The gate used raw liveStream.parts.length, but the bootstrap seeds (stream.parts ?? []).filter(isValidPartFrame) — and it's that count which becomes skipReplayCount, and a skipReplayCount === 0 is what makes a failed join drop the stream. A checkpoint of malformed frames would read as "safe to evict" while seeding nothing, and a failed SSE join would then leave an empty screen: precisely what the gate exists to prevent. Both now use isValidPartFrame.

Validation

typecheck clean · bun run build exits 0 · eslint clean · 770 tests green (120 in the two component suites, +4 pinning the regenerate fallback and the no-spurious-generation guard).

@2witstudios

Copy link
Copy Markdown
Owner Author

Round 6 — the regenerate fallback could destroy a healthy run. Fixed in 25746fb (+1e13c8d15)

Two fixes to the fallback I added last round.

1. Silence is not an answer (ship-blocker)

tryRecover returns false for two semantically opposite outcomes:

  • "the server says nothing is live" → the run died; regenerating is the recovery.
  • "the probe never reached the server" → we know nothing; a run may well still be live.

I was treating the second as if it were the first. And on this path the second is common, not exotic — the first request after an iOS foreground is the one most likely to fail, with the radio still coming up.

It matters because every generation start calls takeOverConversationStreams. So a regenerate issued while the run is in fact still live does not race it — it aborts it. The fallback would have:

  1. killed a healthy, possibly nearly-finished generation and re-run it from scratch;
  2. re-run write tools that had already executed — a turn that created a page creates it twice; the side effects are not rolled back;
  3. billed the discarded tokens;
  4. stranded an orphan partial in the DB — handleRetry deletes the trailing assistant before POSTing, but the aborted run's onFinish persists its partial after that delete.

tryRecover now records reachability separately from its verdict (only a 2xx means the server actually told us what is live). onResume re-probes twice with backoff before concluding — the radio returns well inside that — and regenerates only on an answered probe.

On persistent silence it does nothing, which is safe: the stop() released this channel's consuming mark, so a live run is picked up by the socket-reconnect bootstrap once the network returns, and a dead one leaves the user their prompt and the retry action on it.

2. The regenerate gate was not conversation-scoped

effectiveIsStreaming / displayIsStreaming stay true for a stream still running against a conversation the user has since navigated away from (the useChat id is stable across a switch). Regenerating on that would have fired a generation for the turn the user is now looking at rather than the one that was interrupted — a spurious reply, and a spurious charge, on an untouched conversation.

Both files already carry the conversation-scoped flag for exactly this hazard (isOwnAgentStream/isOwnGlobalStreamForCurrentConversation, isOwnStreamForCurrentConversation — each latching the conversation the stream actually started in). The gate now uses it.

Verified sound (not defects)

  • No concurrent double-generation. takeOverConversationStreams runs on every generation start, so a regenerate takes over rather than running beside the live stream.
  • handleRetry is safe on a stale partial — it strips trailing assistant messages, and AI SDK's regenerate() trims one too. It does not duplicate the user turn.
  • No retry loop. The stop() flips useChat to ready, so the conversation-scoped gate is false on the next resume — it self-clears, and handleRetry cannot fire twice for the same interrupted turn.

Tests

planResume now models ownTurnInFlight and probeAnswered as inputs distinct from the broad streaming flag. Both gates were previously unfalsifiable — reverting either would have kept the suite green. 124 tests in the two component suites, 772 across components/layout + lib/ai/streams.

Validation

typecheck clean · bun run build exits 0 · eslint clean on all touched files.

@2witstudios

Copy link
Copy Markdown
Owner Author

Round 7 — the probe signal was leaky three ways. Fixed in 9bb092f

All three landed on the same failure: regenerating over a healthy run, which takeOverConversationStreams turns into an abort of it (re-running its already-executed write tools, billing its discarded tokens, stranding its partial).

  1. probeAnsweredRef.current = res.ok was set before the body was parsed. res.json() can still throw on a body that dies mid-read — exactly the cold-radio-after-foreground case this path exists for — and the catch swallowed it while we went on believing we had an answer. Now set only after a body we actually parsed.

  2. One shared mutable slot, two callers. tryRecover is called by both the resume handler and useStreamRecovery's network-error retry, and they can be in flight together — so one caller's probe could answer the other caller's question.

  3. The reset sat after tryRecover's early returns. A bail-out (!channelId — e.g. user transiently null while auth rehydrates on foreground, and the token very likely expired while backgrounded) left the ref holding a previous call's true, and the resume path would regenerate on the strength of a probe it never made.

All three are gone by construction: tryRecover now returns { recovered, probeAnswered } (RecoveryAttempt), so each caller reads its own probe and there is no shared state to race or go stale. useStreamRecovery gets a thin adapter reading only .recovered — the reachability half of the answer exists for the resume path, which is the one that would otherwise regenerate over a live run.

Also fixed: tryRecover step 2 read defensively (Array.isArray(data) ? data : data.messages) but wrote data.messages raw, which would blank the surface outright if the route ever answered with a bare array. It now writes the same normalized array its guards were computed from.

Verified sound this round

  • The re-probe loop cannot spin, hang, or double-fire: bounded at 3 probes / ~3s, every success path returns, and useAppStateRecovery's pendingRefreshRef serializes resumes.
  • Re-running step 2 per attempt is harmless — a pure GET plus one full-replace write that returns the instant it applies.
  • No stale closures, no TDZ, deps complete.

Tests

planResume now takes a sequence of attempts, so the re-probe loop is actually modelled: a probe that only answers on a later attempt still regenerates (a cold radio must not strand the turn), one that never answers never does, and a re-probe that finds the live stream rejoins instead. 128 tests in the two component suites; 782 across components/layout + lib/ai/streams.

Validation

typecheck clean · bun run build exits 0 · eslint clean on all touched files.

@2witstudios

Copy link
Copy Markdown
Owner Author

Round 8 — the fallback could DELETE a completed reply. Fixed in 264055d

Three defects; two of them combine into the worst thing this PR could do.

1. Silence from the DB check was read as "nothing is persisted"

Round 7 taught the /active-streams probe to distinguish "the server says nothing is live" from "the probe got no answer." Step 2 had exactly the same hole — and it's the more destructive one.

If the messages GET throws or returns non-ok, we learned nothing about what is persisted. The code treated that as "no reply exists" and regenerated. But handleRetry deletes the trailing assistant by id — and that id is the one the server persisted the reply under (serverAssistantMessageId names the registry row, the UI message via generateId, and the DB row alike).

So a run that finished while backgrounded would have its reply deleted from the DB and regenerated from scratch: content loss, plus a second bill.

RecoveryAttempt now carries dbAnswered alongside probeAnswered, and canConcludeTurnIsLost requires both. The resume loop re-asks until both come back.

2. The "was the user's turn persisted?" guard was broken by pagination

It compared user-message counts (dbUserCount >= localUserCount). But tryRecover's GET is unpaginated, so the route applies its default limit: 50 and returns only the newest 50 rows — while local messages is the initial 50 plus every turn since.

Past one page that guard is permanently false. Step 2 could never recover on a long conversation, and every interrupted turn there fell through to the regenerate in (1) — deleting the very reply it should have refetched.

Now asked by identity: does the DB contain the id of our last local user message? That message is by definition the newest, so it is always inside the returned window. Correct and pagination-proof.

3. tryRecover's writes had no stale-conversation guard

It writes after two awaits into a useChat instance whose id is constant across conversation switches, so a recovery for the conversation the user just left could land in the one they moved to. Every sibling loader guards this (handlePullUpRefresh, loadGlobalMessages) — ironically including the one this PR migrated handleAppResume onto, for exactly this reason. tryRecover now guards too, against the live conversation.

Verified sound this round

  • The tryRecoverForError adapter doesn't disturb useStreamRecovery (it only ever consumed the boolean), and its identity is stable.
  • The normalized step-2 write is the same runtime array the guards were computed from.
  • Hook order, deps, closures all clean.

Tests

The regenerate predicate is now the real canConcludeTurnIsLost, not a mirror of it. Added a case for DB-silence (probe answered, DB didn't → never regenerate). 778 passing.

Validation

typecheck clean · bun run build exits 0 · eslint clean on all touched files.

@2witstudios
2witstudios force-pushed the fix/global-assistant-stream-recovery branch from ac332b3 to 80a8db1 Compare July 14, 2026 21:35
@2witstudios

Copy link
Copy Markdown
Owner Author

Rebased onto master, and the last of the duplication is gone

Rebase (master had landed in these same files)

Master picked up two commits touching GlobalAssistantView.tsx / SidebarChatTab.tsx while this was open — #2072 (deterministic chat rendering + synchronous send with contextRef) and #2071 (per-conversation send serialization). Rebased clean, and I checked the integration rather than trusting the auto-merge:

Duplication (CodeRabbit's thread, and every self-review round before it)

The resume gate and the eviction rule existed in six copies — once per component, then hand-mirrored again in each test file — and the mirrors could not fail. Both are now single pure modules the components and the tests share: canResumeRecovery and evictStalePartial (plus recoveryAttempt's canConcludeTurnIsLost, already shared). The tests assert against the shipped implementation now, and I added the case only the real helper can have: a checkpoint of malformed frames counts as empty, because that is what the bootstrap would seed from — the raw-length version of that gate would have said "safe to evict" and, on a failed SSE join, left an empty screen.

One thing the extraction got wrong on the first pass, caught on review: it swapped the eviction from a setMessages updater to a value read from a render-time ref, so it could compare references and skip a no-op write. Equivalent in GlobalAssistantView (its setters resolve updaters against their own refs anyway) but not in SidebarChatTab, whose setters are the raw useChat ones — passing them an updater resolves against the SDK store's live array, which the transport and useAgentChannelMultiplayer write synchronously without waiting for a render. Reading a ref instead opened a lost-update window. Fixed by splitting the rule rather than weakening it: canEvictStalePartial is exported so callers can ask before writing (an unsafe checkpoint still costs no write), and the eviction goes back through the updater form. evictStalePartial stays self-guarding on the same predicate, and a test pins the two against each other so they cannot drift.

Validation

typecheck clean · bun run build exits 0 · eslint clean · 818 tests green.

@2witstudios

Copy link
Copy Markdown
Owner Author

Final review pass — one real gap closed (7a6c2c8)

A fresh end-to-end read of the final state. The core design verified sound — the eviction has real server data behind it (/active-streams genuinely returns messageId and parts), both stops are the local-only useChat stop rather than the server-aborting wrapper (the one thing that would have silently defeated the whole PR), silence is never treated as an answer, and every post-await write is conversation-guarded. Three findings.

1. onResume and useStreamRecovery could both regenerate the same turn

They watch the same failure from opposite sides — useStreamRecovery fires on status === 'error', onResume on the app-resume event — and share no lock.

If iOS delivers the dead fetch's rejection as an error before the resume listener runs, useStreamRecovery may already have retried. And rawStop() cannot have cancelled it: Chat.stop() returns early unless the status is streaming/submitted, so on an already-errored chat it is a no-op. onResume would then regenerate on top — exactly the double destruction its own comments warn about, since takeOverConversationStreams aborts the run that just started and handleRetry deletes its assistant message on the way in.

The gate now also requires that nothing has restarted the turn, read through a ref because the closure's copy of isStreaming is the value captured before we were frozen.

2. A comment that lied — and contradicted a test in this same PR

It still described neither → falls through to handlePullUpRefresh, which is the DB-refresh fallback deliberately removed two commits earlier, and which the tests explicitly assert never happens ("never a DB refresh"). Fixed in both files.

3. Dead expression

Step 1's guard called decideRecovery with hasLiveStream hardcoded true, so it could only ever answer 'rejoin' — decoration dressed as a decision. The rejoin-first priority is expressed by the order of the two steps; step 2 is where decideRecovery genuinely decides.

Validation

typecheck clean · bun run build exits 0 · eslint clean · 814 tests green.

@2witstudios

Copy link
Copy Markdown
Owner Author

The double-regenerate fix was incomplete — real mutex now (e2268d6)

Verifying the previous commit surfaced that the nothingHasRestarted status check I'd added doesn't actually close the race it was written for.

useStreamRecovery calls clearError() before running its own probes. So for the entire window in which it is deciding whether to regenerate (two fetches, ~1s+), the chat status reads ready and looks idle to us — my status guard passes, and both paths can still reach handleRetry for the same turn. The status check only caught the narrow sub-case where the other path had already reached a running generation and a render had committed.

Closed with an actual lock: one regenerationInFlightRef wraps handleRetry as regenerateTurnOnce, and both callers go through it — useStreamRecovery (via its handleRetry prop) and the resume handler. Held across the whole of handleRetry, whose message DELETEs are a network round-trip.

The status guard stays as the belt to this mutex's braces: the lock releases when handleRetry returns, but the generation it kicked off is still running, and that generation's own submitted status is what keeps the other path out afterwards.

It's a mutex, not a one-shot latch — released in a finally, so a later genuine failure can still recover and a throwing handleRetry can't wedge it shut. Tested with a real concurrent-invocation harness (both paths firing while the first is still awaiting): exactly one regenerate, plus a later-regenerate-allowed case and a throws-still-releases case.

Also fixed a stale comment referencing rawStop() in SidebarChatTab — that file calls the (equally local-only) stop().

Validation

typecheck clean · bun run build exits 0 · eslint clean · 820 tests green (the mutex now has a genuine concurrency test, not just a policy mirror).

Sprite and others added 13 commits July 14, 2026 18:05
Both GlobalAssistantView.tsx and SidebarChatTab.tsx had two defects in their
useAppStateRecovery wiring that orphaned AI streams on mobile:

1. `enabled` was a render-time boolean `!isStreaming` — iOS freezes JS on
   background, so the captured value was `false` (streaming). Recovery was
   gated off in exactly the case it was written for.

2. `onResume` was a dumb DB fetch — no local stop, no server rejoin. Even if
   recovery fired, it would fetch a stale snapshot (reply not persisted yet)
   and clobber the in-progress bubble.

The fix matches AiChatView.tsx's proven pattern exactly:
- `enabled` -> callback form (evaluated at fire time, gates on editing only)
- `onResume` -> resolveResumeAction: on native always returns
  'rejoin-and-refresh', which stops local useChat + rejoins server stream +
  refreshes from DB
Two build-breaking defects from the previous commit, both flagged by Codex:

1. GlobalAssistantView referenced `rejoinAgentStream` in the resume callback,
   but that binding is only initialized further down by useAgentChannelMultiplayer
   — a temporal dead zone that failed the type-check ("Block-scoped variable
   'rejoinAgentStream' used before its declaration") and broke `web#build`.
   Use the existing `rejoinAgentStreamRef`, which exists for exactly this
   hook-ordering problem and is already the in-file idiom in tryRecover.

2. SidebarChatTab called `resolveResumeAction` and `isCapacitorApp` without
   importing either. Added both imports.

Also collapse the agent/global stop branch in GlobalAssistantView to the
already-mode-selected `rawStop`, matching the local-only stop semantics
documented in AiChatView.
The resume wiring had zero coverage on either surface — which is how the
render-time `enabled` boolean survived, and how it regressed a second time.

Follows the file's established pattern (pure mirrors of the hook-heavy
component's logic), but the decision itself is NOT mirrored: planResume calls
the real resolveResumeAction, so these tests pin the wiring against the real
policy rather than a copy of it.

Covers:
- resumeEnabled takes NO streaming argument — the regression is now
  inexpressible in the signature; it gates on user-editing only.
- native mid-stream => stop + rejoin (agent or global) + refresh
- native idle => still stops and rejoins (deterministic, not flag-gated)
- web mid-stream => noop (a live fetch survives a tab switch)
- web idle => refresh only
Found by adversarial self-review, not by a reviewer. The PR fixed only HALF of
its own stated "Defect 2": it added the stop+rejoin, but left the DB refresh as
the same dumb, unguarded fetch — and then wired it to fire on the rejoin path,
where it is at its most dangerous.

The reply is not persisted until the run completes, so a refresh issued while a
stream is live returns a snapshot with NO assistant message. Two consequences:

1. Unguarded write. GlobalAssistantView.handlePullUpRefresh did a raw fetch ->
   setMessages, and SidebarChatTab.handleAppResume bypassed loadGlobalMessages —
   the documented "single writer for the global-mode server->view path" sitting
   directly above it. Neither carried the stale-response check or
   mergeServerAndPending, so the pre-reply snapshot went straight over the live
   assistant bubble. Both now funnel through guarded, reconciled loaders.

2. Racy order. The refresh was fired AFTER the rejoin, so its request could still
   be in flight when the rejoined stream completed — landing last and overwriting
   the just-completed reply. In agent mode nothing heals that (there is no
   completion-triggered refetch), leaving the reply invisible until the user
   reselected the conversation. The order is now stop -> await refresh -> rejoin,
   so no DB request is ever outstanding when the stream completes: the refresh
   catches a run that finished while backgrounded, the rejoin attaches to one
   still live.

Tests updated to pin the ordering (refresh strictly before rejoin), and the
gate's regression test now asserts on arity so an `isStreaming` parameter cannot
be reintroduced — the previous assertion was vacuous.
…ually holds

The previous commit ordered the resume as stop -> await refresh -> rejoin, but in
the sidebar's global path the "await" was a no-op: loadGlobalMessages returned
void and fired its fetch as a floating promise, so handleAppResume resolved
immediately and the rejoin ran with the DB request still outstanding — exactly
the race the reordering was meant to close.

loadGlobalMessages now returns the promise chain, and handleAppResume awaits it.
The other four call sites (load-on-select, refreshSignal, retry) still ignore the
result; they only need to kick a refresh off, not sequence against it.
…tream lookup

Two defects in my own previous commit, both found by self-review.

1. Vacuous stale guard. I copied the "id the load was requested for" ref pattern
   from AiChatView/SidebarChatTab, but that pattern only works there because every
   load path funnels through the single loader that advances the ref — so switching
   conversation advances it and the stale response is dropped. GlobalAssistantView
   does NOT load on select through handlePullUpRefresh (it uses the
   globalInitialMessages / agent-load-signal effects), so nothing else ever moved
   the ref. It always equalled the id its own in-flight fetch was issued for, the
   guard always passed, and a response for the conversation the user just left would
   still be applied to the one they switched to.

   Now guards against the LIVE conversation (currentConversationIdRef, mirrored every
   render), which is the invariant that actually holds on this surface.

2. Unscoped pending-stream lookup. The merge matched on `isOwn && conversationId`
   alone. A conversation id is only unique within its channel, and this surface
   switches between the global channel and per-agent channels, so that could splice a
   different agent's in-flight bubble into this conversation. Now scoped by channel
   via the store's own getOwnStreams(channelId), mirroring AiChatView.
Round-2 self-review found the previous approach was still wrong, and in a way
that made the primary symptom worse.

An OWN stream is deliberately never in the pending-streams store while its POST
body is being consumed (markChannelConsuming -> shouldAttachStream returns false,
so chat:stream_start is skipped). So on the resume path the store is EMPTY for
the stream we are recovering, which means mergeServerAndPending had nothing to
merge and was inert. The "guarded" refresh therefore still wrote a pre-reply DB
snapshot straight over the half-streamed assistant bubble — the partial reply
vanished for a full round-trip until the bootstrap re-seeded it. The very bug the
ordering was supposed to prevent, just moved.

The mistake was reaching for a DB read at all. Both surfaces already have
`tryRecover` — the rejoin-first probe useStreamRecovery uses on a network error —
and a background/foreground cycle IS a network error on iOS, just one we are told
about. It asks /active-streams first (the server's authoritative answer) and only
touches the DB when nothing is live:

  live stream       -> rejoin it, no DB read at all
  already persisted -> refetch the completed reply
  neither           -> fall through to the plain refresh

onResume now stops the local fetch (which also releases the channel's `consuming`
mark, without which the rejoin's bootstrap would skip attaching) and delegates to
tryRecover. No bespoke ordering, no inert merge, and the DB is never read while a
run is still generating.

SidebarChatTab's useAppStateRecovery moves below tryRecover, which it now closes
over. handlePullUpRefresh keeps its stale-guard + reconciliation: it is still
reachable from the pull-up and refreshSignal paths, where a late-landing load can
clobber a conversation the user has since switched away from.

Tests reworked to pin the real invariant — on the rejoin path the DB is NOT read
unless tryRecover comes back empty. Dropped the vacuous arity assertion.
…an unsafe DB refresh

Round-3 self-review found the rejoin was attaching and then rendering NOTHING.

1. Dedup collision (blocker — the PR did not actually work).
   The server mints ONE id and uses it for both the assistant UI message
   (`generateId: () => serverAssistantMessageId`) and the stream registry row, so
   the id useChat holds for the half-streamed bubble IS the live stream's
   messageId. `Chat.stop()` documents that it "keeps the generated tokens", so
   that bubble survives the stop — and the rejoin then re-adds the same stream to
   the pending store. Both surfaces drop a pending stream whose messageId already
   appears in `messages` (dedupRemoteStreams / ChatMessagesArea), so the rejoined
   stream was filtered straight back out: not one token would render, and the user
   would sit in front of a frozen partial until completion.

   tryRecover's rejoin branch now reads the live stream's messageId off
   /active-streams and evicts the matching local message before rejoining. Nothing
   is lost: the bootstrap seeds the stream from the server's registry buffer, which
   holds every part pushed so far — strictly more than the partial we froze with.
   This also fixes the same latent bug on the network-error rejoin path.

2. Unsafe DB fallback (clobber).
   onResume fell through to a blind DB refresh whenever tryRecover returned false —
   which is exactly the two cases where a DB write is UNSAFE:
     - the /active-streams probe FAILED (the first request after a foreground is
       the likeliest to, radio still coming up). A stream may well be live, and the
       DB snapshot cannot contain an unpersisted reply, so the refresh erased the
       in-progress bubble.
     - the DB is BEHIND local state (step 2's dbUserCount >= localUserCount guard
       rejected it) — e.g. a send whose POST never reached the server. The refresh
       erased the user's own prompt.
   The native path now ends at tryRecover, which already refetches when the run
   finished while we were away. Doing nothing in the other cases is correct: local
   state is newer than anything we could fetch, and the reconnect path heals it.

Also corrected two comments that still described the previous (abandoned) ordering.
…t the raw useChat one

Self-review of the previous commit: GlobalAssistantView's eviction called the raw
setAgentMessages/setGlobalLocalMessages, which only touch useChat. The component
has agentSetMessages/globalSetMessages for exactly this — agent mode must also
drop the message from the dashboard store (setConversationMessages), or the store
keeps the stale partial and can serve it back to a co-mounted surface.

Matches how tryRecover's step 2 already writes both.
…inst an empty checkpoint

Round-4 self-review found the previous commit's eviction was being undone, and
could also leave the user with nothing at all.

1. mergeServerAndPending re-inserted the very id the eviction removed.
   It synthesizes an assistant message under the pending stream's messageId when
   that id is absent from the DB snapshot — which is exactly the id both renderers
   dedup on. So after a rejoin, the next refreshSignal (a socket reconnect is
   near-certain right after an iOS foreground) ran handlePullUpRefresh, merged a
   FROZEN snapshot of the stream's parts into `messages` under that id, and the
   live pending stream was deduped away again. The bubble stopped updating until
   completion — the exact symptom the eviction exists to prevent.

   On these surfaces the pending store IS the bubble: an in-flight stream renders
   from `remoteStreams`. Merging a static copy of it into `messages` is not needed
   for visibility and actively kills the live render. The merge is removed — which
   also restores master's behaviour here, since master never had it. (I had added
   it two commits ago while trying to make a mid-stream DB read safe; the read is
   gone now, so the guard it needed is gone too.)

2. Evicting against an empty checkpoint could leave NOTHING on screen.
   /active-streams `parts` is the registry's DEBOUNCED checkpoint (persisted every
   N parts), so it is empty for a stream only a few parts old. If we evicted the
   local partial and the SSE join then failed — the documented multi-instance case,
   where the multicast lives in another process — the bootstrap removes the stream
   and the user is left with nothing, strictly worse than the frozen partial we
   started with. Eviction is now gated on the server actually having parts to
   render in its place; otherwise we keep the partial and still attempt the rejoin.

Tests pin both: eviction only when the checkpoint is non-empty, and never touching
the user's turn.
…ch the seeder's part count

Round-5 self-review. The happy path traced clean end-to-end, but two things remained.

1. The native resume path swallowed the "nothing to recover" case (regression vs master).
   `Chat.stop()` aborts the fetch, so useChat settles at `ready` with NO `error` —
   and useStreamRecovery only fires on `status === 'error'`. So the stop we added
   destroys the very signal that used to drive the fallback, and tryRecover's false
   return was being discarded. A turn whose POST died on the background transition
   (a radio drop right then is common) would find no live stream, no persisted reply
   and no error: the user's prompt sat unanswered forever, with no retry and no
   banner. Master recovered it — the dead fetch raised an error, useStreamRecovery
   probed, came up empty, and regenerated.

   onResume now falls through to handleRetry() when tryRecover comes up empty — the
   same fallback useStreamRecovery applies on the network-error path. Still NOT a DB
   refresh, which remains unsafe here for the reasons documented at the call site.
   Gated on a turn actually having been in flight when we backgrounded, so an
   ordinary resume on an idle conversation can never fire a spurious generation.
   (The frozen render-time streaming flag is a faithful record of that, which is all
   it is used for — deciding whether the TRANSPORT is alive is still
   resolveResumeAction's job.)

2. The eviction gate counted parts differently from the bootstrap that seeds them.
   The gate used the raw `parts.length`, but the bootstrap seeds
   `(stream.parts ?? []).filter(isValidPartFrame)` and it is THAT count which becomes
   skipReplayCount — and a skipReplayCount of 0 is what makes a failed join drop the
   stream. A checkpoint of malformed frames would therefore read as "safe to evict"
   while seeding nothing, and a failed SSE join would leave the user with an empty
   screen: exactly what the gate exists to prevent. Both now use isValidPartFrame.

Also corrected the comment that claimed the reconnect/refreshSignal path heals every
skipped case — it does not heal a run that never started server-side.
Proactive fix, ahead of review. The new handleRetry() fallback was gated on
effectiveIsStreaming/displayIsStreaming, which is NOT "a turn of ours was in
flight for the conversation on screen".

Both flags fold in a stream that is still running against a conversation the user
has since navigated away from — the useChat instance has a stable id, so it keeps
reporting streaming across a conversation switch, for the OLD conversation's
in-flight request. Regenerating on the strength of that would fire a generation
for the turn the user is now LOOKING at rather than the one that was actually
interrupted: a spurious reply, and a spurious charge, on an untouched conversation.

Both files already carry the conversation-scoped flag for exactly this hazard
(isOwnAgentStream/isOwnGlobalStreamForCurrentConversation on the dashboard,
isOwnStreamForCurrentConversation in the sidebar — each latching the conversation
the stream actually started in). The gate now uses it.
…answer

Round-6 self-review found the regenerate fallback could destroy a healthy run.

tryRecover returns false for two OPPOSITE outcomes:
  "the server says nothing is live"   -> the run died; regenerating is the recovery
  "the probe never reached the server" -> we know NOTHING; a run may still be live
and the resume path was treating the second as if it were the first. The first
request after a foreground is the one most likely to fail (cold radio), so on this
path that case is common, not exotic.

It matters because every generation start calls takeOverConversationStreams. A
regenerate issued while the run is in fact still live does NOT race it — it ABORTS
it. So the fallback would have: killed a healthy, possibly nearly-finished
generation; re-run any write tools it had already executed (a turn that created a
page creates it twice — the side effects are not rolled back); billed the discarded
tokens; and stranded its partial in the DB, since onFinish persists the aborted
response AFTER handleRetry's delete of the trailing assistant.

tryRecover now records reachability separately from its verdict (only a 2xx means
the server actually told us what is live). onResume re-probes twice, backing off,
before concluding — the radio returns well inside that — and regenerates ONLY on an
answered probe.

On persistent silence it does nothing, which is safe: the stop released this
channel's `consuming` mark, so a live run is picked up by the socket-reconnect
bootstrap once the network returns, and a dead one leaves the user their prompt and
the retry action on it.

Tests: planResume now models ownTurnInFlight and probeAnswered as inputs distinct
from the broad streaming flag, so neither the conversation-scoped gate nor the
probe-reachability gate can silently regress. Both were previously unfalsifiable.
… ref

Round-7 self-review found the probe-reachability signal was leaky in three ways,
all landing on the same failure — regenerating over a healthy run, which
takeOverConversationStreams turns into an abort of it.

1. `probeAnsweredRef.current = res.ok` was set BEFORE the body was parsed. res.json()
   can still throw on a body that dies mid-read — which is exactly the
   cold-radio-after-foreground case this path exists for — and the catch swallowed it
   while we went on believing we had an answer. Now set only after a body we actually
   parsed.

2. One shared mutable slot, two callers. tryRecover is called by BOTH this surface's
   resume handler and useStreamRecovery's network-error retry, and they can be in
   flight together — so one caller's probe could answer the other caller's question.

3. The reset sat after tryRecover's early returns, so a bail-out (`!channelId`, e.g.
   `user` transiently null while auth rehydrates on foreground — the token very likely
   expired while backgrounded) left the ref holding a PREVIOUS call's `true`, and the
   resume path would regenerate on the strength of a probe it never made.

All three are gone: tryRecover now RETURNS `{ recovered, probeAnswered }`
(RecoveryAttempt), so each caller reads its own probe and there is no shared state to
race or go stale. useStreamRecovery takes a thin adapter reading only `.recovered` —
the reachability half of the answer is for the resume path, which is the one that
would otherwise regenerate over a live run.

Also: step 2 read defensively (`Array.isArray(data) ? data : data.messages`) but wrote
`data.messages` raw, which would blank the surface outright if the route ever answered
with a bare array. It now writes the same normalized array the guards were computed
from.

Tests: planResume takes a SEQUENCE of attempts, so the re-probe loop is modelled — a
probe that only answers on a later attempt still regenerates (a cold radio must not
strand the turn), and one that never answers never does.
Round-8 self-review. Three defects, two of which combine into the worst outcome
this PR could produce: DELETING a completed reply and paying to generate it again.

1. Silence from the DB check was read as "nothing is persisted".
   Round 7 taught the probe to distinguish "the server says nothing is live" from
   "the probe got no answer". Step 2 had exactly the same hole and it is the more
   destructive one: if the messages GET throws or returns non-ok, we learned nothing
   about what is persisted — but the code treated that as "no reply exists" and
   regenerated. handleRetry DELETEs the trailing assistant BY ID, and that id is the
   one the server persisted the reply under (serverAssistantMessageId names the
   registry row, the UI message and the DB row alike). So a run that FINISHED while
   backgrounded would have its reply deleted from the DB and regenerated from
   scratch: content loss plus a second bill.

   RecoveryAttempt now carries dbAnswered alongside probeAnswered, and
   canConcludeTurnIsLost requires BOTH. The resume loop re-asks until both come back.

2. The "was the user's turn persisted?" guard was broken by pagination.
   It compared user-message COUNTS (dbUserCount >= localUserCount). tryRecover's GET
   is unpaginated, so the route applies its default limit of 50 and returns only the
   newest 50 rows — while local `messages` is the initial 50 PLUS every turn since.
   Past one page the guard is PERMANENTLY false, so step 2 could never recover on a
   long conversation, and every interrupted turn there fell through to the regenerate
   in (1) — deleting the very reply it should have refetched.

   Now asked by identity: does the DB contain the id of our last local user message.
   The last user message is by definition the newest, so it is always inside the
   returned window. Correct and pagination-proof.

3. tryRecover's writes had no stale-conversation guard.
   It writes after two awaits into a useChat instance whose id is constant across
   conversation switches, so a recovery for the conversation the user just left could
   land in the one they moved to. Every sibling loader guards this
   (handlePullUpRefresh, loadGlobalMessages); tryRecover now does too, against the
   live conversation.
…d-recovery

Proactive, ahead of review. tryRecover now refuses to WRITE into a conversation the
user has navigated away from — but the regenerate that follows it had no such guard,
and handleRetry always acts on the LIVE conversation.

The recovery spans up to a few seconds of network (three bounded probes), and the
user can switch conversation inside that window. The turn we set out to recover
belongs to the conversation we started from; regenerating after a switch would fire
a generation for the turn they moved TO instead — a spurious reply, and a spurious
charge, on an untouched conversation, and it would delete that conversation's
trailing assistant message on the way in.

onResume now latches the conversation the interrupted turn belongs to and re-checks
it against the live one before retrying.
… the real ones

CodeRabbit, and every self-review round before it, kept landing on the same thing:
the resume gate and the eviction rule existed in six copies — once per component,
then hand-mirrored again in each test file — and the mirrors could not fail. A
component regression would have left the suite green.

Both are now single pure modules that the components and the tests share:

- canResumeRecovery(conversationId, isAnyEditing) — the useAppStateRecovery gate. It
  takes no streaming argument BY CONSTRUCTION, which is the whole point: the original
  bug was a render-time boolean folding in `!isStreaming`, captured before iOS froze
  JS and therefore always reporting the streaming state we went away in.

- evictStalePartial(messages, liveMessageId, serverParts) — owns both halves of the
  rule: drop the frozen bubble whose id collides with the rejoined stream (or the
  dedup filters that stream straight back out and it renders nothing), but ONLY when
  the server's debounced checkpoint has frames the bootstrap can actually seed from,
  counted with the same isValidPartFrame predicate the bootstrap uses. It returns the
  same reference when it declines, so callers skip the write entirely.

The tests now import both, so they assert against the shipped implementation rather
than a copy of it. Added direct coverage for the case only the real helper can have:
a checkpoint of malformed frames counts as empty, because that is what the bootstrap
would seed from.
The extraction in the previous commit swapped the eviction from a setMessages
UPDATER to a value computed from currentMessagesRef, so it could compare references
and skip a no-op write. That was the wrong trade in the sidebar.

In GlobalAssistantView the two are equivalent — agentSetMessages/globalSetMessages
resolve an updater against their OWN refs and always call the setters with a value,
so nothing was reading React's `prev` there anyway. But SidebarChatTab's setters are
the raw useChat ones, and passing them an updater resolves it against the AI SDK
Chat store's LIVE messages array, which the transport and useAgentChannelMultiplayer
write synchronously without waiting for a render. Reading a render-time ref instead
opened a lost-update window: a store write landing after the last render and before
the fetch continuation would be clobbered by our value write.

Split the rule instead of weakening it. `canEvictStalePartial(serverParts)` is now
exported, so the call sites can ask BEFORE writing — an unsafe checkpoint still costs
no state write — and the eviction itself goes back through the updater form, so the
filter runs against the freshest list. evictStalePartial stays self-guarding on the
same predicate, so the rule holds even for a caller that forgets to ask, and a test
pins the two against each other so they cannot drift.
Final review pass. Three findings, one of them a real gap.

1. onResume and useStreamRecovery could BOTH regenerate the same turn.
   They watch the same failure from opposite sides — useStreamRecovery fires on
   `status === 'error'`, onResume on the app-resume event — and share no lock. If
   iOS delivers the dead fetch's rejection as an error before the resume listener
   runs, useStreamRecovery may already have retried. rawStop() cannot have cancelled
   it: Chat.stop() returns early unless the status is streaming/submitted, so on an
   errored chat it is a no-op. onResume would then regenerate on top — exactly the
   double destruction its own comments warn about, since takeOverConversationStreams
   aborts the run that just started and handleRetry deletes its assistant message on
   the way in.

   The gate now also requires that nothing has restarted the turn, read through a ref
   because the closure's copy of `isStreaming` is the value captured before we were
   frozen.

2. A comment that lied, and contradicted a test in the same PR. It still described
   "neither → falls through to handlePullUpRefresh", which is the DB-refresh fallback
   deliberately removed two commits earlier — and which the tests explicitly assert
   never happens ("never a DB refresh").

3. Dead expression. Step 1's guard called decideRecovery with hasLiveStream hardcoded
   true, so it could only ever answer 'rejoin' — decoration dressed as a decision.
   The rejoin-first priority is expressed by the ORDER of the two steps; step 2 is
   where decideRecovery genuinely decides.
Verification of the previous commit showed its `nothingHasRestarted` guard was an
incomplete fix. useStreamRecovery calls clearError() BEFORE running its own probes,
so for the entire window in which it is deciding whether to regenerate, the chat
status reads `ready` and looks idle to us — the status check passes and both paths
can still call handleRetry for the same turn. The status guard only catches the
sub-case where the other path has already reached a running generation.

Close it with an actual lock: a single regenerationInFlightRef wraps handleRetry as
regenerateTurnOnce, and BOTH callers go through it — useStreamRecovery (via its
handleRetry prop) and the resume handler. Held across the whole of handleRetry,
whose message DELETEs are a network round-trip. The status guard stays as the
belt to this mutex's braces: the lock releases when handleRetry returns, but the
generation it started is still running, and that generation's own `submitted` status
is what keeps the other path out afterwards.

The mutex is a mutex, not a one-shot latch — it releases in a finally, so a later
genuine failure can still recover, and a throwing handleRetry does not wedge it shut.
Tested with a real concurrent-invocation harness (both paths firing while the first
is still awaiting): exactly one regenerate.

Also fixed a stale comment referencing `rawStop()` in SidebarChatTab, which has no
such binding — it calls the (equally local-only) `stop()`.
Rebase onto master picked up #2073, which moves the stream registry checkpoint from
a part-count cadence to a time-based one. The eviction logic is unaffected — it gates
on whether the checkpoint has isValidPartFrame frames, whatever the cadence — but the
comments describing it as "persisted every N parts" / "debounced" are now stale.
Reworded to the durable truth: the checkpoint lags the live stream, so it is empty in
a stream's first moments.
@2witstudios
2witstudios force-pushed the fix/global-assistant-stream-recovery branch from e2268d6 to a268a6a Compare July 14, 2026 23:10
@2witstudios
2witstudios merged commit dc85191 into master Jul 14, 2026
3 checks passed
2witstudios added a commit that referenced this pull request Jul 15, 2026
…uck-streaming gaps

Addresses two CodeRabbit findings:

1. Global route had no execute-end durable-persist block at all (only chat/route.ts
   did). If the client disconnected after generation (mobile backgrounding — the exact
   scenario #2065 just fixed client-side for this same route), onFinish might never
   fire and the placeholder stayed 'streaming' forever. Added an execute-end block to
   the global route mirroring chat/route.ts's: unconditional, using whatever
   lifecycle.getBufferedParts() has, status aborted ? 'interrupted' : 'complete', plus
   the same conversations.lastMessageAt bump. onFinish's own no-responseMessage branch
   is now a no-op (execute-end has already terminalized the row by the time it would
   run) — replaced the duplicate persist logic there with a debug log.

2. Both routes' execute-end persist was still gated on `bufferedParts.length > 0 ||
   aborted`. A run that exhausted its retries without ever aborting or producing a
   responseMessage (sustained provider outage) fell through that guard AND onFinish's
   `if (responseMessage)` guard, leaving the placeholder stuck at 'streaming' forever —
   same failure class as the abort gap already fixed, just for a third path (clean exit,
   no content) neither prior fix covered. Both routes' execute-end write is now
   unconditional.

Updated/added tests in both routes' stream-socket-events.test.ts for: non-aborted
empty-buffer persists as 'complete' (chat), aborted-with-content persists via
execute-end (global, migrated off the now-inert onFinish path), lastMessageAt bump via
execute-end (global), and non-aborted-with-content persists as 'complete' via
execute-end (global). Fixed global credit-gate.test.ts's lifecycle mock (missing
getBufferedParts, now called unconditionally). Verified each new/changed assertion
against the pre-fix code before committing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB
2witstudios added a commit that referenced this pull request Jul 15, 2026
…aming|complete|interrupted) (#2076)

* feat(ai): assistant message row at stream start + status column (streaming|complete|interrupted)

Adds status text NOT NULL DEFAULT 'complete' (streaming|complete|interrupted) to
chat_messages and messages, and inserts an empty assistant placeholder row the
moment generation starts (inside startGenerationExclusive's run closure, same
critical section as takeover+lifecycle-create per the PR 4 seam note).

Reader inventory (~30 sites) updated to exclude 'streaming' placeholders:
- Model-context/compaction loads: page + global history loads, chatMessageRepository
  (v1/completions, page-payload-service), ask_agent, consult route (+ fixed a
  pre-existing missing isActive filter on its fallback branch), page-read-tools,
  ask-user-resume's fetchById/fetchLastAssistant (both page + global adapters).
- Previews: conversation-repository's raw-SQL CTEs (last-message preview + count).
- Mutations: edit/delete of a streaming row now 409s (chat + global [messageId]
  routes). Undo: the undo-vs-in-flight-stream UX call (409 whole undo / allow+hidden
  write / allow+abort) is a product decision, left open — ai-undo-service now excludes
  streaming rows from its sweep entirely (preview count + both soft-delete updates) as
  the safe interim default, so undo can never soft-delete a live stream's row.
- Converters: convertDbMessageToUIMessage / convertGlobalAssistantMessageToUIMessage
  (all branches, including a reconstruction-success branch that previously dropped
  extra fields entirely) now propagate status.
- Read APIs: chat/messages, global/[id]/messages, page-agents conversation messages,
  and v1/conversations/[id] GET all gate streaming rows behind includeStreaming=1.
  Client dedup: mergeServerAndPending (shared by AiChatView and SidebarChatTab) now
  replaces a server-loaded streaming placeholder with the richer live stream version
  instead of losing to the empty DB row.
- Exports: tenant-export's column lists and gdpr-export's collectUserMessages both
  carry status now.

Terminal writes: saveMessageToDatabase / saveGlobalAssistantMessageToDatabase's
onConflictDoUpdate sets status: 'complete' (execute-end + onFinish, matching the
existing durable-persist / best-effort-refine split). 'interrupted' via abort/
materialization is PR 3's wiring, per the epic's own "contract fixed here" framing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB

* fix(ai): terminalize aborted/stopped streams as interrupted, not complete

Addresses two Codex review findings on #2076:

1. saveMessageToDatabase / saveGlobalAssistantMessageToDatabase's onConflictDoUpdate
   always set status:'complete', even for a run the user (or credit gate) stopped —
   contradicting the status contract ('interrupted' = terminal, real partial output).
   Both now accept an explicit status param; execute-end and onFinish in both chat
   routes pass 'interrupted' whenever agentRun.terminalReason === 'aborted' or the
   route's abortSignal is aborted.

2. If a stream was stopped before any buffered part or responseMessage existed, no
   saveMessageToDatabase call ever touched the placeholder row — since streaming rows
   are hidden from every reader by default and edit/delete now 409 on them, that left
   an invisible, permanently-locked ghost row. Fixed by writing a (possibly
   empty-content) 'interrupted' row whenever the run was aborted, regardless of
   buffered content:
   - chat/route.ts: execute-end's guard widened from `bufferedParts.length > 0` to
     `bufferedParts.length > 0 || aborted`.
   - global/[id]/messages/route.ts: onFinish gained a fallback branch for
     `!responseMessage && aborted`, using lifecycle.getBufferedParts() the same way
     chat/route.ts's execute-end block does (this route had no execute-end persist
     block at all — a pre-existing architectural difference from chat/route.ts, not
     something this fix introduces).

Also closes the same failure class for the rarer case where createUIMessageStream
itself throws before execute()/onFinish ever run: both routes' outer catch now
does a best-effort terminal write (status:'interrupted'), guarded by a new
assistantMessagePersisted flag so it can never downgrade a row a successful
execute-end/onFinish write already settled.

New tests in both routes' stream-socket-events.test.ts covering: aborted-with-content
→ interrupted, aborted-with-zero-content → still persisted as interrupted,
non-aborted → complete (regression guard), and outer-catch cleanup. Verified each
new assertion fails against the pre-fix code (temporarily reverted, confirmed RED,
restored) before committing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB

* fix(ai): preserve buffered content in abort cleanup; bump lastMessageAt; dedup aborted check

Self-review (8-angle) on the previous fix commit surfaced two real bugs and one
worthwhile simplification, independently confirmed by 3 separate finder angles:

1. CRITICAL: the outer-catch cleanup wrote in the wrong order — it called
   lifecycle.finish(true) BEFORE reading lifecycle.getBufferedParts(). finish()
   deletes the multicast registry entry getBufferedParts() reads from, so the
   cleanup write always persisted EMPTY content, silently discarding any real
   partial content the fix's own comment claimed to preserve. Fixed in both
   routes by capturing getBufferedParts() before finish(). chat/route.ts has a
   SEPARATE inner catch (createUIMessageStream construction failure) that also
   calls finish() before rethrowing to the outer catch — that inner catch now
   captures the buffer too (bufferedPartsAtStreamError), and the outer catch
   prefers that capture when set, since a fresh call by the time it runs would
   already see the inner catch's cleared buffer.

2. The global route's new onFinish fallback branch (no responseMessage, but
   aborted) didn't bump conversations.lastMessageAt the way the sibling
   responseMessage branch does — a conversation whose only new activity was an
   interrupted, no-responseMessage assistant row wouldn't surface in a
   lastMessageAt-sorted conversation list. Fixed to match the sibling branch.

3. `agentRun?.terminalReason === 'aborted' || abortSignal.aborted` was
   duplicated 3x across the two routes (with a comment on one copy claiming it
   was "computed once and reused" — true within its own closure, false across
   the file). Extracted to `isRunAborted()` in run-agent-with-retry.ts, which
   already owns the terminalReason vocabulary.

New tests: both routes now have a regression test that reproduces the real
finish()-then-getBufferedParts() ordering (mocked to return [] only after
finish fires) and asserts the persisted uiMessage.parts still contain the
pre-crash content; a lastMessageAt bump assertion for the fallback branch
(call-count delta across the onFinish boundary, to avoid a false-pass from
other db.update calls elsewhere in the same request); and a unit test suite
for isRunAborted. All three new/changed assertions verified RED against the
prior code before committing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB

* fix(db): regenerate status-column migration against latest master

master gained two new migrations (0208_cloudy_ozymandias, 0209_sprite_reclaim_triggers)
since this branch was last rebased, colliding with this PR's own 0208 migration number.
Rebased onto origin/master, resolved the migration collision by keeping master's
0208/0209 journal entries and dropping this PR's stale 0208 migration + snapshot, then
regenerated via `bun run db:generate` — the status column migration is now
0210_next_mariko_yashida.sql, journal-clean on top of master's latest.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB

* fix(ai): unconditional execute-end terminal write closes remaining stuck-streaming gaps

Addresses two CodeRabbit findings:

1. Global route had no execute-end durable-persist block at all (only chat/route.ts
   did). If the client disconnected after generation (mobile backgrounding — the exact
   scenario #2065 just fixed client-side for this same route), onFinish might never
   fire and the placeholder stayed 'streaming' forever. Added an execute-end block to
   the global route mirroring chat/route.ts's: unconditional, using whatever
   lifecycle.getBufferedParts() has, status aborted ? 'interrupted' : 'complete', plus
   the same conversations.lastMessageAt bump. onFinish's own no-responseMessage branch
   is now a no-op (execute-end has already terminalized the row by the time it would
   run) — replaced the duplicate persist logic there with a debug log.

2. Both routes' execute-end persist was still gated on `bufferedParts.length > 0 ||
   aborted`. A run that exhausted its retries without ever aborting or producing a
   responseMessage (sustained provider outage) fell through that guard AND onFinish's
   `if (responseMessage)` guard, leaving the placeholder stuck at 'streaming' forever —
   same failure class as the abort gap already fixed, just for a third path (clean exit,
   no content) neither prior fix covered. Both routes' execute-end write is now
   unconditional.

Updated/added tests in both routes' stream-socket-events.test.ts for: non-aborted
empty-buffer persists as 'complete' (chat), aborted-with-content persists via
execute-end (global, migrated off the now-inert onFinish path), lastMessageAt bump via
execute-end (global), and non-aborted-with-content persists as 'complete' via
execute-end (global). Fixed global credit-gate.test.ts's lifecycle mock (missing
getBufferedParts, now called unconditionally). Verified each new/changed assertion
against the pre-fix code before committing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB

* fix(ai): 4 self-review findings from 8-angle diff review

- ask-user-resume.ts's persist closures never passed status, silently
  resetting an 'interrupted' row back to 'complete' on every ask_user
  merge/dismiss write. Both page and global adapters now preserve the
  fetched row's status.
- onFinish in both routes wrote a phantom empty 'interrupted' row for
  pre-aborted streams: the AI SDK always calls onFinish with a non-null
  responseMessage (an empty shell) even when execute() wrote nothing,
  but the placeholder INSERT is deliberately skipped when preAborted.
  Guarded on lifecycle.preAborted.
- Outer-catch cleanup in both routes could fabricate a phantom row for
  an exception thrown before `lifecycle` was ever assigned (inside
  startGenerationExclusive's callback, before the placeholder INSERT
  ran) — now requires `lifecycle` itself, not just !preAborted.
- Chat route's credit-gate abort was invisible to runAgentWithRetry's
  classification: the combined AbortSignal.any([...]) was only passed
  to the nested streamText call, not to runAgentWithRetry's own
  abortSignal param, so a mid-stream credit exhaustion was misclassified
  and persisted as 'complete' instead of 'interrupted'.
- search-tools.ts and discovery-service.ts read chat_messages/messages
  without excluding status='streaming', a gap in the PR's own reader
  inventory.

All fixes verified RED->GREEN. Full typecheck+build clean.

* test(ai): strengthen lastMessageAt assertion to check target table, not just call count

CodeRabbit round-4 review nitpick: the assertion only checked that db.update's
call count increased, not that any of the new calls actually targeted
conversations. Any unrelated update in the same code path would have made the
test pass without exercising the lastMessageAt bump. Now asserts one of the
new update calls targets the conversations table specifically.

RED-verified by temporarily removing the lastMessageAt bump from
apps/web/src/app/api/ai/global/[id]/messages/route.ts and confirming the
assertion fails; restored and confirmed GREEN.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013QQTjjDoYto6GHcBBfWEnB

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant