feat(gateway): serve the non-chat families through the pipeline - #475
Draft
Menci wants to merge 71 commits into
Draft
feat(gateway): serve the non-chat families through the pipeline#475Menci wants to merge 71 commits into
Menci wants to merge 71 commits into
Conversation
The first half of migrating the non-chat families: the space every family's pipeline extends, the services a run is given, and the two stages every family's serve pipeline needs. Neither shared stage names a family's own key. Branching is a capability of the framework and what something branches on is a concept in the domain, so a family hands in the two domain-shaped things — how to narrow a candidate, and how to read an attempt's outcome — and the stages stay written against the shared space. That is what lets them compose into a pipeline over any family's larger space with no variance question to lose, since assembly reasons over declarations and declarations are strings. `resolveCandidates` carries both traits, and it is what found the modelling error corrected in the commit below it: refusing a request no upstream can serve means answering with a key the descend path never carries. The families themselves follow. Nothing is routed through this yet, so the gateway's behaviour is unchanged.
The first family, and the one worth doing first: rerank already has a canonical contract — parse, render, serialize, and a usage reading — so it proves the shared stages on real business without needing a protocol written first. Four stages, and the shape every family will repeat. `emitRerank` is the edge: it renders the canonical answer back into whichever of the four protocols the client spoke, which is why the source protocol is an ingress fact and not a request one — it has to survive the switch to whatever the upstream turned out to speak. `callRerankUpstream` is the ending: it dials, parses the body, and provides both the answer and what the call is billable for. Two behaviours change, both because the architecture has no passthrough. The same-protocol path used to forward the upstream's bytes unread; it is now parsed and re-serialized like the translating path, so one code path serves both and the dump shows a parsed answer either way. And an upstream that refused is a value rather than a forwarded Response, so an earlier stage can fail over it. The client's headers reach the upstream from the record rather than from a live request object, which is what lets the dump show what was there to be filtered. The test that matters most is the one that records a hole. `callRerankUpstream` reads `ingress.http.headers`, and the derived entry contract does not mention it: a stage whose only trait is `return` declares no request side, by ruling, so assembly cannot see what an ending stage reads — and every family's ending stage reads something. The consequence is concrete: a caller omitting that key gets a runtime failure at the deepest stage instead of an assembly error, which is the thing the entry contract exists to prevent. The type layer still catches it at the definition site, so it is a gap and not a break. Nothing is routed here yet; the route table still runs the passthrough serve.
The simplest family, and the one that shows the shape with nothing else in the way: one protocol, no translation and no stream, so the array is exactly the four stages every family has. The narrowing is a constant rather than a function of the request, because an embeddings request carries nothing a candidate could be incompatible with. `packages/protocols/src/embeddings` was five lines of index signature — a passthrough shape, not a contract — so the real one is written here: the request's four input arms, `encoding_format`, `dimensions` and `user`, and the response's embeddings, model and usage, each cited against the OpenAI specification. `encoding_format` is why the canonical response holds numbers rather than whatever the upstream wrote. Both official OpenAI SDKs send `encoding_format: base64` when their caller did not choose one, so base64 is the common case on the wire; an upstream that ignores the field and answers with float arrays hands such a client a body it decodes as base64 and turns into noise. Parsing whatever arrived and writing what the client asked for is what closes that, which is why the encoding is an ingress fact and stays put. A base64 embedding is the vector's float32 elements packed little-endian, and reading it into numbers is exact: every float32 is a float64, so a vector survives any number of trips. Two behaviours change. A field outside the protocol is now named in a 400 rather than forwarded, because the request schema is `additionalProperties: false` and both gateways compared against — LiteLLM and copilot-api — draw the same line; this is the one decision worth a second look, since vLLM and Jina document supersets of the endpoint. And an upstream that reports no usage now bills an entity with no quantities, which is how "called and reported nothing" is said, rather than no row at all. The test records what assembly cannot see, and this family shows it twice over: a return-only stage declares no request side, so neither `ingress.http.headers` nor `request.embeddings.canonical` reaches the derived entry contract — and unlike rerank, whose edge needed the payload for rendering, nothing here puts the payload in the contract by accident. The route still runs the passthrough serve. The old handler parses and serializes through the new contract so the two agree until it is deleted.
Five families at once, each with a real protocol contract where it had none.
`completions`, `embeddings`, `images` and `audio` ran on `passthrough-serve.ts`,
which forwards a body it has not parsed — and the architecture has no such
concept, so three of them needed a contract written before they could have a
pipeline at all. `packages/protocols/src/{audio,images,completions,embeddings}/`
carries those now, cited to the OpenAI specification.
`alpha-search` is shaped differently and is not forced into the others' shape: it
has no upstream model and no candidate list, executing locally through the
configured search provider or relaying a selected provider's response. That is an
ending, not a missing family — the same distinction ruling 5 settled for HTTP,
where a provider that does not ultimately speak plain HTTP supplies its own.
Nothing is routed here yet: the route table still mounts the passthrough handlers,
so the gateway's behaviour is unchanged. The wiring, and deleting the passthrough
helpers, is one commit once every family is sound.
Two defects the families found in what they were built on are recorded in the
Pull Request and fixed in the commits that follow. Both were found by building on
the core rather than by reading it, which is what this layer is for.
`failover` declared that it consumes and provides `response.http.body` for every family. Three of the six never produce one — rerank, embeddings and images read their answer to the end, so there is nothing still open by the time the fork sees it — and the runner checks `provides` at handover. Those pipelines composed cleanly and would have thrown on the first real request, at the deepest stage, naming a key their author had never written down. It cannot be a fixed key in either direction: claiming one a family never produces throws, and staying silent about one it does produce and hands up throws the other way. Which keys carry a resource is a statement only the family can make, so it makes it — alongside the failure predicate it already hands in, for the same reason. Branching is the framework's; what something branches on, and what it owns, are the domain's. Found by the completions family, which verified it against a throwaway pipeline rather than reasoning about it. The three affected families' own tests asserted `entryNeeds` and nothing else, so none of them caught it — the same shape of untested claim the core's review found earlier, appearing again one layer up. The test added here asserts the declaration for both kinds of family rather than the absence of the bug, because the absence of the bug is what the other tests were already asserting when it was present.
Settlement is a stage now, and it sits above `failover` so a run bills once however many candidates it tried. Repetition passes through the stage that observes usage, not through this one. It is unconditional. A run that measured rather than generated still writes, and its row simply names no billed entity — emptiness is observed rather than declared, which is why the word "unknown" appears nowhere: the situations are concrete and the list is open. The write is scheduled rather than awaited, because a transient repository failure must not turn an already-flowing upstream response into a 502. `BillableEntity` gained the pricing inputs a rate can need beyond the quantities. Absent is a real reading and not a missing one — most families price on the quantities alone — and making it required broke five families that correctly have none. It also caught a regression I had introduced: the rerank migration dropped the `inputTokens` pricing fact that `settleRerank` passed, so a rerank rate depending on input size would have priced against nothing. Restored, and the reading now travels with the entity it prices rather than being recomputed at the write.
A `ModelCandidate` is two things wearing one type: the selection — which upstream, which model row, which flags — which is data, and the handles — the provider instance, the fetcher, the models cache — which are live. The six families put the whole thing in the record, so `move()` deep-froze all three handles and the SWR cache refresh the provider does on its own schedule broke. The architecture ruled this out before any code existed, and I did not apply it: "Where something is chosen per attempt, the resolver is the service and the selector is a fact. A per-upstream transport is not a fact and is not pinned at the prologue either; what is injected is the thing that resolves one, and what travels is the identifier it resolves from." So `route.candidate` becomes `route.attempt`, carrying the upstream id, the model id and a snapshot of the flags — snapshotted rather than referenced, because the record must show what was true when the attempt was made rather than what the row says now. `resolveAttempt` is a service and the ending stages ask it for the thing that dials. It also fixes the same rule failing the other way: a candidate in the record is walked by the dump encoder, so a run's dump was serializing the provider instance. Nothing caught it because every family test builds a candidate literal with no live handles, and no route mounts a pipeline, so the suite never runs one against a real provider. The test added here asserts the property in both directions — the selector reaches nothing live, and the candidate would have frozen it.
`writeSettlement` was written and composed into nothing, so no migrated family recorded a usage row or a performance sample. Every serve pipeline now carries it, above the fork, so a run bills once however many candidates it tried. Adding it proved its own absence: the completions tests immediately failed with "Repo not initialized", because until now nothing in six families ever reached a write. That error was the fix working. The entry contracts gained `ingress.http.headers`, which five of the six omitted. `compose` cannot derive it — a stage whose only trait is `return` declares no request side, and every family's ending is one — so the hand-written type is the only thing covering for that, and it was covering wrong. One test here is worth reading twice. I first asserted that a run reaching no upstream still records a performance sample; it does not, because `recordPerformance` returns early without an attempt's telemetry and there is no attempt. That matches the replaced surface exactly, so the expectation was invented and the stage was right. The test now says what happens and why.
The two streaming families marked their body with `Object.assign(body, {
[Symbol.asyncDispose]: … })`, which was how a resource was claimed before
ownership stopped being a detection. The runner reads `isOwned` now, so a body
marked that way was invisible: `failover` declared it consumes one, `drain()`
existed, and neither could see anything. A losing attempt's connection stayed
open and the winner's was never drained.
`own(body, release)` makes the claim, and the fact's type says `Owned` rather
than `AsyncDisposable` — the language puts `Symbol.asyncDispose` on every async
generator and on no `ReadableStream`, so a structural type admits an iterator
that is not a resource and rejects the body that is.
The test asserts the property rather than the absence of the bug: a losing 429's
body and a winning stream, the answer handed back before anything is drained, and
the winner drained when the caller says so. Verified by mutation — dropping the
brand from `own()` fails exactly that test and nothing else.
Both families rendered an answer and never said what status it was, so an upstream 429, a resolver's 404 and a 400 all reached the client as a 200 carrying an error envelope. That is not a difference the no-passthrough ruling asks for: a client is not owed the upstream's exact bytes, but it is owed the truth about what happened, and the replaced surface forwarded the status. The edge provides `response.http.status` on both arms — the upstream's own when it refused, the gateway's own when the resolver refused before dialling, and 200 for an answer. The three tests drive the pipeline rather than reading its declarations, which is what the two families were missing: their whole suites asserted `entryNeeds` and nothing else, so a family that could not express a status at all passed. Verified by mutation — pinning the failure arm back to 200 fails both refusal tests.
… unnamed embeddings model The pipeline's shared resolver phrased its own refusal, which changed what a client is told: five families spell the endpoint they could not serve, while rerank spells out which part of the request no candidate could satisfy. The sentence moves to the family as `Narrowing.unsupported`, taking the reasons `reject` gave so a family can use them or ignore them. Copilot's /embeddings answers without the top-level `model` the schema marks required, so parsing refused a body the replaced surface served. The parser now takes the model the request named and completes the record with it.
…'s edge The pipeline dropped the upstream's response headers, so vendor traces, quota state and retry-after stopped reaching clients on the routes it took over. The attempt hands them up unfiltered and the edge decides what a client may see, which keeps both in a dump. A refusal that never dialled has none to carry, so the shared resolver answers with an empty list and no family repeats it. serveThrough now reads the status and headers off the exit facts. A family's render owns the bytes and their media type and nothing else, which is also why content type stays blocked from forwarding. The assembly test builds every family, including the four whose pipeline is built from the request and would otherwise never be composed by a test.
A stage logger wrote to the dump record and to whatever sink the services carried, and the prologue carried none — so on every run without retention configured, a warning or an error reached nothing at all. A failed usage write was the case that showed it. The sink writes warnings and errors to the console the rest of the gateway uses. Debug and info describe one request's progress and would bury the request log at a line per stage, so the threshold is fixed here; a dump still records every level when one is open.
…ot faults Nothing in the domain throws. A refused connection, a timeout or a reset is an outcome the fork has to see so it can try the next candidate, so each ending catches what the platform raised and hands it up as its family's failure. A dial that reached no upstream bills nothing and carries no headers; only the performance row records that the attempt happened. Attribution moves ahead of the dial. It was written from the call's result, so an attempt that never returned left the previous candidate's context in place and misattributed the row. A 2xx body a JSON protocol cannot read is likewise a failure value rather than an unhandled parse error: the gateway that cannot read an answer has not served the request, and the upstream still counts as called having reported nothing. The body reader images already had is now shared, since three families need it.
Both handlers still ran the passthrough scaffold: they read the body, hand-rolled the multipart and JSON edit shapes into a provider request, picked a candidate by endpoint capability and relayed whatever the upstream sent. The images protocol already owns both readings and the family already has a pipeline, so what is left here is a prologue and an epilogue — parse, hand over, write the answer. Reading the body moves to the contract, which reports a malformed request by throwing; the handler turns that into the same 400 envelope the scaffold wrote, with the same sentences. The two endpoints differ only in which parser reads the bytes, so the rest is one shared function.
…r reads them Workspace packages sort after the relative imports, and the two handlers wired to their pipelines had them first. Nothing else changes.
…gs independent
An upstream that refused in its own words is handed on in them. The envelope was
written once for OpenAI-shaped families and applied everywhere, which turned a
rerank client's `{message}` into `{error:{message}}` — a shape its SDK does not
read. What distinguishes the two cases is whether an upstream answered at all,
not which shape it answered in, so that is what the renderer now tests.
Rerank reads usage before results. They are independent readings of one body and
one failing is no reason to discard the other, so an answer this gateway cannot
model still bills for what the upstream metered.
A same-protocol answer is rendered back out unchanged, so results it carries
that the canonical form does not model are no longer a refusal — only a
translation needs to read them. A translation that fails on an answer that
parsed is 502: the upstream answered and the gateway cannot say it in the
client's protocol.
Three tests described the passthrough surface that /v1/embeddings and /v2/rerank no longer run through, and two of them asserted behaviour the design replaces: a 2xx body a JSON protocol cannot read was forwarded verbatim with the upstream's own status. A gateway that never read the answer cannot claim to have served the request, so it refuses in its own words and the upstream still counts as called having reported nothing. The rest are kept as they were, against the pipeline: a failed usage write leaves the answer alone and is still reported, and the last candidate's refusal reaches the client with its status, its headers and its own words.
Whether a request streams is written in the request — `stream: true` in a JSON body, a form field in a multipart upload — and the body can only be read once. The prologue read it internally and took the answer as a parameter, so a family that learns it from the body had no order in which to call it. Guessing `false` is not harmless: the abort controller a streaming run cancels its upstream with is minted from that flag, so a client that disconnected would stop cancelling anything. Reading the ingress is now its own step, and the run opens once the handler knows what was asked for — which also lets the requested model reach the dump through the context that stamps it rather than a second call afterwards.
…t ends A streaming family's answer is the stream, and the seam could only build a response from bytes it already had. It now takes frames as an answer of its own shape: the response is staged on the context first, because hono's SSE helper constructs the response itself and headers passed to a constructor would be lost. A stream's usage arrives with its last chunk, which is after the run has answered — the fact carrying it is a promise, and settling from it is the epilogue's job. Settlement's write is extracted so both paths do the same thing: the stage settles what the ending had already read, and the epilogue settles what only the drained stream could say.
Completions and audio were the only families whose edge handed the upstream's headers on unread, so a body the gateway re-serialized carried the upstream's own content-length and content-encoding — a client would have been told the wrong length for bytes it was actually given. Both now filter as the other families do, and both exits declare the key the seam reads off them.
The release was scheduled the moment the run returned, so for a streaming family it consumed the very frames the client was waiting for — one connection has one reader, and the two raced for every chunk. The terminal [DONE] was the frame most often lost. Reading the frames out to the client is what releases the body they came from, so the drain now waits for that to finish, in a finally so a client that stopped reading still leaves nothing open. A buffered answer was serialized from facts the run already held and still releases at once. Completions serves through its pipeline; the streaming half settles from the usage its last chunk carries.
The reader ran until the upstream closed, so an upstream that holds the connection open past transcript.text.done held the client's stream open with it. Returning at the terminal event closes the read, which cancels the upstream — the behaviour the replaced surface had. The route stays on its existing surface. Migrating it needs a decision this change cannot make: srt and vtt are parsed into cues and rendered back, which the protocol states is not byte-identical to what the upstream sent, and the replaced surface forwarded those documents unchanged.
Two families' endings dialled outside the try the other three had, so a refused connection ended the run as a fault: no failover to the next candidate, no performance row, and a 500 where the replaced surface answered 502. Attribution moves ahead of the dial with it, so an attempt that never returned names its own candidate. A streaming answer returned hono's response without finalizing the gateway context, and finalize is what writes the dump — so a streamed request recorded nothing at all. The upstream's frames are recorded where they are read, before the edge decides which of them the client sees. The prologue's dump sink was a stub that discarded every event while still telling the runner to accumulate them, so a key with retention configured paid for a recording nobody could read. It is absent until it has somewhere to go.
A request carrying stream: true reaches the upstream, and the pipeline cannot read what comes back — the images stream is not carried as facts, so the answer became a 502 after the call had already been made and charged for. The replaced surface forwarded those frames. This is the same blocker audio has: a response shape the fact space does not model yet. The pipeline, its tests and the protocol contracts stay; what is withdrawn is the route, until the shape it cannot carry is carried.
…t two families Completions and audio wrapped an upstream's error body in a second envelope, so a client read an escaped JSON document where the message should have been. They now hand the upstream's own object on, as the other three families do. Three ingress keys were declared and never provided or read, under a comment claiming every request recorded them. The method, path and body are the dump's to record from the request context; what a family actually hands over is the headers, because every ending forwards what a provider may of them. Rerank's attempt module had no referrers left after the pipeline took the route. Two comments cited design documents that are not in the repository.
The family produced a billed set and composed no settlement, so nothing it did was ever written. What stopped it was a request-side need on serve.model that the stage never read: declaring it put a resolved model in the entry contract of a family that resolves none. Settlement reads what came back and nothing on the way down, which is what the declaration now says.
The request detail panel read every record as a pair of edges — request headers, request body, response headers, response body. A pipelined turn is recorded as its whole run instead, so the panel would have read `record.request` off a record that has none. It now dispatches on the record's shape. An edge record draws the four sections it always did; a run record draws its event stream, one block per NDJSON line, labelled with the event's own kind and with whichever stage, object or stream it names. A line that does not parse is shown as it was stored with the failure named, because a record that renders as nothing would read as an empty run. The severity badge reads `meta.status` rather than the response half's copy of it — the metadata is what both shapes have, and the two were always the same number.
…t ended A streaming turn settled as a success whatever happened to it, so a stream that stopped before its terminator wrote a neutral performance row and a dump success — a turn that never finished, reported as one that produced what it said it would. What blocked this was that the write has to be registered while the request is still live for the platform to bind it, and the outcome is only known when the stream ends. The two stop pulling against each other once the family's meter reports both together: it knows the usage and the terminator at the same moment, so it resolves one value carrying each. The audio family arrived at this shape independently and had to write its own epilogue around the seam to use it. Widening the seam is what lets that handler collapse back into it.
The reader dispatch was covered per layer and not end to end, so nothing said that a run record survives the list query, the detail fetch and the wire codec together — which is the path the dashboard actually takes. One list carries a run and a pair of edges, and a detail fetch of each hands back what it is: the run's NDJSON, and the edge's request half. The run fixture encodes the events the way the sink does, object space and folding included, rather than spelling the stored lines out by hand.
Both record shapes fell back to a transport-level read failure when no hook had stamped an error, and each spelled the precedence out for itself — a request-body read failure outranking a response-body one, twice. It belongs beside the assembly it feeds, so it moves there and both call it.
The run's event stream had a writer and no producer: nothing opened one, and a pipelined endpoint still wrote the edge record its handler no longer builds. The shape follows the endpoint. A pipelined turn opens the run recording in its prologue, so one turn is written once and the two shapes never both appear for it. What a turn stamps on either — the requested model, the upstream that answered, what it billed, how it failed — is now a named surface both implement, which is what keeps the metadata common and the stages ignorant of which they are stamping. A frame becomes the stream.frame event the format already specified, which is that event's first producer.
Every non-chat family serves through its pipeline now, so the scaffold whose whole job was to forward a body it had not parsed and hand back one it had not read has no callers. passthrough-serve.ts, passthrough-attempt.ts and their tests go, and PassthroughServeApiName with them — the set of endpoints it named is not a set anything groups by any more. What survives is the one piece that was never about forwarding: reading a JSON body with a model field, which is a request parser and now says so in its name. Audio's respond.ts and usage.ts went the same way — the strategy responder and the usage reader the pipeline replaced, both without a referrer since the family was routed. Audio's handler collapses back into the seam. It wrote its own epilogue because the seam could not carry an upstream's absent media type or a stream's outcome; both now travel, so the two agree again.
# Conflicts: # packages/pipeline/__tests__/boundary_test.ts
Takes the host-independence fix for the ownership tests, which this branch's own suite runs.
The audio test required `Symbol.asyncDispose` to be present on an async generator. That is true on Node 24 and false on Node 22, which CI runs and `apps/platform-node` supports, so the suite passed locally and failed there. What it means to assert is the run's answer, which is the same on every host. Four comments explained the stream wrapper by a mechanism that is gone. They said a bare generator in the record would be adopted as a resource and `return()`ed by the sweep — true when the runner detected ownership structurally, and not since it started claiming it through `own()`. The wrapper still earns its place, for the reason the audio test already stated: it says which value in the record is the resource. That is the upstream's body, and nothing else.
# Conflicts: # packages/gateway/__tests__/data-plane/chat/openai-responses/websocket_test.ts # packages/gateway/src/data-plane/chat/openai-responses/websocket.ts
# Conflicts: # packages/gateway/__tests__/data-plane/openai-audio/usage_test.ts # packages/gateway/__tests__/data-plane/openai-embeddings/failures_test.ts # packages/gateway/__tests__/data-plane/openai-images/stream_test.ts # packages/gateway/__tests__/data-plane/shared/passthrough-attempt_test.ts # packages/gateway/__tests__/data-plane/shared/passthrough-serve_test.ts # packages/gateway/src/data-plane/openai-audio/http.ts # packages/gateway/src/data-plane/openai-audio/pipeline.ts # packages/gateway/src/data-plane/openai-audio/respond.ts # packages/gateway/src/data-plane/openai-audio/usage.ts # packages/gateway/src/data-plane/openai-completions/http.ts # packages/gateway/src/data-plane/openai-completions/pipeline.ts # packages/gateway/src/data-plane/openai-embeddings/http.ts # packages/gateway/src/data-plane/openai-embeddings/pipeline.ts # packages/gateway/src/data-plane/openai-images/http.ts # packages/gateway/src/data-plane/openai-images/pipeline.ts # packages/gateway/src/data-plane/shared/passthrough-serve.ts # packages/protocols/src/openai-audio/index.ts # packages/protocols/src/openai-audio/stream.ts # packages/protocols/src/openai-audio/subtitles.ts # packages/protocols/src/openai-audio/transcription.ts # packages/protocols/src/openai-completions/index.ts # packages/protocols/src/openai-completions/parse.ts # packages/protocols/src/openai-completions/stream.ts # packages/protocols/src/openai-embeddings/index.ts # packages/protocols/src/openai-embeddings/translate.ts # packages/protocols/src/openai-images/request.ts # packages/protocols/src/openai-images/response.ts # packages/protocols/src/openai-images/stream.ts # packages/protocols/src/openai-images/to-sse.ts
…ines
The rename that vendored `completions`, `embeddings`, `imagesGenerations`,
`imagesEdits` and `audioTranscriptions` landed against `main`, where the non-chat
pipelines did not exist yet. This branch's own protocol contracts, pipelines and
handlers kept the bare names, so the merge left a tree where `ModelEndpoints` and
the provider interface spoke the canonical vocabulary and everything this branch
added spoke the old one.
Every identifier, module path, fact key and internal string that names one of the
five now carries its vendor and its endpoint:
completions -> openaiCompletions / OpenAICompletions
embeddings -> openaiEmbeddings / OpenAIEmbeddings
imagesGenerations -> openaiImagesGenerations / OpenAIImagesGenerations
imagesEdits -> openaiImagesEdits / OpenAIImagesEdits
audioTranscriptions -> openaiAudioTranscriptions / OpenAIAudioTranscriptions
The fact keys keep their dotted layout and change only the protocol segment:
`response.completions.rendered` becomes `response.openaiCompletions.rendered`,
`ingress.audioTranscription.responseFormat` becomes
`ingress.openaiAudioTranscription.responseFormat`, and so on across all five
families. The composed pipeline names travel with them, so a `run()` failure now
names `openaiEmbeddingsServe` rather than `embeddingsServe`.
Comments, test names and boundary messages take the prose spelling the chat
rename established: "OpenAI Completions", "OpenAI Embeddings", "OpenAI Images
Generations", "OpenAI Images Edits", "OpenAI Audio Transcriptions", and "OpenAI
Images" for the family that spans two endpoints.
Test files and directories follow the module they cover:
`__tests__/data-plane/{audio,completions,embeddings,images}-pipeline_test.ts`
take the `openai-` prefix, and `protocols/__tests__/{audio,embeddings,images}`
move under `openai-audio`, `openai-embeddings` and `openai-images`.
Deliberately left alone:
- Public URL paths and the fragments built from them: `/v1/completions`,
`/v1/embeddings`, `/v1/images/{generations,edits}`, `/v1/audio/transcriptions`,
and the `Model X does not support the /embeddings endpoint.` refusals that
quote them.
- Persisted, CHECK-constrained `operation` values passed to
`upstreamPerformanceContext`: `text_completion`, `embeddings`,
`image_generation`, `image_edit`, `audio_transcription`.
- `rerank`, which is a model kind fanning out to six vendor wire protocols, and
its pipeline, facts and tests.
- Stale keys seeded on purpose: the endpoint-key migration test's pre-migration
rows, and the import test that pins a pre-rename backup being refused by name.
- Ordinary English: "text completions" and "a text-completion model", the
`embeddings` field of a canonical response and the local that builds it, the
`images` field of a canonical edits request, `parseEmbedding` / `renderEmbedding`
over a bare vector, and "this model is not an embeddings model" about a kind.
- Foreign paths inside reference URLs, including
`vscode-copilot-chat/.../remoteEmbeddingsComputer.ts`, which a mechanical pass
had rewritten.
The one behavioural fix the sweep turned up: `openai-images/stream_test.ts` seeded
an upstream whose stored model declared `imagesGenerations` / `imagesEdits`. Those
keys no longer exist, and the endpoint validator throws on the first key it does
not recognize, so the row would have been refused at load.
The record is closed at the seam — `finalizeGatewayResponse` is what writes it — so an exception escaping past `serveThrough` never reached that call. The turn that most needs explaining was the one that left nothing behind: no row in the dashboard, no event stream, no reason. Only the OpenAI Responses WebSocket turn had ever called `dump.failed`, which is why the gap survived: the transport that owns its own response also owns its own catch. So the seam catches. What goes to the client is the same envelope the app's own handler writes, from the same function, stack included — a gateway's own fault is the operator's to read — and the record is closed with the reason on it and every stage the run did get through still in it, which is what says how far the turn got. `run` drains what it opened before it rethrows, so nothing is left outstanding to release here; only the record, and the reason. The audio family's header said it writes its own epilogue rather than going through the seam. It named the two things the seam owed it — a null content type for a document whose upstream declared none, and a reading that carries both what was billed and whether the stream finished — and both were given, at which point the handler became `serveThrough` and the prose did not follow. Verified by mutation: rethrowing instead of recording leaves zero stored records where the test expects one.
… at it
The format says a stream sits at its key as `{"$stream": n}`, that its frames
arrive afterwards naming that id, and that `stream.end` is when its record is
complete. Only the middle third existed. `streamFact` had no production caller,
so no fact anywhere carried a reference; frames went out under a hardcoded
`streamId: 1`, naming a stream nothing had opened; and `stream.end` had no
producer at all. A reader could see the frames a turn served, in order, and could
not tell which key they belonged to, could not tell two streams apart, and could
not tell a stream that ended from one that was cut off.
`recordStream` is the producer, and what it hands back *is* the reference: the
value a family puts at its fact is the branded iterable, so the fact encodes as
`{"$stream": n}` and every frame that follows names the same n. The id comes from
the recording rather than being fixed. `stream.end` is written where the source
ran out on its own — a reader that stopped early never reaches it, which is what
makes its absence mean "cut short".
One stream is usually held at two keys: the answer the ending provided, and the
framing the edge built over it for the client. `streamReferenceOf` carries the
reference onto the wrapper, so both point at one record — a reframing is not a
second stream.
Recording moves outside the meter in the two families that had threaded a
callback into it. That is what makes "reading is what records" true: the release
path drains the stream underneath the recorder, so a losing attempt nobody read
now contributes nothing instead of contributing frames the client never saw.
Audio recorded nothing at all. Its stream carries bare protocol events rather than
protocol frames, which is why it had been skipped and why `recordStream` asks that
family how one becomes the other — the record holds frames and a cast there would
be the record guessing. A streamed transcription was being stored as stage
boundaries with the transcript missing.
The edge record has one frame log and no way to name anything in it, so its
recording writes no terminator and offers no fact. That is the shape's limit
rather than an omission, and it goes away with the shape.
Verified by mutation: dropping the terminator empties the `stream.end` assertion,
and dropping the reference fails the `$stream` one.
`defer` had no production caller. The type existed, the runner swept for it at teardown, `deferral_test.ts` exercised the whole mechanism — and no fact in the gateway was ever marked, so the sweep never had anything to find. The streamed usage reading is exactly what it names: work a stage starts and does not finish, whose value arrives with the stream's last chunk, long after the run has answered. Three families hand one up and all three now declare it. What that buys is not the settlement — the epilogue already composed the promise — but the runner seeing it: a reading that never settles is reported at teardown instead of silently never billing, and it is legible from the run's own record rather than from a capability someone remembered to pass. `Deferred<T>` is a `Promise<T>` rather than a `PromiseLike<T>`. Branding is all `defer` does, so what comes back is the same object, and the narrower interface only made a caller reach for `Promise.resolve` to get back what it already had.
`searchServePipeline` was complete and had no consumer. The endpoint still ran
the handler it was written to replace, so the family that had been given an edge,
a settlement, two endings and a fact space was reachable only from its own tests.
The seam is `http.ts`, on the shape every other family uses. What it decides
before the run is which ending the chain gets: `passthroughOpenAiSearch` pins an
upstream and everything else runs the commands locally. That is configuration
rather than request, so it is read once and the chain assembled around it — which
is why the family has one pipeline builder and no stage that branches. The
configured backend reaches the chain as a service and never as a fact: it holds
live handles built from the operator's credential, so none of it is dumpable.
The family reads its own body, as every pipelined family does. A validator
middleware would have consumed the bytes the run is given, and those bytes are
what the record holds. The refusal it writes is the same status and the same
`{error: message}` shape the validator wrote.
The pinned ending gains `response.http.headers`, filtered at the edge by the rule
every other family filters by. That replaces the relay's own blocklist, which
differed only in also forwarding `content-type` — a header the seam sets from what
the gateway actually serialized. `relay-response.ts` had no other caller and is
gone.
What the record shows is the proof: the handler this replaced opened no run at
all, so a turn on this path produced no dump however much retention the key had.
It now records the four stages the local ending composes.
The note about pinning `SearchRequest.id` to one provider account travelled with
the dispatch it describes; the port had dropped it.
Brings main's #472 down the stack. No conflicts here — the three files it added merge clean — but one of them fails against this branch, which is the point of merging it down. `/v1/completions reaches a Completions upstream` sent a non-streaming request and answered it with `text/event-stream`. The replaced handler served that; the pipeline reads the upstream's body against what the request asked for, so a non-streaming turn handed a stream has nothing it can parse and says 502 rather than serving a body it never read. The fixture is what is wrong, not the reading: the case exists to prove the ingress header rules reach the upstream on this route, and it now asks to stream, which is what its own upstream answers with.
Six comments in this PR's own files justified a decision by naming what the surface this replaced did. That reads as a citation and resolves to nothing once this ships. Each now states the property itself, and where the point really is a contrast, contrasts with the alternative a reader can still evaluate. These were swept on #476 by mistake — the files are this PR's, and the sweep belongs where they live. The final state was already right either way, which is exactly why a per-PR review catches this and an end-state sweep does not.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #474. Draft until its base lands.
The shared space every non-chat family's pipeline extends, the stages they all run, the seam
their routes serve through — and three families moved onto it.
What is routed
/v1/embeddings/v2/rerank,/jina/v1/rerank, and the other two mounts/v1/completions/v1/images/{generations,edits}/v1/audio/transcriptionssrtandvttAlpha search keeps its own route: its pipeline composes and settles, but nothing dispatches
through it yet. The passthrough surface itself is gone.
The shared parts
Neither shared stage names a family's own key. Branching is a capability of the
framework and what something branches on is a concept in the domain, so a family hands in
the two domain-shaped things — how to narrow a candidate, and how to read an attempt's
outcome — and the stages stay written against the shared space alone. That is what lets them
drop into a pipeline over any family's larger space with no variance question to lose:
assembly reasons over declarations, and declarations are strings.
Reading the request is separate from opening the run. Whether a request streams is
written in the request, and the body can only be read once — so the run is opened after the
handler knows what was asked for. Guessing is not harmless: the abort controller a streaming
run cancels its upstream with is minted from that flag.
A failure is a value, and so is a dial that never connected. Each ending catches what the
platform raised and hands it up, because a refused connection is an outcome failover has to
see. Attribution is set before the dial, so an attempt that never returned names its own
candidate rather than the one before it.
Settlement sits above the fork, so a run bills once however many candidates it tried. A
stream states its usage in the chunk that ends it, which is after the run has answered — the
pipeline hands that up as a promise and the epilogue writes it, so the row is written once
and where the numbers actually are.
What routing the families found
Wiring is what exercised any of this. Every one of these was invisible while no route ran a
pipeline, and each is fixed here:
retry-afterstopped reaching clients.the very frames the client was waiting for. The terminal
[DONE]was what usually vanished.retention configured a warning or an error reached nothing at all.
finalize is what writes the dump, so a streamed request recorded nothing.
belongs to the family: five spell the endpoint, rerank spells out which part of the request
no candidate could satisfy.
read an escaped JSON document where the message should have been.
/embeddingsanswers without the top-levelmodelthe schema marks required, soparsing refused a body the replaced surface served.
back out unchanged; only a translation has to model it.
Two families that were held back, and what unblocked them
Both were the same shape of blocker — a response the fact space did not model — and both are
routed now.
Images. A
stream: truerequest reached the upstream and the pipeline could not read whatcame back, so the call was made, charged for, and answered 502. The stream is carried as facts
now: the events are metered as they pass, so the reading costs one pass and the client's own
stream is what drives it.
Audio.
srtandvttwere parsed into cues and rendered back, which the protocol statesis not byte-identical to what the upstream sent. The decision went the way the replaced surface
had it: a media type is upstream-owned, so the upstream's own document travels with the
upstream's own label, and the edge names a media type only where the gateway wrote the body
out of nothing the upstream sent.
The deferral question, answered
17-facts.mdsays stages have no fire-and-forget, and20-gateway.mddescribes settlementproviding a
Deferred<T>fact composed with.thenand never awaited. That subsystem wasunimplemented when this branch was written, which left the two documents describing something
no code did.
Deferred<T>is implemented in #474 and the runner's teardown settles what a run deferred.Settlement here still reaches the background scheduler rather than providing a deferred fact,
so the mechanism exists and this one caller has not moved onto it — a smaller gap than the one
originally raised, and one that no longer needs a ruling to close.