fix(storage): harden S3 client against stale keep-alive socket hangs - #24
Merged
Conversation
S3StorageProvider (used by backfeed-api, runa-api, moment-api for uploads) built its client with the default @aws-sdk/client-s3 request handler, which pools keep-alive sockets with no aborting request timeout. Against Garage at s3.omni.dev (reached over a NAT/edge that silently idle-drops connections) the pool fills with zombie sockets and every reuse hangs ~30s until the upstream 503s. This is the failure that took down halo-api media on 2026-09-01. Build the client through a new createResilientS3Client (exported for direct use too): keepAlive:false so a socket is never reused, plus requestTimeout with throwOnRequestTimeout (a plain requestTimeout on @smithy/node-http-handler v4 only warns and keeps hanging on Bun) and a connectionTimeout. Added @smithy/node-http-handler as an optional peer (mirroring the @aws-sdk/client-s3 treatment) plus a dev dep, a regression test, and a changeset. Consumers pick up the fix on their next ref bump.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
S3StorageProvider(used by backfeed-api, runa-api, moment-api for uploads) built its client with the default@aws-sdk/client-s3request handler, which pools keep-alive sockets with no aborting request timeout. Against Garage ats3.omni.dev(reached over a NAT/edge that silently idle-drops connections) the pool fills with zombie sockets and every reuse hangs ~30s then 503s. This is the failure that took down halo-api media serving on 2026-09-01.Fix
The client is now built through a new
createResilientS3Client(exported from./storageand the package root for direct use):keepAlive: falseso a socket is never reused (a dropped connection can't become a zombie).throwOnRequestTimeout: truewithrequestTimeout(a plainrequestTimeouton@smithy/node-http-handlerv4 only warns and keeps hanging on Bun) +connectionTimeout.@smithy/node-http-handleradded as an optional peer dependency (mirroring@aws-sdk/client-s3) plus a dev dep; both imported lazily to stay off the module load path for non-S3 consumers. Regression test + changeset included.Consumer propagation
This is a git-ref-pinned library, so backfeed-api, runa-api, and moment-api only pick up the fix once they bump their
@omnidotdev/providersref. (backfeed-api's serve path was hardened directly in a separate PR; this covers its upload path.)Verification
tsc --noEmit,bun run build,bun test src/storage/requestHandler.test.ts(1 pass), biome, knip: clean.