fix: share one HTTP client per provider and size its connection pool - #1021
Open
alexluong wants to merge 1 commit into
Open
fix: share one HTTP client per provider and size its connection pool#1021alexluong wants to merge 1 commit into
alexluong wants to merge 1 commit into
Conversation
Outpost built a separate http.Client for every destination and never configured connection pooling on it, so each one inherited Go's default of two idle connections. Above two concurrent deliveries to a destination, reuse collapsed to roughly one new connection per delivery — latency against fast destinations, TIME_WAIT accumulation against slow ones. Nothing in the client configuration varies per destination: user agent, proxy settings and the transport wrapper all come from provider-level options fixed at registration. So the client is now built once in the provider constructor, which makes the per-host idle limit meaningful and gives a real ceiling on total idle connections. Pool sizing is derived rather than configured: - per-host (depth) from DELIVERY_MAX_CONCURRENCY, floored at Go's default - total (breadth) from RLIMIT_NOFILE — a quarter of the soft limit, floored at 100 and capped at 4096. Fan-out is the normal shape for this product, so Go's default of 100 is the wrong thing to inherit; a deployment with 600 low-rate destinations needs breadth it can't derive from concurrency. The hookdeck provider talks to one host, so it gets a depth-only pool. Deliberately no new env vars: the correct total depends on the active destination count and the host's FD limit, which an operator would have to know both of to set sensibly. Instead the resolved values are logged at startup alongside the FD limit they came from, and a new outpost.delivery_connections metric reports connections opened against connections reused — the signal that the ceiling is binding. Reading RLIMIT_NOFILE is Unix-only; Windows falls back to an assumed 1024. Closes #1017 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Collaborator
Author
alexluong
force-pushed
the
fix/shared-http-client
branch
from
August 7, 2026 07:13
d4d1175 to
9a8a215
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1017.
Outpost built a separate
http.Clientfor every destination and never configured pooling on it, so each inherited Go's default of two idle connections. Above two concurrent deliveries to a destination, reuse collapsed to roughly one new connection per delivery.What changed
One client per provider. Nothing in the client configuration varies per destination — user agent, proxy settings and the transport wrapper all come from provider-level options fixed at registration — so the client moves from
CreatePublisherto the provider constructor (destwebhook,destwebhookstandard,desthookdeck). This also means publisher-cache eviction no longer throws away the connection pool.The rule the code now documents: only connection-level concerns justify a separate client. If per-destination proxy or client certificates are needed later, transports should be keyed by configuration and shared within a key.
Derived pool sizing (
internal/destregistry/connpool.go):DELIVERY_MAX_CONCURRENCYRLIMIT_NOFILE / 4Breadth doesn't derive from concurrency — a deployment with 600 destinations each receiving one delivery a minute has almost no concurrency but needs ~600 warm connections. The
hookdeckprovider talks to one host, so it gets a depth-only pool.No new configuration. The correct total depends on the active destination count and the host's FD limit, which an operator would need to know both of to set sensibly. Instead:
delivery_max_idle_conns,delivery_max_idle_conns_per_host,delivery_conn_pool_fd_limit)outpost.delivery_connectionscounter, dimensioned bytypeandreused— the signal that the ceiling is bindingAdding a knob later is backward-compatible; removing one isn't.
Platform.
RLIMIT_NOFILEis read behind aunixbuild tag; Windows falls back to an assumed 1024.Tests
internal/destregistry/httpclient_pool_test.gocounts TCP connections opened at anhttptestserver viaConnState, swept across concurrency levels against both fast and slow destinations — the two regimes fail differently (latency vs. ephemeral ports). Connections opened track the concurrency level, not the request count.A control case runs the identical workload through a stock-default client so the assertion can't pass trivially: at concurrency 32, stock opens ~120 connections for 320 requests, sized opens 32.
destwebhook_connpool_test.gocovers the provider half — 8 publishers, 40 requests, still bounded by concurrency.Behavior changes worth flagging
DESTINATIONS_WEBHOOK_PROXY_URLnow fails at startup instead of on first publish.Verification
go test -short ./...clean. Pool tests run at-count=15for flake.🤖 Generated with Claude Code