Skip to content

fix: share one HTTP client per provider and size its connection pool - #1021

Open
alexluong wants to merge 1 commit into
mainfrom
fix/shared-http-client
Open

fix: share one HTTP client per provider and size its connection pool#1021
alexluong wants to merge 1 commit into
mainfrom
fix/shared-http-client

Conversation

@alexluong

@alexluong alexluong commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

Closes #1017.

Outpost built a separate http.Client for every destination and never configured pooling on it, so each inherited Go's default of two idle connections. Above two concurrent deliveries to a destination, reuse collapsed to roughly one new connection per delivery.

What changed

One client per provider. Nothing in the client configuration varies per destination — user agent, proxy settings and the transport wrapper all come from provider-level options fixed at registration — so the client moves from CreatePublisher to the provider constructor (destwebhook, destwebhookstandard, desthookdeck). This also means publisher-cache eviction no longer throws away the connection pool.

The rule the code now documents: only connection-level concerns justify a separate client. If per-destination proxy or client certificates are needed later, transports should be keyed by configuration and shared within a key.

Derived pool sizing (internal/destregistry/connpool.go):

source floor / cap
per-host (depth) DELIVERY_MAX_CONCURRENCY floored at 2 (Go's default)
total (breadth) RLIMIT_NOFILE / 4 floored at 100, capped at 4096

Breadth doesn't derive from concurrency — a deployment with 600 destinations each receiving one delivery a minute has almost no concurrency but needs ~600 warm connections. The hookdeck provider talks to one host, so it gets a depth-only pool.

No new configuration. The correct total depends on the active destination count and the host's FD limit, which an operator would need to know both of to set sensibly. Instead:

  • resolved values logged at startup (delivery_max_idle_conns, delivery_max_idle_conns_per_host, delivery_conn_pool_fd_limit)
  • new outpost.delivery_connections counter, dimensioned by type and reused — the signal that the ceiling is binding

Adding a knob later is backward-compatible; removing one isn't.

Platform. RLIMIT_NOFILE is read behind a unix build tag; Windows falls back to an assumed 1024.

Tests

internal/destregistry/httpclient_pool_test.go counts TCP connections opened at an httptest server via ConnState, swept across concurrency levels against both fast and slow destinations — the two regimes fail differently (latency vs. ephemeral ports). Connections opened track the concurrency level, not the request count.

A control case runs the identical workload through a stock-default client so the assertion can't pass trivially: at concurrency 32, stock opens ~120 connections for 320 requests, sized opens 32.

destwebhook_connpool_test.go covers the provider half — 8 publishers, 40 requests, still bounded by concurrency.

Behavior changes worth flagging

  • An invalid DESTINATIONS_WEBHOOK_PROXY_URL now fails at startup instead of on first publish.
  • Idle connections are held on the receiver for up to 90 seconds after a delivery. Those connections were opened anyway; pooling changes whether they're held afterward.
  • The pool is shared across tenants. Connections are per-host so there's no correctness concern, but a busy tenant can evict a quiet tenant's idle connections; the per-host limit bounds the blast radius.

Verification

go test -short ./... clean. Pool tests run at -count=15 for flake.

🤖 Generated with Claude Code

Outpost built a separate http.Client for every destination and never
configured connection pooling on it, so each one inherited Go's default of
two idle connections. Above two concurrent deliveries to a destination,
reuse collapsed to roughly one new connection per delivery — latency
against fast destinations, TIME_WAIT accumulation against slow ones.

Nothing in the client configuration varies per destination: user agent,
proxy settings and the transport wrapper all come from provider-level
options fixed at registration. So the client is now built once in the
provider constructor, which makes the per-host idle limit meaningful and
gives a real ceiling on total idle connections.

Pool sizing is derived rather than configured:

- per-host (depth) from DELIVERY_MAX_CONCURRENCY, floored at Go's default
- total (breadth) from RLIMIT_NOFILE — a quarter of the soft limit,
  floored at 100 and capped at 4096. Fan-out is the normal shape for this
  product, so Go's default of 100 is the wrong thing to inherit; a
  deployment with 600 low-rate destinations needs breadth it can't derive
  from concurrency.

The hookdeck provider talks to one host, so it gets a depth-only pool.

Deliberately no new env vars: the correct total depends on the active
destination count and the host's FD limit, which an operator would have
to know both of to set sensibly. Instead the resolved values are logged
at startup alongside the FD limit they came from, and a new
outpost.delivery_connections metric reports connections opened against
connections reused — the signal that the ceiling is binding.

Reading RLIMIT_NOFILE is Unix-only; Windows falls back to an assumed 1024.

Closes #1017

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@alexluong

alexluong commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator Author

Added #1022 in d4d1175 — pulled back out. #1025 rewrites the same signature formatter constructors this touched, so keeping the two together meant one PR blocking the other on a merge conflict for no reason. #1022 will land separately, based on #1025.

This PR is #1017 only: connection pooling.

@alexluong
alexluong force-pushed the fix/shared-http-client branch from d4d1175 to 9a8a215 Compare August 7, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Webhook deliveries open a new connection for nearly every request

1 participant