egress: suppress per-request HTTP metrics on /ws to fix SigNoz cardinality spike

[myleshorton]

Summary

Fixes the 2× SigNoz ingestion spike the vendor flagged on May 20 (Slack thread), traced to the `unbounded-egress` service.

`otelhttp.NewHandler` on the `/ws` upgrade endpoint emits `http.server.duration.*` histograms with these attributes:

Attribute	Cardinality
`net.sock.peer.addr`	unique per peer
`net.sock.peer.port`	unique per TCP connection (ephemeral port)
`http.user_agent`	high cardinality
`http.client_ip`	unique per peer
`le`	per histogram bucket boundary

Every WebSocket upgrade creates a fresh time series across all five histogram components (`.bucket`, `.count`, `.max`, `.min`, `.sum`), plus the `http.server.request_content_length.` / `http.server.response_content_length.` families. For a service with the throughput pattern of unbounded-egress this lands as thousands of stale time series per day per metric — exactly the 2× ingestion shape SigNoz observed.

SigNoz query confirms the volume: `http.server.duration.count` from `service.name = 'unbounded-egress'` is reporting 2,087 samples/sec rate (~180M samples/day) for a service that has 1–7 concurrent WebSocket connections. The arithmetic doesn't match — almost all of that volume is new-series-per-connection, not per-sample value.

Fix

Attach a noop `MeterProvider` to the `/ws` handler so `otelhttp` doesn't emit any `http.server.*` histograms there. Tracing/span propagation still works — the handler still wraps the context. The four `ObservableUpDownCounters` declared above (`concurrent-websockets`, `concurrent-quic-connections`, `concurrent-quic-streams`, `ingress-bytes`) already cover the only signals worth keeping for this endpoint.

What we lose

Per-request HTTP latency/size histograms scoped to this exact `/ws` route. On a single upgrade endpoint where every request is GET → 101 Switching Protocols, these were:

• Method: always GET (no useful variance)
• Status code: always 101 (no useful variance)
• Duration: handshake-only, not the WebSocket lifetime
• Size: tiny request body, tiny response body, no useful signal

So we lose ~zero useful information and reclaim the SigNoz budget.

Follow-up

Going to open a defensive PR on `getlantern/telemetry` next, adding a metric `View` with an `AttributeFilter` that strips `net.sock.peer.`, `http.client_ip`, and `http.user_agent` from `http.server.` metrics at the SDK layer. That way if another service in the org adds `otelhttp.NewHandler` later, the cardinality explosion can't recur.

Test plan

• `go build ./...` clean
• `go test ./egress/...` passes (2.047s, 0 failures)
• Deploy unbounded-egress with this fix and confirm SigNoz `http.server.duration.*` series for `service.name = 'unbounded-egress'` drop to near-zero within the periodic-reader interval (60s default)
• Confirm `concurrent-websockets` and the other custom counters continue to report normally

cc Reflog (replying in the Slack thread once this lands)

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR reduces OpenTelemetry HTTP server metric cardinality for the unbounded-egress WebSocket upgrade endpoint by preventing otelhttp from emitting per-request http.server.* metrics on /ws, while keeping span/context propagation intact. This aligns with the service’s usage pattern (long-lived WebSockets) and addresses the reported SigNoz ingestion spike caused by high-cardinality attributes on upgrade requests.

Changes:

• Attach a noop MeterProvider specifically to the /ws otelhttp.NewHandler wrapper to suppress http.server.* metrics emission for WebSocket upgrades.
• Add inline documentation explaining the high-cardinality mechanism and why suppressing these metrics is acceptable given existing custom concurrency/ingress counters.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.