Skip to content

ref(snuba): cap snuba-api worker memory via Granian worker settings - #4455

Open
mcujba wants to merge 1 commit into
getsentry:masterfrom
mcujba:ref/snuba-api-worker-memory-cap
Open

ref(snuba): cap snuba-api worker memory via Granian worker settings#4455
mcujba wants to merge 1 commit into
getsentry:masterfrom
mcujba:ref/snuba-api-worker-memory-cap

Conversation

@mcujba

@mcujba mcujba commented Aug 7, 2026

Copy link
Copy Markdown

Fixes #4453

Problem

Snuba has served the API with Granian rather than uWSGI since getsentry/snuba#7566 (January 2026). Two consequences have gone unnoticed since:

  1. UWSGI_MAX_REQUESTS and UWSGI_DISABLE_LOGGING in x-snuba-defaults are inert. Nothing reads them. UWSGI_MAX_REQUESTS: "10000" in particular reads as though it provides worker recycling, which makes the gap harder to spot.

  2. Granian's recycling options are never set, so they default to disabled. snuba/settings/settings_self_hosted.py reads SNUBA_API_WORKERS_MAX_RSS and SNUBA_API_WORKERS_LIFETIME from the environment and passes them to Granian(workers_max_rss=..., workers_lifetime=...). Both are None unless supplied, and this compose file supplies neither.

The result is a snuba-api worker with no memory ceiling. On my instance it reached 7.3 GB after 15 days against a ~200 MB baseline, with VmHWM == VmRSS throughout — monotonic growth, never released. Since no service here sets mem_limit, the eventual OOM is global and kills whichever process happens to be largest rather than the one responsible.

Notably, these knobs were added by getsentry/snuba#7735, titled "ref(api): add server settings, allow self-hosted overrides through env" — they appear to have been built for this deployment, but were never wired up here.

Change

Replace the two dead UWSGI_* entries with pass-through entries, and document defaults in .env. This follows the existing SENTRY_EVENT_RETENTION_DAYS convention in this file.

-    UWSGI_MAX_REQUESTS: "10000"
-    UWSGI_DISABLE_LOGGING: "true"
+    SNUBA_API_WORKERS_MAX_RSS:
+    SNUBA_API_WORKERS_LIFETIME:

Chosen defaults: 1024 MiB and 86400 s. The RSS ceiling gives roughly 5x headroom over a healthy worker, and the lifetime matches what sentry/sentry.conf.py already uses for its own uWSGI workers.

Clearing either value in .env restores current behaviour exactly — an empty string is falsy in settings_self_hosted.py and resolves to None.

Testing

Applied on self-hosted 26.6.0 (Debian 12, 4 vCPU / 32 GB). The worker starts at ~200 MB with the cap in place, and host memory dropped from 95% to 74%. Granian confirmed as the server both in /proc/<pid>/maps and in the container startup log ([INFO] Starting granian (main PID: 1)).

I verified the file still parses and all 64 services resolve.

Two points for reviewers

Placement. I put these in x-snuba-defaults, which applies to every Snuba service rather than just snuba-api. The consumers ignore the vars, and it is where the UWSGI_* entries already lived, so the diff stays minimal. Happy to scope them to the snuba-api service instead if you prefer.

Dropping UWSGI_DISABLE_LOGGING. Granian leaves access logging disabled by default and Snuba's serve() does not enable it, so removing this should not change output. Worth a second opinion in case something else relied on it.

Please also sanity-check the default values — you have far better visibility than I do into what a busy instance needs, and I would rather ship a ceiling that is too generous than one that causes respawn churn.

Snuba has served the API with Granian rather than uWSGI since
getsentry/snuba#7566, so the UWSGI_MAX_REQUESTS and UWSGI_DISABLE_LOGGING
entries in x-snuba-defaults have been inert for months. UWSGI_MAX_REQUESTS
in particular reads as if it provides worker recycling, but nothing
consumes it.

Granian's equivalents are exposed by settings_self_hosted.py as
SNUBA_API_WORKERS_MAX_RSS and SNUBA_API_WORKERS_LIFETIME, both of which
default to None. With neither set, the snuba-api worker is never recycled
and grows unbounded. On a low-traffic instance this reached 7.3 GB after
15 days against a ~200 MB baseline.

Replace the dead vars with pass-through entries and document defaults in
.env, matching the SENTRY_EVENT_RETENTION_DAYS convention. Empty values
are falsy in settings_self_hosted.py and resolve to None, so clearing
them restores today's behaviour.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

snuba-api worker memory recycling is disabled by default, and compose still sets dead UWSGI_* vars after the Granian migration

1 participant