ref(snuba): cap snuba-api worker memory via Granian worker settings - #4455
Open
mcujba wants to merge 1 commit into
Open
ref(snuba): cap snuba-api worker memory via Granian worker settings#4455mcujba wants to merge 1 commit into
mcujba wants to merge 1 commit into
Conversation
Snuba has served the API with Granian rather than uWSGI since getsentry/snuba#7566, so the UWSGI_MAX_REQUESTS and UWSGI_DISABLE_LOGGING entries in x-snuba-defaults have been inert for months. UWSGI_MAX_REQUESTS in particular reads as if it provides worker recycling, but nothing consumes it. Granian's equivalents are exposed by settings_self_hosted.py as SNUBA_API_WORKERS_MAX_RSS and SNUBA_API_WORKERS_LIFETIME, both of which default to None. With neither set, the snuba-api worker is never recycled and grows unbounded. On a low-traffic instance this reached 7.3 GB after 15 days against a ~200 MB baseline. Replace the dead vars with pass-through entries and document defaults in .env, matching the SENTRY_EVENT_RETENTION_DAYS convention. Empty values are falsy in settings_self_hosted.py and resolve to None, so clearing them restores today's behaviour.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4453
Problem
Snuba has served the API with Granian rather than uWSGI since getsentry/snuba#7566 (January 2026). Two consequences have gone unnoticed since:
UWSGI_MAX_REQUESTSandUWSGI_DISABLE_LOGGINGinx-snuba-defaultsare inert. Nothing reads them.UWSGI_MAX_REQUESTS: "10000"in particular reads as though it provides worker recycling, which makes the gap harder to spot.Granian's recycling options are never set, so they default to disabled.
snuba/settings/settings_self_hosted.pyreadsSNUBA_API_WORKERS_MAX_RSSandSNUBA_API_WORKERS_LIFETIMEfrom the environment and passes them toGranian(workers_max_rss=..., workers_lifetime=...). Both areNoneunless supplied, and this compose file supplies neither.The result is a
snuba-apiworker with no memory ceiling. On my instance it reached 7.3 GB after 15 days against a ~200 MB baseline, withVmHWM == VmRSSthroughout — monotonic growth, never released. Since no service here setsmem_limit, the eventual OOM is global and kills whichever process happens to be largest rather than the one responsible.Notably, these knobs were added by getsentry/snuba#7735, titled "ref(api): add server settings, allow self-hosted overrides through env" — they appear to have been built for this deployment, but were never wired up here.
Change
Replace the two dead
UWSGI_*entries with pass-through entries, and document defaults in.env. This follows the existingSENTRY_EVENT_RETENTION_DAYSconvention in this file.Chosen defaults: 1024 MiB and 86400 s. The RSS ceiling gives roughly 5x headroom over a healthy worker, and the lifetime matches what
sentry/sentry.conf.pyalready uses for its own uWSGI workers.Clearing either value in
.envrestores current behaviour exactly — an empty string is falsy insettings_self_hosted.pyand resolves toNone.Testing
Applied on self-hosted 26.6.0 (Debian 12, 4 vCPU / 32 GB). The worker starts at ~200 MB with the cap in place, and host memory dropped from 95% to 74%. Granian confirmed as the server both in
/proc/<pid>/mapsand in the container startup log ([INFO] Starting granian (main PID: 1)).I verified the file still parses and all 64 services resolve.
Two points for reviewers
Placement. I put these in
x-snuba-defaults, which applies to every Snuba service rather than justsnuba-api. The consumers ignore the vars, and it is where theUWSGI_*entries already lived, so the diff stays minimal. Happy to scope them to thesnuba-apiservice instead if you prefer.Dropping
UWSGI_DISABLE_LOGGING. Granian leaves access logging disabled by default and Snuba'sserve()does not enable it, so removing this should not change output. Worth a second opinion in case something else relied on it.Please also sanity-check the default values — you have far better visibility than I do into what a busy instance needs, and I would rather ship a ceiling that is too generous than one that causes respawn churn.