ref(manual-jobs): bound the manual jobs Redis keys - #8397
Merged
Conversation
Every manual job run left four Redis keys behind for good: the start time, the execution status, the log list and the job type. Nothing removed them, and the log list grew one entry per log line with no cap. These sit on a long-lived cluster in US, DE and US2, so the growth was slow but monotonic. The job lock in the same file was already bounded at 24 hours, and this applies the same pattern to the rest. All four keys now expire from one shared constant, 90 days. The record is human-facing history, not operational state, but it is also a re-run guard: run_job refuses a manifest job whose status key exists, and the admin UI shows View Logs instead of Run for the same reason. So a window that is too short makes a job that already ran look re-runnable. 90 days is the cap that snuba/state/retention.py puts on standard retention, and the row-touching jobs write only to tier-1 tables. Past that the rows a job acted on have aged out, so the question the record answers no longer has a target. The log list is capped at 500 lines and keeps the newest ones. view_job_logs already refused to render more than 500 lines, so a longer list was unreadable anyway, and a failing run appends its error and traceback last. The expiry is refreshed on every log line and every status write, so a long run does not lose its own state while it is still going. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ldelvoye
reviewed
Aug 26, 2026
| _redis_client = get_redis_client(RedisClientKey.MANUAL_JOBS) | ||
|
|
||
| # One run of a job may hold the lock for this long | ||
| MANUAL_JOB_LOCK_TTL_SECONDS = 24 * 60 * 60 |
Member
There was a problem hiding this comment.
Thoughts on timedelta(days=1) instead?
Member
Author
There was a problem hiding this comment.
it'd be fine and probably more readable but also every other redis expiry uses plain seconds, and keeping it in plain numbers makes it a bit harder to mix up the units since it's pinned to the type. i'll probably just leave for now 🤷
Member
There was a problem hiding this comment.
Oh whoops, I've been using timedelta for mine
ldelvoye
approved these changes
Aug 27, 2026
MeredithAnya
approved these changes
Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Manual jobs leave four keys behind:
start_time, execution_status, log, job_type. Most especially the log key grows one entry per log line (no cap).lockis already bounded at 24hr, so we're using the same pattern here, w/ a different TTL.Notes:
NOT_STARTEDand the UI will make it look like the job is re-runnable. So we want a TTL that gives re-runs plenty of time to happen and avoid this issue. Also 90 days is the standard retention cap viasnuba/state/retention.py. After that point the rows the job acts on have aged out anywaysview_job_logsalready refuses to render more than 500 lines anywaysRefs INFRENG-480
🤖 Generated with Claude Code