[cuebot] Fix issue on getWhatDependsOn(Frame) (#2278) by aghiles · Pull Request #1 · aghiles/OpenCue

aghiles · 2026-07-23T20:51:01Z

Related Issues

Summarize your change.

86664de Fixes the indentation to make queries legible
ea5abd0 Fixes a bug on the GET_WHAT_DEPENDS_ON_FRAME query. Ready
[cuebot] Missing parentheses in getWhatDependsOn(Frame) SQL changes WHERE clause scope AcademySoftwareFoundation/OpenCue#2277 for more details

Summary by CodeRabbit

Chores
Reformatted internal database access code for improved consistency and
readability. No functional changes or impact to user-facing features.

…ion#2278) ## Related Issues Fixes AcademySoftwareFoundation#2277 ## Summarize your change. - 86664de Fixes the indentation to make queries legible - ea5abd0 Fixes a bug on the `GET_WHAT_DEPENDS_ON_FRAME` query. Ready AcademySoftwareFoundation#2277 for more details  ## Summary by CodeRabbit * **Chores** * Reformatted internal database access code for improved consistency and readability. No functional changes or impact to user-facing features. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2278)

## Summarize your change. Queries were wrapped with `// spotless: off/on` to allow formatting them for legibility. The content of queries has been checked to ensure they contain exactly the same string as before. ## LLM usage disclosure Claude Opus was used to apply the changes and to write a scrip that confirmed the string content is exactly the same.  ## Summary by CodeRabbit * **Style** * Reformatted SQL statement constants across the database access layer for improved code readability and formatting consistency. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2297)

…olumn in two (AcademySoftwareFoundation#2313) ## Related Issues - AcademySoftwareFoundation#2314 ## Summarize your change. Bug: Down host or misreporting hosts can publish free > total (seen on DOWN hosts where /mcp metrics were never refreshed), which makes used negative and the free-fill rect extend past the cell - bleeding the bar's green into the columns to the left. New changes: 1) Bug fix: Temp bar overflow - `_paintDifferenceBar` computed `used = total - free`, which became negative when stale hosts published `free > total`. - The free-fill rect was then built with `rect.adjusted(<negative>, 0, 0, 0)`, shifting its left edge beyond the cell boundary. - Since the Temp delegate paints last in the row, its green free portion bled leftward across Idle Memory, Total Memory, GPU Memory, Physical, and Swap, causing the "green bar spanning many columns". - Clamp `used` to `[0, total]` so misreporting hosts render as fully free (all green) without overflowing the cell. - Add `painter.setClipRect(rect)` as a safeguard so future arithmetic issues cannot paint outside the intended cell. - Tighten the early-return guard to also bail when `total <= 0`. 2) New improvement: "Temp Free" column split - The combined "Temp Free" column value (e.g.: 23.5G (50%)) display made it difficult to sort by absolute free space. Sorting by percentage could rank a `1.0G / 100%` host above a `900G / 45%` host, which is counterproductive when scanning for actual available headroom. - Replace the single Temp Free column with two: a) "Temp Free" (e.g. `"23.5G"`), sorted by "free_mcp" and b) "Temp Free %" (e.g. `"50%"`), sorted by usage ratio and left empty when "total_mcp" is unknown - The adjacent Temp bar continues to sort by ratio for visual continuity. - `_formatTempCell` is replaced by `_formatTempFreeAmount` and `_formatTempFreePercent`. 3) Column width tuning: Increase default widths for columns that were truncating content under production fonts: - GPU Memory - Total Memory - Idle Memory - Temp Free - Temp Free % - Idle Cores - Idle GPUs - GPU Mem - GPU Mem Idle - Ping - Hardware - Locked - ThreadMode - Header labels and common values now fit without clipping or hover-only visibility, making Monitor Hosts easier to scan. 4) Update tests: - Replace `_formatTempCell` tests with dedicated helper coverage: - Amount renders even when total is unknown - Percentage rounds to the nearest integer - Percentage remains empty when total is unavailable - Add a delegate-wiring assertion for the new `Temp Free %` column at index `10`. Docs: - Update `cuecommander-administration-guide.md` to document all three Temp-related columns (`Temp`, `Temp Free`, `Temp Free %`) and their sorting behavior.

…ior (AcademySoftwareFoundation#2316) ## Related Issues - AcademySoftwareFoundation#2314 ## Summarize your change. Revise the 'Temp Free' and 'Temp Free %' columns tooltip to reflect percentage-based /mcp/ free space reporting, ratio-based sorting, and empty values when total /mcp/ size is unavailable.

…wareFoundation#2321) The windows-tests job was timing out because every PR rebuilt all of rqd's transitive deps from scratch on a slow Windows runner, with no job timeout (default 6h). .github/workflows/rust-pipeline.yml: - Add Swatinem/rust-cache@v2 to all four jobs so target/ and the registry are reused across runs (typically 5-10x faster on Windows after the first build). - Add timeout-minutes per job (30-60 min) so stalled runs fail fast. - Set CARGO_INCREMENTAL=0 (no upside on ephemeral CI, hurts clean builds) plus CARGO_NET_RETRY / RUSTUP_MAX_RETRIES = 10 for Windows registry flakiness. - Split windows-tests into `cargo test -p rqd --no-run` then `cargo test -p rqd` so compile-vs-run timing is obvious and the cache lands even when a test fails. - Drop `--verbose` from the Windows jobs . Log volume noticeably slows Windows runners.  ## Summary by CodeRabbit * **Chores** * Updated CI/CD pipeline configuration to improve build reliability and performance through enhanced caching mechanisms and retry configurations.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2321)

Fixes AcademySoftwareFoundation#2311  ## Summary by CodeRabbit * **Bug Fixes** * Corrected exit signal computation for Docker containers when exit codes exceed standard thresholds. * Fixed exit-status interpretation on Unix systems for proper signal detection. * **Tests** * Added validation tests for exit-status handling across container and Unix environments. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2312)

The application os not properly closing the grpc channel at shutdown, leaving this SEVERE warning: ``` Channel ManagedChannelImpl{logId=2787, target=elk0815:8444} was not shutdown properly ``` Explanation: `RqdClientGrpc` (cuebot/src/main/java/com/imageworks/spcue/rqd/RqdClientGrpc.java:71-89) holds a Guava `LoadingCache<String, ManagedChannel>`. The `removalListener` calls `conn.shutdown()` **on cache eviction** — but never on application shutdown. The bean has no `destroy-method` (applicationContext-service.xml:39, no destroy hook), and `RqdClientGrpc` has no `shutdown()` method at all. So when cuebot stops: - The cache is GC'd - Each `ManagedChannel` is finalized **without** `shutdown()` having been called - gRPC logs `SEVERE: Channel ... was not shutdown properly!!!` per leaked channel, with the stack capturing where the channel was first allocated (the `RuntimeException: ManagedChannel allocation site` — that's a diagnostic stack, not a real exception) ## LLM usage disclosure Claude Opus was used for investigating the origin of the log and proposing a fix  ## Summary by CodeRabbit * **Bug Fixes** * Improved application shutdown procedure to properly clean up network resources and prevent potential resource leaks during shutdown. * Enhanced shutdown sequencing to ensure proper initialization and cleanup order of internal services. [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2274)

…d Frame (AcademySoftwareFoundation#2325) ## Related Issues - AcademySoftwareFoundation#2319 ## Summarize your change. Exposes when an object became eligible to run (left DEPEND for WAITING, or the job's submission time when never blocked) so callers can measure how long a frame waited to be picked up by a render proc: wait_for_pickup = frame.startTime() - frame.eligibleTime() Proto files - Adds `eligible_time` field to Frame, Job, NestedJob, and Layer. Cuebot - V41 migration adds `ts_eligible` column to the frame, layer, and job tables. - `layer.ts_eligible` and `job.ts_eligible` default to `current_timestamp`; existing rows are backfilled from `job.ts_started`. - Extends the existing DEPEND -> WAITING trigger so it stamps `frame.ts_eligible` whenever a frame unblocks. - V42 migration adds a SETUP -> WAITING trigger to stamp `ts_eligible` on the path frames actually take through Cuebot (frames are inserted as SETUP and bulk-transitioned to WAITING by `JobManagerService.activateJob`, so V41's BEFORE INSERT trigger never fires in practice). - V42 also backfills `ts_eligible` for frames that already made the SETUP -> WAITING jump before this migration ran. - `WhiteboardDaoJdbc` and `NestedWhiteboardDaoJdbc` map the column into the proto's `eligible_time` via a `getEligibleTimeInEpoch` helper, falling back to the job's submission time when `ts_eligible` is NULL (frames still in DEPEND). PyCue - Adds `Frame.eligibleTime()`. - Adds `Job.eligibleTime(format=None)` and `Layer.eligibleTime(format=None)`, mirroring the format-string behavior of `Job.startTime()`. - Propagates `eligible_time` through `NestedJob.asJob()`. - New unit tests cover all four wrappers. CueGUI - Adds an "Eligible Time" column to `FrameMonitorTree`. - Adds an "Eligible" column to `JobMonitorTree` and `LayerMonitorTree`. - "Eligible" was chosen over "Available" because the latter can imply the object will run next, when other gates (resources, paused state, etc.) may still block dispatch. Docs - Adds `eligibleTime` to the Job and Frame REST API reference schemas and example payloads. - Documents the new monitor-tree columns in the Cuetopia monitoring guide.

…hinx_docs.sh (AcademySoftwareFoundation#2334) ## Summarize your change. - The `aswf/ci-opencue:2023` container has no `pip` binary on PATH, so the job failed with `pip: command not found`. - Switch both `pip install` calls to `python -m pip install`, matching `ci/run_python_tests.sh`.

…SRF) (AcademySoftwareFoundation#2332) ## Related Issues - AcademySoftwareFoundation#2333 ## Summarize your change. - Upgrade `next` from ^14.2.35 to ^15.5.18 (resolves to 15.5.18) to fix GHSA-c4j6-fc7j-m34r: self-hosted Next.js Node server can proxy crafted WebSocket upgrade requests to arbitrary internal/external destinations, enabling SSRF against cloud metadata endpoints and internal services. - Upgrade `eslint-config-next` from 14.0.4 to ^15.5.18 to align with the Next.js major version. - Regenerate `package-lock.json` so `npm ci` in Docker resolves the patched dependency tree correctly. - Address Next.js 15 breaking change: `next/dynamic` no longer supports `ssr: false` inside Server Components. Moved client-only `DataTable` dynamic import into `app/jobs/data-table-client.tsx`, marked with `"use client"`, and imported it from `app/page.tsx` to preserve behavior. - Verified `npm run build` and clean `docker compose build --no-cache cueweb` both succeed; container starts and serves correctly on :3000. Not affected: CueWeb does not use `next/headers`, `next/cache`, middleware, `unstable_*` APIs, or async `params`/`searchParams`, so no additional Next 15 migration changes were required. See: https://cybersecuritynews.com/next-js-vulnerability-exposes-credentials/ Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com> Co-authored-by: Filipe Lacerda <tifilipebr@gmail.com>

…demySoftwareFoundation#2335) ## Related Issues Fixes AcademySoftwareFoundation#2284 ## Summary Adds a bell icon per job row in the cueweb jobs table. Clicking subscribes the browser to a system notification when the job reaches FINISHED. State is stored in localStorage. An app-wide poller checks subscribed jobs every 15 seconds and fires browser notifications via the Web Notifications API. The bell has three visual states: - **Outline bell**: not subscribed → click to subscribe - **Filled bell**: subscribed, waiting → click to cancel - **Filled bell + green dot**: notification fired → click to clear The bell is disabled (faded, with tooltip) on jobs that are already FINISHED when first viewed. ## Screenshots ### 1. First-click permission prompt <img width="800" alt="browser permission prompt on first subscribe" src="https://github.com/user-attachments/assets/69544b4a-b219-4fd0-8de9-4448f7760ffc" /> ### 2. Subscribed (bell turns filled) <img width="800" alt="bell shows filled BellRing icon after subscribe" src="https://github.com/user-attachments/assets/22083793-6a8a-4f66-831c-d115e73bcdf1" /> ### 3. Notified (OS notification fires, bell shows green dot) <img width="800" alt="bell shows filled + green dot after job finished and notification fired" src="https://github.com/user-attachments/assets/094a409e-6f48-45b4-9681-21abcc497ce3" /> ### 4. Disabled bell on already-finished jobs <img width="800" alt="bell faded on FINISHED jobs, tooltip explains why" src="https://github.com/user-attachments/assets/aed24fbc-4e76-4297-9eef-d1edc4461951" /> ### 5. Permission denied (toast refusal) <img width="800" alt="toast warning when browser notifications are blocked" src="https://github.com/user-attachments/assets/33bea528-4ff1-4ea0-ab21-7837f97bbddf" /> ## LLM usage disclosure Assisted-by: Claude / Opus 4.7 Used for implementation planning, initial code drafting, and writing unit tests.  ## Summary by CodeRabbit * **New Features** * Bell icon in the jobs table to subscribe/unsubscribe to job completion notifications. * Background poller that checks subscribed jobs and triggers browser notifications when jobs finish. * Subscriptions persist across sessions and can be managed from the UI. * Permission request for browser notifications with user-facing warning if denied. * **Tests** * Added comprehensive tests covering subscription CRUD, defensive parsing, and notification selection logic.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2335?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)   --------- Signed-off-by: Michael Vallido <vallido.michael@gmail.com>

… another user (AcademySoftwareFoundation#2340) ## Related Issues Fixes AcademySoftwareFoundation#2324 ## Summarize your change. This PR adds a **Take Ownership** action to the CueGUI Hosts context menu so users can reclaim NIMBY-locked hosts that are currently deeded to someone else, without requiring manual pycue scripting or admin intervention. ## Main changes: 1. New Hosts action in CueGUI (`cuegui/MenuActions.py`, `cuegui/HostMonitorTree.py`) - **Take Ownership** entry in the Hosts right-click menu. - Enabled only when exactly one host is selected and that host is `NIMBY_LOCKED` (matches the backend's `OwnerManagerService.takeOwnership` invariant). - Re-checks `lock_state` inside the action itself as defense-in-depth against a stale selection. 2. Explicit ownership-transfer UX - Prompts for the username that should own the host (defaults to `getpass.getuser()`). - Resolves the owner via `opencue.api.getOwner(name)`; if the username doesn't exist yet, lazily creates the owner record via `findShow(SHOW or "pipe").createOwner(name)`, same pattern as `LocalBookingWidget.deedLocalhost`. - Looks up the host's current deed and: - If the host has **no current deed** -> skip the confirmation entirely (nothing to transfer from). - If the host is owned by **another user** -> confirm with `Host <name> is currently owned by <user>. Take ownership?`. - If the host is **already owned by the requested user** -> no-op confirmation. - If the deed lookup **fails** for an unexpected reason -> log the exception and prompt with `Host <name> ownership could not be determined. Take ownership?`. - On confirm, calls `owner.takeOwnership(host_name)` and refreshes the host row. - Owner creation happens **after** confirmation (verified by a dedicated test) so a cancelled prompt never leaves a stray owner record behind. - Errors at any stage (getOwner, createOwner, takeOwnership) surface through `cuegui.Utils.showErrorMessageBox` instead of leaking gRPC tracebacks. 3. Reused existing backend; completed missing client-side plumbing - Uses existing `OwnerInterface.TakeOwnership` flow, no backend API change required. - The backend already replaces any existing deed atomically (`deedDao.deleteDeed(host)` -> `deedDao.insertDeed(owner, host)` in `OwnerManagerService.takeOwnership`). - Added a new pycue wrapper `Host.getDeed()` (`pycue/opencue/wrappers/host.py`) so CueGUI can read the current owner via the deed before confirmation. The underlying gRPC `HostInterface.GetDeed` was already implemented in `ManageHost.java`; only the Python-side wrapper was missing. 4. Tests (`cuegui/tests/test_menu_actions.py`) Added `HostActionsTests` coverage for: - `test_canTakeOwnership`: NIMBY-only enablement gate. - `test_takeOwnership`: Happy path with cross-user confirmation. - `test_takeOwnership_missingOwnerCreatesAfterConfirm`: Verifies `createOwner` is **only** called after the user confirms. - `test_takeOwnership_deedLookupFailureStillPrompts`: Generic deed-lookup failure still prompts so the user can decide. - `test_takeOwnership_unownedHostSkipsConfirmation`: `EntityNotFoundException` (host has no deed) is distinguished from a generic lookup failure, and the confirmation dialog is skipped. - `test_takeOwnership_ownerLookupFailure`: Owner lookup failure surfaces an error dialog without proceeding. - `test_takeOwnership_ignored_for_non_nimby`: Non-NIMBY host is a no-op even if the action is somehow invoked. Focused test slice runs cleanly: QT_QPA_PLATFORM=offscreen python -m pytest cuegui/tests/test_menu_actions.py -k HostActionsTests 5. Documentation (`docs/`) - `docs/_docs/user-guides/cuecommander-administration-guide.md`: Monitor Hosts -> Manage Host States: added the new action with its NIMBY-only gate and confirmation behavior. - `docs/_docs/tutorials/using-cuegui.md`: Added **Take Ownership (NIMBY-locked only)** to the right-click host menu tree. - `docs/_docs/reference/CueGUI-app.md`: Added the action to the Managing hosts "common actions include" list. - pycue Sphinx docs auto-pick up `Host.getDeed()` from its docstring via `automodule :members:`: no manual `.rst` edit needed. ## Why? The backend already supports atomic ownership replacement, but CueGUI had no UI surface for reclaiming a host owned by another user, the only workaround was a pycue shell session or an admin request. This closes that usability gap directly in the Hosts workflow. ## LLM usage disclosure: Olaiwon Ismail Model used: GPT-5.3-Codex (GitHub Copilot) Usage: - Helped me understand the codebase - Analyzed existing CueGUI host action architecture and pycue ownership wrappers - Assisted with syntax and boilerplate generation for the new host action and UI gating logic - Generated unit test scaffolding to execute the focused test slice Co-authored-by: Olaiwon Ismail <olaiwonismail@gmail.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com>

…SoftwareFoundation#2336) ## Related Issues - AcademySoftwareFoundation#2287 ## Summarize your change. The existing `Age` column in the jobs table displays time in HHH:MM format (e.g., `048:32`), which requires mental conversion to understand at a glance. I added a new `Readable Age` column that shows the same datum in a more natural format: - Jobs under a day: `2h 14m` - Jobs over a day: `3d 4h` The column is hidden by default and can be enabled through the column chooser dropdown, so existing workflows aren't disrupted. The header pairs with the existing `Age` column to make the relationship (same value, different format) obvious. Implementation notes: - New formatter `secondsToHumanAge` in `cueweb/app/utils/layers_frames_utils.ts` - New column `readable age` in `cueweb/app/jobs/columns.tsx` with a numeric `sortingFn` so rows sort by actual elapsed seconds (not the formatted string) - `getJobAgeInSeconds` clamps to non-negative and floors to whole seconds, so the sort key always matches what the formatter displays (addresses CodeRabbit feedback on clock-skew / fractional-second edge cases) - Docs updated: new row in the Job Information Columns table in `docs/_docs/user-guides/cueweb-user-guide.md` Testing: - Formatter edge cases verified: negative values, zero, minutes-only, hours-only, multi-day jobs - Confirmed the column appears in the chooser, is hidden by default, and sorts correctly by actual age in seconds - Verified end-to-end in the sandbox stack (`docker compose --profile all up`) Co-authored-by: Vishal Kumar Singh <vishal.kr.singh2021@gmail.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com>

## Summary - add a hover tooltip to CueWeb job progress bars with per-state frame counts and percentages - centralize progress segment and tooltip calculations in a utility - cover progress percentages, tooltip rows, and zero-frame jobs with unit tests Fixes AcademySoftwareFoundation#2285 ## Testing - npm test -- --runTestsByPath app/__tests__/api/utils/job_progress_utils.test.ts - npx tsc --noEmit - git diff --check

## Related Issues - AcademySoftwareFoundation#2290 ## Summarize your change. [cueweb] Add frame state filter chips - Add frame-state filter chips above CueWeb frame tables with per-state counts - Support OR-based filtering for selected frame states - Persist selected frame states in the `frameStates` URL query parameter - Add unit tests covering frame state counts and filtering behavior [cueweb] Improve frame state filter parsing and pagination behavior - Trim whitespace and deduplicate values when parsing the `frameStates` URL parameter, ensuring URLs like `?frameStates=WAITING, RUNNING` correctly preserve valid states - Reset pagination to page 1 whenever frame state filters change, preventing empty result pages after narrowing filters - Preserve the current page during polling-based refreshes via `autoResetPageIndex: false` [cueweb/docs] Document job progress tooltip and frame state filter chips - Update the user guide, reference, tutorial, quick-start, additional guides, and CueWeb README - Document the job progress bar tooltip (AcademySoftwareFoundation#2331), including per-state frame counts and percentages - Document frame state filter chips (AcademySoftwareFoundation#2330), including: - Per-state counts - OR-combined filtering behavior - URL persistence through the `frameStates` query parameter - Whitespace-tolerant and deduplicated parsing - Pagination reset behavior when filters change ## Testing - npm test -- --runTestsByPath app/__tests__/api/utils/frame_columns.test.ts - npx tsc --noEmit - git diff --check Co-authored-by: Mukunda Katta <mukunda.vjcs6@gmail.com> Co-authored-by: Ramon Figueiredo <rfigueiredo@imageworks.com>

…ademySoftwareFoundation#2327) When there's only one active show, the logic to randomize show order inadvertently removes it from the list, leading to a frozen queue.  ## Summary by CodeRabbit * **Bug Fixes** * Improved shuffle behavior so host dispatching now randomizes show order correctly when shuffle is enabled. * **Tests** * Updated unit test expectations to reflect the corrected dispatch behavior.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2327)

…SoftwareFoundation#2343) ## Related Issues - AcademySoftwareFoundation#2344 ## Summarize your change. - Replace QStyle.CE_ProgressBar in ProgressDelegate with manual QPainter rendering (dark background, green chunk, centered text) - macOS aqua style ignores opts.rect inside item-view delegate paints and renders its thin animated indicator at the row's leftmost cell, painting a stray blue line over the Name column - Manual painting matches the cross-platform approach already used by JobProgressBarDelegate, so the Progress column now renders consistently on macOS, Linux, and Windows - Clamp progress to [0, 100] to guard against backend over/underreports

…reFoundation#2346) ## Related Issues - AcademySoftwareFoundation#2347 ## Summarize your change. - Bump @sentry/nextjs from ^8.52.0 to ^10.53.1 to resolve high-severity vulnerabilities in rollup (path traversal) and the Sentry dependency chain. - Run npm audit fix to update transitive dependencies, resolving: - form-data (critical: unsafe random boundary) - @babel/plugin-transform-modules-systemjs (arbitrary code generation) - flatted (DoS / prototype pollution) - lodash (code injection, prototype pollution) - minimatch 3.x/5.x/8.x/9.x (ReDoS) - picomatch 2.x/4.x (ReDoS, method injection) - serialize-javascript (RCE, DoS) - terser-webpack-plugin (via serialize-javascript) Build, type-check, and full test suite (36/36) pass.

…areFoundation#2348) ## Related Issues - AcademySoftwareFoundation#2284 ## Summarize your change. PR AcademySoftwareFoundation#2335 added the per-job subscribe bell to the CueWeb Jobs table but shipped without doc updates. Add coverage across all cueweb doc surfaces that enumerate UI features, columns, or developer components so the feature is discoverable from every entry point. - user-guides/cueweb-user-guide.md: Add Notify column to Job Information Columns and a Job-finished notifications subsection under Real-time Updates and Monitoring covering the three bell states, the permission prompt, the 15s poll cadence, localStorage persistence, and auto-cleanup of deleted jobs. - developer-guide/cueweb-development.md: Register JobSubscriptionPoller under Core Components and SubscribeBell under UI Components; add a Subscription store note covering the cueweb:job-subscriptions key, the cueweb:subscriptions-changed event bus, and the defensive parser. - reference/cueweb.md: Add Notify column to the Jobs Table and a behavior table covering trigger, polling, notification, persistence, auto-cleanup, and cross-component sync. - quick-starts/quick-start-cueweb.md: Add Job-finished Notifications bullet to Expected Interface and step 7 to Frame Operations. - tutorials/cueweb-tutorial.md: Add a Subscribe to Job Completion step to Viewing Your Jobs and a bullet to Auto-refresh Settings.

…cademySoftwareFoundation#2349) ## Related Issues - AcademySoftwareFoundation#2279 ## Summarize your change. Replicates the CueGUI Comments dialog (cuegui/cuegui/Comments.py) in CueWeb: list, add, edit, and delete per-job comments, plus per-browser predefined-comment macros. - New page at app/jobs/[job-name]/comments with comment list, sanitized markdown preview, editor, and New / Save / Delete actions. Edit and delete are gated by author check. - Predefined macro CRUD stored in localStorage (cueweb-comment-macros), matching CueGUI's Add / Edit / Delete predefined comment workflow. - Proxy routes: POST /api/job/getcomments -> JobInterface/GetComments POST /api/job/action/addcomment -> JobInterface/AddComment POST /api/comment/action/save -> CommentInterface/Save POST /api/comment/action/delete -> CommentInterface/Delete - Helpers getJobComments / addJobComment / saveJobComment / deleteJobComment in app/utils. - Sticky-note indicator next to job names when Job.hasComment is true, with username threaded through TanStack Table meta so the indicator click opens the page with the right author context. - "Comments" entry added to the job-row context menu. - Markdown rendered via react-markdown + rehype-sanitize. Docs updated across user guide, reference, REST API reference, developer guide, tutorial, other-guides, quick start, and concepts.

…ySoftwareFoundation#2350) ## Related Issues - AcademySoftwareFoundation#2351 ## Summarize your change. Adds the standard OpenCue Apache 2.0 license header to all CueWeb source files (TS, TSX, JS, JSX, CSS) that were missing it, using the JS/TS-equivalent /* ... */ block comment form of the canonical Python header. - 107 files updated across app/, components/, lib/, public/workers/, jest/, root config (next.config.js, tailwind.config.{js,ts}, postcss.config.js, jest.config.js), and the Sentry config entries. - For files that begin with a "use client" / "use server" / "use strict" directive, the directive remains on line 1 (Next.js / V8 requirement) and the header is inserted immediately below it. - app/__tests__/utils/subscription_utils.test.ts keeps its `@jest-environment jsdom` docblock as the first comment so Jest still picks up the environment override; the license header sits below it. - next-env.d.ts is intentionally skipped, it is auto-generated and carries an explicit "should not be edited" note. No behavior changes. Type check and the full Jest suite (42 tests) pass; the cueweb Docker image builds clean.

…mySoftwareFoundation#2337) ## Related Issues - AcademySoftwareFoundation#2326 ## Summarize your change. Adds `submissionTime()` to `Frame`, exposing the parent job's submission timestamp directly on the frame object. This avoids requiring callers to fetch the parent job or overload `eligibleTime()` to infer submission time. `Frame.startTime()` represents when the frame began executing on a render host, not when the job was submitted. Since `Job.startTime()` and `Layer.startTime()` already serve as submission timestamps for those objects, only `Frame` needed this additional accessor. With `submissionTime()`, callers can now compute frame lifecycle timing directly from a single `Frame` object: - `blocked_by_depends = frame.eligibleTime() - frame.submissionTime()` - `blocked_by_pickup = frame.startTime() - frame.eligibleTime()` - `total_turnaround = frame.stopTime() - frame.submissionTime()` Proto files - Adds `submission_time` field to `Frame`. Cuebot - Updates `WhiteboardDaoJdbc.FRAME_MAPPER` to populate `submission_time` from the existing `job.ts_started` join (already aliased as `job_ts_started` for the `eligibleTime()` fallback). - No database migration is required, since the source column already exists. PyCue - Adds `Frame.submissionTime()`. - Includes a new unit test covering the Python wrapper. CueGUI - Adds a new "Submission Time" column to `FrameMonitorTree`, positioned next to the existing "Eligible Time" column. - Re-anchors the `*_COLUMN` visual-index constants in `FrameMonitorTree`, which had drifted from their intended columns over years of insertions (last touched in 2018). The new "Submission Time" column made the staleness visible by shifting `LASTLINE_COLUMN` further out of place: - `PROC_COLUMN`: 5 -> 6 (was pointing at GPUs; now correctly points to Host). - `CHECKPOINT_COLUMN`: 7 -> 8 (was pointing at Retries; now correctly points to the icon-only `_CheckpointEnabled` column where the checkmark decoration belongs). - `RUNTIME_COLUMN`: 9 -> 10 (was pointing at the hidden `_CheckpointEnabled`; now correctly points to Runtime). - `MEMORY_COLUMN`: 11 -> 12 (was pointing at LLU; now correctly points to Memory (RSS)). - `LASTLINE_COLUMN`: 15 -> 20 (was pointing at Remain; now correctly points to Last Line). - As a side effect, `redrawRunning()` now emits `dataChanged` over the correct Runtime -> Last Line range, restoring smooth repaints for Runtime/Memory/Last Line cells on running frames. The `PROC_COLUMN` foreground-color and alignment, plus the `CHECKPOINT_COLUMN` icon decoration, also land on the right cells now. - Adds a header comment noting these are visual indices that must be updated in lockstep when columns are inserted, removed, or reordered. Docs - Adds `submissionTime` to the Frame REST API reference schema and example payloads. - Updates the Cuetopia monitoring guide to document the new column. VERSION.in - Bumped up to 1.22

…Foundation#2352) The satisfy logic that runs to clean up stale depends would catch both EATEN and SUCCESS frames as a sign its dependents should be cleaned, but this behavior is only acceptable when `depend.satisfy_only_on_frame_success` is false.  ## Summary by CodeRabbit ## Release Notes * **New Features** * Added `depend.satisfy_only_on_frame_success` configuration flag to control how EATEN frames are treated during dependency recovery. When enabled (default), only SUCCEEDED frames satisfy dependencies; when disabled, EATEN frames also count as completion.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2352?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

…cademySoftwareFoundation#2341) ## Related Issues - AcademySoftwareFoundation#2335 ## Summarize your change. Builds on top of the per-job subscribe bell (AcademySoftwareFoundation#2335): - Replace the OS browser Notification API with the existing react-toastify channel (toastSuccess). Removes the first-click permission prompt; the bell now subscribes/unsubscribes immediately. Drops requestNotificationPermission() and the "permission denied" toast warning from the bell click handler. - SSR-guard getSubscriptions(): return {} when window is undefined, matching subscribeToChanges and the (now removed) requestNotificationPermission helper. - Cross-tab sync in subscribeToChanges(): also listen for the browser 'storage' event so a mutation in one tab updates bells in other open tabs. - Harden the poller tick: * Wrap each getJob() call in try/catch so one failed fetch does not reject Promise.all and silently lose the tick. * Wrap the whole tick body in try/catch so failures show up as a console.error instead of an unhandled rejection. * Re-read each entry from localStorage right before firing and skip if notifiedAt is no longer null. Narrows the cross-tab race window where two tabs both pick the same FINISHED entry and toast twice.

…reFoundation#2328) The scheduler was inadvertently booking non-threadable jobs on threadable machines. Going against cuebot's behavior.  ## Summary by CodeRabbit * **Refactor** * Streamlined thread-mode validation logic in the scheduler's core reservation and host matching systems. * Removed redundant code paths and consolidated validation logic. * **Tests** * Enhanced test coverage for thread-mode compatibility validation across different thread-mode configurations.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2328)

…yer (AcademySoftwareFoundation#2338) ## Related Issues - AcademySoftwareFoundation#2339 ## Summarize your change. [cuebot/proto/pycue/cuegui/docs] Add Layer startTime() and stopTime() Layer previously had no start/stop time accessors. Callers that needed the execution window for a layer had to fetch every frame and compute MIN/MAX client-side, which was wasteful in CueGUI (N extra RPCs per refresh for a job with N layers) and unavailable to tools that only queried layers. This change denormalizes layer timing onto `layer_stat` and exposes the values directly on the Layer API: - `Layer.startTime()` = `layer_stat.ts_started` (stamped on first entry to RUNNING) - `Layer.stopTime()` = `layer_stat.ts_stopped` (stamped on each exit from RUNNING, but surfaced as 0 until every frame on the layer has stopped) `stopTime()` remains `0` while any frame is still pending, running, or in DEPEND, mirroring `Job.stopTime()` so callers can use the same "is it done?" idiom consistently. Proto - Add `start_time` (field 24) and `stop_time` (field 25) to the `Layer` message. Cuebot - Add `ts_started` and `ts_stopped` (`TIMESTAMP WITH TIME ZONE`) to `layer_stat`. - Maintain both values through the existing `trigger__update_frame_status_counts` trigger (`AFTER UPDATE ON frame`): * entry to RUNNING stamps `ts_started` from `NEW.ts_started` via `COALESCE(...)` (first-writer-wins; retries do not update it) * exit from RUNNING stamps `ts_stopped` from `NEW.ts_stopped` (latest-writer-wins) - Copy timestamps from the updated frame row instead of sampling `current_timestamp`, ensuring `layer_stat.ts_stopped` exactly matches `MAX(frame.ts_stopped)`. - Preserve `COALESCE(..., current_timestamp)` as a fallback for callers that change state without explicitly updating timestamps. - Update `GET_LAYER` and `GET_LAYER_WITH_LIMITS` to read `layer_stat.ts_started` and `layer_stat.ts_stopped` directly, replacing two correlated frame-table aggregates per layer query. - Move "stopTime stays 0 until all frames are done" logic into `WhiteboardDaoJdbc.LAYER_MAPPER` using existing counters: `int_waiting_count + int_running_count + int_depend_count == 0` - Align layer timing behavior with existing denormalized `job.ts_started` / `job.ts_stopped`. - Add V43 migration to create and backfill both columns from existing frame aggregates, with no manual follow-up required. PyCue - Add `Layer.startTime(format=None)` and `Layer.stopTime(format=None)`, matching `Job.startTime()` formatting behavior. - Add unit tests for both epoch and formatted outputs. CueGUI - Add "Start Time" and "Stop Time" columns to `LayerMonitorTree`, alongside the existing "Eligible" column. Docs - Add `startTime` and `stopTime` to the Layer example payload in the REST API reference. - Document the new monitor-tree columns in the Cuetopia monitoring guide. VERSION.in - Bump version to 1.23.

CueJobMonitorTree was fetching the same job data twice every 22s: once via cached getJobWhiteboard, then once per group via an uncached getJobs(id=...). On a show with N populated groups, one tick cost 1 + N gRPC round-trips and SQL executions. Changes: - Add `repeated Job inline_jobs = 18` to NestedGroup (strictly additive; `repeated string jobs` retained). Cuebot populates it from the existing GET_NESTED_GROUPS row data — the only column added to the SELECT is `str_loki_url`. Reuses WhiteboardDaoJdbc.JOB_MAPPER. - Bump whiteboard cache TTL 5s→10s and add per-show single-flight (synchronized-on-show), so concurrent clients share one SQL execution per TTL window. - cuegui: drop UPDATE_INTERVAL 22s→5s; override _update with a skip-if-running guard; rewrite _processUpdate as an incremental diff (takeChild only stale IDs, no clear()), so selection, scroll, and expansion survive add/remove/reparent. - cuegui: consume inline_jobs; fall back to opencue.api.getJobs against older Cuebot. ## LLM usage disclosure Claude Opus was used to implement this optimization  ## Summary by CodeRabbit * **New Features** * Display inline job data within nested job groups. * **Performance** * Faster job tree refresh (22s → 5s). * Incremental tree updates to avoid full rebuilds. * Improved whiteboard caching with per-show refresh control and longer timeout. * **Bug Fixes** * Ensure work/task completion callbacks fire even on failure. * **Tests** * Added a test verifying inline jobs populate in nested groups. * **Chores** * Project version updated to 1.24.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2370?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)   --------- Signed-off-by: Diego Tavares <dtavares@imageworks.com>

…2371) The following warning has been poluting cuegui's log for a while. This QT warning can be triggered by different locations, this PR treats one of them. ``` WARNING Main Qt thread-affinity violation: File "cuegui/ThreadPool.py", line 218, in run result = work[0]() File "cuegui/HostMonitorTree.py", line 291, in _getUpdate parent.updateOSFilterList(os_values) File "cuegui/HostMonitor.py", line 483, in updateOSFilterList action = QtWidgets.QAction(menu) File "cuegui/Main.py", line 140, in warning_handler logger.warning("Qt thread-affinity violation:\n%s", "".join(traceback.format_stack())) ``` The previous solution called a parent function from a different thread directly. This fix uses a signal to allow triggering the same function on its own thread. ## LLM usage disclosure Claude Code was used to propose a fix once the issue was identified.  ## Summary by CodeRabbit * **Bug Fixes** * Resolved thread-safety issues in host monitoring by implementing proper synchronization for OS filter list updates. This prevents race conditions and potential crashes when the background worker discovers and updates operating system values during host monitoring operations.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2371?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

…oftwareFoundation#2378) For some unknown reason, this change was resulting on a Segmentation Fault when a combination of unrelated factors were involved. For now this PR simple reverts the previous version in an effort to avoid crashes.  ## Summary by CodeRabbit * **Bug Fixes** * Stopped emitting completion signals for tasks that fail, preventing failed background jobs from appearing as completed and avoiding premature cleanup of job state. * Improved background fetch error handling with clearer logging, distinct handling for transient RPC errors, and ensured failed fetches return safely so they are retried on the next update tick.  [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/AcademySoftwareFoundation/OpenCue/pull/2378?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)   --------- Signed-off-by: Diego Tavares <dtavares@imageworks.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>

ThreadMode.ALL hosts (NIMBY workstations by default) run only threadable layers. The planner was blind to the attribute: it parked non-threadable layers on ALL hosts (idle workstations score best), planHost's legacy re-check found zero frames, and the layer burned its one commit per tick forever while legacy booked it elsewhere. Carry int_thread_mode through the host snapshot into the group key (one bit: legacy normalizes every mode but ALL to AUTO) and bind the legacy threadability clause in the candidate query and the DEBUG explain. Parity tests cover refusal and booking on ALL hosts; the refusal test was written first and failed against the unfixed scheduler.

Entire-Checkpoint: f68a943135a3

Fix the batch-start locking javadoc, note the transaction requirement on the stat pre-locks, add a legend to the score formula, explain the advisory lock choice, and clean comment punctuation. Comments and docs only.

Expose the scheduler's behaviour as Prometheus metrics and read them from a Grafana dashboard. Metrics cover tick duration, frames dispatched per show, group pass reasons and totals, per-show farm cores, fragmentation by reason, and active vs inactive host-spec groups, under a human-readable label vocabulary. The dashboard draws per-show throughput and farm share as stacked bars, so each show's contribution and the farm-wide total read at a glance. On the simulator side, add the fragmentation scenario tooling, a multi-show feeder (sim1 to sim5) with per-show priority, and multi-tag hosts, and start each cuebot with the Prometheus collector on so the stack can be scraped.

A permissive layer (loose tags, or plain 'general') is a candidate in every host-spec group it fits. The within-group break already caps it to one host per group per tick, but nothing stopped it being planned again in the next group: each group re-planned it onto its own host, the parallel per-host plan reads then pulled the same waiting frames, and every copy but one lost the frame.int_version race at commit. The waste was real: candidate scans, reads and VirtualProc construction, plus idle cores stolen from siblings that booked nothing. Add a tick-wide placedLayerIds set (cleared with plannedByHost). submitCommit records the placed layer, and the candidate loop skips a layer already placed in an earlier group this tick. The skip sits after seenLayerIds.add, so reservation sweeping still sees the layer, and before any host/cap mutation, so a duplicate consumes no simulated resources. It keys on placement, not candidacy, so a layer that could not fit an earlier group is still tried in later ones. Measured on the simulator under a 120-tag, 30% run-anywhere farm (about 120 host-spec groups, 1553 hosts): raceLost fell from about 97% of planned to 0 (planned now equals committed). Utilisation is unchanged: at this tag count the farm is fragmentation-limited, not planning-limited. sim: guard the fix with a TAGMAX scenario in the --verify battery. tagmax_watch reads the Scheduler's per-window stat line and fails if raceLost exceeds a small fraction of planned (default 0.10) across the fragmented farm, with planned and host-spec-group floors so it cannot pass on an idle run. Add the --tagmax-test flag, its wiring and a README row, plus SIM_GENERAL_FRAC in farm_spec: the fraction of layers that carry no capability tag (run-anywhere 'general' work, a candidate in every group). 0 by default; the scenario uses 120 tags and 0.3.

Break the 527-line doTick into small, single-purpose methods so the tick reads as its phases: the completion drain, the leadership gate, then 1. snapshot, 2. group, 3. plan, 4. commit. The extracted methods (planGroup, planBookings, recordCommitted, stampWarmthAndLaunch, snapshotFarmFill, grantReservations, trimOverFolderCeiling, trimOverLicensePools, clearTickScratch, drainResolvedCompletions, expireDisplacedWarmth, and friends) each carry a plain prose header. runTick stays a thin Quartz harness that times the pass and rolls the leader counters into the window summary; doTick returns the procs dispatched, or -1 for a standby that did not plan.

Documentation, comments, and one rename. No logic changes. Scheduler.java: * Fixed 7 factually wrong or stale comments: the all-hosts query filter, the launch-pool backpressure policy (a drop policy, not caller-runs), the licenses source, the farm-snapshot timing, a nonexistent license property, waitingFrameCount, and dispatchSupport. * Field comments now say what each field is used for and point to the main caller, rather than restating the declared type. * Trimmed duplicated design essays to a gist plus a Scheduler.md pointer (priority lottery, batched accounting, cross-group dedup, E-PVM placement, group fragmentation). * Removed an orphaned tick-algorithm block, lowercased emphasis capitals, dropped leftover HTML and stray dashes, and re-scoped a few comments. * Renamed stampWarmthAndLaunch to launchCommitted, named for its primary job (launch); the cache-warmth stamping it also does is now in the method header. Scheduler.md (full consistency sweep against Scheduler.java): * snapshot: readBookableHosts / SELECT_BOOKABLE_HOSTS renamed to readAllHosts / SELECT_ALL_HOSTS; the query is UP + OPEN (busy or idle), and the minimum idle core cut happens later, per group, in planGroup. * tick loop: document stage 0 (completion drain plus cache warmth expiry on every Cuebot) and the leadership gate that precedes the placement pipeline. * host spec key: full six part tuple (alloc, facility, tags, os, gpu, thread mode), not four. * stat line: add drained, reservedCores, backfilledCores and the optional lic segment to both the sample and the prose. * drop the stale "no schema changes" claims; fix field name glosses (layerCoresMin / layerMemMin) and soften the new file count.

A blocked wide layer could reserve a host and drain it, but two gaps let it starve anyway: - The reservation was not firm. A running frame or a higher-priority reserver could seize the host mid-drain, so the wide job never assembled its block and stranded forever. - Grants were ordered strictly by priority, so a low-priority wide job was starved of the scarce reservation budget by any steady higher-priority stream. Fix both: - Make reservations firm. reservationAllows is owner-only and pickReservationTarget only ever claims a free host, so once a layer holds a host nothing takes it away, not a running frame and not another reserver. Higher-priority work still borrows the draining host's spare cores through EASY backfill (never owning), so the owner is never delayed; the host drains and the wide job runs. - Grant by a priority-weighted lottery, the same one the dispatcher uses (key = random()^(1/priority), Efraimidis-Spirakis), so a low-priority wide job keeps a proportional share of the budget instead of being starved. The RESERVATIONS --verify scenario asserts the stranded wide jobs reserve, drain, and actually run, with a farm-wide throughput floor so a dead farm cannot pass.

Stats only, no scheduling change. Each tick puts every waiting frame from the candidate layers into one of six buckets: flowing, capacity, no fit, limit, no license, held. The tally goes to the Prometheus gauge cue_scheduler_waiting_frames. A live booking ledger feeds cue_scheduler_running_frames, the denominator, so the board shows each cause as a percent of all frames. No SQL is used for stats. The fragmentation metric is removed. The board gains waitlist and utilisation panels. The verify battery asserts each bucket fires.

A legacy trigger rejects any plus that lands over a cap. When a user lowers a job's max cores, or an admin shrinks a subscription burst, below live usage, the batched flush aborts. The pluses wedge in the retry buffer, completions keep subtracting, and the mirror goes negative (seen in production as a job at -14 cores). The flush now writes each guarded table as a cap-neutral pair of updates that the trigger skips; the cap is net unchanged. The planner still enforces caps at plan time. The CAPDROP scenario drops both caps under load and fails on divergence, a negative mirror, or a rejected flush.

PRODENV now runs first in the verify battery: 300 seconds on the full farm while a seeded chaos driver mutates the live environment the way admins do all day. Limit caps churn, capped folders appear and eat running jobs, host tags come and go, job max cores and subscription bursts random-walk, hosts lock and unlock. The watcher asserts decay-style cap invariants (over-cap usage may only drain, never grow), mirror-vs-proc truth everywhere, zero rejected flushes, zero negative counters, and a throughput floor, then reconciles every ledger in a quiet tail. The five feeder shows are renamed to showA..showE in the simulator seed and helpers. SIM_VERIFY_ONLY=NAME[,NAME] now runs a subset of the battery.

Production showed 128 one-core frames of one layer filling a single 128-core host while the rest of the farm sat free: the locality bonus (8.0) dwarfs the E-PVM spread (hundredths), so same-layer frames pile onto one machine until it is full. That concentrates blast radius and phase-locked IO on one chassis. New knob scheduler.layer_host_max_frac (default 0.25; 0 disables): one layer may hold at most this fraction of a host's cores, never below 8 frames so small hosts still anchor a cache-warm batch. Hosts at their share are skipped at selection and the commit is clamped, so the flood spills to the next host. The LAYERCAP scenario reproduces the pile-up with the cap off (128 frames on one host, 2 hosts total) and asserts the cap holds at 0.25 (same flood, 15 hosts, zero violations); LOCALITY still passes with the cap on (refill affinity 27.7% against the 15% floor).

Each RQD host report now feeds an in-memory health ledger: swap use and kernel system time (a new sysTime report attribute; real RQD still needs a small patch to send it). After each tick the scheduler publishes the cue_farm_health_* gauges per host spec group and per hardware shape, with no database read. The dashboard gets a farm health row, the sim plants sick hosts, and the new HEALTH verify scenario asserts that the sickness shows up on the metrics endpoint.

…ions Artists ask for 1 core on memory heavy layers. A few frames fill a host's memory and the rest of its cores sit idle. The scheduler now learns each layer's real memory from the host reports. It keeps the rss peaks of the last 32 frames and takes the median. It then resizes each layer before placement. Cores become the median rss divided by the group's memory per core. Memory becomes the median rss when that is above the declared value. Non threadable layers are never touched. Explicit asks can only grow, never shrink. A threadable layer that asks 1 core with no history runs at most 8 probe frames farm wide. After that the reports show what it needs. One setting: scheduler.mem_per_core. The default 0 reads the ratio from each host group's own machines. Any other value pins one ratio for the whole studio. See Scheduler.md 3.9.

lostProc could stop a frame it no longer owned. The frame may already be freed and booked on a new host. The stale stop freed it again while the new host kept rendering. Our sweep and evict then deleted the new proc row without telling RQD. Result: two hosts rendered the same frame, silently. Two changes. unbookProc now returns true only when it deleted the proc row, and lostProc leaves the frame alone when it gets false. The sweep and evict send a best effort kill for every corpse they delete, except when the corpse sits on the host being booked. They also credit the accounting tables through the same block the drain uses, so job_resource no longer leaks cores. Legacy dispatch is untouched. The maintenance orphaned frame reset must stay: it covers the crash window. The new DOUBLERENDER scenario injects 5 stale stops. Old code: 5 double renders. This code: the 5 leftover renders are killed and no frame runs twice.

A layer that could use the whole farm was stuck at 25% of it. The cap also hid a slower brake: one commit of 8 frames per layer per tick. A lone layer could not even outrun its own completions. The cap now prefers spreading but yields when it is the only blocker. A fitting idle host that only the cap refuses is given to the layer, if the layer's memory is proven. On a busy farm no such host exists, so under contention the cap still holds. Grants keep booking more hosts in the same tick. Each plan pulls its own slice of the waiting list, so parallel plans never fight over the same frames. The 8 frame trickle is gone from the scheduler path. Every commit books up to one 20 frame slice. Fairness comes from the lottery and the caps. All the sizing rules now live in one function, headroomFrames. LAYERCAP_SOLO proves the yield: alone, the layer fills 100% of the farm, 8% before. LAYERCAP proves the hold: contended, zero cap violations with hosts pinned at their cap. PRIORITY, PRIORITY_STARVING, LIMIT, FOLDER and STRANDGROW all stay green.

Production had no view of cold starts, cache reuse or stranded capacity; only the sim could measure them. Record what placement already knows: booked frames by kind (live_warm, cache_warm, cold) and, after each plan, the idle cores no waiting frame can buy. No SQL, per the house rule. Verified live: the sim watcher and the dial agree at 25.1% warm; stranded cores track the no-fit tail.

The tick takes its in flight latch, then starts the launch pools one line above the try whose finally releases it. Pool startup reads eight numeric properties and builds the license source. One malformed value throws there, and the latch is never cleared. Every later tick then stops at the compare and set. The stat line stops with it, so the process books nothing and says nothing for the rest of its life. One bad character in a property file silently ends booking farm wide. Start the pools inside the try. A failure is caught, logged and counted like any other tick failure, and the next tick runs.

A subscription belongs to one show and one allocation, and each one carries its own burst. The planner held used cores in a map keyed on the show alone. A show with two allocations then shared one number between two different limits. The allocation that planned first seeded the counter, and the second read a value that did not belong to it. The cap overshoots or over restricts, and group order decides which. The database trigger that would refuse the overshoot is bypassed on purpose, so nothing caught this. Key the counter on show and allocation. The subscription mirror already built that key, so both sites now call one helper and cannot drift apart again.

The batch wrote its procs in one transaction and their resource counters in later ones. A crash between the two leaves procs whose cores never reached job_resource, folder_resource, point and layer_resource. The release path subtracts those cores anyway, so the mirrors fall below reality and stay there. Only the subscription counter has a repair job. Every cap that reads those tables then sits too loose. The legacy path never had this gap, because it wrote the proc and its counters together. Open one transaction around the batch commit and the counter updates. The dispatch support service joins the caller transaction, so procs and mirrors commit or roll back together. The retry buffers go with it. A failure now rolls the bookings back, so the deltas they held describe procs that never existed. That also drops state the planner carried between ticks.

A busy farm wastes capacity. Hosts keep idle cores because the free remainder is slightly smaller than what the waiting frames request, so nothing can use that space and the farm levels off while frames wait. Let placement buy those cores by booking a frame at reduced cores, and price the choice rather than special case it. Each host offers one candidate to the layer: its full fit score when the whole request fits, otherwise the score of the largest reduced count that does fit, divided by the fraction of the work that count delivers. The memory reservation does not shrink with the cores, so a full fit always costs less per unit of work wherever one exists. The farm slides into reduced bookings as it fills and out of them as it drains, with no mode and no threshold. Cores keep at least 80% of the request, a constant in the code. Memory comes from the layer's observed use, or the declared request pro rated to the reduced count when that is higher. A layer that holds a scene in memory needs its whole footprint at fewer cores, and a smaller reservation would overrun a busy host and start kill cycles, so a layer whose real use was never observed cannot be reduced at all. GPU never shrinks. A reduced booking is still one commit for that layer in that tick, the rule every full booking obeys, so no layer can take host after host by shrinking. Every other gate binds unchanged. The dispatcher receives the exact reduced count and applies it as given, the only path where a booking goes below the request.

OVERDECLARE proves three contracts of the legacy memory balancer. A declaration far above real use heals down to the observed value, the no-optimize guard never shrinks a layer, and a truthful declaration is never changed. SQUEEZE_FLAT covers the layer whose memory does not shrink with its thread count. Hosts short on memory must stay empty, hosts short on cores must still take a reduced booking, and no kill cycle may start. It is the proof that the memory reservation never follows the cores down. UNDERDECLARE moves to a small farm. The old background saturation booked its frames as one-core balloons, because non-threadable layers clamp there, and the flood could then only enter through reservations below real use. The change also repairs three things. The fake RQD now reports pinned memory in completion reports as well as heartbeats. The TAGMAX watcher accepts the stat line's new counter for reduced bookings. The SQUEEZE workload now uses less memory than it declares.

DiegoTavares and others added 30 commits May 11, 2026 19:08

akheffache and others added 3 commits August 11, 2026 22:46

Fix query issues on DispatchQuery

103872b

Entire-Checkpoint: f68a943135a3

cuebot: comment and doc cleanup from review feedback

097a7be

Fix the batch-start locking javadoc, note the transaction requirement on the stat pre-locks, add a legend to the score formula, explain the advisory lock choice, and clean comment punctuation. Comments and docs only.

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from c7f1c00 to aec256d Compare August 13, 2026 02:41

akheffache added 3 commits August 14, 2026 00:05

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from 0bb32a2 to efb0513 Compare August 14, 2026 18:54

akheffache added 2 commits August 15, 2026 02:26

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch 3 times, most recently from c23c771 to abfaafd Compare August 19, 2026 04:22

akheffache added 2 commits August 19, 2026 05:31

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from abfaafd to a6647aa Compare August 19, 2026 05:36

akheffache and others added 7 commits August 19, 2026 15:31

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from fa03c0f to a3a56f1 Compare August 29, 2026 20:28

akheffache added 5 commits August 29, 2026 20:36

aghiles force-pushed the claude/gifted-clarke-0vluzy-v2 branch from a3a56f1 to b3f5b51 Compare August 29, 2026 20:51

Navigation Menu

[cuebot] Fix issue on getWhatDependsOn(Frame) (#2278) - #1

[cuebot] Fix issue on getWhatDependsOn(Frame) (#2278)#1
aghiles wants to merge 135 commits into
masterfrom
claude/gifted-clarke-0vluzy-v2

aghiles commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

Conversation

aghiles commented Jul 23, 2026

Related Issues

Summarize your change.

Summary by CodeRabbit

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants