fix(evaluator): prefer sync SDK for FilesetRef dataset download

[arpitsardhana]

Summary

What failed?

• run_sync creates a new events loop. When async client is passed to it. It attaches client to this loop when function ends, the client is rendered unusable. Any downstream function using client runs into "Event loop closed" issue.
• In this particular case Completed FilesetRef row evals wrote artifacts but GET /eval-results/{job_id} returned 404 — no EvaluateResultEntity was persisted. The same shared-client/run_sync pattern can also break Intake publication after result-entity persistence on agent-eval jobs.

Why? Dataset download (or entity save) and a later run_sync each used asyncio.run() on the same AsyncNeMoPlatform httpx client. The first loop closed after use; the next raise Event loop is closed was swallowed as best-effort (persist) or surfaced as a publication failure.

What is the fix?

1. Prefer the injected sync NeMoPlatform + download_dataset_sync() when available.
2. For async run_sync call sites, run work through run_with_isolated_async_sdk() so each call uses async_sdk.copy(http_client=DefaultAsyncHttpxClient()).

What to review

• Util: plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py (run_with_isolated_async_sdk)
• Call sites:
  • plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py (FilesetRef async download)
  • plugins/nemo-evaluator/src/nemo_evaluator/jobs/result_persistence.py (entity save)
  • plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py (Intake publish)

Related Issue

https://nvbugspro.nvidia.com/bug/6563418

Changes

• Prefer sync SDK for FilesetRef download when both SDKs are injected
• Add jobs/utils.py helper that clones async SDK httpx for each run_sync
• Use the helper for FilesetRef download, result persistence, and Intake publication
• Unit tests for the helper and call-site clone usage

Type of Change

• Code change (feature, bug fix, or refactor)

Quality Gates

• Tests added or updated for changed behavior
• Documentation not applicable — justification: internal job client isolation; public API unchanged

Verification

• Pull request title follows the repository's Conventional Commit format
• Every commit includes an appropriate Signed-off-by: trailer
• uv run pre-commit run --files on changed files passes
• Targeted tests pass
• No secrets, API keys, or credentials are included

Targeted validation:

uv run --frozen pytest \
  plugins/nemo-evaluator/tests/jobs/test_utils.py \
  plugins/nemo-evaluator/tests/test_result_persistence.py \
  plugins/nemo-evaluator/tests/jobs/test_publication.py::test_publishes_and_reports_what_landed \
  plugins/nemo-evaluator/tests/test_evaluate_job.py::TestEvaluateJobRun::test_downloads_fileset_ref_dataset_and_passes_path_to_sdk_evaluator \
  plugins/nemo-evaluator/tests/test_evaluate_job.py::TestEvaluateJobRun::test_prefers_sync_sdk_for_fileset_ref_when_both_sdks_injected -v
# passed

Full plugin suite for the touched files: 111 passed.

Live end-to-end repro

Ran the NVBug 6563418 reproduction steps against a local platform (nemo services run with
entities,files,jobs,evaluator,secrets,models,inference-gateway,auth plus the jobs/entities
controllers): created a real fileset, uploaded a JSONL dataset, submitted a row evaluation through
the public Evaluator SDK using FilesetRef(root="default/<fileset>"), and waited for completion.

Created fileset: default/filesetref-eval-repro
Uploaded dataset.jsonl (96 bytes)
FilesetRef: root='default/filesetref-eval-repro'
Submitted job: nemo-evaluator-lblwzne1
Job finished: nemo-evaluator-lblwzne1 status=completed
Downloaded artifacts: ['aggregate-scores.json', 'row-scores.jsonl']
get_result: rows=2 aggregates=1
PASS: eval_results.retrieve OK job_id=nemo-evaluator-lblwzne1 metric_types=['exact-match'] dataset_ref=default/filesetref-eval-repro
GET /apis/evaluator/v2/workspaces/default/eval-results/nemo-evaluator-lblwzne1 -> 200

evaluation-results was also downloadable from the persisted entity's bundle_ref. The reported
symptom for this bug was a 404 from that final retrieve; it now returns 200 with the aggregate
scores, dataset_ref, and bundle_ref populated.

Summary by CodeRabbit

Summary by CodeRabbit

• Bug Fixes

  • Improved dataset download reliability by prioritizing synchronous downloads when available.
  • Added a safer asynchronous fallback for more consistent evaluation runs.
  • Improved reliability when publishing evaluations and saving results asynchronously.
  • Reduced the risk of event-loop conflicts during background processing.

• Tests

  • Added coverage for synchronous and asynchronous download workflows.
  • Verified reliable publication and result persistence across supported processing paths.
  • Added regression checks for consistent asynchronous processing behavior.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

Evaluator SDK execution

Layer / File(s)	Summary
Isolated SDK utility and coverage plugins/nemo-evaluator/src/nemo_evaluator/jobs/utils.py, plugins/nemo-evaluator/tests/jobs/test_utils.py	Added run_with_isolated_async_sdk. It copies the asynchronous SDK with a managed HTTP client, runs the callback, and returns its result.
Publication and result persistence integration plugins/nemo-evaluator/src/nemo_evaluator/jobs/publication.py, plugins/nemo-evaluator/src/nemo_evaluator/jobs/result_persistence.py, plugins/nemo-evaluator/tests/jobs/test_publication.py, plugins/nemo-evaluator/tests/test_result_persistence.py	Publication and result persistence now use the isolated SDK helper. Tests verify one SDK copy per operation.
Synchronous download selection and fallback coverage plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py, plugins/nemo-evaluator/tests/test_evaluate_job.py	FilesetRef resolution prefers synchronous downloads when both SDK clients exist. The asynchronous fallback uses an isolated SDK copy. Result persistence receives the original asynchronous SDK.

Suggested reviewers: gabwow, sandychapman, ngoncharenko

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 18.18% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the primary fix: preferring the synchronous SDK for FilesetRef dataset downloads.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch nvbug-6563418-filesetref-result-persist/arpsingh

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Suite	Lines Covered	Line Rate	Branch Rate
Unit Tests	32001/40616	78.8%	63.6%
Integration Tests	18560/38542	48.2%	20.8%

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py`:
- Around line 91-93: Update the fallback HTTP-client setup in
AsyncEvaluator.run_local to reuse the custom client configuration from async_sdk
instead of constructing a fresh DefaultAsyncHttpxClient, preserving OAuth hooks,
proxy settings, and transport configuration. Add a regression test that passes a
customized platform client and verifies download_dataset receives those
settings.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a8d2c814-dece-427f-9d96-b07c54594469

📥 Commits

Reviewing files that changed from the base of the PR and between e4df000 and ae8bff2.

📒 Files selected for processing (2)

• plugins/nemo-evaluator/src/nemo_evaluator/jobs/evaluate.py
• plugins/nemo-evaluator/tests/test_evaluate_job.py

🚧 Files skipped from review as they are similar to previous changes (1)

• plugins/nemo-evaluator/tests/test_evaluate_job.py