test(evaluator): run plugin skill evals in NeMo Platform - #1063
test(evaluator): run plugin skill evals in NeMo Platform#1063ngoncharenko wants to merge 4 commits into
Conversation
221fae7 to
86984f1
Compare
08a103b to
4f1b9b9
Compare
…main Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
4f1b9b9 to
2d4c21b
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (8)
🚧 Files skipped from review as they are similar to previous changes (5)
📝 WalkthroughWalkthroughAdds Harbor configuration for the NeMo evaluator plugin. The environment starts a local NeMo Platform with subprocess execution. The evaluation submits a bundled offline spec, waits for completion, retrieves aggregate scores, and reports the exact-match mean. ChangesNeMo evaluator Harbor integration
Sequence Diagram(s)sequenceDiagram
participant HarborTask
participant EvaluateCLI
participant NeMoPlatform
participant SubprocessExecutor
HarborTask->>EvaluateCLI: Submit bundled spec file
EvaluateCLI->>NeMoPlatform: Create evaluation job
NeMoPlatform->>SubprocessExecutor: Execute evaluation
HarborTask->>NeMoPlatform: Poll job status
NeMoPlatform-->>HarborTask: Return terminal status
HarborTask->>NeMoPlatform: Retrieve aggregate scores
NeMoPlatform-->>HarborTask: Return exact-match mean
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.md (1)
1-1: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftMake this a complete HOW-TO page.
Add an H1, prerequisites, primary CLI and Python SDK examples in a tab set, and a
Next Stepssection. This also resolves MD041.As per coding guidelines, Markdown must use one Diataxis type, list prerequisites first, and end with Next Steps.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.md` at line 1, Update the instruction page to be a complete how-to: add a leading H1, place prerequisite requirements before the procedure, and include primary CLI and Python SDK examples in a tab set. Finish the page with a “Next Steps” section, while preserving the existing evaluation workflow and prohibition on the standalone nemo evaluator evaluate run path.Sources: Coding guidelines, Linters/SAST tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap`:
- Around line 22-30: Update the curl invocations in profile_is_ready and
platform_is_ready to include short --connect-timeout and --max-time bounds on
both readiness requests, ensuring stalled endpoints return before
startup_deadline handling is reached.
In
`@skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml`:
- Line 20: Update the task configuration’s network_mode setting from "public" to
"no-network" so the offline bootstrap and evaluation use only loopback services
and bundled data without external network access.
---
Nitpick comments:
In
`@skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.md`:
- Line 1: Update the instruction page to be a complete how-to: add a leading H1,
place prerequisite requirements before the procedure, and include primary CLI
and Python SDK examples in a tab set. Finish the page with a “Next Steps”
section, while preserving the existing evaluation workflow and prohibition on
the standalone nemo evaluator evaluate run path.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: b689dfe7-c7b0-46d7-907e-836db816dcfe
📒 Files selected for processing (8)
skills/nemo-evaluator-plugin/evals/config.ymlskills/nemo-evaluator-plugin/evals/environment/Dockerfileskills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yamlskills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrapskills/nemo-evaluator-plugin/evals/evals.jsonskills/nemo-evaluator-plugin/evals/harbor/dataset.tomlskills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.mdskills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
|
|
/nvskills-ci |
|
CodeRabbit nitpick disposition: |
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
|
/nvskills-ci |
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@skills/nemo-evaluator-plugin/evals/environment/Dockerfile`:
- Line 6: Update the Dockerfile runtime user from root to nvs so evaluation
commands launched by nmp-eval-bootstrap and SubprocessJobBackend run non-root,
and grant nvs ownership or required access to /workspace/.nemo. Preserve root
only for setup steps, switching to nvs before runtime execution.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 017a3838-f980-46bf-996a-abe89b255a15
📒 Files selected for processing (8)
skills/nemo-evaluator-plugin/evals/config.ymlskills/nemo-evaluator-plugin/evals/environment/Dockerfileskills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yamlskills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrapskills/nemo-evaluator-plugin/evals/evals.jsonskills/nemo-evaluator-plugin/evals/harbor/dataset.tomlskills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/instruction.mdskills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
🚧 Files skipped from review as they are similar to previous changes (5)
- skills/nemo-evaluator-plugin/evals/config.yml
- skills/nemo-evaluator-plugin/evals/harbor/dataset.toml
- skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap
- skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
- skills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yaml
|
|
||
| FROM nvcr.io/nvidia/nemo-platform/nmp-api:0.3.0@sha256:852758b994aaca5a9646a163fffaf46a20938b7e0aa10928c275e0d4a32d0723 | ||
|
|
||
| USER 0 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
dockerfile='skills/nemo-evaluator-plugin/evals/environment/Dockerfile'
bootstrap='skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap'
last_user="$(awk '$1 == "USER" { user=$2 } END { print user }' "$dockerfile")"
test "$last_user" != "0"
rg -n 'sudo|su |runuser|setpriv|chown|chmod|mkdir|touch|tee|/etc/nmp|/workspace|/app|/var/lib|/tmp' "$bootstrap" || trueRepository: NVIDIA-NeMo/nemo-platform
Length of output: 163
🏁 Script executed:
#!/bin/bash
set -euo pipefail
dockerfile='skills/nemo-evaluator-plugin/evals/environment/Dockerfile'
bootstrap='skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap'
printf '%s\n' '--- Dockerfile ---'
cat -n "$dockerfile"
printf '%s\n' '--- bootstrap metadata and contents ---'
stat -c '%n %s bytes' "$bootstrap"
cat -n "$bootstrap"
printf '%s\n' '--- related runtime-user references ---'
rg -n -i 'nmp-eval-bootstrap|dockerfile|USER[[:space:]]|user:|runAsUser|privileged|root' \
skills/nemo-evaluator-plugin/evals/environment skills/nemo-evaluator-plugin \
-g '!Dockerfile' -g '!*.lock' || trueRepository: NVIDIA-NeMo/nemo-platform
Length of output: 1290
🏁 Script executed:
#!/bin/bash
set -euo pipefail
dockerfile='skills/nemo-evaluator-plugin/evals/environment/Dockerfile'
bootstrap='skills/nemo-evaluator-plugin/evals/environment/scripts/nmp-eval-bootstrap'
printf '%s\n' '--- bootstrap ---'
wc -l "$bootstrap"
cat -n "$bootstrap"
printf '%s\n' '--- runtime-user references ---'
rg -n -i 'nmp-eval-bootstrap|USER[[:space:]]|user:|runAsUser|privileged|root|sudo|su |runuser|setpriv|chown|chmod|mkdir|touch|tee|/etc/nmp|/workspace|/app|/var/lib|/tmp' \
skills/nemo-evaluator-plugin/evals/environment skills/nemo-evaluator-plugin \
-g '!Dockerfile' -g '!*.lock' || true
printf '%s\n' '--- Dockerfile USER instructions ---'
awk '$1 == "USER" { print NR ":" $0 }' "$dockerfile"Repository: NVIDIA-NeMo/nemo-platform
Length of output: 7656
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- relevant container files ---'
git ls-files | rg '(^|/)(Dockerfile|.*docker.*|.*container.*|.*task\.toml)$|nmp-api|nmp-eval'
printf '%s\n' '--- image and USER references ---'
rg -n -i 'nmp-api|USER[[:space:]]|runAsUser|user[[:space:]]*=|/workspace|/app|/etc/nmp|NMP_DATA_DIR|NMP_CONFIG_FILE_PATH' \
--glob 'Dockerfile*' \
--glob '*.yaml' \
--glob '*.yml' \
--glob '*.toml' \
--glob '*.md' \
.
printf '%s\n' '--- task configuration ---'
cat -n skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
printf '%s\n' '--- evaluator environment configuration ---'
cat -n skills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yamlRepository: NVIDIA-NeMo/nemo-platform
Length of output: 50382
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- target task files ---'
git ls-files skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001 \
skills/nemo-evaluator-plugin/evals/environment
printf '%s\n' '--- target task configuration ---'
cat -n skills/nemo-evaluator-plugin/evals/harbor/nemo-evaluator-plugin-001/task.toml
printf '%s\n' '--- target environment configuration ---'
cat -n skills/nemo-evaluator-plugin/evals/environment/config/nmp-eval-config.yaml
printf '%s\n' '--- analogous evaluator Dockerfile ---'
cat -n tests/agentic-use/evaluator-zero-config-judge-cli/environment/Dockerfile
printf '%s\n' '--- relevant non-root runtime documentation ---'
sed -n '380,410p' tests/agentic-use/README.mdRepository: NVIDIA-NeMo/nemo-platform
Length of output: 6321
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- base-image user setup ---'
for file in Dockerfile.agentic-base Dockerfile*; do
if [ -f "$file" ]; then
printf '\n[%s]\n' "$file"
rg -n -C 4 -i 'FROM|useradd|adduser|groupadd|harbor|USER[[:space:]]|mkdir|chown|chmod|/workspace|/app|/logs|/installed-agent' "$file"
fi
done
printf '%s\n' '--- service privilege and path assumptions ---'
rg -n -C 3 -i 'geteuid|getuid|root|permission|/workspace|NMP_DATA_DIR|working_directory|subprocess' \
services packages \
-g '*.py' -g '*.toml' -g '*.yaml' -g '*.yml' \
| head -n 500
printf '%s\n' '--- task execution user fields ---'
rg -n -i 'run_as|runAs|user[[:space:]]*=|user:|uid|privileged|environment\.env|healthcheck' \
skills/nemo-evaluator-plugin tests \
-g '*.toml' -g '*.yaml' -g '*.yml' -g '*.md' \
| head -n 300Repository: NVIDIA-NeMo/nemo-platform
Length of output: 50381
🌐 Web query:
nvcr.io/nvidia/nemo-platform/nmp-api:0.3.0 Dockerfile USER root non-root runtime user /workspace
💡 Result:
The NVIDIA NeMo Platform API container image (nmp-api:0.3.0) is designed to operate within the NeMo Platform ecosystem, which utilizes a /workspace directory for managing data, configurations, and tasks [1][2]. While specific Dockerfile directives for user permissions in version 0.3.0 are not explicitly detailed in public documentation, the platform architecture generally emphasizes secure deployment and configuration management [3]. In containerized deployments of the NeMo Platform, services often interact with a /workspace directory [1]. Users are encouraged to follow established security practices, such as those outlined in the production hardening documentation, which include managing workspace access through RBAC and restricting administrative roles [3]. For technical details regarding the container's user configuration, you can inspect the image layers or the specific Dockerfile if you have access to the source repository. The repository structure includes specific Dockerfiles for services, such as docker/Dockerfile.nmp-api [4]. If you are working in a local environment, you can verify the runtime user by running: docker run --rm nvcr.io/nvidia/nemo-platform/nmp-api:0.3.0 whoami For production environments, ensure that your deployment adheres to the security requirements described in the NVIDIA NeMo Platform documentation to maintain proper isolation and access control [3].
Citations:
- 1: https://docs.nvidia.com/nemo/microservices/latest/get-started/quickstart.html
- 2: https://docs.nvidia.com/nemo/microservices/26.3.1/get-started/quickstart.html
- 3: https://docs.nvidia.com/nemo-platform/v0.3.0/documentation/access-control/deployment/production-hardening
- 4: 2e8878f
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- API Dockerfile candidates ---'
git ls-files | rg '(^|/)(docker/)?Dockerfile(\.nmp-api)?$|Dockerfile.*nmp|nmp-api'
printf '%s\n' '--- current API Dockerfile content ---'
while IFS= read -r file; do
case "$file" in
*Dockerfile*|*dockerfile*)
printf '\n[%s]\n' "$file"
rg -n -C 5 -i 'FROM|useradd|adduser|USER[[:space:]]|harbor|workspace|chown|chmod|ENTRYPOINT|CMD' "$file" || true
;;
esac
done < <(git ls-files | rg '(^|/)(docker/)?Dockerfile(\.nmp-api)?$|Dockerfile.*nmp|nmp-api')
printf '%s\n' '--- referenced historical API Dockerfile ---'
if git cat-file -e 2e8878f3dff3ce78009aee95ef6d591982726710:docker/Dockerfile.nmp-api 2>/dev/null; then
git show 2e8878f3dff3ce78009aee95ef6d591982726710:docker/Dockerfile.nmp-api | \
nl -ba | rg -C 5 -i 'FROM|useradd|adduser|USER[[:space:]]|harbor|workspace|chown|chmod|ENTRYPOINT|CMD'
else
echo 'historical Dockerfile is not available locally'
fiRepository: NVIDIA-NeMo/nemo-platform
Length of output: 4848
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- docker/Dockerfile.nmp-api ---'
cat -n docker/Dockerfile.nmp-api
printf '%s\n' '--- NGC image metadata ---'
cat -n .github/assets/ngc/containers/nmp-api.mdRepository: NVIDIA-NeMo/nemo-platform
Length of output: 6106
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- nmp-python-base user and workspace setup ---'
rg -n -C 5 -i 'useradd|adduser|groupadd|USER[[:space:]]|WORKDIR|/workspace|/app|chown|chmod|HOME' \
docker/base/Dockerfile.nmp-python-base docker/base/Dockerfile.nmp-workspace
printf '%s\n' '--- subprocess launch implementation ---'
rg -n -C 5 -i 'create_subprocess|Popen|subprocess\.run|subprocess\.|working_directory|cwd=|setuid|setgid|user=' \
services/core/jobs packages/nmp_platform packages/nmp_common \
-g '*.py' -g '*.yaml' -g '*.yml' \
| head -n 800Repository: NVIDIA-NeMo/nemo-platform
Length of output: 40550
Run evaluation commands as a non-root user.
nmp-eval-bootstrap starts the subprocess executor as UID 0, and SubprocessJobBackend does not drop privileges before launching evaluation commands. Switch the runtime to nvs and grant it access to /workspace/.nemo. If the API must remain root, drop privileges for subprocess jobs instead.
🧰 Tools
🪛 Trivy (0.72.0)
[error] 6-6: Image user should not be 'root'
Last USER command in Dockerfile should not be 'root'
Rule: DS-0002
(IaC/Dockerfile)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@skills/nemo-evaluator-plugin/evals/environment/Dockerfile` at line 6, Update
the Dockerfile runtime user from root to nvs so evaluation commands launched by
nmp-eval-bootstrap and SubprocessJobBackend run non-root, and grant nvs
ownership or required access to /workspace/.nemo. Preserve root only for setup
steps, switching to nvs before runtime execution.
Source: Linters/SAST tools
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
2 participants
Summary
nmp-api:0.3.0imagesubprocess/defaultexecution profileStory
Results
Why
The evaluator plugin skill was evaluated in the default SkillEvaluator container, which did not have NeMo Platform running. That only exercised CLI guidance and could not verify the platform submission workflow.
The new harness starts NeMo Platform inside the evaluation container, allows up to 240 seconds for readiness, and submits the bundled two-row exact-match evaluation. It does not use Docker-in-Docker: platform jobs execute through the in-container subprocess runtime.
Validation
uv run --frozen pytest plugins/nemo-evaluator/tests/test_skill_examples.py -q— 22 passednemo-evaluator-hwhtqurncompleted withexact-match.exact-matchmean0.5Stack
This PR is stacked on #1055 so its diff contains only the runtime-evaluation work. Retarget it to
mainafter #1055 merges.Summary by CodeRabbit
New Features
Documentation