feat(nemo-agents): add specialist subagents + rename to Email Security Triage - #1222
Merged
Conversation
Contributor
📝 WalkthroughWalkthroughThe email-phishing example is replaced by an email-security-triage agent. The configuration adds specialist subagents, strict verdict output, IOC dataset tooling, MCP wiring, evaluation support, telemetry, documentation, and Studio registration. ChangesEmail security triage
Sequence Diagram(s)sequenceDiagram
participant Analyst as Email analyst
participant Triage as Triage orchestrator
participant MCP as IOC MCP server
participant Specialists as Specialist subagents
participant Verdict as phishing-analyzer
Analyst->>Triage: Submit email
Triage->>MCP: Extract URLs and domains
MCP-->>Triage: Return IOCs
Triage->>Specialists: Analyze brand, category, and eligible headers
Specialists-->>Triage: Return specialist findings
Triage->>Verdict: Submit email, IOCs, and findings
Verdict-->>Triage: Return YAML verdict
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 2 | ❌ 3❌ Failed checks (3 warnings)
✅ Passed checks (2 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/agent.yaml`:
- Around line 99-105: Update the output schema in the phishing-analyzer prompt
to require a lowercase impersonated_brand value, changing the placeholder under
the existing YAML keys while preserving the “or none” option.
In
`@plugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/README.md`:
- Around line 38-41: Update the README description of the YAML verdict to
identify phishing-analyzer as the verdict owner. Describe attack_type and
impersonated_brand as specialist-informed fields that phishing-analyzer may
override or fill, rather than fields emitted directly by attack-attributor or
url-brand-analyst.
- Around line 53-54: Update the authentication-results example in the
email-phishing-agent README so the SPF interpretation is accurate: either make
the reported SPF, DKIM, and DMARC results consistent with a spoofed From domain,
or explicitly describe spf=fail smtp.mailfrom=evil.example as an envelope-sender
failure without claiming it proves From: paypal.com was spoofed.
- Around line 66-70: Update the trace description near the deployment artifact
and NeMo Studio Intake examples to state that tasks are recorded only for
applicable specialists; clarify that header-auth-analyst appears for emails
containing Authentication-Results: or Received: headers and is absent for the
Step 2 email.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 12ed3c03-3180-4465-9b66-a5754b271d1b
📒 Files selected for processing (3)
plugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/CUSTOMIZE.mdplugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/README.mdplugins/nemo-agents/examples/nemo-agent-config/email-phishing-agent/agent.yaml
Contributor
|
Contributor
Author
|
Naming note (not blocking this PR): with these specialist sub-agents (brand impersonation, attack attribution, SMTP header auth) the example reads as broader than "phishing" now — I'd like to rename it to "Email Security Triage" ( Deferring it deliberately as a post-merge capstone rather than folding it in here: a full identifier rename touches the merged base example (#1117), this PR, and the Studio PR (#1223) all at once, so it's cleanest as one sweep on Locked manifest for whoever runs it later (slug
|
15 tasks
walston
changed the title
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md (1)
16-24: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftKeep
CUSTOMIZE.mdin one Diataxis quadrant.This page mixes a customization HOW-TO with an architecture diagram, a configuration reference table, and contract rules. Move the reference material to a linked reference page, or keep this page procedural.
As per coding guidelines, each documentation page should fit one Diataxis quadrant and should not mix how-tos with architecture explanations or reference tables.
Also applies to: 36-61
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md` around lines 16 - 24, Refocus CUSTOMIZE.md as a procedural customization how-to by removing the architecture explanations, configuration/reference table, and contract rules. Move that material into an appropriate linked reference page, then add links from CUSTOMIZE.md while preserving only the ordered steps needed to customize the example. Apply this consistently to the sections covered by the “Parts & what to change” table and the later referenced content.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md`:
- Around line 3-4: Update the goal statement in CUSTOMIZE.md to explicitly name
both delegated roles: the verdict sub-agent and the specialist sub-agents.
Preserve the existing description of the DeepAgents orchestrator and its tool
usage while replacing the generic “a sub-agent” wording.
---
Nitpick comments:
In
`@plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md`:
- Around line 16-24: Refocus CUSTOMIZE.md as a procedural customization how-to
by removing the architecture explanations, configuration/reference table, and
contract rules. Move that material into an appropriate linked reference page,
then add links from CUSTOMIZE.md while preserving only the ordered steps needed
to customize the example. Apply this consistently to the sections covered by the
“Parts & what to change” table and the later referenced content.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: f640b104-3ae1-44a8-bb4d-620c002a3ff5
⛔ Files ignored due to path filters (2)
plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/data/smaller_test.csvis excluded by!**/*.csvuv.lockis excluded by!**/*.lock
📒 Files selected for processing (11)
plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.mdplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/README.mdplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/agent.yamlplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/data/build_dataset.pyplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/email-security-triage-eval.ymlplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/mcps/iocs.pyplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/pyproject.tomlplugins/nemo-agents/examples/nemo-agent-config/email-security-triage/tests/test_extract_iocs.pypyproject.tomlweb/packages/studio/public/sample-agents/email-security-triage/agent.ymlweb/packages/studio/src/constants/sampleAgents.ts
htolentino-nvidia
approved these changes
Aug 11, 2026
…mple
The orchestrator now fans out to specialist subagents before the verdict, so the
analysis is a traced, tunable chain instead of one opaque classification:
- url-brand-analyst - lookalike / brand-impersonation judgement on the domains
extract_iocs harvested (tool -> subagent composition)
- attack-attributor - names the attack category (bec/credential/malware/spam/benign)
- header-auth-analyst - SPF/DKIM/DMARC reading, gated on the email actually
carrying Authentication-Results:/Received: headers
phishing-analyzer stays the verdict owner and folds the findings in, adding
`attack_type` and `impersonated_brand` to the YAML verdict.
Prompts are ported from the email-security-analyst NAT example (they were removed
from it in #1135 as more protocol than that sample needed). There they were LLM
capability tools behind a router; here each is a declarative subagent, so its
reasoning gets its own trace span and its prompt is tunable in agent.yaml. The
prompt-injection guardrails are ported with them.
analyze_headers is demo-only: the scored dataset has no SMTP headers, and
synthesizing them per row would leak the phishing/benign label into the input and
inflate the eval. README shows a header-bearing email to exercise it by hand.
Verified live (--mode subprocess, real NVIDIA_API_KEY): 3/3 invocations return the
correct schema - phishing (plain + header-bearing) true, benign false - with
attack_type and impersonated_brand populated. The runtime checkpointer shows all
four subagent tasks plus the extract_iocs tool call. AgentConfig validate +
Fabric translate OK; ruff clean; 8/8 tool tests.
Co-authored-by: Octavian Drulea <odrulea@nvidia.com>
Signed-off-by: Nathan Walston <nwalston@nvidia.com>
- agent.yaml: require lowercase impersonated_brand in the phishing-analyzer verdict schema too (the orchestrator copy was already lowercase; the sub-agent copy wasn't - that mismatch is why live output flip-flopped PayPal vs paypal). - README Step 2: phishing-analyzer owns the verdict and emits every field; the attack_type/impersonated_brand values are specialist-informed (advisory), not produced by the specialists instead of the classifier. - README Step 3: fix a self-contradictory synthetic header example - it had spf=fail but dkim=pass/dmarc=pass, which actually authenticates From: paypal.com. Make all three fail and lead the explanation with dmarc=fail on header.from, the check tied to the visible From: domain. - README Step 4: traces carry a task per *applicable* specialist; header-auth-analyst appears only for header-bearing input (the Step 2 email has none). Co-authored-by: Octavian Drulea <odrulea@nvidia.com> Signed-off-by: Nathan Walston <nwalston@nvidia.com>
…ecurity-triage With the specialist sub-agents this example is broader than "phishing" — it's an email-security triage orchestrator — so rename it to match (display name "Email Security Triage", slug `email-security-triage`). Folded into this PR because the specialists are the reason for the rename. Full identifier sweep (collision-checked vs the still-present NAT `email-security-analyst` and `email-phishing-analyzer`): - example dir, agent `name`, telemetry project - package `nemo-agent-config-example-email-phishing` -> `...-email-security-triage` - console script / MCP url `email-phishing-iocs` -> `email-security-triage-iocs` (MCP server key stays `iocs`; module stays `mcps/iocs.py`) - eval config `email-phishing-eval.yml` -> `email-security-triage-eval.yml` - default deployment name in the README - Studio registry key/namePrefix/displayName/agentConfigPath + sample dir - README/CUSTOMIZE branding; re-synced the Studio sample copy to this branch's (specialist) agent.yaml Unchanged on purpose: the `is_likely_phishing` verdict schema, the `phishing-analyzer` sub-agent (it IS the phishing specialist), and domain references to phishing. Verified: AgentConfig validate + Fabric translate; ruff; 8/8 tool tests; Studio loader tests; zero stragglers; and live create -> deploy -> invoke under the new slug returns is_likely_phishing: true (renamed console script resolves). Signed-off-by: Nathan Walston <nwalston@nvidia.com>
steramae-nvidia
approved these changes
Aug 11, 2026
Signed-off-by: Nathan Walston <nwalston@nvidia.com>
walston
force-pushed
the
astd-408-specialist-subagents/nwalston
branch
from
August 11, 2026 19:54
463d214 to
875ec7a
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md (1)
11-25: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winKeep
CUSTOMIZE.mdas a HOW-TO.This section mixes an architecture explanation with a path reference table. Move that content to linked EXPLANATION or REFERENCE pages, or rewrite it as steps for the customization task.
As per coding guidelines, each documentation page must use one Diátaxis quadrant.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md` around lines 11 - 25, Rewrite the “Parts & what to change” section in CUSTOMIZE.md as task-oriented customization steps, removing the architecture overview and path-reference table from this HOW-TO page. Link to separate explanation or reference documentation for those details if it exists, while keeping the page focused on the sequence of changes needed to customize the example.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md`:
- Line 7: Expand the prerequisites section in CUSTOMIZE.md to list the concrete
setup, deployment, invocation, and required access requirements for the
email-security-triage example. Add tested usage examples for both the Python SDK
and CLI, presented in the repository’s standard tab-set format, while preserving
the existing README reference.
---
Nitpick comments:
In
`@plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md`:
- Around line 11-25: Rewrite the “Parts & what to change” section in
CUSTOMIZE.md as task-oriented customization steps, removing the architecture
overview and path-reference table from this HOW-TO page. Link to separate
explanation or reference documentation for those details if it exists, while
keeping the page focused on the sequence of changes needed to customize the
example.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: fbf7381e-5635-47f5-8bd2-08a5645b80eb
📒 Files selected for processing (1)
plugins/nemo-agents/examples/nemo-agent-config/email-security-triage/CUSTOMIZE.md
ryana
pushed a commit
to ryana/nemo-platform
that referenced
this pull request
Aug 12, 2026
…IA-NeMo#1223) * feat(studio): add Fabric email-phishing agent to Create-Example dropdown Adds the Fabric (nemo-agents-spec-v1) email-phishing agent as a canned example in Studio's Create Example Agent modal, alongside the NAT email-security-analyst. - New static asset public/sample-agents/email-phishing-agent/agent.yml: an independent copy of the plugin example's agent.yaml (source of truth) with a Studio-sample header. Body byte-identical, so the two stay diffable. - New SAMPLE_AGENTS entry with configFormat: nemo-agents-spec-v1, so the create API validates it as Fabric and loadSampleAgentConfig injects the selected model at models.default.model (the existing Fabric branch — no loader change needed). - Reworked the INVARIANT comment: it previously only covered NAT _type -> Python package deps; added the Fabric shape (mcp.servers.<n>.url console script must be on PATH — here email-phishing-iocs) and the hand-sync note for the copies. Verified: loadSampleAgentConfig tests 7/7; the asset parses with config_format, models.default.model, and the console-script MCP url the loader/deploy path need. NOTE: the asset is a hand-maintained copy. NVIDIA-NeMo#1222 (specialist sub-agents) changes the plugin agent.yaml; this sample will need a one-line re-sync once that lands. Pre-existing (NOT from this change): studio typecheck fails on src/api/evaluation/agent-evaluations.ts (missing AggregateScalarScore in the generated web SDK) — reproduces on pristine main; needs make update-web-sdk. Signed-off-by: Nathan Walston <nwalston@nvidia.com> * feat(studio): drop the NAT email-security-analyst from Create-Example NAT is being deprecated, so remove the NAT email-security-analyst sample from Studio's Create Example Agent dropdown, leaving the Fabric email-phishing agent (added in this PR) as the example. - Remove the email_security_analyst SAMPLE_AGENTS entry + its INVARIANT _type lines. - Delete its static agent.yml. - KEEP the eval-config assets (eval-config.*.json, dataset.jsonl, READMEs): the eval-config picker (EVAL_CONFIG_SAMPLES) is a separate registry and, per its own contract, configs are not owned by an agent, so the picker keeps working. Only agent.yml is removed from that directory. DEFAULT_SAMPLE_AGENT_KEY now resolves to email_phishing_agent (only entry). Scope: Studio dropdown only. The NAT plugin example (plugins/nemo-agents/examples/email-security-analyst/) is intentionally left in place; removing it is a broader NAT-deprecation call for a separate, cross-team ticket. loadSampleAgentConfig tests 7/7. Signed-off-by: Nathan Walston <nwalston@nvidia.com> --------- Signed-off-by: Nathan Walston <nwalston@nvidia.com> Co-authored-by: Marcus <marcusds@users.noreply.github.com>
9 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
1222-stitched.mov
Summary
Two coherent changes to the Fabric email example (the second motivates the first):
email-phishing-agent→email-security-triage("Email Security Triage") — with the specialists the example is broader than "phishing"; it's an email-security triage orchestrator.1. Specialist sub-agents
url-brand-analystextract_iocsharvested — tool → sub-agent composition in one traceattack-attributorbec/credential/malware/spam/benign)header-auth-analystAuthentication-Results:/Received:headersphishing-analyzerstays the verdict owner and folds the findings in (advisory — it can override or supply them); the verdict gainsattack_typeandimpersonated_brand.Prompts are ported from the
email-security-analystNAT example — co-authored with @nv-odrulea (they were removed from it in #1135 as more protocol than that sample needed). Here each is adeclarative_subagent, so its reasoning gets its own trace span and its prompt is tunable inagent.yaml. Octavian's prompt-injection guardrails are ported with them.analyze_headersis demo-only, on purpose: the scored dataset has no SMTP headers, and synthesizing them per row would leak thephishing/benignlabel into the input and inflate the eval. So the specialist is gated, the dataset stays header-free, and the README ships a header-bearing email to exercise it by hand.2. Rename → Email Security Triage
Full identifier sweep, collision-checked against the still-present NAT
email-security-analystandemail-phishing-analyzer:name, telemetry project →email-security-triagenemo-agent-config-example-email-phishing→…-email-security-triageemail-phishing-iocs→email-security-triage-iocs(MCP key staysiocs; module staysmcps/iocs.py)email-security-triage-eval.yml; default deployment name in the READMEagentConfigPath+ sample dir; re-synced the Studio sample copy to this branch's (specialist)agent.yamlUnchanged on purpose: the
is_likely_phishingverdict schema, thephishing-analyzersub-agent (it is the phishing specialist), and domain references to phishing.Resolves ASTD-411 (rename); the earlier plan to do it as a post-merge capstone is superseded — folding it here avoids a rename that would conflict with this very PR.
Verification
--mode subprocess, real key:create→deploy→invokeunder the new slug returns the correct schema (phishingtrue, incl.attack_type/impersonated_brand); the renamed console script resolves on PATH. 3/3 clean runs (phishing plain + header-bearingtrue, benignfalse) confirmed earlier; runtime checkpointer shows all four sub-agent tasks + theextract_iocscall.AgentConfigvalidate + Fabrictranslate_agent_config;ruff; 8/8 tool tests; StudioloadSampleAgentConfigtests; zero rename stragglers.Tickets
Resolves ASTD-408 (specialists) and ASTD-411 (rename). Docs updated: README gains the header-specialist step;
CUSTOMIZE.mdgains an "Adding or removing a specialist" section.Summary by CodeRabbit
New Features
Documentation
Tests