Skip to content

feat: establish a private graph-native protection runtime - #253

Draft
binaryaaron wants to merge 52 commits into
mainfrom
codex/anonymizer-openshell-intake
Draft

feat: establish a private graph-native protection runtime#253
binaryaaron wants to merge 52 commits into
mainfrom
codex/anonymizer-openshell-intake

Conversation

@binaryaaron

@binaryaaron binaryaaron commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Related issue

Resolves #257

Summary

Completes the private graph-native protection runtime through Phase 7 P8. The new private
stable-substitute-v1 profile compiles the frozen owner contract, carries accepted Phase 6
mentions into bounded scope planning, validates and applies one immutable replacement bundle,
and releases protected text only through Phase 4 terminal accounting.

The public DataFrame API remains compatible. Public Substitute still uses the legacy
replacement workflow; Redact, Annotate, Hash, Rewrite, run, preview, evaluate, display, trace,
validation, and CLI behavior are unchanged. No public graph API or production Intake/OpenShell
integration is added.

Phase status

  • Phases 1–3 established private graph admission and compatibility seams.
  • Phase 4 added hierarchical terminal accounting and fail-closed atomic release.
  • Phase 5 added target/context workframes, reconciliation, cleanup, and conformance evidence.
  • Phase 6 added anchored mention admission, explicit-evidence subject resolution, structural
    roles, private Redact, and the accepted Substitute handoff.
  • Phase 7 P0–P7 froze the owner contract and added manifest compilation, validation, anchored
    application, the NDD candidate backend, planner ledger, and release-gated runtime.
  • Phase 7 P8 now wires the accepted profile through private _ProtectionFlow selection and
    completes privacy/release convergence.

Grouped Rewrite/P9 and the broader public SDK Phase 8 proposal have not started.

P8 material changes

  • Selects default Substitute() only through the private _compile_protection_plan() and
    _ProtectionFlow path, using the frozen stable-substitute-v1 contract.
  • Runs Phase 6 Substitute detection and handoff, Phase 7 scope planning/validation/application,
    and Phase 4 reconciliation through the private protection service.
  • Adds compiler-owned datum application tasks. The sealed Phase 7 plan records the complete
    effective prerequisite set, including scope and prior-stage edges, while Phase 4 accounting
    remains the sole release authority.
  • Attributes anchored application failures per datum, so an affected atomic group is withheld
    without invalidating an independent group in the same coherence scope.
  • Derives release only from Phase 4 _GroupReleased terminal candidates; no side map or raw-input
    fallback can authorize output.
  • Narrows stable-Substitute receipts to an exact five-field content allowlist without changing
    the legacy Redact receipt.
  • Adds compatibility coverage for ordering, filtering, indexes, prompt instructions, no-entity
    bypass, collision/fallback behavior, output shape, exact FailedRecord behavior, display,
    traces, CLI, and all adjacent public strategies.
  • Adds structural privacy allowlists over active and withheld state, workframes, logs, metrics,
    exceptions and causes, tracebacks, receipts, serialization, and cleanup faults. Distinct
    original, synthetic, prompt, source-ID, and digest canaries prove that private material does
    not escape and synthetic values appear only in qualified output locations.

Boundaries and non-goals

The branch does not:

  • change public APIs, signatures, defaults, result fields, or the bundled Anonymizer skill;
  • lower public Substitute through the graph profile;
  • qualify private Annotate, Hash, grouped Rewrite, evaluation, or repair;
  • add production Intake, OpenShell, ATIF, OTLP, OCSF, Sandbox, persistence, retry, retention, or
    delivery integration;
  • call DataDesigner directly outside NddAdapter.run_workflow(); or
  • claim exhaustive PII detection or provider-side secure deletion.

Review and validation

  • Four cumulative maintained review rounds covered behavioral fidelity; fault, privacy, and
    lifecycle safety; architecture and ownership; and final full-diff convergence.
  • Every authenticated Critical or Warning was resolved. The final review accepted signed commit
    7db42f75cfe36ab6928fd2fb4195cb5e01ea8eaf with zero Critical, Warning, or Nit findings.
  • make check passed, including formatting, Ruff, type checks, lock validation, and SPDX checks
    for all 322 files.
  • make test: 4,330 passed, 11 skipped.
  • make coverage: 4,330 passed, 11 skipped; 90% total coverage.
  • Exact-head wheel nemo_anonymizer-0.3.3.post58.dev0+7db42f7-py3-none-any.whl was installed into
    a fresh environment. Imports resolved from site-packages, all frozen resources were present,
    the contract digest matched, and the installed privacy/cleanup suite passed.
  • Wheel SHA-256: 5fdeb4220b121fe5ea1c754a40169cdf724efbbe3d4d91e9500a1314492ab658.
  • P7 evidence
  • P8 evidence
  • P8 CI run 33789131057: Check and Python 3.11–3.13 passed. DCO, linked-issue, conventional-commit, documentation, and required NVSkills checks also passed.

The 11 opt-in Intake dogfood tests remain skipped unless their operator-owned environment is
enabled. The one test warning is an existing third-party wandb_workspaces deprecation warning.

Reviewer guide

  1. Start with docs/development/phase-7-stable-substitute-design.md, the frozen contract
    resources, and the Phase 7 reference model.
  2. Review phase7_admission.py, phase7_validation.py, and phase7_application.py for scope,
    slot, relation, collision, anchored-application, and proof invariants.
  3. Review phase7_ndd_backend.py, phase7_planner_ledger.py, and phase7_runtime.py for the
    adapter boundary, absorbing lifecycle states, cleanup, cancellation/loss, application-task
    accounting, and release evidence.
  4. Review protection_service.py and interface/_protection.py for private-only selection and
    result/receipt materialization.
  5. Review the P8 wiring, privacy-convergence, public-compatibility, accounting, and Phase 7 tests
    for the acceptance matrix and adversarial canaries.

The PR remains draft. Do not mark it ready or merge without explicit operator approval.

@binaryaaron binaryaaron changed the title feat: validate a private protection flow with Intake feat: establish a private graph-native protection runtime Aug 20, 2026
binaryaaron and others added 23 commits August 20, 2026 17:33
Preserve the hardened closed-mapping structured-item harness and exercise complete OCSF 1.7.0 Process Activity JSONL records with local redaction. This remains test-only and adds no production or public streaming surface.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Add an opt-in protected-only Intake dogfood for format fidelity, withholding, delivery failure, and exact-byte retry behavior.

Require stable chat timestamps and agent-name metadata, and document public result atomicity plus adapter/Intake ownership.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron
binaryaaron force-pushed the codex/anonymizer-openshell-intake branch from e20eb93 to fbe2aea Compare August 20, 2026 17:36
@binaryaaron
binaryaaron marked this pull request as ready for review August 20, 2026 17:36
@binaryaaron
binaryaaron requested review from a team as code owners August 20, 2026 17:36
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Route the reachable private flow through the sealed Phase 6 plan and runtime with stage-specific NDD effects. Keep detector work target-only, localize attributable context-peer faults, and materialize only verified released datums.

Freeze executable lifecycle schedules, compare the semantic corpus with production execution/accounting, and add the required public DataFrame compatibility evidence.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron

binaryaaron commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator Author

Phase 7 P0 governance is not yet complete.

Implementation remains blocked. The semantic and execution owners must approve the frozen v1 contract, including all-pairs exact distinctness and the owned semantic/execution decisions. A separate operator checkpoint must then explicitly authorize Phase 7 before any contract-loader, fixture, compiler, planner, or runtime implementation begins.

If governance moves to Linear, update the governing reference and approval record before owner review.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron
binaryaaron force-pushed the codex/anonymizer-openshell-intake branch from 833561f to b1e9805 Compare September 1, 2026 16:40
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Validate complete scope bundles against the frozen contract, then bind replacements to authoritative mention spans and reconstruct source text without fallback or cascading edits.

Add direct P4 oracle conformance and harden the reference mutation harness while preserving the public Substitute path.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Generate one bounded NDD proposal per nonempty scope and reconcile it through exact opaque identities and fail-closed failure evidence. Keep candidate validity with the existing Phase 7 validator and leave the public Substitute path unchanged.

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
@binaryaaron
binaryaaron force-pushed the codex/anonymizer-openshell-intake branch from 4b656cf to 2de3f0f Compare September 3, 2026 04:08
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement private Phase 7 stable Substitute planning

2 participants