Harden Markdown graph intelligence in compass.graph/1

CHANGELOG.md

@@ -2,6 +2,15 @@
 
 ## Unreleased
 
+- Harden Markdown intelligence within `compass.graph/1`. Pipe tables retain
+  table, header, row, and cell nodes while gaining stable semantic identities,
+  header-qualified labels, exact cell-owned references, bounded per-table
+  extraction, and topology-aware containment handling. Nested YAML frontmatter
+  now publishes exact source-backed config-key hierarchies with stable escaped
+  paths, conservative semantic labels, bounded parsing, and fail-closed unsafe
+  syntax handling. Published document nodes remain graph-v1 resources; no graph
+  schema migration is required.
+
 - Make community detail graphs easier to scan in both exported HTML and VS
   Code by grouping node kinds into accessible color-and-shape families,
   coloring edges by relationship purpose while retaining confidence strokes,
@@ -12,7 +21,7 @@
   now the clearer lifecycle states `Qualifying`/`Qualified`; the serialized
   evidence envelope moves from `adapter` to `pipeline`, replaces `profile`
   with `qualification`, and replaces the producer string with `emitter`.
-  Universal evidence schema is now `/2` and extraction semantics is `/3`, so
+  Universal evidence schema is now `/2` and extraction semantics is `/4`, so
   pre-refactor caches and evidence artifacts are intentionally rebuilt.
   Qualification manifests, candidate exports, and TypeScript scorecards now
   use version-2 schemas and call their language identity `producer`.

COMPATIBILITY.md

@@ -182,6 +182,21 @@ with this vocabulary. Strict query, MCP, CLI, VS Code, and viewer consumers
 must use the matching manifest; they must reject an unknown `edgeKind` or
 `nodeRole` instead of filtering it into an older response shape.
 
+Markdown semantic table intelligence remains within `compass.graph/1` by using
+the established resource-node, qualified-name, source-anchor, and relationship
+contracts. Pipe tables continue to publish table, header, row, and cell nodes.
+Their semantic labels, stable identities, exact cell-owned references, and
+bounded extraction are producer-logic improvements; no new graph wire fields
+or schema migration are required. Consumers must continue to reject unknown
+graph majors and must not infer document roles from display labels.
+
+Markdown frontmatter intelligence likewise remains within `compass.graph/1`.
+Bounded nested YAML metadata publishes through established `config_key` nodes,
+Config provenance, exact source anchors, canonical key paths, and `contains`
+relationships. Value-independent IDs and JSON Pointer escaping are producer
+identity rules, not new wire fields. Generic metadata values are not copied
+into graph labels; strict readers need no schema migration.
+
 Swift, Dart, Scala, and Groovy/Gradle now publish through their version-1
 universal evidence pipelines. The four pipelines are intentionally
 `Qualifying`: they use one bounded, source-grounded publication route and may

MIGRATION.md

@@ -80,7 +80,7 @@ dispatch as a missing deterministic fact.
 ## Universal evidence schema reset
 
 The universal evidence envelope is now `compass.languages.evidence/2` and the
-extraction semantics identity is `compass.languages.extraction/3`. The envelope
+extraction semantics identity is `compass.languages.extraction/4`. The envelope
 field is `pipeline` (with `qualification` and `emitter` metadata), replacing
 the provisional `adapter`/`profile`/`producer` shape. Compass intentionally
 does not translate or reuse pre-refactor universal evidence; run a forced
@@ -89,6 +89,11 @@ candidate exports, and TypeScript scorecards likewise require their `/2`
 schemas and use `producer` for the language evidence identity; regenerate
 those audit inputs rather than trying to load the old field names.
 
+Version 4 also invalidates pre-enhancement Markdown caches so nested
+frontmatter can be republished as exact graph-v1 config nodes. Normal builds
+re-extract affected files automatically; no graph schema migration or manual
+artifact editing is required.
+
 ## Python project identity and stubs
 
 Python now publishes version-13 `compass.python` evidence. Static

PERFORMANCE.md

@@ -24,6 +24,20 @@ Compare a proposed change with a previously approved Compass result captured on
 the same runner and corpus. A median regression above 10% requires explicit
 review and evidence explaining the tradeoff.
 
+## Markdown graph-v1 quality qualification
+
+Markdown tables intentionally retain their table, header, row, and cell nodes.
+The implementation bounds each table independently to 20,000 structural nodes,
+16,384 cells, and 512 KiB of retained table text, in addition to the extractor's
+global limits. Exhausting a table budget emits explicit limit evidence and does
+not consume the budget needed to discover later headings. The graph-v1 fixture
+gate runs an independent source oracle for hierarchy, semantic labels, exact
+anchors, reference ownership, and published-schema integrity. The same oracle
+checks frontmatter Config nodes, nested containment, stable canonical paths,
+Config provenance, value-independent identity, and the absence of unapproved
+generic values from graph labels. Frontmatter is capped at 64 KiB and its YAML
+syntax pass is linear in that bounded input.
+
 ## Incremental code-graph qualification
 
 Fact-neutral updates may bypass project-wide resolution only after the changed

advisor-plans/024-markdown-graph-intelligence.md

@@ -0,0 +1,167 @@
+# Markdown graph-v1 intelligence hardening
+
+Status: implementation and release qualification in progress
+
+Public contract: `compass.graph/1`
+Owner boundaries: `compass-languages`, `compass-graph`, `compass-model`,
+`compass-query`, `compass-store`, and `compass-output`
+
+## Objective
+
+Improve Markdown graph usefulness without changing the graph schema or removing
+table, header, row, or cell nodes. The implementation must make those nodes
+semantically useful, stable, source-grounded, searchable, and harmless to
+architecture topology. It must remain deterministic, bounded, local-first, and
+fail closed.
+
+## Compatibility decision
+
+`compass.graph/1` remains the only published graph contract. Rich document facts
+may exist while one file is normalized, but they are not a wire contract. The
+graph publisher resolves their evidence and converts every document node to the
+established graph-v1 `resource` details before validation and publication.
+Validation rejects normalization-only document details if they reach a graph-v1
+artifact.
+
+Document roles are recovered centrally from extractor-owned qualified identity,
+not display text and not a new serialized field. This keeps query, clustering,
+analysis, and output behavior consistent while old strict readers continue to
+receive the graph-v1 shape they already understand.
+
+## Quality target
+
+The reviewed external Markdown graph extractor is the comparative baseline. Its
+public deterministic implementation recognizes ATX headings and document links,
+skips fenced code, and publishes line-level locations. Compass must cover those
+useful facts and additionally qualify:
+
+- ATX and Setext headings with exact byte/line/column anchors;
+- fenced code and the existing Markdown block vocabulary;
+- table, header, row, and cell hierarchy;
+- semantic table labels derived from headers and values;
+- exact link/reference ownership by the smallest containing cell;
+- conservative exact, ambiguous, unresolved, and limited resolution;
+- stable identities across line shifts and non-identity cell edits;
+- search parity between scan and immutable-index paths;
+- isolation of table navigation containment from architecture topology; and
+- explicit per-table limit evidence without suppressing later document facts.
+
+The comparison is source-based. No external graph implementation is added as a
+runtime, test, configuration, artifact, or fallback dependency.
+
+## Phase 1: Contract boundary
+
+Context: typed document facts previously risked becoming an accidental new
+public schema.
+
+Execution:
+
+1. Keep publication and strict loading on `CODE_GRAPH_SCHEMA_V1`.
+2. Reject normalization-only document details in the graph-v1 validator.
+3. Resolve document references before converting document details to graph-v1
+   resource details.
+4. Remove graph-v2 adapters, gates, workflow targets, and migration claims.
+
+Acceptance criteria:
+
+- every newly published graph reports `compass.graph/1`;
+- every document node uses graph-v1 resource details on the wire;
+- unknown majors fail explicitly;
+- strict graph-v1 load/round-trip tests pass; and
+- stable IDs, edge direction, multiplicity, anchors, and provenance survive the
+  normalization projection.
+
+## Phase 2: Semantic table extraction
+
+Context: generic `pipe_table_row` and `pipe_table_cell` labels provide syntax
+volume but little retrieval or inspection value.
+
+Execution:
+
+1. Retain table, header, row, and cell nodes.
+2. Give each node a section-qualified, occurrence-safe identity.
+3. Label tables with section/header context, rows as `Header=value`, and cells
+   as `Header: value`, including explicit empty/limited labels.
+4. Preserve exact source anchors and the full containment hierarchy.
+5. Assign inline links and backtick code references to the smallest exact cell.
+
+Acceptance criteria:
+
+- all four table roles are present for a normal pipe table;
+- labels are meaningful without consulting private attributes;
+- nested anchors are contained by their parent anchors;
+- row and cell IDs survive unrelated line insertion and non-identity edits;
+- references originate at the containing cell; and
+- output is byte-deterministic for equivalent input.
+
+## Phase 3: Bounds and failure truthfulness
+
+Context: a giant early table must not consume the global block budget and hide
+later headings.
+
+Execution:
+
+1. Enforce independent table caps of 20,000 structural nodes, 16,384 cells, and
+   512 KiB retained text.
+2. Count omitted facts and emit bounded diagnostics.
+3. Continue scanning the document after the table budget is exhausted.
+
+Acceptance criteria:
+
+- no table exceeds any configured cap;
+- a limit is not reported as an empty table;
+- omitted counts are deterministic and truthful;
+- source anchors remain ordered and in bounds; and
+- headings after an oversized table are still extracted.
+
+## Phase 4: Retrieval and topology
+
+Context: semantic table content must be discoverable, but navigation
+containment must not inflate architecture centrality.
+
+Execution:
+
+1. Index semantic node names and qualified identities in both scan and
+   immutable snapshot paths.
+2. Keep exact document-to-code and document-to-file references as ordinary
+   graph-v1 reference edges.
+3. Exclude containment edges touching table navigation nodes from architecture
+   degree, clustering, and topology summaries only.
+4. Keep those nodes and edges available for search, traversal, inspection, and
+   source navigation.
+
+Acceptance criteria:
+
+- scan and immutable-index rankers retrieve the same table cell query;
+- exact references resolve to their unique code/file targets;
+- ambiguous and unresolved references never acquire invented targets;
+- topology scores do not change merely because a table gains cells; and
+- table nodes remain present in the graph and viewer.
+
+## Phase 5: Independent qualification
+
+Context: parser snapshots alone can reproduce extractor mistakes. Quality needs
+a source-derived oracle.
+
+Execution:
+
+1. Add an adversarial Markdown fixture with tables and exact local/code links.
+2. Run an independent source oracle against the published graph-v1 artifact.
+3. Integrate it into `qualify_code_graph_v1.sh --fixtures-only`.
+4. Verify the product boundary so no Graphify dependency enters Compass.
+
+Acceptance criteria:
+
+- the oracle verifies schema integrity, role counts, semantic labels,
+  hierarchy, anchors, reference ownership, and exact targets;
+- repeated qualification builds are byte-identical;
+- the quality score meets the checked-in threshold;
+- the graph-v1 fixture release gate passes; and
+- `scripts/check_product_boundary.sh` passes.
+
+## Rollback
+
+Revert the extractor semantic-label/identity logic, graph-v1 normalization
+projection, central role inference, topology filtering, and the independent
+oracle together. Do not change the schema string, rewrite history, or remove
+published table/header/row/cell records during rollback.

advisor-plans/README.md

@@ -90,6 +90,12 @@ determinism, lifecycle, and performance evidence before promoting any pack
 claim. Python language promotion remains a separate complete-capability
 decision.
 
+Plan 024 deepens Plan 009's parser-backed Markdown support without changing the
+`compass.graph/1` wire contract. It retains table, header, row, and cell nodes,
+adds semantic identities and labels, resolves cell-owned references before the
+normalization facts are projected back to graph-v1 resources, keeps table
+navigation out of architecture topology, and adds an independent quality gate.
+
 ## Execution order and status
 
 | Plan | Title | Priority | Effort | Depends on | Status |
@@ -117,6 +123,7 @@ decision.
 | 021 | Make React frontend framework graphs enterprise-ready | P1 | XXL | 013 production hard cut; final gate should consume 005 or equivalent | DONE |
 | 022 | Add bounded, quality-gated OCR to document processing | P1 | XL | 006, 007, 008, 010 | IN PROGRESS |
 | 023 | Make Python framework graphs source-proven and production-qualified | P1 | XXL | —; final gate should consume 005 or equivalent | BLOCKED |
+| 024 | Harden Markdown graph-v1 intelligence | P1 | XL | 009; coordinate with 012 | IN PROGRESS |
 
 Status values: `TODO`, `IN PROGRESS`, `DONE`, `BLOCKED`, or `REJECTED`.
 
@@ -183,6 +190,10 @@ helper runtimes prerequisites for native document support.
   gates, the native structural qualification. The final public claim should
   consume Plan 005's exact-production-evidence model or an equivalent release
   gate.
+- Plan 024 keeps graph-v1 publication strict. Rich typed document facts are an
+  internal normalization representation only; the publisher resolves their
+  evidence and downgrades them to established resource details before strict
+  validation. Table navigation nodes remain public and searchable.
 
 ## Direction options not promoted to implementation plans
 
@@ -209,6 +220,8 @@ helper runtimes prerequisites for native document support.
   010.
 - **ODT/ODS/ODP and EPUB:** architecturally fit the document artifact after the
   core formats are qualified, but have their own package and semantic rules.
+- **HTML and Office table convergence in Plan 024:** deferred to Plan 012 or a
+  focused follow-up; this change is limited to Markdown extraction semantics.
 
 ## Findings considered and rejected
 

crates/compass-core/src/pipeline.rs

@@ -36,8 +36,8 @@ use compass_languages::{
     ResolutionConstraint, ScopeFact, SemanticEvidenceBatch, file_stem, make_id,
 };
 use compass_model::code_graph::{
-    CommunityMetadata, CoverageRecord, DiagnosticSeverity, ExtractionStatus, FileNodeDetails,
-    GraphDiagnostic, GraphDocument as V1GraphDocument, NodeDetails, NodeKind,
+    CommunityMetadata, CoverageRecord, DiagnosticSeverity, ExtractionStatus, GraphDiagnostic,
+    GraphDocument as V1GraphDocument, NodeKind,
 };
 use compass_model::provenance::{
     COALESCED_NODE_EVIDENCE_ATTRIBUTE, CONSUME_INCREMENTAL_ENDPOINT_REMAP_ATTRIBUTE,
@@ -1860,16 +1860,14 @@ fn prepare_fact_neutral_document(
             continue;
         };
         let source = relative_fact_path(Path::new(source), root);
-        let Some(file) = files.get(source.as_str()) else {
+        if !files.contains_key(source.as_str()) {
             continue;
-        };
-        if node.kind == NodeKind::File {
-            node.details = Some(NodeDetails::File(FileNodeDetails {
-                content_digest: file.content_digest.clone(),
-                byte_size: file.byte_size,
-                generated: file.generated,
-            }));
         }
+        // The canonical graph-v1 file inventory owns digest, size, and generated
+        // state. Do not synthesize an optional file-node payload only on the
+        // fact-neutral route: full publication intentionally preserves the
+        // established graph-v1 node shape, and adding it here makes an
+        // edit-then-restore build differ from a clean build.
         let refreshed_envelope = source_digests.contains_key(&source).then(|| {
             anchors
                 .get(&source)

crates/compass-graph/src/analyze.rs

@@ -1726,6 +1726,9 @@ impl<'a> AnalysisGraph<'a> {
                 right: *right,
                 record,
             });
+            if zero_topology_document_containment(record, nodes[*left], nodes[*right]) {
+                continue;
+            }
             let key = if document.directed || left <= right {
                 (*left, *right)
             } else {
@@ -1856,6 +1859,18 @@ fn is_json_key_node(node: &NodeRecord) -> bool {
     attribute(node, "source_file").is_some_and(|source| source.to_lowercase().ends_with(".json"))
         && JSON_NOISE_LABELS.contains(&node.label().trim().to_lowercase().as_str())
 }
+
+fn zero_topology_document_containment(
+    edge: &EdgeRecord,
+    source: &NodeRecord,
+    target: &NodeRecord,
+) -> bool {
+    if edge_string(edge, "relation") != "contains" {
+        return false;
+    }
+    source.is_table_navigation_node() || target.is_table_navigation_node()
+}
+
 fn attribute<'a>(node: &'a NodeRecord, key: &str) -> Option<&'a str> {
     match key {
         "source_file" => node.source_file(),