diff --git a/CHANGELOG.md b/CHANGELOG.md index 1ab500448..aea5188b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,15 @@ ## Unreleased +- Harden Markdown intelligence within `compass.graph/1`. Pipe tables retain + table, header, row, and cell nodes while gaining stable semantic identities, + header-qualified labels, exact cell-owned references, bounded per-table + extraction, and topology-aware containment handling. Nested YAML frontmatter + now publishes exact source-backed config-key hierarchies with stable escaped + paths, conservative semantic labels, bounded parsing, and fail-closed unsafe + syntax handling. Published document nodes remain graph-v1 resources; no graph + schema migration is required. + - Make community detail graphs easier to scan in both exported HTML and VS Code by grouping node kinds into accessible color-and-shape families, coloring edges by relationship purpose while retaining confidence strokes, @@ -12,7 +21,7 @@ now the clearer lifecycle states `Qualifying`/`Qualified`; the serialized evidence envelope moves from `adapter` to `pipeline`, replaces `profile` with `qualification`, and replaces the producer string with `emitter`. - Universal evidence schema is now `/2` and extraction semantics is `/3`, so + Universal evidence schema is now `/2` and extraction semantics is `/4`, so pre-refactor caches and evidence artifacts are intentionally rebuilt. Qualification manifests, candidate exports, and TypeScript scorecards now use version-2 schemas and call their language identity `producer`. diff --git a/COMPATIBILITY.md b/COMPATIBILITY.md index 72964931a..84b4dc7af 100644 --- a/COMPATIBILITY.md +++ b/COMPATIBILITY.md @@ -182,6 +182,21 @@ with this vocabulary. Strict query, MCP, CLI, VS Code, and viewer consumers must use the matching manifest; they must reject an unknown `edgeKind` or `nodeRole` instead of filtering it into an older response shape. +Markdown semantic table intelligence remains within `compass.graph/1` by using +the established resource-node, qualified-name, source-anchor, and relationship +contracts. Pipe tables continue to publish table, header, row, and cell nodes. +Their semantic labels, stable identities, exact cell-owned references, and +bounded extraction are producer-logic improvements; no new graph wire fields +or schema migration are required. Consumers must continue to reject unknown +graph majors and must not infer document roles from display labels. + +Markdown frontmatter intelligence likewise remains within `compass.graph/1`. +Bounded nested YAML metadata publishes through established `config_key` nodes, +Config provenance, exact source anchors, canonical key paths, and `contains` +relationships. Value-independent IDs and JSON Pointer escaping are producer +identity rules, not new wire fields. Generic metadata values are not copied +into graph labels; strict readers need no schema migration. + Swift, Dart, Scala, and Groovy/Gradle now publish through their version-1 universal evidence pipelines. The four pipelines are intentionally `Qualifying`: they use one bounded, source-grounded publication route and may diff --git a/MIGRATION.md b/MIGRATION.md index b441edc4b..789a1ebfd 100644 --- a/MIGRATION.md +++ b/MIGRATION.md @@ -80,7 +80,7 @@ dispatch as a missing deterministic fact. ## Universal evidence schema reset The universal evidence envelope is now `compass.languages.evidence/2` and the -extraction semantics identity is `compass.languages.extraction/3`. The envelope +extraction semantics identity is `compass.languages.extraction/4`. The envelope field is `pipeline` (with `qualification` and `emitter` metadata), replacing the provisional `adapter`/`profile`/`producer` shape. Compass intentionally does not translate or reuse pre-refactor universal evidence; run a forced @@ -89,6 +89,11 @@ candidate exports, and TypeScript scorecards likewise require their `/2` schemas and use `producer` for the language evidence identity; regenerate those audit inputs rather than trying to load the old field names. +Version 4 also invalidates pre-enhancement Markdown caches so nested +frontmatter can be republished as exact graph-v1 config nodes. Normal builds +re-extract affected files automatically; no graph schema migration or manual +artifact editing is required. + ## Python project identity and stubs Python now publishes version-13 `compass.python` evidence. Static diff --git a/PERFORMANCE.md b/PERFORMANCE.md index e4973c037..184d43e19 100644 --- a/PERFORMANCE.md +++ b/PERFORMANCE.md @@ -24,6 +24,20 @@ Compare a proposed change with a previously approved Compass result captured on the same runner and corpus. A median regression above 10% requires explicit review and evidence explaining the tradeoff. +## Markdown graph-v1 quality qualification + +Markdown tables intentionally retain their table, header, row, and cell nodes. +The implementation bounds each table independently to 20,000 structural nodes, +16,384 cells, and 512 KiB of retained table text, in addition to the extractor's +global limits. Exhausting a table budget emits explicit limit evidence and does +not consume the budget needed to discover later headings. The graph-v1 fixture +gate runs an independent source oracle for hierarchy, semantic labels, exact +anchors, reference ownership, and published-schema integrity. The same oracle +checks frontmatter Config nodes, nested containment, stable canonical paths, +Config provenance, value-independent identity, and the absence of unapproved +generic values from graph labels. Frontmatter is capped at 64 KiB and its YAML +syntax pass is linear in that bounded input. + ## Incremental code-graph qualification Fact-neutral updates may bypass project-wide resolution only after the changed diff --git a/advisor-plans/024-markdown-graph-intelligence.md b/advisor-plans/024-markdown-graph-intelligence.md new file mode 100644 index 000000000..5a81f4dcf --- /dev/null +++ b/advisor-plans/024-markdown-graph-intelligence.md @@ -0,0 +1,167 @@ +# Markdown graph-v1 intelligence hardening + +Status: implementation and release qualification in progress + +Public contract: `compass.graph/1` +Owner boundaries: `compass-languages`, `compass-graph`, `compass-model`, +`compass-query`, `compass-store`, and `compass-output` + +## Objective + +Improve Markdown graph usefulness without changing the graph schema or removing +table, header, row, or cell nodes. The implementation must make those nodes +semantically useful, stable, source-grounded, searchable, and harmless to +architecture topology. It must remain deterministic, bounded, local-first, and +fail closed. + +## Compatibility decision + +`compass.graph/1` remains the only published graph contract. Rich document facts +may exist while one file is normalized, but they are not a wire contract. The +graph publisher resolves their evidence and converts every document node to the +established graph-v1 `resource` details before validation and publication. +Validation rejects normalization-only document details if they reach a graph-v1 +artifact. + +Document roles are recovered centrally from extractor-owned qualified identity, +not display text and not a new serialized field. This keeps query, clustering, +analysis, and output behavior consistent while old strict readers continue to +receive the graph-v1 shape they already understand. + +## Quality target + +The reviewed external Markdown graph extractor is the comparative baseline. Its +public deterministic implementation recognizes ATX headings and document links, +skips fenced code, and publishes line-level locations. Compass must cover those +useful facts and additionally qualify: + +- ATX and Setext headings with exact byte/line/column anchors; +- fenced code and the existing Markdown block vocabulary; +- table, header, row, and cell hierarchy; +- semantic table labels derived from headers and values; +- exact link/reference ownership by the smallest containing cell; +- conservative exact, ambiguous, unresolved, and limited resolution; +- stable identities across line shifts and non-identity cell edits; +- search parity between scan and immutable-index paths; +- isolation of table navigation containment from architecture topology; and +- explicit per-table limit evidence without suppressing later document facts. + +The comparison is source-based. No external graph implementation is added as a +runtime, test, configuration, artifact, or fallback dependency. + +## Phase 1: Contract boundary + +Context: typed document facts previously risked becoming an accidental new +public schema. + +Execution: + +1. Keep publication and strict loading on `CODE_GRAPH_SCHEMA_V1`. +2. Reject normalization-only document details in the graph-v1 validator. +3. Resolve document references before converting document details to graph-v1 + resource details. +4. Remove graph-v2 adapters, gates, workflow targets, and migration claims. + +Acceptance criteria: + +- every newly published graph reports `compass.graph/1`; +- every document node uses graph-v1 resource details on the wire; +- unknown majors fail explicitly; +- strict graph-v1 load/round-trip tests pass; and +- stable IDs, edge direction, multiplicity, anchors, and provenance survive the + normalization projection. + +## Phase 2: Semantic table extraction + +Context: generic `pipe_table_row` and `pipe_table_cell` labels provide syntax +volume but little retrieval or inspection value. + +Execution: + +1. Retain table, header, row, and cell nodes. +2. Give each node a section-qualified, occurrence-safe identity. +3. Label tables with section/header context, rows as `Header=value`, and cells + as `Header: value`, including explicit empty/limited labels. +4. Preserve exact source anchors and the full containment hierarchy. +5. Assign inline links and backtick code references to the smallest exact cell. + +Acceptance criteria: + +- all four table roles are present for a normal pipe table; +- labels are meaningful without consulting private attributes; +- nested anchors are contained by their parent anchors; +- row and cell IDs survive unrelated line insertion and non-identity edits; +- references originate at the containing cell; and +- output is byte-deterministic for equivalent input. + +## Phase 3: Bounds and failure truthfulness + +Context: a giant early table must not consume the global block budget and hide +later headings. + +Execution: + +1. Enforce independent table caps of 20,000 structural nodes, 16,384 cells, and + 512 KiB retained text. +2. Count omitted facts and emit bounded diagnostics. +3. Continue scanning the document after the table budget is exhausted. + +Acceptance criteria: + +- no table exceeds any configured cap; +- a limit is not reported as an empty table; +- omitted counts are deterministic and truthful; +- source anchors remain ordered and in bounds; and +- headings after an oversized table are still extracted. + +## Phase 4: Retrieval and topology + +Context: semantic table content must be discoverable, but navigation +containment must not inflate architecture centrality. + +Execution: + +1. Index semantic node names and qualified identities in both scan and + immutable snapshot paths. +2. Keep exact document-to-code and document-to-file references as ordinary + graph-v1 reference edges. +3. Exclude containment edges touching table navigation nodes from architecture + degree, clustering, and topology summaries only. +4. Keep those nodes and edges available for search, traversal, inspection, and + source navigation. + +Acceptance criteria: + +- scan and immutable-index rankers retrieve the same table cell query; +- exact references resolve to their unique code/file targets; +- ambiguous and unresolved references never acquire invented targets; +- topology scores do not change merely because a table gains cells; and +- table nodes remain present in the graph and viewer. + +## Phase 5: Independent qualification + +Context: parser snapshots alone can reproduce extractor mistakes. Quality needs +a source-derived oracle. + +Execution: + +1. Add an adversarial Markdown fixture with tables and exact local/code links. +2. Run an independent source oracle against the published graph-v1 artifact. +3. Integrate it into `qualify_code_graph_v1.sh --fixtures-only`. +4. Verify the product boundary so no Graphify dependency enters Compass. + +Acceptance criteria: + +- the oracle verifies schema integrity, role counts, semantic labels, + hierarchy, anchors, reference ownership, and exact targets; +- repeated qualification builds are byte-identical; +- the quality score meets the checked-in threshold; +- the graph-v1 fixture release gate passes; and +- `scripts/check_product_boundary.sh` passes. + +## Rollback + +Revert the extractor semantic-label/identity logic, graph-v1 normalization +projection, central role inference, topology filtering, and the independent +oracle together. Do not change the schema string, rewrite history, or remove +published table/header/row/cell records during rollback. diff --git a/advisor-plans/README.md b/advisor-plans/README.md index 2c8b9bedb..894fd8129 100644 --- a/advisor-plans/README.md +++ b/advisor-plans/README.md @@ -90,6 +90,12 @@ determinism, lifecycle, and performance evidence before promoting any pack claim. Python language promotion remains a separate complete-capability decision. +Plan 024 deepens Plan 009's parser-backed Markdown support without changing the +`compass.graph/1` wire contract. It retains table, header, row, and cell nodes, +adds semantic identities and labels, resolves cell-owned references before the +normalization facts are projected back to graph-v1 resources, keeps table +navigation out of architecture topology, and adds an independent quality gate. + ## Execution order and status | Plan | Title | Priority | Effort | Depends on | Status | @@ -117,6 +123,7 @@ decision. | 021 | Make React frontend framework graphs enterprise-ready | P1 | XXL | 013 production hard cut; final gate should consume 005 or equivalent | DONE | | 022 | Add bounded, quality-gated OCR to document processing | P1 | XL | 006, 007, 008, 010 | IN PROGRESS | | 023 | Make Python framework graphs source-proven and production-qualified | P1 | XXL | —; final gate should consume 005 or equivalent | BLOCKED | +| 024 | Harden Markdown graph-v1 intelligence | P1 | XL | 009; coordinate with 012 | IN PROGRESS | Status values: `TODO`, `IN PROGRESS`, `DONE`, `BLOCKED`, or `REJECTED`. @@ -183,6 +190,10 @@ helper runtimes prerequisites for native document support. gates, the native structural qualification. The final public claim should consume Plan 005's exact-production-evidence model or an equivalent release gate. +- Plan 024 keeps graph-v1 publication strict. Rich typed document facts are an + internal normalization representation only; the publisher resolves their + evidence and downgrades them to established resource details before strict + validation. Table navigation nodes remain public and searchable. ## Direction options not promoted to implementation plans @@ -209,6 +220,8 @@ helper runtimes prerequisites for native document support. 010. - **ODT/ODS/ODP and EPUB:** architecturally fit the document artifact after the core formats are qualified, but have their own package and semantic rules. +- **HTML and Office table convergence in Plan 024:** deferred to Plan 012 or a + focused follow-up; this change is limited to Markdown extraction semantics. ## Findings considered and rejected diff --git a/crates/compass-core/src/pipeline.rs b/crates/compass-core/src/pipeline.rs index 644f38a0d..98b139145 100644 --- a/crates/compass-core/src/pipeline.rs +++ b/crates/compass-core/src/pipeline.rs @@ -36,8 +36,8 @@ use compass_languages::{ ResolutionConstraint, ScopeFact, SemanticEvidenceBatch, file_stem, make_id, }; use compass_model::code_graph::{ - CommunityMetadata, CoverageRecord, DiagnosticSeverity, ExtractionStatus, FileNodeDetails, - GraphDiagnostic, GraphDocument as V1GraphDocument, NodeDetails, NodeKind, + CommunityMetadata, CoverageRecord, DiagnosticSeverity, ExtractionStatus, GraphDiagnostic, + GraphDocument as V1GraphDocument, NodeKind, }; use compass_model::provenance::{ COALESCED_NODE_EVIDENCE_ATTRIBUTE, CONSUME_INCREMENTAL_ENDPOINT_REMAP_ATTRIBUTE, @@ -1860,16 +1860,14 @@ fn prepare_fact_neutral_document( continue; }; let source = relative_fact_path(Path::new(source), root); - let Some(file) = files.get(source.as_str()) else { + if !files.contains_key(source.as_str()) { continue; - }; - if node.kind == NodeKind::File { - node.details = Some(NodeDetails::File(FileNodeDetails { - content_digest: file.content_digest.clone(), - byte_size: file.byte_size, - generated: file.generated, - })); } + // The canonical graph-v1 file inventory owns digest, size, and generated + // state. Do not synthesize an optional file-node payload only on the + // fact-neutral route: full publication intentionally preserves the + // established graph-v1 node shape, and adding it here makes an + // edit-then-restore build differ from a clean build. let refreshed_envelope = source_digests.contains_key(&source).then(|| { anchors .get(&source) diff --git a/crates/compass-graph/src/analyze.rs b/crates/compass-graph/src/analyze.rs index 68be0050c..3dad66ac6 100644 --- a/crates/compass-graph/src/analyze.rs +++ b/crates/compass-graph/src/analyze.rs @@ -1726,6 +1726,9 @@ impl<'a> AnalysisGraph<'a> { right: *right, record, }); + if zero_topology_document_containment(record, nodes[*left], nodes[*right]) { + continue; + } let key = if document.directed || left <= right { (*left, *right) } else { @@ -1856,6 +1859,18 @@ fn is_json_key_node(node: &NodeRecord) -> bool { attribute(node, "source_file").is_some_and(|source| source.to_lowercase().ends_with(".json")) && JSON_NOISE_LABELS.contains(&node.label().trim().to_lowercase().as_str()) } + +fn zero_topology_document_containment( + edge: &EdgeRecord, + source: &NodeRecord, + target: &NodeRecord, +) -> bool { + if edge_string(edge, "relation") != "contains" { + return false; + } + source.is_table_navigation_node() || target.is_table_navigation_node() +} + fn attribute<'a>(node: &'a NodeRecord, key: &str) -> Option<&'a str> { match key { "source_file" => node.source_file(), diff --git a/crates/compass-graph/src/cluster.rs b/crates/compass-graph/src/cluster.rs index 495bd66ea..be4b39632 100644 --- a/crates/compass-graph/src/cluster.rs +++ b/crates/compass-graph/src/cluster.rs @@ -445,8 +445,9 @@ pub fn label_communities_by_hub( .iter() .filter_map(|member| positions.get(member.as_str()).map(|index| (member, *index))) .min_by(|(left_id, left), (right_id, right)| { - is_pipe_table_structure(&document.nodes[*left]) - .cmp(&is_pipe_table_structure(&document.nodes[*right])) + document.nodes[*left] + .is_table_navigation_node() + .cmp(&document.nodes[*right].is_table_navigation_node()) .then_with(|| degrees[*right].cmp(°rees[*left])) .then_with(|| left_id.cmp(right_id)) }); @@ -454,7 +455,7 @@ pub fn label_communities_by_hub( let (base, context, context_required) = hub .and_then(|(_, index)| document.nodes.get(index)) .map(|node| { - let table_structure = is_pipe_table_structure(node); + let table_structure = node.is_table_navigation_node(); ( if table_structure { "Table".to_owned() @@ -520,11 +521,15 @@ pub fn label_communities_by_hub( labels.into_iter().collect() } -fn is_pipe_table_structure(node: &NodeRecord) -> bool { - matches!( - node.string("document_kind").as_str(), - "pipe_table" | "pipe_table_header" | "pipe_table_row" | "pipe_table_cell" - ) +fn zero_topology_document_containment( + edge: &EdgeRecord, + source: &NodeRecord, + target: &NodeRecord, +) -> bool { + if edge.string("relation") != "contains" { + return false; + } + source.is_table_navigation_node() || target.is_table_navigation_node() } fn concise_community_label(node: &NodeRecord) -> Option { @@ -1097,6 +1102,13 @@ impl WeightedGraph { else { continue; }; + if zero_topology_document_containment( + edge, + &document.nodes[*left], + &document.nodes[*right], + ) { + continue; + } let weight = edge.number("weight").unwrap_or(1.0); let candidate = selected .entry((*left, *right)) @@ -1781,6 +1793,32 @@ mod tests { assert_eq!(weighted.edge_count(), 0); } + #[test] + fn semantic_table_containment_is_navigation_only_for_topology() { + let mut document = graph( + &["heading", "table", "row", "code"], + &[("heading", "table"), ("table", "row"), ("row", "code")], + ); + for edge in &mut document.links { + edge.attributes + .insert("relation".to_owned(), json!("contains")); + } + document.links[2] + .attributes + .insert("relation".to_owned(), json!("references")); + document.nodes[1] + .attributes + .insert("qualifiedName".to_owned(), json!("Guide::pipe_table#1")); + document.nodes[2].attributes.insert( + "qualifiedName".to_owned(), + json!("Guide::pipe_table#1::pipe_table_row#graph-1"), + ); + let weighted = WeightedGraph::from_document(&document); + assert_eq!(weighted.edge_count(), 1); + assert_eq!(weighted.degree_unweighted(1), 0); + assert_eq!(weighted.degree_unweighted(2), 1); + } + #[test] fn duplicate_hub_labels_add_source_context_only_when_needed() { let mut document = graph(&["a", "b", "c", "d", "unique"], &[("a", "b"), ("c", "d")]); @@ -1815,16 +1853,10 @@ mod tests { } #[test] - fn meaningful_document_anchor_outranks_pipe_table_hub() { + fn meaningful_document_anchor_outranks_table_navigation_hub() { let mut document = graph( - &["heading", "table", "header", "row", "cell-a", "cell-b"], - &[ - ("heading", "table"), - ("table", "header"), - ("table", "row"), - ("header", "cell-a"), - ("row", "cell-b"), - ], + &["heading", "table", "row"], + &[("heading", "table"), ("table", "row")], ); document.nodes[0] .attributes @@ -1832,13 +1864,7 @@ mod tests { document.nodes[0] .attributes .insert("document_kind".to_owned(), json!("heading")); - for (index, kind) in [ - (1, "pipe_table"), - (2, "pipe_table_header"), - (3, "pipe_table_row"), - (4, "pipe_table_cell"), - (5, "pipe_table_cell"), - ] { + for (index, kind) in [(1, "table"), (2, "table_row")] { document.nodes[index] .attributes .insert("label".to_owned(), json!(kind.replace('_', " "))); @@ -1858,15 +1884,15 @@ mod tests { } #[test] - fn pipe_table_only_communities_use_source_anchored_table_labels() { + fn table_only_communities_use_source_anchored_table_labels() { let mut document = graph(&["left", "right"], &[]); for (index, line) in [(0, 12), (1, 44)] { document.nodes[index] .attributes - .insert("label".to_owned(), json!("pipe table")); + .insert("label".to_owned(), json!("table")); document.nodes[index] .attributes - .insert("document_kind".to_owned(), json!("pipe_table")); + .insert("document_kind".to_owned(), json!("table")); document.nodes[index] .attributes .insert("source_file".to_owned(), json!("docs/reference/outputs.md")); diff --git a/crates/compass-graph/src/lib.rs b/crates/compass-graph/src/lib.rs index 72b5fcb4f..aac7169aa 100644 --- a/crates/compass-graph/src/lib.rs +++ b/crates/compass-graph/src/lib.rs @@ -291,6 +291,14 @@ fn build_from_owned_extraction( endpoint_remap.extend(doc_remap); endpoint_remap.extend(ghost_remap); + // Document reference evidence is embedded in node attributes rather than + // represented by an edge endpoint at this assembly boundary. Carry the + // same deterministic endpoint remaps into that evidence so a later strict + // publication pass can resolve the retained target/candidate IDs instead + // of downgrading otherwise exact links to unresolved merely because a + // semantic or document-twin alias changed the raw ID. + remap_document_reference_attributes(&mut nodes, &endpoint_remap); + let mut normalized = EndpointAliases::new(); for node in &nodes { normalized @@ -515,6 +523,44 @@ fn publish_legacy_edge(record: RawEdgeRecord) -> LegacyEdgeRecord { } } +fn remap_document_reference_attributes( + nodes: &mut [NodeRecord], + endpoint_remap: &HashMap, +) { + for node in nodes { + let Some(Value::Array(references)) = node.attributes.get_mut("document_references") else { + continue; + }; + for reference in references { + let Some(object) = reference.as_object_mut() else { + continue; + }; + if let Some(target) = object + .get("target") + .and_then(Value::as_str) + .map(str::to_owned) + { + let remapped = remap_endpoint(&target, endpoint_remap); + object.insert("target".to_owned(), Value::String(remapped)); + } + if let Some(candidates) = object.get_mut("candidates").and_then(Value::as_array_mut) { + for candidate in candidates { + let Some(candidate) = candidate.as_object_mut() else { + continue; + }; + for key in ["nodeId", "node_id"] { + let Some(node_id) = candidate.get(key).and_then(Value::as_str) else { + continue; + }; + let remapped = remap_endpoint(node_id, endpoint_remap); + candidate.insert(key.to_owned(), Value::String(remapped)); + } + } + } + } + } +} + fn profile_internal(label: &str, started: &mut Instant) { if std::env::var_os("COMPASS_PROFILE_INTERNAL").is_some() { eprintln!( diff --git a/crates/compass-graph/src/v1.rs b/crates/compass-graph/src/v1.rs index 87ff7360e..9d672aee1 100644 --- a/crates/compass-graph/src/v1.rs +++ b/crates/compass-graph/src/v1.rs @@ -7,12 +7,14 @@ use ahash::{AHashMap as HashMap, AHashSet as HashSet}; use compass_languages::{Extraction, RawEdgeRecord, RawNodeRecord, Registry}; use compass_model::code_graph::{ BuildMetadata, CommunityMetadata, ConfigNodeDetails, CoverageRecord, CoverageStatus, - DatabaseNodeDetails, DiagnosticSeverity, EdgeDetails, EdgeKind, EdgeRecord, ExtractionStatus, - FileNodeDetails, FileRecord, GraphDiagnostic, GraphDocument, GraphMetadata, - ImportExportNodeDetails, MessagingNodeDetails, NodeDetails, NodeKind, NodeRecord, NodeRole, - QueryNodeDetails, RenderEdgeDetails, RenderKind, ResourceKind, ResourceNodeDetails, - RouteEdgeDetails, RouteNodeDetails, RouteStage, RouteStageDetails, SchemaNodeDetails, - SymbolNodeDetails, + DatabaseNodeDetails, DiagnosticSeverity, DocumentFormat, DocumentNodeDetails, + DocumentReferenceResolution, DocumentRole, DocumentSignificance, DocumentTableCellDetails, + DocumentTableColumnDetails, DocumentTableDetails, DocumentTableRowDetails, EdgeDetails, + EdgeKind, EdgeRecord, ExtractionStatus, FileNodeDetails, FileRecord, GraphDiagnostic, + GraphDocument, GraphMetadata, ImportExportNodeDetails, MessagingNodeDetails, NodeDetails, + NodeKind, NodeRecord, NodeRole, QueryNodeDetails, RenderEdgeDetails, RenderKind, ResourceKind, + ResourceNodeDetails, RouteEdgeDetails, RouteNodeDetails, RouteStage, RouteStageDetails, + SchemaNodeDetails, SymbolNodeDetails, TableAlignment, TableCellState, }; use compass_model::identity::{ database_entity_id, domain_id, edge_id, file_id, messaging_id, normalize_repository_path, @@ -35,8 +37,10 @@ use serde_json::{Map, Value}; use crate::inference::{InferenceLevel, prefilter_extraction_inference}; use crate::quarantine::{PublicationOutcome, QuarantineCollector}; -/// Version of the normalization and publication contract behind graph schema v1. +/// Normalization/publication semantics used by `compass.graph/1`. pub const V1_PUBLICATION_SEMANTICS_VERSION: &str = "compass.graph.publication/1"; +const MAX_DOCUMENT_REFERENCE_CANDIDATES: usize = 20; +const MAX_DOCUMENT_REFERENCE_PROBES: usize = 100_000; use sha2::{Digest, Sha256}; const TRUSTED_NODE_RECORD: &str = TRUSTED_NODE_RECORD_ATTRIBUTE; @@ -47,6 +51,9 @@ const CANONICAL_EXTERNAL_SYMBOL: &str = "_canonical_external_symbol"; const CANONICAL_RAW_ORDER: &str = "_compass_v1_canonical_raw_order"; const COALESCED_EDGE_EVIDENCE: &str = "_coalesced_edge_evidence"; const MAX_EXTERNAL_REFERENCE_DIAGNOSTICS: usize = 100; +const MAX_DOCUMENT_TEXT_BYTES: usize = 4 * 1024; +const MAX_DOCUMENT_TABLE_ITEMS: usize = 128; +const MAX_DOCUMENT_REFERENCES: usize = 128; #[derive(Clone, Copy, Debug, Eq, PartialEq)] enum PublicationMode { @@ -1057,6 +1064,293 @@ fn finalize_prepared_edge( } } +#[derive(Default)] +struct DocumentReferenceIndex { + paths: BTreeMap, + qualified_names: BTreeMap>, + names: BTreeMap>, +} + +fn document_reference_index( + nodes: &HashMap, + file_facts: &HashMap, +) -> DocumentReferenceIndex { + let mut index = DocumentReferenceIndex::default(); + for path in file_facts.keys() { + index.paths.insert(path.clone(), file_id(path)); + } + for node in nodes.values() { + if node.kind == NodeKind::File { + if let Some(source) = node.source.as_ref() { + index.paths.insert(source.file.clone(), node.id.clone()); + } + continue; + } + if node.kind == NodeKind::Resource { + continue; + } + index + .qualified_names + .entry(node.qualified_name.clone()) + .or_default() + .push(node.id.clone()); + index + .names + .entry(node.name.clone()) + .or_default() + .push(node.id.clone()); + } + for values in index + .qualified_names + .values_mut() + .chain(index.names.values_mut()) + { + values.sort(); + values.dedup(); + } + index +} + +fn document_reference_path(source_file: &str, spelling: &str) -> Option { + let before_fragment = spelling.split_once('#').map_or(spelling, |(path, _)| path); + let path = before_fragment + .split_once('?') + .map_or(before_fragment, |(path, _)| path) + .trim(); + if path.is_empty() + || (!path.contains('/') + && !path.starts_with('.') + && !path.starts_with('/') + && Path::new(path).extension().is_none()) + { + return None; + } + let normalized = if path.starts_with('/') { + normalize_repository_path(path.trim_start_matches('/')) + } else { + let parent = Path::new(source_file) + .parent() + .unwrap_or_else(|| Path::new("")); + normalize_repository_path(&parent.join(path).to_string_lossy()) + }; + Some(normalized) +} + +fn document_reference_candidates( + spelling: &str, + source_file: &str, + index: &DocumentReferenceIndex, +) -> Vec { + if let Some(path) = document_reference_path(source_file, spelling) { + let mut paths = vec![path.clone()]; + if Path::new(&path).extension().is_none() { + paths.push(format!("{path}.md")); + } + for path in paths { + if let Some(target) = index.paths.get(&path) { + return vec![target.clone()]; + } + } + } + if let Some(targets) = index.qualified_names.get(spelling) { + return targets.clone(); + } + index.names.get(spelling).cloned().unwrap_or_default() +} + +fn document_reference_edge( + source_raw: &str, + target_raw: &str, + relation: &str, + reference: &compass_model::code_graph::DocumentReferenceEvidence, +) -> RawEdgeRecord { + let mut attributes = Map::new(); + attributes.insert("relation".to_owned(), Value::String(relation.to_owned())); + attributes.insert( + "confidence".to_owned(), + Value::String("EXTRACTED".to_owned()), + ); + attributes.insert( + "source_file".to_owned(), + Value::String(reference.site.file.clone()), + ); + attributes.insert("_origin".to_owned(), Value::String("artifact".to_owned())); + attributes.insert( + "link_kind".to_owned(), + Value::String("inline_code".to_owned()), + ); + attributes.insert( + "start_byte".to_owned(), + Value::from(reference.site.start_byte), + ); + attributes.insert("end_byte".to_owned(), Value::from(reference.site.end_byte)); + attributes.insert( + "line_start".to_owned(), + Value::from(reference.site.start_line), + ); + attributes.insert("line_end".to_owned(), Value::from(reference.site.end_line)); + attributes.insert( + "column_start".to_owned(), + Value::from(reference.site.start_column), + ); + attributes.insert( + "column_end".to_owned(), + Value::from(reference.site.end_column), + ); + RawEdgeRecord { + source: source_raw.to_owned(), + target: target_raw.to_owned(), + attributes, + } +} + +/// Resolve document reference evidence after all raw nodes have acquired +/// their strict IDs. This keeps extraction conservative and lets a unique +/// code/path target be proven from the complete repository inventory. +fn resolve_document_references( + nodes: &mut HashMap, + id_remap: &HashMap, + file_facts: &HashMap, +) -> (Vec, Vec) { + let index = document_reference_index(nodes, file_facts); + let mut published_to_raw = BTreeMap::::new(); + for (raw, published) in id_remap { + published_to_raw + .entry(published.clone()) + .and_modify(|existing| { + if raw < existing { + existing.clone_from(raw); + } + }) + .or_insert_with(|| raw.clone()); + } + let mut probes = 0usize; + let mut synthetic_edges = Vec::new(); + let mut diagnostics = Vec::new(); + let node_ids = nodes.keys().cloned().collect::>(); + let node_kinds = nodes + .iter() + .map(|(id, node)| (id.clone(), node.kind)) + .collect::>(); + for node in nodes.values_mut() { + let owner_id = node.id.clone(); + let Some(NodeDetails::Document(details)) = node.details.as_mut() else { + continue; + }; + for reference in &mut details.references { + if let Some(target) = reference.target.as_ref() { + let remapped = id_remap.get(target).cloned().or_else(|| { + // The compatibility graph assembler may have rewritten + // an edge endpoint through a normalized alias without + // being able to rewrite the nested evidence payload. A + // path/qualified-name lookup is the same exact index used + // for inline-code resolution and repairs that transport + // alias without weakening the evidence to a guess. + let candidates = document_reference_candidates( + &reference.spelling, + &reference.site.file, + &index, + ) + .into_iter() + .filter(|candidate| node_ids.contains(candidate)) + .collect::>(); + (candidates.len() == 1).then(|| candidates[0].clone()) + }); + if let Some(remapped) = remapped { + reference.target = Some(remapped); + } else if !node_ids.contains(target) { + reference.target = None; + reference.resolution = DocumentReferenceResolution::Unresolved; + } + } + for candidate in &mut reference.candidates { + if let Some(published) = id_remap.get(&candidate.node_id) { + candidate.node_id.clone_from(published); + } + } + reference + .candidates + .retain(|candidate| node_ids.contains(&candidate.node_id)); + sort_dedup_candidates(&mut reference.candidates); + if reference.kind != "inline_code" + || reference.resolution != DocumentReferenceResolution::Unresolved + || reference.target.is_some() + || !reference.candidates.is_empty() + { + continue; + } + if probes >= MAX_DOCUMENT_REFERENCE_PROBES { + reference.resolution = DocumentReferenceResolution::Limited; + diagnostics.push(GraphDiagnostic { + severity: DiagnosticSeverity::Warning, + code: "document_reference_resolution_limited".to_owned(), + message: format!( + "document reference resolution stopped after {MAX_DOCUMENT_REFERENCE_PROBES} probes" + ), + anchor: Some(reference.site.clone()), + related_ids: vec![owner_id.clone()], + }); + continue; + } + probes = probes.saturating_add(1); + let targets = + document_reference_candidates(&reference.spelling, &reference.site.file, &index) + .into_iter() + .filter(|target| node_ids.contains(target)) + .collect::>(); + if targets.len() > MAX_DOCUMENT_REFERENCE_CANDIDATES { + reference.resolution = DocumentReferenceResolution::Limited; + diagnostics.push(GraphDiagnostic { + severity: DiagnosticSeverity::Warning, + code: "document_reference_candidates_limited".to_owned(), + message: format!( + "document reference {:?} has more than {MAX_DOCUMENT_REFERENCE_CANDIDATES} exact candidates", + reference.spelling + ), + anchor: Some(reference.site.clone()), + related_ids: vec![owner_id.clone()], + }); + continue; + } + match targets.as_slice() { + [target] => { + reference.resolution = DocumentReferenceResolution::Exact; + reference.target = Some(target.clone()); + if let (Some(source_raw), Some(target_raw)) = ( + published_to_raw.get(&owner_id), + published_to_raw.get(target), + ) { + let relation = if node_kinds.get(target) == Some(&NodeKind::File) { + "documents" + } else { + "references" + }; + synthetic_edges.push(document_reference_edge( + source_raw, target_raw, relation, reference, + )); + } + } + [] => {} + targets => { + reference.resolution = DocumentReferenceResolution::Ambiguous; + reference.candidates = targets + .iter() + .map(|target| ResolutionCandidate { + node_id: target.clone(), + reason: "multiple exact document reference candidates".to_owned(), + confidence: EvidenceConfidence::Ambiguous, + score: None, + anchor: None, + }) + .collect(); + sort_dedup_candidates(&mut reference.candidates); + } + } + } + } + (synthetic_edges, diagnostics) +} + fn normalize_v1_with_mode( mut extraction: Extraction, mut evidence: BuildEvidence, @@ -1214,6 +1508,7 @@ fn normalize_v1_with_mode( id_remap.insert(raw_id, published_id); } } + coalesce_external_placeholder_nodes(&mut nodes, &mut id_remap, mode)?; for node in nodes.values_mut() { remap_provenance_candidates(&mut node.evidence, &id_remap); let Some(NodeDetails::Route(details)) = node.details.as_mut() else { @@ -1233,6 +1528,10 @@ fn normalize_v1_with_mode( } recompute_route_resolution(details); } + let (document_reference_edges, document_reference_diagnostics) = + resolve_document_references(&mut nodes, &id_remap, &file_facts); + extraction.edges.extend(document_reference_edges); + evidence.diagnostics.extend(document_reference_diagnostics); profile_v1("v1 node normalization", &mut profile_started); // Edge publication consults endpoint kinds and unresolved placeholder @@ -1474,6 +1773,8 @@ fn normalize_v1_with_mode( file_id.clone_from(published); } } + ensure_external_placeholder_details(&mut nodes); + downgrade_document_details_for_graph_v1(&mut nodes); nodes.par_sort_unstable_by(|left, right| left.id.cmp(&right.id)); links.par_sort_unstable_by(|left, right| { left.id @@ -1528,24 +1829,195 @@ fn normalize_v1_with_mode( }) } +/// Document details are a bounded normalization IR used to resolve references +/// and derive stable identities. `compass.graph/1` keeps its established wire +/// shape: document nodes publish as `Resource(document)` and express Markdown +/// structure through source-backed nodes, meaningful names, qualified names, +/// containment, and reference edges. +fn downgrade_document_details_for_graph_v1(nodes: &mut [NodeRecord]) { + for node in nodes { + let details = match node.details.take() { + Some(NodeDetails::Document(details)) => details, + other => { + node.details = other; + continue; + } + }; + let media_type = match details.format { + DocumentFormat::Markdown => Some("text/markdown".to_owned()), + DocumentFormat::Html => Some("text/html".to_owned()), + DocumentFormat::Text => Some("text/plain".to_owned()), + DocumentFormat::Pdf => Some("application/pdf".to_owned()), + DocumentFormat::Docx => Some( + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + .to_owned(), + ), + DocumentFormat::Xlsx => { + Some("application/vnd.openxmlformats-officedocument.spreadsheetml.sheet".to_owned()) + } + DocumentFormat::Pptx => Some( + "application/vnd.openxmlformats-officedocument.presentationml.presentation" + .to_owned(), + ), + DocumentFormat::Rtf => Some("application/rtf".to_owned()), + DocumentFormat::Other => None, + }; + node.details = Some(NodeDetails::Resource(ResourceNodeDetails { + resource_kind: ResourceKind::Document, + uri: details.uri, + media_type, + })); + } +} + +/// Coalesce equivalent unresolved external symbols emitted by independent +/// producers. Framework extraction and universal semantic resolution can both +/// observe the same unresolved import at one exact source site, but their raw +/// IDs intentionally differ. Publishing both records creates duplicate +/// placeholders and violates the one-wiring-occurrence identity contract. +/// +/// The key includes every exact wiring site carried by the node, so symbols +/// used at different source occurrences (or in different scopes) remain +/// distinct. The lexicographically smallest existing ID is retained as the +/// canonical identity; all raw aliases are remapped before references and +/// edges are published. +fn coalesce_external_placeholder_nodes( + nodes: &mut HashMap, + id_remap: &mut HashMap, + mode: PublicationMode, +) -> Result<(), GraphError> { + let mut groups = BTreeMap::>::new(); + for node in nodes.values() { + if let Some(identity) = external_placeholder_identity(node) { + groups.entry(identity).or_default().push(node.id.clone()); + } + } + for ids in groups.values_mut() { + if ids.len() < 2 { + continue; + } + ids.sort(); + let canonical_id = ids[0].clone(); + let Some(mut canonical) = nodes.remove(&canonical_id) else { + continue; + }; + for duplicate_id in ids.iter().skip(1) { + let Some(duplicate) = nodes.remove(duplicate_id) else { + continue; + }; + if let Err(error) = merge_normalized_node(&mut canonical, duplicate) { + // The identity key is deliberately stricter than the merge + // contract. Preserve the normal strict/best-effort behavior + // if a producer nevertheless supplied incompatible details. + if mode == PublicationMode::Strict { + return Err(error); + } + continue; + } + for published in id_remap.values_mut() { + if published == duplicate_id { + published.clone_from(&canonical_id); + } + } + } + canonical.id = canonical_id.clone(); + nodes.insert(canonical_id, canonical); + } + Ok(()) +} + +#[derive(Clone, Debug, Eq, Ord, PartialEq, PartialOrd)] +struct ExternalPlaceholderIdentity { + kind: &'static str, + qualified_name: String, + language: Option, + framework: Option, + wiring_sites: Vec<(String, u64, u64)>, +} + +fn external_placeholder_identity(node: &NodeRecord) -> Option { + if node.source.is_some() { + return None; + } + let mut wiring_sites = node + .evidence + .iter() + .filter(|evidence| { + evidence.origin == EvidenceOrigin::Heuristic + && evidence.confidence == EvidenceConfidence::Inferred + && evidence.rule.as_deref() == Some("external-symbol-placeholder") + }) + .filter_map(|evidence| evidence.wiring_site.as_ref()) + .map(|site| (site.file.clone(), site.start_byte, site.end_byte)) + .collect::>(); + if wiring_sites.is_empty() { + return None; + } + wiring_sites.sort(); + wiring_sites.dedup(); + Some(ExternalPlaceholderIdentity { + kind: node.kind.as_str(), + qualified_name: node.qualified_name.clone(), + language: node.language.clone(), + framework: node.framework.clone(), + wiring_sites, + }) +} + +/// An unresolved external symbol is still a typed symbol occurrence. Some +/// producer paths carry no signature fields, so their otherwise-empty symbol +/// payload can disappear while facts are recomposed. Restore that closed +/// payload before graph-v1 publication instead of making consumers infer a +/// node category from a heuristic provenance record. +fn ensure_external_placeholder_details(nodes: &mut [NodeRecord]) { + for node in nodes { + if node.details.is_some() || external_placeholder_identity(node).is_none() { + continue; + } + node.details = Some(NodeDetails::Symbol(SymbolNodeDetails { + signature: None, + modifiers: Vec::new(), + overload_discriminator: None, + declaring_type: None, + signature_digest: None, + implementation_digest: None, + source_digest: None, + })); + } +} + fn normalize_trusted_node(value: Value, raw_id: &str) -> Result { let mut node = serde_json::from_value::(value) .map_err(|error| raw_error(raw_id, &error.to_string()))?; - // Trusted records already carry typed semantics. Markdown headings with a - // retained fragment URI have a hierarchical identity that survives source - // movement; other document resources remain positional occurrences so - // repeated blocks cannot quarantine one another. + // Trusted records already carry typed semantics. Recompute document IDs + // from the same semantic/occurrence rules used for raw normalization so a + // legacy producer's global document ID cannot collapse repeated blocks. if node.kind == NodeKind::Resource - && matches!( - node.details, + && let Some(site) = node.source.as_ref() + { + let semantic = match node.details.as_ref() { + Some(NodeDetails::Document(details)) => Some(matches!( + details.role, + DocumentRole::Document + | DocumentRole::Heading + | DocumentRole::Table + | DocumentRole::TableRow + )), Some(NodeDetails::Resource(ResourceNodeDetails { resource_kind: ResourceKind::Document, + uri, .. - })) - ) - && let Some(site) = node.source.as_ref() - { - node.id = if typed_markdown_heading(&node) { + })) => Some( + uri.as_deref().is_some_and(|value| value.starts_with('#')) + || graph_v1_table_qualified_name(&node.qualified_name) + || (site.start_byte == 0 && node.name == node.qualified_name), + ), + _ => None, + }; + let Some(semantic) = semantic else { + return Ok(node); + }; + node.id = if semantic { domain_id(NodeKind::Resource, &site.file, &node.qualified_name) } else { let positional_name = format!( @@ -1567,6 +2039,13 @@ fn normalize_trusted_node(value: Value, raw_id: &str) -> Result bool { + qualified_name.contains("::pipe_table#") + || qualified_name.contains("::pipe_table_header#") + || qualified_name.contains("::pipe_table_row#") + || qualified_name.contains("::pipe_table_cell#") +} + fn sanitize_document( document: &mut GraphDocument, quarantine: &mut QuarantineCollector, @@ -3218,6 +3697,7 @@ fn insert_raw_evidence(attributes: &mut Map, evidence: &Provenanc fn insert_raw_node_details(attributes: &mut Map, details: &NodeDetails) { match details { NodeDetails::File(_) | NodeDetails::Resource(_) => {} + NodeDetails::Document(details) => insert_raw_document_details(attributes, details), NodeDetails::Symbol(details) => { insert_optional_string(attributes, "signature", details.signature.as_ref()); insert_optional_string( @@ -3336,6 +3816,93 @@ fn insert_raw_node_details(attributes: &mut Map, details: &NodeDe } } +fn insert_raw_document_details(attributes: &mut Map, details: &DocumentNodeDetails) { + attributes.insert( + "document_format".to_owned(), + serde_json::to_value(details.format).unwrap_or(Value::Null), + ); + attributes.insert( + "document_kind".to_owned(), + serde_json::to_value(details.role).unwrap_or(Value::Null), + ); + attributes.insert("block_index".to_owned(), Value::from(details.ordinal)); + if let Some(section) = &details.section { + attributes.insert( + "document_section".to_owned(), + Value::String(section.clone()), + ); + } + if let Some(uri) = &details.uri { + attributes.insert("uri".to_owned(), Value::String(uri.clone())); + } + if let Some(content) = &details.content { + attributes.insert( + "document_content".to_owned(), + Value::String(content.clone()), + ); + } + attributes.insert( + "document_significance".to_owned(), + serde_json::to_value(details.significance).unwrap_or(Value::Null), + ); + if let Some(table) = &details.table { + attributes.insert( + "table_columns".to_owned(), + serde_json::to_value(&table.columns).unwrap_or(Value::Null), + ); + attributes.insert( + "table_headers".to_owned(), + Value::Array( + table + .columns + .iter() + .map(|column| Value::String(column.header.clone())) + .collect(), + ), + ); + attributes.insert( + "table_alignments".to_owned(), + Value::Array( + table + .columns + .iter() + .map(|column| serde_json::to_value(column.alignment).unwrap_or(Value::Null)) + .collect(), + ), + ); + attributes.insert( + "table_body_row_count".to_owned(), + Value::from(table.body_row_count), + ); + attributes.insert( + "table_omitted_row_count".to_owned(), + Value::from(table.omitted_row_count), + ); + attributes.insert( + "table_omitted_column_count".to_owned(), + Value::from(table.omitted_column_count), + ); + attributes.insert("table_truncated".to_owned(), Value::Bool(table.truncated)); + } + if let Some(row) = &details.table_row { + attributes.insert("table_row_index".to_owned(), Value::from(row.row_index)); + if let Some(index) = row.identity_cell_index { + attributes.insert("table_identity_cell_index".to_owned(), Value::from(index)); + } + attributes.insert( + "table_cells".to_owned(), + serde_json::to_value(&row.cells).unwrap_or(Value::Null), + ); + attributes.insert("table_truncated".to_owned(), Value::Bool(row.truncated)); + } + if !details.references.is_empty() { + attributes.insert( + "document_references".to_owned(), + serde_json::to_value(&details.references).unwrap_or(Value::Null), + ); + } +} + fn insert_raw_edge_details(attributes: &mut Map, details: &EdgeDetails) { match details { EdgeDetails::Call(details) => { @@ -4343,6 +4910,353 @@ fn map_edge_kind(raw: &str) -> Option<(EdgeKind, Option<&'static str>, bool)> { Some(mapped) } +fn document_format(attributes: &Map) -> DocumentFormat { + let value = optional_any_string(attributes, &["document_format", "language", "lang"]); + match value.as_deref().map(str::to_ascii_lowercase).as_deref() { + Some("markdown" | "md" | "mdx") => DocumentFormat::Markdown, + Some("html" | "htm") => DocumentFormat::Html, + Some("text" | "txt" | "plain") => DocumentFormat::Text, + Some("pdf") => DocumentFormat::Pdf, + Some("docx") => DocumentFormat::Docx, + Some("xlsx") => DocumentFormat::Xlsx, + Some("pptx") => DocumentFormat::Pptx, + Some("rtf") => DocumentFormat::Rtf, + _ => DocumentFormat::Other, + } +} + +fn document_role(attributes: &Map) -> DocumentRole { + match optional_string(attributes, "document_kind").as_deref() { + Some("document") => DocumentRole::Document, + Some("heading") => DocumentRole::Heading, + Some("paragraph") => DocumentRole::Paragraph, + Some("list") => DocumentRole::List, + Some("list_item") => DocumentRole::ListItem, + Some("block_quote" | "quote") => DocumentRole::Quote, + Some("fenced_code_block" | "indented_code_block" | "code") => DocumentRole::Code, + Some("thematic_break") => DocumentRole::ThematicBreak, + Some("table" | "pipe_table") => DocumentRole::Table, + Some("table_row" | "pipe_table_row") => DocumentRole::TableRow, + Some("link_reference_definition") => DocumentRole::LinkDefinition, + Some("footnote_definition") => DocumentRole::FootnoteDefinition, + Some("page") => DocumentRole::Page, + Some("sheet") => DocumentRole::Sheet, + Some("slide") => DocumentRole::Slide, + Some("note") => DocumentRole::Note, + _ => DocumentRole::Other, + } +} + +fn document_significance( + role: DocumentRole, + attributes: &Map, +) -> DocumentSignificance { + if let Some(value) = optional_string(attributes, "document_significance") { + return match value.as_str() { + "container" => DocumentSignificance::Container, + "scaffolding" => DocumentSignificance::Scaffolding, + _ => DocumentSignificance::Content, + }; + } + match role { + DocumentRole::List | DocumentRole::Quote | DocumentRole::Table => { + DocumentSignificance::Container + } + DocumentRole::LinkDefinition | DocumentRole::FootnoteDefinition => { + DocumentSignificance::Scaffolding + } + _ => DocumentSignificance::Content, + } +} + +fn document_alignment(value: Option<&Value>) -> TableAlignment { + match value.and_then(Value::as_str) { + Some("left") => TableAlignment::Left, + Some("center") => TableAlignment::Center, + Some("right") => TableAlignment::Right, + _ => TableAlignment::Unspecified, + } +} + +fn normalize_document_anchor( + value: &Value, + root: &Path, + file_facts: &HashMap, + record: &str, +) -> Result { + let mut anchor = structured_source_anchor(value) + .map_err(|_| raw_error(record, "invalid nested document source anchor"))?; + anchor.file = portable_path(&anchor.file, root)?; + if !anchor.is_valid() { + return Err(raw_error( + record, + "nested document source anchor is invalid", + )); + } + let Some(file) = file_facts.get(&anchor.file) else { + return Err(raw_error( + record, + "nested document source anchor references an unknown file", + )); + }; + if anchor.end_byte > file.byte_size { + return Err(raw_error( + record, + "nested document source anchor exceeds file bounds", + )); + } + Ok(anchor) +} + +fn document_table_columns( + attributes: &Map, + root: &Path, + file_facts: &HashMap, + record: &str, +) -> Result, GraphError> { + if let Some(Value::Array(values)) = attributes.get("table_columns") { + if values.len() > MAX_DOCUMENT_TABLE_ITEMS { + return Err(raw_error( + record, + "table_columns exceeds the bounded item limit", + )); + } + let mut columns = Vec::with_capacity(values.len()); + for value in values { + let Some(object) = value.as_object() else { + return Err(raw_error(record, "table_columns entries must be objects")); + }; + let index = object + .get("index") + .and_then(Value::as_u64) + .and_then(|value| u32::try_from(value).ok()) + .ok_or_else(|| raw_error(record, "table column index is invalid"))?; + let header = object + .get("header") + .and_then(Value::as_str) + .ok_or_else(|| raw_error(record, "table column header is invalid"))? + .to_owned(); + let source = object + .get("source") + .filter(|value| !value.is_null()) + .map(|value| normalize_document_anchor(value, root, file_facts, record)) + .transpose()?; + columns.push(DocumentTableColumnDetails { + index, + header, + alignment: document_alignment(object.get("alignment")), + source, + }); + } + return Ok(columns); + } + + let headers = attributes + .get("table_headers") + .and_then(Value::as_array) + .map(|values| { + values + .iter() + .filter_map(Value::as_str) + .map(str::to_owned) + .collect::>() + }) + .unwrap_or_default(); + if headers.len() > MAX_DOCUMENT_TABLE_ITEMS { + return Err(raw_error( + record, + "table_headers exceeds the bounded item limit", + )); + } + let alignments = attributes + .get("table_alignments") + .and_then(Value::as_array) + .map(|values| values.iter().collect::>()) + .unwrap_or_default(); + if alignments.len() > MAX_DOCUMENT_TABLE_ITEMS { + return Err(raw_error( + record, + "table_alignments exceeds the bounded item limit", + )); + } + let count = headers.len().max(alignments.len()); + Ok((0..count) + .map(|index| DocumentTableColumnDetails { + index: index as u32, + header: headers.get(index).cloned().unwrap_or_default(), + alignment: document_alignment(alignments.get(index).copied()), + source: None, + }) + .collect()) +} + +fn document_table_cells( + attributes: &Map, + root: &Path, + file_facts: &HashMap, + record: &str, +) -> Result, GraphError> { + let Some(Value::Array(values)) = attributes.get("table_cells") else { + return Ok(Vec::new()); + }; + if values.len() > MAX_DOCUMENT_TABLE_ITEMS { + return Err(raw_error( + record, + "table_cells exceeds the bounded item limit", + )); + } + let mut cells = Vec::with_capacity(values.len()); + for value in values { + let Some(object) = value.as_object() else { + return Err(raw_error(record, "table_cells entries must be objects")); + }; + let index = object + .get("columnIndex") + .or_else(|| object.get("column_index")) + .and_then(Value::as_u64) + .and_then(|value| u32::try_from(value).ok()) + .ok_or_else(|| raw_error(record, "table cell column index is invalid"))?; + let state = match object.get("state").and_then(Value::as_str) { + Some("present") => TableCellState::Present, + Some("empty") => TableCellState::Empty, + Some("missing") => TableCellState::Missing, + Some("limited") => TableCellState::Limited, + _ => return Err(raw_error(record, "table cell state is invalid")), + }; + let text = object + .get("text") + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(); + let source = object + .get("source") + .filter(|value| !value.is_null()) + .map(|value| normalize_document_anchor(value, root, file_facts, record)) + .transpose()?; + cells.push(DocumentTableCellDetails { + column_index: index, + state, + text, + source, + }); + } + Ok(cells) +} + +fn document_references( + attributes: &Map, + root: &Path, + file_facts: &HashMap, + record: &str, +) -> Result, GraphError> { + let Some(Value::Array(values)) = attributes.get("document_references") else { + return Ok(Vec::new()); + }; + if values.len() > MAX_DOCUMENT_REFERENCES { + return Err(raw_error( + record, + "document_references exceeds the bounded item limit", + )); + } + let mut references = Vec::with_capacity(values.len()); + for value in values { + let mut reference = serde_json::from_value::< + compass_model::code_graph::DocumentReferenceEvidence, + >(value.clone()) + .map_err(|error| raw_error(record, &format!("invalid document reference: {error}")))?; + if reference.spelling.len() > MAX_DOCUMENT_TEXT_BYTES { + return Err(raw_error( + record, + "document reference spelling exceeds the bounded limit", + )); + } + reference.site = normalize_document_anchor( + &serde_json::to_value(&reference.site) + .map_err(|error| raw_error(record, &error.to_string()))?, + root, + file_facts, + record, + )?; + for candidate in &mut reference.candidates { + if let Some(anchor) = candidate.anchor.take() { + candidate.anchor = Some(normalize_document_anchor( + &serde_json::to_value(anchor) + .map_err(|error| raw_error(record, &error.to_string()))?, + root, + file_facts, + record, + )?); + } + } + references.push(reference); + } + Ok(references) +} + +fn document_node_details( + attributes: &Map, + _source_path: &str, + file_facts: &HashMap, + record: &str, + root: &Path, +) -> Result { + let role = document_role(attributes); + let table = if role == DocumentRole::Table + || attributes.contains_key("table_columns") + || attributes.contains_key("table_headers") + { + Some(DocumentTableDetails { + columns: document_table_columns(attributes, root, file_facts, record)?, + body_row_count: optional_u32(attributes, "table_body_row_count").unwrap_or(0), + omitted_row_count: optional_u32(attributes, "table_omitted_row_count").unwrap_or(0), + omitted_column_count: optional_u32(attributes, "table_omitted_column_count") + .unwrap_or(0), + truncated: attributes + .get("table_truncated") + .and_then(Value::as_bool) + .unwrap_or(false), + }) + } else { + None + }; + let table_row = if role == DocumentRole::TableRow || attributes.contains_key("table_cells") { + Some(DocumentTableRowDetails { + row_index: optional_u32(attributes, "table_row_index").unwrap_or(0), + identity_cell_index: optional_u32(attributes, "table_identity_cell_index"), + cells: document_table_cells(attributes, root, file_facts, record)?, + truncated: attributes + .get("table_truncated") + .and_then(Value::as_bool) + .unwrap_or(false), + }) + } else { + None + }; + let content = optional_any_string(attributes, &["document_content", "content"]); + if content + .as_ref() + .is_some_and(|value| value.len() > MAX_DOCUMENT_TEXT_BYTES) + { + return Err(raw_error( + record, + "document content exceeds the bounded limit", + )); + } + Ok(DocumentNodeDetails { + format: document_format(attributes), + role, + ordinal: optional_u32(attributes, "block_index").unwrap_or(0), + section: optional_string(attributes, "document_section"), + uri: optional_string(attributes, "uri") + .or_else(|| optional_string(attributes, "anchor_slug").map(|slug| format!("#{slug}"))), + content, + significance: document_significance(role, attributes), + table, + table_row, + references: document_references(attributes, root, file_facts, record)?, + }) +} + fn node_details( kind: NodeKind, resource_kind: Option, @@ -4387,16 +5301,51 @@ fn node_details( middleware_count: optional_u32(attributes, "middleware_count").unwrap_or(0), stages: route_stage_details(attributes, record, root)?, })), - NodeKind::Resource => Some(NodeDetails::Resource(ResourceNodeDetails { - resource_kind: resource_kind.unwrap_or(ResourceKind::Document), - uri: optional_string(attributes, "uri").or_else(|| { - raw_markdown_heading(attributes) - .then(|| optional_string(attributes, "anchor_slug")) - .flatten() - .map(|slug| format!("#{slug}")) - }), - media_type: optional_string(attributes, "media_type"), - })), + NodeKind::Resource => { + // Framework domain resources are emitted through the generic + // `resource` node vocabulary but carry a more specific semantic + // discriminator in `resource_kind`. Do not let an unrecognised + // discriminator fall through to the historical + // `Resource(document)` payload. Source-backed document fragments + // temporarily use bounded normalization facts below, then publish + // through the established graph/1 resource shape. + // A file-set is still a resource in the closed graph vocabulary, + // so represent it as a concept resource while retaining the + // framework-specific discriminator in the flat compatibility + // attributes. + let resource_kind = resource_kind + .or_else( + || match optional_string(attributes, "resource_kind").as_deref() { + Some("paper") => Some(ResourceKind::Paper), + Some("image") => Some(ResourceKind::Image), + Some("concept" | "framework_file_set") => Some(ResourceKind::Concept), + Some("rationale") => Some(ResourceKind::Rationale), + Some("document") => Some(ResourceKind::Document), + _ => None, + }, + ) + .unwrap_or(ResourceKind::Document); + if resource_kind == ResourceKind::Document && attributes.contains_key("document_kind") { + Some(NodeDetails::Document(document_node_details( + attributes, + source_path, + file_facts, + record, + root, + )?)) + } else { + Some(NodeDetails::Resource(ResourceNodeDetails { + resource_kind, + uri: optional_string(attributes, "uri").or_else(|| { + raw_markdown_heading(attributes) + .then(|| optional_string(attributes, "anchor_slug")) + .flatten() + .map(|slug| format!("#{slug}")) + }), + media_type: optional_string(attributes, "media_type"), + })) + } + } NodeKind::Event | NodeKind::Message | NodeKind::Topic | NodeKind::Queue => { Some(NodeDetails::Messaging(MessagingNodeDetails { transport: required_string(attributes, "transport", record)?, @@ -4590,6 +5539,41 @@ fn node_identity( { domain_id(kind, source_path, qualified_name) } + NodeKind::Resource + if matches!( + details, + Some(NodeDetails::Document(DocumentNodeDetails { + format: DocumentFormat::Markdown, + role: DocumentRole::Heading, + .. + })) + ) => + { + domain_id(kind, source_path, qualified_name) + } + NodeKind::Resource if matches!(details, Some(NodeDetails::Document(_))) => { + let semantic = matches!( + details, + Some(NodeDetails::Document(DocumentNodeDetails { + role: DocumentRole::Document | DocumentRole::Table | DocumentRole::TableRow, + .. + })) + ) || raw_markdown_table_structure(attributes); + if semantic { + // Table, header, row, and cell qualified names are derived + // from normalized headers, identity cells, and column indexes, + // so their IDs survive line shifts and non-identity edits. + domain_id(kind, source_path, qualified_name) + } else { + // Other blocks are occurrences. Keep the exact source range + // to disambiguate repeated prose and parser recovery nodes. + let positional_name = identity_site.map_or_else( + || qualified_name.to_owned(), + |site| format!("{qualified_name}@{}:{}", site.start_byte, site.end_byte), + ); + domain_id(kind, source_path, &positional_name) + } + } NodeKind::Resource if matches!( details, @@ -4674,20 +5658,20 @@ fn node_identity_source( if !source_path.is_empty() { return source_path.to_owned(); } - // External placeholders are occurrences of one unresolved wiring site. - // A route candidate and the corresponding unresolved import can arrive - // with different transient package/module scopes, but they still denote - // the same source expression when their language, qualified name, and - // exact anchor agree. Prefer that anchor for identity so normalization - // coalesces the compatible records instead of publishing duplicate - // placeholders that manufacture ambiguity. + // External placeholders are scoped by the deterministic source context + // gathered during placeholder splitting. All references in one source + // scope should coalesce even when they occur at different wiring sites; + // the scope still keeps same-named placeholders in different files or + // language families distinct. If no scope was derived, fall back to the + // exact wiring site so an unscoped occurrence cannot merge by accident. if attributes .get("rule") .and_then(Value::as_str) .is_some_and(|rule| rule == "external-symbol-placeholder") - && let Some(site) = identity_site + && let Some(scope) = optional_string(attributes, "external_identity_scope") + && !scope.trim().is_empty() { - return format!("{}#{}:{}", site.file, site.start_byte, site.end_byte); + return scope; } optional_string(attributes, "external_identity_scope") .filter(|scope| !scope.trim().is_empty()) @@ -4708,16 +5692,12 @@ fn raw_markdown_heading(attributes: &Map) -> bool { ) } -fn typed_markdown_heading(node: &NodeRecord) -> bool { - node.language.as_deref() == Some("markdown") - && matches!( - node.details.as_ref(), - Some(NodeDetails::Resource(ResourceNodeDetails { - resource_kind: ResourceKind::Document, - uri: Some(uri), - .. - })) if uri.starts_with('#') - ) +fn raw_markdown_table_structure(attributes: &Map) -> bool { + matches!( + optional_string(attributes, "document_kind").as_deref(), + Some("pipe_table" | "pipe_table_header" | "pipe_table_row" | "pipe_table_cell") + ) && optional_string(attributes, "qualified_name") + .is_some_and(|name| name.contains("::pipe_table")) } fn raw_anchor( diff --git a/crates/compass-graph/tests/domain_normalization.rs b/crates/compass-graph/tests/domain_normalization.rs index 61a5cc7ce..229876d23 100644 --- a/crates/compass-graph/tests/domain_normalization.rs +++ b/crates/compass-graph/tests/domain_normalization.rs @@ -170,6 +170,118 @@ fn json_config_keys_publish_with_config_provenance_and_stable_paths() Ok(()) } +#[test] +fn markdown_frontmatter_publishes_nested_config_graph_without_changing_graph_v1() +-> Result<(), Box> { + let directory = tempfile::tempdir()?; + let root = directory.path(); + let relative = Path::new("docs/guide.md"); + let source = br#"--- +title: Graph Guide +tags: [markdown, compass] +api_token: do-not-publish +authors: + - name: Ada + role: editor +site: + navigation: + label: Guide +--- +# Body +"#; + fs::create_dir_all(root.join("docs"))?; + fs::write(root.join(relative), source)?; + + let extraction = Engine::default().extract_source(relative, source)?; + let evidence = BuildEvidence::from_extraction(root, &extraction, "sha256:test-config")?; + let graph = normalize_v1(extraction, evidence)?; + assert_eq!(graph.graph.schema, "compass.graph/1"); + assert!(graph.nodes.iter().all(|node| { + !matches!( + node.details, + Some(compass_model::code_graph::NodeDetails::Document(_)) + ) + })); + + let document = graph + .nodes + .iter() + .find(|node| node.qualified_name == "docs/guide.md") + .ok_or("missing Markdown document")?; + assert_eq!(document.name, "Graph Guide"); + let keys = graph + .nodes + .iter() + .filter(|node| node.kind == NodeKind::ConfigKey) + .collect::>(); + for expected in [ + "frontmatter/title", + "frontmatter/tags", + "frontmatter/api_token", + "frontmatter/authors", + "frontmatter/authors/0", + "frontmatter/authors/0/name", + "frontmatter/authors/0/role", + "frontmatter/site", + "frontmatter/site/navigation", + "frontmatter/site/navigation/label", + ] { + assert!( + keys.iter().any(|node| node.qualified_name == expected), + "missing {expected}: {keys:#?}" + ); + } + assert!(keys.iter().all(|node| { + node.source.is_some() + && node.evidence[0].origin == EvidenceOrigin::Config + && matches!( + node.details, + Some(compass_model::code_graph::NodeDetails::Config( + ref details + )) if details.format == "yaml_frontmatter" + && details.key_path.starts_with('/') + ) + })); + let token = keys + .iter() + .find(|node| node.qualified_name == "frontmatter/api_token") + .ok_or("missing token key")?; + assert_eq!(token.name, "api_token"); + assert!(!serde_json::to_string(&graph)?.contains("do-not-publish")); + let author = keys + .iter() + .find(|node| node.qualified_name == "frontmatter/authors/0") + .ok_or("missing author item")?; + let author_name = keys + .iter() + .find(|node| node.qualified_name == "frontmatter/authors/0/name") + .ok_or("missing author name")?; + assert!(graph.links.iter().any(|edge| { + edge.source == author.id + && edge.target == author_name.id + && edge.kind == EdgeKind::Contains + && edge.evidence[0].origin == EvidenceOrigin::Config + })); + + let stable_ids = keys + .iter() + .map(|node| (node.qualified_name.clone(), node.id.clone())) + .collect::>(); + let changed = String::from_utf8(source.to_vec())?.replace("Graph Guide", "Compass Graph Guide"); + fs::write(root.join(relative), &changed)?; + let extraction = Engine::default().extract_source(relative, changed.as_bytes())?; + let evidence = BuildEvidence::from_extraction(root, &extraction, "sha256:test-config")?; + let changed_graph = normalize_v1(extraction, evidence)?; + let changed_ids = changed_graph + .nodes + .iter() + .filter(|node| node.kind == NodeKind::ConfigKey) + .map(|node| (node.qualified_name.clone(), node.id.clone())) + .collect::>(); + assert_eq!(changed_ids, stable_ids); + Ok(()) +} + #[test] fn package_manifests_publish_dependency_endpoints_instead_of_dangling_edges() -> Result<(), Box> { diff --git a/crates/compass-graph/tests/graph_v1_normalization.rs b/crates/compass-graph/tests/graph_v1_normalization.rs index 00b09163e..5209683ce 100644 --- a/crates/compass-graph/tests/graph_v1_normalization.rs +++ b/crates/compass-graph/tests/graph_v1_normalization.rs @@ -12,7 +12,7 @@ use compass_graph::{ use compass_languages::{Extraction, RawEdgeRecord, RawNodeRecord}; use compass_model::code_graph::{ BuildMetadata, CoverageRecord, CoverageStatus, DiagnosticSeverity, EdgeKind, ExtractionStatus, - FileRecord, GraphDiagnostic, NodeKind, + FileRecord, GraphDiagnostic, NodeDetails, NodeKind, ResourceKind, }; use compass_model::identity::edge_id; use compass_model::provenance::{ @@ -146,6 +146,45 @@ fn file_nodes_canonicalize_extension_preserving_identity_before_coalescing() Ok(()) } +#[test] +fn framework_file_set_resources_do_not_fall_back_to_document_details() +-> Result<(), Box> { + let directory = tempfile::tempdir()?; + let root = directory.path(); + let mut file_set = raw_node(root, "raw:file-set", "import.meta.glob", 10); + file_set + .attributes + .insert("symbol_kind".to_owned(), json!("resource")); + file_set + .attributes + .insert("qualified_name".to_owned(), json!("framework_file_set")); + file_set + .attributes + .insert("resource_kind".to_owned(), json!("framework_file_set")); + file_set + .attributes + .insert("component_type".to_owned(), json!("framework_file_set")); + file_set + .attributes + .insert("framework".to_owned(), json!("vite")); + + let outcome = normalize_v1( + Extraction { + nodes: vec![file_set], + ..Extraction::default() + }, + build_evidence(root)?, + )?; + + assert_eq!(outcome.nodes.len(), 1); + assert!(matches!( + &outcome.nodes[0].details, + Some(NodeDetails::Resource(details)) + if details.resource_kind == ResourceKind::Concept + )); + Ok(()) +} + #[test] fn node_navigation_extent_preserves_and_contains_exact_provenance() -> Result<(), Box> { @@ -2347,6 +2386,11 @@ fn canonical_external_exact_binding_is_published_as_inferred_placeholder() .iter() .find(|node| node.qualified_name == "flask.Blueprint") .ok_or("missing canonical external placeholder")?; + assert!( + matches!(external.details.as_ref(), Some(NodeDetails::Symbol(_))), + "unexpected placeholder details: {:?}", + external.details + ); assert!(external.evidence.iter().any(|evidence| { evidence.extractor == "compass.graph.external-placeholder" && evidence.origin == EvidenceOrigin::Heuristic diff --git a/crates/compass-graph/tests/markdown_identity.rs b/crates/compass-graph/tests/markdown_identity.rs index 128522370..d9b7f76f6 100644 --- a/crates/compass-graph/tests/markdown_identity.rs +++ b/crates/compass-graph/tests/markdown_identity.rs @@ -2,9 +2,10 @@ use std::collections::BTreeMap; use std::error::Error; use std::fs; -use compass_graph::{build_from_extraction, normalize_document_v1}; +use compass_graph::{RawNodeRecord, build_from_extraction, normalize_document_v1}; use compass_languages::Engine; -use compass_model::code_graph::{NodeDetails, NodeKind}; +use compass_model::code_graph::{NodeDetails, NodeKind, ResourceKind}; +use serde_json::{Map, json}; #[test] fn repeated_markdown_headings_use_stable_hierarchical_identities() -> Result<(), Box> { @@ -32,7 +33,10 @@ Second problem. .filter(|node| node.kind == NodeKind::Resource && node.name == "Problem") .map(|node| { let uri = match node.details.as_ref() { - Some(NodeDetails::Resource(resource)) => resource.uri.clone(), + Some(NodeDetails::Resource(resource)) => { + assert_eq!(resource.resource_kind, ResourceKind::Document); + resource.uri.clone() + } _ => None, }; (node.qualified_name.clone(), uri) @@ -70,3 +74,342 @@ Second problem. assert_eq!(after, before); Ok(()) } + +#[test] +fn markdown_tables_keep_structural_nodes_and_stable_semantic_identities() +-> Result<(), Box> { + let directory = tempfile::tempdir()?; + let root = directory.path(); + let path = root.join("docs/ownership.md"); + fs::create_dir_all(path.parent().ok_or("missing source parent")?)?; + let source = r#"# Ownership +| Area | Owner | Status | +| :--- | :---: | ---: | +| Graph | `compass-model` | active | +| Empty | | values | +| Missing | +"#; + fs::write(&path, source)?; + + let graph_for = + |contents: &str| -> Result> { + fs::write(&path, contents)?; + let extraction = Engine::default().extract(&path)?; + let flexible = build_from_extraction(&extraction, true, Some(root)); + Ok(normalize_document_v1(&flexible, root, "sha256:test", None)?) + }; + + let graph = graph_for(source)?; + assert_eq!(graph.graph.schema, "compass.graph/1"); + assert!( + graph + .nodes + .iter() + .all(|node| { !matches!(node.details.as_ref(), Some(NodeDetails::Document(_))) }) + ); + let table = graph + .nodes + .iter() + .find(|node| { + node.qualified_name + .rsplit("::") + .next() + .is_some_and(|part| part.starts_with("pipe_table#")) + }) + .ok_or("missing markdown table")?; + assert!(table.name.contains("Area | Owner | Status")); + let mut rows = graph + .nodes + .iter() + .filter(|node| { + node.qualified_name + .rsplit("::") + .next() + .is_some_and(|part| part.starts_with("pipe_table_row#")) + }) + .collect::>(); + rows.sort_by_key(|node| node.source.as_ref().map(|source| source.start_byte)); + assert_eq!(rows.len(), 3); + assert_eq!( + rows[0].name, + "Area=Graph · Owner=`compass-model` · Status=active" + ); + assert_eq!(rows[1].name, "Area=Empty · Status=values"); + assert_eq!(rows[2].name, "Area=Missing"); + let cells = graph + .nodes + .iter() + .filter(|node| node.qualified_name.contains("::pipe_table_cell#")) + .collect::>(); + assert_eq!(cells.len(), 10); + assert!(cells.iter().any(|node| node.name == "Owner: (empty)")); + assert!(graph.links.iter().any(|edge| { + edge.kind == compass_model::code_graph::EdgeKind::Contains + && edge.source == table.id + && rows.iter().any(|node| edge.target == node.id) + })); + + let legacy = graph.to_legacy_document()?; + let legacy_table = legacy + .nodes + .iter() + .find(|node| node.id == table.id) + .ok_or("legacy table projection lost table")?; + assert_eq!(legacy_table.property("file_type"), Some(json!("document"))); + assert_eq!(legacy_table.document_role(), Some("pipe_table")); + let legacy_row = legacy + .nodes + .iter() + .find(|node| node.id == rows[0].id) + .ok_or("legacy row projection lost document role")?; + assert_eq!(legacy_row.document_role(), Some("pipe_table_row")); + let round_tripped = compass_model::code_graph::GraphDocument::from_legacy_document(legacy)?; + assert_eq!(round_tripped, graph); + + let before = rows + .iter() + .map(|node| (node.qualified_name.clone(), node.id.clone())) + .collect::>(); + let shifted = format!("Introductory prose.\n\n{source}").replace("active", "planned"); + let after_graph = graph_for(&shifted)?; + let after = after_graph + .nodes + .iter() + .filter(|node| { + node.qualified_name + .rsplit("::") + .next() + .is_some_and(|part| part.starts_with("pipe_table_row#")) + }) + .map(|node| (node.qualified_name.clone(), node.id.clone())) + .collect::>(); + assert_eq!(after, before); + Ok(()) +} + +#[test] +fn markdown_document_references_resolve_only_unique_exact_targets() -> Result<(), Box> { + let directory = tempfile::tempdir()?; + let root = directory.path(); + fs::create_dir_all(root.join("docs"))?; + fs::create_dir_all(root.join("src"))?; + let markdown = root.join("docs/guide.md"); + let rust = root.join("src/lib.rs"); + fs::write( + &markdown, + "# Ownership\n\n| Name | Target |\n| --- | --- |\n| Widget | `Widget` |\n| Source | `../src/lib.rs` |\n| Missing | `NoSuchThing` |\n", + )?; + fs::write(&rust, "pub struct Widget;\n")?; + + let mut engine = Engine::default(); + let markdown_extraction = engine.extract(&markdown)?; + let mut combined = markdown_extraction; + combined.nodes.extend([ + RawNodeRecord { + id: "raw-file".to_owned(), + attributes: Map::from_iter([ + ("label".to_owned(), json!("lib.rs")), + ("qualified_name".to_owned(), json!("src/lib.rs")), + ("symbol_kind".to_owned(), json!("file")), + ("file_type".to_owned(), json!("code")), + ("language".to_owned(), json!("rust")), + ("extractor".to_owned(), json!("test.rust")), + ("source_file".to_owned(), json!("src/lib.rs")), + ( + "source_anchor".to_owned(), + json!({"file":"src/lib.rs","startByte":0,"endByte":19,"startLine":1,"startColumn":0,"endLine":2,"endColumn":0}), + ), + ]), + }, + RawNodeRecord { + id: "raw-widget".to_owned(), + attributes: Map::from_iter([ + ("label".to_owned(), json!("Widget")), + ("qualified_name".to_owned(), json!("crate::Widget")), + ("symbol_kind".to_owned(), json!("struct")), + ("file_type".to_owned(), json!("code")), + ("language".to_owned(), json!("rust")), + ("extractor".to_owned(), json!("test.rust")), + ("source_file".to_owned(), json!("src/lib.rs")), + ( + "source_anchor".to_owned(), + json!({"file":"src/lib.rs","startByte":11,"endByte":17,"startLine":1,"startColumn":11,"endLine":1,"endColumn":17}), + ), + ]), + }, + ]); + let document = build_from_extraction(&combined, true, Some(root)); + let graph = normalize_document_v1(&document, root, "sha256:test", None)?; + + let widget_target = graph + .nodes + .iter() + .find(|node| node.kind == NodeKind::Struct && node.name == "Widget") + .ok_or("missing Widget target")?; + let file_target = graph + .nodes + .iter() + .find(|node| node.kind == NodeKind::File && node.qualified_name == "src/lib.rs") + .ok_or("missing file target")?; + let widget_owner = graph + .nodes + .iter() + .find(|node| node.name == "Target: `Widget`") + .ok_or("missing Widget cell")?; + let path_owner = graph + .nodes + .iter() + .find(|node| node.name == "Target: `../src/lib.rs`") + .ok_or("missing path cell")?; + let missing_owner = graph + .nodes + .iter() + .find(|node| node.name == "Target: `NoSuchThing`") + .ok_or("missing unresolved cell")?; + assert!(graph.links.iter().any(|edge| { + edge.kind == compass_model::code_graph::EdgeKind::References + && edge.source == widget_owner.id + && edge.target == widget_target.id + && edge.relationship_site.is_some() + })); + assert!(graph.links.iter().any(|edge| { + edge.kind == compass_model::code_graph::EdgeKind::Documents + && edge.source == path_owner.id + && edge.target == file_target.id + && edge.relationship_site.is_some() + })); + assert!( + !graph + .links + .iter() + .any(|edge| edge.source == missing_owner.id) + ); + Ok(()) +} + +#[test] +fn markdown_document_reference_ambiguity_and_limits_never_guess() -> Result<(), Box> { + let directory = tempfile::tempdir()?; + let root = directory.path(); + fs::create_dir_all(root.join("docs"))?; + fs::create_dir_all(root.join("src"))?; + let markdown = root.join("docs/guide.md"); + let source_path = root.join("src/lib.rs"); + fs::write(&markdown, "# Guide\n\nSee `Duplicate`.\n")?; + fs::write(&source_path, "pub struct Duplicate;\n")?; + + let mut extraction = Engine::default().extract(&markdown)?; + extraction.nodes.push(RawNodeRecord { + id: "raw-file".to_owned(), + attributes: Map::from_iter([ + ("label".to_owned(), json!("lib.rs")), + ("qualified_name".to_owned(), json!("src/lib.rs")), + ("symbol_kind".to_owned(), json!("file")), + ("file_type".to_owned(), json!("code")), + ("language".to_owned(), json!("rust")), + ("extractor".to_owned(), json!("test.rust")), + ("source_file".to_owned(), json!("src/lib.rs")), + ( + "source_anchor".to_owned(), + json!({"file":"src/lib.rs","startByte":0,"endByte":22,"startLine":1,"startColumn":0,"endLine":2,"endColumn":0}), + ), + ]), + }); + for index in 0..21 { + extraction.nodes.push(RawNodeRecord { + id: format!("raw-duplicate-{index}"), + attributes: Map::from_iter([ + ("label".to_owned(), json!("Duplicate")), + ( + "qualified_name".to_owned(), + json!(format!("crate::Duplicate{index}")), + ), + ("symbol_kind".to_owned(), json!("struct")), + ("file_type".to_owned(), json!("code")), + ("language".to_owned(), json!("rust")), + ("extractor".to_owned(), json!("test.rust")), + ("source_file".to_owned(), json!("src/lib.rs")), + ( + "source_anchor".to_owned(), + json!({"file":"src/lib.rs","startByte":11,"endByte":20,"startLine":1,"startColumn":11,"endLine":1,"endColumn":20}), + ), + ]), + }); + } + let document = build_from_extraction(&extraction, true, Some(root)); + let graph = normalize_document_v1(&document, root, "sha256:test", None)?; + let owner = graph + .nodes + .iter() + .find(|node| { + node.language.as_deref() == Some("markdown") && node.name.contains("Duplicate") + }) + .ok_or("missing Duplicate owner")?; + assert!(!graph.links.iter().any(|edge| edge.source == owner.id + && edge.kind == compass_model::code_graph::EdgeKind::References)); + assert!(graph.graph.diagnostics.iter().any(|diagnostic| { + diagnostic.code == "document_reference_candidates_limited" + && diagnostic.related_ids.contains(&owner.id) + && diagnostic.anchor.is_some() + })); + Ok(()) +} + +#[test] +fn markdown_explicit_link_evidence_survives_absolute_path_aliases() -> Result<(), Box> { + let directory = tempfile::tempdir()?; + let root = directory.path(); + fs::create_dir_all(root.join("docs"))?; + fs::create_dir_all(root.join("src"))?; + let markdown = root.join("docs/guide.md"); + let source = root.join("src/lib.rs"); + fs::write(&markdown, "# Guide\n\n[implementation](../src/lib.rs)\n")?; + fs::write(&source, "pub struct Widget;\n")?; + + // Engine::extract uses absolute source paths, while the compatibility + // assembler canonicalizes endpoints relative to `root`. The nested + // document evidence must follow that same remap and remain exact. + let mut engine = Engine::default(); + let markdown_extraction = engine.extract(&markdown)?; + let mut extraction = markdown_extraction; + extraction.nodes.push(RawNodeRecord { + id: compass_languages::make_id(&[&source.to_string_lossy()]), + attributes: Map::from_iter([ + ("label".to_owned(), json!("lib.rs")), + ("qualified_name".to_owned(), json!(source.to_string_lossy())), + ("symbol_kind".to_owned(), json!("file")), + ("file_type".to_owned(), json!("code")), + ("language".to_owned(), json!("rust")), + ("extractor".to_owned(), json!("test.rust")), + ("source_file".to_owned(), json!(source.to_string_lossy())), + ( + "source_anchor".to_owned(), + json!({"file": source.to_string_lossy(), "startByte": 0, "endByte": 19, "startLine": 1, "startColumn": 0, "endLine": 2, "endColumn": 0}), + ), + ]), + }); + let document = build_from_extraction(&extraction, true, Some(root)); + let graph = normalize_document_v1(&document, root, "sha256:test", None)?; + let owner = graph + .nodes + .iter() + .find(|node| node.name.contains("implementation")) + .ok_or("missing explicit link owner")?; + let edge = graph + .links + .iter() + .find(|edge| { + edge.source == owner.id && edge.kind == compass_model::code_graph::EdgeKind::Documents + }) + .ok_or("missing exact document edge")?; + assert_eq!( + graph + .nodes + .iter() + .find(|node| node.id == edge.target) + .map(|node| node.kind), + Some(NodeKind::File) + ); + assert!(edge.relationship_site.is_some()); + Ok(()) +} diff --git a/crates/compass-graph/tests/store_snapshot.rs b/crates/compass-graph/tests/store_snapshot.rs index 2a04a69f7..347be5c5a 100644 --- a/crates/compass-graph/tests/store_snapshot.rs +++ b/crates/compass-graph/tests/store_snapshot.rs @@ -8,7 +8,8 @@ use compass_graph::{ }; use compass_model::code_graph::{ BuildMetadata, CommunityMetadata, EdgeKind, EdgeRecord, ExtractionStatus, FileNodeDetails, - FileRecord, GraphDocument, NodeDetails, NodeKind, NodeRecord, + FileRecord, GraphDocument, NodeDetails, NodeKind, NodeRecord, ResourceKind, + ResourceNodeDetails, }; use compass_model::identity::{edge_id, file_id}; use compass_model::provenance::{ @@ -358,6 +359,51 @@ fn nodes_for_terms_matches_diacritic_normalized_queries() -> Result<(), Box Result<(), Box> { + let store = MemoryStore::default(); + let builder = GraphSnapshotBuilder::new(); + let mut document = graph(); + let mut table = node("table"); + table.kind = NodeKind::Resource; + table.name = "Owners — Owner | Status".to_owned(); + table.qualified_name = "Operations::pipe_table#1".to_owned(); + table.language = Some("markdown".to_owned()); + table.details = Some(NodeDetails::Resource(ResourceNodeDetails { + resource_kind: ResourceKind::Document, + uri: Some("#owners".to_owned()), + media_type: Some("text/markdown".to_owned()), + })); + + let mut row = node("table-row"); + row.kind = NodeKind::Resource; + row.name = "Owner=platform-team · Status=deploy".to_owned(); + row.qualified_name = "Operations::pipe_table#1::pipe_table_row#platform-team-1".to_owned(); + row.language = Some("markdown".to_owned()); + row.details = Some(NodeDetails::Resource(ResourceNodeDetails { + resource_kind: ResourceKind::Document, + uri: None, + media_type: Some("text/markdown".to_owned()), + })); + document.nodes.extend([table, row]); + + let prepared = builder.prepare(&store, &document)?; + builder.activate(&store, &prepared)?; + let reader = GraphSnapshotReader::open_active(&store)?.ok_or("active snapshot missing")?; + + for term in ["owner", "platform", "deploy"] { + let (nodes, truncated) = reader.nodes_for_terms(&[term.to_owned()], limits(128))?; + assert!(!truncated, "{term}"); + assert!( + nodes + .iter() + .any(|node| node.id == "table" || node.id == "table-row"), + "term {term} was not indexed in the immutable snapshot" + ); + } + Ok(()) +} + #[test] fn identifier_subword_capability_is_found_in_a_multilevel_terms_tree() -> Result<(), Box> { diff --git a/crates/compass-languages/src/lib.rs b/crates/compass-languages/src/lib.rs index cb0c97c9f..43cf30947 100644 --- a/crates/compass-languages/src/lib.rs +++ b/crates/compass-languages/src/lib.rs @@ -17,7 +17,7 @@ mod fortran; pub mod frameworks; /// Version of the extraction contract consumed by graph publication. -pub const EXTRACTION_SEMANTICS_VERSION: &str = "compass.languages.extraction/3"; +pub const EXTRACTION_SEMANTICS_VERSION: &str = "compass.languages.extraction/4"; mod go; mod html; mod ids; diff --git a/crates/compass-languages/src/markdown.rs b/crates/compass-languages/src/markdown.rs index f5d99395e..833cbbbbb 100644 --- a/crates/compass-languages/src/markdown.rs +++ b/crates/compass-languages/src/markdown.rs @@ -5,6 +5,7 @@ use crate::facts::stamp_source_range; use crate::{RawEdgeRecord as EdgeRecord, RawNodeRecord as NodeRecord}; use serde_json::{Map, Value, json}; use tree_sitter::{Node, Parser}; +use tree_sitter_language_pack::{DataNode, DataNodeKind, ProcessConfig}; use tree_sitter_md::{INLINE_LANGUAGE, LANGUAGE}; const FRONTMATTER_MAX_BYTES: usize = 64 * 1024; @@ -14,7 +15,19 @@ const MAX_DIAGNOSTICS: usize = 256; const MAX_METADATA_KEYS: usize = 256; const MAX_METADATA_STRING_BYTES: usize = 16 * 1024; const MAX_METADATA_ARRAY_ITEMS: usize = 256; +const MAX_METADATA_DEPTH: usize = 12; +const MAX_METADATA_GRAPH_NODES: usize = 512; const MAX_LABEL_CHARS: usize = 512; +const MAX_TABLE_COLUMNS: usize = 128; +const MAX_TABLE_ROWS: usize = 10_000; +const MAX_TABLE_CELL_BYTES: usize = 4 * 1024; +const MAX_TABLE_TEXT_BYTES: usize = 4 * 1024 * 1024; +const MAX_TABLE_CELLS: usize = 100_000; +const MAX_TABLE_NODES_PER_TABLE: usize = 20_000; +const MAX_TABLE_CELLS_PER_TABLE: usize = 16_384; +const MAX_TABLE_TEXT_BYTES_PER_TABLE: usize = 512 * 1024; +const MAX_DOCUMENT_REFERENCES: usize = 128; +const MAX_REFERENCE_SPELLING_BYTES: usize = 4 * 1024; /// Extract Markdown from bytes supplied by the caller. /// @@ -47,9 +60,10 @@ pub(crate) fn extract_source( detail: error.to_string(), })?; - let (metadata, frontmatter_diagnostic) = parse_frontmatter(source); + let (frontmatter, frontmatter_diagnostic) = parse_frontmatter(source); let stem = crate::file_stem(path); let file_id = crate::make_id(&[source_file]); + let line_starts = newline_offsets(source); let mut state = State { path, source, @@ -70,13 +84,28 @@ pub(crate) fn extract_source( pending_links: Vec::new(), unresolved_links: Vec::new(), external_links: Vec::new(), + document_references: HashMap::new(), diagnostics: Vec::new(), inline_parser, + line_starts, next_block_index: 1, other_count: 0, + table_occurrences: HashMap::new(), + table_cells_retained: 0, + table_text_bytes: 0, + table_limit_diagnostics: HashSet::new(), + document_reference_limit_reported: false, }; - state.add_root(file_id, metadata); + state.add_root( + file_id, + frontmatter + .as_ref() + .map(|frontmatter| frontmatter.metadata.clone()), + ); + if let Some(frontmatter) = frontmatter { + state.add_frontmatter_nodes(frontmatter.facts); + } if let Some(diagnostic) = frontmatter_diagnostic { state.add_diagnostic(diagnostic); } @@ -106,6 +135,7 @@ pub(crate) fn extract_source( state.scan_footnotes(); state.scan_other_constructs(); state.finalize_links(); + state.publish_document_references(); state .extraction @@ -175,10 +205,17 @@ struct State<'source, 'path> { pending_links: Vec, unresolved_links: Vec, external_links: Vec, + document_references: HashMap>, diagnostics: Vec, inline_parser: Parser, + line_starts: Vec, next_block_index: usize, other_count: usize, + table_occurrences: HashMap, + table_cells_retained: usize, + table_text_bytes: usize, + table_limit_diagnostics: HashSet<&'static str>, + document_reference_limit_reported: bool, } #[derive(Clone)] @@ -211,19 +248,310 @@ struct DocumentTargetHint { root_relative: bool, } +struct FrontmatterExtraction { + metadata: Map, + facts: Vec, +} + +struct FrontmatterFact { + key: String, + key_path: String, + parent_path: Option, + value: Value, + start_byte: usize, + end_byte: usize, +} + +#[derive(Default)] +struct MetadataBudget { + keys: usize, + array_items: usize, +} + +#[derive(Clone)] +struct TableCellFact { + text: String, + raw_start: usize, + raw_end: usize, +} + +#[derive(Clone, Copy)] +enum TableAlignment { + Left, + Center, + Right, + Unspecified, +} + +impl TableAlignment { + const fn as_str(self) -> &'static str { + match self { + Self::Left => "left", + Self::Center => "center", + Self::Right => "right", + Self::Unspecified => "unspecified", + } + } +} + +fn table_children<'tree>( + node: Node<'tree>, +) -> (Option>, Option>, Vec>) { + let mut header = None; + let mut delimiter = None; + let mut rows = Vec::new(); + let mut cursor = node.walk(); + for child in node.children(&mut cursor).filter(Node::is_named) { + match child.kind() { + "pipe_table_header" => header = Some(child), + "pipe_table_delimiter_row" => delimiter = Some(child), + "pipe_table_row" => rows.push(child), + _ => {} + } + } + (header, delimiter, rows) +} + +fn table_cells(node: Node<'_>, source: &[u8]) -> Vec { + let mut cells = Vec::new(); + let mut cursor = node.walk(); + for child in node + .children(&mut cursor) + .filter(|child| child.kind() == "pipe_table_cell") + { + cells.push(TableCellFact::empty( + child.start_byte(), + child.end_byte(), + node_text(source, child.start_byte(), child.end_byte()), + )); + } + cells +} + +fn table_alignments(node: Node<'_>, source: &[u8]) -> Vec { + let mut alignments = Vec::new(); + let mut cursor = node.walk(); + for cell in node + .children(&mut cursor) + .filter(|child| child.kind() == "pipe_table_delimiter_cell") + { + let text = node_text(source, cell.start_byte(), cell.end_byte()); + let text = text.trim(); + let alignment = match (text.starts_with(':'), text.ends_with(':')) { + (true, true) => TableAlignment::Center, + (true, false) => TableAlignment::Left, + (false, true) => TableAlignment::Right, + (false, false) => TableAlignment::Unspecified, + }; + alignments.push(alignment); + } + alignments +} + +fn normalize_table_text(text: &str) -> String { + text.split_whitespace().collect::>().join(" ") +} + +fn table_label(section: &str, headers: &[String]) -> String { + let header = headers + .iter() + .filter(|header| !header.is_empty()) + .cloned() + .collect::>() + .join(" | "); + let title = if header.is_empty() { + "table".to_owned() + } else { + format!("table: {header}") + }; + if section.is_empty() { + bounded_label(&title) + } else { + bounded_label(&format!("{section} — {title}")) + } +} + +fn row_label(headers: &[String], cells: &[String]) -> String { + let values = cells + .iter() + .enumerate() + .filter(|(_, cell)| !cell.is_empty()) + .take(4) + .map( + |(index, cell)| match headers.get(index).filter(|header| !header.is_empty()) { + Some(header) => format!("{header}={cell}"), + None => cell.clone(), + }, + ) + .collect::>(); + if values.is_empty() { + "table row".to_owned() + } else { + bounded_label(&values.join(" · ")) + } +} + +fn table_cell_label(header: Option<&str>, text: &str, column_index: usize) -> String { + let value = if text.is_empty() { "(empty)" } else { text }; + let column = header + .filter(|header| !header.is_empty()) + .map(str::to_owned) + .unwrap_or_else(|| format!("column {}", column_index.saturating_add(1))); + bounded_label(&format!("{column}: {value}")) +} + +fn compact_identity(value: &str) -> String { + let mut output = String::new(); + for character in value.chars() { + if output.len() >= 64 { + break; + } + if character.is_ascii_alphanumeric() || matches!(character, '_' | '-') { + output.push(character.to_ascii_lowercase()); + } else if !output.ends_with('-') { + output.push('-'); + } + } + let trimmed = output.trim_matches('-'); + if trimmed.is_empty() { + "row".to_owned() + } else { + trimmed.to_owned() + } +} + +fn truncate_utf8(text: &str, limit: usize) -> String { + if text.len() <= limit { + return text.to_owned(); + } + let mut end = limit; + while end > 0 && !text.is_char_boundary(end) { + end = end.saturating_sub(1); + } + text[..end].to_owned() +} + +fn is_inline_reference_candidate(value: &str) -> bool { + if value.is_empty() + || value.len() > MAX_REFERENCE_SPELLING_BYTES + || value.chars().any(char::is_whitespace) + || value.contains('`') + { + return false; + } + let valid = value.chars().all(|character| { + character.is_ascii_alphanumeric() + || matches!(character, '_' | '-' | '/' | '.' | ':' | '$' | '#' | '@') + }); + if !valid { + return false; + } + // A path, qualified symbol, or identifier with an explicit separator is + // useful evidence. Keep ordinary single-word spans too: the backticks are + // the syntax proof that the author intended a literal identifier. + value + .chars() + .any(|character| character.is_ascii_alphanumeric()) +} + +/// Encode a nested source anchor using the same camel-case contract as the +/// strict graph model. Raw extraction attributes otherwise use snake case, +/// but table columns/cells are typed payloads and must be round-trippable. +fn source_anchor_json( + source_file: &str, + source: &[u8], + line_starts: &[usize], + start: usize, + end: usize, +) -> Value { + let start = start.min(source.len()); + let end = end.clamp(start, source.len()); + let (start_line, start_column) = indexed_source_point(line_starts, start); + let (end_line, end_column) = indexed_source_point(line_starts, end); + json!({ + "file": source_file, + "startByte": start as u64, + "endByte": end as u64, + "startLine": start_line as u64, + "startColumn": start_column as u64, + "endLine": end_line as u64, + "endColumn": end_column as u64, + }) +} + +fn newline_offsets(source: &[u8]) -> Vec { + let mut offsets = vec![0]; + offsets.extend( + source + .iter() + .enumerate() + .filter_map(|(index, byte)| (*byte == b'\n').then_some(index.saturating_add(1))), + ); + offsets +} + +fn indexed_source_point(line_starts: &[usize], offset: usize) -> (usize, usize) { + let line_index = match line_starts.binary_search(&offset) { + Ok(index) => index, + Err(index) => index.saturating_sub(1), + }; + let line_start = line_starts.get(line_index).copied().unwrap_or(0); + ( + line_index.saturating_add(1), + offset.saturating_sub(line_start), + ) +} + +fn stamp_source_range_indexed( + attributes: &mut Map, + source: &[u8], + line_starts: &[usize], + start: usize, + end: usize, +) { + let start = start.min(source.len()); + let end = end.clamp(start, source.len()); + let (start_line, start_column) = indexed_source_point(line_starts, start); + let (end_line, end_column) = indexed_source_point(line_starts, end); + attributes.insert("start_byte".to_owned(), Value::from(start as u64)); + attributes.insert("end_byte".to_owned(), Value::from(end as u64)); + attributes.insert("line_start".to_owned(), Value::from(start_line as u64)); + attributes.insert("line_end".to_owned(), Value::from(end_line as u64)); + attributes.insert("column_start".to_owned(), Value::from(start_column as u64)); + attributes.insert("column_end".to_owned(), Value::from(end_column as u64)); +} + +impl TableCellFact { + fn empty(start: usize, end: usize, text: String) -> Self { + Self { + text, + raw_start: start, + raw_end: end, + } + } +} + impl State<'_, '_> { fn add_root(&mut self, id: String, metadata: Option>) { self.seen_nodes.insert(id.clone()); let mut attributes = Map::new(); + let source_name = self + .path + .file_name() + .and_then(|name| name.to_str()) + .unwrap_or_default(); + let label = metadata + .as_ref() + .and_then(|metadata| metadata.get("title")) + .and_then(Value::as_str) + .map(str::trim) + .filter(|title| !title.is_empty()) + .map(bounded_label) + .unwrap_or_else(|| source_name.to_owned()); + attributes.insert("label".to_owned(), Value::String(label)); attributes.insert( - "label".to_owned(), - Value::String( - self.path - .file_name() - .and_then(|name| name.to_str()) - .unwrap_or_default() - .to_owned(), - ), + "qualified_name".to_owned(), + Value::String(self.source_file.clone()), ); attributes.insert("file_type".to_owned(), Value::String("document".to_owned())); attributes.insert( @@ -251,6 +579,98 @@ impl State<'_, '_> { self.file_id = id; } + fn add_frontmatter_nodes(&mut self, facts: Vec) { + let mut ids = HashMap::new(); + for fact in facts { + let id = crate::make_id(&[&self.source_file, "markdown_frontmatter", &fact.key_path]); + let parent = fact + .parent_path + .as_ref() + .and_then(|path| ids.get(path)) + .cloned() + .unwrap_or_else(|| self.file_id.clone()); + let mut attributes = Map::new(); + attributes.insert( + "symbol_kind".to_owned(), + Value::String("config_key".to_owned()), + ); + attributes.insert("file_type".to_owned(), Value::String("code".to_owned())); + attributes.insert( + "label".to_owned(), + Value::String(frontmatter_fact_label(&fact.key, &fact.value)), + ); + attributes.insert( + "qualified_name".to_owned(), + Value::String(format!("frontmatter{}", fact.key_path)), + ); + attributes.insert("key_path".to_owned(), Value::String(fact.key_path.clone())); + attributes.insert( + "format".to_owned(), + Value::String("yaml_frontmatter".to_owned()), + ); + attributes.insert( + "namespace".to_owned(), + Value::String(self.source_file.clone()), + ); + attributes.insert( + "source_file".to_owned(), + Value::String(self.source_file.clone()), + ); + attributes.insert( + "source_location".to_owned(), + Value::String(format!("L{}", self.line_at(fact.start_byte))), + ); + attributes.insert("_origin".to_owned(), Value::String("config".to_owned())); + attributes.insert( + "rule".to_owned(), + Value::String("markdown-frontmatter-key".to_owned()), + ); + stamp_source_range_indexed( + &mut attributes, + self.source, + &self.line_starts, + fact.start_byte, + fact.end_byte, + ); + self.seen_nodes.insert(id.clone()); + self.extraction.nodes.push(NodeRecord { + id: id.clone(), + attributes, + }); + self.add_frontmatter_relation(&parent, &id, fact.start_byte, fact.end_byte); + ids.insert(fact.key_path, id); + } + } + + fn add_frontmatter_relation(&mut self, source: &str, target: &str, start: usize, end: usize) { + let mut attributes = Map::new(); + attributes.insert("relation".to_owned(), Value::String("contains".to_owned())); + attributes.insert( + "confidence".to_owned(), + Value::String("EXTRACTED".to_owned()), + ); + attributes.insert( + "source_file".to_owned(), + Value::String(self.source_file.clone()), + ); + attributes.insert( + "source_location".to_owned(), + Value::String(format!("L{}", self.line_at(start))), + ); + attributes.insert("_origin".to_owned(), Value::String("config".to_owned())); + attributes.insert( + "rule".to_owned(), + Value::String("markdown-frontmatter-containment".to_owned()), + ); + stamp_source_range_indexed(&mut attributes, self.source, &self.line_starts, start, end); + attributes.insert("weight".to_owned(), json!(1.0)); + self.extraction.edges.push(EdgeRecord { + source: source.to_owned(), + target: target.to_owned(), + attributes, + }); + } + fn add_diagnostic(&mut self, diagnostic: String) { if self.diagnostics.len() < MAX_DIAGNOSTICS { self.diagnostics.push(diagnostic); @@ -373,6 +793,10 @@ impl State<'_, '_> { self.emit_heading(node, parent); return; } + "pipe_table" => { + self.emit_pipe_table(node, parent); + return; + } "inline" | "block_continuation" | "minus_metadata" | "plus_metadata" => return, _ => {} } @@ -407,13 +831,6 @@ impl State<'_, '_> { extra.insert("task_checked".to_owned(), Value::Bool(checked)); } } - if kind == "pipe_table_header" { - extra.insert("table_role".to_owned(), Value::String("header".to_owned())); - } else if kind == "pipe_table_row" { - extra.insert("table_role".to_owned(), Value::String("row".to_owned())); - } else if kind == "pipe_table_cell" { - extra.insert("table_role".to_owned(), Value::String("cell".to_owned())); - } if let Some(section) = self.heading_stack.last() { extra.insert( "document_section".to_owned(), @@ -432,6 +849,12 @@ impl State<'_, '_> { Value::String(format!("{}::{kind}#{}", self.stem, self.next_block_index)), ); } + if !matches!(kind, "table" | "table_row") { + let content = truncate_utf8(&normalize_table_text(&text), MAX_TABLE_CELL_BYTES); + if !content.is_empty() { + extra.insert("document_content".to_owned(), Value::String(content)); + } + } let id = self.add_block_node( id, &label, @@ -453,6 +876,537 @@ impl State<'_, '_> { } } + fn emit_pipe_table(&mut self, node: Node<'_>, parent: Option<&str>) { + if self.next_block_index > MAX_BLOCKS { + self.add_table_limit_diagnostic("blocks"); + return; + } + + let (header_node, delimiter_node, rows) = table_children(node); + let headers = header_node + .map(|header| table_cells(header, self.source)) + .unwrap_or_default(); + let delimiter_alignments = delimiter_node + .map(|delimiter| table_alignments(delimiter, self.source)) + .unwrap_or_default(); + let row_facts = rows + .iter() + .map(|row| table_cells(*row, self.source)) + .collect::>(); + + let source_section = self + .heading_stack + .last() + .map(|heading| heading.qualified_name.clone()) + .unwrap_or_else(|| self.stem.clone()); + let header_signature = headers + .iter() + .take(MAX_TABLE_COLUMNS) + .map(|cell| truncate_utf8(&normalize_table_text(&cell.text), MAX_TABLE_CELL_BYTES)) + .collect::>() + .join("\u{1f}"); + let table_key = format!("{source_section}\u{1e}{header_signature}"); + let occurrence = self.table_occurrences.entry(table_key).or_default(); + *occurrence = occurrence.saturating_add(1); + let table_ordinal = *occurrence; + let table_qualified_name = format!("{source_section}::pipe_table#{table_ordinal}"); + let table_id = crate::make_id(&[ + &self.source_file, + "markdown_table", + &source_section, + &header_signature, + &table_ordinal.to_string(), + ]); + + let column_count = headers + .len() + .max(row_facts.iter().map(Vec::len).max().unwrap_or(0)); + let retained_columns = column_count.min(MAX_TABLE_COLUMNS); + let omitted_columns = column_count.saturating_sub(retained_columns); + if omitted_columns > 0 { + self.add_table_limit_diagnostic("columns"); + } + let mut retained_headers = Vec::with_capacity(retained_columns); + let mut header_truncated = false; + for cell in headers.iter().take(retained_columns) { + let mut header = normalize_table_text(&cell.text); + if header.len() > MAX_TABLE_CELL_BYTES { + header = truncate_utf8(&header, MAX_TABLE_CELL_BYTES); + header_truncated = true; + self.add_table_limit_diagnostic("cell_text"); + } + if self.table_text_bytes.saturating_add(header.len()) > MAX_TABLE_TEXT_BYTES { + header.clear(); + header_truncated = true; + self.add_table_limit_diagnostic("text"); + } else { + self.table_text_bytes = self.table_text_bytes.saturating_add(header.len()); + } + retained_headers.push(header); + } + let table_initially_truncated = header_truncated || omitted_columns > 0; + let table_label = table_label(&source_section, &retained_headers); + let mut table_extra = Map::new(); + table_extra.insert( + "qualified_name".to_owned(), + Value::String(table_qualified_name.clone()), + ); + table_extra.insert("table_role".to_owned(), Value::String("table".to_owned())); + table_extra.insert( + "table_headers".to_owned(), + Value::Array( + retained_headers + .iter() + .map(|header| Value::String(header.clone())) + .collect(), + ), + ); + table_extra.insert( + "table_alignments".to_owned(), + Value::Array( + (0..retained_columns) + .map(|index| { + Value::String( + delimiter_alignments + .get(index) + .copied() + .unwrap_or(TableAlignment::Unspecified) + .as_str() + .to_owned(), + ) + }) + .collect(), + ), + ); + table_extra.insert( + "table_columns".to_owned(), + Value::Array( + retained_headers + .iter() + .enumerate() + .map(|(index, header)| { + let mut column = Map::new(); + column.insert("index".to_owned(), json!(index)); + column.insert("header".to_owned(), Value::String(header.clone())); + column.insert( + "alignment".to_owned(), + Value::String( + delimiter_alignments + .get(index) + .copied() + .unwrap_or(TableAlignment::Unspecified) + .as_str() + .to_owned(), + ), + ); + if let Some(cell) = headers.get(index) { + column.insert( + "source".to_owned(), + source_anchor_json( + &self.source_file, + self.source, + &self.line_starts, + cell.raw_start, + cell.raw_end, + ), + ); + } + Value::Object(column) + }) + .collect(), + ), + ); + table_extra.insert( + "table_body_row_count".to_owned(), + json!(rows.len().min(MAX_TABLE_ROWS)), + ); + table_extra.insert("table_omitted_row_count".to_owned(), json!(0)); + table_extra.insert( + "table_omitted_column_count".to_owned(), + json!(omitted_columns), + ); + table_extra.insert( + "table_truncated".to_owned(), + Value::Bool(table_initially_truncated), + ); + let table_id = self.add_block_node( + table_id, + &table_label, + "pipe_table", + node.start_byte()..node.end_byte(), + parent, + table_extra, + ); + + // Keep the complete parser-backed table hierarchy in graph/1, but + // give each record bounded semantic labels and stable structural + // identities. Consumers can navigate exact header/cell evidence while + // architecture analysis treats these containment edges as zero-weight. + let mut table_node_count = 1usize; + let mut table_cell_count = 0usize; + let mut table_text_bytes = retained_headers.iter().map(String::len).sum::(); + if let Some(header) = header_node + && self.next_block_index <= MAX_BLOCKS + && table_node_count < MAX_TABLE_NODES_PER_TABLE + { + let header_cells = table_cells(header, self.source); + let header_qualified_name = format!("{table_qualified_name}::pipe_table_header#1"); + let header_id = crate::make_id(&[&table_id, "markdown_table_header"]); + let mut header_extra = Map::new(); + header_extra.insert( + "qualified_name".to_owned(), + Value::String(header_qualified_name.clone()), + ); + header_extra.insert("table_role".to_owned(), Value::String("header".to_owned())); + let header_content = retained_headers.join(" | "); + if !header_content.is_empty() { + header_extra.insert( + "document_content".to_owned(), + Value::String(header_content.clone()), + ); + } + let header_id = self.add_block_node( + header_id, + &bounded_label(&format!("header: {header_content}")), + "pipe_table_header", + header.start_byte()..header.end_byte(), + Some(&table_id), + header_extra, + ); + table_node_count = table_node_count.saturating_add(1); + + for (column_index, cell) in header_cells.iter().take(retained_columns).enumerate() { + if self.next_block_index > MAX_BLOCKS + || table_node_count >= MAX_TABLE_NODES_PER_TABLE + || table_cell_count >= MAX_TABLE_CELLS_PER_TABLE + || self.table_cells_retained >= MAX_TABLE_CELLS + { + header_truncated = true; + self.add_table_limit_diagnostic("cells"); + break; + } + let text = retained_headers + .get(column_index) + .cloned() + .unwrap_or_default(); + let cell_id = crate::make_id(&[ + &table_id, + "markdown_table_header_cell", + &column_index.to_string(), + ]); + let mut cell_extra = Map::new(); + cell_extra.insert( + "qualified_name".to_owned(), + Value::String(format!( + "{header_qualified_name}::pipe_table_cell#{}", + column_index.saturating_add(1) + )), + ); + cell_extra.insert( + "table_role".to_owned(), + Value::String("header_cell".to_owned()), + ); + cell_extra.insert("table_column_index".to_owned(), json!(column_index)); + cell_extra.insert( + "table_alignment".to_owned(), + Value::String( + delimiter_alignments + .get(column_index) + .copied() + .unwrap_or(TableAlignment::Unspecified) + .as_str() + .to_owned(), + ), + ); + cell_extra.insert( + "table_cell_state".to_owned(), + Value::String(if text.is_empty() { "empty" } else { "present" }.to_owned()), + ); + if !text.is_empty() { + cell_extra.insert("document_content".to_owned(), Value::String(text.clone())); + } + let cell_id = self.add_block_node( + cell_id, + &table_cell_label(None, &text, column_index), + "pipe_table_cell", + cell.raw_start..cell.raw_end, + Some(&header_id), + cell_extra, + ); + self.collect_inline_text_range(cell.raw_start, cell.raw_end, &cell_id); + table_node_count = table_node_count.saturating_add(1); + table_cell_count = table_cell_count.saturating_add(1); + self.table_cells_retained = self.table_cells_retained.saturating_add(1); + } + } + + let mut row_occurrences = HashMap::::new(); + let mut retained_rows = 0usize; + let mut table_truncated = table_initially_truncated || header_truncated; + for (row_index, row) in rows.iter().enumerate() { + if row_index >= MAX_TABLE_ROWS { + table_truncated = true; + self.add_table_limit_diagnostic("rows"); + break; + } + if self.next_block_index > MAX_BLOCKS { + table_truncated = true; + self.add_table_limit_diagnostic("blocks"); + break; + } + if table_node_count >= MAX_TABLE_NODES_PER_TABLE { + table_truncated = true; + self.add_table_limit_diagnostic("per_table_nodes"); + break; + } + if table_cell_count >= MAX_TABLE_CELLS_PER_TABLE + || self.table_cells_retained >= MAX_TABLE_CELLS + { + table_truncated = true; + self.add_table_limit_diagnostic("cells"); + break; + } + let cells = table_cells(*row, self.source); + let normalized_cells = cells + .iter() + .take(retained_columns) + .map(|cell| truncate_utf8(&normalize_table_text(&cell.text), MAX_TABLE_CELL_BYTES)) + .collect::>(); + let identity = normalized_cells + .iter() + .enumerate() + .find(|(_, value)| !value.is_empty()) + .map(|(index, value)| (index, value.clone())); + let identity_key = identity.as_ref().map_or_else( + || format!("ordinal:{row_index}"), + |(index, value)| format!("column:{index}:{value}"), + ); + let identity_occurrence = row_occurrences.entry(identity_key.clone()).or_default(); + *identity_occurrence = identity_occurrence.saturating_add(1); + let row_qualified_name = format!( + "{table_qualified_name}::pipe_table_row#{}-{}", + compact_identity(&identity_key), + *identity_occurrence + ); + let row_id = crate::make_id(&[ + &table_id, + "markdown_table_row", + &identity_key, + &identity_occurrence.to_string(), + ]); + let mut row_extra = Map::new(); + row_extra.insert( + "qualified_name".to_owned(), + Value::String(row_qualified_name.clone()), + ); + row_extra.insert("table_role".to_owned(), Value::String("row".to_owned())); + row_extra.insert("table_row_index".to_owned(), json!(row_index)); + if let Some((index, _)) = identity.as_ref() { + row_extra.insert("table_identity_cell_index".to_owned(), json!(*index)); + } + let mut row_truncated = omitted_columns > 0; + let mut serialized_cells = Vec::with_capacity(retained_columns); + let mut emitted_cells = Vec::<(usize, String, &'static str, usize, usize)>::new(); + for column_index in 0..retained_columns { + if self.next_block_index.saturating_add(emitted_cells.len()) > MAX_BLOCKS + || table_node_count + .saturating_add(1) + .saturating_add(emitted_cells.len()) + >= MAX_TABLE_NODES_PER_TABLE + || table_cell_count.saturating_add(emitted_cells.len()) + >= MAX_TABLE_CELLS_PER_TABLE + || self + .table_cells_retained + .saturating_add(emitted_cells.len()) + >= MAX_TABLE_CELLS + { + row_truncated = true; + table_truncated = true; + self.add_table_limit_diagnostic("cells"); + break; + } + let Some(cell) = cells.get(column_index) else { + serialized_cells.push(json!({ + "columnIndex": column_index, + "state": "missing", + "text": "" + })); + continue; + }; + let mut text = normalize_table_text(&cell.text); + let mut state = if text.is_empty() { "empty" } else { "present" }; + if text.len() > MAX_TABLE_CELL_BYTES { + text = truncate_utf8(&text, MAX_TABLE_CELL_BYTES); + row_truncated = true; + self.add_table_limit_diagnostic("cell_text"); + } + let text_bytes = text.len(); + if self.table_text_bytes.saturating_add(text_bytes) > MAX_TABLE_TEXT_BYTES + || table_text_bytes.saturating_add(text_bytes) > MAX_TABLE_TEXT_BYTES_PER_TABLE + { + text.clear(); + state = "limited"; + row_truncated = true; + table_truncated = true; + self.add_table_limit_diagnostic("text"); + } else { + self.table_text_bytes = self.table_text_bytes.saturating_add(text_bytes); + table_text_bytes = table_text_bytes.saturating_add(text_bytes); + } + serialized_cells.push(json!({ + "columnIndex": column_index, + "state": state, + "text": text, + "source": source_anchor_json( + &self.source_file, + self.source, + &self.line_starts, + cell.raw_start, + cell.raw_end, + ) + })); + emitted_cells.push((column_index, text, state, cell.raw_start, cell.raw_end)); + } + if cells.len() > retained_columns { + row_truncated = true; + } + row_extra.insert("table_cells".to_owned(), Value::Array(serialized_cells)); + row_extra.insert("table_truncated".to_owned(), Value::Bool(row_truncated)); + let row_label = row_label(&retained_headers, &normalized_cells); + let row_id = self.add_block_node( + row_id, + &row_label, + "pipe_table_row", + row.start_byte()..row.end_byte(), + Some(&table_id), + row_extra, + ); + table_node_count = table_node_count.saturating_add(1); + + for (column_index, text, state, start, end) in emitted_cells { + let cell_id = + crate::make_id(&[&row_id, "markdown_table_cell", &column_index.to_string()]); + let mut cell_extra = Map::new(); + cell_extra.insert( + "qualified_name".to_owned(), + Value::String(format!( + "{}::pipe_table_cell#{}", + row_qualified_name, + column_index.saturating_add(1) + )), + ); + cell_extra.insert( + "table_role".to_owned(), + Value::String("body_cell".to_owned()), + ); + cell_extra.insert("table_column_index".to_owned(), json!(column_index)); + cell_extra.insert( + "table_header".to_owned(), + Value::String( + retained_headers + .get(column_index) + .cloned() + .unwrap_or_default(), + ), + ); + cell_extra.insert( + "table_cell_state".to_owned(), + Value::String(state.to_owned()), + ); + if !text.is_empty() { + cell_extra.insert("document_content".to_owned(), Value::String(text.clone())); + } + let label = if state == "limited" { + table_cell_label( + retained_headers.get(column_index).map(String::as_str), + "(limited)", + column_index, + ) + } else { + table_cell_label( + retained_headers.get(column_index).map(String::as_str), + &text, + column_index, + ) + }; + let cell_id = self.add_block_node( + cell_id, + &label, + "pipe_table_cell", + start..end, + Some(&row_id), + cell_extra, + ); + if state != "limited" { + self.collect_inline_text_range(start, end, &cell_id); + } + table_node_count = table_node_count.saturating_add(1); + table_cell_count = table_cell_count.saturating_add(1); + self.table_cells_retained = self.table_cells_retained.saturating_add(1); + } + retained_rows = retained_rows.saturating_add(1); + } + let omitted_rows = rows.len().saturating_sub(retained_rows); + table_truncated |= omitted_rows > 0; + self.update_table_metadata(&table_id, retained_rows, omitted_rows, table_truncated); + } + + fn update_table_metadata( + &mut self, + table_id: &str, + retained_rows: usize, + omitted_rows: usize, + truncated: bool, + ) { + if let Some(table) = self + .extraction + .nodes + .iter_mut() + .find(|node| node.id == table_id) + { + table + .attributes + .insert("table_body_row_count".to_owned(), json!(retained_rows)); + table + .attributes + .insert("table_omitted_row_count".to_owned(), json!(omitted_rows)); + table + .attributes + .insert("table_truncated".to_owned(), Value::Bool(truncated)); + } + } + + fn collect_inline_text_range(&mut self, start: usize, end: usize, owner_id: &str) { + let start = start.min(self.source.len()); + let end = end.clamp(start, self.source.len()); + if start >= end { + return; + } + let inline_source = &self.source[start..end]; + let Some(tree) = self.inline_parser.parse(inline_source, None) else { + self.add_diagnostic("Markdown inline parser was cancelled".to_owned()); + return; + }; + self.walk_inline_node(tree.root_node(), start, owner_id); + self.scan_reference_links(start, inline_source, owner_id); + self.scan_wikilinks(start, inline_source, owner_id); + self.scan_inline_code_references(start, inline_source, owner_id); + } + + fn add_table_limit_diagnostic(&mut self, class: &'static str) { + if self.table_limit_diagnostics.insert(class) { + self.add_diagnostic(format!("Markdown table {class} limit exceeded")); + self.extraction.extensions.insert( + crate::EXTRACTION_QUALITY_EXTENSION.to_owned(), + json!(crate::EXTRACTION_QUALITY_PARTIAL), + ); + self.extraction.extensions.insert( + crate::EXTRACTION_QUALITY_REASON_EXTENSION.to_owned(), + json!("markdown_table_limit"), + ); + } + } + fn add_block_node( &mut self, id: String, @@ -469,6 +1423,10 @@ impl State<'_, '_> { self.next_block_index = self.next_block_index.saturating_add(1); extra.insert("label".to_owned(), Value::String(bounded_label(label))); extra.insert("file_type".to_owned(), Value::String("document".to_owned())); + extra.insert( + "document_format".to_owned(), + Value::String("markdown".to_owned()), + ); extra.insert("document_kind".to_owned(), Value::String(kind.to_owned())); extra.insert( "source_file".to_owned(), @@ -480,7 +1438,13 @@ impl State<'_, '_> { ); extra.insert("block_index".to_owned(), json!(block_index)); extra.insert("_origin".to_owned(), Value::String("artifact".to_owned())); - stamp_source_range(&mut extra, self.source, range.start, range.end); + stamp_source_range_indexed( + &mut extra, + self.source, + &self.line_starts, + range.start, + range.end, + ); self.extraction.nodes.push(NodeRecord { id: id.clone(), attributes: extra, @@ -522,6 +1486,193 @@ impl State<'_, '_> { self.walk_inline_node(tree.root_node(), start, owner_id); self.scan_reference_links(start, inline_source, owner_id); self.scan_wikilinks(start, inline_source, owner_id); + self.scan_inline_code_references(start, inline_source, owner_id); + } + + /// Retain only explicitly delimited code spans that have a shape useful + /// to a deterministic repository resolver. Ordinary prose remains prose; + /// a backtick span is evidence of intentional reference syntax, but it is + /// still resolved fail-closed later by the graph publisher. + fn scan_inline_code_references(&mut self, base: usize, source: &[u8], owner_id: &str) { + let mut offset = 0usize; + while offset < source.len() { + if source[offset] != b'`' { + offset = offset.saturating_add(1); + continue; + } + let mut run = 1usize; + while offset.saturating_add(run) < source.len() && source[offset + run] == b'`' { + run = run.saturating_add(1); + } + let body_start = offset.saturating_add(run); + let mut search = body_start; + let mut close = None; + while search < source.len() { + if source[search] != b'`' { + search = search.saturating_add(1); + continue; + } + let mut close_run = 1usize; + while search.saturating_add(close_run) < source.len() + && source[search + close_run] == b'`' + { + close_run = close_run.saturating_add(1); + } + if close_run == run { + close = Some(search); + break; + } + search = search.saturating_add(close_run); + } + let Some(close) = close else { + break; + }; + let absolute_start = base.saturating_add(offset); + let absolute_end = base + .saturating_add(close.saturating_add(run)) + .min(self.source.len()); + let body = String::from_utf8_lossy(&source[body_start..close]); + let spelling = body.trim(); + if is_inline_reference_candidate(spelling) { + self.record_document_reference( + owner_id, + spelling, + "inline_code", + self.link_site(absolute_start, absolute_end), + "unresolved", + None, + Vec::new(), + ); + } + offset = close.saturating_add(run); + } + } + + #[allow(clippy::too_many_arguments)] + fn record_document_reference( + &mut self, + owner_id: &str, + spelling: &str, + kind: &str, + site: LinkSite, + resolution: &str, + target: Option<&str>, + mut candidates: Vec, + ) { + let spelling = truncate_utf8(spelling.trim(), MAX_REFERENCE_SPELLING_BYTES); + if spelling.is_empty() { + return; + } + let references = self + .document_references + .entry(owner_id.to_owned()) + .or_default(); + if references.len() >= MAX_DOCUMENT_REFERENCES { + if !self.document_reference_limit_reported { + self.document_reference_limit_reported = true; + self.add_diagnostic("Markdown document reference limit exceeded".to_owned()); + self.extraction.extensions.insert( + crate::EXTRACTION_QUALITY_EXTENSION.to_owned(), + json!(crate::EXTRACTION_QUALITY_PARTIAL), + ); + self.extraction.extensions.insert( + crate::EXTRACTION_QUALITY_REASON_EXTENSION.to_owned(), + json!("markdown_reference_limit"), + ); + } + return; + } + let duplicate = references.iter().any(|value| { + value.as_object().is_some_and(|object| { + object.get("kind").and_then(Value::as_str) == Some(kind) + && object.get("spelling").and_then(Value::as_str) == Some(&spelling) + && object + .get("site") + .and_then(Value::as_object) + .and_then(|site| site.get("startByte")) + .and_then(Value::as_u64) + == Some(site.start_byte as u64) + }) + }); + if duplicate { + return; + } + candidates.sort_by_cached_key(|value| { + let object = value.as_object(); + ( + object + .and_then(|object| object.get("nodeId")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + object + .and_then(|object| object.get("reason")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + object + .and_then(|object| object.get("confidence")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + ) + }); + candidates.dedup(); + let mut reference = Map::new(); + reference.insert("spelling".to_owned(), Value::String(spelling)); + reference.insert("kind".to_owned(), Value::String(kind.to_owned())); + reference.insert( + "site".to_owned(), + source_anchor_json( + &self.source_file, + self.source, + &self.line_starts, + site.start_byte, + site.end_byte, + ), + ); + reference.insert( + "resolution".to_owned(), + Value::String(resolution.to_owned()), + ); + if let Some(target) = target.filter(|target| !target.is_empty()) { + reference.insert("target".to_owned(), Value::String(target.to_owned())); + } + if !candidates.is_empty() { + reference.insert("candidates".to_owned(), Value::Array(candidates)); + } + references.push(Value::Object(reference)); + } + + fn publish_document_references(&mut self) { + for node in &mut self.extraction.nodes { + let Some(mut references) = self.document_references.remove(&node.id) else { + continue; + }; + references.sort_by_cached_key(|value| { + let object = value.as_object(); + ( + object + .and_then(|object| object.get("site")) + .and_then(Value::as_object) + .and_then(|site| site.get("startByte")) + .and_then(Value::as_u64) + .unwrap_or_default(), + object + .and_then(|object| object.get("kind")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + object + .and_then(|object| object.get("spelling")) + .and_then(Value::as_str) + .unwrap_or_default() + .to_owned(), + ) + }); + node.attributes + .insert("document_references".to_owned(), Value::Array(references)); + } } fn walk_inline_node(&mut self, node: Node<'_>, base: usize, owner_id: &str) { @@ -998,7 +2149,21 @@ impl State<'_, '_> { ); } } - Some(_) => self.add_unresolved(&pending, "ambiguous_footnote", &pending.raw), + Some(candidates) => self.add_unresolved_with_candidates( + &pending, + "ambiguous_footnote", + &pending.raw, + candidates + .iter() + .map(|target| { + json!({ + "nodeId": target, + "reason": "multiple footnote definitions", + "confidence": "ambiguous" + }) + }) + .collect(), + ), None => { self.add_unresolved(&pending, "missing_footnote_definition", &pending.raw) } @@ -1036,6 +2201,15 @@ impl State<'_, '_> { "column_start": pending.site.start_byte.saturating_sub(pending.site.line_start), "column_end": pending.site.end_byte.saturating_sub(self.line_start(pending.site.end_byte)), })); + self.record_document_reference( + &pending.owner_id, + raw, + pending.kind, + pending.site, + "unresolved", + None, + Vec::new(), + ); continue; } let (path_part, fragment) = @@ -1089,13 +2263,30 @@ impl State<'_, '_> { ); } } - Some(_) => self.add_unresolved(&pending, "ambiguous_fragment", fragment), + Some(candidates) => self.add_unresolved_with_candidates( + &pending, + "ambiguous_fragment", + fragment, + candidates + .iter() + .map(|target| { + json!({ + "nodeId": target, + "reason": "multiple matching heading anchors", + "confidence": "ambiguous" + }) + }) + .collect(), + ), None => self.add_unresolved(&pending, "missing_fragment", fragment), } + } else { + self.add_unresolved(&pending, "same_file_without_fragment", raw); } continue; } if !is_supported_local_link(&target_path) { + self.add_unresolved(&pending, "unsupported_local_target", raw); continue; } let target_id = crate::make_id(&[&target_path.to_string_lossy()]); @@ -1124,6 +2315,19 @@ impl State<'_, '_> { relation: Option<(&str, Option<&str>)>, ) { let (relation, fragment) = relation.unwrap_or(("references", None)); + let spelling = pending + .reference_label + .as_deref() + .unwrap_or(pending.raw.as_str()); + self.record_document_reference( + &pending.owner_id, + spelling, + pending.kind, + pending.site, + "exact", + Some(&target), + Vec::new(), + ); self.add_relation_with_site( &pending.owner_id, &target, @@ -1157,7 +2361,28 @@ impl State<'_, '_> { } fn add_unresolved(&mut self, pending: &PendingLink, reason: &str, target: &str) { + self.add_unresolved_with_candidates(pending, reason, target, Vec::new()); + } + + fn add_unresolved_with_candidates( + &mut self, + pending: &PendingLink, + reason: &str, + target: &str, + candidates: Vec, + ) { if self.unresolved_links.len() >= MAX_DIAGNOSTICS { + // Keep the typed evidence bounded independently of the diagnostic + // list. A limit is still an observable partial result. + self.record_document_reference( + &pending.owner_id, + target, + pending.kind, + pending.site, + "limited", + None, + Vec::new(), + ); return; } self.unresolved_links.push(json!({ @@ -1175,6 +2400,28 @@ impl State<'_, '_> { "column_start": pending.site.start_byte.saturating_sub(pending.site.line_start), "column_end": pending.site.end_byte.saturating_sub(self.line_start(pending.site.end_byte)), })); + let spelling = pending + .reference_label + .as_deref() + .unwrap_or(if target.is_empty() { + pending.raw.as_str() + } else { + target + }); + let resolution = if candidates.is_empty() { + "unresolved" + } else { + "ambiguous" + }; + self.record_document_reference( + &pending.owner_id, + spelling, + pending.kind, + pending.site, + resolution, + None, + candidates, + ); } fn add_relation( @@ -1248,19 +2495,16 @@ impl State<'_, '_> { } fn line_at(&self, offset: usize) -> usize { - self.source[..offset.min(self.source.len())] - .iter() - .filter(|byte| **byte == b'\n') - .count() - .saturating_add(1) + indexed_source_point(&self.line_starts, offset.min(self.source.len())).0 } fn line_start(&self, offset: usize) -> usize { - let prefix = &self.source[..offset.min(self.source.len())]; - prefix - .iter() - .rposition(|byte| *byte == b'\n') - .map_or(0, |newline| newline.saturating_add(1)) + let offset = offset.min(self.source.len()); + let line_index = match self.line_starts.binary_search(&offset) { + Ok(index) => index, + Err(index) => index.saturating_sub(1), + }; + self.line_starts.get(line_index).copied().unwrap_or(0) } fn pending_link_count(&self) -> usize { @@ -1274,7 +2518,7 @@ impl State<'_, '_> { } } -fn parse_frontmatter(source: &[u8]) -> (Option>, Option) { +fn parse_frontmatter(source: &[u8]) -> (Option, Option) { let Some((first_end, first_line)) = next_line(source, 0) else { return (None, None); }; @@ -1306,10 +2550,18 @@ fn parse_frontmatter(source: &[u8]) -> (Option>, Option(&yaml) { + let Ok(yaml) = std::str::from_utf8(yaml) else { + return ( + None, + Some("Markdown frontmatter must be valid UTF-8".to_owned()), + ); + }; + return match serde_yaml_ng::from_str::(yaml) { Ok(value) => match yaml_metadata(&value) { - Ok(metadata) => (Some(metadata), None), + Ok(metadata) => match frontmatter_facts(yaml, first_end, &metadata) { + Ok(facts) => (Some(FrontmatterExtraction { metadata, facts }), None), + Err(diagnostic) => (None, Some(diagnostic)), + }, Err(diagnostic) => (None, Some(diagnostic.to_owned())), }, Err(_) => ( @@ -1378,45 +2630,31 @@ fn yaml_metadata(value: &serde_yaml_ng::Value) -> Result, &'s let serde_yaml_ng::Value::Mapping(mapping) = value else { return Err("Markdown frontmatter must be a mapping"); }; - if mapping.len() > MAX_METADATA_KEYS { - return Err("Markdown frontmatter has too many keys"); + let mut budget = MetadataBudget::default(); + yaml_mapping(mapping, 0, &mut budget) +} + +fn yaml_mapping( + mapping: &serde_yaml_ng::Mapping, + depth: usize, + budget: &mut MetadataBudget, +) -> Result, &'static str> { + if depth > MAX_METADATA_DEPTH { + return Err("Markdown frontmatter exceeds the nesting-depth limit"); } let mut entries = Vec::with_capacity(mapping.len()); for (key, value) in mapping { let serde_yaml_ng::Value::String(key) = key else { return Err("Markdown frontmatter keys must be strings"); }; + budget.keys = budget.keys.saturating_add(1); + if budget.keys > MAX_METADATA_KEYS { + return Err("Markdown frontmatter has too many keys"); + } if key.len() > MAX_METADATA_STRING_BYTES { return Err("Markdown frontmatter key exceeds the byte limit"); } - let json_value = match value { - serde_yaml_ng::Value::Null => Value::Null, - serde_yaml_ng::Value::Bool(value) => Value::Bool(*value), - serde_yaml_ng::Value::Number(value) => { - serde_json::to_value(value).map_err(|_| "Markdown frontmatter number is invalid")? - } - serde_yaml_ng::Value::String(value) => { - if value.len() > MAX_METADATA_STRING_BYTES { - return Err("Markdown frontmatter value exceeds the byte limit"); - } - Value::String(value.clone()) - } - serde_yaml_ng::Value::Sequence(values) => { - if values.len() > MAX_METADATA_ARRAY_ITEMS { - return Err("Markdown frontmatter array exceeds the item limit"); - } - let mut output = Vec::with_capacity(values.len()); - for value in values { - let scalar = yaml_scalar(value) - .ok_or("Markdown frontmatter arrays must contain scalars")?; - output.push(scalar); - } - Value::Array(output) - } - serde_yaml_ng::Value::Mapping(_) | serde_yaml_ng::Value::Tagged(_) => { - return Err("Markdown frontmatter nested values are not supported"); - } - }; + let json_value = yaml_value(value, depth.saturating_add(1), budget)?; entries.push((key.clone(), json_value)); } entries.sort_by(|left, right| left.0.cmp(&right.0)); @@ -1427,18 +2665,218 @@ fn yaml_metadata(value: &serde_yaml_ng::Value) -> Result, &'s Ok(output) } -fn yaml_scalar(value: &serde_yaml_ng::Value) -> Option { +fn yaml_value( + value: &serde_yaml_ng::Value, + depth: usize, + budget: &mut MetadataBudget, +) -> Result { + if depth > MAX_METADATA_DEPTH { + return Err("Markdown frontmatter exceeds the nesting-depth limit"); + } match value { - serde_yaml_ng::Value::Null => Some(Value::Null), - serde_yaml_ng::Value::Bool(value) => Some(Value::Bool(*value)), - serde_yaml_ng::Value::Number(value) => serde_json::to_value(value).ok(), - serde_yaml_ng::Value::String(value) if value.len() <= MAX_METADATA_STRING_BYTES => { - Some(Value::String(value.clone())) + serde_yaml_ng::Value::Null => Ok(Value::Null), + serde_yaml_ng::Value::Bool(value) => Ok(Value::Bool(*value)), + serde_yaml_ng::Value::Number(value) => { + serde_json::to_value(value).map_err(|_| "Markdown frontmatter number is invalid") } - _ => None, + serde_yaml_ng::Value::String(value) => { + if value.len() > MAX_METADATA_STRING_BYTES { + return Err("Markdown frontmatter value exceeds the byte limit"); + } + Ok(Value::String(value.clone())) + } + serde_yaml_ng::Value::Sequence(values) => { + budget.array_items = budget.array_items.saturating_add(values.len()); + if values.len() > MAX_METADATA_ARRAY_ITEMS + || budget.array_items > MAX_METADATA_ARRAY_ITEMS + { + return Err("Markdown frontmatter array exceeds the item limit"); + } + values + .iter() + .map(|value| yaml_value(value, depth.saturating_add(1), budget)) + .collect::, _>>() + .map(Value::Array) + } + serde_yaml_ng::Value::Mapping(mapping) => { + yaml_mapping(mapping, depth.saturating_add(1), budget).map(Value::Object) + } + serde_yaml_ng::Value::Tagged(_) => Err("Markdown frontmatter YAML tags are not supported"), } } +fn frontmatter_facts( + yaml: &str, + source_offset: usize, + metadata: &Map, +) -> Result, String> { + let config = ProcessConfig::new("yaml") + .minimal() + .with_data_extraction(true); + let parsed = tree_sitter_language_pack::process(yaml, &config) + .map_err(|error| format!("Markdown frontmatter source anchoring failed: {error}"))?; + let root = parsed + .data + .ok_or_else(|| "Markdown frontmatter source anchoring produced no data".to_owned())?; + let root_value = Value::Object(metadata.clone()); + let mut facts = Vec::new(); + let mut seen_paths = HashSet::new(); + collect_frontmatter_facts( + &root.children, + "", + None, + &root_value, + source_offset, + yaml.len(), + &mut seen_paths, + &mut facts, + )?; + if !metadata.is_empty() && facts.is_empty() { + return Err("Markdown frontmatter keys could not be source-anchored".to_owned()); + } + Ok(facts) +} + +#[allow(clippy::too_many_arguments)] +fn collect_frontmatter_facts( + nodes: &[DataNode], + parent_path: &str, + parent_fact_path: Option<&str>, + root: &Value, + source_offset: usize, + yaml_len: usize, + seen_paths: &mut HashSet, + facts: &mut Vec, +) -> Result<(), &'static str> { + for node in nodes { + let Some(segment) = node.key.as_deref() else { + continue; + }; + let key_path = frontmatter_key_path(parent_path, segment); + let Some(value) = json_pointer_value(root, &key_path) else { + return Err("Markdown frontmatter syntax and normalized keys disagree"); + }; + if !seen_paths.insert(key_path.clone()) { + return Err("Markdown frontmatter contains a duplicate key path"); + } + let scalar_sequence_item = node.kind == DataNodeKind::Sequence + && node.children.is_empty() + && !matches!(value, Value::Array(_) | Value::Object(_)); + let emitted_path = if scalar_sequence_item { + parent_fact_path.map(str::to_owned) + } else { + if facts.len() >= MAX_METADATA_GRAPH_NODES { + return Err("Markdown frontmatter graph-node limit exceeded"); + } + if node.span.start_byte >= node.span.end_byte || node.span.end_byte > yaml_len { + return Err("Markdown frontmatter contains an invalid source range"); + } + facts.push(FrontmatterFact { + key: if node.kind == DataNodeKind::Sequence { + segment + .parse::() + .ok() + .map_or_else(|| segment.to_owned(), |index| format!("item {}", index + 1)) + } else { + segment.to_owned() + }, + key_path: key_path.clone(), + parent_path: parent_fact_path.map(str::to_owned), + value: value.clone(), + start_byte: source_offset.saturating_add(node.span.start_byte), + end_byte: source_offset.saturating_add(node.span.end_byte), + }); + Some(key_path.clone()) + }; + collect_frontmatter_facts( + &node.children, + &key_path, + emitted_path.as_deref(), + root, + source_offset, + yaml_len, + seen_paths, + facts, + )?; + } + Ok(()) +} + +fn frontmatter_key_path(parent: &str, segment: &str) -> String { + let escaped = segment.replace('~', "~0").replace('/', "~1"); + if parent.is_empty() { + format!("/{escaped}") + } else { + format!("{parent}/{escaped}") + } +} + +fn json_pointer_value<'value>(root: &'value Value, pointer: &str) -> Option<&'value Value> { + root.pointer(pointer) +} + +fn frontmatter_fact_label(key: &str, value: &Value) -> String { + let semantic_key = key.to_ascii_lowercase().replace(['-', '_'], ""); + let show_value = matches!( + semantic_key.as_str(), + "title" + | "tag" + | "tags" + | "alias" + | "aliases" + | "author" + | "authors" + | "description" + | "summary" + | "category" + | "categories" + | "layout" + | "status" + | "draft" + | "date" + | "published" + | "updated" + | "slug" + | "permalink" + | "navlabel" + | "contenttype" + | "audience" + | "owner" + | "owners" + ); + let Some(summary) = show_value + .then(|| frontmatter_value_summary(value)) + .flatten() + else { + return bounded_label(key); + }; + bounded_label(&format!("{key}: {summary}")) +} + +fn frontmatter_value_summary(value: &Value) -> Option { + let summary = match value { + Value::Null => "null".to_owned(), + Value::Bool(value) => value.to_string(), + Value::Number(value) => value.to_string(), + Value::String(value) => compact_label(value), + Value::Array(values) if values.len() <= 16 => { + let values = values + .iter() + .map(|value| match value { + Value::Null => Some("null".to_owned()), + Value::Bool(value) => Some(value.to_string()), + Value::Number(value) => Some(value.to_string()), + Value::String(value) => Some(compact_label(value)), + Value::Array(_) | Value::Object(_) => None, + }) + .collect::>>()?; + values.join(", ") + } + Value::Array(_) | Value::Object(_) => return None, + }; + (!summary.is_empty()).then(|| truncate_utf8(&summary, MAX_LABEL_CHARS)) +} + fn next_line(source: &[u8], start: usize) -> Option<(usize, &[u8])> { if start >= source.len() { return None; diff --git a/crates/compass-languages/tests/markdown_coverage.rs b/crates/compass-languages/tests/markdown_coverage.rs index e2e0cede0..60ffe033c 100644 --- a/crates/compass-languages/tests/markdown_coverage.rs +++ b/crates/compass-languages/tests/markdown_coverage.rs @@ -3,6 +3,8 @@ use std::fs; use compass_languages::Engine; +const MAX_TEST_FRONTMATTER_DEPTH: usize = 13; + #[test] fn markdown_extracts_heading_hierarchy_and_only_local_document_links() -> Result<(), Box> { @@ -157,6 +159,18 @@ fn markdown_source_path_supports_frontmatter_blocks_and_section_links() -> Resul title: Guide tags: [rust, graph] draft: false +site: + navigation: + label: Graph guide +routes: + "docs/url": /guide +authors: + - name: Ada + roles: [editor, reviewer] +reviewers: [{name: Grace, team: Docs}] +aliases: + - Compass guide + - Graph handbook --- # Intro {#start} @@ -185,11 +199,90 @@ let hidden = "[not a link](ignored.md)"; .first() .ok_or("missing Markdown document root")?; assert_eq!(root.string("document_format"), "markdown"); - assert_eq!(root.attributes["document_metadata"]["title"], "Guide"); + let metadata = root + .attributes + .get("document_metadata") + .unwrap_or_else(|| panic!("extensions={:#?}", extraction.extensions)); + assert_eq!(metadata["title"], "Guide"); assert_eq!( root.attributes["document_metadata"]["tags"], serde_json::json!(["rust", "graph"]) ); + assert_eq!( + root.attributes["document_metadata"]["site"]["navigation"]["label"], + "Graph guide" + ); + assert_eq!( + root.attributes["document_metadata"]["authors"][0]["roles"], + serde_json::json!(["editor", "reviewer"]) + ); + assert_eq!(root.label(), "Guide"); + assert_eq!(root.string("qualified_name"), "docs/guide.md"); + + let config_nodes = extraction + .nodes + .iter() + .filter(|node| node.string("symbol_kind") == "config_key") + .collect::>(); + for expected in [ + "frontmatter/title", + "frontmatter/tags", + "frontmatter/site", + "frontmatter/site/navigation", + "frontmatter/site/navigation/label", + "frontmatter/routes", + "frontmatter/routes/docs~1url", + "frontmatter/authors", + "frontmatter/authors/0", + "frontmatter/authors/0/name", + "frontmatter/authors/0/roles", + "frontmatter/reviewers", + "frontmatter/reviewers/0", + "frontmatter/reviewers/0/name", + "frontmatter/reviewers/0/team", + "frontmatter/aliases", + ] { + assert!( + config_nodes + .iter() + .any(|node| node.string("qualified_name") == expected), + "missing {expected}: {config_nodes:#?}" + ); + } + assert!(config_nodes.iter().all(|node| { + node.string("format") == "yaml_frontmatter" + && node.string("file_type") == "code" + && node.string("_origin") == "config" + && node + .attributes + .get("start_byte") + .and_then(serde_json::Value::as_u64) + .zip( + node.attributes + .get("end_byte") + .and_then(serde_json::Value::as_u64), + ) + .is_some_and(|(start, end)| start < end) + })); + let title = config_nodes + .iter() + .find(|node| node.string("qualified_name") == "frontmatter/title") + .ok_or("missing title metadata node")?; + assert_eq!(title.label(), "title: Guide"); + let author = config_nodes + .iter() + .find(|node| node.string("qualified_name") == "frontmatter/authors/0") + .ok_or("missing author metadata node")?; + let author_name = config_nodes + .iter() + .find(|node| node.string("qualified_name") == "frontmatter/authors/0/name") + .ok_or("missing author name metadata node")?; + assert!(extraction.edges.iter().any(|edge| { + edge.source == author.id + && edge.target == author_name.id + && edge.string("relation") == "contains" + && edge.string("_origin") == "config" + })); let kinds = extraction .nodes @@ -209,6 +302,49 @@ let hidden = "[not a link](ignored.md)"; ] { assert!(kinds.contains(&expected), "missing {expected}: {kinds:?}"); } + let table = extraction + .nodes + .iter() + .find(|node| node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table"))) + .ok_or("missing table")?; + assert_eq!(table.label(), "Intro::Setext heading — table: Name | Value"); + assert_eq!( + table.attributes["table_headers"], + serde_json::json!(["Name", "Value"]) + ); + assert_eq!( + table.attributes["table_alignments"], + serde_json::json!(["left", "right"]) + ); + assert_eq!( + table.attributes["table_body_row_count"], + serde_json::json!(1) + ); + let row = extraction + .nodes + .iter() + .find(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table_row")) + }) + .ok_or("missing table row")?; + assert_eq!(row.label(), "Name=alpha · Value=`one`"); + assert_eq!(row.attributes["table_row_index"], serde_json::json!(0)); + assert_eq!( + row.attributes["table_identity_cell_index"], + serde_json::json!(0) + ); + assert_eq!( + row.attributes["table_cells"][0]["state"], + serde_json::json!("present") + ); + assert_eq!( + row.attributes["table_cells"][0]["text"], + serde_json::json!("alpha") + ); + assert_eq!( + row.attributes["table_cells"][1]["text"], + serde_json::json!("`one`") + ); assert!(extraction.nodes.iter().any(|node| { node.attributes.get("heading_style") == Some(&serde_json::json!("setext")) && node.attributes.get("heading_level") == Some(&serde_json::json!(2)) @@ -260,6 +396,210 @@ let hidden = "[not a link](ignored.md)"; Ok(()) } +#[test] +fn markdown_tables_publish_semantic_rows_and_retain_cell_links() -> Result<(), Box> { + let source = br#"# Ownership + +| Area | Owner | Status | +| :--- | :---: | ---: | +| [Graph](../graph.md) | `compass-model` | active | +| Empty | | values | +| Missing | + +## After +"#; + let extraction = + Engine::default().extract_source(std::path::Path::new("docs/index.md"), source)?; + let tables = extraction + .nodes + .iter() + .filter(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table")) + }) + .collect::>(); + let rows = extraction + .nodes + .iter() + .filter(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table_row")) + }) + .collect::>(); + let headers = extraction + .nodes + .iter() + .filter(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table_header")) + }) + .collect::>(); + let cells = extraction + .nodes + .iter() + .filter(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table_cell")) + }) + .collect::>(); + assert_eq!(tables.len(), 1); + assert_eq!(headers.len(), 1); + assert_eq!(rows.len(), 3); + assert_eq!(cells.len(), 10); + assert_eq!( + tables[0].attributes["table_body_row_count"], + serde_json::json!(3) + ); + assert_eq!( + tables[0].attributes["table_headers"], + serde_json::json!(["Area", "Owner", "Status"]) + ); + assert_eq!( + tables[0].attributes["table_alignments"], + serde_json::json!(["left", "center", "right"]) + ); + assert_eq!( + rows[0].label(), + "Area=[Graph](../graph.md) · Owner=`compass-model` · Status=active" + ); + assert_eq!(rows[1].label(), "Area=Empty · Status=values"); + assert_eq!( + rows[1].attributes["table_cells"][1]["state"], + serde_json::json!("empty") + ); + assert_eq!(rows[2].label(), "Area=Missing"); + assert_eq!( + rows[2].attributes["table_cells"][1]["state"], + serde_json::json!("missing") + ); + assert_eq!( + rows[2].attributes["table_cells"][2]["state"], + serde_json::json!("missing") + ); + let table_id = &tables[0].id; + let linked_row = rows[0]; + let link_cell = cells + .iter() + .find(|node| node.label().starts_with("Area: [Graph]")) + .ok_or("missing linked body cell")?; + assert!(extraction.edges.iter().any(|edge| { + edge.source == link_cell.id + && edge.attributes.get("relation") == Some(&serde_json::json!("references")) + && edge.attributes.get("link_kind") == Some(&serde_json::json!("inline")) + })); + let references = link_cell + .attributes + .get("document_references") + .and_then(serde_json::Value::as_array) + .ok_or("missing cell reference evidence")?; + assert!(references.iter().any(|reference| { + reference.get("kind") == Some(&serde_json::json!("inline")) + && reference.get("resolution") == Some(&serde_json::json!("exact")) + && reference["site"]["startByte"].as_u64().is_some() + })); + let code_cell = cells + .iter() + .find(|node| node.label() == "Owner: `compass-model`") + .ok_or("missing inline-code body cell")?; + let code_references = code_cell + .attributes + .get("document_references") + .and_then(serde_json::Value::as_array) + .ok_or("missing inline-code cell evidence")?; + assert!(code_references.iter().any(|reference| { + reference.get("kind") == Some(&serde_json::json!("inline_code")) + && reference.get("spelling") == Some(&serde_json::json!("compass-model")) + && reference.get("resolution") == Some(&serde_json::json!("unresolved")) + })); + assert_eq!( + references + .iter() + .filter(|reference| reference.get("kind") == Some(&serde_json::json!("inline"))) + .count(), + 1 + ); + assert!(extraction.edges.iter().any(|edge| { + edge.source == *table_id + && edge.target == linked_row.id + && edge.attributes.get("relation") == Some(&serde_json::json!("contains")) + })); + Ok(()) +} + +#[test] +fn markdown_table_limits_are_truthful_and_later_blocks_survive() -> Result<(), Box> { + let directory = tempfile::tempdir()?; + let path = directory.path().join("limits.md"); + let mut source = String::from("# Limits\n| Key | Value |\n| --- | --- |\n"); + for index in 0..10_005 { + source.push_str(&format!("| item-{index} | retained |\n")); + } + source.push_str("\n# After\nThe later section remains extractable.\n"); + std::fs::write(&path, source)?; + + let extraction = Engine::default().extract(&path)?; + let table = extraction + .nodes + .iter() + .find(|node| node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table"))) + .ok_or("missing limited table")?; + let retained = table.attributes["table_body_row_count"] + .as_u64() + .ok_or("missing retained row count")?; + let omitted = table.attributes["table_omitted_row_count"] + .as_u64() + .ok_or("missing omitted row count")?; + assert!(retained > 0 && retained < 10_005); + assert_eq!(retained + omitted, 10_005); + assert_eq!(table.attributes["table_truncated"], serde_json::json!(true)); + assert_eq!( + extraction.extensions[compass_languages::EXTRACTION_QUALITY_EXTENSION], + serde_json::json!(compass_languages::EXTRACTION_QUALITY_PARTIAL) + ); + assert!(extraction.nodes.iter().any(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("heading")) + && node.label() == "After" + })); + Ok(()) +} + +#[test] +fn markdown_table_anchors_are_byte_exact_across_crlf_and_unicode() -> Result<(), Box> { + let source = "# Зона\r\n\r\n| Name | Value |\r\n| --- | --- |\r\n| café | `one\\|two` |\r\n"; + let extraction = Engine::default() + .extract_source(std::path::Path::new("docs/guide.md"), source.as_bytes())?; + let table = extraction + .nodes + .iter() + .find(|node| node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table"))) + .ok_or("missing table")?; + let column_anchor = &table.attributes["table_columns"][0]["source"]; + let name_start = source.find("Name").ok_or("missing header text")?; + assert_eq!(column_anchor["startByte"], serde_json::json!(name_start)); + // Tree-sitter's cell span includes the trailing cell whitespace while + // preserving the visible text's exact byte start. + assert_eq!(column_anchor["endByte"], serde_json::json!(name_start + 5)); + assert_eq!(column_anchor["startLine"], serde_json::json!(3)); + assert_eq!(column_anchor["endLine"], serde_json::json!(3)); + + let row = extraction + .nodes + .iter() + .find(|node| { + node.attributes.get("document_kind") == Some(&serde_json::json!("pipe_table_row")) + }) + .ok_or("missing row")?; + assert_eq!( + row.attributes["table_cells"][0]["text"], + serde_json::json!("café") + ); + assert_eq!( + row.attributes["table_cells"][0]["source"]["startLine"], + serde_json::json!(5) + ); + assert_eq!( + row.attributes["table_cells"][1]["text"], + serde_json::json!("`one\\|two`") + ); + Ok(()) +} + #[test] fn markdown_duplicate_heading_slugs_follow_source_order_and_explicit_ids_remain_ambiguous() -> Result<(), Box> { @@ -421,6 +761,72 @@ fn markdown_frontmatter_is_bounded_and_diagnosed_without_swallowing_body() .contains_key("document_metadata") ); assert!(unsafe_yaml.nodes.iter().any(|node| node.label() == "Body")); + + let duplicate = Engine::default().extract_source( + std::path::Path::new("guide.md"), + b"---\ntitle: first\ntitle: second\n---\n# Body\n", + )?; + assert!( + !duplicate.nodes[0] + .attributes + .contains_key("document_metadata") + ); + assert!( + duplicate + .extensions + .get("markdown_diagnostics") + .and_then(serde_json::Value::as_array) + .is_some_and(|diagnostics| diagnostics.iter().any(|value| value + .as_str() + .is_some_and(|message| message.contains("frontmatter")))) + ); + assert!(duplicate.nodes.iter().any(|node| node.label() == "Body")); + + let invalid_utf8 = Engine::default().extract_source( + std::path::Path::new("guide.md"), + b"---\ntitle: \xff\n---\n# Body\n", + )?; + assert!( + invalid_utf8 + .extensions + .get("markdown_diagnostics") + .and_then(serde_json::Value::as_array) + .is_some_and(|diagnostics| diagnostics.iter().any(|value| value + .as_str() + .is_some_and(|message| message.contains("valid UTF-8")))) + ); + assert!(invalid_utf8.nodes.iter().any(|node| node.label() == "Body")); + + let mut deeply_nested = String::from("---\n"); + for depth in 0..=MAX_TEST_FRONTMATTER_DEPTH { + deeply_nested.push_str(&format!("{}level{depth}:\n", " ".repeat(depth))); + } + deeply_nested.push_str(&format!( + "{}value: final\n---\n# Body\n", + " ".repeat(MAX_TEST_FRONTMATTER_DEPTH + 1) + )); + let deeply_nested = Engine::default() + .extract_source(std::path::Path::new("guide.md"), deeply_nested.as_bytes())?; + assert!( + !deeply_nested.nodes[0] + .attributes + .contains_key("document_metadata") + ); + assert!( + deeply_nested + .extensions + .get("markdown_diagnostics") + .and_then(serde_json::Value::as_array) + .is_some_and(|diagnostics| diagnostics.iter().any(|value| value + .as_str() + .is_some_and(|message| message.contains("nesting-depth")))) + ); + assert!( + deeply_nested + .nodes + .iter() + .any(|node| node.label() == "Body") + ); Ok(()) } diff --git a/crates/compass-model/src/code_graph.rs b/crates/compass-model/src/code_graph.rs index 745178c6a..8f9ad0af6 100644 --- a/crates/compass-model/src/code_graph.rs +++ b/crates/compass-model/src/code_graph.rs @@ -4,7 +4,7 @@ use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicU64, Ordering}; use serde::{Deserialize, Serialize}; -use serde_json::Value; +use serde_json::{Map, Value}; use sha2::{Digest, Sha256}; use crate::provenance::{ @@ -12,9 +12,10 @@ use crate::provenance::{ }; use crate::{GraphError, validate_code_graph}; +/// The strict graph schema emitted and accepted by Compass. pub const CODE_GRAPH_SCHEMA_V1: &str = "compass.graph/1"; -/// The closed structural and enterprise node vocabulary for `compass.graph/1`. +/// The closed structural and enterprise node vocabulary for Compass graphs. #[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] pub enum NodeKind { @@ -496,6 +497,172 @@ pub struct ComponentNodeDetails { pub component_type: String, } +/// Format of a source-backed document structure node. +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DocumentFormat { + Markdown, + Html, + Text, + Pdf, + Docx, + Xlsx, + Pptx, + Rtf, + Other, +} + +/// Semantic role of a source-backed document structure node. +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DocumentRole { + Document, + Heading, + Paragraph, + List, + ListItem, + Quote, + Code, + ThematicBreak, + Table, + TableRow, + LinkDefinition, + FootnoteDefinition, + Page, + Sheet, + Slide, + Note, + Other, +} + +/// Derived significance used by query, topology, and human-facing projections. +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DocumentSignificance { + Content, + Container, + Scaffolding, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct DocumentTableColumnDetails { + pub index: u32, + pub header: String, + pub alignment: TableAlignment, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, +} + +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TableAlignment { + Left, + Center, + Right, + Unspecified, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct DocumentTableDetails { + pub columns: Vec, + pub body_row_count: u32, + #[serde(default)] + pub omitted_row_count: u32, + #[serde(default)] + pub omitted_column_count: u32, + #[serde(default)] + pub truncated: bool, +} + +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TableCellState { + Present, + Empty, + Missing, + /// The cell exists, but its text was omitted by a bounded extractor. + Limited, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct DocumentTableCellDetails { + pub column_index: u32, + pub state: TableCellState, + pub text: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub source: Option, +} + +#[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct DocumentTableRowDetails { + pub row_index: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub identity_cell_index: Option, + pub cells: Vec, + #[serde(default)] + pub truncated: bool, +} + +/// Outcome of resolving a reference embedded in a document. +/// +/// This is intentionally separate from the code resolver's `ResolutionState`: +/// document probing has an additional bounded/limited outcome, while widening +/// the shared resolver enum would make every existing exhaustive resolver +/// branch silently reinterpret a limit as a normal unresolved result. +#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum DocumentReferenceResolution { + Exact, + Ambiguous, + Unresolved, + Limited, +} + +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct DocumentReferenceEvidence { + pub spelling: String, + pub kind: String, + pub site: SourceAnchor, + pub resolution: DocumentReferenceResolution, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub target: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub candidates: Vec, +} + +/// Bounded document normalization facts. +/// +/// These facts are used while publishing stable graph/1 identities and +/// references. The graph/1 validator rejects this details discriminant on a +/// published artifact; document nodes are downgraded to `Resource(document)` +/// after their structural nodes and relationships have been derived. +#[derive(Clone, Debug, PartialEq, Serialize, Deserialize)] +#[serde(rename_all = "camelCase", deny_unknown_fields)] +pub struct DocumentNodeDetails { + pub format: DocumentFormat, + pub role: DocumentRole, + pub ordinal: u32, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub section: Option, + /// Stable document fragment, when the source construct declares one. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub uri: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub content: Option, + pub significance: DocumentSignificance, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub table: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + pub table_row: Option, + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub references: Vec, +} + #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] #[serde(rename_all = "camelCase", deny_unknown_fields)] pub struct ResourceNodeDetails { @@ -569,6 +736,7 @@ pub enum NodeDetails { ImportExport(ImportExportNodeDetails), Route(RouteNodeDetails), Component(ComponentNodeDetails), + Document(DocumentNodeDetails), Resource(ResourceNodeDetails), Messaging(MessagingNodeDetails), Job(JobNodeDetails), @@ -876,6 +1044,7 @@ impl NodeRecord { self.details .as_ref() .and_then(|details| match details { + NodeDetails::Document(_) => Some("document"), NodeDetails::Resource(details) => { Some(resource_kind_str(details.resource_kind)) } @@ -887,6 +1056,112 @@ impl NodeRecord { } .to_owned(), )), + "document_format" => self.document_details().map(|details| { + serde_json::to_value(details.format) + .unwrap_or_else(|_| Value::String("other".to_owned())) + }), + "document_kind" | "document_role" => self.document_details().map(|details| { + serde_json::to_value(details.role) + .unwrap_or_else(|_| Value::String("other".to_owned())) + }), + "document_section" => self + .document_details() + .and_then(|details| details.section.clone()) + .map(Value::String), + "uri" => self + .document_details() + .and_then(|details| details.uri.clone()) + .or_else(|| { + self.details.as_ref().and_then(|details| match details { + NodeDetails::Resource(details) => details.uri.clone(), + _ => None, + }) + }) + .map(Value::String), + "document_content" => self + .document_details() + .and_then(|details| details.content.clone()) + .map(Value::String), + "document_significance" => self.document_details().map(|details| { + serde_json::to_value(details.significance) + .unwrap_or_else(|_| Value::String("content".to_owned())) + }), + "document_ordinal" => self + .document_details() + .map(|details| Value::from(details.ordinal)), + "table_columns" => self.document_details().and_then(|details| { + details + .table + .as_ref() + .map(|table| serde_json::to_value(&table.columns).unwrap_or(Value::Null)) + }), + "table_headers" => self.document_details().and_then(|details| { + details.table.as_ref().map(|table| { + Value::Array( + table + .columns + .iter() + .map(|column| Value::String(column.header.clone())) + .collect(), + ) + }) + }), + "table_alignments" => self.document_details().and_then(|details| { + details.table.as_ref().map(|table| { + Value::Array( + table + .columns + .iter() + .map(|column| { + serde_json::to_value(column.alignment).unwrap_or(Value::Null) + }) + .collect(), + ) + }) + }), + "table_body_row_count" => self + .document_details() + .and_then(|details| details.table.as_ref()) + .map(|table| Value::from(table.body_row_count)), + "table_omitted_row_count" => self + .document_details() + .and_then(|details| details.table.as_ref()) + .map(|table| Value::from(table.omitted_row_count)), + "table_omitted_column_count" => self + .document_details() + .and_then(|details| details.table.as_ref()) + .map(|table| Value::from(table.omitted_column_count)), + "table_truncated" => self.document_details().and_then(|details| { + details + .table + .as_ref() + .map(|table| Value::Bool(table.truncated)) + .or_else(|| { + details + .table_row + .as_ref() + .map(|row| Value::Bool(row.truncated)) + }) + }), + "table_row_index" => self + .document_details() + .and_then(|details| details.table_row.as_ref()) + .map(|row| Value::from(row.row_index)), + "table_identity_cell_index" => self + .document_details() + .and_then(|details| details.table_row.as_ref()) + .and_then(|row| row.identity_cell_index) + .map(Value::from), + "table_cells" => self.document_details().and_then(|details| { + details + .table_row + .as_ref() + .map(|row| serde_json::to_value(&row.cells).unwrap_or(Value::Null)) + }), + "document_references" => self.document_details().and_then(|details| { + (!details.references.is_empty()) + .then(|| serde_json::to_value(&details.references).unwrap_or(Value::Null)) + }), "language" => self.language.clone().map(Value::String), "framework" => self.framework.clone().map(Value::String), "source_file" => self @@ -986,6 +1261,19 @@ impl NodeRecord { _ => None, } } + + fn document_details(&self) -> Option<&DocumentNodeDetails> { + match self.details.as_ref()? { + NodeDetails::Document(details) => Some(details), + _ => None, + } + } + + /// Return the validated significance profile for document nodes. + #[must_use] + pub fn document_significance(&self) -> Option { + self.document_details().map(|details| details.significance) + } } const NODE_PROPERTY_KEYS: &[&str] = &[ @@ -995,6 +1283,25 @@ const NODE_PROPERTY_KEYS: &[&str] = &[ "kind", "roles", "file_type", + "document_format", + "document_kind", + "document_role", + "document_section", + "uri", + "document_content", + "document_significance", + "document_ordinal", + "table_columns", + "table_headers", + "table_alignments", + "table_body_row_count", + "table_omitted_row_count", + "table_omitted_column_count", + "table_truncated", + "table_row_index", + "table_identity_cell_index", + "table_cells", + "document_references", "language", "framework", "source_file", @@ -1477,12 +1784,114 @@ fn legacy_node_record(node: &NodeRecord) -> Result, + details: &DocumentNodeDetails, +) -> Result<(), GraphError> { + attributes.insert("file_type".to_owned(), Value::String("document".to_owned())); + attributes.insert( + "document_format".to_owned(), + serialize_json_value(&details.format)?, + ); + attributes.insert( + "document_kind".to_owned(), + serialize_json_value(&details.role)?, + ); + attributes.insert( + "document_role".to_owned(), + serialize_json_value(&details.role)?, + ); + attributes.insert("block_index".to_owned(), Value::from(details.ordinal)); + attributes.insert("document_ordinal".to_owned(), Value::from(details.ordinal)); + if let Some(section) = &details.section { + attributes.insert( + "document_section".to_owned(), + Value::String(section.clone()), + ); + } + if let Some(uri) = &details.uri { + attributes.insert("uri".to_owned(), Value::String(uri.clone())); + } + if let Some(content) = &details.content { + attributes.insert( + "document_content".to_owned(), + Value::String(content.clone()), + ); + } + attributes.insert( + "document_significance".to_owned(), + serialize_json_value(&details.significance)?, + ); + if let Some(table) = &details.table { + attributes.insert( + "table_columns".to_owned(), + serialize_json_value(&table.columns)?, + ); + attributes.insert( + "table_headers".to_owned(), + Value::Array( + table + .columns + .iter() + .map(|column| Value::String(column.header.clone())) + .collect(), + ), + ); + attributes.insert( + "table_alignments".to_owned(), + Value::Array( + table + .columns + .iter() + .map(|column| serialize_json_value(&column.alignment)) + .collect::, _>>()?, + ), + ); + attributes.insert( + "table_body_row_count".to_owned(), + Value::from(table.body_row_count), + ); + attributes.insert( + "table_omitted_row_count".to_owned(), + Value::from(table.omitted_row_count), + ); + attributes.insert( + "table_omitted_column_count".to_owned(), + Value::from(table.omitted_column_count), + ); + attributes.insert("table_truncated".to_owned(), Value::Bool(table.truncated)); + } + if let Some(row) = &details.table_row { + attributes.insert("table_row_index".to_owned(), Value::from(row.row_index)); + if let Some(index) = row.identity_cell_index { + attributes.insert("table_identity_cell_index".to_owned(), Value::from(index)); + } + attributes.insert("table_cells".to_owned(), serialize_json_value(&row.cells)?); + attributes.insert("table_truncated".to_owned(), Value::Bool(row.truncated)); + } + if !details.references.is_empty() { + attributes.insert( + "document_references".to_owned(), + serialize_json_value(&details.references)?, + ); + } + Ok(()) +} + +fn serialize_json_value(value: &T) -> Result { + serde_json::to_value(value).map_err(GraphError::Corrupt) +} + fn legacy_edge_record(edge: &EdgeRecord) -> Result { let value = serde_json::to_value(edge).map_err(GraphError::Corrupt)?; let mut object = value.as_object().cloned().ok_or_else(|| { @@ -1508,6 +1917,40 @@ fn legacy_edge_record(edge: &EdgeRecord) -> Result Result { let mut object = node.attributes; + // `legacy_node_record` adds flat document aliases for compatibility + // consumers. They are deliberately not part of the strict typed DTO, so + // discard only those aliases when reconstructing typed records. + let typed_document = object + .get("details") + .and_then(Value::as_object) + .is_some_and(|details| details.get("type").and_then(Value::as_str) == Some("document")); + if typed_document { + for key in [ + "file_type", + "document_format", + "document_kind", + "document_role", + "block_index", + "document_ordinal", + "document_section", + "uri", + "document_content", + "document_significance", + "table_columns", + "table_headers", + "table_alignments", + "table_body_row_count", + "table_omitted_row_count", + "table_omitted_column_count", + "table_truncated", + "table_row_index", + "table_identity_cell_index", + "table_cells", + "document_references", + ] { + object.remove(key); + } + } object.insert("id".to_owned(), Value::String(node.id)); serde_json::from_value(Value::Object(object)).map_err(GraphError::Corrupt) } diff --git a/crates/compass-model/src/document.rs b/crates/compass-model/src/document.rs index 1dca679fc..bd7210d43 100644 --- a/crates/compass-model/src/document.rs +++ b/crates/compass-model/src/document.rs @@ -20,6 +20,85 @@ pub struct NodeRecord { } impl NodeRecord { + /// Return the normalized document role when this compatibility node has + /// document semantics. Legacy parser-shaped roles are kept here as a + /// read-only adaptation detail so graph consumers do not duplicate syntax + /// vocabulary or treat it as product meaning. + #[must_use] + pub fn document_role(&self) -> Option<&str> { + self.attributes + .get("document_kind") + .and_then(Value::as_str) + .or_else(|| { + let qualified_name = self + .attributes + .get("qualified_name") + .or_else(|| self.attributes.get("qualifiedName")) + .and_then(Value::as_str)?; + if qualified_name.contains("::pipe_table_cell#") { + Some("pipe_table_cell") + } else if qualified_name.contains("::pipe_table_header#") { + Some("pipe_table_header") + } else if qualified_name.contains("::pipe_table_row#") { + Some("pipe_table_row") + } else if qualified_name.contains("::pipe_table#") { + Some("pipe_table") + } else { + None + } + }) + } + + /// Return the derived document significance used by consumer profiles. + /// Legacy records may not carry the field, so derive the same conservative + /// fallback as the typed graph adapter. + #[must_use] + pub fn document_significance(&self) -> Option { + let role = self.document_role()?; + let explicit = self + .attributes + .get("document_significance") + .and_then(Value::as_str); + Some(match explicit { + Some("container") => crate::code_graph::DocumentSignificance::Container, + Some("scaffolding") => crate::code_graph::DocumentSignificance::Scaffolding, + Some("content") => crate::code_graph::DocumentSignificance::Content, + _ => match role { + "list" | "block_quote" | "quote" | "table" => { + crate::code_graph::DocumentSignificance::Container + } + "link_reference_definition" | "footnote_definition" => { + crate::code_graph::DocumentSignificance::Scaffolding + } + _ => crate::code_graph::DocumentSignificance::Content, + }, + }) + } + + /// Whether this node is one of the historical Markdown parser scaffolding + /// records. In strict graph/1 projections the role is recovered from the + /// extractor-owned qualified-name grammar rather than a new wire field. + #[must_use] + pub fn is_legacy_table_scaffolding(&self) -> bool { + matches!( + self.document_role(), + Some("pipe_table" | "pipe_table_header" | "pipe_table_row" | "pipe_table_cell") + ) + } + + /// Whether this node is a compact semantic table or body-row record. + #[must_use] + pub fn is_semantic_table_structure(&self) -> bool { + matches!(self.document_role(), Some("table" | "table_row")) + } + + /// Whether this node should stay out of architecture/topology summaries + /// while remaining available for navigation and detail inspection. + #[must_use] + pub fn is_table_navigation_node(&self) -> bool { + self.is_legacy_table_scaffolding() || self.is_semantic_table_structure() + } + #[must_use] pub fn string(&self, key: &str) -> String { self.attributes @@ -1807,7 +1886,9 @@ fn insert_optional_value(attributes: &mut Map, key: &str, value: mod tests { use std::fs; - use super::{GraphDocument, affected_cache_path, query_cache_path, traversal_cache_path}; + use super::{ + GraphDocument, NodeRecord, affected_cache_path, query_cache_path, traversal_cache_path, + }; #[test] fn omitted_multigraph_uses_networkx_legacy_default() { @@ -1824,6 +1905,18 @@ mod tests { assert!(!document.multigraph); } + #[test] + fn graph_v1_markdown_table_roles_are_derived_from_qualified_identity() { + let node: NodeRecord = serde_json::from_value(serde_json::json!({ + "id": "cell", + "name": "Owner: compass-model", + "qualifiedName": "Ownership::pipe_table#1::pipe_table_row#graph-1::pipe_table_cell#2" + })) + .unwrap_or_else(|_| std::process::abort()); + assert_eq!(node.document_role(), Some("pipe_table_cell")); + assert!(node.is_table_navigation_node()); + } + #[test] fn typed_records_project_compatibility_source_locations() -> Result<(), Box> { diff --git a/crates/compass-model/src/validation.rs b/crates/compass-model/src/validation.rs index 3d0ff1347..40baff7d7 100644 --- a/crates/compass-model/src/validation.rs +++ b/crates/compass-model/src/validation.rs @@ -4,15 +4,21 @@ use std::fmt::Write as _; use rayon::prelude::*; use serde_json::Value; +use crate::code_graph::DocumentReferenceResolution; use crate::code_graph::{ - CODE_GRAPH_SCHEMA_V1, EdgeKind, GraphDocument as CodeGraphDocument, NodeDetails, NodeKind, - NodeRole, + CODE_GRAPH_SCHEMA_V1, DocumentRole, DocumentSignificance, EdgeKind, + GraphDocument as CodeGraphDocument, NodeDetails, NodeKind, NodeRole, TableCellState, }; use crate::identity::{edge_id, file_id}; use crate::provenance::{Provenance, SourceAnchor}; const VALID_FILE_TYPES: [&str; 6] = ["code", "concept", "document", "image", "paper", "rationale"]; const VALID_CONFIDENCES: [&str; 3] = ["AMBIGUOUS", "EXTRACTED", "INFERRED"]; +const MAX_DOCUMENT_TABLE_COLUMNS: usize = 128; +const MAX_DOCUMENT_TABLE_ROWS: usize = 10_000; +const MAX_DOCUMENT_TABLE_CELLS: usize = 100_000; +const MAX_DOCUMENT_TEXT_BYTES: usize = 4 * 1024; +const MAX_DOCUMENT_REFERENCES: usize = 128; // CPython's set iteration order with the compatibility harness' PYTHONHASHSEED=0. const REQUIRED_NODE_FIELDS: [&str; 4] = ["file_type", "id", "source_file", "label"]; @@ -296,6 +302,33 @@ pub fn validate_code_graph_records(document: &CodeGraphDocument) -> CodeGraphVal node.kind.as_str() )); } + if let Some(NodeDetails::Document(details)) = node.details.as_ref() { + validate_document_details(&node.id, details, node.source.as_ref(), &files, &mut errors); + for reference in &details.references { + if let Some(target) = reference.target.as_ref() + && !nodes.contains_key(target.as_str()) + { + errors.push(format!( + "node {} document reference target {} is not a published node", + node.id, target + )); + } + for candidate in &reference.candidates { + if !nodes.contains_key(candidate.node_id.as_str()) { + errors.push(format!( + "node {} document reference candidate {} is not a published node", + node.id, candidate.node_id + )); + } + } + } + } + if matches!(node.details.as_ref(), Some(NodeDetails::Document(_))) { + errors.push(format!( + "node {} carries normalization-only document details in compass.graph/1", + node.id + )); + } if !errors.is_empty() { Some(RecordValidationErrors { id: node.id.clone(), @@ -548,6 +581,7 @@ fn details_match_kind(kind: NodeKind, details: Option<&NodeDetails>) -> bool { } Some(NodeDetails::Route(_)) => kind == NodeKind::Route, Some(NodeDetails::Component(_)) => kind == NodeKind::Component, + Some(NodeDetails::Document(_)) => kind == NodeKind::Resource, Some(NodeDetails::Resource(_)) => kind == NodeKind::Resource, Some(NodeDetails::Messaging(_)) => matches!( kind, @@ -572,6 +606,307 @@ fn details_match_kind(kind: NodeKind, details: Option<&NodeDetails>) -> bool { } } +fn validate_document_details( + owner: &str, + details: &crate::code_graph::DocumentNodeDetails, + node_source: Option<&SourceAnchor>, + files: &HashMap<&str, u64>, + errors: &mut Vec, +) { + let expected_significance = match details.role { + DocumentRole::List | DocumentRole::Quote | DocumentRole::Table => { + DocumentSignificance::Container + } + DocumentRole::LinkDefinition | DocumentRole::FootnoteDefinition => { + DocumentSignificance::Scaffolding + } + _ => DocumentSignificance::Content, + }; + if details.significance != expected_significance { + errors.push(format!( + "{owner}: document significance {:?} does not match role {:?}", + details.significance, details.role + )); + } + + match details.role { + DocumentRole::Table => { + if details.table.is_none() || details.table_row.is_some() { + errors.push(format!( + "{owner}: table document details require table payload only" + )); + } + } + DocumentRole::TableRow => { + if details.table_row.is_none() || details.table.is_some() { + errors.push(format!( + "{owner}: table row document details require table_row payload only" + )); + } + } + _ => { + if details.table.is_some() || details.table_row.is_some() { + errors.push(format!( + "{owner}: non-table document role carries table payload" + )); + } + } + } + + if let Some(table) = &details.table { + if table.columns.len() > MAX_DOCUMENT_TABLE_COLUMNS { + errors.push(format!( + "{owner}: table has {} columns, maximum is {MAX_DOCUMENT_TABLE_COLUMNS}", + table.columns.len() + )); + } + if table.body_row_count > MAX_DOCUMENT_TABLE_ROWS as u32 { + errors.push(format!( + "{owner}: table has {} retained rows, maximum is {MAX_DOCUMENT_TABLE_ROWS}", + table.body_row_count + )); + } + if table.omitted_row_count > MAX_DOCUMENT_TABLE_ROWS as u32 { + errors.push(format!( + "{owner}: table omitted row count exceeds {MAX_DOCUMENT_TABLE_ROWS}" + )); + } + if table.omitted_column_count > MAX_DOCUMENT_TABLE_COLUMNS as u32 { + errors.push(format!( + "{owner}: table omitted column count exceeds {MAX_DOCUMENT_TABLE_COLUMNS}" + )); + } + if !table.truncated && (table.omitted_row_count != 0 || table.omitted_column_count != 0) { + errors.push(format!( + "{owner}: table omission counts require truncated=true" + )); + } + for (index, column) in table.columns.iter().enumerate() { + if column.index != index as u32 { + errors.push(format!( + "{owner}: table column index {} is not contiguous at position {index}", + column.index + )); + } + if column.header.len() > MAX_DOCUMENT_TEXT_BYTES { + errors.push(format!( + "{owner}: table column {index} header exceeds {MAX_DOCUMENT_TEXT_BYTES} bytes" + )); + } + if let Some(anchor) = &column.source { + validate_document_anchor(owner, anchor, node_source, files, errors); + } + } + } + + if let Some(row) = &details.table_row { + if row.cells.len() > MAX_DOCUMENT_TABLE_COLUMNS + || row.cells.len() > MAX_DOCUMENT_TABLE_CELLS + { + errors.push(format!( + "{owner}: table row carries too many cells ({}), maximum is {MAX_DOCUMENT_TABLE_COLUMNS}", + row.cells.len() + )); + } + let mut previous_index = None; + for cell in &row.cells { + if cell.column_index >= MAX_DOCUMENT_TABLE_COLUMNS as u32 { + errors.push(format!( + "{owner}: table cell column index {} exceeds the maximum", + cell.column_index + )); + } + if previous_index.is_some_and(|previous| previous >= cell.column_index) { + errors.push(format!( + "{owner}: table cell column indexes are not strictly increasing" + )); + } + previous_index = Some(cell.column_index); + if cell.text.len() > MAX_DOCUMENT_TEXT_BYTES { + errors.push(format!( + "{owner}: table cell {} exceeds {MAX_DOCUMENT_TEXT_BYTES} bytes", + cell.column_index + )); + } + match cell.state { + TableCellState::Present if cell.text.is_empty() => errors.push(format!( + "{owner}: present table cell {} has empty text", + cell.column_index + )), + TableCellState::Empty | TableCellState::Missing | TableCellState::Limited + if !cell.text.is_empty() => + { + errors.push(format!( + "{owner}: non-present table cell {} carries text", + cell.column_index + )) + } + _ => {} + } + if let Some(anchor) = &cell.source { + validate_document_anchor(owner, anchor, node_source, files, errors); + } + } + if row + .identity_cell_index + .is_some_and(|index| index >= MAX_DOCUMENT_TABLE_COLUMNS as u32) + { + errors.push(format!( + "{owner}: table identity cell index exceeds the maximum" + )); + } + if let Some(identity_index) = row.identity_cell_index { + let identity_cell = row + .cells + .iter() + .find(|cell| cell.column_index == identity_index); + let identity_is_present = identity_cell + .is_some_and(|cell| cell.state == TableCellState::Present && !cell.text.is_empty()); + if !identity_is_present && !row.truncated { + errors.push(format!( + "{owner}: table identity cell must reference a present non-empty cell" + )); + } + } + } + + if details + .section + .as_ref() + .is_some_and(|section| section.len() > MAX_DOCUMENT_TEXT_BYTES) + { + errors.push(format!( + "{owner}: document section exceeds {MAX_DOCUMENT_TEXT_BYTES} bytes" + )); + } + if details + .uri + .as_ref() + .is_some_and(|uri| uri.len() > MAX_DOCUMENT_TEXT_BYTES) + { + errors.push(format!( + "{owner}: document URI exceeds {MAX_DOCUMENT_TEXT_BYTES} bytes" + )); + } + + if details.references.len() > MAX_DOCUMENT_REFERENCES { + errors.push(format!( + "{owner}: document references exceed {MAX_DOCUMENT_REFERENCES}" + )); + } + for reference in &details.references { + if reference.spelling.len() > MAX_DOCUMENT_TEXT_BYTES { + errors.push(format!( + "{owner}: document reference spelling exceeds {MAX_DOCUMENT_TEXT_BYTES} bytes" + )); + } + if reference.kind.len() > MAX_DOCUMENT_TEXT_BYTES { + errors.push(format!( + "{owner}: document reference kind exceeds {MAX_DOCUMENT_TEXT_BYTES} bytes" + )); + } + if reference + .target + .as_ref() + .is_some_and(|target| target.is_empty()) + { + errors.push(format!( + "{owner}: document reference target must not be empty" + )); + } + if reference.target.is_some() && !reference.candidates.is_empty() { + errors.push(format!( + "{owner}: document reference cannot carry both target and candidates" + )); + } + match reference.resolution { + DocumentReferenceResolution::Exact + if reference.target.is_none() || !reference.candidates.is_empty() => + { + errors.push(format!( + "{owner}: exact document reference requires only a target" + )) + } + DocumentReferenceResolution::Ambiguous + if reference.target.is_some() || reference.candidates.is_empty() => + { + errors.push(format!( + "{owner}: ambiguous document reference requires candidates and no target" + )) + } + DocumentReferenceResolution::Unresolved + if reference.target.is_some() || !reference.candidates.is_empty() => + { + errors.push(format!( + "{owner}: unresolved document reference cannot carry target candidates" + )); + } + DocumentReferenceResolution::Limited + if reference.target.is_some() || !reference.candidates.is_empty() => + { + errors.push(format!( + "{owner}: limited document reference cannot carry target candidates" + )); + } + _ => {} + } + let candidate_keys = reference + .candidates + .iter() + .map(|candidate| { + ( + candidate.node_id.as_str(), + candidate.reason.as_str(), + candidate.confidence.as_str(), + ) + }) + .collect::>(); + if candidate_keys.windows(2).any(|pair| pair[0] >= pair[1]) { + errors.push(format!( + "{owner}: document reference candidates must be sorted and unique" + )); + } + for candidate in &reference.candidates { + if candidate.node_id.is_empty() || candidate.reason.len() > MAX_DOCUMENT_TEXT_BYTES { + errors.push(format!( + "{owner}: document reference candidate is empty or exceeds the bounded limit" + )); + } + } + validate_document_anchor(owner, &reference.site, node_source, files, errors); + for candidate in &reference.candidates { + if let Some(anchor) = &candidate.anchor { + validate_document_anchor(owner, anchor, node_source, files, errors); + } + } + } +} + +fn validate_document_anchor( + owner: &str, + anchor: &SourceAnchor, + node_source: Option<&SourceAnchor>, + files: &HashMap<&str, u64>, + errors: &mut Vec, +) { + validate_anchor(owner, anchor, files, errors); + if let Some(node_source) = node_source + && !source_anchor_contains(node_source, anchor) + { + errors.push(format!( + "{owner}: nested document source anchor is outside the node source" + )); + } +} + +fn source_anchor_contains(outer: &SourceAnchor, inner: &SourceAnchor) -> bool { + outer.file == inner.file + && outer.start_byte <= inner.start_byte + && inner.end_byte <= outer.end_byte + && (outer.start_line, outer.start_column) <= (inner.start_line, inner.start_column) + && (inner.end_line, inner.end_column) <= (outer.end_line, outer.end_column) +} + fn endpoint_kinds_are_valid( source: &crate::code_graph::NodeRecord, kind: EdgeKind, diff --git a/crates/compass-model/tests/code_graph_loading.rs b/crates/compass-model/tests/code_graph_loading.rs index 4c1776801..5948da14d 100644 --- a/crates/compass-model/tests/code_graph_loading.rs +++ b/crates/compass-model/tests/code_graph_loading.rs @@ -153,15 +153,73 @@ fn strict_loading_rejects_pre_contract_and_unknown_graphs() -> Result<(), Box Result<(), Box> { + let directory = tempfile::tempdir()?; + let graph_path = directory.path().join("graph.json"); + let build = BuildMetadata { + builder_version: "test".to_owned(), + schema_fingerprint: "schema".to_owned(), + source_tree_digest: "tree".to_owned(), + configuration_digest: "config".to_owned(), + generation_id: "generation".to_owned(), + source_commit: None, + }; + fs::write( + &graph_path, + serde_json::to_vec(&GraphDocument::empty_v1(build))?, + )?; + let loaded = GraphDocument::load(&graph_path)?; + assert_eq!(loaded.graph.schema, "compass.graph/1"); + Ok(()) +} + +#[test] +fn graph_v1_rejects_normalization_only_document_details() -> Result<(), Box> +{ + let directory = tempfile::tempdir()?; + let graph_path = directory.path().join("graph.json"); + let mut value = serde_json::to_value(document())?; + value["nodes"] = serde_json::json!([{ + "id": "document:normalization-only", + "kind": "resource", + "name": "Heading", + "qualifiedName": "Guide::heading#heading", + "details": { + "type": "document", + "data": { + "format": "markdown", + "role": "heading", + "ordinal": 0, + "significance": "content" + } + }, + "evidence": [] + }]); + fs::write(&graph_path, serde_json::to_vec(&value)?)?; + + let error = match GraphDocument::load(&graph_path) { + Ok(_) => return Err("normalization-only document details were published".into()), + Err(error) => error, + }; + assert!( + error + .to_string() + .contains("normalization-only document details in compass.graph/1"), + "{error}" + ); + Ok(()) +} + #[test] fn exact_recluster_loading_does_not_require_a_json_extension() -> Result<(), Box> { diff --git a/crates/compass-output/src/html.rs b/crates/compass-output/src/html.rs index 2557fdf77..b58395697 100644 --- a/crates/compass-output/src/html.rs +++ b/crates/compass-output/src/html.rs @@ -743,24 +743,32 @@ fn source_anchor_unsigned(node: &NodeRecord, legacy_key: &str, v1_key: &str) -> fn document_value(node: &NodeRecord) -> Option { const DOCUMENT_VIEWER_TEXT_LIMIT: usize = 4_096; - let is_document = - node.string("file_type") == "document" || node.property("document_kind").is_some(); + let is_document = node + .property("file_type") + .and_then(|value| value.as_str().map(|value| value == "document")) + .unwrap_or(false) + || node.document_role().is_some(); if !is_document { return None; } - let kind = if node.string("document_kind").is_empty() { - node.string("symbol_kind") - } else { - node.string("document_kind") + let direct_string = |key: &str| { + node.property(key) + .and_then(|value| value.as_str().map(str::to_owned)) }; + let kind = node + .document_role() + .map(str::to_owned) + .unwrap_or_else(|| node.string("symbol_kind")); let role = if kind == "document" { "root" } else { "block" }; let mut document = Map::new(); document.insert("role".into(), Value::String(role.to_owned())); if !kind.is_empty() { document.insert("kind".into(), Value::String(kind)); } + if let Some(value) = direct_string("document_format").map(Value::String) { + document.insert("format".to_owned(), value); + } for (target, source) in [ - ("format", "document_format"), ("visualCoverage", "document_visual_coverage"), ("ocrMode", "document_ocr_mode"), ] { @@ -771,9 +779,7 @@ fn document_value(node: &NodeRecord) -> Option { document.insert(target.to_owned(), value); } } - if let Some(text) = node - .property("document_text") - .and_then(|value| value.as_str().map(str::to_owned)) + if let Some(text) = direct_string("document_text").or_else(|| direct_string("document_content")) { document.insert( "text".to_owned(), @@ -1237,7 +1243,22 @@ fn degrees(document: &GraphDocument) -> HashMap<&str, usize> { .iter() .map(|node| (node.id.as_str(), 0)) .collect::>(); + let positions = document + .nodes + .iter() + .map(|node| (node.id.as_str(), node)) + .collect::>(); for edge in &document.links { + if edge.relation() == "contains" + && (positions + .get(edge.source.as_str()) + .is_some_and(|node| node.is_table_navigation_node()) + || positions + .get(edge.target.as_str()) + .is_some_and(|node| node.is_table_navigation_node())) + { + continue; + } *degrees.entry(edge.source.as_str()).or_default() += 1; *degrees.entry(edge.target.as_str()).or_default() += 1; } @@ -2698,6 +2719,29 @@ mod tests { Ok(()) } + #[test] + fn document_value_recovers_markdown_role_from_graph_v1_identity() -> Result<(), Box> + { + let cell: NodeRecord = serde_json::from_value(json!({ + "id": "cell", + "kind": "resource", + "name": "Area: Graph", + "qualifiedName": "Guide::pipe_table#1::pipe_table_row#graph-1::pipe_table_cell#1", + "details": { + "type": "resource", + "data": { + "resourceKind": "document", + "uri": "docs/guide.md#L4C1-L4C8", + "mediaType": "text/markdown" + } + } + }))?; + let value = document_value(&cell).ok_or("graph-v1 document omitted")?; + assert_eq!(value["role"], json!("block")); + assert_eq!(value["kind"], json!("pipe_table_cell")); + Ok(()) + } + #[test] fn html_inspector_is_accessible_and_responsive() -> Result<(), Box> { let graph: GraphDocument = serde_json::from_value(json!({ diff --git a/crates/compass-output/src/report.rs b/crates/compass-output/src/report.rs index fd1de2009..b0f8d31cc 100644 --- a/crates/compass-output/src/report.rs +++ b/crates/compass-output/src/report.rs @@ -448,7 +448,7 @@ pub fn agent_orientation( let hub_total = god_node_list.len(); let hubs = god_node_list .iter() - .filter(|node| !graph.is_pipe_table_structure_node_id(&node.id)) + .filter(|node| !graph.is_document_table_node_id(&node.id)) .filter(|node| graph.node_identity_and_anchor_are_safe(&node.id, &node.label)) .take(HUB_LIMIT) .map(|node| build_hub(&graph, node, &node_communities)) @@ -1003,13 +1003,13 @@ fn build_communities( (!real.is_empty()).then_some(( *community, real.iter() - .all(|member| graph.is_pipe_table_structure_node_id(member)), + .all(|member| graph.is_document_table_node_id(member)), real, )) }) .collect::>(); let total = eligible.len(); - eligible.retain(|(_, pipe_table_only, _)| !pipe_table_only); + eligible.retain(|(_, document_table_only, _)| !document_table_only); eligible.sort_by(|(left_id, _, left_members), (right_id, _, right_members)| { right_members .len() @@ -1042,8 +1042,8 @@ fn build_communities( // when two candidates have equal connectivity. representatives.sort_by(|left, right| { graph - .is_pipe_table_structure_node_id(&left.id) - .cmp(&graph.is_pipe_table_structure_node_id(&right.id)) + .is_document_table_node_id(&left.id) + .cmp(&graph.is_document_table_node_id(&right.id)) .then_with(|| graph.degree(&right.id).cmp(&graph.degree(&left.id))) }); representatives.truncate(if detailed { @@ -1184,7 +1184,7 @@ fn build_risks( .filter(|node| { graph.degree(&node.id) <= 1 && !graph.is_file_node_id(&node.id) - && !graph.is_pipe_table_structure_node_id(&node.id) + && !graph.is_document_table_node_id(&node.id) && !is_concept_node(node) && node.string("file_type") != "rationale" }) @@ -1193,15 +1193,15 @@ fn build_risks( .values() .filter(|members| { let mut count = 0_usize; - let mut pipe_table_only = true; + let mut document_table_only = true; for member in *members { if graph.is_file_node_id(member) { continue; } count = count.saturating_add(1); - pipe_table_only &= graph.is_pipe_table_structure_node_id(member); + document_table_only &= graph.is_document_table_node_id(member); } - count > 0 && count < min_size && !pipe_table_only + count > 0 && count < min_size && !document_table_only }) .count(); let mut risks = Vec::new(); @@ -2287,7 +2287,7 @@ fn render_orientation_markdown_with_community_limit( "## Architecture Map".to_owned(), "- Leading communities are ranked by member count, connectivity, label, and stable ID. See the complete bounded directory of architecture-relevant communities later in this report." .to_owned(), - "- Markdown communities containing only pipe-table parser blocks are excluded from this architecture view and counted in omitted coverage; their source-backed nodes remain available in the graph." + "- Markdown communities containing only pipe-table parser blocks are excluded from this architecture view and counted in omitted coverage; these are semantic table navigation records whose source-backed nodes remain available in the graph." .to_owned(), disclosure(SectionOmission::from_total_shown( model.omissions.communities.total, @@ -2669,7 +2669,7 @@ fn append_community_directory( "## Community Directory".to_owned(), "- Start here, then use labels with `compass path` or the exact scope with `compass query`. Architecture-relevant communities are ranked by member count, connectivity, label, and stable ID." .to_owned(), - "- Markdown communities containing only pipe-table parser blocks are excluded and counted in omitted coverage; mixed communities retain their meaningful headings or symbols as entry points." + "- Markdown communities containing only pipe-table parser blocks are excluded and counted in omitted coverage; these semantic table navigation records retain source anchors, while mixed communities retain their meaningful headings or symbols as entry points." .to_owned(), disclosure(model.omissions.communities), ]); @@ -2882,9 +2882,12 @@ impl<'a> ReportGraph<'a> { { edge_visits = edge_visits.saturating_add(1); } - *degrees.entry(edge.source.as_str()).or_default() += 1; - if edge.target != edge.source { - *degrees.entry(edge.target.as_str()).or_default() += 1; + let zero_topology = zero_topology_document_containment(edge, &positions); + if !zero_topology { + *degrees.entry(edge.source.as_str()).or_default() += 1; + if edge.target != edge.source { + *degrees.entry(edge.target.as_str()).or_default() += 1; + } } let relation = relation(edge); let confidence = confidence(edge); @@ -2902,34 +2905,36 @@ impl<'a> ReportGraph<'a> { } } } - record_node_connectivity( - node_connectivity.entry(edge.source.as_str()).or_default(), - &relation, - &confidence, - document.directed.then_some(EndpointDirection::Outgoing), - ); - if edge.target == edge.source { - if document.directed { - node_connectivity - .entry(edge.source.as_str()) - .or_default() - .incoming += 1; - } - } else { + if !zero_topology { record_node_connectivity( - node_connectivity.entry(edge.target.as_str()).or_default(), + node_connectivity.entry(edge.source.as_str()).or_default(), &relation, &confidence, - document.directed.then_some(EndpointDirection::Incoming), + document.directed.then_some(EndpointDirection::Outgoing), + ); + if edge.target == edge.source { + if document.directed { + node_connectivity + .entry(edge.source.as_str()) + .or_default() + .incoming += 1; + } + } else { + record_node_connectivity( + node_connectivity.entry(edge.target.as_str()).or_default(), + &relation, + &confidence, + document.directed.then_some(EndpointDirection::Incoming), + ); + } + record_community_connectivity( + &mut community_connectivity, + node_communities.get(edge.source.as_str()).copied(), + node_communities.get(edge.target.as_str()).copied(), + &relation, + document.directed, ); } - record_community_connectivity( - &mut community_connectivity, - node_communities.get(edge.source.as_str()).copied(), - node_communities.get(edge.target.as_str()).copied(), - &relation, - document.directed, - ); } Self { nodes: &document.nodes, @@ -3005,14 +3010,26 @@ impl<'a> ReportGraph<'a> { || (label.ends_with("()") && self.degree(id) <= 1) } - fn is_pipe_table_structure_node_id(&self, id: &str) -> bool { - self.positions.get(id).is_some_and(|node| { - matches!( - node.string("document_kind").as_str(), - "pipe_table" | "pipe_table_header" | "pipe_table_row" | "pipe_table_cell" - ) - }) + fn is_document_table_node_id(&self, id: &str) -> bool { + self.positions + .get(id) + .is_some_and(|node| node.is_table_navigation_node()) + } +} + +fn zero_topology_document_containment( + edge: &EdgeRecord, + positions: &HashMap<&str, &NodeRecord>, +) -> bool { + if edge.relation() != "contains" { + return false; } + positions + .get(edge.source.as_str()) + .is_some_and(|node| node.is_table_navigation_node()) + || positions + .get(edge.target.as_str()) + .is_some_and(|node| node.is_table_navigation_node()) } #[derive(Clone, Copy)] diff --git a/crates/compass-query/src/score.rs b/crates/compass-query/src/score.rs index 06949ace2..237957009 100644 --- a/crates/compass-query/src/score.rs +++ b/crates/compass-query/src/score.rs @@ -762,6 +762,38 @@ mod tests { Ok(()) } + #[test] + fn graph_v1_markdown_cell_labels_are_retrievable_in_both_rankers() -> Result<(), Box> + { + let document: GraphDocument = serde_json::from_value(json!({ + "directed": true, + "multigraph": true, + "graph": {}, + "nodes": [ + { + "id": "table-row", + "label": "Owner: compass-model", + "kind": "resource", + "qualifiedName": "Ownership::pipe_table#1::pipe_table_row#graph-1::pipe_table_cell#2" + }, + {"id": "unrelated", "label": "unrelated"} + ], + "links": [] + }))?; + let graph = Graph::from_document(document)?; + for profile in [TextRankProfile::FullScanV1, TextRankProfile::Bm25V1] { + let scores = + score_nodes_with_profile(&graph, &["compass-model".to_owned()], true, profile); + let first = scores + .scores + .ranked + .first() + .ok_or("missing semantic result")?; + assert_eq!(graph.node(first.node).id, "table-row"); + } + Ok(()) + } + #[test] fn query_tier_and_singleton_arithmetic_are_exact() { assert_eq!( diff --git a/docs/design/code-graph-v1-qualification.md b/docs/design/code-graph-v1-qualification.md index 32dc4b930..cbb697222 100644 --- a/docs/design/code-graph-v1-qualification.md +++ b/docs/design/code-graph-v1-qualification.md @@ -65,8 +65,11 @@ The script: 4. compares canonical graph bytes from clean, unchanged warm, forced, and edit-then-restore updates; 5. verifies that the checked-in source fixtures were not changed; -6. executes every semantic assertion over the restored production graph; and -7. prints one canonical JSON summary to standard output. +6. executes every semantic assertion over the restored production graph; +7. derives Markdown table and reference expectations independently from the + fixture source, then checks graph-v1 roles, labels, hierarchy, and exact + anchors; and +8. prints canonical machine summaries to standard output. Oracle self-tests and manifest-only validation can be run independently: diff --git a/docs/design/document-processing.md b/docs/design/document-processing.md index 1660630f5..f081d9105 100644 --- a/docs/design/document-processing.md +++ b/docs/design/document-processing.md @@ -33,7 +33,7 @@ bounded source bytes v compass-languages::Engine | - +--> pinned Markdown block/inline grammars + +--> pinned Markdown block/inline + YAML grammars +--> pinned HTML grammar and shared renderer +--> bounded frontmatter/entity/URL decoders | @@ -54,11 +54,13 @@ source ranges. The standalone compatibility path still accepts a `Path` and reads it once. The Markdown grammars are statically linked through the pinned `tree-sitter-md` -crate and HTML uses the exact pinned `tree-sitter-html` crate. The vendored -language pack remains the owner for the general language registry; the direct -HTML binding is deliberately parser-only because this release's pack build -does not expose an HTML static loader. Neither path downloads a grammar at -runtime, invokes Python, calls a model, or follows a URL. +crate, YAML frontmatter source anchoring uses the pinned statically linked YAML +grammar in the vendored language pack, and HTML uses the exact pinned +`tree-sitter-html` crate. The vendored language pack remains the owner for the +general language registry; the direct HTML binding is deliberately parser-only +because this release's pack build does not expose an HTML static loader. None +of these paths downloads a grammar at runtime, invokes Python, calls a model, +or follows a URL. ## Markdown projection @@ -66,7 +68,8 @@ Every file has one root node with: - `document_format: "markdown"` and `document_kind: "document"`; - the source file and exact whole-document byte/line range; -- deterministic `document_metadata` when bounded frontmatter is valid. +- deterministic nested `document_metadata` when bounded frontmatter is valid; +- source-anchored `config_key` nodes and containment for frontmatter paths. The structural projection emits ordered nodes for headings, paragraphs, lists and list items, block quotes, thematic breaks, fenced and indented code, @@ -89,11 +92,21 @@ published as blocks. Frontmatter is recognized only when the source begins with a whole-line `---` (an optional UTF-8 BOM is accepted) and a whole-line closing delimiter appears -within 64 KiB. It is parsed with the workspace YAML implementation and only -JSON-compatible scalars and bounded scalar arrays are published. Mappings, -aliases, tags, oversized values, and arrays containing non-scalars produce a -bounded diagnostic and do not become graph attributes. Keys are deterministic -and capped at 256 entries; individual strings and arrays are bounded. +within 64 KiB. The workspace YAML implementation validates and normalizes +JSON-compatible scalars, nested mappings, and arrays. A second, statically +linked YAML syntax pass must source-anchor the same canonical paths before any +metadata is accepted. Disagreement, duplicate paths, invalid UTF-8, non-string +keys, aliases, tags, oversized values, or exceeded key/item/depth/node budgets +produce a bounded diagnostic and publish no partial metadata graph. + +The raw root retains the bounded nested `document_metadata` map. Graph-v1 +publication expresses that structure with the existing `ConfigKey` contract: +canonical JSON Pointer key paths, `yaml_frontmatter` format, Config provenance, +stable source-file/path identity, exact pair/item ranges, and nested `contains` +edges. Only an allowlist of semantic content fields receives a value summary in +the node display name; generic and credential-shaped values are never copied +into the public graph. This is producer logic inside `compass.graph/1`, not a +new graph wire field. Frontmatter is metadata, not visible Markdown body text. Body node ranges still point into the original bytes, including CRLF and non-UTF-8 input (labels use a diff --git a/docs/design/security-and-privacy.md b/docs/design/security-and-privacy.md index 35193ba0e..68907beaf 100644 --- a/docs/design/security-and-privacy.md +++ b/docs/design/security-and-privacy.md @@ -228,6 +228,14 @@ Treat graph artifacts with the same or higher classification as the source corpus. Do not upload them to a public artifact store merely because they contain less text than the repository. +Markdown frontmatter is untrusted source. Compass validates it within byte, +key, item, depth, and graph-node budgets; rejects YAML aliases and tags; and +requires parser-backed source ranges before publication. Public ConfigKey +labels include values only for a conservative set of content metadata such as +title, tags, aliases, authors, dates, layout, and status. Generic values, +including credential-shaped fields, remain out of graph artifacts. This is a +disclosure reduction, not permission to store credentials in frontmatter. + HTML and SVG exports must remain self-contained and avoid loading untrusted external scripts/fonts/resources. diff --git a/docs/implementation/universal-evidence.md b/docs/implementation/universal-evidence.md index 269b5bef9..b1de1db26 100644 --- a/docs/implementation/universal-evidence.md +++ b/docs/implementation/universal-evidence.md @@ -29,7 +29,7 @@ future work. | Status | Implementation | | --- | --- | -| Available now | `compass-languages` owns the source registry, parsers, established extractors, and universal evidence schema version 2 (extraction semantics version 3) | +| Available now | `compass-languages` owns the source registry, parsers, established extractors, and universal evidence schema version 2 (extraction semantics version 4) | | Available now | C#, Dart, Go, Groovy, Java, Kotlin, PHP, Python, Ruby, Rust, Scala, Swift, TypeScript, and JavaScript are entries in the hard-cut `UniversalEvidenceRegistry`; each entry pairs a `UniversalEvidenceProducer` with a `UniversalEvidenceQualification` state | | Available now | `EvidenceBuilder` emits bounded `SemanticEvidenceBatch` values for all registered universal languages; Swift, Dart, Scala, and Groovy use direct language modules backed by a shared bounded AST-first traversal, while each retains a distinct version-1 producer identity | | Available now | The language-wave parity profiles preserve quoted Groovy/Spock feature declarations, Dart library namespaces and bounded `part`/`part of` plus import/export selectors, Swift enum/struct/extension/type-alias/member identities, and Scala companion plus import-selector identities without enabling unaudited test or dynamic-dispatch capabilities | diff --git a/docs/reference/document-formats.md b/docs/reference/document-formats.md index 182941a18..90a0a5815 100644 --- a/docs/reference/document-formats.md +++ b/docs/reference/document-formats.md @@ -14,14 +14,14 @@ alone. | Lists and task items | List/list-item blocks; `task_checked` when present | `contains` + exact ranges | | Block quotes and thematic breaks | Structural block | exact node range | | Fenced / indented code | Code block; fenced info string becomes `language` | exact node range | -| Pipe tables | Table, header, row, and cell blocks | nested `contains` edges | +| Pipe tables | Table, header, row, and cell blocks with header-qualified labels | nested `contains` edges and exact cell-owned references | | Reference definitions | Definition block and definition relationship | definition range | | Inline/reference/autolinks | Link relationship with `link_kind` | link-site range | | Footnotes | Bounded definition nodes and reference relationships | exact definition/reference ranges | | Wikilinks | Local link evidence with `link_kind: "wikilink"` | link-site range | | MDX / Quarto extensions | Bounded `other` blocks; no execution | exact source range | | Images | Ignored as document relationships | no fetch or edge | -| Frontmatter | Bounded `document_metadata` map | root metadata; body offsets unchanged | +| Frontmatter | Bounded nested metadata plus `config_key` graph nodes | exact key/value-pair ranges; body offsets unchanged | | Malformed syntax | Recovered evidence plus bounded diagnostic | extraction quality extension | Markdown is parsed from the caller-supplied bytes with statically linked @@ -50,10 +50,29 @@ attributes and must not parse stable IDs as path components. ### Frontmatter limits - opening and closing delimiters must be whole lines within 64 KiB; -- at most 256 metadata keys and 256 scalar-array items are published; +- at most 256 metadata keys and 256 total array items are normalized; +- mappings and arrays can nest to 12 levels, with at most 512 source-backed + metadata graph nodes; - individual metadata keys and strings are capped at 16 KiB; -- nested mappings, YAML tags/aliases, and non-scalar arrays are diagnosed and - omitted rather than projected as arbitrary graph data. +- YAML tags/aliases, duplicate paths, non-string mapping keys, invalid UTF-8, + and unanchorable parser recovery are diagnosed and omitted. + +Valid mappings and object arrays publish established `config_key` nodes under +the Markdown document. `details.format` is `yaml_frontmatter`; `keyPath` is a +canonical JSON Pointer and `qualifiedName` is prefixed with `frontmatter`. +Pointer escaping (`~0` and `~1`) prevents dotted or slash-containing keys from +colliding. Nested `contains` edges and nodes carry exact Config provenance and +source ranges. IDs depend on source file and key path, not the metadata value, +so ordinary value edits do not churn graph identity. + +The document root uses a bounded string `title` as its display name when one is +present. Metadata node labels summarize a conservative set of content fields +such as title, tags, aliases, authors, dates, layout, and status. Generic values +remain out of the public graph label, preventing credential-shaped frontmatter +from being copied into graph artifacts. The full bounded `document_metadata` +map remains available at the raw extraction boundary; `compass.graph/1` +projects its structure through existing typed config nodes rather than adding a +new wire field or schema major. ### Link boundary diff --git a/docs/reference/outputs.md b/docs/reference/outputs.md index ccb9d97d5..a249b0498 100644 --- a/docs/reference/outputs.md +++ b/docs/reference/outputs.md @@ -267,8 +267,8 @@ json` compares these exact IDs across immutable realizations; realizations without the sidecar are counted as observations without graph insights. Community evidence labels prefer a meaningful symbol or document heading over -Markdown pipe-table parser blocks, even when a table container has more -structural edges. A community containing only pipe-table blocks receives a +Markdown semantic table navigation records, even when a table container has more +structural edges. A community containing only table navigation records receives a source-anchored `Table (path:line)` label. When other communities share a hub name, Compass adds a compact source or wiring-site anchor and, only if needed, the graph-local community ID. These labels are deterministic navigation aids, @@ -293,7 +293,7 @@ completeness, overview omissions, and architecture quality are separate signals. The Architecture Map and Community Directory omit communities made entirely -of Markdown pipe-table parser blocks. Those communities count toward the +of Markdown table navigation records. Those communities count toward the report's omitted-community coverage and their source-backed nodes remain in the graph; the report does not present parser partitions as architectural subsystems. diff --git a/docs/reference/universal-semantic-evidence.md b/docs/reference/universal-semantic-evidence.md index 101ed0bba..bb7e66521 100644 --- a/docs/reference/universal-semantic-evidence.md +++ b/docs/reference/universal-semantic-evidence.md @@ -14,7 +14,7 @@ mode, terminal-name fallback, or runtime dependency on Graphify. ## Evidence contract The serialized evidence schema is `compass.languages.evidence/2`. The -extraction cache identity is `compass.languages.extraction/3`; changing either +extraction cache identity is `compass.languages.extraction/4`; changing either major version invalidates older artifacts instead of attempting an implicit translation. diff --git a/fixtures/code-graph/qualification/guide.md b/fixtures/code-graph/qualification/guide.md index 64f15a618..de06f1210 100644 --- a/fixtures/code-graph/qualification/guide.md +++ b/fixtures/code-graph/qualification/guide.md @@ -1,3 +1,17 @@ +--- +title: Qualification Guide +tags: [markdown, graph] +owners: + quality: + team: Compass +api_token: fixture-private-value +--- + # Qualification guide [Runtime source](rich.rs) + +| Area | Owner | Status | +| --- | --- | --- | +| Graph | `compass-model` | active | +| Runtime | [Rust source](rich.rs) | maintained | diff --git a/scripts/markdown_graph_quality_oracle.py b/scripts/markdown_graph_quality_oracle.py new file mode 100755 index 000000000..6889263a3 --- /dev/null +++ b/scripts/markdown_graph_quality_oracle.py @@ -0,0 +1,410 @@ +#!/usr/bin/env python3 +"""Independent structural quality oracle for Markdown in compass.graph/1.""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + + +def fail(invariant: str, identity: str, detail: str) -> None: + raise ValueError(f"{invariant} [{identity}]: {detail}") + + +def load_object(path: Path) -> dict[str, Any]: + value = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(value, dict): + fail("json_object", str(path), "root must be an object") + return value + + +def source_file(node: dict[str, Any]) -> str | None: + source = node.get("source") + return source.get("file") if isinstance(source, dict) else None + + +def source_order(node: dict[str, Any]) -> tuple[int, int, str]: + source = node.get("source") + if not isinstance(source, dict): + return (sys.maxsize, sys.maxsize, str(node.get("id", ""))) + return ( + int(source.get("startByte", sys.maxsize)), + int(source.get("endByte", sys.maxsize)), + str(node.get("id", "")), + ) + + +def terminal_role(node: dict[str, Any]) -> str | None: + qualified = node.get("qualifiedName") + if not isinstance(qualified, str): + return None + terminal = qualified.rsplit("::", 1)[-1] + for role in ("pipe_table", "pipe_table_header", "pipe_table_row", "pipe_table_cell"): + if terminal.startswith(role + "#"): + return role + return None + + +def line_column(source: bytes, position: int) -> tuple[int, int]: + prefix = source[:position] + return prefix.count(b"\n") + 1, len(prefix.rsplit(b"\n", 1)[-1]) + + +def anchor( + node: dict[str, Any], identity: str, sources: dict[str, bytes] +) -> tuple[int, int]: + value = node.get("source") + if not isinstance(value, dict): + fail("source_anchor", identity, "missing source object") + required = { + "file", "startByte", "endByte", "startLine", "startColumn", "endLine", "endColumn" + } + if not required <= set(value): + fail("source_anchor", identity, "incomplete exact range") + start = value["startByte"] + end = value["endByte"] + if not isinstance(start, int) or not isinstance(end, int) or start >= end: + fail("source_anchor", identity, "range must be non-empty and ordered") + file = value["file"] + source = sources.get(file) + if source is None: + fail("source_anchor", identity, f"source {file!r} is not in the fixture corpus") + if end > len(source): + fail("source_anchor", identity, f"end byte {end} exceeds source length {len(source)}") + if (value["startLine"], value["startColumn"]) != line_column(source, start): + fail("source_anchor", identity, "start line/column does not match source bytes") + if (value["endLine"], value["endColumn"]) != line_column(source, end): + fail("source_anchor", identity, "end line/column does not match source bytes") + return start, end + + +def contains_index(graph: dict[str, Any]) -> dict[str, list[str]]: + children: dict[str, list[str]] = {} + for edge in graph.get("links", []): + if edge.get("kind") == "contains": + children.setdefault(str(edge.get("source")), []).append(str(edge.get("target"))) + for values in children.values(): + values.sort() + return children + + +def table_cells(line: str) -> list[str]: + stripped = line.strip() + if stripped.startswith("|"): + stripped = stripped[1:] + if stripped.endswith("|"): + stripped = stripped[:-1] + return [cell.strip() for cell in stripped.split("|")] + + +def source_table(root: Path, expected: dict[str, Any]) -> tuple[int, int]: + path = root / str(expected.get("source")) + lines = path.read_text(encoding="utf-8").splitlines() + headers = expected.get("headers") + for index in range(len(lines) - 1): + if table_cells(lines[index]) != headers: + continue + separators = table_cells(lines[index + 1]) + if len(separators) != len(headers) or not all( + cell.strip(":").replace("-", "") == "" and "-" in cell + for cell in separators + ): + continue + rows = 0 + cells = 0 + for line in lines[index + 2:]: + if "|" not in line: + break + values = table_cells(line) + rows += 1 + cells += len(values) + return rows, cells + fail("source_table", str(expected.get("id")), "declared pipe table not found in source") + + +def source_frontmatter_paths(source: bytes) -> set[str]: + lines = source.decode("utf-8").splitlines() + if not lines or lines[0].lstrip("\ufeff") != "---": + return set() + paths: set[str] = set() + stack: list[tuple[int, str]] = [] + for line in lines[1:]: + if line == "---": + return paths + if not line.strip() or line.lstrip().startswith("#"): + continue + indent = len(line) - len(line.lstrip(" ")) + stripped = line.strip() + if stripped.startswith("-") or ":" not in stripped: + continue + key, value = stripped.split(":", 1) + key = key.strip().strip("\"'") + if not key: + continue + while stack and stack[-1][0] >= indent: + stack.pop() + segment = key.replace("~", "~0").replace("/", "~1") + path = (stack[-1][1] if stack else "") + "/" + segment + paths.add(path) + if not value.strip(): + stack.append((indent, path)) + fail("source_frontmatter", "source", "frontmatter has no closing delimiter") + + +def load_sources(root: Path, graph: dict[str, Any]) -> dict[str, bytes]: + sources: dict[str, bytes] = {} + for node in graph.get("nodes", []): + if not isinstance(node, dict): + continue + file = source_file(node) + if isinstance(file, str) and file not in sources: + path = root / file + if path.is_file(): + sources[file] = path.read_bytes() + return sources + + +def assert_markdown_quality( + graph: dict[str, Any], manifest: dict[str, Any], fixture_root: Path +) -> dict[str, Any]: + if manifest.get("schema") != "compass.markdown-graph-qualification/3": + fail("manifest_schema", "manifest", repr(manifest.get("schema"))) + if graph.get("graph", {}).get("schema") != manifest.get("graphSchema"): + fail("graph_schema", "graph", repr(graph.get("graph", {}).get("schema"))) + + nodes = graph.get("nodes") + links = graph.get("links") + if not isinstance(nodes, list) or not isinstance(links, list): + fail("graph_shape", "graph", "nodes and links must be arrays") + sources = load_sources(fixture_root, graph) + node_index = {node.get("id"): node for node in nodes if isinstance(node, dict)} + if len(node_index) != len(nodes): + fail("node_identity", "graph", "node IDs must be present and unique") + children = contains_index(graph) + markdown_nodes = [ + node for node in nodes + if isinstance(source_file(node), str) + and source_file(node).lower().endswith((".md", ".mdx", ".qmd", ".markdown")) + ] + if not markdown_nodes: + fail("markdown_nodes", "graph", "no Markdown nodes found") + + score = 0 + score += 1 # graph/1 contract retained + if all((node.get("details") or {}).get("type") != "document" for node in markdown_nodes): + score += 1 + else: + fail("graph_v1_details", "graph", "normalization-only document details leaked") + if all(anchor(node, str(node.get("id")), sources) for node in markdown_nodes): + score += 1 + if any(node.get("kind") == "resource" for node in markdown_nodes): + score += 1 + if any(node.get("name") and terminal_role(node) is None for node in markdown_nodes): + score += 1 + + frontmatter_count = 0 + for expected in manifest.get("frontmatter", []): + identity = str(expected.get("id", "")) + source_name = str(expected.get("source")) + source = sources.get(source_name) + if source is None: + fail("frontmatter_source", identity, f"source {source_name!r} is absent") + source_paths = source_frontmatter_paths(source) + documents = [ + node for node in markdown_nodes + if source_file(node) == source_name + and node.get("kind") == "resource" + and node.get("qualifiedName") == source_name + ] + if len(documents) != 1: + fail("frontmatter_document", identity, f"expected one document, found {len(documents)}") + if documents[0].get("name") != expected.get("documentName"): + fail("frontmatter_title", identity, repr(documents[0].get("name"))) + config_nodes = { + str(node.get("qualifiedName")): node + for node in markdown_nodes + if source_file(node) == source_name and node.get("kind") == "config_key" + } + for path in expected.get("paths", []): + qualified = str(path.get("qualifiedName")) + node = config_nodes.get(qualified) + if node is None: + fail("frontmatter_path", identity, f"missing {qualified!r}") + key_path = path.get("keyPath") + if key_path not in source_paths: + fail("frontmatter_source_path", identity, f"{key_path!r} is absent from source") + details = node.get("details") + data = details.get("data") if isinstance(details, dict) else None + if ( + not isinstance(data, dict) + or details.get("type") != "config" + or data.get("format") != "yaml_frontmatter" + or data.get("keyPath") != key_path + ): + fail("frontmatter_details", qualified, repr(details)) + if node.get("name") != path.get("name"): + fail("frontmatter_label", qualified, repr(node.get("name"))) + start, end = anchor(node, qualified, sources) + first_line = source[start:end].splitlines()[0].decode("utf-8").strip() + terminal = str(key_path).rsplit("/", 1)[-1].replace("~1", "/").replace("~0", "~") + if not first_line.startswith(terminal + ":"): + fail("frontmatter_anchor", qualified, repr(first_line)) + evidence = node.get("evidence") + if ( + not isinstance(evidence, list) + or not evidence + or evidence[0].get("origin") != "config" + ): + fail("frontmatter_provenance", qualified, repr(evidence)) + parent_qualified = path.get("parent") + parent = config_nodes.get(str(parent_qualified)) if parent_qualified else documents[0] + if parent is None or not any( + edge.get("kind") == "contains" + and edge.get("source") == parent.get("id") + and edge.get("target") == node.get("id") + for edge in links + ): + fail("frontmatter_containment", qualified, repr(parent_qualified)) + frontmatter_count += 1 + encoded_graph = json.dumps(graph, sort_keys=True) + for forbidden in expected.get("forbiddenGraphValues", []): + if str(forbidden) in encoded_graph: + fail("frontmatter_value_disclosure", identity, repr(forbidden)) + if frontmatter_count: + score += 5 + + table_count = 0 + row_count = 0 + cell_count = 0 + for expected in manifest.get("tables", []): + identity = str(expected.get("id", "")) + source = expected.get("source") + tables = [ + node for node in markdown_nodes + if source_file(node) == source and terminal_role(node) == "pipe_table" + ] + if len(tables) != 1: + fail("table_count", identity, f"expected one table, found {len(tables)}") + table = tables[0] + table_count += 1 + source_rows, source_cells = source_table(fixture_root, expected) + if source_rows != expected.get("bodyRows") or source_cells != expected.get("bodyCells"): + fail("manifest_source", identity, "checked-in counts do not match source") + table_start, table_end = anchor(table, identity, sources) + direct = [node_index[item] for item in children.get(str(table["id"]), [])] + headers = [node for node in direct if terminal_role(node) == "pipe_table_header"] + rows = [node for node in direct if terminal_role(node) == "pipe_table_row"] + rows.sort(key=source_order) + if len(headers) != 1: + fail("header_count", identity, f"expected one header, found {len(headers)}") + if len(rows) != expected.get("bodyRows"): + fail("row_count", identity, f"expected {expected.get('bodyRows')}, found {len(rows)}") + header_cells = [ + node_index[item] + for item in children.get(str(headers[0]["id"]), []) + if terminal_role(node_index[item]) == "pipe_table_cell" + ] + header_cells.sort(key=source_order) + actual_headers = [str(node.get("name", "")).split(": ", 1)[-1] for node in header_cells] + if actual_headers != expected.get("headers"): + fail("header_content", identity, repr(actual_headers)) + body_cells = [] + for row in rows: + row_start, row_end = anchor(row, str(row["id"]), sources) + if row_start < table_start or row_end > table_end: + fail("row_anchor", str(row["id"]), "row escapes table") + if row.get("name") == "pipe table row": + fail("row_label", str(row["id"]), "generic row label") + current = [ + node_index[item] + for item in children.get(str(row["id"]), []) + if terminal_role(node_index[item]) == "pipe_table_cell" + ] + current.sort(key=source_order) + for cell in current: + cell_start, cell_end = anchor(cell, str(cell["id"]), sources) + if cell_start < row_start or cell_end > row_end: + fail("cell_anchor", str(cell["id"]), "cell escapes row") + if cell.get("name") == "pipe table cell": + fail("cell_label", str(cell["id"]), "generic cell label") + body_cells.extend(current) + if len(body_cells) != expected.get("bodyCells"): + fail("cell_count", identity, f"expected {expected.get('bodyCells')}, found {len(body_cells)}") + row_count += len(rows) + cell_count += len(header_cells) + len(body_cells) + + if table_count: + score += 1 + if row_count: + score += 1 + if cell_count: + score += 1 + score += 1 # semantic header text verified + score += 1 # semantic row labels verified + score += 1 # exact nested anchors verified + + references = 0 + for expected in manifest.get("references", []): + identity = str(expected.get("id", "")) + reference_source = fixture_root / str(expected.get("source")) + if str(expected.get("ownerContains")) not in reference_source.read_text(encoding="utf-8"): + fail("source_reference", identity, "owner text is absent from source") + if not (fixture_root / str(expected.get("targetSource"))).is_file(): + fail("source_reference", identity, "target source is absent from corpus") + owners = [ + node for node in markdown_nodes + if source_file(node) == expected.get("source") + and expected.get("ownerContains") in str(node.get("name", "")) + ] + targets = [node for node in nodes if source_file(node) == expected.get("targetSource")] + matches = [ + edge for edge in links + if edge.get("kind") == expected.get("relationship") + and any(edge.get("source") == owner.get("id") for owner in owners) + and any(edge.get("target") == target.get("id") for target in targets) + and isinstance(edge.get("relationshipSite"), dict) + ] + if len(matches) != 1: + fail("reference", identity, f"expected one exact edge, found {len(matches)}") + references += 1 + if references: + score += 1 + + minimum = manifest.get("minimumQualityScore") + if not isinstance(minimum, int) or score < minimum: + fail("quality_score", "graph", f"score {score} is below {minimum}") + return { + "schema": "compass.markdown-graph-quality-result/1", + "graphSchema": manifest["graphSchema"], + "qualityScore": score, + "markdownNodes": len(markdown_nodes), + "tables": table_count, + "rows": row_count, + "cells": cell_count, + "exactReferences": references, + "frontmatterKeys": frontmatter_count, + } + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--graph", type=Path, required=True) + parser.add_argument("--manifest", type=Path, required=True) + parser.add_argument("--fixture-root", type=Path, required=True) + args = parser.parse_args() + try: + result = assert_markdown_quality( + load_object(args.graph), load_object(args.manifest), args.fixture_root + ) + except (OSError, json.JSONDecodeError, ValueError) as error: + print(f"Markdown graph quality qualification failed: {error}", file=sys.stderr) + return 1 + print(json.dumps(result, sort_keys=True, separators=(",", ":"))) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/qualify_code_graph_v1.sh b/scripts/qualify_code_graph_v1.sh index 36bd4a279..54d439d36 100755 --- a/scripts/qualify_code_graph_v1.sh +++ b/scripts/qualify_code_graph_v1.sh @@ -378,6 +378,12 @@ python3 scripts/check_code_graph_v1_coverage.py \ --compass-revision "$(git rev-parse HEAD)" \ --comparisons "$QUALIFY_TMP/comparisons.json" +echo "[code-graph-v1] execute independent Markdown graph-quality assertions" +python3 scripts/markdown_graph_quality_oracle.py \ + --manifest tests/qualification/markdown-intelligence.json \ + --fixture-root "$CORPUS" \ + --graph "$QUALIFY_TMP/restored.graph.json" + quality_json="$("$COMPASS_BIN" diagnose quality --graph "$QUALIFY_TMP/restored.graph.json" --json)" python3 - "$quality_json" <<'PY' import json diff --git a/tests/qualification/markdown-intelligence.json b/tests/qualification/markdown-intelligence.json new file mode 100644 index 000000000..e1e1674e2 --- /dev/null +++ b/tests/qualification/markdown-intelligence.json @@ -0,0 +1,53 @@ +{ + "schema": "compass.markdown-graph-qualification/3", + "graphSchema": "compass.graph/1", + "minimumQualityScore": 17, + "frontmatter": [ + { + "id": "qualification-guide-frontmatter", + "source": "fixtures/code-graph/qualification/guide.md", + "documentName": "Qualification Guide", + "paths": [ + {"qualifiedName": "frontmatter/title", "keyPath": "/title", "name": "title: Qualification Guide"}, + {"qualifiedName": "frontmatter/tags", "keyPath": "/tags", "name": "tags: markdown, graph"}, + {"qualifiedName": "frontmatter/owners", "keyPath": "/owners", "name": "owners"}, + {"qualifiedName": "frontmatter/owners/quality", "keyPath": "/owners/quality", "name": "quality", "parent": "frontmatter/owners"}, + {"qualifiedName": "frontmatter/owners/quality/team", "keyPath": "/owners/quality/team", "name": "team", "parent": "frontmatter/owners/quality"}, + {"qualifiedName": "frontmatter/api_token", "keyPath": "/api_token", "name": "api_token"} + ], + "forbiddenGraphValues": ["fixture-private-value"] + } + ], + "tables": [ + { + "id": "qualification-guide-table", + "source": "fixtures/code-graph/qualification/guide.md", + "headers": ["Area", "Owner", "Status"], + "bodyRows": 2, + "bodyCells": 6 + } + ], + "references": [ + { + "id": "qualification-guide-link", + "source": "fixtures/code-graph/qualification/guide.md", + "ownerContains": "Runtime source", + "targetSource": "fixtures/code-graph/qualification/rich.rs", + "relationship": "documents" + }, + { + "id": "qualification-guide-cell-link", + "source": "fixtures/code-graph/qualification/guide.md", + "ownerContains": "Rust source", + "targetSource": "fixtures/code-graph/qualification/rich.rs", + "relationship": "documents" + }, + { + "id": "qualification-semantic-doc-link", + "source": "fixtures/code-graph/qualification/semantic-doc.md", + "ownerContains": "Documented runtime", + "targetSource": "fixtures/code-graph/routes/typescript/express.ts", + "relationship": "documents" + } + ] +} diff --git a/vendor/compass-tree-sitter-language-pack/build.rs b/vendor/compass-tree-sitter-language-pack/build.rs index 1fb0b122a..1d2700e51 100644 --- a/vendor/compass-tree-sitter-language-pack/build.rs +++ b/vendor/compass-tree-sitter-language-pack/build.rs @@ -90,6 +90,7 @@ const COMPASS_STATIC_LANGUAGES: &[&str] = &[ "typescript", "verilog", "vue", + "yaml", "zig", ]; diff --git a/vendor/compass-tree-sitter-language-pack/src/intel/data_extraction.rs b/vendor/compass-tree-sitter-language-pack/src/intel/data_extraction.rs index e11ca34c9..c377cc91e 100644 --- a/vendor/compass-tree-sitter-language-pack/src/intel/data_extraction.rs +++ b/vendor/compass-tree-sitter-language-pack/src/intel/data_extraction.rs @@ -609,7 +609,7 @@ fn yaml_children(node: &Node, source: &str) -> Vec { "block_mapping" | "flow_mapping" => { result.extend(yaml_children(&child, source)); } - "block_sequence" => { + "block_sequence" | "flow_sequence" => { let items = yaml_sequence_items(&child, source); result.extend(items); } @@ -674,7 +674,7 @@ fn yaml_sequence_items(node: &Node, source: &str) -> Vec { let mut result = Vec::new(); let mut cursor = node.walk(); for (idx, child) in node.named_children(&mut cursor).enumerate() { - if child.kind() == "block_sequence_item" { + if matches!(child.kind(), "block_sequence_item" | "flow_node") { let sub = yaml_children(&child, source); let value = if sub.is_empty() { let mut c2 = child.walk(); @@ -1251,6 +1251,35 @@ mod tests { assert!(server.is_some(), "should find nested server key"); } + #[test] + fn test_yaml_flow_sequence_retains_nested_mapping_items() { + let source = "authors: [{name: Ada, role: editor}]\n"; + let Some(root) = extract(source, "yaml") else { + return; + }; + let authors = root + .children + .iter() + .find(|child| child.key.as_deref() == Some("authors")); + let Some(author) = authors.and_then(|authors| authors.children.first()) else { + std::process::abort(); + }; + assert_eq!(author.kind, DataNodeKind::Sequence); + assert_eq!(author.key.as_deref(), Some("0")); + assert!( + author + .children + .iter() + .any(|child| child.key.as_deref() == Some("name")) + ); + assert!( + author + .children + .iter() + .any(|child| child.key.as_deref() == Some("role")) + ); + } + #[test] fn test_csv_rows() { let source = "a,b,c\n1,2,3\n";