Add benchmarks and improvements for contextSelector#2644
Open
GeorgeNgMsft wants to merge 19 commits into
Open
Conversation
Comprehensive Tier-1 jest coverage for the raw-turns -> context-vector extraction subsystem: realistic multi-turn accumulation, recency ordering, tokenization robustness (punctuation/NFKC/emoji), protected patterns, plural->singular stemming consistency, window eviction, degenerate all-glue turns, and the documented adversarial limitations (negation, quoted speech). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Comprehensive Tier-1 jest coverage for the context-vector + candidates -> resolve/abstain decision subsystem via TfIdfStrategy and TfIdfScorer: clear-winner resolve, shared-token cancellation (candidate-local IDF), margin/minMass/coverage gates at their boundaries, determinism under candidate reordering, and the documented adversarial score patterns (broad-token leak, negated-word pile-up). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Suppress negated content words from the conversation context vector so "not the spreadsheet" no longer deposits its topic and misroutes a collision. New opt-in tokenize option (dropNegatedSpans) enabled by the conversation signal via a negationGuard config lever (default on); keyword extraction stays byte-identical. A negation scope opens at a cue (not/no/never/without/...) and closes at a clause boundary, a reset connector (but/instead/...), or turn end. Clause-boundary detection requires whitespace adjacency so intra-token punctuation (decimals 2.5, times 3:30, versions v3.2) does not reopen the scope. Measured on the offline benchmark: adversarial wrong-target 13 -> 8, resolution accuracy 7.1 -> 20.0 percent, retrieval share 32.7 -> 56.6 percent, zero safe-tier regressions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…report fix Add a CS_NEGATION_GUARD toggle to the offline runner so the pre-guard baseline can be replayed for before/after measurement. Document the shipped negation guard and the measured-net-negative global-IDF finding in the README, and add a "Methodology & relationship to the static-collision benchmark" section relating this deterministic lexical benchmark to the LLM/embedding @collision corpus probe. Compute the adversarial wrong-target count dynamically in the generated report instead of a stale hard-coded value. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ites metricRunner/measureMetrics: full head-to-head of contextSelector vs every context-blind collision strategy (first-match, score-rank, priority, user-clarify), plus the deployed routing-lift of contextSelector layered on each silent auto-resolver; rendered to console and the markdown report. reproduce.mts: one-command run-all harness (measureMetrics + cached LLM arm) with a per-child V8 heap-cap safeguard (CS_BENCH_MAX_OLD_SPACE_MB, default 2 GB) so a memory regression OOM-fails fast instead of exhausting system RAM; --out is forwarded to both arms so their reports land in one directory. Tests: contextSelectorTrigger (150 trigger/abstain collisions incl. multi-turn) and contextSelectorExtractionAccuracy (75 extraction cases) - deterministic, offline, no LLM. README documents the negation-scope guard and strategy comparison. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
All arms now write a single contextSelector-report.md at a fixed local path (next to the scripts; --out overrides), instead of scattering contextSelector-metrics.{md,json} + contextSelector-vs-llm.md to cwd.
New reportFile.mts provides the shared target: path resolution, the base-report write, and an idempotent marker-delimited LLM-section upsert - so any run order or repetition yields the metrics report followed by exactly one LLM section. measureMetrics writes the base; compareLlm upserts its section; reproduce coordinates and prints the path.
Also drops the unused machine-readable JSON sidecar (nothing in the repo consumed it) and fixes the llm-cache.json trailing-newline churn so re-runs keep it prettier-clean.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…define retrieval The per-tier tables (and console) showed 'spurious / wrong-target' as a single '18 / 8' cell that reads like a ratio. Split into two separate percentage rows: Spurious (over should-abstain fixtures, e.g. 54.5% (18/33)) and Wrong-target (over should-resolve fires, e.g. 80.0% (8/10)), each with its denominator. Metric 1 now defines 'appropriately retrieve the conversation's topic' in plain language (the signal source sorts recent, recency-weighted words into three buckets - intended topic / look-alike distractor / unrelated noise - and checks most weight landed on the topic), with a one-line gloss per measure. Also adds a 'Reading the columns' note defining the combined column (the full realistic calibration corpus, with the adversarial tier excluded). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…tric goals Readability pass so the report explains itself to a reader who has never seen it. Adds a plain-language 'What this measures' overview (the grammar-collision problem, resolve/abstain, the wrong-target=0 safety promise, the offline/deterministic nature, the corpus makeup), a 'Key terms' section with a four-outcome table (correct / safe-miss / wrong-target / spurious) and one-line metric definitions, and a 'How to regenerate this report' block with the run command. Gives Metrics 2, 3, Safety, and the threshold sweep explicit 'Goal:' statements (Metric 1 already had one), and adds the same framing plus a caching note to the LLM-comparison section. De-jargons row labels (Gate A/Gate D/WRR/B-6/L2/'vice versa' -> plain language), fixes the self-contradictory 'Safety is threshold-robust: N cells produced a wrong-target' line, and drops the now-unused easy-slice metric. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
New '## Summary of results' section near the top (after the overview, before Key terms): a plain-language bottom-line conclusion plus five headline bullets — safety (0 silent misroutes across the realistic corpus), helpfulness (yield + resolution accuracy), net routing gain over first-match, how it compares with every silent strategy, and the adversarial limitation. A reader gets the conclusion up front and can scroll down for the detailed findings. All figures are interpolated from the live metrics, so the summary stays accurate on every regeneration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…ub.com/microsoft/TypeAgent into dev/georgeng/benchmark_contextselector
…examples Adds a 'How the benchmark is built' section to the generated report: an at-a-glance one-paragraph summary of the whole benchmark (offline, deterministic, real pipeline, corpus sizes, no LLM), the three-layer structure (foundation / corpus generators / scoring engine), the Fixture shape, a generators comparison table, and a worked example fixture for each of the four corpus generators (easy / siblings / real-pairs / dialogue). Restructures the README's flat file list into the same three layers with the Fixture shape and a generator table, so the code layout and the report tell the same story. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
The report is now a tracked artifact at docs/architecture/collision/contextSelector-report.md, but the generator defaulted to the (untracked) source folder, so the committed copy drifted. Make that docs path the default output of resolveReportPath() (still overridable with --out), so a plain 'reproduce.mts' keeps the tracked report current. reproduce.mts now runs 'prettier --write' on the report as a final step: the generator emits valid markdown but not prettier-normalized table alignment, and the tracked file must pass CI's 'prettier --check .'. This bakes in the manual format pass that previously kept it clean. Regenerates the report (now including the 'How the benchmark is built' section) into the tracked location; updates README + header comments to state the docs default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
The LLM arm is a proxy, not the production fallback: it asks a default chat model to pick between the two colliding agents (described by one-line blurbs) and is explicitly told to watch for negation/sarcasm/quoting, whereas the real fallback is full schema translation across all agents on the configured model. Document that this makes the LLM look stronger than production would — so the realistic-tier 0 regressions is a robust floor, but the adversarial regression count is a worst-case (not expected) cost — and point at an L3 live replay as the faithful measurement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
Flip the shipped default contextSelector.detect to true. It runs as a deterministic, LLM-free first pass on grammar collisions: a confident topical pick resolves the collision, otherwise it abstains and (via the unchanged defer-to-strategy fallback) hands off to first-match — strictly additive over today's routing, never a silent misroute. Backed by the offline benchmark (docs/architecture/collision/contextSelector-report.md): 0 wrong-target across the ~1690-collision realistic corpus, +29% routing lift over first-match with no per-class regression. Verified the full dispatcher offline suite still passes (78 suites / 1407 tests) with the default on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 61308b63-2247-4538-ab35-593de5423150
…k_contextselector
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Strategy Comparison (Standalone Comparison 1000 tests)
Full Route Comparisons (250 dialogues)
Aggregate on normal traffic: the 88 collisions contextSelector confidently resolves (simple+realistic+hard) would each be an LLM round-trip under escalate-to-LLM. That's ~88 × 531 ms ≈ 47 s of serial LLM latency replaced by 88 × 8 µs ≈ 0.7 ms of local compute — same-or-better routing, zero regressions, effectively instant.
Note: the speed win is vs the LLM path only. Against first-match, both are cache-path (µs-scale) — there the win is accuracy (+29%), not latency. contextSelector adds only ~8 µs per collision it evaluates, far under the design's <5 ms cache-path budget.
Bottom line
• Normal traffic (150 scenarios): strictly additive vs both baselines — as-accurate-or-better routing with 0 regressions, while eliminating 88 LLM calls (~47 s → <1 ms).
• No-context: abstains 50/50 — never makes the no-signal case worse.
• Adversarial (50 scenarios): worse than the LLM (28% vs 56%); the negation guard shrank the gap (wrong-target 13→8, LLM-regressions 19→14) but sarcasm/quotes/dense ties need the LLM. Mitigated by shipping detect: false (off) and abstainFallback: escalate-to-llm .
What & why: Adds a deterministic offline benchmark and comprehensive test coverage for the dispatcher's contextSelector collision-resolution tier, plus a negation-scope guard that stops loaded-negation phrases like "not the spreadsheet" from misrouting a collision to the rejected agent.
What's included
• Negation-scope guard (the only production behavior change) — the conversation signal now suppresses negated topic words before scoring, behind a negationGuard config lever (default on). Keyword extraction is unchanged, and the whole tier stays off by default ( detect: false ), so it's inert until enabled.
• Tier-1 tests (CI-gated, deterministic) — new context-extraction and scoring suites plus expanded tokenize/signal specs (127 contextSelector tests).
• Offline benchmark — a 250-scenario hand-authored dialogue corpus across 5 difficulty tiers replayed through the real scorer, an LLM-comparison arm, and a methodology write-up relating it to the static @collision corpus benchmark.
Results (measured)
• Adversarial tier: the guard cuts wrong-target misroutes 13 → 8 with zero safe-tier regressions.
• Normal traffic: contextSelector matches or beats both first-match (+29% routing accuracy) and the LLM with 0 regressions, while avoiding LLM calls