fix(importers): rank pre-filter candidates by relevance before the caps - #153
Merged
jramos merged 3 commits intoSep 1, 2026
Merged
Conversation
RelevanceFilter qualified candidates with a boolean predicate and then truncated at max_examples * 3 in source-then-import order, so whenever heuristic matches overflowed the cap the strongest matches were discarded before the LLM scorer ever saw them. Confirming the defect surfaced a second, wider path to the same loss: the scoring loop breaks at len(examples) >= max_examples, so candidate order decides the output set on every run, not only when the cap engages. Replace the boolean with _relevance_score, which returns a tiered tuple (name_match, name_words, keyword_overlap); _is_relevant_to_skill becomes any() over that tuple. A tuple rather than weighted integers because skill_name is caller-supplied and unbounded in length, so no fixed set of weights can stop a long name's word count from outranking a full-name match. The keyword tier contributes 0 below two overlaps, which is the one way this change could have widened the qualifying set rather than reordering it. Candidates now sort strongest-first ahead of both caps. sorted() is stable, so equally-scored messages keep their import order and the seeded backfill is untouched. Tests: 6 added, including a 2000-case oracle comparison against a verbatim copy of the previous predicate that proves the qualifying set is unchanged. The pre-existing relevance tests are deliberately unmodified — they are the equivalence proof. Full non-slow suite green (1769 passed, +6), ruff clean. Eval sets drawn after this change differ from earlier ones in composition and in train/val/holdout assignment; noted in the triage doc so they are not compared naively across the boundary.
… tests Review found that every ordering test used a single-word skill name. For a one-word name the full-name tier and the name-word tier fire on identical conditions, so no test could tell them apart — a weighted-sum implementation, which is exactly what the tuple key was chosen over, would have passed all six of the original tests. Adds a multi-word-name case where the tiers genuinely disagree: a message matching one name word must outrank a message overlapping ~14 keywords. Verified by mutation — swapping the tuple for name_match * 100 + name_words * 10 + keyword_overlap fails the two new tier tests and no others, confirming both that they have teeth and that the original set could not distinguish the designs. A companion test guards the premise, so the tiers collapsing to equal values would fail loudly rather than making the ordering test vacuous. Also from review: - Pin tie ordering instead of only comparing two runs to each other. The old determinism test would have passed for any deterministic-but-wrong tie order; it now asserts ties resolve to import order, which is the property the caller relies on for source priority to survive the caps. - Add a case for ties exceeding the candidate cap, so the stable sort is tested rather than only asserted in a comment. - Add the backfill-only path, the one case where the sort receives an empty list. - Name the two caps in the sorting comment rather than pointing "below" at code 25 lines away, which would go stale with no proximity cue. - Say in the docstring that keyword_overlap is thresholded rather than a raw intersection size. 10 tests total. Full non-slow suite green (1773 passed, +10), ruff clean.
jramos
force-pushed
the
fix/importer-relevance-ranking
branch
from
August 31, 2026 20:48
9c0ebcf to
49b9d53
Compare
…rvives Follow-up from review of this change. - Cache the skill-keyword set on skill_text. Scoring every tier removed the old predicate's short-circuit, so the set was rebuilt per message even on the name-match path the old code skipped entirely: 0.729s vs 0.008s over 20k name-matching messages. Cached, that is 0.024s — a ~3x regression instead of ~90x, and negligible ahead of a pipeline that then makes hundreds of LLM calls. - Document why _is_relevant_to_skill stays. It now has no production callers and survives only as the boolean view that keeps the set-preservation guarantee under test via the pre-filter's original tests. Without that note a reader would take it for live code, or delete it and silently drop the guarantee. - Point the class docstring at the function actually in the pipeline. - Widen the equivalence fuzz. Its charset was 19 lowercase ASCII characters with a single constant skill_text, which left case folding and both punctuation strippers as no-ops on every case, and never exercised the 500-char truncation or the keyword-length filter. Now spans case, punctuation, digits, non-ASCII, and seven skill_texts including two straddling the truncation boundary (uppercase 528, punctuation 984, non-ASCII 704 cases, all previously zero). Score-tuple diversity stays low, so it is an equivalence check over the normalisation paths rather than a broad exploration of the score space; the triage note says so rather than overstating it. Also records in the triage doc that for a single-word skill name the top two tiers coincide, leaving keyword overlap as the only intra-tier discriminator — a count that grows with message length, so eval-set composition now skews toward verbose messages. Parked rather than tuned on intuition. Full non-slow suite green (1773 passed), ruff clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The defect
RelevanceFilter.filter_and_scorequalified candidates with a boolean predicate, then truncated:No ranking signal existed anywhere before the cap, so on any skill where heuristic matches overflow it, the strongest matches were discarded before the LLM scorer ever saw them.
build_dataset_from_externalconcatenates per importer, so an overflowing source could crowd out a later one entirely.Confirming the defect surfaced a second, wider path to the same loss: the scoring loop breaks at
len(examples) >= max_examples, so candidate order decides the output set on every run — not only when the cap engages.The existing cap test asserted the resulting count and never which candidates survived, which is why the suite couldn't see this.
The fix
_relevance_scorereturns a tiered tuple(name_match, name_words, keyword_overlap), and_is_relevant_to_skillbecomesany()over it. Candidates sort strongest-first ahead of both caps.A tuple rather than weighted integers, deliberately:
skill_nameis caller-supplied and unbounded in length (traced to a free-form--skillCLI option with no length validation), so no fixed set of weights can stop a long name's word count from outranking a full-name match. A tuple key cannot be violated by input.The keyword tier contributes 0 below two overlaps. That is the one way this change could have silently widened the qualifying set instead of reordering it, so it has its own test.
sorted()is stable, so equally-scored messages keep import order and the seeded backfill is untouched — no new nondeterminism.Verification
10 tests. Three are genuine behavioral reds that fail against the base commit with real assertion failures — one per cap, plus one pinning the scorer's call order. Two more fail pre-change only because the new function doesn't exist yet, which is an artifact of testing a new symbol and is not evidence the old code behaved differently; the rest are invariant and characterization tests, labeled as such.
name_match * 100 + name_words * 10 + keyword_overlapfails exactly the two new tier tests and no others.Known non-goal
The duplicated
mock_dspyfixture is left alone. Hoisting it would touch three test classes and weaken the "no existing test line altered" property that makes the pre-existing tests a clean equivalence proof; it belongs in its own cleanup.Note for future runs
Eval sets drawn after this change differ from earlier ones in both composition and train/val/holdout assignment, so they should not be compared naively across this boundary. Recorded in the triage doc.
Stacking
Based on the triage-doc branch so the action-item row it flips exists in the same diff. GitHub will retarget this to
mainwhen that PR merges.Addresses the relevance-ranking item raised upstream as
NousResearch/hermes-agent-self-evolution#149, and closes the recall follow-up deferred fromNousResearch/hermes-agent-self-evolution#26. Implemented natively; no upstream diff applied.