Skip to content

fix(importers): rank pre-filter candidates by relevance before the caps - #153

Merged
jramos merged 3 commits into
docs/triage-sweep-and-quarterly-cadencefrom
fix/importer-relevance-ranking
Sep 1, 2026
Merged

fix(importers): rank pre-filter candidates by relevance before the caps#153
jramos merged 3 commits into
docs/triage-sweep-and-quarterly-cadencefrom
fix/importer-relevance-ranking

Conversation

@jramos

@jramos jramos commented Aug 31, 2026

Copy link
Copy Markdown
Owner

The defect

RelevanceFilter.filter_and_score qualified candidates with a boolean predicate, then truncated:

candidates = [m for m in messages if _is_relevant_to_skill(...)]   # boolean, no strength
candidates = candidates[:max_examples * 3]                         # source-then-import order

No ranking signal existed anywhere before the cap, so on any skill where heuristic matches overflow it, the strongest matches were discarded before the LLM scorer ever saw them. build_dataset_from_external concatenates per importer, so an overflowing source could crowd out a later one entirely.

Confirming the defect surfaced a second, wider path to the same loss: the scoring loop breaks at len(examples) >= max_examples, so candidate order decides the output set on every run — not only when the cap engages.

The existing cap test asserted the resulting count and never which candidates survived, which is why the suite couldn't see this.

The fix

_relevance_score returns a tiered tuple (name_match, name_words, keyword_overlap), and _is_relevant_to_skill becomes any() over it. Candidates sort strongest-first ahead of both caps.

A tuple rather than weighted integers, deliberately: skill_name is caller-supplied and unbounded in length (traced to a free-form --skill CLI option with no length validation), so no fixed set of weights can stop a long name's word count from outranking a full-name match. A tuple key cannot be violated by input.

The keyword tier contributes 0 below two overlaps. That is the one way this change could have silently widened the qualifying set instead of reordering it, so it has its own test.

sorted() is stable, so equally-scored messages keep import order and the seeded backfill is untouched — no new nondeterminism.

Verification

10 tests. Three are genuine behavioral reds that fail against the base commit with real assertion failures — one per cap, plus one pinning the scorer's call order. Two more fail pre-change only because the new function doesn't exist yet, which is an artifact of testing a new symbol and is not evidence the old code behaved differently; the rest are invariant and characterization tests, labeled as such.

  • Set preservation is proven by a 2000-case fuzz plus 7 hand-picked edges (empty skill name, whitespace name, empty text, overlap of exactly 1, case mismatch, short name words, empty skill text) against a verbatim copy of the previous predicate as an oracle.
  • Tier separation is mutation-verified. Review caught that every ordering test used a single-word skill name, making the middle tier indistinguishable from the top — a weighted sum would have passed all of them. A multi-word case was added where the tiers disagree, and mutating the implementation into name_match * 100 + name_words * 10 + keyword_overlap fails exactly the two new tier tests and no others.
  • Tie ordering is pinned, not merely compared run-to-run; ties exceeding the cap and the backfill-only path are both covered.
  • The pre-existing relevance tests are deliberately unmodified — they are the equivalence proof.
  • Full non-slow suite: 1773 passed (1763 baseline + 10), ruff clean.

Known non-goal

The duplicated mock_dspy fixture is left alone. Hoisting it would touch three test classes and weaken the "no existing test line altered" property that makes the pre-existing tests a clean equivalence proof; it belongs in its own cleanup.

Note for future runs

Eval sets drawn after this change differ from earlier ones in both composition and train/val/holdout assignment, so they should not be compared naively across this boundary. Recorded in the triage doc.

Stacking

Based on the triage-doc branch so the action-item row it flips exists in the same diff. GitHub will retarget this to main when that PR merges.

Addresses the relevance-ranking item raised upstream as NousResearch/hermes-agent-self-evolution#149, and closes the recall follow-up deferred from NousResearch/hermes-agent-self-evolution#26. Implemented natively; no upstream diff applied.

jramos added 2 commits August 31, 2026 14:44
RelevanceFilter qualified candidates with a boolean predicate and then
truncated at max_examples * 3 in source-then-import order, so whenever
heuristic matches overflowed the cap the strongest matches were discarded
before the LLM scorer ever saw them. Confirming the defect surfaced a second,
wider path to the same loss: the scoring loop breaks at
len(examples) >= max_examples, so candidate order decides the output set on
every run, not only when the cap engages.

Replace the boolean with _relevance_score, which returns a tiered tuple
(name_match, name_words, keyword_overlap); _is_relevant_to_skill becomes
any() over that tuple. A tuple rather than weighted integers because
skill_name is caller-supplied and unbounded in length, so no fixed set of
weights can stop a long name's word count from outranking a full-name match.
The keyword tier contributes 0 below two overlaps, which is the one way this
change could have widened the qualifying set rather than reordering it.

Candidates now sort strongest-first ahead of both caps. sorted() is stable, so
equally-scored messages keep their import order and the seeded backfill is
untouched.

Tests: 6 added, including a 2000-case oracle comparison against a verbatim
copy of the previous predicate that proves the qualifying set is unchanged.
The pre-existing relevance tests are deliberately unmodified — they are the
equivalence proof. Full non-slow suite green (1769 passed, +6), ruff clean.

Eval sets drawn after this change differ from earlier ones in composition and
in train/val/holdout assignment; noted in the triage doc so they are not
compared naively across the boundary.
… tests

Review found that every ordering test used a single-word skill name. For a
one-word name the full-name tier and the name-word tier fire on identical
conditions, so no test could tell them apart — a weighted-sum implementation,
which is exactly what the tuple key was chosen over, would have passed all six
of the original tests.

Adds a multi-word-name case where the tiers genuinely disagree: a message
matching one name word must outrank a message overlapping ~14 keywords.
Verified by mutation — swapping the tuple for name_match * 100 + name_words * 10
+ keyword_overlap fails the two new tier tests and no others, confirming both
that they have teeth and that the original set could not distinguish the
designs. A companion test guards the premise, so the tiers collapsing to equal
values would fail loudly rather than making the ordering test vacuous.

Also from review:

- Pin tie ordering instead of only comparing two runs to each other. The old
  determinism test would have passed for any deterministic-but-wrong tie order;
  it now asserts ties resolve to import order, which is the property the caller
  relies on for source priority to survive the caps.
- Add a case for ties exceeding the candidate cap, so the stable sort is tested
  rather than only asserted in a comment.
- Add the backfill-only path, the one case where the sort receives an empty
  list.
- Name the two caps in the sorting comment rather than pointing "below" at code
  25 lines away, which would go stale with no proximity cue.
- Say in the docstring that keyword_overlap is thresholded rather than a raw
  intersection size.

10 tests total. Full non-slow suite green (1773 passed, +10), ruff clean.
@jramos
jramos force-pushed the fix/importer-relevance-ranking branch from 9c0ebcf to 49b9d53 Compare August 31, 2026 20:48
…rvives

Follow-up from review of this change.

- Cache the skill-keyword set on skill_text. Scoring every tier removed the old
  predicate's short-circuit, so the set was rebuilt per message even on the
  name-match path the old code skipped entirely: 0.729s vs 0.008s over 20k
  name-matching messages. Cached, that is 0.024s — a ~3x regression instead of
  ~90x, and negligible ahead of a pipeline that then makes hundreds of LLM
  calls.
- Document why _is_relevant_to_skill stays. It now has no production callers and
  survives only as the boolean view that keeps the set-preservation guarantee
  under test via the pre-filter's original tests. Without that note a reader
  would take it for live code, or delete it and silently drop the guarantee.
- Point the class docstring at the function actually in the pipeline.
- Widen the equivalence fuzz. Its charset was 19 lowercase ASCII characters with
  a single constant skill_text, which left case folding and both punctuation
  strippers as no-ops on every case, and never exercised the 500-char truncation
  or the keyword-length filter. Now spans case, punctuation, digits, non-ASCII,
  and seven skill_texts including two straddling the truncation boundary
  (uppercase 528, punctuation 984, non-ASCII 704 cases, all previously zero).
  Score-tuple diversity stays low, so it is an equivalence check over the
  normalisation paths rather than a broad exploration of the score space; the
  triage note says so rather than overstating it.

Also records in the triage doc that for a single-word skill name the top two
tiers coincide, leaving keyword overlap as the only intra-tier discriminator — a
count that grows with message length, so eval-set composition now skews toward
verbose messages. Parked rather than tuned on intuition.

Full non-slow suite green (1773 passed), ruff clean.
@jramos
jramos merged commit 43a2899 into main Sep 1, 2026
5 checks passed
@jramos
jramos deleted the fix/importer-relevance-ranking branch September 1, 2026 01:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant