Skip to content

fix: stop dropping token labels with more than 100 rows - #118

Merged
dawsbot merged 2 commits into
v1from
fix/token-pagination
Jul 10, 2026
Merged

fix: stop dropping token labels with more than 100 rows#118
dawsbot merged 2 commits into
v1from
fix/token-pagination

Conversation

@dawsbot

@dawsbot dawsbot commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Problem

scripts/HtmlParser/HtmlParser.ts had a long-standing TODO ("repull tokens which have a page length > 100"). selectAllLabels skipped any token label whose record count was >= 100 (the token page-size cap), so those categories were silently never pulled at all — in the etherscan labelcloud fixture alone that is 35 labels, including defi (2,101 tokens) and stablecoin. The URLs it did emit also lacked the start cursor that ApiParser.fetchTokens requires, which would crash that fetch path.

Fix

  • HtmlParser.selectAllLabels now always includes token labels, emitting ?size=100&start=0 — matching CheerioParser.selectAllLabels, the variant ChainPuller uses at runtime.
  • Pagination happens in the existing fetch layer: ApiParser.fetchTokens keeps advancing start by 100 until a page comes back with fewer than 100 rows, so all rows are captured.
  • The accounts guard (skip labels with >= 10,000 records to avoid freezing on e.g. beacon-depositor) is unchanged.

Tests

  • New ApiParser unit test drives fetchTokens through a full 100-row page plus a 20-row page via a stubbed browser fetcher (rows built from the real aave.json fixture HTML with unique addresses), asserting the start cursor advances [0, 100] and all 120 unique rows are captured.
  • Labelcloud fixture tests extended: previously-dropped >100-row token labels (defi on etherscan/arbiscan/optimism, blockchain on bscscan) are now asserted present with the pagination cursor, and expected label counts updated (etherscan 899 → 934, bscscan 240 → 263, arbiscan 167 → 170, optimism 101 → 102).

bun test:unit (31 pass), eslint, and tsc all pass. Dataset intentionally not regenerated — code fix + tests only.

🤖 Generated with Claude Code

dawsbot and others added 2 commits July 6, 2026 13:08
HtmlParser.selectAllLabels skipped any token label whose record count
was >= 100 (the token page-size cap), so those categories were silently
never pulled, and the urls it did emit lacked the "start" cursor that
ApiParser.fetchTokens requires for pagination. Token labels are now
always included with "?size=100&start=0", matching CheerioParser, and
fetchTokens pages through them 100 rows at a time. The accounts guard
(skip >= 10,000 to avoid freezing on labels like beacon-depositor) is
unchanged.

Adds a unit test that drives fetchTokens through a full 100-row page
plus a 20-row page via a stubbed browser fetcher, and extends the
labelcloud fixture tests to cover previously-dropped >100-row labels.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@dawsbot
dawsbot merged commit a0e3b35 into v1 Jul 10, 2026
1 check passed
@dawsbot
dawsbot deleted the fix/token-pagination branch July 10, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant