fix(email): honour part charset and skip binary parts in list snippets

[shuvamk]

Summary

extractListSnippet in apps/email/server/src/lib/imap-driver.ts (added by 532753a, 2026-07-31) decodes the bounded BODY[TEXT] partial fetch with a fixed utf8 decode, on whatever leaf resolveLeafPart lands on, and unescapes six hard-coded HTML entities. That produces mojibake on non-UTF-8 mail, raw binary in inbox rows on S/MIME / PGP / attachment-first mail, and literal ’/— in HTML mail.

Motivation

The clearest statement of the first bug is that the same message renders two different ways in the same UI. A text/plain; charset=ISO-8859-1 quoted-printable part:

Code path	Output
list row — extractListSnippet	Hola Bob, la reuni�n de ma�ana se traslada a las diez. Un saludo.
reading pane — getThread → simpleParser (imap-driver.ts:936 on main)	Hola Bob, la reunión de mañana se traslada a las diez. Un saludo.

getThread gets it right because mailparser reads the part's charset. The list path throws that information away: resolveLeafPart returns only { type, encoding }, even though imapflow already parses parameters.charset for every leaf (imapflow/lib/tools.js getStructuredParams, keys lowercased, values preserved).

Full main vs branch differential, produced by running both versions of extractListSnippet side by side on the same inputs (script in the Verification section):

Body structure	main snippet	this branch
text/plain; charset=ISO-8859-1, quoted-printable	Hola Bob, la reuni�n de ma�ana se traslada a las diez. Un saludo.	Hola Bob, la reunión de mañana se traslada a las diez. Un saludo.
text/plain; charset=windows-1252, base64	Your order shipped � �track it� for updates.	Your order shipped — “track it” for updates.
application/pkcs7-mime (S/MIME), base64 DER	0�\u0003 \u0006 *�H�� \u0001\u0007\u0002��\u0002�0�\u0002�\u0002\u0001\u00011\u000f0	`` (empty)
multipart/mixed, application/pdf first (scan-to-email)	%PDF-1.7 %���� 1 0 obj endobj	`` (empty)
multipart/encrypted, application/pgp-encrypted first	Version: 1	`` (empty)
multipart/related, image/jpeg first (inline logo)	����\u0000\u0010JFIF\u0000\u0001\u0001\u0001\u0000H\u0000H\u0000\u0000��\u0000C\u0000\b\u0006\u0006\u0007\u0006	`` (empty)
text/html, numeric + named references	We’re excited — your order’s on its way. Save 20% today & tomorrow. © 2026 Acme	We’re excited — your order’s on its way. Save 20% today & tomorrow. © 2026 Acme
text/html, already-escaped &amp;lt;script&amp;gt;	Escaped tag: <script> stays escaped.	Escaped tag: &lt;script&gt; stays escaped.
multipart/mixed, message/rfc822 first (forwarded)	From: Ops Subject: Nightly backup report Content-Type: text/plain; charset=utf-8 All volumes backed up successfully.	`` (empty)
text/plain; charset=utf-8 (control)	Standing meeting moved to Thursday at ten.	Standing meeting moved to Thursday at ten.

10 cases, 9 differ, 1 identical. Cell values are the JSON.stringify'd return value, so \u00XX sequences are literal control bytes landing in the inbox row and � is U+FFFD.

The three causes:

1. Charset dropped. imap-driver.ts:691 on main is decoded.toString('utf8'), and resolveLeafPart (:606) never carried the charset to that call site.
2. Non-text leaves decoded. resolveLeafPart descends childNodes[0] unconditionally, and extractListSnippet never checks that the leaf is text/* before base64-decoding it. The function's own JSDoc says it "always degrades to ''"; these four cases are where it doesn't.
3. Entity table too small, and applied in the wrong order. :703-708 handles only   & < > " ' '. & is replaced before </>, so < double-unescapes to <.

Related issue

None.

Changes

apps/email/server/src/lib/imap-driver.ts:

• LeafPartInfo gains charset, populated from the leaf's parameters.charset (trimmed, lowercased). New decodeCharset helper decodes via TextDecoder; labels TextDecoder rejects throw and fall through to today's utf8 decode, so this can only widen what decodes correctly. us-ascii/ascii deliberately stay on the utf8 path. WHATWG resolves both labels to windows-1252 (new TextDecoder('us-ascii').encoding === 'windows-1252' on bun 1.3.14 and node v26.4.0 alike), so routing them through TextDecoder would turn the very common part that declares ASCII but carries UTF-8 bytes from Cafés — closing at 5. into Cafés — closing at 5. — a fresh bug introduced by the fix. A test pins this.
• extractListSnippet returns '' when the resolved leaf is not text/*. A null leaf keeps the current behaviour, so a body structure that can't be parsed is not newly suppressed.
• New decodeHtmlEntities: decimal, hexadecimal and a 24-entry named table, resolved in one left-to-right pass. The single pass is what removes the double-unescape — after & is consumed, scanning resumes past its ;. Unrecognised references are left verbatim.
• TextDecoder is imported from node:util because bun-types narrows the global constructor to Bun.Encoding = "utf-8" | "windows-1252" | "utf-16"; without the import tsc --noEmit in apps/email/server fails with error TS2345: Argument of type 'string' is not assignable to parameter of type 'Encoding | undefined'. The node:util export takes a string and is the same class at runtime. There is precedent for node: imports in this workspace (src/env.ts, src/lib/crypto.ts, src/lib/branding.ts).
• extractListSnippet is exported so the test can call it, following formatFromAddress in src/trpc/routes/mail.ts, which is exported for exactly the same reason.

apps/email/server/test/list-snippet.test.ts (new, 13 tests, bun:test to match the existing test/from-header.test.ts).

Verification

Please note where this does and does not run in CI. apps/email defines no test script, so turbo run test skips it:

$ npx turbo run test --dry=json | jq -r '.tasks[] | "\(.taskId) | \(.command)"'
@repo/adapters#test | vitest run
@repo/api#test | vitest run
@repo/core#test | vitest run
@repo/dashboard#test | vitest run
@repo/db#test | vitest run
@repo/db-email#test | <NONEXISTENT>
@repo/desktop#test | vitest run
@repo/email#test | <NONEXISTENT>
@repo/onboarding#test | <NONEXISTENT>
@repo/ui#test | <NONEXISTENT>
@repo/web#test | <NONEXISTENT>
openship#test | vitest run

That is already true of test/from-header.test.ts on main, so this PR follows the existing convention rather than changing the build — but it does mean the new tests will not turn the CI Test job red or green. They run with bun test from apps/email/server.

I deliberately did not touch apps/email/package.json, because #219 is already proposing to wire this workspace up — it adds "test": "cd server && bun run test" to apps/email/package.json and "test": "vitest run" to apps/email/server/package.json. If #219 lands first I'll swap this file's bun:test imports for vitest (test/from-header.test.ts on main will need the same). Happy to add the script here instead if you'd rather not wait.

New tests, run locally (bun 1.3.14):

$ cd apps/email/server && bun test test/
bun test v1.3.14 (0d9b296a)

 19 pass
 0 fail
 21 expect() calls
Ran 19 tests across 2 files. [107.00ms]

I verified the tests fail without the fix. With all three behaviour changes reverted in place (keeping only the export and the node:util import, so the module still loads), 9 of 13 fail:

$ bun test test/list-snippet.test.ts 2>&1 | grep -E '^\(fail\)| pass$| fail$'
(fail) extractListSnippet > charset > decodes an ISO-8859-1 quoted-printable text/plain part
(fail) extractListSnippet > charset > decodes a windows-1252 base64 text/plain part
(fail) extractListSnippet > non-text leaf parts > returns empty for an S/MIME application/pkcs7-mime message
(fail) extractListSnippet > non-text leaf parts > returns empty when a multipart/mixed leads with a PDF attachment
(fail) extractListSnippet > non-text leaf parts > returns empty for the PGP/MIME version identification part
(fail) extractListSnippet > non-text leaf parts > returns empty when a multipart/related leads with an inline image
(fail) extractListSnippet > HTML character references > decodes numeric and named references
(fail) extractListSnippet > HTML character references > decodes hexadecimal references
(fail) extractListSnippet > HTML character references > does not double-unescape an already-escaped reference

 4 pass
 9 fail

Reverting each of the three independently isolates them cleanly: charset only → 2 fail; non-text guard only → 4 fail; entity decoding only → 3 fail.

The remaining 4 tests pass on main as well, so I want to be explicit that they are boundary assertions rather than regression tests. Each one constrains a real decision in this diff, and I proved that by mutating the fix and watching the test go red:

Test	Mutation applied	Result
decodes UTF-8 bytes in a part mislabelled as us-ascii	drop us-ascii/ascii from UTF8_COMPATIBLE_CHARSETS	fails
falls back to utf8 for a charset label TextDecoder does not know	remove the try/catch in decodeCharset	fails
still extracts a snippet when the body structure is unusable	if (leaf && …) → if (!leaf || …)	fails
leaves an unrecognised reference verbatim	HTML_ENTITIES[…] ?? match → ?? ''	fails

Full suite, forced (no turbo cache), on this branch:

$ bun run test -- --continue --force
@repo/desktop:test    Test Files 1 passed (1)      Tests 3 passed (3)
@repo/core:test       Test Files 29 passed (29)    Tests 325 passed (325)
@repo/dashboard:test  Test Files 19 passed (19)    Tests 236 passed (236)
@repo/adapters:test   Test Files 85 passed (85)    Tests 744 passed (744)
openship:test         Test Files 22 passed (22)    Tests 181 passed (181)
@repo/db:test         Test Files 11 passed (11)    Tests 55 passed (55)
@repo/api:test        Test Files 171 passed | 4 skipped (175)
                            Tests 1849 passed | 15 skipped (1864)

 Tasks:    7 successful, 7 total
Cached:    0 cached, 7 total

I ran the identical command on pristine main @ 2d7fa591 (also --force, 0 cached) and got the same seven lines, so main is fully green and there is no pre-existing failure to discount here.

Typecheck. CI's Typecheck job has two steps (.github/workflows/ci.yml:31-45), apps/api and apps/dashboard; apps/email/server is not typechecked in CI, so I ran it separately:

$ bun run --cwd apps/api lint                            # CI step 1 — tsc --noEmit, exit 0
$ cd apps/dashboard && npx tsc --noEmit | grep -v fumadocs   # CI step 2 — 0 errors, gate passes
$ cd apps/email/server && npx tsc --noEmit               # not in CI — exit 0

Formatting: imap-driver.ts carries pre-existing prettier drift on main — 465 lines differ under the repo's own .prettierrc (npx prettier --check on main already reports it as unformatted). Per CONTRIBUTING I did not run the formatter over it; my lines are hand-formatted to the file's own style (single quotes, ≤100 cols). The new test file is prettier-clean (npx prettier --check passes). This is the one honest qualification on the bun format checklist item below.

The differential table above was produced by extracting extractListSnippet and its helpers from origin/main and from this branch into two modules and calling both on the same fixtures, in the apps/email/server workspace so the real imapflow (1.3.3) and mailparser (3.9.8) resolve. The reading-pane row was produced by feeding the identical RFC822 message to simpleParser and taking (parsed.text ?? '').slice(0, 240) — what getThread does at imap-driver.ts:936 on main (:1021 on this branch). For this fixture that value carries no trailing whitespace, so the cell is verbatim; a message with a blank line before the closing boundary would return a trailing \n. I can push the script if you want it in the repo.

Residuals and trade-offs — things I did not fix

1. message/rfc822 first parts now yield an empty snippet. This is the one behaviour change beyond the three bugs, and it is a direct consequence of the text/* guard. Measured above: main leaks From: Ops Subject: Nightly backup report Content-Type: … into the row, the branch yields ''.

   To be precise about what an empty snippet does in the UI, since this is the riskiest thing in the PR — there are exactly two consumers and neither substitutes anything:

   • mail-list.tsx:506 — {latestMessage.snippet ? highlightText(…) : null}, so the <p> renders empty and the snippet line goes blank.
   • command-palette-context.tsx:991 — {thread.snippet || ''}, so the palette row shows Sender - with nothing after the dash.

   The subject is unaffected: it is rendered in its own <p> at mail-list.tsx:488, independent of the snippet. So the row goes from subject + header soup to subject + blank line, not to a subject fallback. (In the Sent folder the snippet slot is occupied by the recipient list at :498, so nothing changes there at all.)

   The retained JSDoc on extractListSnippet says "the client falls back to the subject line when snippet is empty" — that is upstream's own wording from 532753a and it does not match the client code. I left it untouched as out of scope; happy to correct it here if you want.

   I think blank is better than header soup, but the right answer is probably to descend into message/rfc822 the way multipart/ is descended (depth + 1 lands correctly, since the inner message contributes exactly one header block). I left it out to keep this to one concern. Say the word and I'll add it here or in a follow-up.

2. The named-entity table is a fixed 24 entries, not full HTML5 coverage. entities@4.5.0 is already in apps/email/server/node_modules as a transitive dep of cheerio, so decodeHTML from it would be complete and shorter — but that means promoting it to a declared dependency, which I did not want to do unasked. Happy to switch if you prefer that.

3. Charset conversion still happens after truncation, not before. The BODY[TEXT] window is 320 bytes and the cut can land mid-character; TextDecoder yields U+FFFD there, and the existing .replace(/�+$/, '') strips it at the tail. Unchanged from main, just now reachable for more charsets.

4. Bun's TextDecoder covers fewer labels than Node's, so the win is partial under the runtime this server actually uses (apps/email/server starts with bun run src/main.ts). I measured both, decoding real bytes rather than just constructing the decoder — bun 1.3.14 vs node v26.4.0:

   Label	bun 1.3.14	node v26.4.0
   iso-8859-1	Hola óñ	Hola óñ
   windows-1252	—“”	—“”
   koi8-u	При	При
   shift_jis	日本	日本
   euc-kr	한국	한국
   big5 / gbk	中文	中文
   iso-8859-2	RangeError → utf8 fallback	Złó
   iso-8859-15	RangeError → utf8 fallback	€
   windows-1250 / 1251 / 1256	RangeError → utf8 fallback	ŚŁó / При / ال
   koi8-r	RangeError → utf8 fallback	При
   x-mac-roman	RangeError → utf8 fallback	é
   genuinely unknown label	RangeError → utf8 fallback	RangeError → utf8 fallback

   Every RangeError row falls back to bytes.toString('utf8'), which is byte-for-byte what main produces today — so nothing regresses, but Central/Eastern European, Cyrillic (koi8-r, windows-1251) and Arabic mail is still mojibake on Bun. (koi8-u working while koi8-r throws is a Bun quirk, not a typo on my part.) If you want those covered too, the options are a small hand-rolled single-byte table or iconv-lite; I'd rather you pick than assume.

5. resolveLeafPart still follows childNodes[0] only. A multipart/alternative that lists text/html before text/plain will use the HTML part, as on main.

Checklist

• One change per PR — one bug, or one agreed feature, with nothing unrelated bundled in
• The diff is scoped — no reformatting or lint fixes on lines I wasn't otherwise changing
• A test fails without this change and passes with it (or I explained above why there isn't one)
• bun run test, bun run --cwd <workspace> lint, and bun format all pass locally — with one qualification: bun run test and the typechecks pass as shown above, but I did not run bun format. It would rewrite the 465 pre-existing prettier-drifted lines in imap-driver.ts and bury a +110/-24 fix in ~465 lines of unrelated churn. My own lines are hand-formatted to the file's style; the new test file is prettier-clean. Say the word if you'd rather have the file formatted and I'll do it as a separate commit on this branch so it stays reviewable.
• I understand every line of this diff and can explain it in review