Local embeddings: on-device model as the default — complete epic (all platforms, field-tested)

.gitignore

@@ -43,3 +43,9 @@ Desktop.ini
 
 # Scratchpad
 scratchpad/
+
+# Local-embedding assets (downloaded by `make assets`, never committed).
+# Runtime assets that get go:embed-ed into the localembed build:
+internal/embeddings/local/embedded/
+# Link-time static tokenizer lib (pulled in via CGO_LDFLAGS):
+internal/embeddings/local/lib/

CLAUDE.md

@@ -1,16 +1,24 @@
 # Agent Memory
 
-Local-first desktop app + MCP server for semantic file search. Watches directories, embeds file contents via OpenAI, stores vectors in SQLite (sqlite-vec), and provides KNN semantic search.
+Local-first desktop app + MCP server for semantic file search. Watches directories, embeds file contents (a bundled local model by default; OpenAI opt-in), stores vectors in SQLite (sqlite-vec), and provides KNN semantic search.
 
 ## Build & Run
 
 ```bash
-make build          # wails build -skipbindings (DO NOT use plain `wails build` — hangs on binding generation due to CGo)
+make assets         # download pinned local model/tokenizer/ORT libs into assets/embedded/ (~150 MB, checksummed via assets/manifest.json)
+make build          # wails build -skipbindings, builds with -tags localembed (DO NOT use plain `wails build` — hangs on binding generation due to CGo)
+make build-darwin-amd64  # cross-build the Intel-mac app from an arm64 Mac (swaps in darwin-amd64 assets)
 make dev            # hot-reload dev mode
-make test           # go test ./...
+make test           # go test ./... (default build, no localembed tag — stays native-lib-free)
 make clean          # rm -rf build/bin
 ```
 
+> `make build` depends on `make assets`, which downloads ~150 MB of native artifacts (model,
+> tokenizer, ONNX Runtime lib) the first time — needs network access. The build sets
+> `-tags localembed` plus `CGO_LDFLAGS` to link the tokenizer library; the resulting binary is
+> ~180 MB (it bundles the model). The native code is build-tag-isolated, so plain
+> `go test ./...` and CI need no assets.
+
 > **macOS 26+ gotcha:** Go ≤ 1.24 produces CGo binaries the kernel kills instantly on launch
 > (exit 137, `dyld: missing LC_UUID`) — this breaks `make build`, the `wails` CLI, and the built
 > app. Fix: `go env -w GOTOOLCHAIN=go1.26.4`, then `go install github.com/wailsapp/wails/v2/cmd/wails@v2.11.0`.
@@ -44,7 +52,8 @@ Delivery (main.go, mcp/, tray.go, frontend/)
 |---------|---------|
 | `internal/engine` | Core orchestrator: scan → extract → chunk → embed → store |
 | `internal/store` | SQLite + sqlite-vec persistence (config, directories, files, chunks, vectors) |
-| `internal/embeddings` | OpenAI embedding client with batching (2048/req) and retry |
+| `internal/embeddings` | Embedder interface + OpenAI client (opt-in) with batching (2048/req) and retry |
+| `internal/embeddings/local` | Bundled in-process local embedder (default): ONNX Runtime + HF tokenizer, `multilingual-e5-small` (384-dim), offline |
 | `internal/chunker` | Token-based text splitting (tiktoken, cl100k_base) |
 | `internal/watcher` | fsnotify recursive directory watcher with 500ms debounce |
 | `internal/extractor` | Multi-format content extraction (text, docx, xlsx, pptx, pdf, images) |

Documentation/ARCHITECTURE.md

@@ -17,7 +17,7 @@ Delivery  ->  Service (Engine)  ->  Domain (interfaces)  ->  Infrastructure (imp
 
 **Domain** — each package defines its own interface in `iface.go`. No cross-domain imports.
 
-**Infrastructure** — concrete implementations that satisfy domain interfaces: SQLite, OpenAI API, fsnotify, file extractor.
+**Infrastructure** — concrete implementations that satisfy domain interfaces: SQLite, the embedding providers (local ONNX model and OpenAI API), fsnotify, file extractor.
 
 ## Composition Root
 
@@ -30,7 +30,7 @@ Six interfaces, each in its own package:
 | Interface | Package | Defined in | Implemented by |
 |-----------|---------|------------|----------------|
 | `Store` | `internal/store` | `iface.go` | `sqlite.go` (SQLite + sqlite-vec) |
-| `Embedder` | `internal/embeddings` | `iface.go` | `openai.go` (OpenAI API) |
+| `Embedder` | `internal/embeddings` | `iface.go` | `local/` (bundled ONNX model, **default**) and `openai.go` (OpenAI API, opt-in) |
 | `Chunker` | `internal/chunker` | `iface.go` | `chunker.go` (tiktoken, cl100k_base) |
 | `Watcher` | `internal/watcher` | `iface.go` | `fswatcher.go` (fsnotify) |
 | `FileEventHandler` | `internal/watcher` | `iface.go` | `engine.go` (Engine implements OnCreate/OnModify/OnDelete) |
@@ -63,7 +63,8 @@ tray.go              System tray (macOS CGo)
 internal/
   domain/types.go    Shared types: Directory, File, Chunk, SearchResult, IndexStats, ActivityLogEntry
   store/             Store interface + SQLite implementation
-  embeddings/        Embedder interface + OpenAI implementation
+  embeddings/        Embedder interface + OpenAI implementation (opt-in)
+    local/           Bundled ONNX local embedder (default provider)
   chunker/           Chunker interface + token-based splitter
   watcher/           Watcher + FileEventHandler interfaces + fsnotify implementation
   extractor/         Extractor interface + multi-format file extraction
@@ -96,17 +97,36 @@ One binary, two modes controlled by the `--mcp` flag:
 
 Both share the same SQLite database. WAL mode supports concurrent readers with one writer. The GUI writes, the stdio process reads.
 
-## Embedding Model Change Flow
+## Embedding Providers
 
-When the user changes the embedding model in Settings:
+Two infrastructure implementations satisfy the same `Embedder` interface, so the engine, store, watcher, and MCP layers stay provider-agnostic:
 
-1. `app.SetConfig("embedding_model", newModel)` detects the change
-2. Persists the new model to the config table
-3. Creates a new embedder via the injected `EmbedderFactory` (so `app.go` never imports concrete embeddings)
-4. Calls `engine.SetEmbedder()` to swap it
+- **Local (default)** — `internal/embeddings/local/`. A bundled `multilingual-e5-small` ONNX model (384 dimensions) runs in-process on the CPU via ONNX Runtime (`yalue/onnxruntime_go`), with a Hugging Face tokenizer (`daulet/tokenizers`). The session is lazy-initialized on first use; inference tokenizes, runs the model, mean-pools over the attention mask, and L2-normalizes. In stdio (`--mcp`) mode the model loads lazily on the first search so the MCP handshake stays instant. Works fully offline.
+- **OpenAI (opt-in)** — `openai.go`. Used only when the user supplies an API key and selects the OpenAI provider in Settings.
+
+`main.go` picks the implementation from the `embedding_provider` config key; the local branch also wires the model-matched tokenizer into the chunker (via `chunker.WithTokenizer`) so chunk boundaries are measured in the model's own tokens.
+
+### Asset Bundling
+
+The native artifacts (ONNX model weights, tokenizer, ONNX Runtime shared library) are **not** committed to git. Instead:
+
+- `assets/manifest.json` (in git) pins each artifact's URL and SHA-256 checksum.
+- `make assets` downloads and checksum-verifies them into `assets/embedded/` (gitignored).
+- The `localembed`-tagged build embeds them into the binary via `go:embed`; on first use they are extracted (atomically, checksum-verified) to `~/.agent-memory/runtime/` because the ORT library must be `dlopen`-ed from a real file path.
+
+The native-lib-dependent code lives behind the `//go:build localembed` tag, so the default `go build`/`go test ./...` (and CI) stays lib-free and green; the pure-Go embedding pipeline is unit-testable without the tag. `make build` sets `-tags localembed` plus the `CGO_LDFLAGS` to link the tokenizer library.
+
+## Embedding Provider / Model Change Flow
+
+When the user changes the embedding provider or model in Settings:
+
+1. `app.SetConfig` detects the change (`embedding_provider` or `embedding_model`)
+2. Persists the new value to the config table
+3. Creates a new embedder via the injected `EmbedderFactory` (so `app.go` never imports concrete embeddings) — the factory maps the provider to `local.New(...)` or `NewOpenAIEmbedder(...)`
+4. Calls `engine.SetEmbedder()` (and, for a provider switch, swaps the matching chunker/tokenizer) to swap it
 5. Calls `engine.Reset()` which stops the watcher, drops and recreates the vector table with the new dimension, and restarts
 
-The same factory pattern applies when the API key changes — the embedder is swapped so new requests use the updated key immediately.
+The same factory pattern applies when the OpenAI API key changes — the embedder is swapped so new requests use the updated key immediately.
 
 ## Testing
 

Documentation/Bugs/fswatcher-create-event-swallowed-linux.md

@@ -0,0 +1,54 @@
+# Bug: watcher swallows Create events on Linux (debounce replaces instead of merges)
+
+**Date found:** 2026-07-16
+**Found during:** local-embeddings epic, Phase 5 Linux verification (first-ever test run on Linux)
+**Status:** Fix submitted — PR #5 (`fix/watcher-create-swallowed-linux`, based on `main`, independent of the epic PR stack)
+**Severity:** Low today (no user-visible breakage), latent correctness risk
+
+## Symptom
+
+`go test ./internal/watcher/` fails deterministically on Linux (3/3 runs, Docker
+`golang:1.26-bookworm`, arm64):
+
+```
+--- FAIL: TestOnCreate (2.01s)
+    fswatcher_test.go:121: expected OnCreate to be called
+```
+
+`TestOnModify` and `TestOnDelete` pass. The full suite is green on macOS.
+
+## Root cause
+
+Writing a new file on Linux (inotify) emits **two events** for the same path within
+milliseconds: `CREATE`, then `WRITE`. In `internal/watcher/fswatcher.go`,
+`handleEvent` → `debounce(path, fn)` stores **one pending closure per path and
+replaces it** on each new event (`fswatcher.go:114-128`): the `WRITE` event cancels
+the `CREATE` timer and substitutes a closure that only sees `WRITE`. When the
+debounce fires, the handler classifies the event as a modification —
+`handler.OnCreate` is never called for newly written files.
+
+macOS (fsnotify kqueue/FSEvents backend) delivers/coalesces these events differently,
+so the swallow never manifests there.
+
+## Impact
+
+- **Today: effectively none for users.** `engine.OnCreate` and `engine.OnModify`
+  are identical (`engine.go:717-747` — both call `IndexFile`), so new files on
+  Linux are still indexed, merely misclassified as modifications.
+- **Latent risk:** any future divergence between create and modify handling
+  (e.g. create-only bookkeeping, activity-log semantics, per-event UX) silently
+  breaks on Linux only.
+- Blocks a fully green `go test ./...` on Linux (Phase 5 verify gate) until fixed.
+
+## Fix (applied in PR #5)
+
+In the debouncer, **accumulate the fsnotify op bits per path** instead of replacing
+the closure — e.g. keep `pendingOps map[string]fsnotify.Op`, OR-ing each event's op;
+when the timer fires, classify with precedence Remove/Rename > Create > Write/Chmod
+from the merged bits, then clear the entry. Semantics on macOS are unchanged
+(single-op case degenerates to today's behavior); Linux create+write merges to
+Create. `TestOnCreate` then passes on both platforms.
+
+Fix delivered as its own small PR per working rules (the watcher is outside the
+local-embeddings epic's scope): **PR #5**. Verified there: watcher suite 3/3 pass
+on Linux arm64 (was 3/3 fail) and 3/3 pass on macOS.

Documentation/Bugs/local-embed-token-budget-overflow.md

@@ -0,0 +1,54 @@
+# Bug: local embedder overflows the model context — every multi-chunk file silently dropped
+
+**Date found:** 2026-07-17
+**Found during:** Phase 5 Windows GUI field test (first real-world vault ever indexed)
+**Status:** Fixed — branch `fix/local-embed-token-budget` (stacks on the epic PR chain)
+**Severity:** Critical for the local provider — affects ALL platforms, latent since Phase 3
+
+## Symptom
+
+Indexing a real vault (109 files, D:\Brain) silently indexed only 26 files. No errors
+anywhere user-visible (see the companion visibility bug, PR #8). The 26 survivors were
+exactly the files ≤ 1.6 KB; every file ≥ 1.7 KB — i.e., anything needing more than one
+chunk — was dropped. All previous platform verifications used tiny single-chunk test
+files, so five platforms' smokes all passed while the pipeline was broken for real
+workloads everywhere.
+
+## Root cause (the "512 by 516" error)
+
+ONNX Runtime fails hard when a sequence exceeds mE5-small's 512-token context:
+
+```
+BroadcastIterator::Append ... Attempting to broadcast an axis by a dimension other than 1. 512 by 516
+```
+
+`buildChunker` passed `embedder.MaxInputTokens()` (512) straight through as the chunk
+budget. But the embedder prepends the E5 instruction prefix ("passage: ") and the
+tokenizer adds special tokens **after** chunking, so a chunk cut at exactly 512 tokens
+reached the model at 516. The epic specified the defense precisely ("effective chunk
+size = min(chunk_size, MaxInputTokens − prefix − special) → default 480"); the
+implementation skipped the reservation.
+
+The unchunked **query path** was also exposed: `EmbedQuery` has no chunker, so a search
+query longer than ~500 tokens crashed inference on every platform.
+
+## Fix (two layers)
+
+1. `app.go buildChunker`: chunk budget = `MaxInputTokens() − local.EmbedTokenReserve`
+   (32 → effective 480, matching the epic).
+2. `LocalEmbedder.embedBatch`: defensively truncate any tokenized input to the model
+   limit, preserving the trailing EOS token — protects queries and any future caller
+   regardless of chunking correctness.
+
+## Verification
+
+- Unit: truncation table test + fake-session proof the model never receives >512 tokens.
+- Integration (permanent): `TestIntegrationLargeDocument` — real pipeline, multi-chunk
+  document + over-long query. Failed before the fix on macOS AND Windows; passes on both.
+- Field: clean re-index of the same vault on Windows → **105 files / 932 chunks, all 5
+  dossier files present, 79 multi-chunk files, zero errors** (was 26/26/0).
+
+## Lesson recorded
+
+Platform verifications used single-chunk corpora only; a large-document case is now a
+permanent integration test, and the Phase 6 checklist gained a real-vault field test.

Documentation/Bugs/pdf-extraction-passthrough.md

@@ -0,0 +1,43 @@
+# Bug: PDF "extraction" embeds raw bytes, not text
+
+**Status:** Open · **Severity:** Medium (silent quality loss) · **Reported:** 2026-07-06
+**Component:** `internal/extractor` · **Track:** separate from the local-embeddings epic (pre-existing)
+
+## Summary
+
+PDF files are treated as "supported," but their text is never actually extracted.
+`extractPDFPassthrough` (`internal/extractor/extractor.go:177`) does `os.ReadFile(path)` and returns
+the **raw PDF bytes** as the `Text` to chunk and embed. PDFs are binary (compressed content streams,
+xref tables, object dictionaries), so what gets embedded is mostly non-text noise, not the
+document's readable content.
+
+## Impact
+
+- Semantic search over PDFs is effectively broken — matches are against binary noise, not content.
+- The README claims "PDF — text content extraction," which the code does not do.
+- Under the local-embeddings epic, the same garbage would burn **local CPU** instead of API dollars.
+
+## Evidence
+
+`internal/extractor/extractor.go`:
+
+```go
+func extractPDFPassthrough(path string) (Result, error) {
+    data, err := os.ReadFile(path)
+    if err != nil { return Result{}, err }
+    return Result{Text: string(data)}, nil   // raw PDF bytes, not extracted text
+}
+```
+
+`.pdf` is registered in `binaryExtractors`, so the file is reported as supported and indexed.
+
+## Proposed fix (separate task)
+
+Replace the passthrough with real PDF text extraction (a pure-Go PDF text library returning
+concatenated page text), and add a unit test with a small sample PDF. Keep this **out of the
+local-embeddings epic scope** — it is an independent defect.
+
+## Notes
+
+Found during the Phase-0 "get familiar / try to break it" pass. Logged per the bug-report workflow;
+not a blocker for the epic.