Phase 5 Windows: DLL + source-built tokenizer, on-device verified [draft, stacks on #6]

Documentation/Epics/local-embeddings.md

@@ -1,7 +1,10 @@
 # Epic: Local Embeddings as the Primary Provider
 
 **Date:** 2026-07-02
-**Status:** In Progress — Phase 5 started (foundation: per-platform go:embed split, branch `feat/xplat-foundation`); Phases 0–4 complete and in PR #2 (draft, awaiting review); Phase 6 (cleanup) remains
+**Status:** In Progress — Phase 5 nearly complete: all four targets built & verified (macOS
+arm64 shipped in PR #2; Linux, macOS x86_64, Windows x64 in stacked PRs #3/#4/#6/#7). Only the
+linux-amd64 *runtime* smoke (artifacts pinned, needs an x64 Linux box) and Phase 6 (cleanup)
+remain. PRs all draft, awaiting Bo's review.
 **Owner:** Bo Motlagh
 
 ## Goal
@@ -486,6 +489,36 @@ and the indexing-progress UX all already exist and are the extension points.
   x86_64 Wails app builds, extracts its assets to its own arch dir, and answers an MCP
   search correctly with both arch dirs coexisting. Native arm64 re-verified after.
 
+**Windows x64 findings (recorded 2026-07-17, branch `feat/xplat-windows`):**
+
+- **Built on-device over SSH** (Tailscale) to the Windows PC, driven from the dev Mac.
+  Toolchain installed via scoop: Go 1.26.5, MinGW gcc 16.1.0, rustup-gnu + cargo 1.97.1,
+  make. Recipe + provenance in `Documentation/windows-build.md`.
+- **libtokenizers.a source-built** (no upstream Windows prebuilt): `daulet/tokenizers` v1.27.0
+  tag, **GNU** Rust toolchain (must match MinGW gcc; MSVC would produce an unlinkable `.lib`).
+  The v1.27.0 layout emits `libtokenizers_ffi.a` — renamed to `libtokenizers.a` on packaging.
+  Published as repo release **`tokenizers-1.27.0-windows-x64`**, SHA-256 pinned. ONNX Runtime
+  DLL is the **official** Microsoft `onnxruntime-win-x64-1.26.0.zip` (member-pinned).
+- **`assets_embed_windows.go`** embeds `onnxruntime.dll`; **Makefile** learned to extract
+  `.zip` archives (unzip → Windows `tar.exe` fallback) so `make assets` runs on Windows.
+- **Two Windows-only build gaps found and fixed in the Makefile** (both invisible on
+  macOS/Linux, guarded by `GOOS=windows`):
+  1. The Rust static lib needs NT/Winsock/crypto syscall libs MinGW doesn't link by default
+     (`undefined reference to Nt*/Rtl*`) → `LINK_LIBS` appends `-lntdll -lws2_32 -lbcrypt
+     -luserenv -ladvapi32 -lkernel32 -lncrypt`.
+  2. sqlite-vec's cgo `#include "sqlite3.h"` has no system header on Windows (macOS SDK /
+     Linux libsqlite3 supplied it) → new `winhdr` step stages the headers mattn/go-sqlite3
+     bundles (its `sqlite3-binding.h` IS the amalgamation `sqlite3.h`) into `build/winhdr`,
+     version-matched to the SQLite mattn compiles in.
+- **Verified natively on Windows x64:** `make assets` downloads + checksum-verifies all four
+  artifacts (incl. `.zip` extraction and our published tokenizer release); integration test
+  links and passes (cosine ordering 0.140 < 0.171 < 0.286, matching every platform); full
+  `make build` produces `agent-memory.exe` (193 MB PE32+ GUI); the exe extracts its embedded
+  assets to `windows-amd64-<fingerprint>/` and answers an MCP search on a Mac-indexed DB
+  ("how do I cook italian pasta" → carbonara-recipe.md) — cross-platform vector compatibility
+  confirmed. (Network not forcibly disabled — SSH session — but the local provider makes no
+  outbound calls by design; the `--network none` Linux/Intel runs already proved that property.)
+
 1. Linux x64/arm64: assets manifest entries, CI build, smoke test.
 2. Windows x64: CI job builds `libtokenizers.a` with Rust toolchain (no published binary);
    ORT DLL from official release; smoke test.
@@ -502,7 +535,24 @@ and the indexing-progress UX all already exist and are the extension points.
 2. Architecture rule check per `Documentation/ARCHITECTURE.md` (inward deps, wiring in main.go,
    mocks for all interfaces, thin delivery layer).
 3. `go vet ./...`, `go test ./...`, `make build`, full manual test both modes.
-4. Update this epic's Status to Complete; record the chosen default model and measured numbers.
+4. **Verification gaps carried from Phase 5** (recorded 2026-07-17 — coverage gaps, not known
+   defects; each Phase 5 platform's core inference + search path IS verified):
+   - [ ] **Windows GUI smoke** (~5 min, human at the PC screen — headless SSH couldn't do it):
+     launch `agent-memory.exe`, keyless onboarding, index a folder, search from the UI.
+     All Windows verification so far was headless (MCP stdio path only).
+   - [x] **Full untagged `go test ./...` on Windows** — DONE 2026-07-17: every package green
+     except `TestOnCreate`, which fails exactly as on Linux (assumption CONFIRMED — Windows
+     delivers the same create+write double-event). PR #5's fix branch verified on Windows:
+     watcher suite 3/3 pass. So the sole Windows failure is the known bug with a proven fix;
+     re-run once #5 merges into the stack for the final green checkmark.
+   - [x] **linux-amd64 runtime smoke** — DONE 2026-07-17 on real x64 hardware (Surface Book
+     i7, Pop!_OS 22.04): binary built in an Ubuntu 22.04 amd64 container (glibc-matched;
+     `go build` with wails production tags — the full wails-CLI build path was already proven
+     on linux-arm64), integration test passed in-container (0.140 < 0.171 < 0.286), then the
+     binary ran natively on the Surface with **zero library installs** (`ldd` clean against
+     stock webkit2gtk-4.0/gtk-3), extracted assets to `linux-amd64-<fingerprint>/`, and
+     answered the MCP search correctly against a macOS-indexed DB.
+5. Update this epic's Status to Complete; record the chosen default model and measured numbers.
 
 ## Phase 3 — Detailed Plan (DRAFT, pending Bo review)
 

Documentation/windows-build.md

@@ -0,0 +1,90 @@
+# Windows x64 Build & Artifact Provenance (local-embeddings, Phase 5)
+
+How the Windows local-embedding build is produced and how to reproduce its one
+source-built artifact. Records what was actually run (versions below are what
+shipped), so a future maintainer can rebuild after a dependency bump.
+
+## Why Windows needs a build step at all
+
+Every other platform gets both native libraries as official prebuilts. Windows
+is the exception on one of them:
+
+| Artifact | Windows source |
+|---|---|
+| ONNX Runtime DLL | **Official** Microsoft release (`onnxruntime-win-x64-1.26.0.zip`) — pinned directly in `assets/manifest.json`. |
+| `libtokenizers.a` (static) | **No upstream Windows prebuilt** — `daulet/tokenizers` publishes macOS/Linux only. We build it from source and host it in this repo's releases (`tokenizers-1.27.0-windows-x64`). |
+
+## Toolchain (installed via scoop)
+
+```
+scoop install go mingw rustup-gnu make
+rustup default stable-x86_64-pc-windows-gnu
+```
+
+Verified versions: Go 1.26.5, MinGW gcc 16.1.0, rustc/cargo 1.97.1 (GNU),
+GNU make 4.4.1.
+
+**The GNU pairing is load-bearing.** CGo links the Go binary with MinGW `gcc`,
+so the Rust static lib must come from the `*-windows-gnu` toolchain. Building it
+with the default MSVC toolchain yields a `.lib` in a format MinGW cannot link.
+
+## Reproducing `libtokenizers.a`
+
+```
+git clone --depth 1 --branch v1.27.0 https://github.com/daulet/tokenizers
+cd tokenizers
+cargo build --release
+# Output: target/release/libtokenizers_ffi.a  (~40 MB)
+```
+
+Note the **name change**: this repo layout (v1.27.0) emits `libtokenizers_ffi.a`
+from the `tokenizers-ffi` crate. The macOS/Linux prebuilts — and therefore the
+`-ltokenizers` link flag and the manifest's `member: libtokenizers.a` — expect
+`libtokenizers.a`. Same archive, so we simply rename on packaging:
+
+```
+cp target/release/libtokenizers_ffi.a libtokenizers.a
+tar czf libtokenizers.windows-x86_64.tar.gz libtokenizers.a
+# publish as a repo release, pin archive + member SHA-256 in the manifest
+```
+
+## Building the app
+
+```
+git clone https://github.com/unitedeffectslabs/agent-memory
+cd agent-memory && git checkout feat/xplat-windows
+cd frontend && npm install && cd ..
+make assets      # downloads DLL + model + tokenizer + our published static lib; verifies all checksums
+make build       # wails build -skipbindings -tags localembed  →  build/bin/agent-memory.exe
+```
+
+`make assets` extracts the `.zip` ORT archive via `unzip` if present, else falls
+back to Windows' bundled `C:\Windows\System32\tar.exe` (Git Bash's GNU tar
+cannot read zips). Checksums use `sha256sum` if present, else `shasum -a 256`
+(both provided by Git for Windows).
+
+## Verification (the Phase 5 gate)
+
+1. Integration test — real tokenizer + ORT + model, as a native Windows binary:
+   ```
+   CGO_LDFLAGS="-L$PWD/internal/embeddings/local/lib -ltokenizers" \
+     go test -tags localembed -count=1 -run TestIntegrationEmbed -v ./internal/embeddings/local/
+   ```
+   Expect PASS, cosine ordering ≈ related 0.14 < cross-lingual 0.17 < unrelated
+   0.29 (matches macOS arm64/x86_64 and Linux).
+2. Offline app run: disconnect network, `agent-memory.exe --db test.db`, onboard
+   with **no API key**, index a small folder, confirm search returns matches.
+
+## Troubleshooting (observed / anticipated)
+
+- **Missing Windows system symbols at link** (`undefined reference to Nt*` /
+  `Rtl*`, `ws2_32`, `bcrypt`, `userenv`): the Rust static lib pulls in NT/Winsock/
+  crypto syscalls MinGW doesn't link by default. **Handled** — the Makefile's
+  `LINK_LIBS` appends `-lntdll -lws2_32 -lbcrypt -luserenv -ladvapi32 -lkernel32
+  -lncrypt` when `GOOS=windows` (empty elsewhere). Observed and fixed during the
+  first Windows build; listed here in case a tokenizers/Rust bump adds more.
+- **`onnxruntime_providers_shared.dll` load error**: the official zip ships this
+  second DLL; CPU-only use normally doesn't need it, but if ORT fails to load,
+  pin it as an extra `embedded/` artifact beside the main DLL.
+- **Defender quarantines the fresh unsigned exe**: add a temp exclusion for the
+  repo folder while testing.

Makefile

@@ -1,4 +1,4 @@
-.PHONY: build build-darwin-amd64 dev test clean assets
+.PHONY: build build-darwin-amd64 dev test clean assets winhdr
 
 # --- Local-embedding asset bundling -----------------------------------------
 # Artifacts (model, tokenizer, ONNX Runtime dylib, static tokenizer lib) are
@@ -15,8 +15,35 @@ PLATFORM  := $(shell go env GOOS)-$(shell go env GOARCH)
 # "<hash>  <file>", so the awk '{print $1}' callers work with either.
 SHA256    := $(shell command -v sha256sum >/dev/null 2>&1 && echo sha256sum || echo shasum -a 256)
 
-build: assets
-	CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) -ltokenizers" wails build -skipbindings -tags localembed
+# Windows-only: the source-built libtokenizers.a (Rust std, GNU toolchain) pulls
+# in low-level NT/Winsock/crypto syscalls that MinGW does not link by default.
+# Naming them here resolves "undefined reference to Nt*/Rtl*" at link time.
+# Empty on macOS/Linux, so those builds are unaffected.
+LINK_LIBS := -ltokenizers
+CGO_EXTRA_CFLAGS :=
+WIN_PREREQ :=
+ifeq ($(shell go env GOOS),windows)
+LINK_LIBS += -lntdll -lws2_32 -lbcrypt -luserenv -ladvapi32 -lkernel32 -lncrypt
+# sqlite-vec's cgo build #includes sqlite3.h / sqlite3ext.h, which macOS and
+# Linux supply from the system but Windows does not. Stage the exact headers
+# mattn/go-sqlite3 bundles (its sqlite3-binding.h IS the amalgamation sqlite3.h),
+# so they match the SQLite that mattn compiles in — see the winhdr target.
+CGO_EXTRA_CFLAGS := -I$(PWD)/build/winhdr
+WIN_PREREQ := winhdr
+endif
+
+# Stage sqlite headers for the Windows build from the mattn/go-sqlite3 module
+# (Windows has no system sqlite3.h). No-op / unused on macOS and Linux.
+winhdr:
+	@mkdir -p "$(PWD)/build/winhdr"
+	@d=$$(go list -m -f '{{.Dir}}' github.com/mattn/go-sqlite3); \
+	  d=$$(cygpath -u "$$d" 2>/dev/null || echo "$$d"); \
+	  cp "$$d/sqlite3-binding.h" "$(PWD)/build/winhdr/sqlite3.h"; \
+	  cp "$$d/sqlite3ext.h" "$(PWD)/build/winhdr/sqlite3ext.h"; \
+	  echo ">> staged Windows sqlite headers from $$d"
+
+build: assets $(WIN_PREREQ)
+	CGO_CFLAGS="$(CGO_EXTRA_CFLAGS)" CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) $(LINK_LIBS)" wails build -skipbindings -tags localembed
 
 # Cross-build the Intel-mac app from an arm64 Mac. GOARCH=amd64 makes the
 # assets target fetch the darwin-amd64 artifacts (the embedded/ and lib/ dirs
@@ -26,8 +53,8 @@ build-darwin-amd64:
 	GOARCH=amd64 $(MAKE) assets
 	CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) -ltokenizers" wails build -skipbindings -tags localembed -platform darwin/amd64
 
-dev: assets
-	CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) -ltokenizers" wails dev -tags localembed
+dev: assets $(WIN_PREREQ)
+	CGO_CFLAGS="$(CGO_EXTRA_CFLAGS)" CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) $(LINK_LIBS)" wails dev -tags localembed
 
 test:
 	go test ./...
@@ -66,7 +93,12 @@ assets:
 	  mkdir -p "$$(dirname "$$destpath")"; \
 	  if [ -n "$$member" ]; then \
 	    xd=$$(mktemp -d); \
-	    tar xzf "$$tmp" -C "$$xd" "$$member"; \
+	    case "$$url" in \
+	      *.zip) if command -v unzip >/dev/null 2>&1; then unzip -q "$$tmp" "$$member" -d "$$xd"; \
+	             elif [ -x /c/Windows/System32/tar.exe ]; then /c/Windows/System32/tar.exe xf "$$tmp" -C "$$xd" "$$member"; \
+	             else tar xf "$$tmp" -C "$$xd" "$$member"; fi ;; \
+	      *) tar xzf "$$tmp" -C "$$xd" "$$member" ;; \
+	    esac; \
 	    cp "$$xd/$$member" "$$destpath"; \
 	    rm -rf "$$xd"; \
 	    got2=$$($(SHA256) "$$destpath" | awk '{print $$1}'); \

app.go

@@ -10,6 +10,7 @@ import (
 	"log"
 	"os"
 	"path/filepath"
+	goruntime "runtime"
 	"strconv"
 
 	"github.com/borzou/vecstore/internal/chunker"
@@ -384,10 +385,28 @@ func generateToken() (string, error) {
 	return hex.EncodeToString(b), nil
 }
 
-// defaultClaudeDesktopConfigPath returns the standard Claude Desktop config location.
+// defaultClaudeDesktopConfigPath returns the standard Claude Desktop config
+// location for the current OS. Claude Desktop reads a different path per
+// platform; writing the macOS path on Windows produces a config it never sees
+// (while still reporting success — the Phase 5 Windows smoke caught exactly that).
 func defaultClaudeDesktopConfigPath() string {
-	home, _ := os.UserHomeDir()
-	return filepath.Join(home, "Library", "Application Support", "Claude", "claude_desktop_config.json")
+	switch goruntime.GOOS {
+	case "windows":
+		if appData := os.Getenv("APPDATA"); appData != "" {
+			return filepath.Join(appData, "Claude", "claude_desktop_config.json")
+		}
+		home, _ := os.UserHomeDir()
+		return filepath.Join(home, "AppData", "Roaming", "Claude", "claude_desktop_config.json")
+	case "darwin":
+		home, _ := os.UserHomeDir()
+		return filepath.Join(home, "Library", "Application Support", "Claude", "claude_desktop_config.json")
+	default: // linux
+		if xdg := os.Getenv("XDG_CONFIG_HOME"); xdg != "" {
+			return filepath.Join(xdg, "Claude", "claude_desktop_config.json")
+		}
+		home, _ := os.UserHomeDir()
+		return filepath.Join(home, ".config", "Claude", "claude_desktop_config.json")
+	}
 }
 
 // GetClaudeDesktopConfigPath returns the current config path (custom or default).

assets/manifest.json

@@ -1,6 +1,6 @@
 {
   "schema": 1,
-  "comment": "Pinned local-embedding artifacts (URLs + SHA-256). Weights/libs are NEVER committed; `make assets` downloads and checksum-verifies these into internal/embeddings/local/embedded/ (go:embed runtime assets) and internal/embeddings/local/lib/ (link-time static lib). For tarball artifacts, `sha256` is the checksum of the downloaded archive and `member_sha256` is the checksum of the extracted member. `dest` is relative to internal/embeddings/local/. Model + tokenizer entries are identical bytes on every platform (per-platform keyed for a uniform `make assets`). Remaining Phase 5 platform: windows-amd64. darwin-amd64's onnxruntime is our own source build (official mac-Intel prebuilts stopped at ORT 1.23) hosted in this repo's GitHub releases — see the ort-1.26.0-darwin-x64 release notes for the reproducible recipe.",
+  "comment": "Pinned local-embedding artifacts (URLs + SHA-256). Weights/libs are NEVER committed; `make assets` downloads and checksum-verifies these into internal/embeddings/local/embedded/ (go:embed runtime assets) and internal/embeddings/local/lib/ (link-time static lib). For tarball artifacts, `sha256` is the checksum of the downloaded archive and `member_sha256` is the checksum of the extracted member. `dest` is relative to internal/embeddings/local/. Model + tokenizer entries are identical bytes on every platform (per-platform keyed for a uniform `make assets`). darwin-amd64's onnxruntime is our own source build (official mac-Intel prebuilts stopped at ORT 1.23) hosted in this repo's GitHub releases — see the ort-1.26.0-darwin-x64 release notes for the reproducible recipe. windows-amd64's tokenizers_static has no upstream prebuilt: it is built from Rust source per Documentation/windows-build.md and hosted in this repo's releases (tokenizers-1.27.0-windows-x64).",
   "platforms": {
     "darwin-arm64": {
       "model": {
@@ -28,6 +28,32 @@
         "dest": "lib/libtokenizers.a"
       }
     },
+    "windows-amd64": {
+      "model": {
+        "url": "https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx",
+        "sha256": "f80102d3f2a1229f387d3c81909990d8945513e347b0eab049f7de3c6f98c193",
+        "dest": "embedded/model_quantized.onnx"
+      },
+      "tokenizer": {
+        "url": "https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/tokenizer.json",
+        "sha256": "0b44a9d7b51c3c62626640cda0e2c2f70fdacdc25bbbd68038369d14ebdf4c39",
+        "dest": "embedded/tokenizer.json"
+      },
+      "onnxruntime": {
+        "url": "https://github.com/microsoft/onnxruntime/releases/download/v1.26.0/onnxruntime-win-x64-1.26.0.zip",
+        "sha256": "6ebe99b5564bf4d029b6e93eac9ff423682b6212eade769e9ca3f685eaf500b4",
+        "member": "onnxruntime-win-x64-1.26.0/lib/onnxruntime.dll",
+        "member_sha256": "b2ba7ca16e0e4fe71ad5148744ab885a2f5809e52a0c3de4d9ba3853a03977f9",
+        "dest": "embedded/onnxruntime.dll"
+      },
+      "tokenizers_static": {
+        "url": "https://github.com/unitedeffectslabs/agent-memory/releases/download/tokenizers-1.27.0-windows-x64/libtokenizers.windows-x86_64.tar.gz",
+        "sha256": "882f520174a6cb14dcf4dca559375d65915a69a075bd85c22e40463bc0b466a8",
+        "member": "libtokenizers.a",
+        "member_sha256": "5e0815434a9d9eea40638ae9303f239c85059868673dc7e2a5a1eee72a9ad692",
+        "dest": "lib/libtokenizers.a"
+      }
+    },
     "darwin-amd64": {
       "model": {
         "url": "https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx",

internal/embeddings/local/assets_embed_windows.go

@@ -0,0 +1,19 @@
+//go:build localembed && windows
+
+package local
+
+import "embed"
+
+// Per-platform ONNX Runtime shared library (Windows x64). One binary can embed
+// only one platform's ORT lib (epic Open Concern #8); the official Microsoft
+// release ships the DLL — see assets/manifest.json (windows-amd64) and
+// Documentation/windows-build.md for the full Windows build procedure
+// (the static tokenizer lib has no upstream prebuilt and is built from Rust
+// source there).
+//
+//go:embed embedded/onnxruntime.dll
+var embeddedORTLib embed.FS
+
+// ortLibFile is the base name of this platform's embedded ONNX Runtime shared
+// library, both inside embedded/ and after extraction to the runtime dir.
+const ortLibFile = "onnxruntime.dll"