diff --git a/Documentation/Epics/local-embeddings.md b/Documentation/Epics/local-embeddings.md index 61bba43..5f4b74f 100644 --- a/Documentation/Epics/local-embeddings.md +++ b/Documentation/Epics/local-embeddings.md @@ -1,7 +1,10 @@ # Epic: Local Embeddings as the Primary Provider **Date:** 2026-07-02 -**Status:** In Progress — Phase 5 started (foundation: per-platform go:embed split, branch `feat/xplat-foundation`); Phases 0–4 complete and in PR #2 (draft, awaiting review); Phase 6 (cleanup) remains +**Status:** In Progress — Phase 5 nearly complete: all four targets built & verified (macOS +arm64 shipped in PR #2; Linux, macOS x86_64, Windows x64 in stacked PRs #3/#4/#6/#7). Only the +linux-amd64 *runtime* smoke (artifacts pinned, needs an x64 Linux box) and Phase 6 (cleanup) +remain. PRs all draft, awaiting Bo's review. **Owner:** Bo Motlagh ## Goal @@ -486,6 +489,36 @@ and the indexing-progress UX all already exist and are the extension points. x86_64 Wails app builds, extracts its assets to its own arch dir, and answers an MCP search correctly with both arch dirs coexisting. Native arm64 re-verified after. +**Windows x64 findings (recorded 2026-07-17, branch `feat/xplat-windows`):** + +- **Built on-device over SSH** (Tailscale) to the Windows PC, driven from the dev Mac. + Toolchain installed via scoop: Go 1.26.5, MinGW gcc 16.1.0, rustup-gnu + cargo 1.97.1, + make. Recipe + provenance in `Documentation/windows-build.md`. +- **libtokenizers.a source-built** (no upstream Windows prebuilt): `daulet/tokenizers` v1.27.0 + tag, **GNU** Rust toolchain (must match MinGW gcc; MSVC would produce an unlinkable `.lib`). + The v1.27.0 layout emits `libtokenizers_ffi.a` — renamed to `libtokenizers.a` on packaging. + Published as repo release **`tokenizers-1.27.0-windows-x64`**, SHA-256 pinned. ONNX Runtime + DLL is the **official** Microsoft `onnxruntime-win-x64-1.26.0.zip` (member-pinned). +- **`assets_embed_windows.go`** embeds `onnxruntime.dll`; **Makefile** learned to extract + `.zip` archives (unzip → Windows `tar.exe` fallback) so `make assets` runs on Windows. +- **Two Windows-only build gaps found and fixed in the Makefile** (both invisible on + macOS/Linux, guarded by `GOOS=windows`): + 1. The Rust static lib needs NT/Winsock/crypto syscall libs MinGW doesn't link by default + (`undefined reference to Nt*/Rtl*`) → `LINK_LIBS` appends `-lntdll -lws2_32 -lbcrypt + -luserenv -ladvapi32 -lkernel32 -lncrypt`. + 2. sqlite-vec's cgo `#include "sqlite3.h"` has no system header on Windows (macOS SDK / + Linux libsqlite3 supplied it) → new `winhdr` step stages the headers mattn/go-sqlite3 + bundles (its `sqlite3-binding.h` IS the amalgamation `sqlite3.h`) into `build/winhdr`, + version-matched to the SQLite mattn compiles in. +- **Verified natively on Windows x64:** `make assets` downloads + checksum-verifies all four + artifacts (incl. `.zip` extraction and our published tokenizer release); integration test + links and passes (cosine ordering 0.140 < 0.171 < 0.286, matching every platform); full + `make build` produces `agent-memory.exe` (193 MB PE32+ GUI); the exe extracts its embedded + assets to `windows-amd64-/` and answers an MCP search on a Mac-indexed DB + ("how do I cook italian pasta" → carbonara-recipe.md) — cross-platform vector compatibility + confirmed. (Network not forcibly disabled — SSH session — but the local provider makes no + outbound calls by design; the `--network none` Linux/Intel runs already proved that property.) + 1. Linux x64/arm64: assets manifest entries, CI build, smoke test. 2. Windows x64: CI job builds `libtokenizers.a` with Rust toolchain (no published binary); ORT DLL from official release; smoke test. @@ -502,7 +535,24 @@ and the indexing-progress UX all already exist and are the extension points. 2. Architecture rule check per `Documentation/ARCHITECTURE.md` (inward deps, wiring in main.go, mocks for all interfaces, thin delivery layer). 3. `go vet ./...`, `go test ./...`, `make build`, full manual test both modes. -4. Update this epic's Status to Complete; record the chosen default model and measured numbers. +4. **Verification gaps carried from Phase 5** (recorded 2026-07-17 — coverage gaps, not known + defects; each Phase 5 platform's core inference + search path IS verified): + - [ ] **Windows GUI smoke** (~5 min, human at the PC screen — headless SSH couldn't do it): + launch `agent-memory.exe`, keyless onboarding, index a folder, search from the UI. + All Windows verification so far was headless (MCP stdio path only). + - [x] **Full untagged `go test ./...` on Windows** — DONE 2026-07-17: every package green + except `TestOnCreate`, which fails exactly as on Linux (assumption CONFIRMED — Windows + delivers the same create+write double-event). PR #5's fix branch verified on Windows: + watcher suite 3/3 pass. So the sole Windows failure is the known bug with a proven fix; + re-run once #5 merges into the stack for the final green checkmark. + - [x] **linux-amd64 runtime smoke** — DONE 2026-07-17 on real x64 hardware (Surface Book + i7, Pop!_OS 22.04): binary built in an Ubuntu 22.04 amd64 container (glibc-matched; + `go build` with wails production tags — the full wails-CLI build path was already proven + on linux-arm64), integration test passed in-container (0.140 < 0.171 < 0.286), then the + binary ran natively on the Surface with **zero library installs** (`ldd` clean against + stock webkit2gtk-4.0/gtk-3), extracted assets to `linux-amd64-/`, and + answered the MCP search correctly against a macOS-indexed DB. +5. Update this epic's Status to Complete; record the chosen default model and measured numbers. ## Phase 3 — Detailed Plan (DRAFT, pending Bo review) diff --git a/Documentation/windows-build.md b/Documentation/windows-build.md new file mode 100644 index 0000000..1df3cca --- /dev/null +++ b/Documentation/windows-build.md @@ -0,0 +1,90 @@ +# Windows x64 Build & Artifact Provenance (local-embeddings, Phase 5) + +How the Windows local-embedding build is produced and how to reproduce its one +source-built artifact. Records what was actually run (versions below are what +shipped), so a future maintainer can rebuild after a dependency bump. + +## Why Windows needs a build step at all + +Every other platform gets both native libraries as official prebuilts. Windows +is the exception on one of them: + +| Artifact | Windows source | +|---|---| +| ONNX Runtime DLL | **Official** Microsoft release (`onnxruntime-win-x64-1.26.0.zip`) — pinned directly in `assets/manifest.json`. | +| `libtokenizers.a` (static) | **No upstream Windows prebuilt** — `daulet/tokenizers` publishes macOS/Linux only. We build it from source and host it in this repo's releases (`tokenizers-1.27.0-windows-x64`). | + +## Toolchain (installed via scoop) + +``` +scoop install go mingw rustup-gnu make +rustup default stable-x86_64-pc-windows-gnu +``` + +Verified versions: Go 1.26.5, MinGW gcc 16.1.0, rustc/cargo 1.97.1 (GNU), +GNU make 4.4.1. + +**The GNU pairing is load-bearing.** CGo links the Go binary with MinGW `gcc`, +so the Rust static lib must come from the `*-windows-gnu` toolchain. Building it +with the default MSVC toolchain yields a `.lib` in a format MinGW cannot link. + +## Reproducing `libtokenizers.a` + +``` +git clone --depth 1 --branch v1.27.0 https://github.com/daulet/tokenizers +cd tokenizers +cargo build --release +# Output: target/release/libtokenizers_ffi.a (~40 MB) +``` + +Note the **name change**: this repo layout (v1.27.0) emits `libtokenizers_ffi.a` +from the `tokenizers-ffi` crate. The macOS/Linux prebuilts — and therefore the +`-ltokenizers` link flag and the manifest's `member: libtokenizers.a` — expect +`libtokenizers.a`. Same archive, so we simply rename on packaging: + +``` +cp target/release/libtokenizers_ffi.a libtokenizers.a +tar czf libtokenizers.windows-x86_64.tar.gz libtokenizers.a +# publish as a repo release, pin archive + member SHA-256 in the manifest +``` + +## Building the app + +``` +git clone https://github.com/unitedeffectslabs/agent-memory +cd agent-memory && git checkout feat/xplat-windows +cd frontend && npm install && cd .. +make assets # downloads DLL + model + tokenizer + our published static lib; verifies all checksums +make build # wails build -skipbindings -tags localembed → build/bin/agent-memory.exe +``` + +`make assets` extracts the `.zip` ORT archive via `unzip` if present, else falls +back to Windows' bundled `C:\Windows\System32\tar.exe` (Git Bash's GNU tar +cannot read zips). Checksums use `sha256sum` if present, else `shasum -a 256` +(both provided by Git for Windows). + +## Verification (the Phase 5 gate) + +1. Integration test — real tokenizer + ORT + model, as a native Windows binary: + ``` + CGO_LDFLAGS="-L$PWD/internal/embeddings/local/lib -ltokenizers" \ + go test -tags localembed -count=1 -run TestIntegrationEmbed -v ./internal/embeddings/local/ + ``` + Expect PASS, cosine ordering ≈ related 0.14 < cross-lingual 0.17 < unrelated + 0.29 (matches macOS arm64/x86_64 and Linux). +2. Offline app run: disconnect network, `agent-memory.exe --db test.db`, onboard + with **no API key**, index a small folder, confirm search returns matches. + +## Troubleshooting (observed / anticipated) + +- **Missing Windows system symbols at link** (`undefined reference to Nt*` / + `Rtl*`, `ws2_32`, `bcrypt`, `userenv`): the Rust static lib pulls in NT/Winsock/ + crypto syscalls MinGW doesn't link by default. **Handled** — the Makefile's + `LINK_LIBS` appends `-lntdll -lws2_32 -lbcrypt -luserenv -ladvapi32 -lkernel32 + -lncrypt` when `GOOS=windows` (empty elsewhere). Observed and fixed during the + first Windows build; listed here in case a tokenizers/Rust bump adds more. +- **`onnxruntime_providers_shared.dll` load error**: the official zip ships this + second DLL; CPU-only use normally doesn't need it, but if ORT fails to load, + pin it as an extra `embedded/` artifact beside the main DLL. +- **Defender quarantines the fresh unsigned exe**: add a temp exclusion for the + repo folder while testing. diff --git a/Makefile b/Makefile index 1a04c93..1dc8e30 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: build build-darwin-amd64 dev test clean assets +.PHONY: build build-darwin-amd64 dev test clean assets winhdr # --- Local-embedding asset bundling ----------------------------------------- # Artifacts (model, tokenizer, ONNX Runtime dylib, static tokenizer lib) are @@ -15,8 +15,35 @@ PLATFORM := $(shell go env GOOS)-$(shell go env GOARCH) # " ", so the awk '{print $1}' callers work with either. SHA256 := $(shell command -v sha256sum >/dev/null 2>&1 && echo sha256sum || echo shasum -a 256) -build: assets - CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) -ltokenizers" wails build -skipbindings -tags localembed +# Windows-only: the source-built libtokenizers.a (Rust std, GNU toolchain) pulls +# in low-level NT/Winsock/crypto syscalls that MinGW does not link by default. +# Naming them here resolves "undefined reference to Nt*/Rtl*" at link time. +# Empty on macOS/Linux, so those builds are unaffected. +LINK_LIBS := -ltokenizers +CGO_EXTRA_CFLAGS := +WIN_PREREQ := +ifeq ($(shell go env GOOS),windows) +LINK_LIBS += -lntdll -lws2_32 -lbcrypt -luserenv -ladvapi32 -lkernel32 -lncrypt +# sqlite-vec's cgo build #includes sqlite3.h / sqlite3ext.h, which macOS and +# Linux supply from the system but Windows does not. Stage the exact headers +# mattn/go-sqlite3 bundles (its sqlite3-binding.h IS the amalgamation sqlite3.h), +# so they match the SQLite that mattn compiles in — see the winhdr target. +CGO_EXTRA_CFLAGS := -I$(PWD)/build/winhdr +WIN_PREREQ := winhdr +endif + +# Stage sqlite headers for the Windows build from the mattn/go-sqlite3 module +# (Windows has no system sqlite3.h). No-op / unused on macOS and Linux. +winhdr: + @mkdir -p "$(PWD)/build/winhdr" + @d=$$(go list -m -f '{{.Dir}}' github.com/mattn/go-sqlite3); \ + d=$$(cygpath -u "$$d" 2>/dev/null || echo "$$d"); \ + cp "$$d/sqlite3-binding.h" "$(PWD)/build/winhdr/sqlite3.h"; \ + cp "$$d/sqlite3ext.h" "$(PWD)/build/winhdr/sqlite3ext.h"; \ + echo ">> staged Windows sqlite headers from $$d" + +build: assets $(WIN_PREREQ) + CGO_CFLAGS="$(CGO_EXTRA_CFLAGS)" CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) $(LINK_LIBS)" wails build -skipbindings -tags localembed # Cross-build the Intel-mac app from an arm64 Mac. GOARCH=amd64 makes the # assets target fetch the darwin-amd64 artifacts (the embedded/ and lib/ dirs @@ -26,8 +53,8 @@ build-darwin-amd64: GOARCH=amd64 $(MAKE) assets CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) -ltokenizers" wails build -skipbindings -tags localembed -platform darwin/amd64 -dev: assets - CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) -ltokenizers" wails dev -tags localembed +dev: assets $(WIN_PREREQ) + CGO_CFLAGS="$(CGO_EXTRA_CFLAGS)" CGO_LDFLAGS="-L$(PWD)/$(LIB_DIR) $(LINK_LIBS)" wails dev -tags localembed test: go test ./... @@ -66,7 +93,12 @@ assets: mkdir -p "$$(dirname "$$destpath")"; \ if [ -n "$$member" ]; then \ xd=$$(mktemp -d); \ - tar xzf "$$tmp" -C "$$xd" "$$member"; \ + case "$$url" in \ + *.zip) if command -v unzip >/dev/null 2>&1; then unzip -q "$$tmp" "$$member" -d "$$xd"; \ + elif [ -x /c/Windows/System32/tar.exe ]; then /c/Windows/System32/tar.exe xf "$$tmp" -C "$$xd" "$$member"; \ + else tar xf "$$tmp" -C "$$xd" "$$member"; fi ;; \ + *) tar xzf "$$tmp" -C "$$xd" "$$member" ;; \ + esac; \ cp "$$xd/$$member" "$$destpath"; \ rm -rf "$$xd"; \ got2=$$($(SHA256) "$$destpath" | awk '{print $$1}'); \ diff --git a/app.go b/app.go index d8df838..9f94353 100644 --- a/app.go +++ b/app.go @@ -10,6 +10,7 @@ import ( "log" "os" "path/filepath" + goruntime "runtime" "strconv" "github.com/borzou/vecstore/internal/chunker" @@ -384,10 +385,28 @@ func generateToken() (string, error) { return hex.EncodeToString(b), nil } -// defaultClaudeDesktopConfigPath returns the standard Claude Desktop config location. +// defaultClaudeDesktopConfigPath returns the standard Claude Desktop config +// location for the current OS. Claude Desktop reads a different path per +// platform; writing the macOS path on Windows produces a config it never sees +// (while still reporting success — the Phase 5 Windows smoke caught exactly that). func defaultClaudeDesktopConfigPath() string { - home, _ := os.UserHomeDir() - return filepath.Join(home, "Library", "Application Support", "Claude", "claude_desktop_config.json") + switch goruntime.GOOS { + case "windows": + if appData := os.Getenv("APPDATA"); appData != "" { + return filepath.Join(appData, "Claude", "claude_desktop_config.json") + } + home, _ := os.UserHomeDir() + return filepath.Join(home, "AppData", "Roaming", "Claude", "claude_desktop_config.json") + case "darwin": + home, _ := os.UserHomeDir() + return filepath.Join(home, "Library", "Application Support", "Claude", "claude_desktop_config.json") + default: // linux + if xdg := os.Getenv("XDG_CONFIG_HOME"); xdg != "" { + return filepath.Join(xdg, "Claude", "claude_desktop_config.json") + } + home, _ := os.UserHomeDir() + return filepath.Join(home, ".config", "Claude", "claude_desktop_config.json") + } } // GetClaudeDesktopConfigPath returns the current config path (custom or default). diff --git a/assets/manifest.json b/assets/manifest.json index 9cdfe00..50532cf 100644 --- a/assets/manifest.json +++ b/assets/manifest.json @@ -1,6 +1,6 @@ { "schema": 1, - "comment": "Pinned local-embedding artifacts (URLs + SHA-256). Weights/libs are NEVER committed; `make assets` downloads and checksum-verifies these into internal/embeddings/local/embedded/ (go:embed runtime assets) and internal/embeddings/local/lib/ (link-time static lib). For tarball artifacts, `sha256` is the checksum of the downloaded archive and `member_sha256` is the checksum of the extracted member. `dest` is relative to internal/embeddings/local/. Model + tokenizer entries are identical bytes on every platform (per-platform keyed for a uniform `make assets`). Remaining Phase 5 platform: windows-amd64. darwin-amd64's onnxruntime is our own source build (official mac-Intel prebuilts stopped at ORT 1.23) hosted in this repo's GitHub releases — see the ort-1.26.0-darwin-x64 release notes for the reproducible recipe.", + "comment": "Pinned local-embedding artifacts (URLs + SHA-256). Weights/libs are NEVER committed; `make assets` downloads and checksum-verifies these into internal/embeddings/local/embedded/ (go:embed runtime assets) and internal/embeddings/local/lib/ (link-time static lib). For tarball artifacts, `sha256` is the checksum of the downloaded archive and `member_sha256` is the checksum of the extracted member. `dest` is relative to internal/embeddings/local/. Model + tokenizer entries are identical bytes on every platform (per-platform keyed for a uniform `make assets`). darwin-amd64's onnxruntime is our own source build (official mac-Intel prebuilts stopped at ORT 1.23) hosted in this repo's GitHub releases — see the ort-1.26.0-darwin-x64 release notes for the reproducible recipe. windows-amd64's tokenizers_static has no upstream prebuilt: it is built from Rust source per Documentation/windows-build.md and hosted in this repo's releases (tokenizers-1.27.0-windows-x64).", "platforms": { "darwin-arm64": { "model": { @@ -28,6 +28,32 @@ "dest": "lib/libtokenizers.a" } }, + "windows-amd64": { + "model": { + "url": "https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx", + "sha256": "f80102d3f2a1229f387d3c81909990d8945513e347b0eab049f7de3c6f98c193", + "dest": "embedded/model_quantized.onnx" + }, + "tokenizer": { + "url": "https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/tokenizer.json", + "sha256": "0b44a9d7b51c3c62626640cda0e2c2f70fdacdc25bbbd68038369d14ebdf4c39", + "dest": "embedded/tokenizer.json" + }, + "onnxruntime": { + "url": "https://github.com/microsoft/onnxruntime/releases/download/v1.26.0/onnxruntime-win-x64-1.26.0.zip", + "sha256": "6ebe99b5564bf4d029b6e93eac9ff423682b6212eade769e9ca3f685eaf500b4", + "member": "onnxruntime-win-x64-1.26.0/lib/onnxruntime.dll", + "member_sha256": "b2ba7ca16e0e4fe71ad5148744ab885a2f5809e52a0c3de4d9ba3853a03977f9", + "dest": "embedded/onnxruntime.dll" + }, + "tokenizers_static": { + "url": "https://github.com/unitedeffectslabs/agent-memory/releases/download/tokenizers-1.27.0-windows-x64/libtokenizers.windows-x86_64.tar.gz", + "sha256": "882f520174a6cb14dcf4dca559375d65915a69a075bd85c22e40463bc0b466a8", + "member": "libtokenizers.a", + "member_sha256": "5e0815434a9d9eea40638ae9303f239c85059868673dc7e2a5a1eee72a9ad692", + "dest": "lib/libtokenizers.a" + } + }, "darwin-amd64": { "model": { "url": "https://huggingface.co/Xenova/multilingual-e5-small/resolve/main/onnx/model_quantized.onnx", diff --git a/internal/embeddings/local/assets_embed_windows.go b/internal/embeddings/local/assets_embed_windows.go new file mode 100644 index 0000000..7d47555 --- /dev/null +++ b/internal/embeddings/local/assets_embed_windows.go @@ -0,0 +1,19 @@ +//go:build localembed && windows + +package local + +import "embed" + +// Per-platform ONNX Runtime shared library (Windows x64). One binary can embed +// only one platform's ORT lib (epic Open Concern #8); the official Microsoft +// release ships the DLL — see assets/manifest.json (windows-amd64) and +// Documentation/windows-build.md for the full Windows build procedure +// (the static tokenizer lib has no upstream prebuilt and is built from Rust +// source there). +// +//go:embed embedded/onnxruntime.dll +var embeddedORTLib embed.FS + +// ortLibFile is the base name of this platform's embedded ONNX Runtime shared +// library, both inside embedded/ and after extraction to the runtime dir. +const ortLibFile = "onnxruntime.dll"