toroid-kernel

toroid-kernel is a Go package for running tool-using LLM agents with persistent traces, resumable history, gateway-truth cost accounting, and a built-in tool registry.

~6,824 lines of Go across the kernel (.), the LLM wire layer (llm/), and the tools (tools/). No third-party model SDK.

Runner binaries (built with -trimpath -ldflags="-s -w"; linux is amd64):

runner	macOS (arm64)	Linux (amd64)	Linux + upx
examples/cli	11.8 MB	12.1 MB	5.2 MB

Features

• Self-contained LLM layer (in-repo llm/ package) speaking OpenAI-compatible chat completions to a LiteLLM gateway — SSE streaming, tool calls, multimodal content blocks, retries on 429/5xx
• Kernel-owned tool loop: one llm-step per turn via the Step interface
• Gateway-truth cost: every non-streaming llm-step carries the gateway's authoritative x-litellm-response-cost; there is no local pricing table to drift out of date. Usage.PricingOK is true only when the gateway reported a cost
• Hard USD spend limits: WithMaxTurnSpendUSD caps one Run/Stream call and Config.MaxTranscriptSpendUSD caps cumulative transcript spend; reaching either stops further LLM steps, including structured-output and wake paths
• Structured output (WithSchema) as a forced tool call — works across upstreams, including Bedrock-backed Anthropic
• Multimodal input: images and PDFs inline in prompts (![](path), 5 MiB cap, capability-gated per model) and as tool-result media (the read tool returns images a vision model can see)
• Single-file SQLite persistence (traces, costs, events) with OpenTelemetry-compatible trace/span IDs and a Langfuse OTLP exporter
• Always-on transcript: every session appends its observable events (tool calls included) as OTEL-shaped NDJSON to ~/.toroid/sessions/<session-id>/transcript.jsonl — independent of Save, so there is a durable trace even without SQLite
• Conversation compaction, loop guards (MaxIter, default 100, + repeat-call spin guard), and history reconstruction for resume
• Lean default toolset — read, write, edit, multiedit, and bash (search/list/find go through bash). Skills appear only when discovered; subagents are opt-in with IncludeSubagentTools; MCP tools appear only when configured. Truncated tool outputs spill to ~/.toroid/sessions/<session>/tool-output/ (the result names the file) so nothing is lost; when rtk is on PATH, simple read-only bash commands are auto-routed through it for compressed output. The bash tool runs non-interactively with a per-command timeout (default 120s, overridable) and a process-group kill, so a command that opens an editor or otherwise blocks on a terminal can never hang the kernel
• Cache-stable prompt compilation — startup capabilities (skills, MCP, and optional subagents) are compiled once into the system prompt after discovery; tool definitions are deterministically ordered and remain unchanged across loop turns and later runs. Capability guidance appears once in that stable prefix instead of being re-injected as cache-busting history
• Background agents — async subagents that wake an idle kernel on completion

Verified models

Live-verified (tool loop, structured output, SSE streaming, billed cost, OTEL store round-trip, plus image and PDF input where the model supports vision):

model	tools	schema	stream	cost	image input	document (PDF) input
llmgateway/claude-haiku-4-5	✅	✅	✅	✅	✅	✅
llmgateway/gpt-5.4-mini	✅	✅	✅	✅	✅	✅
llmgateway/kimi-k2p6	✅	✅	✅	✅	n/a (text-only)	n/a
llmgateway/glm-5p1	✅	✅	✅	✅	n/a (text-only)	n/a
llmgateway/minimax-m2p7	✅	✅	✅	✅	n/a (text-only)	n/a
openai/gpt-5.4-mini	✅	✅	✅	✅ (cached rates)	✅	✅
anthropic/claude-haiku-4-5	✅	✅	✅	✅ (cached rates)	✅	✅

Text-only models never receive media: an inline ![](path) ref is left as text and a warning is surfaced, so nothing is silently dropped or paid for.

Architecture

Host → Kernel (turns, tools, compaction, events, store) → Step (one LLM call = one llm-step) → llm.Client (OpenAI-compatible wire to the gateway).

For the deep dive — construction, the turn loop, tools, the SQLite store, OTEL telemetry, context management, sub/background agents — see the architecture explainer.

Install

go get github.com/yashbonde/toroid-kernel

Requires Go 1.26.4 or newer.

Providers

The model id prefix picks the provider; the key comes from that provider's env var:

prefix	wire	key env	cost source
llmgateway/<name> (default)	OpenAI-compatible chat completions to a LiteLLM gateway (LLM_GATEWAY_BASE_URL, including /v1)	LLM_GATEWAY_KEY	gateway x-litellm-response-cost header (authoritative)
openai/<name>	OpenAI API direct (same wire)	OPENAI_API_KEY	cached in-code rates (Model.Price)
anthropic/<name>	native Anthropic messages API (/v1/messages)	ANTHROPIC_API_KEY	cached in-code rates (Model.Price)

Neither OpenAI nor Anthropic expose pricing via API, so per-token rates for their families are cached in the kernel code and used whenever the gateway cost header is absent. Unknown families stay honestly unpriced (Usage.PricingOK == false, warning logged) — never assumed free.

The Anthropic route implements Anthropic's explicit prompt caching (they have no automatic caching): cache_control breakpoints on the system prompt and the last user message every loop step. Brutally verified live on anthropic/claude-haiku-4-5: turn 2 wrote 6,643 tokens to cache, turn 3 read them back, and a follow-up Run on the same kernel read 11,837 cached tokens with only 3 fresh input tokens. Tool loop, structured output (native forced tool), SSE streaming, mid-stream abort (partial kept), thinking (budget_tokens), image, and PDF input all pass on the native wire.

Quick Start

package main

import (
	"context"
	"fmt"
	"os"

	toroid "github.com/yashbonde/toroid-kernel"
)

func main() {
	ctx := context.Background()

	kernel, err := toroid.NewKernel(ctx, toroid.Config{
		Model:   "llmgateway/claude-haiku-4-5",
		APIKey:  os.Getenv("LLM_GATEWAY_KEY"),
		WorkDir: ".",
		Save:    true,
	})
	if err != nil {
		panic(err)
	}
	defer kernel.Close()

	out, _, err := kernel.Run(ctx, "Summarize the repository and point out release blockers.")
	if err != nil {
		panic(err)
	}

	fmt.Println(out)
	fmt.Printf("cost: $%.6f\n", kernel.RunningCostUSD())
}

Runners

The cli runner in examples/ does double duty:

• cli — interactive terminal chat with live tool-call display, cost footer, /cost, /model, /reset.
• cli --run '<prompt>' — one-shot: drive the kernel from the command line, emitting every kernel event as NDJSON on stdout (the bridge for hosts in other languages; --plain prints just the final answer). Shares the same flags, so --model, --thinking, --save all apply.

export LLM_GATEWAY_BASE_URL=https://my-gateway.example.com/v1
export LLM_GATEWAY_KEY=sk-...

go run ./examples/cli
go run ./examples/cli --run 'what files are in this directory?' --plain

Examples

The examples/ directory has small, focused, runnable programs — one per usage pattern. Start with examples/README.md.

If you are an AI agent, read examples/README.md and the per-directory main.go files first — they are the canonical, up-to-date reference for how to use this library.

go run ./examples/running                 # the one example: blocking + streaming + events + guardrails
                                         #   + delegation + multimodal + structured output + OTEL
go run ./examples/running --guardrails   # loop guards only, no network needed (scripted FauxStep)
go run ./examples/langfuse                # push a persisted trace to Langfuse over OTLP
go test ./examples/e2e-test               # offline skill + MCP + cache-prefix integration test

Cost accounting

The kernel's turn loop is non-streaming per llm-step, so every step gets the gateway's x-litellm-response-cost header — the number LiteLLM actually bills. Per-step costs land in EventTurnCost, roll up into RunningCostUSD() (subagent spend included), and persist to SQLite with the trace. There is no client-side price table; if the gateway does not report a cost (Usage.PricingOK == false), the cost is unknown, never assumed free, and a warning is logged.

Hosts can enforce hard stop boundaries with WithMaxTurnSpendUSD for one Run/Stream call and Config.MaxTranscriptSpendUSD for the kernel's cumulative transcript. Because providers report cost after a response, the step that crosses a limit is billed; the kernel then runs no subsequent LLM step for that call. A non-positive limit disables it.

Prompt caching: Claude-family models get cache_control breakpoints (system + last user/tool message) on every loop step, so each turn re-reads the prior conversation at cache-read price. History is never mutated mid-chat (that would bust the cache prefix); trimming happens only at compaction. The full token/cost audit and its resolutions live in assets/adversarial-cost-review.md.

Persistence & telemetry

When Save: true, traces, spans, costs, and events are written to a single SQLite database at ~/.toroid/sql.db. Span IDs are time-ordered Snowflake IDs and the trace/span/parent graph maps directly onto OpenTelemetry — toroid.LangfuseOTLP(ctx, traceID, …) projects a stored trace into OTLP spans and pushes it to Langfuse. Call kernel.Close() when done to checkpoint the store.

Background agents

Set IncludeSubagentTools: true to expose subagent and subagent_async. subagent_async runs a subtask in the background and returns immediately. When it finishes, its result is queued and the kernel is woken to process it — even if Run/Stream had already returned. The MasterIdle and TaskCompleted events let a host observe these transitions.

Skills

When Config.LoadSkills is unset or true (the default), the kernel scans ~/.toroid/skills/*.md at startup and reads only each file's frontmatter (name + description) into the system prompt — the full body is not loaded. When at least one skill exists, the kernel registers a skill tool that loads its full contents on demand. Set LoadSkills to false to disable discovery.

Release Notes

• The module path is github.com/yashbonde/toroid-kernel.
• The stable system prompt and built-in tool contracts are compiled in prompt_compiler.go; prompts/ holds only task-specific prompts such as conversation compaction.
• Runtime state is stored under ~/.toroid/ (single SQLite DB at sql.db).