Add section: "Core concepts"

src/content/_meta.ts

@@ -21,6 +21,7 @@ const meta: MetaRecord = {
   'run-on-lightpanda-cloud': 'Run on Lightpanda Cloud',
   usage: 'Usage',
   guides: 'Guides',
+  'core-concepts': 'Core concepts',
 }
 
 export default meta

src/content/core-concepts/_meta.ts

@@ -0,0 +1,10 @@
+import type { MetaRecord } from 'nextra'
+
+const meta: MetaRecord = {
+  'what-is-lightpanda': 'What is Lightpanda?',
+  'architecture-overview': 'Architecture overview',
+  'local-vs-cloud': 'When to use local vs cloud',
+  benchmarks: 'Benchmarks',
+}
+
+export default meta

src/content/core-concepts/architecture-overview.mdx

@@ -0,0 +1,111 @@
+---
+title: Architecture Overview
+description: How Lightpanda's architecture works, from its Zig core and V8 engine to native Web APIs, and why it runs headless automation with a small memory footprint.
+---
+
+import { Callout } from 'nextra/components'
+
+# Architecture overview
+
+Lightpanda is a headless browser built in Zig to be driven by machines, not viewed by people. Its defining choice is that it has no rendering engine and never draws a page to a screen.
+
+It keeps the parts automation needs, a JavaScript engine, the DOM and its Web APIs, and a network layer, and drops the graphics pipeline that makes a normal browser heavy. A single native binary exposes that engine through four entry points, from a CDP server to a built-in AI agent.
+
+## Why there's no rendering engine
+
+A normal browser runs a rendering pipeline to turn a page into pixels for a human: style resolution, layout, paint, and compositing. Machines rarely need those pixels. A client sends commands over a protocol (usually CDP, the Chrome DevTools Protocol) and reads the page's structure, the DOM, back. The picture on screen is irrelevant.
+
+Lightpanda drops the rendering pipeline. What remains is the set of components automation uses, the same ones detailed further down this page:
+
+- **Control surface.** The four entry points you drive it through: `serve` (CDP), `fetch`, `agent`, and `mcp`.
+- **JavaScript engine.** Runs the page's scripts. Lightpanda uses V8, the same engine as Chrome.
+- **Web APIs and the DOM.** The in-memory document that scripts read and mutate.
+- **HTML parser.** Turns fetched HTML into that DOM.
+- **Network layer.** Fetches the main document and its subresources over HTTP.
+
+<Callout type="info">
+  Because Lightpanda does not render, it does not compute visual layout. APIs
+  that depend on pixel geometry (for example element bounding boxes) return
+  best-effort values, and `Page.captureScreenshot` returns a placeholder image,
+  not a real render. Reach for a full browser when you need actual pixels.
+</Callout>
+
+## How the parts fit together
+
+Lightpanda ships as a single native binary. At runtime it has one process. That process holds state shared across the whole run (the `App`) and creates browser instances on demand. Each browser instance owns one V8 isolate and the page hierarchy that runs inside it.
+
+The stack, top to bottom:
+
+```
+Control surface   serve (CDP) · fetch · agent · mcp                       ← how you drive it
+                        │
+App               Network · SQLite storage · Arena pool · V8 Platform · Snapshot · Telemetry  ← process-wide
+                        │
+Browser           one V8 isolate · HTTP client
+                        │
+Session           cookie jar · Web Storage
+                        │
+Page              a document, a tab: DOM · Web APIs
+                        │
+Frame             a document frame or iframe: DOM · Web APIs
+```
+
+Every box in the diagram maps to a directory in the [browser repository](https://github.com/lightpanda-io/browser):
+
+| Path | Where it sits in the diagram |
+| --- | --- |
+| `src/cdp/`, `src/agent/`, `src/mcp/` | Control surface |
+| `src/main.zig`, `src/App.zig` | `App` and its process-wide services |
+| `src/network/` | The `App`'s network layer (libcurl, robots.txt, WebSocket) |
+| `src/browser/` | Browser, Session, Page, Frame |
+| `src/browser/js/` | The Browser's JavaScript engine, a V8 isolate (thin wrappers over the V8 C++ API) |
+| `src/browser/webapi/` | Inside each Page and Frame: the Web APIs and the DOM |
+| `src/browser/parser/` | Inside each Page and Frame: the HTML parser (html5ever, Rust) |
+
+### Entry points
+
+The same engine runs behind four commands. Each is a different way to drive it.
+
+- [`serve`](/run-locally/commands/serve) starts a WebSocket CDP server. Clients like [Puppeteer](/usage/cdp/puppeteer), [Playwright](/usage/cdp/playwright), and [chromedp](/usage/cdp/chromedp) connect over the Chrome DevTools Protocol on port `9222` by default. This is the mode most automation uses.
+- [`fetch`](/run-locally/commands/fetch) loads one or more URLs and dumps the result to stdout as HTML or Markdown. It is a one-shot command with no server.
+- [`agent`](/usage/agent) starts an interactive AI agent that browses the web from natural language and can record reproducible scripts.
+- [`mcp`](/usage/mcp) starts a Model Context Protocol server over stdio, so an LLM host can use the browser as a tool.
+
+The agent-oriented modes stack the same engine under different amounts of the browser. With `serve`, your client drives the engine over CDP. With `mcp`, the binary also carries the tool server. With `agent`, it carries the whole loop, so no protocol sits between the model and the engine.
+
+![The browser agent stack: model, harness, browser driver, and engine layers across lightpanda serve, mcp, and agent, showing what the single binary covers in each mode](https://cdn.lightpanda.io/website/assets/images/blog/posts/the-browser-agent-stack-explained/browser-agent-stack-diagram-2.svg)
+
+### The browser engine
+
+Below the entry points, one object hierarchy runs pages. It separates state that lives for the whole process from state scoped to a browsing session.
+
+`App` is created once at startup and owns the process-wide services shown above, and everything below borrows from them. Two are worth naming: the arena pool (a shared set of memory arenas, see [Memory model](#memory-model)) and telemetry (anonymous usage metrics, off in debug builds and [disabled](/run-locally/installation/nightly-builds#telemetry) with `LIGHTPANDA_DISABLE_TELEMETRY`).
+
+- **Browser** wraps a single V8 isolate. An isolate has thread affinity, so a browser is created and used on one thread. It also holds a per-browser HTTP client and borrows arenas from the `App` pool. A browser contains one session.
+- **Session** is a browsing context group. It owns the cookie jar (the session's cookie store) and Web Storage (`localStorage` and `sessionStorage`), the state that outlives navigation. This is per-session and separate from the `App`'s SQLite storage backend.
+- **Page** is one top-level document, the equivalent of a tab.
+- **Frame** is one document frame, either the main frame or an iframe.
+
+Cookies and permissions belong to the session, so they survive when a page navigates. The isolate belongs to the browser, so it survives across pages but never crosses threads.
+
+**JavaScript engine.** Lightpanda runs page scripts on V8 and does not reimplement JavaScript. The Zig code in `src/browser/js/` is a thin wrapper around V8's C++ API: isolates, contexts, values, promises, and modules. The Web API objects that scripts touch are implemented in Zig, so `document`, `window`, and `fetch` are backed by native code, not more JavaScript. Startup is fast because of a V8 startup snapshot: a serialized, pre-initialized V8 heap embedded in the binary at build time. On each start V8 loads it instead of rebuilding its built-in objects from scratch, which removes most of the isolate warm-up cost.
+
+**Web APIs and the DOM.** These are implemented natively in Zig, grouped by area: the DOM (`Document`, `Element`, `Node`), events, `fetch` and networking, `Crypto`, observers (mutation, intersection, resize), and storage. Coverage is partial and grows over time. Lightpanda implements the APIs headless automation exercises, not the entire web platform. The source of truth for what exists is `src/browser/webapi/` in the browser repository. When a script calls one of these APIs it runs compiled Zig, which is why DOM-heavy pages stay cheap in both time and memory.
+
+**HTML parser.** Parsing is delegated to [html5ever](https://github.com/servo/html5ever), the spec-compliant Rust parser from the Servo project. Lightpanda calls it over a C ABI and builds the DOM tree from the parser's callbacks. This is why building the browser from source needs a Rust toolchain in addition to Zig.
+
+**Network layer.** It sits on top of [libcurl](https://curl.se/libcurl/), fetches the main document and subresources, manages the cookie jar, and can honor `robots.txt` when you pass `--obey-robots`.
+
+<Callout type="warning">
+  Lightpanda is fast, so it is easy to send a high volume of requests. Respect
+  `robots.txt` and avoid hammering small sites. Pass `--obey-robots` to have
+  Lightpanda follow `robots.txt` for you.
+</Callout>
+
+## Memory model
+
+A small, predictable memory footprint is one of Lightpanda's defining traits, so how the engine handles memory is part of its architecture. Dropping the rendering pipeline removes the largest cost. The allocator design keeps what remains bounded under load.
+
+Debug builds use an allocator that detects leaks on exit, and the custom test runner fails any test that allocates without freeing. Release builds use the C allocator directly.
+
+Short-lived allocations tied to a request or a navigation go through an arena. An arena is a single region you allocate many small objects into and then free all at once, instead of tracking each object. Lightpanda keeps a process-wide pool of these arenas on the `App`: a page borrows one when it loads and returns it when the page goes away. That keeps per-page overhead flat and predictable, and makes teardown a single free.

src/content/core-concepts/benchmarks.mdx

@@ -0,0 +1,80 @@
+---
+title: Benchmarks
+description: Benchmark results comparing Lightpanda to headless Chrome for crawling, page automation, and AI agent workloads, with full methodology.
+---
+
+import { Callout } from 'nextra/components'
+
+# Benchmarks
+
+Lightpanda publishes three benchmarks against headless Chrome:
+
+- **Crawling at scale**: following every link on a 933-page demo site.
+- **Single-page automation**: repeated load-and-extract cycles over CDP.
+- **AI agent task completion**: task accuracy on AssistantBench and GAIA.
+
+Across all three, Lightpanda uses less memory and CPU than Chrome and finishes faster. Each section below states the source of its numbers. For exact commands and raw output, see [Reproduce these results](#reproduce-these-results).
+
+## How results are measured
+
+The crawling and single-page benchmarks track peak memory (via `smem`, using PSS so shared pages aren't double-counted), CPU utilization (via `ps aux`), and wall-clock duration, sampled every 100ms during the run. Both run on an AWS m5.xlarge instance with a fresh Ubuntu install, comparing Lightpanda against Google Chrome 143.0.7499.169.
+
+The AI agent benchmark grades differently: each task's final answer is checked against a known correct answer, not a screenshot, since Lightpanda has no rendered page to screenshot in the first place.
+
+## Crawling at scale
+
+Source: [Crawler Benchmark, BENCHMARKS.md](https://github.com/lightpanda-io/demo/blob/main/BENCHMARKS.md#crawler-benchmark) in the demo repository.
+
+This benchmark crawls [demo-browser.lightpanda.io/amiibo](https://demo-browser.lightpanda.io/amiibo/), a demo product catalog, following every link from the index page to its subpages: 933 URLs in total. A Go program using [chromedp](https://github.com/chromedp/chromedp) drives the crawl over CDP, so both browsers are exercised through the same client code.
+
+Chrome runs one browser with multiple tabs, since that's how most people scale automation with a normal browser. Lightpanda can't open multiple tabs in one process, so it runs multiple processes instead, each on its own port. This mirrors how each browser is actually deployed in production, not just a same-process comparison.
+
+<Callout type="info">
+  Chrome shares a lot of infrastructure (renderer processes, V8 heaps) across
+  tabs in the same browser. Lightpanda processes are fully independent. The
+  comparison below is tab count against process count, because that's the
+  unit each browser scales with.
+</Callout>
+
+| Parallel tabs/processes | Lightpanda duration | Lightpanda memory peak | Chrome duration | Chrome memory peak |
+| --- | --- | --- | --- | --- |
+| 1 | 0:51.68 | 27.2M | 1:22.83 | 1.3G |
+| 2 | 0:29.79 | 31.7M | 0:53.11 | 1.3G |
+| 5 | 0:11.70 | 43.9M | 0:45.66 | 1.6G |
+| 10 | 0:06.76 | 63.7M | 0:45.62 | 1.7G |
+| 25 | 0:04.81 | 123.0M | 0:46.70 | 2.0G |
+| 100 | 0:05.23 | 410.2M | 1:09.37 | 4.2G |
+
+At 25 parallel tasks, the point where both browsers are near their best throughput, Lightpanda finishes in 4.81 seconds using 123MB, against Chrome's 46.70 seconds and 2.0GB: about 9x faster and 16x lighter. Chrome's duration plateaus past 5 tabs because its tabs share a process and start contending for resources. Lightpanda keeps improving up to 25 processes, since each process is fully isolated and the machine still has headroom.
+
+## Single-page automation
+
+Source: [Campfire e-commerce Benchmark, BENCHMARKS.md](https://github.com/lightpanda-io/demo/blob/main/BENCHMARKS.md#campfire-e-commerce-benchmark) in the demo repository.
+
+Crawling measures fetching and following links. This benchmark measures what happens once a script needs to load a page, wait on network requests, and read data back, the pattern behind most scraping and testing code. It uses a [homemade e-commerce demo page](https://demo-browser.lightpanda.io/campfire-commerce/) that loads product details and reviews over two XHR requests, served from a local web server to keep network latency out of the result. A [Puppeteer](https://pptr.dev/) script connects over CDP and repeats the same load-and-extract task 100 times.
+
+| Browser | Avg run duration | Total duration (100 runs) | Memory peak | CPU peak |
+| --- | --- | --- | --- | --- |
+| Lightpanda | 16ms | 1,698ms | 21.2M | 4.6% |
+| Chrome 143.0.7499.109 | 185ms | 18,551ms | 402.1M | 158.6% |
+
+Lightpanda completes the same 100 runs about 11x faster, using roughly 19x less peak memory. See [Architecture overview](/core-concepts/architecture-overview) for why.
+
+## AI agent task completion
+
+Source: [agent-benchmarks](https://github.com/lightpanda-io/agent-benchmarks), the [current results](https://github.com/lightpanda-io/agent-benchmarks#current-results) and [cross-framework comparison](https://github.com/lightpanda-io/agent-benchmarks#cross-framework-comparison-lightpanda-vs-agent-browser-vs-browser-use-claudemcp) sections of its README.
+
+Crawl and page-load benchmarks measure raw speed, but they don't tell you whether an AI agent actually completes its task. Running `lightpanda agent`, the built-in agent loop with no MCP or CDP round-trip, against AssistantBench's 33-task validation split and GAIA Level 1's 53-task validation split, with Claude Sonnet 4.6 and a 1,800-second per-task timeout, scores 69.7% strict accuracy on AssistantBench and 83.0% on GAIA, with zero timeouts on either. Cost per task, computed from token usage: $1.94 on AssistantBench, $0.34 on GAIA.
+
+To isolate whether that result comes from the engine or the tool surface, Lightpanda also ran the same Claude Sonnet 4.6 session over MCP against three other setups: [agent-browser](https://github.com/vercel-labs/agent-browser) driving Chromium, agent-browser driving Lightpanda as its engine instead of Chrome, and [browser-use](https://github.com/browser-use/browser-use) driving Chromium. Only the browser and its tool surface change between rows.
+
+| Suite | Lightpanda MCP | agent-browser + Chromium | agent-browser + Lightpanda | browser-use (Chromium) |
+| --- | --- | --- | --- | --- |
+| AssistantBench, strict | 66.7% | 57.6% | 57.6% | 39.4% |
+| GAIA Level 1, strict | 86.8% | 84.9% | 81.1% | 47.2% |
+
+AssistantBench accuracy is identical whether agent-browser drives Chrome or Lightpanda (57.6% either way), so the gap to Lightpanda's own MCP tool surface (66.7%) comes from the tools, not the engine underneath them. On GAIA, swapping Lightpanda in under agent-browser costs a few points of accuracy (81.1% vs 84.9%), on pages where Lightpanda's text-only output misses content a rendered page would show.
+
+## Reproduce these results
+
+The crawler and CDP benchmark scripts, exact commands, and raw `/usr/bin/time` output live in [BENCHMARKS.md in the demo repository](https://github.com/lightpanda-io/demo/blob/main/BENCHMARKS.md), alongside the [chromedp crawler](https://github.com/lightpanda-io/demo/tree/main/chromedp/crawler) source. The AI agent benchmark's suites, runners, and raw results live in the [agent-benchmarks repository](https://github.com/lightpanda-io/agent-benchmarks).