Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/astro.config.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ export default defineConfig({
{ label: '<MeasureTool />', link: '/plugins/measure-tool/' },
{ label: '<SelectionTool />', link: '/plugins/selection/' },
{ label: '<Skybox />', link: '/plugins/skybox/' },
{ label: '<LLMSceneBuilder />', link: '/plugins/llm-scene-builder/' },
],
},
{
Expand Down
126 changes: 126 additions & 0 deletions docs/src/content/docs/plugins/llm-scene-builder.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
---
title: <LLMSceneBuilder />
description: Natural-language frame editing — describe a spatial change in plain text and an LLM proposes the frame deltas.
---

import { Aside } from '@astrojs/starlight/components'

`<LLMSceneBuilder />` adds a **Frame Builder** panel to the visualizer dashboard. You type a natural-language instruction ("Move the arm 200 mm forward along X"), the plugin calls your `onInfer` callback with the current frame state, and presents a diff of every proposed field change before anything is applied. The user confirms or cancels — no frame is mutated until confirmation.

The plugin is model-agnostic: you wire in whatever LLM backend you prefer via the `onInfer` prop.

## Usage

```svelte
<script lang="ts">
import { Visualizer } from '@viamrobotics/motion-tools'
import { LLMSceneBuilder } from '@viamrobotics/motion-tools/plugins'
import type { ComponentFrameInfo, FrameDelta } from '@viamrobotics/motion-tools/plugins'

async function handleInfer(
prompt: string,
components: ComponentFrameInfo[]
): Promise<{ updates: FrameDelta[]; explanation: string }> {
const response = await fetch('/api/infer-frames', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ prompt, components }),
})
return response.json()
}
</script>

<div class="h-screen w-screen">
<Visualizer>
<LLMSceneBuilder onInfer={handleInfer} />
</Visualizer>
</div>
```

A robot-outline button appears in the dashboard. Clicking it opens the **Frame Builder** floating panel. Enter a prompt and press **Submit** (or `Enter`) — the panel shows a diff table while the LLM responds, then lets the user confirm or cancel.

## Props

| Prop | Type | Default | Description |
| --------- | --------------- | ------- | ------------------------------------------------------------------------------------------------------------- |
| `onInfer` | `InferCallback` | — | **Required.** Called with the prompt and current frame state; must return proposed deltas and an explanation. |

## InferCallback

```ts
type InferCallback = (
prompt: string,
components: ComponentFrameInfo[]
) => Promise<{ updates: FrameDelta[]; explanation: string }>
```

The plugin passes every component that has a frame defined:

```ts
interface ComponentFrameInfo {
name: string
frame: {
parent: string | undefined
translation: { x?: number; y?: number; z?: number } | undefined
orientation: { roll: number; pitch: number; yaw: number } // degrees
}
}
```

Your callback should return:

- **`updates`** — an array of `FrameDelta` objects describing changes (see below). Omit any field that should remain unchanged.
- **`explanation`** — a human-readable summary shown above the diff table.

### FrameDelta

```ts
interface FrameDelta {
componentName: string
translation?: { x?: number; y?: number; z?: number } // mm, absolute
orientation?: { roll?: number; pitch?: number; yaw?: number } // degrees, delta applied to current
parent?: string
explanation?: string // per-component note shown in the diff
}
```

<Aside type="caution">
`translation` values are **absolute** — they replace the current translation field directly.
`orientation` values are **deltas** added to the current Euler angles.
</Aside>

The plugin validates every delta before showing the diff: unknown component names, self-referential parent assignments, and non-finite numbers are surfaced as errors without blocking the rest of the update batch.

## Example: using the Anthropic SDK

Set `ANTHROPIC_API_KEY` in your environment (see `.env.example`). A minimal server-side route using the Anthropic SDK:

```ts
// src/routes/api/infer-frames/+server.ts (SvelteKit)
import Anthropic from '@anthropic-ai/sdk'

export async function POST({ request }) {
const { prompt, components } = await request.json()

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY })

const message = await client.messages.create({
model: 'claude-sonnet-4-6',
max_tokens: 1024,
messages: [
{
role: 'user',
content: [
{
type: 'text',
text: `Current robot frame configuration (JSON):\n${JSON.stringify(components, null, 2)}\n\nUser request: ${prompt}\n\nRespond with JSON matching the schema: { updates: FrameDelta[], explanation: string }`,
},
],
},
],
})

const text = message.content.find((b) => b.type === 'text')?.text ?? '{}'
return new Response(text, { headers: { 'Content-Type': 'application/json' } })
}
```
12 changes: 5 additions & 7 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@
"@changesets/cli": "2.29.6",
"@connectrpc/connect": "1.7.0",
"@connectrpc/connect-web": "1.7.0",
"@langchain/anthropic": "^1.4.0",
"@dimforge/rapier3d-compat": "0.18.2",
"@eslint/compat": "2.0.2",
"@eslint/js": "10.0.1",
Expand Down Expand Up @@ -115,11 +116,13 @@
"type-fest": "^5.0.1",
"typescript": "5.9.2",
"typescript-eslint": "8.56.1",
"uuid-tool": "^2.0.3",
"vite": "7.3.2",
"vite-plugin-devtools-json": "1.0.0",
"vite-plugin-glsl": "^1.5.5",
"vite-plugin-mkcert": "1.17.9",
"vitest": "3.2.6"
"vitest": "3.2.6",
"zod": "^4.4.3"
},
"peerDependencies": {
"@ag-grid-community/client-side-row-model": ">=32.3.0",
Expand Down Expand Up @@ -209,16 +212,11 @@
],
"dependencies": {
"@bufbuild/protobuf": "1.10.1",
"@connectrpc/connect": "1.7.0",
"@connectrpc/connect-web": "1.7.0",
"@langchain/anthropic": "^1.4.0",
"@neodrag/svelte": "^2.3.3",
"d3-force": "^3.0.0",
"filtrex": "^3.1.0",
"koota": "0.6.5",
"lodash-es": "4.18.1",
"three-mesh-bvh": "^0.9.8",
"uuid-tool": "^2.0.3",
"zod": "^4.4.3"
"three-mesh-bvh": "^0.9.8"
}
}
30 changes: 15 additions & 15 deletions pnpm-lock.yaml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading