App 16360 add llm 3 d scene generation by sucrammal · Pull Request #740 · viamrobotics/visualization

sucrammal · 2026-06-01T22:03:26Z

This is a prototype for an LLM-powered frame builder. It allows you to type in a natural language query to edit frame orientation, position, and parent of one or more components in the visualizer.

Here's a short, imperfect demo that shows me moving an arm to a new position, as well as resetting it back to its origin.

basicdemo.mov

Here's another demo of using the frame builder alongside the other editor. Making edits in both ways creates a config diff together.

dragSessionAndFrameBuild.mov

However, I'm having a little trouble with local position vs. world position. I had a peek at the worldPose/worldMatrix implementation, and heard from @mattmacf98 that there was an ongoing bug there. In this demo, an update from the frame builder, although confirmed, does not show a visual preview in its new position. Looking at the component edit tab shows a stale World Position that is only later updated. Not sure if this is because it is waiting for it to update from the network, or something else.

bug.mov

TODOs

Investigate world position vs. local position discrepency bug.
Currently, the chat is single-shot (i.e. only accepts one prompt at a time, no conversation history). If it's a desirable feature, we can extend the chat functionality.
There is no place to place your own LLM provider key. When the visualizer is wired into the app, we can reuse the app/inference routes that are already there. But as suggested by @mattmacf98 , might be easier to have a manual entry in the settings widget for development.
General stress testing and robust support simultaneous editing (use frame builder, drag edit session, manual entry).

Introduce a new SceneBuilder overlay feature: add SceneBuilder.svelte component, frameDeltaAdapter implementation and unit tests, and a server route (server/routes/scene-builder.ts). Wire the route into server.ts and update +layout.svelte to include the overlay. package.json updated for required dependencies.

Refactor frame-delta workflow to separate validation/preparation from application. Introduces validateProposedFrameDeltas which computes PreparedUpdate records (previous/new pose & parent) and returns errors without mutating config, and add applyPreparedUpdates to apply those prepared updates via updateFrame. Updated tests to cover both validation and application flows, and updated SceneBuilder.svelte to use pendingUpdates, derive diffRows for the preview table, call validateProposedFrameDeltas on submit, and applyPreparedUpdates on confirm (replaced revert->cancel and applied->pendingUpdates naming). Minor debug helper $inspect(localConfigProps, 'LOCAL') added to App.svelte.

changeset-bot · 2026-06-01T22:03:30Z

🦋 Changeset detected

Latest commit: 1366025

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@viamrobotics/motion-tools	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

mattmacf98 · 2026-06-02T13:05:26Z

this is dope!

mattmacf98

This looks great! I left a couple of nits to think about when doing the final PR (I think this is non-invasive enough to just do one PR)

biggest thing is just to move this to a plugin see DrawService for an example on how to do that visualization/src/lib/plugins/DrawService/DrawService.svelte

mattmacf98 · 2026-06-02T13:22:57Z

 		() => partID,
 		() => localConfigProps
 	)
+	provideSceneBuilder()


we will probably want to put this provide only in the LLMSceneBuilder plugin, look at how the draw service sets it up

visualization/src/lib/plugins/DrawService/DrawService.svelte

Line 2 in 78579a2

import { provideDrawAPI } from './useDrawAPI.svelte'

as inspo

…pdates and Deps from frameDeltaAdapter.ts.

…ally, not viam's OV. Allow partial Euler orientation updates instead of replacing the full ov_degrees vector. Changes include: - frameDeltaAdapter: FrameDelta.orientation now accepts optional roll/pitch/yaw and applies Euler deltas to the previous pose (via applyEulerDeltaToPose) rather than overwriting the OV orientation. - transform: add poseToEulerDegrees and applyEulerDeltaToPose helpers to convert between poses and Euler angles and apply deltas. - useSceneBuilder: improve diff formatting (rounding for mm/deg) and show per-axis orientation changes (yaw/pitch/roll) using poseToEulerDegrees. - API (+server): update FrameDelta schema to accept optional roll/pitch/yaw, add field descriptions and enhance the system prompt with coordinate conventions, examples, and guidance to convert Viam orientation vectors to Euler angles. - Tests: update expectations and add cases covering yaw/pitch application and combining deltas with existing orientations. These changes make it easier to express incremental, intuitive rotations (e.g. "yaw 90°") and surface axis-specific diffs in the UI.

…gs).

micheal-parks · 2026-06-02T19:06:26Z

 * `Frame.svelte` uses to blend live kinematics with user-staged edits;
 * `worldMatrix.ts` premultiplies the result by the parent's `WorldMatrix`.
 */
+export const poseToEulerDegrees = (pose: Partial<Pose>): { roll: number; pitch: number; yaw: number } => {


Interesting, does the Typescript SDK not expose Viam's euler type signature?

Yes, the TS SDK includes the Euler type signature (https://ts.viam.dev/classes/appRobotApi.Orientation_EulerAngles.html). It's just that Orientation_eulerAngles is defined in radians. The callers of these functions are intentionally in degrees just because degrees are a more natural unit for the end user for frame editing. I could edit this function to use the SDK defined type signature – will just need to add conversions from deg <-> radians. what do you think?

Everything must be prerenderable.

github-actions · 2026-06-02T20:09:31Z

PR Preview Action v1.8.1
Preview removed because the pull request was closed.
2026-06-09 18:41 UTC

viamrobotics-overwatch · 2026-06-08T15:17:43Z

Hey @sucrammal — CI is green and no reviewer is assigned yet. Could you request one when you have a chance?

Auto-comment from overwatch. Will not re-nudge for 7 days.

mattmacf98

very close! there are just a couple more extractions to be made to fully plugin-ize this

nothing should go in App.svelte your provideAnthropicKey, LLMSceneBuilder component etc... should just go in the +layout.svelte so they are used in local motion tools without baking them into the exported MotionTools component
pull the AISettings out and inject from the top layer

mattmacf98 · 2026-06-08T22:34:12Z

 			{ label: 'Vision', component: VisionSettings },
 			{ label: 'Widgets', component: WidgetSettings },
 			{ label: 'Debug', component: DebugSettings },
+			{ label: 'AI', component: AISettings },


move AISettings into the plugin dir for AI and then inject it in via settingsTabs from the top level look at XRSettings as an example https://github.com/viamrobotics/visualization/blob/main/src/routes/%2Blayout.svelte#L70

mattmacf98 · 2026-06-08T22:37:55Z

 				<Logs />
 				<AddFrames />
+				{#if !localConfigProps || onInfer}
+					<LLMSceneBuilder onInfer={onInfer ?? standaloneInfer} />


this should not go here, this App gets exported as the component, put it here https://github.com/viamrobotics/visualization/blob/main/src/routes/%2Blayout.svelte#L78-L81 like the other local plugins

mattmacf98 · 2026-06-09T17:27:34Z

+	</fieldset>
+</Portal>
+
+<FloatingPanel


we can actually put this in <Portal id="dom"> I just found out (instead of having to put in the dashboard snippet

mattmacf98

LGTM 3 things before merge

run npx @changesets/cli to create a .changeset minor bump
remove the changes in ‎src/lib/hooks/useFrames.svelte.ts‎
wrap the floatingPanel in <Portal id="dom"> instead of putting in the dashboard snippet

sucrammal added 6 commits May 29, 2026 17:19

Merge branch 'main' into APP-16360-Add-LLM-3D-Scene-generation

86b02d1

Refactor SceneBuilder to Context Provider Pattern

f78c833

Lint

b640b7b

Rederive frame builder deltas; Recompute pose while save pending

5335d06

sucrammal requested review from mattmacf98 and micheal-parks June 1, 2026 22:03

mattmacf98 reviewed Jun 2, 2026

View reviewed changes

sucrammal added 6 commits June 2, 2026 09:31

Merge branch 'main' into APP-16360-Add-LLM-3D-Scene-generation

172df62

LLM explanation for complex prompts

4314dbe

Use sveltekit +server route instead of standalone file

1dc8215

fix packages

2559673

inline the apply loop directly in confirm() and delete applyPreparedU…

a9e10d6

…pdates and Deps from frameDeltaAdapter.ts.

micheal-parks reviewed Jun 2, 2026

View reviewed changes

Comment thread src/lib/hooks/useFrames.svelte.ts Outdated

Move SceneBuilder out of layout. copied pattern as cog button (settin…

41cb4c5

…gs).

micheal-parks reviewed Jun 2, 2026

View reviewed changes

sucrammal added 2 commits June 2, 2026 15:10

Revert scene builder route back into bun server

da6ceac

Everything must be prerenderable.

Convert config's OV --> euler angle for LLM interprebility

b808ec8

sucrammal added 4 commits June 2, 2026 16:25

Clamp LLM outputed Euler angles

a143eeb

Add own Anthropic API key for frame builder

5922be2

Lint fixes

5758ba4

Merge branch 'main' into APP-16360-Add-LLM-3D-Scene-generation

e3a0828

sucrammal mentioned this pull request Jun 3, 2026

APP-16360a: Fix baseline matrix and EditedMatrix update conditions in useFrames #742

Merged

sucrammal added 2 commits June 3, 2026 10:45

Merge branch 'main' into APP-16360-Add-LLM-3D-Scene-generation

f5d03bf

Move frame builder out of core and into plugin

92ad3b8

sucrammal requested a review from mattmacf98 June 8, 2026 19:00

sucrammal added 2 commits June 8, 2026 15:11

Merge; follow Settings refactoring into components

997ed0c

lint and CI

92099d6

mattmacf98 reviewed Jun 8, 2026

View reviewed changes

sucrammal added 2 commits June 9, 2026 13:09

Refactor properly out of app into plugin

fa2166d

Unify context naming, lint.

e40d3b5

mattmacf98 reviewed Jun 9, 2026

View reviewed changes

mattmacf98 approved these changes Jun 9, 2026

View reviewed changes

sucrammal added 4 commits June 9, 2026 14:21

fix settings context usage

8ad8a01

merge main into branch

ee662a9

changeset

20d0a7e

Revert useFrames changes

1366025

mattmacf98 approved these changes Jun 9, 2026

View reviewed changes

mattmacf98 merged commit 64db2c5 into main Jun 9, 2026
8 checks passed

mattmacf98 deleted the APP-16360-Add-LLM-3D-Scene-generation branch June 9, 2026 18:41

This was referenced Jun 10, 2026

LLM Scene builder cleanup: docs #767

Merged

Update useFrames.svelte.ts #769

Merged

Conversation

sucrammal commented Jun 1, 2026

TODOs

Uh oh!

changeset-bot Bot commented Jun 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

mattmacf98 commented Jun 2, 2026

Uh oh!

mattmacf98 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattmacf98 Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

micheal-parks Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sucrammal Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

viamrobotics-overwatch Bot commented Jun 8, 2026

Uh oh!

mattmacf98 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mattmacf98 Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

mattmacf98 Jun 8, 2026

Choose a reason for hiding this comment

Uh oh!

mattmacf98 Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mattmacf98 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

changeset-bot Bot commented Jun 1, 2026 •

edited

Loading

micheal-parks Jun 2, 2026 •

edited

Loading

github-actions Bot commented Jun 2, 2026 •

edited

Loading

mattmacf98 left a comment •

edited

Loading

mattmacf98 Jun 9, 2026 •

edited

Loading

mattmacf98 left a comment •

edited

Loading