Benchmark replay by Dreamsorcerer · Pull Request #3347 · dimensionalOS/dimos

Dreamsorcerer · 2026-08-04T12:52:49Z

No description provided.

mintlify · 2026-08-04T12:52:53Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
dimensional	🟢 Ready	View Preview	Aug 4, 2026, 12:53 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

greptile-apps · 2026-08-04T12:56:08Z

Greptile Summary

The PR adds replay-window controls and a CodSpeed benchmark that measures draining a recorded Go2 workload.

Adds configurable replay speed, seek offset, and duration.
Forwards the new settings through the Go2 replay connection.
Adds a replay-drain benchmark, supporting tests, CI workflow, documentation, and the pytest-codspeed dependency.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
.github/workflows/codspeed.yml	Adds the ARM64 CodSpeed workflow, replay-data preparation, diagnostics, smoke checks, and benchmark execution.
dimos/core/global_config.py	Adds process-wide replay speed, seek, and duration configuration fields.
dimos/robot/unitree/go2/connection.py	Forwards replay controls from global configuration into the SQLite replay implementation.
dimos/robot/unitree/go2/test_replay_benchmark.py	Adds a bounded replay-drain benchmark with stream validity floors, quiescence detection, and teardown handling.
pyproject.toml	Adds the pinned pytest-codspeed test dependency.
uv.lock	Locks pytest-codspeed 5.0.3 and its resolved artifacts and dependencies.

Sequence Diagram

sequenceDiagram
    participant CI as CodSpeed workflow
    participant Pytest as pytest-codspeed
    participant Coord as ModuleCoordinator
    participant Go2 as ReplayConnection
    participant DB as SQLite replay
    participant Topics as Watched transports

    CI->>Pytest: Run replay benchmark
    Pytest->>Coord: Build unitree-go2 blueprint
    Coord->>Go2: Start configured replay
    Go2->>DB: Open recording with speed/window
    DB-->>Topics: Emit odom, lidar, and images
    Topics-->>Pytest: Record arrivals
    Pytest->>Pytest: Wait for count floors and quiescence
    Pytest-->>CI: Report wall-time sample

_{Reviews (8): Last reviewed commit: "Fix" | Re-trigger Greptile}

codecov · 2026-08-04T13:01:31Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
3965	1	3964	68

View the top 1 failed test(s) by shortest run time

dimos.robot.unitree.go2.test_replay_benchmark::test_go2_replay_drain_walltime

Stack Traces | 120s run time

+ Exception Group Traceback (most recent call last):
  |   File ".../dimos/dimos/.venv/lib/python3.12........./site-packages/_pytest/runner.py", line 341, in from_call
  |     result: TResult | None = func()
  |                              ^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12........./site-packages/_pytest/runner.py", line 242, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_hooks.py", line 512, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 167, in _multicall
  |     raise exception
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
  |     yield
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
  |     yield
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/logging.py", line 829, in _runtest_for
  |     yield
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.../site-packages/_pytest/capture.py", line 898, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.../site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 121, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12........./site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
  |     item.runtest()
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/python.py", line 1627, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_hooks.py", line 512, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 167, in _multicall
  |     raise exception
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 121, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../unitree/go2/test_replay_benchmark.py", line 181, in test_go2_replay_drain_walltime
  |     benchmark.pedantic(build_and_drain, teardown=teardown, rounds=1, warmup_rounds=0)
  |   File ".../dimos/dimos/.venv/lib/python3.12.../site-packages/pytest_codspeed/plugin.py", line 396, in pedantic
  |     result = target(*args, **kwargs)
  |              ^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../unitree/go2/test_replay_benchmark.py", line 155, in build_and_drain
  |     state["coordinator"] = ModuleCoordinator.build(blueprint)
  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../core/coordination/module_coordinator.py", line 368, in build
  |     coordinator.start_all_modules()
  |   File ".../core/coordination/module_coordinator.py", line 257, in start_all_modules
  |     safe_thread_map(modules, lambda m: m.start())
  |   File ".../dimos/utils/safe_thread_map.py", line 84, in safe_thread_map
  |     raise ExceptionGroup("safe_thread_map failed", errors)
  | ExceptionGroup: safe_thread_map failed (1 sub-exception)
  +-+---------------- 1 ----------------
    | dimos.protocol.rpc.rpc_utils.RemoteError: [Remote builtins.RuntimeError] Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.
    | e.g. jpeg = TurboJPEG(lib_path)
    | 
    | Remote traceback:
    | Traceback (most recent call last):
    |   File ".../protocol/rpc/pubsubrpc.py", line 280, in execute_and_respond
    |     response = f(*args[0], **args[1])
    |                ^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../protocol/rpc/spec.py", line 116, in override_f
    |     return getattr(module, fname)(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../unitree/go2/connection.py", line 348, in start
    |     self.register_disposable(self.connection.video_stream().subscribe(onimage))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 142, in subscribe
    |     current_thread_scheduler.schedule(set_disposable)
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampolinescheduler.py", line 49, in schedule
    |     return self.schedule_absolute(self.now, action, state=state)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampolinescheduler.py", line 95, in schedule_absolute
    |     self.get_trampoline().run(item)
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampoline.py", line 30, in run
    |     self._run()
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampoline.py", line 51, in _run
    |     item.invoke()
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/scheduleditem.py", line 25, in invoke
    |     ret = self.scheduler.invoke_action(self.action, state=self.state)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/scheduler.py", line 105, in invoke_action
    |     ret = action(self, state)
    |           ^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 127, in set_disposable
    |     if not auto_detach_observer.fail(ex):
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observer/autodetachobserver.py", line 64, in fail
    |     self._on_error(exn)
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/internal/basic.py", line 34, in default_error
    |     raise err
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 125, in set_disposable
    |     subscriber = self._subscribe_core(auto_detach_observer, scheduler)
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 49, in _subscribe_core
    |     return self._subscribe(observer, scheduler) if self._subscribe else Disposable()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/memory2/replay.py", line 250, in subscribe
    |     first_ts, first_data = next(iterator)
    |                            ^^^^^^^^^^^^^^
    |   File ".../dimos/memory2/replay.py", line 243, in make_iterator
    |     yield (obs.ts, decode(obs))
    |                    ^^^^^^^^^^^
    |   File ".../dimos/memory2/replay.py", line 136, in _decode
    |     data = obs.data
    |            ^^^^^^^^
    |   File ".../memory2/type/observation.py", line 181, in data
    |     loaded = self._loader()
    |              ^^^^^^^^^^^^^^
    |   File ".../dimos/memory2/backend.py", line 91, in loader
    |     return codec.decode(raw)
    |            ^^^^^^^^^^^^^^^^^
    |   File ".../memory2/codecs/jpeg.py", line 39, in decode
    |     return Image.lcm_jpeg_decode(data)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../msgs/sensor_msgs/Image.py", line 604, in lcm_jpeg_decode
    |     jpeg = TurboJPEG()
    |            ^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12/site-packages/turbojpeg.py", line 301, in __init__
    |     self.__find_turbojpeg() if lib_path is None else lib_path)
    |     ^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12/site-packages/turbojpeg.py", line 996, in __find_turbojpeg
    |     raise RuntimeError(
    | RuntimeError: Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.
    | e.g. jpeg = TurboJPEG(lib_path)
    | 
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File ".../dimos/utils/safe_thread_map.py", line 68, in safe_thread_map
    |     outcomes[idx] = fut.result()
    |                     ^^^^^^^^^^^^
    |   File "........./usr/lib/python3.12....../concurrent/futures/_base.py", line 449, in result
    |     return self.__get_result()
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "........./usr/lib/python3.12....../concurrent/futures/_base.py", line 401, in __get_result
    |     raise self._exception
    |   File "........./usr/lib/python3.12.../concurrent/futures/thread.py", line 58, in run
    |     result = self.fn(*self.args, **self.kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../core/coordination/module_coordinator.py", line 257, in <lambda>
    |     safe_thread_map(modules, lambda m: m.start())
    |                                        ^^^^^^^^^
    |   File ".../dimos/core/rpc_client.py", line 93, in __call__
    |     result, unsub_fn = self._rpc.call_sync(
    |                        ^^^^^^^^^^^^^^^^^^^^
    |   File ".../protocol/rpc/spec.py", line 88, in call_sync
    |     raise result
    | RuntimeError: Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.
    | e.g. jpeg = TurboJPEG(lib_path)
    +------------------------------------

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

… into sam/benchmark-replay

Benchmark replay

1b8c639

Dreamsorcerer requested review from leshy, mustafab0, paul-nechifor and spomichter as code owners August 4, 2026 12:52

mintlify Bot deployed to staging - docs August 4, 2026 12:53 View deployment

Merge branch 'main' into sam/benchmark-replay

77c8f6b

mintlify Bot deployed to staging - docs August 4, 2026 14:04 View deployment

Dreamsorcerer added 2 commits August 4, 2026 17:06

Fix

badf42c

Merge branch 'sam/benchmark-replay' of github.com:dimensionalOS/dimos…

c0079c7

… into sam/benchmark-replay

mintlify Bot deployed to staging - docs August 4, 2026 16:09 View deployment

Dreamsorcerer added 6 commits August 4, 2026 20:11

Fix

11a09b1

Fix

82a52d1

Fix

92ca3fa

Fix

8c41c87

Fix

8189b7b

Fix

628aeb6

Navigation Menu

Uh oh!

Benchmark replay - #3347

Benchmark replay#3347
Dreamsorcerer wants to merge 10 commits into
mainfrom
sam/benchmark-replay

Dreamsorcerer commented Aug 4, 2026

Uh oh!

mintlify Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Dreamsorcerer commented Aug 4, 2026

Uh oh!

mintlify Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Sequence Diagram

Uh oh!

codecov Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mintlify Bot commented Aug 4, 2026 •

edited

Loading

greptile-apps Bot commented Aug 4, 2026 •

edited

Loading

codecov Bot commented Aug 4, 2026 •

edited

Loading