reset: modular rebuild on TDD pipeline architecture#108
Closed
Exelord wants to merge 25 commits into
Closed
Conversation
The previous codebase had accreted enough cross-cutting concerns
(cache + sandbox + remote-cache + watch all tangled with the runner
and orchestrator) that further iteration was paying compound interest
on past refactors. Wipe the working tree and start over with a hard
constraint: each concern lives in its own module behind a stable
interface; the CLI is the only place modules are composed.
What's shipped in this PR
-------------------------
- src/workspace/ — discover projects (pnpm-workspace.yaml /
package.json workspaces / single-package); walk up to find a root.
- src/config/ — load + validate vx.config.{ts,mts,js,mjs}. Minimal
base schema (description / exec / dependsOn); extension modules
read their own fields off the same object without bloating it.
- src/graph/ — build the task DAG: dependency-spec parser
('name' / '^name' / 'pkg#task' / wildcards / negation), cycle
detection (iterative DFS coloring), Kahn topo sort with
deterministic tie-breaking, text/json/dot renderers.
- src/cli/ — argv parser + dispatcher. Ships one subcommand: vx graph.
- src/bin.ts + src/index.ts — binary entry + public re-exports.
Each module has its own README describing the contract + replacement
points, types.ts with the interface, a default implementation, and
collocated *.test.ts + *.bench.ts files. Underscored folders
(_bench/, _testkit/) are internal-only.
Tests + benches
---------------
- 93 tests across 9 files (84 collocated unit tests + 8 end-to-end
tests that spawn the real bin against fixture workspaces + 1 cycle
test). bun test runs in <500ms cold.
- mitata benchmarks per module: buildGraph at 200 projects x 8 tasks
runs in ~3ms. Discovery + config-load benches included for the
speed budget going forward.
Tooling
-------
- Dropped @anthropic-ai/sandbox-runtime (sandbox returns as a
standalone module later).
- Added mitata as a devDependency.
- CI gate is now install -> format-check -> lint -> test. No sandbox
setup, no apparmor toggles, no dogfood-vx step until the runner
module ships.
Next module: runner (vx run [tasks...] executes the graph in topo
order via Bun.spawn). Scheduler / logger / package-graph / cache
follow as separate PRs.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Mirrors the src/ tree under tests/. Benches (*.bench.ts) stay collocated since they reference internal helpers (src/_testkit) and measure performance against the source they sit next to. End-to-end tests already lived under tests/e2e/ — unchanged. Import paths in each moved test rewritten to '../../src/<module>/...'. The defineProject helper test in tests/config/load.test.ts now computes the absolute path to src/config/index.ts via join(import.meta.dir, '../../src/config') instead of import.meta.dir. CLAUDE.md + docs/README.md updated accordingly. 93 tests still pass; oxfmt + oxlint clean. https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Renames the public contract of `vx graph`: it now emits a pure JSON
inventory of the workspace (projects + their declared targets), with
no DAG resolution. The graph module's buildGraph stays in the tree but
isn't wired into the CLI yet — it'll come back when the runner ships.
Why
---
Two things were broken in the previous shape:
1. `vx graph` with no positional args returned "no tasks" even when
the workspace had configs. Confusing UX: discovery succeeded but the
output gave no signal.
2. `vx graph <task>` resolved `dependsOn`, which meant `^build` errored
out with "^name deps require the package-graph module". That makes
it impossible to inspect a turbo-style monorepo until package-graph
ships.
The new shape sidesteps both: emit a lossless, AI-first JSON dump of
the configuration. Authors can `vx graph | jq …` to query their setup;
LLMs can ingest it directly. Raw `dependsOn` strings are emitted
unresolved — the runner does the resolution work when it lands.
Inventory shape
---------------
```json
{
"workspace": { "root": "/abs/path" },
"projects": [
{
"name": "@scope/pkg",
"dir": "/abs/path/packages/pkg",
"targets": [
{
"name": "build",
"description": "compile",
"command": "tsc -b",
"dependsOn": ["^build", "compile"]
}
]
}
]
}
```
Projects without a `vx.config.ts` are emitted with `targets: []` so
users can see "vx found this package but it declared no tasks".
Repository restructure
----------------------
`src/` is now pure production code. Tests + benches + helpers all
moved out:
- `src/_testkit/` → `tests/_testkit/`
- `src/_bench/harness.ts` → `bench/_harness.ts`
- `src/<module>/*.bench.ts` → `bench/<module>/*.bench.ts`
Test/bench imports rewritten to point at `../../src/<module>/...`.
Module changes
--------------
- New: `src/inventory/` — types + buildInventory + tests + README.
- Dropped: `src/graph/format.ts` (text/DOT renderers) and its test.
- `src/cli/graph-cmd.ts` rewritten — JSON inventory only, no flags,
rejects positional args with a clear message.
- HELP text updated; `--json` / `--dot` no longer documented.
94 tests pass. oxfmt + oxlint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Pull back from the multi-module sprawl. Everything except config gone:
src/workspace, src/graph, src/inventory, src/cli, src/bin.ts, docs/,
plus all their tests and benches. The package no longer ships a CLI.
The config module is reduced to the minimum: discover the file, import
it, return whatever the user `export default`-ed. No validation, no
schema, no defineProject helper. ProjectConfig is `{}`.
Contract:
```ts
type LoadConfigs = (sources: readonly ConfigSource[]) =>
Promise<readonly LoadedConfig[]>
interface ConfigSource { name: string; dir: string }
interface LoadedConfig { source: ConfigSource; config: ProjectConfig }
interface ProjectConfig {} // intentionally empty
```
Sources without a vx.config file are silently omitted from the result.
Discovery order is .ts > .mts > .js > .mjs. First match wins.
5 tests pass. Format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Drop the loadConfigs(sources) shape entirely. The config module knows
nothing about workspaces, projects, names, file discovery, or
extensions. It cares about itself only.
API:
async function loadConfig(path: string): Promise<ProjectConfig>
interface ProjectConfig {}
The caller picks the path. The module dynamic-imports it and returns
the default export. No validation, no schema, no fallbacks.
The whole load.ts is now four lines:
export async function loadConfig(path: string): Promise<ProjectConfig> {
const mod = await import(path)
return mod.default as ProjectConfig
}
Tests reduced to 2 cases (empty config, arbitrary exported object).
Bench reduced to a single scenario.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Module folder src/config → src/project. Tests and benches moved too.
The names reflect what the module is actually about: a single
project's config file.
API:
async function loadProject(path: string): Promise<Project>
function defineProject<T extends Project>(project: T): T
interface Project {}
Both functions are stateless and isolated. The module knows nothing
about who calls it, how the path was chosen, or what the loaded
project will be used for. `defineProject` is identity at runtime —
it exists for type inference inside vx.config.ts files.
3 tests pass.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
ProjectSchema is the source of truth. The Project type is inferred
from it (`z.infer<typeof ProjectSchema>`) rather than declared by
hand. loadProject parses through the schema and throws ZodError on
invalid input.
Currently empty + strict: `z.strictObject({})`. Extension modules will
extend it later via `ProjectSchema.extend({...})`.
const ProjectSchema = z.strictObject({})
type Project = z.infer<typeof ProjectSchema>
async function loadProject(path) {
const mod = await import(path)
return ProjectSchema.parse(mod.default)
}
zod@4 added as a runtime dependency.
Tests now cover: empty config loads, unknown fields throw (strict),
non-object default throws, missing default export throws. 5 tests
pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Caller hands over a project directory; the module discovers
vx.config.{ts,mts,js,mjs} inside it (.ts > .mts > .js > .mjs, first
match wins) and imports + parses through ProjectSchema. Throws if the
directory has no config file at all.
This puts the extension-resolution responsibility inside the module
that owns it — callers don't need to know which extension a project
chose. dynamic `import()` requires an exact path, so the discovery
step was necessary anyway.
async function loadProject(dir: string): Promise<Project>
Tests grew to 10: load empty config, throws on missing config,
extension priority + each fallback, strict-schema rejection,
non-object default, missing default export.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
await import(join(dir, 'vx.config')) — Bun resolves the extension at
import time. No CONFIG_FILES array, no findConfigPath loop, no
Bun.file().exists() probe per candidate. The whole loader is now five
lines.
Bun's native priority is .mts > .ts > .mjs > .js (verified with all
four files present in one dir). That's now the documented order;
tests cover loading each extension individually rather than asserting
a specific priority.
export async function loadProject(dir: string): Promise<Project> {
const mod = await import(join(dir, 'vx.config'))
return ProjectSchema.parse(mod.default)
}
9 tests, all passing. Bun's "Cannot find module" error bubbles up when
no config exists — clear enough; no custom wrapper.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
A standalone validator for callers that already have data (HTTP body, JSON file, in-memory object) and want to assert it conforms to ProjectSchema before using it. Throws ZodError on mismatch. function validateProject(input: unknown): Project loadProject is now layered on top — it imports the user's vx.config.* and pipes the default export through validateProject, so validation lives in one place. 12 tests pass. https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Two changes:
1. ProjectSchema is no longer re-exported from src/project/index.ts or
src/index.ts. It stays inside the module, used by validateProject
and (formerly) loadProject. External callers reach the schema only
through validateProject — no direct .extend() access for now.
2. loadProject no longer validates. It returns the dynamic-import's
default export typed as `unknown`. Callers that want a Project pass
the result through validateProject:
const project = validateProject(await loadProject(dir))
Load and validate are separate concerns; composition belongs at the
call site.
11 tests pass.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
The module is small enough that splitting it into schema.ts +
load.ts + validate.ts + define.ts was ceremony, not structure.
Everything lives in src/project/index.ts now — 17 lines including
imports.
Tests collapse the same way: tests/project/{load,validate,define}.test.ts
become a single tests/project.test.ts file. 11 tests, three describe
blocks, one source-of-truth import.
Module README dropped — the whole file is short enough to read.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
The bench/ folder is gone. Bench files live alongside test files:
tests/
_harness.ts # mitata wrapper
_testkit/fixtures.ts # tmp-dir builder
project.test.ts # unit tests
project.bench.ts # mitata benchmarks
Same convention applies to future modules: tests/<module>.test.ts
and tests/<module>.bench.ts. tsconfig include narrowed to
["src/**/*", "tests/**/*"].
11 tests pass; bench runs (~3µs per loadProject); format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
…WorkspaceRoot
Mirrors the project module's surface for vx.workspace.{ts,mts,js,mjs}
files, plus adds findWorkspaceRoot which walks up from a path to the
workspace root.
API:
type Workspace // inferred, currently {}
async function loadWorkspace(dir: string): Promise<unknown>
function validateWorkspace(input: unknown): Workspace
function defineWorkspace<T extends Workspace>(w: T): T
async function findWorkspaceRoot(start: string): Promise<string>
Whole module is one file: src/workspace/index.ts (~21 lines).
findWorkspaceRoot is a thin wrapper over pkg-types' findWorkspaceDir.
pkg-types is a small unjs package that handles the marker-file
heuristic (pnpm-workspace.yaml, lerna.json, turbo.json, rush.json,
deno.json, .git/config, lockfiles, package.json). Throws with
"Cannot detect workspace root from <path>" when nothing is found.
Tests: 14 cases in tests/workspace.test.ts (load each extension,
validate strict, define identity, find root from same dir + from a
subdirectory, throw when no marker). Bench: findWorkspaceRoot
(~12µs), loadWorkspace (~12µs).
zod stays the only runtime dep besides pkg-types now.
25 tests total pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Both load functions now run their schema as part of loading. The
return type is the typed Project / Workspace instead of unknown.
Callers don't have to remember to validate.
async function loadProject(dir: string): Promise<Project>
async function loadWorkspace(root: string): Promise<Workspace>
validateProject and validateWorkspace stay exported for callers that
already have data from elsewhere (HTTP body, JSON file, in-memory).
Also renamed loadWorkspace's first arg from `dir` to `root` to match
its semantics — it expects the workspace root, not just any dir.
loadProject still takes `dir` (a project dir, not the root).
The workspace surface is intentionally root-aware but never embeds
an absolute path in any structured return value. Workspace stays {}
for now; findWorkspaceRoot is the only function that returns an
absolute path, by necessity.
25 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
The Workspace type carries the list of project locations declared by
the user. Each entry is a raw string — a glob (`packages/*`) or a
concrete path (`libs/core`). The module doesn't expand globs and
doesn't read package-manager files; vx.workspace.{ts,mts,js,mjs} is
the single source of truth.
const WorkspaceSchema = z.strictObject({
packages: z.array(z.string()).readonly(),
})
type Workspace = z.infer<typeof WorkspaceSchema>
async function loadWorkspace(root: string): Promise<Workspace>
function validateWorkspace(input: unknown): Workspace
function defineWorkspace<T extends Workspace>(w: T): T
async function findWorkspaceRoot(start: string): Promise<string>
Strict: `packages` is required, unknown fields throw, non-string
entries throw. Empty list is valid.
28 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Mirrors nx's project-graph in spirit: walks the workspace's `packages`
patterns, finds each vx.config.{ts,mts,js,mjs}, loads + validates
through the project schema, returns the workspace + a Map of relative
dir → Project.
interface Graph {
workspace: Workspace
projects: ReadonlyMap<string, Project> // relative dir → loaded project
}
async function loadGraph(root: string): Promise<Graph>
Implementation:
- Calls loadWorkspace(root) for the packages patterns.
- For each pattern, `new Bun.Glob(\`<pattern>/vx.config.{ts,mts,js,mjs}\`)`
to find candidate config files. Bun's brace expansion handles the
four extensions in one scan.
- Strips the /vx.config.<ext> suffix to get the dir.
- Dedupes via Set, sorts for stable ordering.
- Loads each project in parallel via Promise.all.
Dirs matched by the pattern but lacking a vx.config are silently
skipped (they're glob matches, not vx projects). Projects with
invalid vx.config files throw ZodError — propagated to the caller.
No edges yet — the Project schema is still empty, so there's nothing
to derive edges from. When fields like dependsOn ship, edges follow.
36 tests; format + lint clean. New file: src/graph/index.ts (~25 lines).
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
The workspace module shouldn't have known how to find the root —
that's an orchestration concern. Removed findWorkspaceRoot from
src/workspace/index.ts. Graph now owns the walk.
Graph walks up from `start` looking for vx.workspace.{ts,mts,js,mjs}
— our own marker, not whatever pkg-types finds. Dropped the pkg-types
dependency entirely; zod is the only runtime dep again.
async function loadGraph(start: string): Promise<Graph>
// walks up from start to find vx.workspace.* → root
// loadWorkspace(root) → workspace.packages
// glob each pattern for vx.config.{ts,mts,js,mjs} → project dirs
// loadProject in parallel
// returns { workspace, projects }
34 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Was four file.exists() checks per dir in a loop over an extension
list. Now one Bun.Glob('vx.workspace.{ts,mts,js,mjs}') scan per dir
— same effect, the loop is the walk-up, not the extension match.
The walk itself is inherently iterative (filesystems don't have an
upward glob), but the inner check is one call instead of four.
async function findRoot(start: string): Promise<string> {
let current = isAbsolute(start) ? start : resolve(start)
while (true) {
for await (const _ of WORKSPACE_MARKER.scan({ cwd: current })) return current
const parent = dirname(current)
if (parent === current) throw new Error(`no vx workspace found from ${start}`)
current = parent
}
}
Globs hoisted to module-top constants so they're reused across calls.
34 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Was two passes (resolveProjectDirs collected dirs, then a second map
loaded each). Now one loop: glob the pattern, fire loadProject as
each match comes in, collect promises, Promise.all at the end.
for (const pattern of workspace.packages) {
const glob = new Bun.Glob(`${pattern}/vx.config.{ts,mts,js,mjs}`)
for await (const match of glob.scan({ cwd: root })) {
const dir = match.slice(0, match.lastIndexOf('/vx.config.'))
if (seen.has(dir)) continue
seen.add(dir)
loads.push(loadProject(join(root, dir)).then(p => [dir, p]))
}
}
return { workspace, projects: new Map(await Promise.all(loads)) }
Workspace stays as raw user input (packages = whatever strings the
author wrote — globs or paths). Graph handles the glob expansion +
project loading in a single pass.
34 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
ProjectConfig / WorkspaceConfig are the user-authored schemas
(currently empty / { packages: string[] }). Project / Workspace are
what load* returns — the config wrapped with whatever else loading
needs to produce.
// project module
type ProjectConfig = z.infer<typeof ProjectConfigSchema> // {}
interface Project { config: ProjectConfig }
async function loadProject(dir): Promise<Project>
function validateProject(input): ProjectConfig
function defineProject<T extends ProjectConfig>(c: T): T
// workspace module
type WorkspaceConfig = z.infer<typeof WorkspaceConfigSchema> // { packages: string[] }
interface Workspace {
config: WorkspaceConfig
projects: ReadonlyMap<string, Project> // inferred from config.packages
}
async function loadWorkspace(root): Promise<Workspace>
function validateWorkspace(input): WorkspaceConfig
function defineWorkspace<T extends WorkspaceConfig>(c: T): T
// graph module
type Graph = Workspace
async function loadGraph(start): Promise<Graph> // findRoot + loadWorkspace
Workspace infers projects: glob each pattern with onlyFiles:false,
stat-filter to directories, loadProject on each in parallel. vx.config
is optional — a dir without it loads as { config: {} }.
loadProject is now in project module; the loadProjects helper is gone
(its work was absorbed into workspace's project inference).
29 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
…orts
Three changes folded together:
1. WorkspaceConfig schema dropped its `packages` field — empty `{}`
now. The PM (pnpm/yarn/npm/bun) is the source of truth for what
packages exist. vx.workspace.ts is optional; absent => empty config.
2. workspace.loadWorkspace uses @manypkg/get-packages to enumerate
workspace projects. Reads pnpm-workspace.yaml or package.json
`workspaces` field directly — no hand-rolled glob expansion, no
per-PM matrix.
3. loadProject simplified to just `await import(join(dir, 'vx.config'))`
with a `.catch(() => ({ default: {} }))` for the missing-config
case. No more pre-flight Bun.Glob existence check.
graph.loadGraph uses @manypkg/find-root for the walk-up.
New runtime deps: @manypkg/get-packages, @manypkg/find-root.
Workspace.projects keys are relative dirs (from @manypkg's
`relativeDir`). Project/Workspace are plain interfaces (not `extends
XConfig`) to avoid TS2411 from zod's `strictObject({})` inferring an
implicit `[k: string]: never` index signature.
Tests use fixture workspaces with a fake `bun.lock` / `pnpm-lock.yaml`
since @manypkg detects the PM by lockfile presence.
23 tests pass; format + lint clean.
https://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
1 task
Member
Author
|
Closing in favor of adopting this PR's principles in place on The core judgment: this branch's architecture instincts were right (modules behind contracts, composition only at the edges, TDD, benchmarks), but at its tip it carries only workspace/project/graph (28 files, 93 tests) while What's landed from this PR's agenda so far:
Also shipped from the same effort: a six-reviewer bug sweep that fixed a critical remote-cache exit-code bug (#109), corrupt-artifact poisoning (#113), the macOS sandbox Reopen if you disagree with the in-place call — the audit doc has the evidence either way. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The previous codebase had accreted enough cross-cutting concerns (cache + sandbox + remote-cache + watch all tangled with the runner and orchestrator) that further iteration was paying compound interest on past refactors. This PR wipes the working tree and starts over with a hard constraint: each concern lives in its own module behind a stable interface; the CLI is the only place modules are composed.
History is preserved in git —
git logstill walks back to the previous architecture.Shipped in this PR
workspace/config/vx.config.{ts,mts,js,mjs}. Minimal base schema; extension modules read their own fields off the same object.graph/cli/vx graph.bin.ts+index.tsEach module has its own
README.mddescribing the contract + replacement points,types.tswith the interface, a default implementation, and collocated*.test.ts+*.bench.tsfiles. Underscored folders (_bench/,_testkit/) are internal-only.Architecture principles (in CLAUDE.md)
index.ts.workspace → config → graph → runner → …. Each step is replaceable.Tests + benches
bun testruns in <500ms cold.buildGraphat 200 projects × 8 tasks runs in ~3ms.Tooling changes
@anthropic-ai/sandbox-runtime(sandbox returns as a standalone module later).mitataas a devDependency.Try it
Next
The runner module —
vx run [tasks...]executes the graph in topo order viaBun.spawn. Scheduler / logger / package-graph / cache follow as separate PRs.Test plan
bun test— 93/93 passbun x oxfmt --check .— cleanbun x oxlint --type-aware --type-check— cleanbun src/graph/build.bench.ts— runs, prints comparison summarybun src/bin.ts graph buildon a fixture workspace workshttps://claude.ai/code/session_01S6hT7kbvDDS8nqCTqvMWGJ
Generated by Claude Code