Roadmap questions

[spunkedy]

@markovejnovic we talked on Reddit.

I figured having a diagram and thoughts here might be a little easier to communicate some of the ideas. I was thinking through ways of possibly separating out some of the layers to make them portable as an interface to be able to do things like:

• create extensions like vfkit / apple containers for the runtime
• scale out without having to need postgres as a hard dependency ( thinking through auto scaling groups etc )

If you are open to something like this, I can flush it out much more. In a PR

Proposal: split hyper_control / hyper_agent, behind a pluggable registry interface

Problem

:hyper is one Mix package today. Every node — whether it's fielding gRPC calls and
scheduling, or just running Firecracker VMs — runs identical code and links ecto_sql +
postgrex with live DB credentials, because agent-side code calls straight into
Hyper.Img.Db.* (Ecto/Postgres) rather than through an interface.

Concretely, three call sites do this today:

Call site	What it does	File
Hyper.Node.Img.Server.resolve_layers/1 → Db.Image.resolve_chain/1	VM boot: img_id → ordered layer/blob chain	lib/hyper/node/img/server.ex
Hyper.Node.Img.with_image_lease/3 → Db.Lease.bump/3, Db.Lease.release/1	Per-VM lease heartbeat while a VM lives	lib/hyper/node/img.ex
Hyper.Metering.Usage.record/1	Periodic CPU-usage row insert, called from FireVMM.Meter	lib/hyper/metering/usage.ex

Everything else that's Postgres-hardwired (Hyper.Img.create/2, create_derived/3, all of
Hyper.Img.Db.Gc) is write/ingest or singleton-GC work that belongs to the control side
regardless of any split.

Notably, the equivalent problem on the bytes side is already solved:
Hyper.Node.Layer.Repo (lib/hyper/node/layer/repo.ex) only ever resolves
layer_id -> path, and its moduledoc already says it may be "backed by anything you like:
a plain filesystem, an NFS drive." There's no Ecto in it. The gap is purely on the
metadata side.

Proposed split

Three Mix packages:

• hyper_control — scheduler, gRPC API, Postgres (Hyper.Img.Db.Repo + migrations),
  GC. 1–3 nodes, HA quorum.
• hyper_agent — VM execution: Node.VMSupervisor/FireVMM, Budget, Reaper,
  Users, the setuid suidhelper. N nodes.
• hyper_proto — shared contracts: Vm.Id/Vm.Spec, Unit.*, generated gRPC +
  Firecracker bindings. No Ecto, no scheduler, no Postgres.

Control and agents stay joined on the same Distributed Erlang mesh they use today —
Horde's CRDT gossip (Hyper.Cluster.Routing, Hyper.Cluster.Budget) keeps working
unmodified; Hyper.Cluster.Gc moves into hyper_control since it needs Postgres.

Module map

Module / area	Today	Proposed package	Port / adapter
Hyper.Img.Db.Repo + migrations	lib/hyper/img/db/repo.ex, priv/repo/	hyper_control	—
Hyper.Cluster.Scheduler	lib/hyper/cluster/scheduler.ex	hyper_control	—
Hyper.Img, Hyper.Img.Db.Gc	lib/hyper/img.ex, lib/hyper/img/db/gc*.ex	hyper_control	—
Hyper.Grpc.Server (external API)	lib/hyper/grpc/*.ex	hyper_control	—
Hyper.Node.Img / Img.Server — chain resolve, lease bump/release	lib/hyper/node/img.ex, img/server.ex	hyper_agent	Hyper.Img.Registry — Postgres or Remote adapter
Hyper.Metering.Usage (called from FireVMM.Meter)	lib/hyper/metering/usage.ex	hyper_agent	Hyper.Img.Registry.record_usage/1
Hyper.Node.Layer.Repo — blob path lookup	lib/hyper/node/layer/repo.ex	hyper_agent	Layer.Repo — local/NFS or blob-proxy adapter
Hyper.Node.* — VMSupervisor, FireVMM, Users, Budget, Reaper	lib/hyper/node/**	hyper_agent	—
suidhelper, guest-agent	native/**	hyper_agent	—
Hyper.Cluster.Routing / Budget (CRDTs)	lib/hyper/cluster/{routing,budget}.ex	started by both	—
Vm.Id/Vm.Spec, Unit.*, generated bindings	lib/hyper/vm/{id,spec}.ex, lib/unit/**, proto/**	hyper_proto	—

The interface layer

Two ports, each with a direct (colocated) adapter and a remote (funneled through
hyper_control) adapter — chosen independently of each other:

defmodule Hyper.Img.Registry do
  @moduledoc "Port: agent-side access to image lineage and lease/usage state."

  @callback resolve_chain(Hyper.Img.id()) :: [layer()]
  @callback lease_bump(Hyper.Img.id(), Hyper.Vm.Id.t(), Unit.Time.t()) ::
              {:ok, term()} | {:error, term()}
  @callback lease_release(Hyper.Vm.Id.t()) :: :ok
  @callback record_usage(Hyper.Metering.Usage.attrs()) :: :ok | {:error, term()}
end

Port	Adapter	Backing	Extra hop?	Best for
Hyper.Img.Registry (new)	Registry.Postgres	This node's own Ecto pool — Db.Image, Db.Lease, Metering.Usage	No	Trusted subnet — today's behavior, unchanged
Hyper.Img.Registry (new)	Registry.Remote	Call to hyper_control over the existing libcluster mesh	Yes — RTT on VM boot + lease heartbeat	NAT'd / cross-region agents with no direct DB reachability
Hyper.Node.Layer.Repo (exists today)	local / NFS mount	File.stat/1 against Cfg.Dirs.layer_dir()	No	Already shipped — works as-is
Hyper.Node.Layer.Repo (exists today)	blob-store proxy	Streamed / range-fetched from hyper_control or an object store	Yes — per-layer fetch on first mount	Cross-region deploys with no shared POSIX filesystem

Both remote adapters can start as plain Distributed Erlang calls, since agents already join
the same cluster as hyper_control for Horde's CRDTs — no new wire protocol needed until an
agent has to leave that mesh entirely, at which point the call becomes a gRPC RPC on
hyper.proto instead.

Trade-offs

Pros

• Execution nodes never ship Postgres drivers or DB credentials — smaller trust surface per VM host.
• Scheduler, GC, and the gRPC API redeploy independently of hundreds of live-VM agents.
• Bounded connection-pool pressure: control nodes' pools, not N-agents × pool_size.
• Reaches topologies today's design can't: NAT'd, cross-region, or edge agents with no DB line-of-sight.
• Agent-side tests run against a fake port — no Postgres needed in CI for that code path.

Cons

• New latency + failure mode on VM boot: resolve_chain becomes a network call, not a local query.
• Lease heartbeat now depends on control-plane reachability, adding a partition-vs-DB-down failure mode to classify.
• Control plane must be its own HA target, or it's a new bottleneck — today "HA" is just "Postgres is HA."
• Loses free transactionality: Ecto.Multi writes need a purpose-built transactional RPC when remote.
• More to operate: an internal RPC surface with its own versioning, auth boundary, and retry policy.

[spunkedy]

Current

Rough idea

[markovejnovic]

This is almost completely AI generated. Could you kindly spend some time rewriting this proposal by hand? I do not want to spend my human time reviewing AI proposals.

Thanks

[markovejnovic]

For future reference, I plan on keeping the roadmap here Hyper Roadmap.

[spunkedy]

Yes it is AI written as I asked it to summarize a couple implementations that I thought through.

I don't mind refining it completely, but I don't want to go through and invest the 2 hours towards something if you want to keep some of the features coupled.

To be clear, help is only helpful if it helps. If this isn't where you want to take it, out of respect. I don't want to waste your time or mine.

Do you like ideas of:

• a pluggable interface/adapter pattern that allows for other implementations like broadway, broadway_sqs ecto etc.
  • firecfracker
  • kvm
  • apple virtualization
  • bhyve
• a decoupling of hard requirement of postgres where it becomes optional for the executor nodes
  • runner nodes that can be decoupled from being in vpc/region etc

If so I don't mind drawing up some implementation plans / PRs off of some of those ideas to get alignment.

[markovejnovic]

got it, sorry for the cold shoulder and late reply, got a little busy.

Generally, I do like the idea of making stuff more modular.

---

I don't see how Broadway would fit the picture here. I haven't personally used broadway in production, but it feels to me like it's a mechanism for queue-based computing, something like an interface to rmq or kafka. I'm not really sure what it would serve to service in Hyper. What are you thinking of replacing with Broadway?

---

As for postgres specifically, I do, think it would be a great idea to tear out the explicit postgres dependency and enable users to pick and choose their data storage. I had a vibe-coded PR which attempted to support sqlite, but the code quality was junk so I threw it out the window.

We use postgres in Hyper for two applications -- an OLAP-like use-case wherein we store usage metrics for each VM (generally useful for billing applications) and, more importantly, the registry which holds the graph of all filesystems, as they are layered together.

This page goes into more detail. Unfortunately, the rendering seems broken (and a fix won't land until 0.2.0, since breaking gRPC changes are already on main), but I have a render here:

flowchart LR
  subgraph I1
    L1[(L1)]
    M1
    L1 -->|cow| M1
  end
  VM1[[VM1]]
  M1 -.->|mount| VM1

  subgraph I2
    L2[(L2)]
    M2
    L2 -->|cow| M2
  end

  L1 -->|cow| L2
  VM2[[VM2]]
  M2 -.->|mount| VM2

  subgraph I2'
    L2'[(L2')]
    M2'
    L2' -->|cow| M2'
  end

  L1 -->|cow| L2'
  VM2'[[VM2']]
  M2' -.->|mount| VM2'

  subgraph I3
    L3[(L3)]
    M3
    L3 -->|cow| M3
  end

  L2 -->|cow| L3
  VM3[[VM3]]
  M3 -.->|mount| VM3

Loading

If we can store this graph, somehow, with some sort of generic Hyper.Img.Registry behaviour, I think that would be awesome.

---

Going back to the usage, we have #77 tracking that. Part of that ticket is designing the interface. I would prefer if these designs were not AI-generated. That code is not load bearing, and has been poisoned by AI more than I would prefer -- I'd like to have a human-generated generic design that would serve that use-case.

---

As for having a pluggable interface adapter pattern, I'd like to clear up -- firecracker does go through KVM, explicitly. I do not want to have further support on Linux. Firecracker is "good enough" for managing VMs, and gives us a lot of niceties in terms of security, jailing, etc. that I would prefer not to manage ourselves.

Now, that being said, we'll need to support Windows and macOS VMs, and I'm guessing that we'll need to design a generic interface there. I think we should track that as a separate ticket, since I think it's probably a larger issue.

---

If you want to hop on a call regarding this, feel free to email me and we can schedule a call to discuss the architecture for any of these.

[spunkedy]

got it, sorry for the cold shoulder and late reply, got a little busy.

no worries

I don't see how Broadway would fit the picture here. I haven't personally used broadway in production, but it feels to me like it's a mechanism for queue-based computing, something like an interface to rmq or kafka. I'm not really sure what it would serve to service in Hyper. What are you thinking of replacing with Broadway?

I meant as an example of how they build out the composability, not using it sorry.

If we can store this graph, somehow, with some sort of generic Hyper.Img.Registry behaviour, I think that would be awesome.

That's the behavior I was thinking about.

Let me think through some practical ways that might allow for an implmentation that benefits going towards something like apple virtualization or shim layer.

Even having a linux vm with nested virtualization + firecracker might be better than having to go native as most mac usage isn't server grade but development grade.