Background
This is a follow-up to the summit, where we identified this pattern and validated it with a PoC that suspends a real agent actor (openclaw) mid WebSocket and LLM/MCP calls.
Today both ingress and egress traffic reach the actor over TCP connections that cross the sandbox boundary: inbound requests are forwarded from outside the sandbox to a port the actor listens on, and the actor's outbound connections have their peer endpoint outside the sandbox.
A TCP connection's state lives at both endpoints, so any connection crossing the boundary is broken by suspend, for two reasons: checkpointing freezes only the sandbox side, so the outside half is closed or lost and the actor's frozen half has no peer on restore; and the actor may resume on a different worker with a different IP, which TCP cannot follow. But connections with both endpoints inside the sandbox - loopback/localhost - survive checkpoint/restore intact, since both halves are snapshotted together and are insensitive to where the sandbox resumes. That idea is the basis of this proposal.
Problem
An actor that is serving an inbound HTTP request, or waiting on an outbound HTTP call, cannot be safely suspended: the socket drops and the request fails. It forces actors to be suspend-aware, forces the platform to avoid suspending actors with in-flight traffic, and makes long-running calls (e.g. an agent waiting minutes on an LLM response) effectively pin the actor as running.
Being able to suspend an actor at any point, in-flight traffic included, greatly boosts the value of suspend/resume and the system as a whole: it turns it from an idle-only optimization into a general platform mechanism:
-
Density and cost: agents spend most of their wall-clock time waiting on slow calls (LLM inference, tools, humans). If waiting pins the actor as running, the workloads that would benefit most from suspension are exactly the ones that can never be suspended. Suspend-during-wait lets a node oversubscribe to actual CPU use rather than open connections.
-
Upgrades and maintenance: draining a worker for a node upgrade, or rebalancing no longer needs to wait for actors to go quiet or fail their in-flight requests - network-wise, any actor can be checkpointed immediately and resumed elsewhere.
-
Scheduling freedom: the scheduler can possibly preempt and migrate actors based on resource pressure alone, without reasoning about connection state or coordinating with the traffic path.
-
Failure recovery: a resumed-from-checkpoint actor comes back with its in-flight work intact instead of with a lap full of broken sockets it must detect and retry.
Proposal
Note: this is a proposal we want community feedback on: should this be part of substrate? The real benefit is the set of capabilities listed in the Problem section. It could also ship as an opt-in mode in substrate rather than the behavior for all users.
Inject a pair of substrate-owned, substrate-aware proxy containers into every actor sandbox:
-
Ingress proxy: a reverse proxy in front of the actor. It receives inbound requests as messages, replays them to the actor container over localhost HTTP, and publishes the response as a message.
-
Egress proxy: a forward proxy for the actor's outbound HTTP calls. It accepts them on localhost (becoming their connection endpoint), publishes the request as a message, and replays the response to the actor when it arrives.
The actor container only ever holds loopback connections to the two proxies. Since the proxies live inside the sandbox, those connections are checkpointed and restored together with the actor - the actor can be suspended at any point, including mid-request, and observes nothing.
The suspend-sensitive hop (sandbox ↔ outside world) could become message-based instead of connection-based:
external client
=> ingress gateway (holds the client HTTP connection, or async API)
=> downstream queue [http request message]
=> ingress proxy --http/localhost--> actor container
=> actor container --http/localhost--> egress proxy
=> downstream queue [http request message]
=> egress gateway / PEP => external destination
(responses flow back the same way via upstream queues)
External semantics are unchanged: the ingress gateway still holds the client connection and returns a normal synchronous HTTP response once the response message arrives. Because the sandbox side is decoupled from the client connection, an async API is also possible as an alternative (submit a request, receive a request ID, poll or get a callback for the response) for calls that outlive reasonable HTTP timeouts.
Transport between proxies and the outside
The queues need to be durable across suspend, ordered per connection, and consumable from wherever the sandbox resumes. Redis Streams is one candidate backing: it is durable with per-consumer offsets (so a message published while the sandbox is suspended is still there on resume), and Redis is well suited to peer-to-peer messaging across many per-actor queues. A gRPC stream between the proxies and the gateways is an alternative that avoids a broker dependency, but it reintroduces a connection that must be re-established on resume and creates stickiness between the sandbox and a specific gateway instance, so a queue is the cleaner fit.
Security considerations
The injected proxies hold no credentials and no policy authority - they are delivery/replay agents, so compromising them gains the actor nothing. Actor identity attestation and policy enforcement stay outside the sandbox, attached where messages cross the boundary.
Open questions
-
Timeline: when should this be proposed and built, relative to the in-flight ingress/egress work?
-
Opt-in per substrate deployment vs. the long-term default networking path.
-
Queue backing (Redis Streams/pub-sub, substrate-native buffer, pluggable broker) and delivery semantics (at-least-once + idempotent replay vs. exactly-once).
-
Handling of streaming responses, SSE, and WebSockets across suspend.
-
Timeout ownership: who fails a request whose actor stays suspended too long - the ingress gateway, the queue TTL, or policy?
-
Proxy footprint inside the sandbox (checkpoint size, startup latency) and whether the proxies are one container or two.
Non-goals
-
Changing how ingress extracts actor identity, how actors are resumed and located, or how egress policy is enforced.
-
Supporting non-HTTP protocols in the first iteration.
Relationship to the existing ingress/egress proposals
This proposal is orthogonal to the pluggable ingress (#326, #431) and egress (#430, #126, #460) work, for the high-level reasons above:
-
The trusted components must stay outside the sandbox. Identity stamping, policy enforcement, routing, and resume stay exactly where those proposals put them; the injected proxies are untrusted and credential-free, so nothing about the trust model or the gateway/PEP design moves.
-
No placement of an outside component can make the last hop suspend-safe: a socket crossing the sandbox boundary cannot be turned into loopback, because loopback requires both endpoints inside the checkpoint. Those proposals define where traffic enters, exits, and is policed; this one only changes how the final hop crosses the boundary - messages instead of a live socket.
xref:
Background
This is a follow-up to the summit, where we identified this pattern and validated it with a PoC that suspends a real agent actor (openclaw) mid WebSocket and LLM/MCP calls.
Today both ingress and egress traffic reach the actor over TCP connections that cross the sandbox boundary: inbound requests are forwarded from outside the sandbox to a port the actor listens on, and the actor's outbound connections have their peer endpoint outside the sandbox.
A TCP connection's state lives at both endpoints, so any connection crossing the boundary is broken by suspend, for two reasons: checkpointing freezes only the sandbox side, so the outside half is closed or lost and the actor's frozen half has no peer on restore; and the actor may resume on a different worker with a different IP, which TCP cannot follow. But connections with both endpoints inside the sandbox - loopback/localhost - survive checkpoint/restore intact, since both halves are snapshotted together and are insensitive to where the sandbox resumes. That idea is the basis of this proposal.
Problem
An actor that is serving an inbound HTTP request, or waiting on an outbound HTTP call, cannot be safely suspended: the socket drops and the request fails. It forces actors to be suspend-aware, forces the platform to avoid suspending actors with in-flight traffic, and makes long-running calls (e.g. an agent waiting minutes on an LLM response) effectively pin the actor as running.
Being able to suspend an actor at any point, in-flight traffic included, greatly boosts the value of suspend/resume and the system as a whole: it turns it from an idle-only optimization into a general platform mechanism:
Density and cost: agents spend most of their wall-clock time waiting on slow calls (LLM inference, tools, humans). If waiting pins the actor as running, the workloads that would benefit most from suspension are exactly the ones that can never be suspended. Suspend-during-wait lets a node oversubscribe to actual CPU use rather than open connections.
Upgrades and maintenance: draining a worker for a node upgrade, or rebalancing no longer needs to wait for actors to go quiet or fail their in-flight requests - network-wise, any actor can be checkpointed immediately and resumed elsewhere.
Scheduling freedom: the scheduler can possibly preempt and migrate actors based on resource pressure alone, without reasoning about connection state or coordinating with the traffic path.
Failure recovery: a resumed-from-checkpoint actor comes back with its in-flight work intact instead of with a lap full of broken sockets it must detect and retry.
Proposal
Note: this is a proposal we want community feedback on: should this be part of substrate? The real benefit is the set of capabilities listed in the Problem section. It could also ship as an opt-in mode in substrate rather than the behavior for all users.
Inject a pair of substrate-owned, substrate-aware proxy containers into every actor sandbox:
Ingress proxy: a reverse proxy in front of the actor. It receives inbound requests as messages, replays them to the actor container over localhost HTTP, and publishes the response as a message.
Egress proxy: a forward proxy for the actor's outbound HTTP calls. It accepts them on localhost (becoming their connection endpoint), publishes the request as a message, and replays the response to the actor when it arrives.
The actor container only ever holds loopback connections to the two proxies. Since the proxies live inside the sandbox, those connections are checkpointed and restored together with the actor - the actor can be suspended at any point, including mid-request, and observes nothing.
The suspend-sensitive hop (sandbox ↔ outside world) could become message-based instead of connection-based:
External semantics are unchanged: the ingress gateway still holds the client connection and returns a normal synchronous HTTP response once the response message arrives. Because the sandbox side is decoupled from the client connection, an async API is also possible as an alternative (submit a request, receive a request ID, poll or get a callback for the response) for calls that outlive reasonable HTTP timeouts.
Transport between proxies and the outside
The queues need to be durable across suspend, ordered per connection, and consumable from wherever the sandbox resumes. Redis Streams is one candidate backing: it is durable with per-consumer offsets (so a message published while the sandbox is suspended is still there on resume), and Redis is well suited to peer-to-peer messaging across many per-actor queues. A gRPC stream between the proxies and the gateways is an alternative that avoids a broker dependency, but it reintroduces a connection that must be re-established on resume and creates stickiness between the sandbox and a specific gateway instance, so a queue is the cleaner fit.
Security considerations
The injected proxies hold no credentials and no policy authority - they are delivery/replay agents, so compromising them gains the actor nothing. Actor identity attestation and policy enforcement stay outside the sandbox, attached where messages cross the boundary.
Open questions
Timeline: when should this be proposed and built, relative to the in-flight ingress/egress work?
Opt-in per substrate deployment vs. the long-term default networking path.
Queue backing (Redis Streams/pub-sub, substrate-native buffer, pluggable broker) and delivery semantics (at-least-once + idempotent replay vs. exactly-once).
Handling of streaming responses, SSE, and WebSockets across suspend.
Timeout ownership: who fails a request whose actor stays suspended too long - the ingress gateway, the queue TTL, or policy?
Proxy footprint inside the sandbox (checkpoint size, startup latency) and whether the proxies are one container or two.
Non-goals
Changing how ingress extracts actor identity, how actors are resumed and located, or how egress policy is enforced.
Supporting non-HTTP protocols in the first iteration.
Relationship to the existing ingress/egress proposals
This proposal is orthogonal to the pluggable ingress (#326, #431) and egress (#430, #126, #460) work, for the high-level reasons above:
The trusted components must stay outside the sandbox. Identity stamping, policy enforcement, routing, and resume stay exactly where those proposals put them; the injected proxies are untrusted and credential-free, so nothing about the trust model or the gateway/PEP design moves.
No placement of an outside component can make the last hop suspend-safe: a socket crossing the sandbox boundary cannot be turned into loopback, because loopback requires both endpoints inside the checkpoint. Those proposals define where traffic enters, exits, and is policed; this one only changes how the final hop crosses the boundary - messages instead of a live socket.
xref:
Pluggable Ingress Support #326
Pluggable Egress Support #430
Enhancing Network Ingress #431
Add transparent actor egress policy and credential injection #126
Implement Milestone2 Networking (Ingress and Egress) #460