Feature Request: Resource-usage telemetry at actor / ActorTemplate granularity

[git286]

Problem

Substrate cannot answer "how much CPU and memory is this actor using?" — nor the same question rolled up per ActorTemplate.

Resource usage is observable only at worker-pod granularity, via kubectl ate top workers. actor is the unit users create, name, bill for, debug, and reason about — and it is invisible.

Why this matters

Concretely, four groups are blocked:

Who	What they cannot do today
Agent developers	Diagnose why an actor was OOM-killed. Size an ActorTemplate's resource requests with anything but guesswork. Tell which of their actors is the expensive one.
Platform operators	Detect noisy neighbours. Attribute cost or chargeback to a tenant, when atespace is the tenant boundary. Set quotas informed by observed usage.
Substrate developers	Validate the density claim, or tune multiplexing against real numbers. #44 (concurrent actors per worker, P0) has no way to measure whether it succeeded.
Load testing	Explain bottlenecks or confirm fleet health under load — the original motivation for #174, and still open in #168 / #176.

This also gates scheduling work. Demand-reactive capacity (#198) and suspend-on-inactivity (#424) both make better decisions with a real actor resource footprint than without one.

---

Related

• Observability At-Scale #174 Observability At-Scale — parent discussion, granularity taxonomy, cardinality decisions
• Initial set of platform metrics around lifecycle, scheduling, and fleet-saturation #433 Initial set of platform metrics — the metrics workstream this complements
• All actors are under the pause cgroup #288 All actors are under the pause cgroup — closed, was the stated blocker
• ateom gvisor: drop privileged: true #496 ateom gvisor: drop privileged: true — delivered the per-actor cgroup leaves
• fix(atelet): set OCI CgroupsPath so each actor gets its own cgroup #161 fix(atelet): set OCI CgroupsPath — open, superseded by ateom gvisor: drop privileged: true #496
• Feature request: Implement 'kubectl ate top workers' command for real-time worker pod resource utilization #515 / feat(kubectl-ate): implement kubectl ate top workers command (#515) #516 kubectl ate top workers — the pod-level precedent
• High Concurrent Active Actors per Worker #44 High Concurrent Active Actors per Worker — will invalidate pod-as-proxy-for-actor
• feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools #198 WorkerPool autoscaling / Feature request: suspend actor on inactivity #424 suspend on inactivity — consumers of actor usage data
• OpenTelemetry Application Telemetry Data Loss and Delivery Delays Across Actor Lifecycle Events #503 OpenTelemetry data loss across actor lifecycle events — adjacent, application-side

[krisztianfekete]

+1 for driving this!
Most important question to officially decide I think is to what signals to use for what purpose.

To follow #174 and OTel best practices, these shouldn't be on metrics: actor, atespace, session, and we should add high-cardinality attrs on traces/logs/events. So individual actor usage is probably best as events that atelet can emit as something like ate.actor.usage.

ActorTemplate/pool/sandbox labels can be on metrics, as those are fairly bounded. I guess we can start with two histograms, once for CPU, and one for memory?

[git286]

+1 for driving this! Most important question to officially decide I think is to what signals to use for what purpose.

To follow #174 and OTel best practices, these shouldn't be on metrics: actor, atespace, session', and we should add high-cardinality attrs on traces/logs/events. So individual actor usage is probably best as [events](https://opentelemetry.io/docs/specs/semconv/general/events/) that ateletcan emit as something likeate.actor.usage`.

ActorTemplate/pool/sandbox labels can be on metrics, as those are fairly bounded. I guess we can start with two histograms, once for CPU, and one for memory?

Thanks and agreed on the signal split. actor / atespace / session as event attributes, ActorTemplate / pool / sandbox class as metric labels is the right division, and it's the constraint I'd want this issue to inherit from #174 rather than relitigate. Two deliverables, not one: aggregated template-keyed metrics for the TSDB, and per-actor events for drill-down.

I'd put the emitter in ateom rather than atelet

Two reasons:

atelet can't see the data. It's a node-level DaemonSet in a different pod, and its only host mount is /var/lib/ateom-gvisor. No /sys/fs/cgroup, no hostPID. Reading the leaves from atelet would mean adding a host cgroupfs mount and reconstructing host cgroup paths from container IDs, which is close to the coupling #496 just removed. The leaves are only addressable as /sys/fs/cgroup/<containerName> from inside ateom's own cgroup namespace.

ateom already has the identity, and already emits actor-scoped signals. RunWorkloadRequest carries atespace, actor_name, actor_uid, actor_template_namespace, actor_template_name, and internal/actorlog already emits lifecycle events tagged with exactly that ate.dev/actor_* label set, shared by both the gVisor and micro-VM runtimes. An ate.actor.usage event is a natural sibling of EmitLifecycleLog — same emitter, same attributes, same code path for both sandbox classes.

atelet still has a role if we want the aggregation rollup to happen off the worker, but the read has to originate in ateom.

[krisztianfekete]

Ah right, that was a typo from my side! Btw. ateam is not wired in to OTel path yet, was planning to do that here in the split chunks (#433 (comment)). Happy to do item 5) now so you're unblocked.

[git286]

Ah right, that was a typo from my side! Btw. ateam is not wired in to OTel path yet, was planning to do that here in the split chunks (#433 (comment)). Happy to do item 5) now so you're unblocked.

Thanks and that would be appreciated!

[baizhenyu]

Full design proposal below, inheriting the signal split Krisztian F (@krisztianfekete) and I agreed on above: actor / atespace / session on events only, template / sandbox class as the metric dimensions.

One correction to the issue body first, because it changes the implementation.

#496 gave us the cgroup leaves, but they are empty

The issue lists #288 as the stated blocker and #496 as having delivered per-actor leaves. The directories exist, but they contain no processes and will always report zero.

ensureContainerCgroupsPath (cmd/ateom-gvisor/runsc.go:39-72) hands each container an absolute /<containerName>, and runsc creates it — but setupCgroupForSubcontainer installs it with empty resources ("Use empty resources, just want the directory structure created"), and container.go:133-135 is explicit that CompatCgroup exists only so tools that expect container cgroups find them. Every application process runs inside the sentry, and the sentry is a single process in pause/.

/sys/fs/cgroup/          the ateom container scope, carries memory.max
├── ateom/               the ateom daemon
├── pause/               sentry + gofers — all actor CPU and memory
└── main/, sidecar/      empty

So the readable unit is the sandbox, which today equals the actor. Per-container breakdown for gVisor needs runsc events / ContainerUsage from the sentry, which I have put in a later phase. Worth editing the issue body so this is not rediscovered in review.

Architecture: read in ateom, emit from atelet

The read does have to originate in ateom, but I no longer think ateom should be the emitter.

  Worker pod (one per actor)          Node
 ┌─────────────────────────┐   ┌────────────────────┐
 │ ateom                   │   │ atelet             │
 │  reads cgroup / guest   │◄──┤  reads on start,   │
 │  returns raw counters   │──►│  timer, teardown   │      ┌──────────────┐
 └─────────────────────────┘   │                    │      │              │
                               │  aggregate by      │─────►│ OTel endpoint│
 ┌─────────────────────────┐   │  template → metrics│ OTLP │              │
 │ ateom                   │◄─►│  per actor → events│─────►│              │
 └─────────────────────────┘   └────────────────────┘      └──────────────┘

Every ateom process registers as a distinct OTLP source, since serverboot.newResource stamps a per-process service.instance.id. If ateom emitted the metrics, series count would scale with worker pods rather than nodes, and every worker restart would mint a fresh instance ID and therefore a fresh timeseries. atelet is a stable DaemonSet that already emits template-labeled metrics today (atelet.snapshot.size, cmd/atelet/main.go:277-311).

A useful side effect: because ateom only serves a read, this work does not block on the ateom OTel wiring. Krisztian F (@krisztianfekete), your item 5 is still worth doing — ateom currently calls InitTracing but worker pods have no OTEL_EXPORTER_OTLP_ENDPOINT (workerpool_apply.go:38-44 sets only POD_UID, and applyWorkerPoolPodTemplate has no env path), so those spans are going to localhost:4317 and being dropped. That is a live bug independent of this issue.

What ateom reads

gVisor: from pause/ — memory.current, memory.peak, memory.stat:inactive_file, cpu.stat:usage_usec. memory.peak needs kernel 5.19+.

Micro-VM: the host cgroup is not usable. Guest RAM is a fixed allocation (default 2048 MiB via kata.ParseConfig), so host memory stays roughly constant regardless of actor behaviour, and Shared: true backs it with a memfd, putting those pages on the file LRU — which makes the working-set calculation subtract idle guest RAM and understate the footprint. (This is already quietly skewing kubectl ate top workers for micro-VM workers.) ateom should query the guest agent (StatsContainer) instead, which gives genuine per-container numbers.

Interface

message StatsWorkloadResponse {
  string atespace = 1;                  // identity retained by ateom from
  string actor_name = 2;                // the original RunWorkloadRequest
  string actor_uid = 3;
  string actor_template_namespace = 4;
  string actor_template_name = 5;

  string sandbox_class = 6;             // "gvisor" | "microvm"
  string source = 7;                    // "cgroup" | "guest-agent"

  uint64 memory_current_bytes = 8;
  uint64 memory_peak_bytes = 9;
  uint64 memory_working_set_bytes = 10;
  uint64 cpu_usage_usec = 11;           // cumulative within the current epoch
  int64  observed_at_unix_nano = 12;
}

source exists so consumers do not compare cgroup readings against guest readings as though they were measured the same way. ateom-gvisor needs to retain the identity fields (it is currently stateless between RPCs); ateom-microvm already keeps them in running.

When atelet reads

Polling alone does not handle the actor lifecycle, so the design does not rely on it. atelet already drives Run, Restore, and Checkpoint, so it reads at those boundaries in addition to the timer:

Trigger	Event outcome	Also does
Run or Restore	running	Sets the CPU baseline for the epoch
Timer, every 60s	running	Updates gauges and the CPU counter
Checkpoint	checkpointed	Records both histogram samples
Read fails, worker restarted	oom_killed	Records histograms from last known values

An actor living less than one interval still contributes one histogram sample and its full CPU total. Restore is a new epoch — the sandbox is recreated so cumulative CPU restarts at zero, and atelet treats the decrease as a reset rather than emitting a negative delta.

Metrics

Metric	Instrument	Unit	Labels	Recorded
ate.actor.memory.usage	Observable gauge	By	template ns, template name, sandbox class	Each read
ate.actor.cpu.usage	Observable gauge	{cpu}	template ns, template name, sandbox class	Each read
ate.actor.cpu.time	Counter	s	template ns, template name, sandbox class	Each read
ate.actor.memory.peak	Histogram	By	template ns, template name, sandbox class, outcome	Once at teardown
ate.actor.cpu.time.total	Histogram	s	template ns, template name, sandbox class, outcome	Once at teardown

Series per node is namespaces × templates × 2, independent of how many actors are running. ate.workerpool.name is omitted because atelet does not know it today (see open questions).

Krisztian F (@krisztianfekete) — agreed on the two histograms, with one adjustment: they should record one value per actor at teardown, not one per sample. A histogram fed from 60-second samples measures a distribution over sampling ticks, which weights long-lived actors more heavily and makes the percentiles unusable. Recorded once per actor lifetime, p99 is directly the number to put in the template's resource request.

The outcome label on the histograms matters for the same reason. Actors killed at the container limit have a truncated peak that would drag the percentile down and cause under-provisioning; keeping them in a separate series lets you exclude them from sizing queries and count them separately as a health signal.

Two other correctness constraints:

• ate.actor.cpu.time is incremented by per-actor deltas, never set from the raw cgroup value. usage_usec is monotonic within one cgroup, but the population of cgroups behind a template label churns constantly, so summing raw values is non-monotonic and breaks rate(). Because atelet is the stable emitter, churn only changes the increment size.
• Gauges are observable, not synchronous, so a template with no running actors produces an absent series rather than a stale value.

Events

One per actor per read: the five identity fields, sandbox_class, source, the four measurements, and outcome. This label set already exists in internal/actorlog, so the usage event is a direct sibling of EmitLifecycleLog and can ship on the existing stdout path with no new dependencies. Once a logs bridge exists in serverboot (there is no LoggerProvider today, only InitTracing and InitMetrics), the same records become native OTLP LogRecords with shared Resource attributes and automatic trace correlation. The content does not change, only the transport.

The periodic stream is what makes the events useful. A single teardown event says an actor peaked at 3 GB; the stream says when, which is what you need to correlate a spike against a request or a neighbouring actor.

OOM handling, and why periodic reads are mandatory

This is the first row of the issue's table and the case most likely to be missed by a teardown-only design.

There is no per-actor memory limit in Substrate today — ActorTemplate has no Resources field, WorkloadSpec.Container carries none, and cmd/atelet/oci.go:270-289 writes no Linux.Resources. The only binding limit is WorkerPool.spec.template.resources.limits.memory on the ateom container, and post-#496 the sentry nests inside that scope.

Kubernetes defaults SingleProcessOOMKill to false on cgroup v2 (kubelet.go:781-782), so memory.oom.group=1 applies and the kernel kills the entire subtree atomically: ateom daemon, sentry, every gofer. No teardown, no checkpoint, no surviving process in the worker to report anything. atelet is the only component that still holds the numbers, which is why the terminal event is emitted there.

Reporting memory.peak rather than only memory.current is what makes those last-known values accurate, since the kernel maintains it as a high-water mark across the whole interval.

Adjacent finding, probably its own issue: gVisor clamps --total-memory to s.CgroupJSON.Cgroup.MemoryLimit() (sandbox.go:1333-1338), but that reads the pause cgroup, where atelet wrote no limit. The clamp never engages, so the sentry advertises roughly host RAM to the guest and the application allocates past the pod limit with no backpressure and no guest-side OOM. Setting Linux.Resources.Memory on the pause container's spec would make the limit bind on pause/ and kill only the sandbox, leaving ateom alive to report the death — but that needs a Resources field on ActorTemplate and a carrier in WorkloadSpec.

Phases

Phase	Scope	Unblocks
0	Stats method: proto, both reads, identity retention in ateom	Independently testable, nothing user-facing
1	atelet reader, live-actor registry, events, OOM terminal path	OOM diagnosis, cost attribution, expensive-actor identification — three of the four groups in the issue table, at zero metric cardinality cost
2	Two gauges and the counter	Density validation (#44), autoscaling (#198), inactivity suspend (#424)
3	Teardown histograms; per-container detail via sentry and guest agent	Resource sizing, multi-container attribution

Open questions

1. Registry durability. atelet's live-actor list is in memory, so an atelet restart loses it and it recovers only as actors cycle. The alternative is listing worker pods on the node through the API server — more robust, but adds a Kubernetes client dependency atelet does not currently have.
2. ate.workerpool.name as a label. Add it to RunWorkloadRequest, or omit it in v1? Including it multiplies series count by the number of pools.
3. Read interval. 60s default. At 5,000 concurrent actors that is roughly 7M events per day. Worth suppressing unchanged readings, or leave it flat initially?
4. Event transport. Ship on the actorlog stdout path now, or wait for the logs bridge?

Happy to split phases 0–3 into tracking issues if that is easier to review.

[krisztianfekete]

ateom currently calls InitTracing but worker pods have no OTEL_EXPORTER_OTLP_ENDPOINT (workerpool_apply.go:38-44 sets only POD_UID, and applyWorkerPoolPodTemplate has no env path), so those spans are going to localhost:4317 and being dropped. That is a live bug independent of this issue.

This PR fixes that as well: #562, can you please review it?

[krisztianfekete]

Agree with the histogram adjustments as well 👍

As per the open questions:

1. Registry durability. atelet's live-actor list is in memory, so an atelet restart loses it and it recovers only as actors cycle. The alternative is listing worker pods on the node through the API server — more robust, but adds a Kubernetes client dependency atelet does not currently have.

In-memory is probably fine for the first milestone? I'd not add Kubernetes client dependency right now. cc. Eitan Yarmush (@EItanya), wdyt?

2. ate.workerpool.name as a label. Add it to RunWorkloadRequest, or omit it in v1? Including it multiplies series count by the number of pools.

I'd include it, it's already a low cardinality dimension we have, e.g. ate.workerpool.workers{pool}. I'd rather have it early then add it later.

3. Read interval. 60s default. At 5,000 concurrent actors that is roughly 7M events per day. Worth suppressing unchanged readings, or leave it flat initially?

Sampling should be a collector/pipeline concern, so let's keep the components as simple as possible from this point of view. E.g. ObservableGauges are missing when idle anyway.

4. Event transport. Ship on the actorlog stdout path now, or wait for the logs bridge?

I think actorlog on stdout is fine for now. The content should be identical and it's easy to onboard to the bridge once it's available.