Feature request: Implement 'kubectl ate top workers' command for real-time worker pod resource utilization

[baizhenyu]

🚀 Feature Description

Platform operators and SREs running Substrate workloads need real-time visibility into WorkerPool resource utilization to determine if worker pools are CPU-throttled, memory-pressured, or in need of autoscaling (WorkerPool.spec.replicas).

While kubectl ate get workers lists worker pods and their assigned actors, there is currently no CLI top command to view live hardware utilization (CPU and Memory) across physical worker pods.

This issue proposes adding kubectl ate top workers to kubectl-ate. This command can be implemented with zero backend daemon changes by combining Substrate's existing ateapi.ListWorkers() RPC with the standard Kubernetes Metrics API (metrics.k8s.io/v1beta1).

---

📐 Proposed CLI UX & Command Format

kubectl ate top workers [flags]

Supported Flags

• -n, --namespace <ns>: Scope output to a specific Kubernetes namespace.
• -a, --atespace <space>: Filter worker pods hosting actors in a specific atespace.
• -l, --selector <labels>: Filter by worker pool labels (e.g. ate.dev/worker-pool=counter).
• -o, --output json|yaml: Output format option (defaults to formatted terminal table).

Example Terminal Output

$ kubectl ate top workers -n ate-demo-counter

NAME                             POOL      STATUS    ASSIGNED ACTOR   CPU(CORES)   MEMORY(WORKING SET)
counter-worker-pool-7b9f8-x123   counter   CLAIMED   my-counter-1     342m         412Mi
counter-worker-pool-7b9f8-y456   counter   IDLE      <none>           2m           64Mi
counter-worker-pool-7b9f8-z789   counter   CLAIMED   actor-test-99    1850m        1920Mi

---

🏗️ Architecture & Data Sources

The command executes entirely within the kubectl-ate CLI client by joining two existing APIs:

                               +----------------------------------------+
                               | kubectl ate top workers                |
                               +----------------------------------------+
                                   /                        \
                                  /                          \
 1. Query Worker List & Actor    /                            \ 2. Query Pod CPU/RAM Usage
    Assignments                 /                              \    via K8s Metrics API
                               v                                v
         +--------------------------+           +----------------------------------+
         | ate-api-server (gRPC)    |           | K8s Metrics Server               |
         | apiClient.ListWorkers()  |           | metrics.k8s.io/v1beta1           |
         +--------------------------+           +----------------------------------+
                                   \                            /
                                    \                          /
                                     v                        v
                               +----------------------------------------+
                               | Merge & Render Terminal Table          |
                               +----------------------------------------+

1. Substrate ListWorkers RPC (ateapi):
   Returns worker records containing WorkerNamespace, WorkerPod, WorkerPool, and current Assignment (actor.name and actor.atespace).

2. Kubernetes Metrics API (metrics.k8s.io/v1beta1):
   Uses standard k8s.io/metrics/pkg/client/clientset/versioned to fetch PodMetrics for each worker pod's ateom container.

3. Data Join:
   kubectl-ate matches PodMetrics to Worker records by (namespace, pod_name) and prints combined resource utilization.

---

📋 Implementation Checklist

• Command Structure (cmd/kubectl-ate/internal/cmd/)
  • Add top.go parent command: kubectl ate top.
  • Add top_workers.go subcommand: kubectl ate top workers.
• Metrics Client Integration
  • Add K8s metrics.k8s.io/v1beta1 clientset initialization using existing kubeconfig flags.
• Printer Implementation (cmd/kubectl-ate/internal/printer/)
  • Add PrintWorkerTopTable() and PrintWorkerTopJSON() methods.
• Error Handling & Fallbacks
  • Gracefully handle clusters where metrics-server is unavailable by displaying "metrics unavailable".
• Testing
  • Unit tests for data joining logic in top_workers_test.go.
  • CLI integration tests with mocked ListWorkers gRPC response and mocked PodMetricses client.

---

🔗 Related Issues & PRs

• PR feat(otel): add ate.workerpool.workers metric with e2e suite #492: feat(otel): add ate.workerpool.workers metric with e2e suite (Control Plane worker counts)
• cmd/kubectl-ate/internal/cmd/get_workers.go (Base ListWorkers implementation)