JEP-0013 Phase 3 - exporter MetricsStream by RoddieKieley · Pull Request #1062 · jumpstarter-dev/jumpstarter

RoddieKieley · 2026-09-02T22:56:12Z

Summary

JEP-0013 Phase 3 PR C**: the exporter MetricsStream spoke. After existing GetServiceEndpoints / PushLogs setup, the exporter opens a long-lived MetricsStream and answers scrapes with the same get_registry().generate_latest() bytes as local GET /metrics.

Depends on PR A: #1060

Independent of PR B (#1061) at the protocol layer. In-cluster reverse-scrape still needs B’s metrics port :8080. Do not merge until A is on main, then rebase. Related PRs D–E will be linked here as they are opened.

Does not replace the Phase 2 loopback :0 lab scrape.
Reconnect with backoff; local counters are not reset (JEP).
First stream message is MetricsRegister with identity=exporter_name (JWT exporter name), not Metadata.name.

Lab / #1058: #1058 stopped sending jumpstarter.dev/name on Register, so identity=self.name became "unknown" and the hub (A) correctly rejected the stream. This PR registers with self.exporter_name. _session_labels() still injects jumpstarter.dev/name only into local Session / /metrics so series are not exporter="unknown", without putting that label back on Register.

Lab / exemplars: after driver ops, generate_latest() includes OpenMetrics exemplars (# {lease_id=…}). The hub from A omits that snapshot and increments jumpstarter_metrics_parse_errors_total. This PR does not strip or re-encode exemplars.

DEMO

The lab walkthrough for the stacked Phase 3 work (A+B+C+D+E plus lab-only Route) is on the fork demo branch, not this PR:

DEMO.md notes that MetricsStream must register with identity=exporter_name after #1058. See MetricsStream during the lease for the DD-3 exemplar limitation (A), which is visible only after this spoke is connected.

How this PR fits the series

flowchart TB
  subgraph hub ["In-cluster — PRs A and B"]
    Tel["telemetry MetricsStream + GET /metrics :8080"]
  end
  subgraph edge ["This PR"]
    Exp["exporter MetricsStreamClient"]
    Local["local HTTP /metrics :0"]
  end
  Reg["Phase 2 MetricsRegistry"] --> Exp
  Reg --> Local
  Exp <--> Tel

flowchart TB
  A["PR A #1060: proto + hub"]
  B["PR B #1061: image + metrics port + scrape CR"]
  C["PR C this PR: exporter MetricsStream client"]
  D["PR D: Loki HTTP push"]
  E["PR E: jmp PushLogs"]
  A --> B
  A --> C
  B --> D
  D --> E

PR	Branch	Status
A	`jep-0013-phase3-metricsstream`	#1060
B	`jep-0013-phase3-operator-image`	#1061
C	`jep-0013-phase3-exporter-metricsstream`	This PR
D	`jep-0013-phase3-loki-push`	#1063
E	`jep-0013-phase3-client-pushlogs`	#1064

Unique work vs PR A: b84f9524 (client) + 841250fa (identity after #1058). Python-only besides the stubs already in A. If GitHub shows A's commits, that is stacking against main; review those two commits only.

Out of scope

MetricsStream protocol, parse-error omit, OpenMetrics exemplar decode (A / JEP-0013 Phase 3 - MetricsStream #1060)
Operator image / metrics port / scrape CR (B / JEP-0013 Phase 3 - operator image #1061)
Loki / PushLogs (D, E), including exporter TFTP errors that only show up on the log path
Advertising a Route/hostname for out-of-cluster exporters (lab-only on the demo branch)
Phase 4 driver telemetry API
#1059 (Session.exporter_name for local counters). Complementary to _session_labels() here; on rebase onto current main, keep both identity=exporter_name and pass exporter_name= into Session(...).

Add the MetricsStream protocol and Go hub so Prometheus can scrape merged exporter OpenMetrics from telemetry without an exporter client yet. Generated Python stubs are included for proto consistency. Co-authored-by: Cursor <cursoragent@cursor.com>

Stop silently dropping unparseable exporter snapshots. Log the exporter and error, and increment jumpstarter_metrics_parse_errors_total so reverse-scrape omissions are visible on the same /metrics response. Co-authored-by: Cursor <cursoragent@cursor.com>

…e 3) Open a long-lived MetricsStream after GetServiceEndpoints so the telemetry hub can pull generate_latest() snapshots without replacing local /metrics.

…1058 jumpstarter-dev#1058 stopped sending jumpstarter.dev/name on Register, so identity=self.name was "unknown" and the hub rejected the stream. Co-authored-by: Cursor <cursoragent@cursor.com>

coderabbitai · 2026-09-02T22:56:18Z

Warning

Review limit reached

Next included review available in 55 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: ab00dd94-2c32-42ab-8a49-a9026c023448

📥 Commits

Reviewing files that changed from the base of the PR and between f1b7f24 and 841250f.

⛔ Files ignored due to path filters (2)

controller/internal/protocol/jumpstarter/v1/telemetry.pb.go is excluded by !**/*.pb.go
controller/internal/protocol/jumpstarter/v1/telemetry_grpc.pb.go is excluded by !**/*.pb.go

📒 Files selected for processing (18)

controller/cmd/telemetry/main.go
controller/internal/service/metrics_merge.go
controller/internal/service/metrics_merge_test.go
controller/internal/service/metrics_stream.go
controller/internal/service/metrics_stream_test.go
controller/internal/service/telemetry_http.go
controller/internal/service/telemetry_identity.go
controller/internal/service/telemetry_service.go
protocol/proto/jumpstarter/v1/telemetry.proto
python/packages/jumpstarter-protocol/jumpstarter_protocol/jumpstarter/v1/telemetry_pb2.py
python/packages/jumpstarter-protocol/jumpstarter_protocol/jumpstarter/v1/telemetry_pb2.pyi
python/packages/jumpstarter-protocol/jumpstarter_protocol/jumpstarter/v1/telemetry_pb2_grpc.py
python/packages/jumpstarter-protocol/jumpstarter_protocol/jumpstarter/v1/telemetry_pb2_grpc.pyi
python/packages/jumpstarter/jumpstarter/exporter/exporter.py
python/packages/jumpstarter/jumpstarter/exporter/exporter_telemetry_test.py
python/packages/jumpstarter/jumpstarter/exporter/exporter_test.py
python/packages/jumpstarter/jumpstarter/exporter/metrics_stream.py
python/packages/jumpstarter/jumpstarter/exporter/metrics_stream_test.py

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

bennyz · 2026-09-03T05:20:01Z

                if await self._apply_status(message, tg):
                    break

+    def _session_labels(self) -> dict[str, str]:


is this still necessary after #1059?

@RoddieKieley questions to @cursoragent:

Questions to answer:

Is fix: add missing exporter name in session #1059 included in our jep-0013-phase3-exporter-metricsstream branch for pull request JEP-0013 Phase 3 - exporter MetricsStream #1062?

If it is not, does fix: add missing exporter name in session #1059 impact our implementation and thus are there required changes once we rebase on the latest main which contains fix: add missing exporter name in session #1059?

Analysis from @cursoragent:
tl;dr
#1059 is not in this branch (we’re still on #1058). _session_labels was a local workaround so Session.name wasn’t "unknown" after Register stopped sending jumpstarter.dev/name. After we rebase onto main, Session uses exporter_name= from #1059, so this helper isn’t needed for metrics/logs. We’ll drop it and keep identity=exporter_name for MetricsStream, which #1059 doesn’t cover.

In detail:

Details

#1059 is not on jep-0013-phase3-exporter-metricsstream. This PR still sits on #1058 (600a2fd2). Current main is #1059 (f1b7f240).

_session_labels() copies labels: dict[str, str] and injects jumpstarter.dev/name for local Session only, not Register.

This PR (C) #1059 (on main)

Problem after #1058 Metadata.name is "unknown" (labels.get("jumpstarter.dev/name", "unknown")) Same "unknown" on Session metrics/logs

MetricsStream Register identity=self.exporter_name (hub rejects "unknown") Not touched

Local active_sessions / log exporter= Inject name into Session labels so self.name works New Session.exporter_name, passed as exporter_name=self.exporter_name; Session uses that field, not self.name

They overlap on “don’t show exporter=unknown locally.” They do not overlap on MetricsStream identity.

On rebase onto main:

Keep identity=self.exporter_name. fix: add missing exporter name in session #1059 does not fix Register. Without this, MetricsStream still fails the hub identity check.

Keep fix: add missing exporter name in session #1059’s exporter_name=self.exporter_name on every Session(...). This branch’s Session sites are labels=self._session_labels() and do not pass exporter_name=. After rebase, session.py uses self.exporter_name (default "unknown"). If we keep _session_labels and drop exporter_name=, local metrics go back to "unknown".

_session_labels is not required for the fix: add missing exporter name in session #1059 metrics/log fix. After rebase, Session no longer reads self.name for adjust_active_sessions / set_log_context.

Residual effect if we keep _session_labels: GetReport still returns session.labels, so clients would see jumpstarter.dev/name on the lease session without putting that label back on Register. #1059 does not restore that. Optional, not needed for MetricsStream.

Plan: drop _session_labels and use labels=self.labels plus exporter_name=self.exporter_name, matching #1059. Keep identity=self.exporter_name. Do not put jumpstarter.dev/name on Register.

RoddieKieley and others added 4 commits September 2, 2026 10:39

feat: exporter MetricsStream client for reverse-scrape (JEP-0013 Phas…

b84f952

…e 3) Open a long-lived MetricsStream after GetServiceEndpoints so the telemetry hub can pull generate_latest() snapshots without replacing local /metrics.

fix: register MetricsStream with exporter_name after jumpstarter-dev#…

841250f

…1058 jumpstarter-dev#1058 stopped sending jumpstarter.dev/name on Register, so identity=self.name was "unknown" and the hub rejected the stream. Co-authored-by: Cursor <cursoragent@cursor.com>

This was referenced Sep 2, 2026

JEP-0013 Phase 3 - MetricsStream #1060

Open

JEP-0013 Phase 3 - loki push #1063

Open

JEP-0013 - Phase 3 - client PushLogs #1064

Open

RoddieKieley marked this pull request as ready for review September 2, 2026 23:20

RoddieKieley mentioned this pull request Sep 2, 2026

JEP-0013 Phase 3 - operator image #1061

Open

bennyz reviewed Sep 3, 2026

View reviewed changes

Navigation Menu

Uh oh!

JEP-0013 Phase 3 - exporter MetricsStream - #1062

JEP-0013 Phase 3 - exporter MetricsStream#1062
RoddieKieley wants to merge 4 commits into
jumpstarter-dev:mainfrom
RoddieKieley:jep-0013-phase3-exporter-metricsstream

RoddieKieley commented Sep 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Sep 2, 2026 •

edited

Loading

Review limit reached

Uh oh!

bennyz Sep 3, 2026

Uh oh!

RoddieKieley Sep 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	This PR (C)	#1059 (on `main`)
Problem after #1058	`Metadata.name` is `"unknown"` (`labels.get("jumpstarter.dev/name", "unknown")`)	Same `"unknown"` on Session metrics/logs
MetricsStream Register	`identity=self.exporter_name` (hub rejects `"unknown"`)	Not touched
Local `active_sessions` / log `exporter=`	Inject name into Session labels so `self.name` works	New `Session.exporter_name`, passed as `exporter_name=self.exporter_name`; Session uses that field, not `self.name`

Uh oh!

Conversation

RoddieKieley commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

DEMO

How this PR fits the series

Out of scope

Uh oh!

coderabbitai Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

bennyz Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

RoddieKieley Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

RoddieKieley commented Sep 2, 2026 •

edited

Loading

coderabbitai Bot commented Sep 2, 2026 •

edited

Loading