Skip to content

Add max-connection-age support for Triple server (connection rotation for load balancing) - #16454

Open
digyear wants to merge 2 commits into
apache:3.3from
digyear:feat/triple-max-connection-age
Open

digyear wants to merge 2 commits into
apache:3.3from
digyear:feat/triple-max-connection-age

Conversation

@digyear

@digyear digyear commented Sep 6, 2026

Copy link
Copy Markdown

What is the purpose of the change

Add server-side max-connection-age support for the Triple protocol, aligned with the standard gRPC capability (gRFC A9: Server-Side Connection Management).

HTTP/2 multiplexes all requests over a single long-lived connection, so a Triple consumer stays pinned to the same provider instance forever — even after new instances are scaled up or traffic should be rebalanced. gRPC solves this with maxConnectionAge / maxConnectionAgeGrace: when a connection exceeds the configured age, the server sends an advisory GOAWAY (NO_ERROR, last-stream-id = MAX_INT) so in-flight requests keep running while clients migrate to a new connection (and get redistributed by the load balancer), then closes the connection after the grace period.

Triple currently has no equivalent: there is no way for a Triple server to proactively rotate long-lived connections. This PR adds it, disabled by default.

Brief changelog

  • TripleConfig: add maxConnectionAge (ms, default -1 = disabled) and maxConnectionAgeGrace (ms, default 10000) with validation, following the existing field conventions (getXxxOrDefault()); matching TripleBuilder methods in dubbo-config-api.
  • TripleServerConnectionHandler: on channelActive, if maxConnectionAge > 0, schedule a one-shot task on the channel's EventLoop with ±10% jitter (same as grpc-java) to avoid mass simultaneous reconnections when many connections are established at the same time. On expiry:
    1. send the advisory GOAWAY via the existing GracefulShutdown.sendGoAwayFrame (reuses the production code path — in-flight streams are unaffected);
    2. after the grace period, initiate close through ctx.channel().close() so the request traverses this handler's close() override and runs the existing graceful-shutdown sequence (final GOAWAY + PING) instead of an abrupt disconnect;
    3. pending tasks are cancelled on channelInactive (no leaks when the client disconnects first).
  • TripleHttp2Protocol: pass the config into the handler on both server pipeline paths (direct h2 and h1-upgrade); raise the codec's gracefulShutdownTimeoutMillis to at least the configured grace so Netty's built-in backstop never fires before it.

Usage:

dubbo:
  protocol:
    name: tri
    triple:
      max-connection-age: 3600000      # rotate connections after ~1h
      max-connection-age-grace: 10000  # let in-flight requests finish

Verifying this change

New tests (all passing):

  • TripleServerConnectionHandlerTest — EmbeddedChannel with virtual clock: advisory GOAWAY (NO_ERROR + extraStreamIds=MAX_INT) fires within the jitter window and never before it; disabled by default (24h, no frames); tasks cancelled on channelInactive; graceful-shutdown sequence (final GOAWAY + PING) initiated after the grace period.
  • TripleServerConnectionHandlerConcurrencyTest — real NIO server/client, 100 connections sharing 2 EventLoop threads: concurrent rotation of all connections; client disconnects racing age expiry (random delays within ±20% of the window); server-side close() racing pending age tasks; 5 rounds of connection churn under ResourceLeakDetector.PARANOID with zero leak reports.
  • TripleConfigTest / TripleBuilderTest — defaults, validation and builder wiring.

Regression: full test suites of dubbo-rpc-triple (492), dubbo-common (1169) and dubbo-config-api (695) pass locally.

Note for reviewers: with the current consumer-side GOAWAY handling (#16344), each rotation costs one request failure window on the consumer, so this feature is best paired with the graceful consumer migration in #16345. This PR is independent and safe to merge on its own (default: disabled).

Does this pull request potentially affect one of the following parts

  • Dependencies (does it add or upgrade a dependency)
  • The public API (new optional TripleConfig fields; default values keep behavior unchanged)
  • The persistence format of the configurations
  • The default values of configurations (new fields only; existing defaults untouched)
  • The serialization protocol
  • The compatibility with previous versions

Documentation

  • Does this pull request introduce a new feature? yes
  • If yes, how is the feature documented? JavaDoc on the new TripleConfig fields (website docs can follow after merge)

Add server-side connection rotation for the Triple protocol, aligned
with the gRPC standard capability (see gRFC A9). Long-lived HTTP/2
connections pin clients to the same provider instance, preventing load
balancers from redistributing traffic after scale-up or deployment.

- TripleConfig: add maxConnectionAge (ms, -1 = disabled by default) and
  maxConnectionAgeGrace (ms, default 10s) with validation, plus the
  matching TripleBuilder methods.
- TripleServerConnectionHandler: on channelActive schedule a one-shot
  task when maxConnectionAge > 0, with +/-10% jitter (same as
  grpc-java) to avoid mass simultaneous reconnections. On expiry send
  an advisory GOAWAY (NO_ERROR, last-stream-id = MAX_INT) so in-flight
  requests keep running while clients migrate to a new connection;
  after the grace period initiate the existing graceful shutdown
  sequence (final GOAWAY + PING + close). Tasks are cancelled on
  channelInactive.
- TripleHttp2Protocol: pass the config into the handler on both the h2
  direct and h1-upgrade server pipeline paths; raise the codec
  gracefulShutdownTimeoutMillis so it never fires before the age grace
  period.

Backward compatible: the feature is disabled by default and no existing
behavior changes unless maxConnectionAge is configured.

Note: for zero request failures during rotation this pairs with the
consumer-side graceful GOAWAY migration in apache#16345.

Signed-off-by: Jamie Wilson <a2v1h4.76g1nr7j@gmail.com>
@codecov-commenter

codecov-commenter commented Sep 6, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.65079% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.94%. Comparing base (dab47b7) to head (d88fc01).

Files with missing lines Patch % Lines
...he/dubbo/rpc/protocol/tri/TripleHttp2Protocol.java 66.66% 2 Missing ⚠️
...l/tri/transport/TripleServerConnectionHandler.java 94.59% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##                3.3   #16454      +/-   ##
============================================
+ Coverage     60.91%   60.94%   +0.02%     
- Complexity       15    11768   +11753     
============================================
  Files          1953     1953              
  Lines         89271    89331      +60     
  Branches      13473    13479       +6     
============================================
+ Hits          54383    54440      +57     
+ Misses        29309    29306       -3     
- Partials       5579     5585       +6     
Flag Coverage Δ
integration-tests-java21 32.07% <19.04%> (-0.06%) ⬇️
integration-tests-java8 32.19% <19.04%> (-0.02%) ⬇️
samples-tests-java21 32.11% <3.17%> (-0.06%) ⬇️
samples-tests-java8 29.71% <3.17%> (-0.13%) ⬇️
unit-tests-java11 59.21% <93.65%> (+0.04%) ⬆️
unit-tests-java17 58.67% <93.65%> (+0.02%) ⬆️
unit-tests-java21 58.67% <93.65%> (+0.02%) ⬆️
unit-tests-java25 58.64% <93.65%> (+0.04%) ⬆️
unit-tests-java8 59.20% <93.65%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

- TripleServerConnectionHandlerGuardTest: cover the race-guard branches
  (channel dies between scheduling and task execution) via mocked context,
  unreachable through EmbeddedChannel lifecycle alone since channelInactive
  cancels the tasks first.
- TripleHttp2ProtocolPipelineTest: verify the server pipeline wires
  maxConnectionAge/maxConnectionAgeGrace from TripleConfig into
  TripleServerConnectionHandler (both enabled and disabled defaults).
- TripleConfigTest: cover the null-setter branch keeping defaults.
- Enable debug logging for TripleServerConnectionHandler in log4j2-test.xml
  so the grace-period debug branch is exercised.

Signed-off-by: Jamie Wilson <a2v1h4.76g1nr7j@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants