-
Notifications
You must be signed in to change notification settings - Fork 98
feat(blog): Add new blog featuring adoption of in-place Pod resource updates
#976
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
6239e0f
3b71e72
6588dd3
f5d4ada
3c85e3c
efc8ed4
87b0c8b
ab2ed11
eedb079
1f549b3
3a9fcf4
e5912f4
33e464b
9f65958
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| --- | ||
| title: Adoption of In-Place Pod Resource Updates in Gardener | ||
| linkTitle: Adoption of In-Place Pod Resource Updates in Gardener | ||
| newsSubtitle: May 19, 2026 | ||
| publishdate: 2026-05-19 | ||
| authors: | ||
| - avatar: https://avatars.githubusercontent.com/vitanovs | ||
| login: vitanovs | ||
| name: Stoyan Vitanov | ||
| aliases: [/blog/2026/05/19/in-place-pod-resource-updates] | ||
| tags: | ||
| - autoscaling | ||
| - vpa | ||
| --- | ||
|
|
||
| As Kubernetes workloads evolve, the need for dynamic resource adjustments becomes increasingly critical. Traditional `Pod` resource updates require `Pod` recreation, leading to increased operational costs and potential service disruptions. | ||
| To address this shortcoming, Kubernetes introduced a new way of updating `Pod` resources _in-place_, without the need to _evict_ workloads in order to apply the configuration change. With the [v1.27](https://kubernetes.io/releases/1.27/) release, a new `resize` subresource got introduced to the `Pod`'s API, acting as an interface to the underlying CRI implementations that manage the _cgroups_ settings. | ||
|
|
||
| ## What are the key benefits of _in-place_ Pod resource updates? | ||
|
|
||
| Having the ability to bypass the rollout process when updating `Pod` resources drastically improves the scaling efficiency. Eliminating the overhead of `Pod` scheduling and application initialization is among the primary benefits of the new _update_ mechanism. The following points summarize the key factors when considering using _in-place_ updates: | ||
|
|
||
| - __Zero-downtime scaling__: Resources are adjusted without `Pod` recreation or service interruption | ||
| - __Reduced scheduling overhead__: No need to re-schedule `Pod`s across the cluster | ||
| - __Reduced initialization overhead__: Applications __do not__ go through full initialization all over again | ||
| - __Preserved `Pod` identity__: `Pod` names, IPs, and volumes remain unchanged | ||
| - __Improved resource efficiency__: More granular and responsive resource optimization | ||
|
ialidzhikov marked this conversation as resolved.
|
||
|
|
||
| To read more about the feature, refer to the official [documentation](https://kubernetes.io/blog/2025/12/19/kubernetes-v1-35-in-place-pod-resize-ga/). | ||
|
|
||
| ## How does Gardener utilize the new _update_ mechanism? | ||
|
|
||
| Gardener relies heavily on the [Vertical Pod Autoscaler](https://github.com/kubernetes/autoscaler) to advance the resource usage optimization and keep the _out-of-the-box_ components running as efficiently as possible. With the [v1.4.0](https://github.com/kubernetes/autoscaler/releases/tag/vertical-pod-autoscaler-1.4.0) release, Vertical Pod Autoscaler introduced the ability to configure a new `.spec.updatePolicy.updateMode` for the `VerticalPodAutoscaler` resources: | ||
|
|
||
| ```yaml | ||
| apiVersion: autoscaling.k8s.io/v1 | ||
| kind: VerticalPodAutoscaler | ||
| metadata: | ||
| name: demo-vpa | ||
| spec: | ||
| targetRef: | ||
| apiVersion: "apps/v1" | ||
| kind: Deployment | ||
| name: demo | ||
| updatePolicy: | ||
| updateMode: "InPlaceOrRecreate" | ||
| ``` | ||
|
|
||
| effectively performing _in-place_ resource updates and _falling back_ to _eviction_ in case of failure. Adopting this release created the opportunity to leverage the new _update_ mechanism, and with Gardener [v1.137](https://github.com/gardener/gardener/releases/tag/v1.137.0), a new automatic migration mechanism was introduced. | ||
|
|
||
| Historically, `VerticalPodAutoscaler` resources, created for the different Gardener components, were configured to use _update mode_ `Auto` as it was the default option that mimics the behavior of `Recreate` - _evicting_ `Pod`s to apply newly [calculated resource recommendations](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/components.md#implementation-of-the-recommender). The subsequent Vertical Pod Autoscaler [v1.5.0](https://github.com/kubernetes/autoscaler/releases/tag/vertical-pod-autoscaler-1.5.0) release deprecated the `Auto` _update mode_, leaving only two viable options for continuous scaling: `Recreate` and `InPlaceOrRecreate`. The following graph illustrates the _flow_ of `Pod` resource updates used in Gardener: | ||
|
|
||
| ```mermaid | ||
|
vitanovs marked this conversation as resolved.
|
||
| graph LR | ||
| A[VPA Recommender] -->|new recommendation| B[VPA Updater] | ||
| B -->|PATCH /resize subresource| C[API Server] | ||
| C -->|sends update request| D[Kubelet] | ||
| D -->|sends update request| E[CRI Runtime] | ||
| E -->|updates cgroups| F[Running Container] | ||
| F -->|updates status| D | ||
| D -->|updates Pod status| C | ||
| ``` | ||
|
|
||
| The _key_ functionality behind this improved resource management is Linux [cgroups](https://www.man7.org/linux/man-pages/man7/cgroups.7.html), which are responsible for grouping processes and limiting the host resources they can utilize. | ||
|
|
||
| ### How are `VerticalPodAutoscaler` resources configured? | ||
|
|
||
| Like any cluster-wide configuration change, migrating the `VerticalPodAutoscaler` resources' _update mode_ presented a unique challenge that solved this by leveraging the [Gardener Resource Manager](https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md) and its extensible architecture. | ||
|
|
||
| We developed a dedicated `MutatingWebhook` that automatically filters relevant `VerticalPodAutoscaler` resources and applies the _update mode_ change, making the migration seamless. The webhook is deployed through the `VPAInPlaceUpdates` _feature gate_, available in both `gardenlet` and [Gardener Operator](https://github.com/gardener/gardener/blob/master/docs/concepts/operator.md). | ||
| In case the feature gate gets disabled, there is also a _rollback_ mechanism, present in both components. During the initialization of either the `gardenlet` or the Gardener operator, the _update mode_ gets automatically reverted. These changes have been available since Gardener [v1.137](https://github.com/gardener/gardener/releases/tag/v1.137.0), making _in-place_ update mode adoption fully operational and ready for production use. | ||
|
|
||
| For detailed information on [usage](https://github.com/gardener/gardener/blob/master/docs/usage/autoscaling/in-place-resource-updates.md) and [enablement](https://github.com/gardener/gardener/blob/master/docs/operations/enabling-in-place-resource-updates.md), refer to the official documentation. | ||
|
|
||
| ## How does Gardener benefit from the _in-place_ Pod resource updates adoption ? | ||
|
|
||
| Apart from the above-mentioned advantages of using _in-place_ resource updates, adopting the mechanism allows mitigation of a few shortcomings that have the potential to cause | ||
| severe troubles. | ||
| Since Gardener offers the ability to configure _non-HA_ `Shoot` clusters, their `etcd` data stores become vulnerable to restarts. Having a `VerticalPodAutoscaler` resource with an | ||
| _update mode_ `Recreate`, referencing a _single-node_ `etcd` store, causes a short control plane downtime each time the recommender decides to apply new resource values. | ||
| An identical situation can be witnessed in the Gardener _monitoring_ and _logging_ stacks, where [Prometheus](https://prometheus.io) and [Vali](https://github.com/credativ/vali) run as _single-instances_, facing the _single-node_ `etcd` problem when _evicted_. | ||
|
|
||
| Another good example of a Kubernetes flaw that could have been resolved by using _in-place_ resources _update mode_ has to do with the `kube-scheduler` and its prior behavior of getting `Pod`s, with volumes, scheduled on `Node`s that have already reached their volume attachment limit. Documented in a dedicated Kubernetes [issue](https://github.com/kubernetes/kubernetes/issues/126921), this problem had appeared when the CSI `Node` plugin `Pod` gets restarted during _eviction_, exactly how VPA was configured to do at the time of reporting it. | ||
| During the gap between the deletion and the new `Pod` creation, the `CSINode` object temporarily has an empty drivers section. The `kube-scheduler`'s `NodeVolumeLimits` plugin treated this missing information as "no limit" and incorrectly scheduled volume-backed pods to fully-saturated `Node`s. | ||
|
|
||
| The impact of adopting _in-place_ resource updates is best illustrated by the numbers: enabling the feature on the `Seed` clusters and the runtime cluster results in roughly 98% of the `Pod` resource updates being applied _in-place_, drastically reducing the amount of disruptive `Pod` recreations across the Gardener landscape. The exact ratio may vary from one Seed or runtime cluster to another, depending on its configuration. | ||
|
|
||
| ## Monitoring | ||
|
|
||
| Performing configuration migrations can become an exhausting task without a convenient dashboard to evaluate the process state. For this reason, as part of the effort to support _in-place_ Pod resource updates, we introduced a brand new _dashboard_ for the `vpa-updater` component. | ||
|
|
||
|  | ||
|
|
||
| With sections covering `VerticalPodAutoscaler` resource overviews (segregated by _update mode_) and panels displaying success rates per resource, the new dashboard can be used for both monitoring and generating status reports on applied resource recommendations. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We could add a section with the achievements on our side. We together checked with you that in dev environments 90% of the Pod evictions were replaced with in-place updates.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I suggest adding a sentence about it. Concrete suggestion: diff --git a/website/blog/2026/05/05-19-in-place-pod-resource-updates.md b/website/blog/2026/05/05-19-in-place-pod-resource-updates.md
index 43d9e5b7..7dcde589 100644
--- a/website/blog/2026/05/05-19-in-place-pod-resource-updates.md
+++ b/website/blog/2026/05/05-19-in-place-pod-resource-updates.md
@@ -83,6 +83,8 @@ An identical situation can be witnessed in the Gardener _monitoring_ and _loggin
Another good example of a Kubernetes flaw that could have been resolved by using _in-place_ resources _update mode_ has to do with the `kube-scheduler` and its prior behavior of getting `Pod`s, with volumes, scheduled on `Node`s that have already reached their volume attachment limit. Documented in a dedicated Kubernetes [issue](https://github.com/kubernetes/kubernetes/issues/126921), this problem had appeared when the CSI `Node` plugin `Pod` gets restarted during _eviction_, exactly how VPA was configured to do at the time of reporting it.
During the gap between the deletion and the new `Pod` creation, the `CSINode` object temporarily has an empty drivers section. The `kube-scheduler`'s `NodeVolumeLimits` plugin treated this missing information as "no limit" and incorrectly scheduled volume-backed pods to fully-saturated `Node`s.
+The impact of adopting _in-place_ resource updates is best illustrated by the numbers: enabling the feature on the `Seed` clusters and the runtime cluster results in roughly 98% of the `Pod` resource updates being applied _in-place_, drastically reducing the amount of disruptive `Pod` recreations across the Gardener landscape. The exact ratio may vary from one Seed or runtime cluster to another, depending on its configuration.
+
## Monitoring
Performing configuration migrations can become an exhausting task without a convenient dashboard to evaluate the process state. For this reason, as part of the effort to support _in-place_ Pod resource updates, we introduced a brand new _dashboard_ for the `vpa-updater` component.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for the suggestion, pushed it with the latest commits. |
||
|
|
||
| ## References | ||
|
ialidzhikov marked this conversation as resolved.
|
||
|
|
||
| - [In-Place Pod Resize (GA in v1.35)](https://kubernetes.io/blog/2025/12/19/kubernetes-v1-35-in-place-pod-resize-ga/) | ||
| - [Resize Container Resources](https://kubernetes.io/docs/tasks/configure-pod-container/resize-container-resources/) | ||
| - [Pod v1 API Reference - resize subresource](https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#operations-pod-v1-resize) | ||
| - [VPA v1.4.0 Release Notes](https://github.com/kubernetes/autoscaler/releases/tag/vertical-pod-autoscaler-1.4.0) (introduces _update mode_ `InPlaceOrRecreate`) | ||
| - [VPA v1.5.0 Release Notes](https://github.com/kubernetes/autoscaler/releases/tag/vertical-pod-autoscaler-1.5.0) (deprecates _update mode_ `Auto`) | ||
| - [VPA In-Place Updates Documentation](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/features.md#in-place-updates-inplaceorrecreate) | ||
| - [Gardener Usage: In-Place Resource Updates](https://github.com/gardener/gardener/blob/master/docs/usage/autoscaling/in-place-resource-updates.md) | ||
| - [Gardener Operations: Enabling In-Place Resource Updates](https://github.com/gardener/gardener/blob/master/docs/operations/enabling-in-place-resource-updates.md) | ||
| - [Gardener Resource Manager](https://github.com/gardener/gardener/blob/master/docs/concepts/resource-manager.md) | ||
| - [Gardener v1.137](https://github.com/gardener/gardener/releases/tag/v1.137.0) | ||
| - [Linux cgroups v2](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html) | ||
| - [CRI (Container Runtime Interface) Specification](https://github.com/kubernetes/cri-api) | ||
Uh oh!
There was an error while loading. Please reload this page.