✨ feat: introduce NodeReadinessEvaluation (NRE) CRD and controller

[Karthik-K-N]

Description

Introduces the NodeReadinessEvaluation (NRE) custom resource and its dedicated reconciler. NRE provides a per-node, read-only mirror of the evaluated state of every applicable NodeReadinessRule, giving operators a single object to kubectl get nre <node-name> to see the full readiness picture of any node without having to cross-reference multiple rule statuses.

Discussion document: https://docs.google.com/document/d/1DOP1G6i__nQN8qbhlSvdswkUquMeSKQY4LU554AbcgM/edit?usp=sharing

Feature Flag

The controller is opt-in via: --enable-node-readiness-evaluation

Related Issue

Type of Change

Testing

Checklist

• make test passes
• make lint passes

Does this PR introduce a user-facing change?

Doc #(issue)

[netlify]

✅ Deploy Preview for node-readiness-controller canceled.

Name	Link
🔨 Latest commit	6c00c70
🔍 Latest deploy log	https://app.netlify.com/projects/node-readiness-controller/deploys/6a958bb0cb2fb80008bbceb3

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Karthik-K-N
Once this PR has been reviewed and has the lgtm label, please assign sergeykanzhelev for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ajaysundark]

@Karthik-K-N IIUC, this would trigger two parallel reconciliations for node updates (NodeReconciler and NREReconciler) and one is handling the output of the other. I wonder if a separate controller for NRE is a better fitting pattern for this or should be bridged into NodeReconciler's watch?

[Karthik-K-N]

yes I thought about that and I missed to point out why I chose this way, here are my thoughts

1. Initially tried combining both into the NodeReconciler, but then when I checked with best practices of writing controller, if we do so we will be merging two different operations into one, Node controller adding/removing taints,NRE just storing the result, One failure should not cause requeue for other and I think NodeReconcile should be as quick as possible as it affects the workloads.
2. Followed the existing pattern of having separate controllers like NRR, Node and NRE
3. Even if there is race and NRE reconciles first and later the Node, Since the NRE also watches for condition, taint and label, if something updated by the Node Rec then eventually it triggers the NRE. We will be eventually consistent.

These are the thoughts and I lean towards keeping them separate, but let me know what do you think.

[ajaysundark]

My concern with another reconciler is that we are reintroducing a shared (Node) state again between two controllers. looking at buildRuleEvaluation it is very similar to evaluateRuleForNode. reevaluating the node status between two controllers with caches seem like a bad idea (and that seems like repeating what we did and trying to change with rule.status). :/

I saw #343 emitting individual nodeStatusDelta in rule-reconciliation, which could make adding NRE even cheaper than before. Happy to discuss more / hear your thoughts further on this.

[DsThakurRawat]

i tested this branch and found two behaviours worth knowing about, plus one interaction with #315. all three are reproduced with runnable tests, happy to share them.

first, a rule-creation window that never heals. the NRE fan-out fires on the rule Create event (GenerationChangedPredicate only filters updates), but the shared ruleCache is only populated on the rule controller's second reconcile pass, because the first one adds the finalizer and returns RequeueAfter one second. so the NRE reconciles for every matching node run against a cache that cannot contain the new rule yet. for rules whose evaluation then changes a node, the taint write re-triggers the node watch and everything heals. but for a rule whose conditions are already satisfied everywhere, nothing mutates any node, the rule's status patch doesn't bump generation, heartbeats don't pass the node predicate (it compares only condition type to status), and informer resyncs are filtered as no-ops. the NRE just permanently misses the rule until some unrelated node change. deletion is fine, reconcileDelete empties the cache before dropping the finalizer. i think this is the concrete version of the question ajaysundark raised above: the reconciler's correctness depends on another controller's queue having run first. listing rules from the informer inside Reconcile instead of reading the private cache would remove the ordering dependency entirely.

second, this branch merges cleanly with #315, and the two disagree once combined. evaluateRuleForNode there honours conditionPolicy anyOf, while buildRuleEvaluation here ANDs every condition unconditionally. an anyOf rule with one of two conditions satisfied ends up enforced as satisfied (no taint) while the NRE for the same node reports it Unmatched. a shared evaluation helper honouring GetConditionPolicy() in both paths would keep them from drifting.

small one: nothing currently produces RuleStatusError, so the Errors count, State Pending, and the Evaluated False condition are unreachable in this iteration. fine if that's intentional groundwork, might deserve a TODO.

[ajaysundark]

@Karthik-K-N is this ready for review?

[Karthik-K-N]

@Karthik-K-N is this ready for review?

yes , its ready for review

[DsThakurRawat]

The rule-creation window I flagged here on Aug 11 has a second victim now, and this one is worse because it's the reporting surface. Walked the sequence on your head 7c0ecc42: a rule Create passes GenerationChangedPredicate into this controller too, ruleToNodeRequests enqueues the matching nodes immediately, but the NRE reconciler reads getApplicableRulesForNode, which serves only the private ruleCache, and the cache isn't populated until the rule controller's second reconcile, because the first one just adds the finalizer and returns RequeueAfter one second (nodereadinessrule_controller.go:118-125, cache write at :139). So for a rule whose conditions are already satisfied on every matching node and whose taint is absent everywhere, the NRE usually reconciles against the empty cache and writes state: Ready with zero rule entries. Nothing corrects it after that: evaluation mutates no node, so no node event fires, and resync updates die on your equality predicates in the For(). Until something else happens to touch that node, the NRE keeps telling any reader (#327's history story, dashboards) that no rule applies to it while one actively does.

The Aug 11 fix suggestion covers both consumers, and it got cheaper to justify: have this controller read rules through the informer instead of the private cache, and the ordering dependency disappears rather than getting patched around.

Separately, I reproduced prow's lint failure locally so it can be named precisely: gocritic ifElseChain on the four-way taint transition at nodereadinessevaluation_controller.go:290, and unparam's unused ctx in the test's sharedSetup at :91. Both trivial.

What I checked that holds: rule deletions do reach this controller (in controller-runtime v0.24.1 a nil DeleteFunc defaults to passing, so GenerationChangedPredicate only filters updates), and your D1 shows the prune works once a reconcile runs; the CEL immutability on nodeSelector makes mapping by selector safe across rule updates; node deletion is covered by the ownerReference; and the single-writer separation is real, this reconciler writes nothing but its own status. One RBAC nit: the marker asks delete on nodereadinessevaluations but nothing in the code ever calls it, GC does that work.

[ajaysundark]

The Matched / Unmatched rules on status is bit unclear to me. Sorry I missed to get this at the doc, will look into it again.

[Karthik-K-N]

So the RuleStatus field helps to understand "whether node satisfy every condition listed in the rule"
Matched - All rule.Spec.Conditions are satisfied - ideally no taint
Unmatched - Any condition is not satisfied - there is taint on node

Do you think we should use better field name to convey this?

Instead of Matched should I change that to RuleStatusSatisfied means the rule is satisfied by the node?

[ajaysundark]

does 'unmatched' (node-selector escaped?) add value to the per-node status? As a node-owner I would less-likely be interested in rules that are not affecting my node.

Imagine a large cluster with many different node-pools. Setting the status of a node in pool-A for other rules associated with every other pools may be rather confusing to the user.

[Karthik-K-N]

True, I think we can drop the UnmatchedRules field.

[ajaysundark]

I expect, adding another controller will also increase the API interaction volume to a larger extent at scale.

[Karthik-K-N]

I think if we can finalize on this aspect, then other things will be easy to tackle
Since we use controller-runtime and it uses a Shared Informer Cache, additional controller does not increase API server watch traffic or our memory footprint. but It does duplicate the reconcile queue (may be CPU usage) and small window of race, but it provides less blast radius and separate of concern, we have two controller doing different things.
Also I think we should consdier at what scale level we can expect this latency of having new controller

[ajaysundark]

hmm. I maybe wrong about the API scale. but my primary concern is that NRE suppose to be a 'record' of what the controller enforced at node but a second reconciler recomputes the decision by further observations. This could bring other race and complexity of state handling.

For example, how do we distinguish even between kubelet added it vs another reconciler added it, when NRE is computed separately in its reconciler?

blast radius and separate of concern

I understand your concern on the implementation risks injecting this detail throughout the controller code. I was imagining #343 could simplify it for the rule, and maybe eventually #389 when we would have to handle the node updates all at just one place.

[ajaysundark]

Thanks for looking into this @Karthik-K-N!

Left some comments, mostly on the need for separate controller loop for NRE. We could find sometime to align on the implementation plan to avoid adding to your work and delaying this.

[ajaysundark]

NRE reconciler reads getApplicableRulesForNode, which serves only the private ruleCache, and the cache isn't populated until the rule controller's second reconcile

Thanks for the callout @DsThakurRawat. agree, there are some risks with a managed cache and two controllers reconciling at two different snapshots of the cached data. we need to carefully handle the race-conditions.

[yindia]

@ajaysundark On the cache-ordering window: reading rules via an informer List in Reconcile, instead of the private ruleCache, closes it. The fan-out fires on the rule watch event, so the informer already has the new rule, whereas ruleCache isn't written until RuleReconciler's second pass. It also drops the ordering dependency, with no need to remove ruleCache itself. wdyt ?

[ajaysundark]

Is it possible to get Ready=False AND Evaluated=True on a NREStatus? I'm trying to see whether Evaluated is functionally identical signal as Ready or how it should be interpreted.

[ajaysundark]

Do I understand your comments correctly: you think Evaluated=False suggests 'errors' processing this node with NRC, whereas Ready=False means Taint is present (pending readiness on this node). So a normal user would subscribe to "kubectl wait --for=condition=Ready".

who / how do you see Evaluated should be consumed? and could you also clarify what 'errors' do you see this to be. is it like permission (for eg: to remove taint) or something else?

[Karthik-K-N]

Yes Correct, What I thought is, when the Ready/Available is False, User can check the status of Evaluated

1. when Its True that means the Rule is successfully evaluated but its not satisfied on the Node.
2. when its False it can be Inprogress means the Rule is currently being reconciled
3. when its False it can be error means, there can be any intermittent error like api error or selector error

[ajaysundark]

is this needed in addition to Ready=True/False/Unknown to show combinational status (compose all rule outcomes)? your comments seem to map to True, False & Unknown respectively.

[Karthik-K-N]

It feels redundant but having it will enhance the user experience

1. Helps in printing the result when queried the NRE using kubectl as we can't easily print the conditions

kubectl get nre
NAME             NODE             STATE       AGE
nre-node-worker1 node-worker1     Available   5m
nre-node-worker2 node-worker2     NotAvailable 2m

2. Currently its being markes as selectable field so we can do something like this

kubectl get nre --field-selector status.state=NotAvailable

Apart from this there is no other intention we should be good to drop as well.

[ajaysundark]

I understand why you want this top level summary object here.

Maybe I am being extra careful since we are now moving out of a collapsing entity to isolated objects -- do we want this summary at the top, or is this something we could plan later? since we anticipate a handful or <10 rule matchings per node, I wonder if we could get away with having only the Conditions and RuleEvaluation[] array here?

[Karthik-K-N]

Sure, I will update, Even this was added to provide the better UX for user, may be we can consider it later if needed, I will remove it for now.

[ajaysundark]

super nit: could we keep RuleName and RuleUID directly here? Since these are keys to these objects, avoiding an extra deref would be more convenient for the consumers.

[ajaysundark]

NA - I'm not suggesting to change this. but I'll be worried about a node having to deal with 100 taints!~ .

[Karthik-K-N]

True that, do you think as a controller we should provide some recommendation? or controller optimal performance when the Node has X taints?

[ajaysundark]

listType=map with Key=ruleName?

[ajaysundark]

we would need this for multiple rule-workers working in parallel. and ref is not compatible with map, so it'd be good to consider flattening the rule references here.

[ajaysundark]

Trying to compare with the states we introduced here: #344.

We added new state conventions for nodes: 'held' vs 'released' - but I think we saw them as "node's" state per rule. these are rule's state so "satisfied" makes sense. is that how you're thinking?

[Karthik-K-N]

Yes, Correct, Its the Rule's State

[ajaysundark]

how will unparseable NodeSelector error is attributable to any specific node? I think rule-configuration error on the node's evaluation error is confusing.

I could imagine a more generic 'Blocked' for RuleStatus on the node - but then the question about - when do we decide unsatisfied vs blocked comes up as we do not have any 'time' budget on the rule-reconciliation.

[Karthik-K-N]

The Error is not attributed to Node rather the particular Rule of the Node. But yes having unparseble Nodeselector means it belong to all Nodes and which does not makes sense, I just removed this state. I think with this we no more need NodeEvaluationStatePending Node state as well, I just removed that.

[ajaysundark]

should we add the taintKey/effect here to help for self-reporting (instead of the active taints 'count' at the summary)?

I dunno, it would be redundant to rule.spec.taint, but for the headlamp kind of clients we are building, the CR may give a singular view of the readiness state on a 'node' level, without have to lookup again on the rule).

[Karthik-K-N]

We can add, its provides better clarity and better UX, avoid looks up

[ajaysundark]

is it intentionally 10K?

[ajaysundark]

I was surprised at this thinking why should they be this long, but looks like k8s api gives 32k for Message and 1024! for Reason as well: https://github.com/kubernetes/apimachinery/blob/50d9b4a672b474db2e0bf61c968d87465b55eb56/pkg/apis/meta/v1/types.go#L1702. so ours is fine!

[ajaysundark]

do we update this in continuous when Taint is reapplied?

[ajaysundark]

or should we split into FirstAppliedAt and CurrentTaintAppliedAt?

[Karthik-K-N]

so currently we support only one taint lifecycle Absent->Present cycle, do you expect to support Absent→Present→Absent→Present cycle as well, does it have any impact on telemetry?

[ajaysundark]

nit: is this for observing possible rule-reconciliation delay on the node? I think the other TaintAt times are also set at the first rule-reconciliation times. but I think I'm okay to keep this as it may bring some additional insights on rule's actions later.

[Karthik-K-N]

Yes, This is the time the NRC saw the rule. helps to see how long it took to evaluate the rule on tainted node

[ajaysundark]

This means we are updating all the matching nodes' RuleEvals, irrespective of whether they are satisfied.

Rule-reconciliations should be less frequent - only on rule modifications - so this seems fine, and 'freshness' details of evaluations maybe helpful.

[ajaysundark]

ah, now I see where do you see these enums for state.

could we use Status=(?Condition==Ready).status?

[Karthik-K-N]

Array filtering was not supported when I check last time, but I can see if there are any alternatives if we really want to drop the state field.

[AnuragThePathak]

Haven't got a chance to look into the controller code, which I will get back soon.

About the API, loved the recent update, to drop error and loading states (loading was unnecessary anyway for our controller as it's not time taking by nature). Error being displayed at the rule level solved a lot of problems.

[AnuragThePathak]

Considering the fact that spec is usually meant for defining the desired state of a CRD, may I know if you considered the trade-offs while putting NodeName as a spec?

My first thought was that we will be better off using 1:1 metadata.name for node and nre.

With on-going updates, the line maybe slightly misplaced, putting the context in the comment only

NodeName string `json:"nodeName,omitempty"`

[Karthik-K-N]

Yeah, The idea was to keep 1:1 mapping only and currently even the metadata.name is also set to nodename, We should be good to drop this field, but only thing is then the spec will become empty so I thought to keep the nodename as immutable field for better clarity.

[AnuragThePathak]

Yes that's also right, spec will be empty. But unless there's some guideline or practice in common projects not to do so, we should be good I guess.

[AnuragThePathak]

I would be against reusing the old ConditionEvaluationResult. When adding default status, I ran into an issue of not being able to properly provide visibility into what's the actual status and what's the status we are calculating after applying default status, because of the name 'CurrentStatus'.

Now that we have a chance to introduce new API, I think replacing 'CurrentStatus' with something like 'EffectiveStatus' and 'ObservedStatus' can improve the observability.

[Karthik-K-N]

I tend to agree with you, I think instead of currentState we should rename it to ObservedState and we can possibly add another field as EffectiveState which can help the user to understand the effective state which will be ObservedState if present or Default state if CurrentState not present
but
I think if we do this we should do it at both NRE and NRR level to avoid further confusion and keep them simple, Given that both are bit dependent or it may give two different results to user. I can take this up as a follow up task.

[AnuragThePathak]

Ah state suffix would be confusing I guess, status would be more consistent.

About the NRE and NRR, not quite sure about that. My personal preference would have been to do directly in NRR but obviously not possible due to schema constraints breaking existing schemas, so thought while introducing a new one, should be better doing it from day 1.

Anyway that's your call.