Filter kube informers server-side to operator-generated objects

[gregakinman]

Problem

The kube informer factory watches Pods, Services, EndpointSlices, ConfigMaps and StatefulSets across the whole cluster whenever watchNamespaces is anything other than a single literal namespace name — GetInformerNamespace() returns meta.NamespaceAll in every other case, including regexp patterns.

Filtering then happens client-side in isTrackedObject(), i.e. after every object in the cluster has already been transferred, decoded and queued in the DeltaFIFO. Informer memory therefore scales with total cluster size rather than with the number of managed installations, which OOM-kills the operator on large clusters with many namespaces.

Change

Push the same filter server-side with kubeinformers.WithTweakListOptions, setting opts.LabelSelector to the chop-generated-object label.

The selector is built through the same labeler that IsCHOPGeneratedObject() resolves through, rather than from the raw LabelAppName constant — that constant is the literal placeholder "APIGroupName" + "/" + "app" and is substituted at runtime, so building the selector any other way would let it drift from the client-side predicate. Resolved, it is clickhouse.altinity.com/app=chop.

CHOp custom resource informers (CHI/CHIT/CHOpConfig) are deliberately left unfiltered — they are cluster-wide by necessity and small.

24 added lines in one file, plus tests. Nothing is removed or renamed, and no default changes.

Why this is behavior-preserving

• Every handler registered on kubeInformerFactory already gates on isTrackedObject(), which requires IsCHOPGeneratedObject(). No handler regresses.
• Nothing reads these caches expecting objects the operator did not generate: there are no Lister() or GetIndexer() reads against the kube informers anywhere in the tree, so they function purely as event sources.
• The metrics exporter reads through direct API calls, not the informer cache.
• The CHK watcher builds its own dynamic informer, independent of this factory.

Two cases worth a reviewer's attention:

EndpointSlices are populated by Kubernetes, not by the operator. They inherit the label from the chop-created Service, since the endpointslice controller mirrors Service labels onto slices except for reserved prefixes. Verified on two live clusters — 179/179 and 19/19 chop-labelled Services had a correspondingly labelled EndpointSlice, zero misses. The legacy Endpoints informer is never constructed (addEventHandlersEndpoints is commented out at the registration site), so it is unaffected either way.

CHI and CHK use different label keys. clickhouse.altinity.com/app versus clickhouse-keeper.altinity.com/app, and a label selector cannot express a disjunction over two keys. This is fine because isTrackedObject() uses the CHI labeler exclusively, so CHK-generated objects were already dropped client-side; the test table pins that case so it cannot silently regress.

One pre-existing sharp edge this does not change: CR-provided labels are merged over generated ones, so a CHI that sets clickhouse.altinity.com/app to another value strips the label from its own generated objects. That already defeated isTrackedObject() identically, so the net behavior is the same — no reconcile either way.

Testing

• go build ./... clean.
• go vet produces an identical finding set before and after, compared in the same tree with the same build cache — no new findings.
• New tests pin the literal resolved selector and assert it agrees with IsCHOPGeneratedObject() across a table of label sets, including the wrong-value, bare-app, and keeper-labelled cases.
• Existing tests pass. Note go test currently needs -vet=off on master due to pre-existing vet failures, and TestEnforceVerifiedLegacyTLS fails on macOS independently of this change.
• Operator image builds and starts.

On one production-shaped cluster with a regexp watch config, cached objects across the five watched types drop from 22,194 to 761.

🤖 Generated with Claude Code