[Feature request] Topology Spread Constraints & Per-Deployment Affinity

[kevin-lindsay-1]

It's fairly normal for k8s deployments that live in a multi-zone cluster to have topology constraints or affinity rules, however it's fairly normal to have an affinity rule per deployment, as the label selectors for pods are typically different per-deployment.

Expected Behaviour

Be able to set anti-affinity and topology spread constraints per deployment.

Current Behaviour

Cannot set affinity per deployment, and topology spread constraints are not in the chart.

Are you a GitHub Sponsor (Yes/No?)

Check at: https://github.com/sponsors/openfaas

• Yes
• No
• No, but I sponsor Alex

List All Possible Solutions and Workarounds

Affinity is currently a global option for what appears to be all deployments. Per-deployment overrides is a reasonable option.

Topology Spread Constraints are not currently in the chart, and I don't think a global configuration option would really even work, as the labelSelector would need to be different per-deployment.

Which Solution Do You Recommend?

Make overrides for affinity and make topology spread configurable, not implementing a global variant.

Steps to Reproduce (for bugs)

Context

Trying to have HA for faas-netes/openfaas.

Your Environment

• FaaS-CLI version ( Full output from: faas-cli version ):
  0.13.13

• Docker version docker version (e.g. Docker 17.0.05 ):
  20.10.8

• What version and distriubtion of Kubernetes are you using? kubectl version
  server v1.21.3
  client v1.22.2

• Operating System and version (e.g. Linux, Windows, MacOS):
  MacOS

• Link to your project or a code example to reproduce issue:

• What network driver are you using and what CIDR? i.e. Weave net / Flannel

[alexellis]

Hi @kevin-lindsay-1

I think what we are looking for here is more description of the the actual behaviour vs. the desired behaviour. What is happening now in the cluster, why is that bad, and what would make that better?

Have you looked into Profiles for affinity yet? I didn't see you mention it in your issue.

Regards,

Alex

[kevin-lindsay-1]

Hey @alexellis

Right now I have no guarantee of regional, zonal, or nodal high availability on services. We currently use EKS with spot instances preferred, and part of how EKS works is that a node might be gracefully drained and cordoned at any time when the node's underlying spot instance is reclaimed.

As such, making sure multiple pods for each service exist and that they're not on the same node (and preferably not in the same zone) is a strong desire. We currently do this through Topology Spread Constraints.

Typically you need to set something like:

labelSelector:
  app: gateway

Per app, and with the current way affinity is exposed, I don't think I have enough expressivity to easily select OF components and specify their desired affinities, especially if they differ in some way now or down the road.

---

Have you looked into Profiles for affinity yet? I didn't see you mention it in your issue.

Interesting; I never really looked into profiles, as the CRD exposes up most of what I want, and since it's easy enough to write a helm chart with values per-environment, this really hasn't come up.

Now that I look at profiles, I guess I'm wondering why I'd use them instead of a CRD. One use case I can think of is reasonable defaults for a set of functions, but our helmfile setup already has defaults on top of a custom chart, which already has defaults. Don't really think a 3rd layer would be particularly advantageous here at this time, and would instead probably prefer what profiles exposes exposed in the CRD.

On that note, I'm starting to wonder if it would be easier for the CRD to simply expose a deployment spec or something, because manually exposing up every option that kubernetes exposes to a deployment seems like a never-ending chore.

[alexellis]

I don't follow the comparison of "CRD" vs Profiles.

You can use the OpenFaaS REST API or Function CRD to create your function, there's an annotation to tie that function to a Profile.

Profiles enable affinity/anti-affinity rules.

Clarify for me: Is this request for functions or the core services?

Do you also run the core services on spot instances?

[kevin-lindsay-1]

I don't follow the comparison of "CRD" vs Profiles.

To me, both just look like different ways to specify certain things in k8s config. I've only given a cursory look to profiles, but they look to me like a way to apply multiple sets of certain k8s config to multiple functions, similar to having multiple queue workers for different "types" of functions.

Personally, I don't really see the need for profiles as I understand them now, as if the currently exposed options:

• runtimeClassName
• tolerations
• affinity
• podSecurityContext

were just part of the deployment created when the CRD is applied, it doesn't seems like the profiles would add anything. Those are all just normal k8s config options, just like annotations on the CRD, so it's unclear to me what the separation of concerns is between these two.

That, and if, for example, there was some kind of overlap between what the CRD for the function would expose, and what the profiles would expose. For example, if the CRD let you specify a podSecurityContext, which would override which, the CRD or the profile?

Clarify for me: Is this request for functions or the core services?

This is for the core services, because currently we don't expect functions to be always present (we expect to use scale-to-zero as much as reasonably possible for a given workload), and as such are not particularly concerned with the spread of the pods at this time. It will probably come, and would definitely be better to have than not, but right now the focus is core services.

Do you also run the core services on spot instances?

Yes; with the exception of OpenFaaS, all of our current workloads are highly available and individual pods are "stateless", i.e. assuming that a pod handling a workload is not interrupted by a SIGKILL, everything has time to gracefully terminate within the eviction period given to us by EKS' spot instances.

The question/issue that I have is that in regards to queues in OpenFaaS, I've noticed in local testing that terminating certain pods seems to cause queues to be cleared while they should still have a workload. That's definitely not ideal, and I would like to know if that is expected given the current implementation/arch, and if so, whether or not persistent queues could be implemented. I've only recently begun to look into that, but I expect to create a separate issue for that once I do a little more digging.

Assuming that a queue is in-memory, and that termination of a specific core service, graceful or not, currently clears a queue, then perhaps a PVC or the like could be used to flush the contents of a queue during graceful termination, or some kind of "hand-off", where pod A (draining) waits for pod B (pending/running) to come online, and once available, pod A sends its queue to pod B.

[kevin-lindsay-1]

ftr I could probably pretty easily add these to the chart, as long I know whether or not we'd like to use global defaults or something.

[LucasRoesler]

just a small thing while i read and think about the the thread. The comment about the queue persistence probably deserves its own issue because it will require us to modify this

faas-netes/chart/openfaas/templates/nats-dep.yaml

Line 44 in 1b2546c

- memory

and the options are file and sql, per this doc https://docs.nats.io/nats-streaming-server/configuring/persistence but both of these options come with their issues that need to be carefully considered and we would need to provide a lot more documentation about error states

[kevin-lindsay-1]

Maybe a cluster would solve for both NATS high availability and this shared state? Assuming you could mount a file via a PVC and let RAFT take care of the rest, would allow for multiple NATS pods at once, and be persistent, killing two birds with one stone.

A SQL database running in k8s would probably have the same issue, meaning I'd either have to set up an external k8s database, or it would need some kind of k8s-level persistence, so probably just more config for the same end value.

Although, you could theoretically create a db in the OF chart, allowing you to turn it off and change the endpoint, port, and auth. That would basically give you in-memory storage by default (the db is lost upon termination), and persistence if you want it (the db is remote).

PVC and cluster sounds simpler, though.

[kevin-lindsay-1]

PVC and cluster sounds simpler, though.

That said, using it in AWS in a (potentially) multi-regional, multi-zonal, multi-nodal spread, EFS or FSx are probably the best choices, as both appear to support multi-zone/region use cases.

People installing a NATS cluster and expecting multi-zonal persistence will likely need something similar, which is potentially an extra wrench to throw in, but we would probably be able to handle the persistence just fine with EFS; people say "it's kinda slow" at lower workloads, but I don't think that's necessarily a big deal for something like a queue.

I imagine most people who need both HA and multi-AZ persistence have a good storageclass option at their disposal from their preferred k8s implementation/cloud provider.

[kevin-lindsay-1]

I have broken out the NATS issues discussed into two separate issues, persistence and high availability; that way we can keep discussion focused to Topology Spread and Affinity.

[kevin-lindsay-1]

@alexellis if my proposal of just exposing up different topologySpreadConstraints and affinity for each deployment sounds good to you, I can just go ahead and make a PR for that.

For the sake of keeping the existing pattern:

# ./values.yaml

affinity: {...} # [current] global "default"

topologySpreadConstraints: # [proposed] global "default"
  - {...}
  - {...}

gateway:
  affinity: {...} # [proposed] overrides global
  topologySpreadConstraints: # [proposed] overrides global
    - ...
      labelSelector:
        app: gateway
    - ...
      labelSelector:
        app: gateway

faasIdler:
  affinity: {...} # [proposed] overrides global
  topologySpreadConstraints: # [proposed] overrides global
    - ...
      labelSelector:
        app: faas-idler
    - ...
      labelSelector:
        app: faas-idler

[alexellis]

The additional context is useful, but please don't take us not getting to requests within 24 hours meaning that we are giving a silent approval on the design.

[kevin-lindsay-1]

Understood. Not trying to pressure you, I'd just rather be a little noisy in a (hopefully) useful way than quiet in an unuseful way; not trying to signal that I think the response is slow.

[LucasRoesler]

@kevin-lindsay-1 your proposal here #856 (comment) is this to apply just to the openfaas components (gateway, provider, nats, etc) or is this a global value to apply to all functions? it looks like it is just the components, right?

[kevin-lindsay-1]

@LucasRoesler only the core OF components in faas-netes/openfaas. I'm not worried about functions in this immediate moment, because their transience makes it less of a big deal to me; why worry about availability or failover when you're scaling to zero?

That, and most of our functions at this time are primarily async.

[LucasRoesler]

in that case, it seems like we could treat this the same as affinity, tolorations, and nodeSelector. These are empty and omitted by default, but the template can provide the required values to insert them where needed. This is essentially what you proposed here #856 (comment)