fix(k3s): raise ARP neighbour table limits to stop table overflow

[bborbe]

Problem

nuke-k3s-prod-0 logged 31,904 neighbour: arp_cache: neighbor table overflow! today — ~7000/hour, continuously since boot.

Once arp_tbl hits gc_thresh3, neigh_alloc() fails and any packet needing a new neighbour entry is silently dropped. That caused:

• ~35% DNS failure rate (getent hosts <registry>: 0/40 succeeded)
• recurring ImagePullBackOff across prod (peaked at 8 pods), including the MT5 tick pipeline
• systemd-resolved reporting zero timeouts — its queries never left the box

Why it wasn't obvious

arp_tbl is a single global kernel table counting entries across every netns, but ip neigh show only displays the host namespace. prod-0 showed 404 entries against the 1024 default — apparent 60% headroom — while overflowing constantly, because ~389 pod netns held the rest.

Thresholds were stock kernel defaults (128/512/1024), never tuned for node density.

Change

Adds sysctl.options to the k3s group (13 hm.nuke-k3s-* nodes):

net.ipv4.neigh.default.gc_thresh1 = 4096
net.ipv4.neigh.default.gc_thresh2 = 8192
net.ipv4.neigh.default.gc_thresh3 = 16384

Group-scoped rather than node-scoped deliberately: dev-0 runs 379 veth against prod-0's 389 — clean today only because it hasn't crossed the line.

Entries are allocated on demand, so the raised ceiling is free on small nodes (~4 MB worst case if a table ever actually filled). No reboot: the sysctl bundle triggers systemd-sysctl restart.

Verification — applied via bw apply -i k3s, 13/13 fixed, 0 failed

Metric	Before	After
getent hosts <registry>	0/40	40/40
overflow msgs / 2 min	~230	0
unhealthy pods across all 3 clusters	8	0
ip neigh show	404	404 (unchanged — never the signal)

Note

hm.hell runs k3s but does not match this group's .*k3s.* member pattern, so it is not covered. Pre-existing gap, left alone here.

[ben-s-pull-request-reviewer]

{
  "verdict": "approve",
  "summary": "The PR raises ARP neighbour table gc_thresh limits (4096/8192/16384) to prevent table overflow on k3s nodes running many pods. The change is well-documented with a production incident reference. Values are correctly ordered (4096 < 8192 < 16384) and typed as strings per the config schema. The blank-line removal in metadata.py is incidental formatting with no functional impact.",
  "comments": [],
  "concerns_addressed": [
    "correctness: blank line removal in bundles/intellij/metadata.py is incidental formatting, not a bug",
    "correctness: sysctl gc_thresh values are string types as the config schema expects",
    "correctness: gc_thresh invariant (4096 < 8192 < 16384) is satisfied"
  ]
}