https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
How exactly did it break things? Do you mean you broke your cluster networking?
b
I attempted to remove the Helm release using Rancher, then using Rancher selected my Git repo that I'd created as a fork of the RKE2 charts
it let me pick the Multus chart and try to install it, but it wouldn't install - it just stayed in Stalled state forever
c
Yeahhhhh that is highly likely to mess things up. What specifically are you trying to accomplish?
b
I have some NoSchedule nodes, and I want to use Multus DHCP on them
c
Don’t use Rancher to manage the charts bundled with RKE2
👍 1
b
but the chart doesn't support adding Tolerations
or rather, it does - just not for Multus DHCP, only for Multus
I was planning to submit a feature request (maybe even a PR), but wanted to fix it short term for me
c
are you talking about the whereabouts subchart?
b
No, specifically the Multus DHCP feature
It deploys the DHCP daemonset separately from Multus itself, as an optional addon
c
ah the dhcp daemonset/
👍 1
b
and that daemonset does not support adding tolerations even though the Helm chart has a values field for them
correct
c
Rancher manages helm charts using a completely different set of tools than RKE2. If you try to upgrade the RKE2 packaged charts using Rancher it will conflict with the helm controller built into RKE2.
😲 1
b
(My cluster is all good by the way, I was able to simply change a Helm value and restart rke2-server to trigger a reinstall of Multus)
Ahh. I did... not know that.
c
We put annotations on the charts to prevent the Rancher UI from suggesting you upgrade them, but I guess it doesn’t stop you from going in there and manually editing the rke2 charts repo to point at something else and then manually trying to upgrade them
but you’re not supposed to do that
b
Ah, I didn't change the repo
I added my own
and then uninstalled old, went to install new
and... ded
c
ah ok
b
😅
c
At the moment the rancher UI doesn’t let us prevent users from doing that. but you shouldn’t.
b
Definitely noted!
🦶 + 🔫
c
for sure!
b
As for accomplishing what I wanted, though - what's the "right" way to do it
other than hoping people want my PR merged 👼
basically just want to take the latest multus, apply a four line patch, and apply that version of the chart
c
Well if you can open an issue I can make sure it gets into the April release
what are you using for your primary CNI at the moment?
b
When is that one? That will definitely be helpful medium term, but the reason I'm doing this is we need to use 
dhcp
instead of 
whereabouts
for a 10K pod workload, because Whereabouts takes ~1 to 2s per pod to spawn pods
(Obviously April, but early or late, I mean)
c
it’ll be in April lol. a week or so after the upstream patch release. We do a release a month, same as upstream.
😂 1
🙏 1
b
We're using Calico for the main CNI + bridge CNI for the workload + we were prototyping with whereabouts as the IPAM for it
Is there any way to "bodge" it til April, assuming that this cluster is only used for this testing/simulation work?
(IE: if I break it while trying to make simulations work better, it's not the end of the world, just.. annoying)
c
ok so assuming your cluster currently has 
cni: multus,calico
you could try switching that to 
cni: calico
and then just drop a replacement HelmChart for your forked Multus chart at 
/var/lib/rancher/rke2/server/manifests/custom-multus.yaml
Copy code
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChart
metadata:
  name: rke2-multus
  namespace: kube-system
spec:
  bootstrap: true
  chart: <https://rke2-charts.rancher.io/assets/rke2-multus/rke2-multus-v4.0.2-build2024020800.tgz>
Replace that chart URL with the URL of your forked chart of course.
that should be equivalent to what you have now
I will see if I can get the tolerations addressed here: https://github.com/rancher/rke2-charts/pull/429#issuecomment-2023870270 If not I think we can get it handled in another PR.
b
thank you brandon 🙂 you do, sincerely, fight for the users
Interesting. So I used the workaround...
and now I get this error when starting pods:
Copy code
│   Warning  FailedCreatePodSandBox  106s (x70 over 19m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown d │
│ esc = failed to setup network for sandbox "61895d3687d140f681f71356fba9fcafc2abe6800c931b400851a2438be9a2d0": plugin type="multus" name="multus-cni-network" failed  │
│ (add): [zerotesting/bootstrap-0/12423915-1f79-4eed-90d1-5763e974e280:calico-default]: error adding container to network "calico-default": failed to add IPv4 route f │
│ or 0.0.0.0/0 via 169.254.1.1: file exists
it's odd because I'm using the same charts, with the same settings and the same Calico network, but it works with the stock Multus charts and not with my customised one 😞
(and all I changed was tolerations)
Aha! I figured it out. When I was using Whereabouts, it was not assigning a default route (which is the correct behaviour)
When using the DHCP plugin, my DHCP server was handing out a default route.
16 Views
Open in Slack
PreviousNext
Powered by