https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
m
It doesn't have to be wireguard - if there's a way to migrate to vxlan, etc.
My best guess is something is broken with the wireguard driver in the newer Elemental kernel
c
no. that’s the agent tunnel trying to connect to something, but the node hosting that pod is down
Not sure at all how wireguard is related, but that message just means that whatever node that pod is on is not connected to the server.
That is a symptom of something else. Make sure rke2 is running on all your nodes. if it is not, what do the logs say?
m
wireduard is the flannel backend in canal
*wireguard
c
yes but those messages are not from wireguard
m
Ah, the support engineer I was working with thought it was
c
no, that’s clearly from the rke2 process
m
rke2 is running on all nodes. kubectl works. cluster agent can't connect to rancher manager
c
what node should the pod with ip 
10.42.2.64
be running on? was it connected at the time that message was logged?
m
Yes - can't ping it from the node itself though
c
you also didn’t mention what version of rke2 you’re running
m
Upgraded to v1.34.5+rke2r1
Support had me downgrade canal charts but that didn't fix it either
#01779519 if it helps
c
I’m an engineer, I don’t have access to SF
m
Fair enough. First support person was working with me, when he left for the day the next person told me wireguard wasn't supported for RKE2 so here I am
c
that sounds like a symptom of the CNI being down - it is generally expected that nodes can reach pods running on any other node. that message is just confirming what you’ve already seen; you can’t hit pods from other nodes.
m
We ran a privileged sak pod and saw that the wireguard had anomalies from a working cluster vs not
Right - can't hit pods from nodes
c
can you hit pods from the node they’re running on?
m
yes
but not pods on other nodes
etcd talks fine. node to node is fine. node to pod on another node doesn't talk
c
what does the wireguard status show?
m
So this is what's weird - there's some duplicates with no ip allowed on the broken cluster that's on the newer elemental version
I don't see that on the same cluster setup on an older elemental version
That's 
wg show flannel-wg
There are duplicate peers with the same endpoint, but with no allowed ips
It's very weird
I've never had to go this far in the weeds with wireguard before - this had just worked for us
c
are the node IPs correct?
m
Yes - they're dhcp reservations and haven't changed
c
If you do 
kubectl get node
is it showing the correct internal and external IPs?
m
Yes
And the duplicate entries have peer pubkey's that aren't in the annotations which also seems like it shouldn't happen
c
I take it just rebooting the nodes doesn’t clear the duplicates? They come back after a reboot?
m
Correct - Rebooting doesn't clear them. Nor rolling out the ds.
c
Can you show the rke2-canal HelmChartConfig you are using to configure the flannel backend?
m
Well, it might clear them for a few seconds but they return shortly after
It's just flannel.backend: wireguard in the cluster yaml. In the helmchartconfig object that gets rendered as: 
valuesContent: '{"flannel":{"backend":"wireguard"},"global":{"cattle":{"clusterId":"c-m-p6rqzcv9"}}}'
(for rke2-canal)
We see it with 6.4.0-32-default kernel but not (so far) with 6.4.0-24-default. Unfortunately, trying to trigger an elemental downgrade seems to create jobs in the cluster that run for a few seconds and "succeed" without obviously doing anything.
Is there any way to migrate to vxlan without causing a split brain?
c
I’m not sure, we don’t generally test or support switching CNIs or CNI backends
m
Yeah, it doesn't seem like a low risk thing to do
c
but since you’ve already got a broken CNI here… doesn’t seem likely to make it much worse
m
Especially with Longhorn
That's a fair point
Would restoring etcd from backup possible do something here?
c
you could try it but if the problem is due to something on the elemental side it seems unlikely to fix anything
m
Yeah, that makes sense
c
I take it you’ve already looked at the kube-flannel container logs and don’t see anything interesting there?
Copy code
kubectl logs -n kube-system   rke2-canal-XXXXXX -c kube-flannel
there should be a bunch of stuff in there for it adding routes to various nodes
m
I see wireguard stuff in there. Mainly leases and added subnets
c
no errors?
m
I don't see any obvious error messages
A few "no subnet found for key " warning messages
Copy code
W0324 22:35:55.535796       1 main.go:577] no subnet found for key: FLANNEL_NETWORK in file: /run/flannel/subnet.env                                                                                               
W0324 22:35:55.535814       1 main.go:577] no subnet found for key: FLANNEL_SUBNET in file: /run/flannel/subnet.env                                                                                                
W0324 22:35:55.535827       1 main.go:612] no subnet found for key: FLANNEL_IPV6_NETWORK in file: /run/flannel/subnet.env                                                                                          
W0324 22:35:55.535837       1 main.go:612] no subnet found for key: FLANNEL_IPV6_SUBNET in file: /run/flannel/subnet.env
I0324 22:35:55.535847       1 iptables.go:101] Current network or subnet (10.42.0.0/16, 10.42.3.0/24) is not equal to previous one (0.0.0.0/0, 0.0.0.0/0), trying to recycle old iptables rules
Not sure how much of that is normal?
It's interesting that 
kubectl get po -A
shows all pods are healthy. It seems k8s itself thinks everything is fine.
c
yeah they can be healthy while not being able to communicate with each other lol
is it picking the right interface for inter-node traffic?
Copy code
I0324 23:49:40.082817       1 match.go:211] Determining IP address of default interface
I0324 23:49:40.083239       1 match.go:269] Using interface with name eth0 and address 172.17.0.5
I0324 23:49:40.083256       1 match.go:291] Defaulting external address to interface address (172.17.0.5)
m
Just checked the logs and yes, it is
c
there is not really a whole lot TO the flannel backend
m
At least for flannel on k3s there's a difference between wireguard (deprecated) and wireguard-native. The rke2-canal chart seems to only understand vxlan and wireguard. Does it end up being wireguard-native?
c
yes. wireguard-native for k3s is the wireguard built in to flannel
which uses native golang bindings to interact with wireguard, not the cli
can you show what you see in 
wg show all dump
?
m
sure, just a sec
Copy code
wg show all dump
flannel-wg      MI1YY+Ew/a3HyfHxRiO2oJgxG0q934BiyO1USD6Y/W0=    HkGvAhSbuIBV+BxGoy71wPd49JULZN9wB/Xfiijv7HU=    51820   off
flannel-wg      D4Z6YIwzXr+KiqsfEakOMlPSP37xBVMlVUczqy7nJl0=    (none)  10.16.0.82:51820        10.42.1.0/24    1774397500      19392   476800  off
flannel-wg      Q0/5ZktMwXXFc6CMC7/PljM8yYkEq8IeZgBDKwjToCI=    (none)  10.16.0.85:51820        10.42.2.0/24    1774397400      21280   13876208        off
flannel-wg      tGjBXOMUfWjy2wEkQ8pe6asBNmlr6QdYtIrg3c8W+AI=    (none)  10.16.0.87:51820        (none)  0       0       2960    off
flannel-wg      To84s+beDgnW7+GXZqaZqPEDbqbs5qx0MPLbHPI26Vo=    (none)  10.16.0.31:51820        10.42.6.0/24    1774397469      1256736 753552  off
flannel-wg      IihXrd+JLTEcj/jQAhW+qDjulTSAr1/aalrF2kNmr2w=    (none)  10.16.0.39:51820        10.42.5.0/24    1774396673      5880    5692    off
flannel-wg      BcyAVyhSDj2Azzl/ohk2wP5B4AepAjgsKPaAW2mvjzs=    (none)  10.16.0.87:51820        10.42.0.0/24    1774397492      16612   217676  off
.87 is duplicated there. Some of the duplicates come and go
c
on mine it looks like this:
Copy code
brandond@dev01:~$ ssh rke2-server-001 wg show all dump
flannel-wg	wJFmA6axjY8vjO7+Ei8gBMk7tV3UJj2WZ+8WmnYmcX0=	yeHTFooNaHNFMwsFjlZeX6IGFVRb1KIGZxwyOZp6Znk=	51820	off
flannel-wg	mPM8UUGC4/9P6GTxF/wfd7K4aVNS68miXAt8mKJqdEI=	(none)	172.17.0.5:51820	(none)	0	0	0	off
flannel-wg	C9iinnW40QZ6t1IVVkyQaCJTKEpvGts7L6NXD8BqpAk=	(none)	172.17.0.5:51820	10.42.1.0/24	0	0	0	off
brandond@dev01:~$ ssh rke2-agent-001 wg show all dump
flannel-wg	sJwNRPQCafozN7Fc8v6uoologLvH7mkIoUl3Kx64zE4=	C9iinnW40QZ6t1IVVkyQaCJTKEpvGts7L6NXD8BqpAk=	51820	off
flannel-wg	yeHTFooNaHNFMwsFjlZeX6IGFVRb1KIGZxwyOZp6Znk=	(none)	172.17.0.4:51820	10.42.0.0/24	0	0	0	off
m
Yeah, on the near identical cluster that works, it looks like that too. Nice and neat.
Copy code
wg show all dump  
flannel-wg      2OXT42yFTl6V6HEaf7viCNFU1Jjs9hxYuw+F8RTySk8=    jyIaJ8beApHQVOKKO7RcvPkRb2s07JGOAL/y87sY8Ss=    51820   off
flannel-wg      f0r6e3c3N76NaUf4D4aZdRd9vcFwATPP0XCPeQpZsVI=    (none)  10.16.0.86:51820        10.42.0.0/24    0       0       0       off
flannel-wg      nE9V8B1shZxgju1Bi1Yl+CZqCF/Ikrf6XhV4vtJ+AD8=    (none)  10.16.0.81:51820        10.42.6.0/24    0       0       0       off
flannel-wg      Vzvv4ayBd+gSV007c8ZUf636TFdJg7a5+qD3gd2D+3w=    (none)  10.16.0.83:51820        10.42.12.0/24   0       0       0       off
flannel-wg      WAHf8aJpfwWSaztRg89skeN2aETuJDJxDN3bknaMgU4=    (none)  10.16.0.92:51820        10.42.8.0/24    0       0       0       off
flannel-wg      YejT0NKaPgw7vQDXIlqTJt+zy2yIMGemducGD5zOayQ=    (none)  10.16.0.90:51820        10.42.14.0/24   0       0       0       off
flannel-wg      /Mvk+Y4+JI/LeHQaK7X51m7MOeS6DQFi+thNX7LHejk=    (none)  10.16.0.88:51820        10.42.7.0/24    0       0       0       off
flannel-wg      juTofK1unuQSeQQdcXS2bAbQjE+fjYk8xvrZKtIQNQQ=    (none)  10.16.0.89:51820        10.42.13.0/24   0       0       0       off
flannel-wg      FZs52FvXS5mvIeTylDh7YgFY9Rdf+w70fE/T05t+BEI=    (none)  10.16.0.34:51820        10.42.2.0/24    0       0       0       off
flannel-wg      m3WsNN8TEQz7ggp45nP+nOwvCyt9bev9cfgR2Mb6/Go=    (none)  10.16.0.35:51820        10.42.4.0/24    0       0       0       off
flannel-wg      /C/n4Z0CbNIx0fNswG3+JX4/KjtlWRUjnwC+m9uhWCs=    (none)  10.16.0.84:51820        10.42.11.0/24   0       0       0       off
flannel-wg      Nu6DY5eu23oY4drsZhTWO1bFHI/jjebjC3V/owrvvT4=    (none)  10.16.0.91:51820        10.42.5.0/24    0       0       0       off
Same hardware, versions, except for it's running an older 6.1 elemental version
c
compare that to another node in the cluster - does it have the 
HkGvAhSbuIBV+BxGoy71wPd49JULZN9wB/Xfiijv7HU=
public key listed for any peers? is the peer address and subnet correct?
m
Yes - the previous 
wg dump all
was from the .32 node. This one is from the .39
Copy code
peer: HkGvAhSbuIBV+BxGoy71wPd49JULZN9wB/Xfiijv7HU=
  endpoint: 10.16.0.32:51820
  allowed ips: 10.42.3.0/24
  latest handshake: 19 minutes, 40 seconds ago
  transfer: 5.56 KiB received, 5.74 KiB sent
What I don't know is where the public key is coming from for the duplicate. Is there some AI hallucination in the latest wireguard driver 😅
c
I think that is just a wireguard thing
Once I actually start throwing traffic between nods I see that too. I think it is normal.
👍 1
just on the server nodes, idk why
Copy code
brandond@dev01:~$ ssh rke2-server-001 wg show all dump
flannel-wg	wJFmA6axjY8vjO7+Ei8gBMk7tV3UJj2WZ+8WmnYmcX0=	yeHTFooNaHNFMwsFjlZeX6IGFVRb1KIGZxwyOZp6Znk=	51820	off
flannel-wg	mPM8UUGC4/9P6GTxF/wfd7K4aVNS68miXAt8mKJqdEI=	(none)	172.17.0.5:51820	(none)	0	0	0	off
flannel-wg	C9iinnW40QZ6t1IVVkyQaCJTKEpvGts7L6NXD8BqpAk=	(none)	172.17.0.5:51820	10.42.1.0/24	1774398008	2108	2116	off
brandond@dev01:~$ ssh rke2-agent-001 wg show all dump
flannel-wg	sJwNRPQCafozN7Fc8v6uoologLvH7mkIoUl3Kx64zE4=	C9iinnW40QZ6t1IVVkyQaCJTKEpvGts7L6NXD8BqpAk=	51820	off
flannel-wg	yeHTFooNaHNFMwsFjlZeX6IGFVRb1KIGZxwyOZp6Znk=	(none)	172.17.0.4:51820	10.42.0.0/24	1774398008	2116	2108	off
odd that on your “working” cluster the counters are all zero…
is this perhaps a network policy thing?
The wireguard stats claim that traffic is passing between peers. If it’s just node-to-pod traffic that’s getting blocked, it could be netpol.
m
I think they were all zeros on your working one too - on mine it looks like this:
Copy code
brandond@dev01:~$ ssh rke2-server-001 wg show all dump
flannel-wg	wJFmA6axjY8vjO7+Ei8gBMk7tV3UJj2WZ+8WmnYmcX0=	yeHTFooNaHNFMwsFjlZeX6IGFVRb1KIGZxwyOZp6Znk=	51820	off
flannel-wg	mPM8UUGC4/9P6GTxF/wfd7K4aVNS68miXAt8mKJqdEI=	(none)	172.17.0.5:51820	(none)	0	0	0	off
flannel-wg	C9iinnW40QZ6t1IVVkyQaCJTKEpvGts7L6NXD8BqpAk=	(none)	172.17.0.5:51820	10.42.1.0/24	0	0	0	off
brandond@dev01:~$ ssh rke2-agent-001 wg show all dump
flannel-wg	sJwNRPQCafozN7Fc8v6uoologLvH7mkIoUl3Kx64zE4=	C9iinnW40QZ6t1IVVkyQaCJTKEpvGts7L6NXD8BqpAk=	51820	off
flannel-wg	yeHTFooNaHNFMwsFjlZeX6IGFVRb1KIGZxwyOZp6Znk=	(none)	172.17.0.4:51820	10.42.0.0/24	0	0	0	off
Sorry - copy paste error
I'm still also trying to see a way to get Elemental to give the college try at downgrading that cluster
c
Yeah that with all zeros was right after I brought it up, after I passed some traffic between pods the counters were all nonzero
👍 1
m
Yeah, it's odd that the working cluster is all zeros and the non-working one has numbers
It looks like an etcd restore might have fixed it?
I can access it in Rancher Manager again
Yup, restoring etcd fix it. So weird.
I'm happy that ectd restore fixed it, but it also means I don't have a clue what caused it and if it's sane to upgrade the OS on prod. I can't isolate it to the CNI chart, RKE2 version, hardware, or kernel. My only guess is somehow in the upgrades there was some orphaned CNI or node data in etcd that manifested itself in a wonky flannel/wireguard
17 Views
Open in Slack
PreviousNext
Powered by