https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
… that’s not the correct way to change that. You should customize the canal chart configuration as described here: https://docs.rke2.io/helm#customizing-packaged-components-with-helmchartconfig
e
sure, we can use the alternate method, but wouldn't updating the ENV vars still ultimately change the policy?
or are you saying the policy cannot be changed once depoyed?
c
1. I believe changing the interface requires a reboot of the node, but I’m not sure. 2. you can’t just poke directly at things that are managed by a helm chart; your changes will be reverted next time the chart is upgraded or the chart values changed. This is true of anything managed by helm, not just our stuff.
e
yeah we deff get this, we are currently troubleshooting the issue @bright-lifeguard-9803 pointed out before moving the config into HelmChartConfig hence his question
@creamy-pencil-82913 a reboot does not fix the issue, ultimately we are trying to understand how canal determines the src address it looks like the first interface is always used regardles
c
What do the pod logs show? It should tell you which iface it’s using when it starts.
b
I'm going to try this:
Copy code
cat /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml
apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-canal
  namespace: kube-system
spec:
  valuesContent: |-
    calico:
      ipAutoDetectionMethod: interface=lom0
e
@creamy-pencil-82913 I do not see a clear log which sates the interface it is listening on when looking at the rke2-canal pod logs, at least nothing obvious
b
(this is a different system I repoed the issue on in my home lab, thus the interface change to 
lom0
)
c
there are multiple containers in that pod, which ones are you looking at?
e
calico-node also has kube-flanel, install-cni (init), and flexol-driver (init)
b
looks like that did the right thing (in so far as pluming in the env var, about to test the interface stuff now)
Copy code
kubectl get ds -n kube-system rke2-canal -o yaml | grep -A1 AUTO
        - name: IP_AUTODETECTION_METHOD
          value: interface=lom0
e
@creamy-pencil-82913 as far as the kube-flannel logs are concerned it shows the correct interface
Copy code
I0627 17:59:49.443577       1 match.go:263] Using interface with name eth0 and address 10.1.148.230
I0627 17:59:49.443600       1 match.go:285] Defaulting external address to interface address (10.1.148.230)
Hence our ultimate question, we are explaining our observation about the wrongly sourced packets, however, I don't actually know what's responsible for setting that , my understanding IP_AUTODETECTION_METHOD had more to do with node to node than the local systems srcing of packets
i will not weigh in tho, ill let @bright-lifeguard-9803 describe what he is seeing
c
IIRC the IP autodetection stuff determines what interface IP it advertises to the cluster, when it comes to sending traffic it does not do anything special - it just generates packets to the other node, and allows the kernel routing table to select the source address and interface for that traffic as it would any other client traffic. If you have multiple interfaces that the kernel considers as viable for that traffic, you’d probably want to modify your routing tables to get the kernel to select the one you want when traffic is destined for that node.
@bland-account-99790 would probably know more
b
yeah even with 
interface=lom0
the packets are still sourced from the wrong interface
but when I swap the interfaces the packets are sources from the right interface
e
well you should clarify... they are the "correct" interface only because the management interface in your case happens to be the 1st interface... However, in other situations the management interface would be the second interface and thus it would be sourcing from the wrong interface (ifidx 1)
b
yeah, basiclly, 
interface=lom0
does not seem to change the behavior at all.
c
is that the interface with the lowest cost to the other nodes?
or if they have equal cost in the routing table, the first one in the if table?
b
we're single node
c
oh so are you just talking about the interface selected for traffic egressing the cluster?
not for CNI traffic between nodes?
e
traffic inside the cluster host
c
you just said you have only a single node
how can there be traffic within the cluster, leaving the node, if you have only a single node
b
he calls a single node a cluster for reasons
c
what specific traffic are you looking at
I mean you can have a single-node cluster, that is fine and makes sense
but there should be no inter-node traffic managed by the CNI if you have only a single node…
b
im looking at probe traffic as a sentinel
c
CNI manages traffic between pods using the overlay network
if all the pods are on a single node, there would not be any CNI traffic leaving the node
the only traffic from pods leaving the node would be traffic egressing the cluster, to the internet or the like
b
what we are seeing is that due to the source IP being "wrong" "sometimes", that when we apply Policy Routing to one of the interfaces that the probes go out the PBR interface that's not intended for k8s traffic (in this case, egress of the single node deployment.)
e
i wouldn't even complicate it that much @bright-lifeguard-9803... the kubelet when health checking a pod originates traffic from the first interface on the linux system, which in our case is not correct, even tho the interfaces are setup with the appropriate route metric
c
afaik kubelet health checks occur within the pod’s network namespace.
b
pcaps say otherwise
c
but I may be thinking of just the exec-based health checks, I guess if you are using HTTP or GRPC health checks those would go from host ns -> pod ns as the kubelet process is initiating the connection
b
thats what the pcaps show me
c
how did you end up poking at CNI IP autodetection for this? I don’t think that’s involved at all. If you’re just talking host network -> pod network, within the same node, I don’t believe the CNI interface selection is a part of the equation
kubelet just opens a TCP connection to the pod, and the kernel decides what the source IP for that should be.
b
because cali seems to control the egrerss paths
c
but this isn’t egress. it’s host network ns -> pod network ns, within a single node
correct?
b
yeah thats at least one case
I just picked the probes because thats how we found the problem,
cali has it hands in egress and local node traffic as well, at least it seems that way.
e
@creamy-pencil-82913 yeah I ssume it's just pure kernel routing but what we see 192.0.0.4 for example is coredns, the kernel wants to route it through 10.1.252.132 which is the wrong interface... if you down the interface it does the correct thing, looking in the cali logs it had appears it causes this...
Copy code
sudo ip r g 192.0.0.4
192.0.0.4 dev calibcb3fe8370f src 10.1.252.132 uid 0
    cache
routes
Copy code
[broala@ip-10-1-148-230 bin]$ ip route
default via 10.1.148.1 dev eth0 src 10.1.148.230 metric 203 mtu 9001
default via 10.1.252.1 dev eth1 src 10.1.252.132 metric 210 mtu 9001
10.1.148.0/22 dev eth0 proto kernel scope link src 10.1.148.230 metric 203 mtu 9001
10.1.252.0/22 dev eth1 proto kernel scope link src 10.1.252.132 metric 210 mtu 9001
192.0.0.2 dev calie87d0f9759f scope link
192.0.0.3 dev calie56f4c00502 scope link
192.0.0.4 dev calibcb3fe8370f scope link
192.0.0.5 dev cali0b4cbd83c6e scope link
192.0.0.6 dev cali86937c25236 scope link
192.0.0.7 dev calice4109fc67f scope link
192.0.0.8 dev cali2c85d7dd4da scope link
192.0.0.9 dev cali94d23b88e64 scope link
192.0.0.10 dev cali18df69b6a18 scope link
192.0.0.11 dev cali00ec06cb7dd scope link
192.0.0.13 dev calif8b843d9ef2 scope link
192.0.0.14 dev cali83295eb1102 scope link
192.0.0.15 dev cali47264bd43d4 scope link
192.0.0.16 dev cali995f34fdcbb scope link
192.0.0.17 dev cali7853826c62c scope link
192.0.0.18 dev cali1de2de92208 scope link
192.0.0.19 dev cali8d1b2a7e7d1 scope link
192.0.0.20 dev cali65e0d43664c scope link
192.0.0.21 dev cali0d961dbf0ba scope link
192.0.0.22 dev calic6c090420a5 scope link
192.0.0.23 dev cali006646e50fc scope link
192.0.0.24 dev cali4efb7326ad0 scope link
we explicitly set node-external-ip and node-ip to 10.1.148.230 which is the "management" network
c
you have two default routes!?
why would you want to have a default route that sends traffic over your management network?
If you remove the default route via the management network, and add specific routes via that IF that cover only your internal network instead, I bet you’d see it doing what you want.
I am pretty sure this is all just kernel behavior, since the kubelet doesn’t bind to a specific address when creating the outbound connection for the health check, and if the client doesn’t provide an address, it is assigned by the kernel
e
Yeah, that's just silly aws, i've deleted the default route many times over which doesn't resolve the issue, each leg is on a diff subnet which happens to dhcp.. i didn't ignore routes in netplan on the alternate interface
c
b
yeah so our "management" network is for k8s.
Were going to try to isolate the problem interface into a VRF
VRF worked for us to isolate the traffic.
30 Views
Open in Slack
PreviousNext
Powered by