https://rancher.com/ logo
Powered by
#rke2
Title
# rke2
n

narrow-noon-75604

08/24/2022, 5:04 AM
Hi, I am trying to deploy a HA RKE2 cluster on baremetal machines with ubuntu20 using kube-vip. The first server node is up and running and be able to list the VIP so I have added 2 more server nodes and a worker node. The installation is successful without any issue and the VIP has appeared after the overall RKE2 installation. Then I have deployed a sample nginx application to test if the service is accessible through VIP address. I am able to access the application through all the server node IP Addresses but could not be able to access it using VIP. Then I tried to list the IP addresses in the interface that I am using and found that the VIP has disappeared. Right after the installation is finished,
Copy code
ip a list $INTERFACE
Output:
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:9b:3a:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.71/24 brd 192.168.10.255 scope global ens160
      valid_lft forever preferred_lft forever
    inet 192.168.10.74/32 scope global ens160
      valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe9b:3acb/64 scope link
      valid_lft forever preferred_lft forever
After sometime VIP is disappearing,
Copy code
ip a list $INTERFACE
Output:
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:9b:3a:cb brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.71/24 brd 192.168.10.255 scope global ens160
      valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fe9b:3acb/64 scope link
      valid_lft forever preferred_lft forever
Please let me know if I am missing anything.
c

creamy-pencil-82913

08/24/2022, 5:59 AM
Have you looked at the kube-vip logs? This is all kube-vip stuff, not rke2 specific really.
n

narrow-noon-75604

08/24/2022, 6:00 AM
I guess this might be because of etcd latency as mentioned in a similar issue - https://github.com/Azure/aks-hci/issues/167
Please find the logs of 3 VIP pods -
Copy code
$ kubectl logs kube-vip-ds-554th -n kube-system
time="2022-08-24T06:03:39Z" level=info msg="Starting <http://kube-vip.io|kube-vip.io> [v0.5.0]"
time="2022-08-24T06:03:39Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2022-08-24T06:03:39Z" level=info msg="prometheus HTTP server started"
time="2022-08-24T06:03:39Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2022-08-24T06:03:39Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [<http://master3.192.168.10.73.nip.io|master3.192.168.10.73.nip.io>]"
I0824 06:03:39.600009       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2022-08-24T06:03:39Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [<http://master3.192.168.10.73.nip.io|master3.192.168.10.73.nip.io>]"
I0824 06:03:39.600649       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-cp-lock...
time="2022-08-24T06:03:39Z" level=info msg="Node [<http://master2.192.168.10.72.nip.io|master2.192.168.10.72.nip.io>] is assuming leadership of the cluster"
time="2022-08-24T06:03:39Z" level=info msg="new leader elected: <http://master2.192.168.10.72.nip.io|master2.192.168.10.72.nip.io>"

--------------------------------------------------------------------------------------------------------
$ kubectl logs kube-vip-ds-rqhn5 -n kube-system
time="2022-08-24T06:05:17Z" level=info msg="Starting <http://kube-vip.io|kube-vip.io> [v0.5.0]"
time="2022-08-24T06:05:17Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2022-08-24T06:05:17Z" level=info msg="prometheus HTTP server started"
time="2022-08-24T06:05:17Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2022-08-24T06:05:17Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [<http://master2.192.168.10.72.nip.io|master2.192.168.10.72.nip.io>]"
I0824 06:05:17.604621       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2022-08-24T06:05:17Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [<http://master2.192.168.10.72.nip.io|master2.192.168.10.72.nip.io>]"
I0824 06:05:17.605137       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-cp-lock...
time="2022-08-24T06:05:17Z" level=info msg="new leader elected: <http://master1.192.168.10.71.nip.io|master1.192.168.10.71.nip.io>"
time="2022-08-24T06:05:17Z" level=info msg="Node [<http://master3.192.168.10.73.nip.io|master3.192.168.10.73.nip.io>] is assuming leadership of the cluster"


--------------------------------------------------------------------------------------------------------
$ kubectl logs kube-vip-ds-vdwdq -n kube-system
time="2022-08-24T06:03:33Z" level=info msg="Starting <http://kube-vip.io|kube-vip.io> [v0.5.0]"
time="2022-08-24T06:03:33Z" level=info msg="namespace [kube-system], Mode: [ARP], Features(s): Control Plane:[true], Services:[true]"
time="2022-08-24T06:03:33Z" level=info msg="prometheus HTTP server started"
time="2022-08-24T06:03:33Z" level=info msg="Starting Kube-vip Manager with the ARP engine"
time="2022-08-24T06:03:33Z" level=info msg="beginning services leadership, namespace [kube-system], lock name [plndr-svcs-lock], id [<http://master1.192.168.10.71.nip.io|master1.192.168.10.71.nip.io>]"
I0824 06:03:33.407599       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-svcs-lock...
time="2022-08-24T06:03:33Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plndr-cp-lock], id [<http://master1.192.168.10.71.nip.io|master1.192.168.10.71.nip.io>]"
I0824 06:03:33.408068       1 leaderelection.go:248] attempting to acquire leader lease kube-system/plndr-cp-lock...
time="2022-08-24T06:03:33Z" level=info msg="new leader elected: <http://master2.192.168.10.72.nip.io|master2.192.168.10.72.nip.io>"
time="2022-08-24T06:03:33Z" level=info msg="Node [<http://master2.192.168.10.72.nip.io|master2.192.168.10.72.nip.io>] is assuming leadership of the cluster"
time="2022-08-24T06:05:07Z" level=info msg="Node [<http://master3.192.168.10.73.nip.io|master3.192.168.10.73.nip.io>] is assuming leadership of the cluster"
I0824 06:05:11.368625       1 leaderelection.go:258] successfully acquired lease kube-system/plndr-svcs-lock
6 Views
Powered by