https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
p
Does the VM run on the node that the VIP is on? @faint-art-23779 @red-king-19196 is this the same as https://github.com/harvester/harvester/issues/3960?
s
yeah
r
Probably hitting issue 3960. Have you created any new cluster network and network config other than the 
mgmt
one? Or just created the L2Vlan VM Network associated with the 
mgmt
cluster network?
s
yeah wr have a separate nic for cluster network
@red-king-19196 what can we do to fix this?
@adventurous-portugal-91104 cc
a
ok? if this is true, this is not very good. Though we have a 32gb node we could use in this role and rebuild our harvestercluster (again..... sigh).
We don't have this kind of setup @some-addition-13540. We have a dedicated vm network on a totally different vland id. Remember we had a similar issue earlier when we did something like this, when we ran vms on the mgmt network.
r
Can you directly curl port 6443 with any of the three VMs’ IP addresses from the bastion VM?
s
Yes I can
@red-king-19196 I also created a plain Ubuntu VM with nginx the Proxy pod gives
Copy code
time="2024-07-10T09:54:55Z" level=info msg="probe error, I/O timeout, address: 10.0.0.227:80, timeout: 3s"
But I can 
curl 10.0.0.227:80
totally fine from the Bastion itself
LB doesn't work
This seems wrong? The mgmt-br has all of the IP addresses associated with it and not 
hypervisor-br
which I thought is the one I have assigned it to in the LB IPAM Pool
Copy code
8: mgmt-br: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000                       
    link/ether 1c:98:ec:5c:18:28 brd ff:ff:ff:ff:ff:ff                                                                                                         
    inet 10.0.2.10/24 brd 10.0.2.255 scope global mgmt-br                                                                                                      
       valid_lft forever preferred_lft forever                                                                                                                 
    inet 10.0.2.20/32 scope global mgmt-br                                                                                                                     
       valid_lft forever preferred_lft forever                                                                                                                 
    inet 10.0.0.30/32 scope global mgmt-br                                                                                                                     
       valid_lft forever preferred_lft forever                                                                                                                 
    inet 10.0.0.31/32 scope global mgmt-br                                                                                                                     
       valid_lft forever preferred_lft forever
@red-king-19196 any idea?
f
Seems there're 3 LB IP associated. 10.0.2.20, 30, 31. Only 10.0.2.10 can be ping by the IP of the same subnet. Can you do 
ip route show
and 
ip neigh
on the node and provide the output?
s
10.0.0.2.10 is the harvester node, 10.0.2.30 is the harvester vip on management net 10.0.0.30 is a talos cluster lb vip on hypervisor / vm network 10.0.0.31 is a nginx test on hypervisor / vm network
f
Sorry that I was confused by the IP 10.0.0.30 and 10.0.2.30. The IP addresses with postfix /`32` imply they're LB IPs and there're no subnet for them. It means they can't not be found by L2 broadcast (ARP). The VIP is reached by some magic iptables rules to route the traffic destined to VIP from mgmt-br's IP (10.0.2.10). So you mean curl to 10.0.2.10 is OK but curl 10.0.2.30 fails?
s
10.0.2.10 is a node ip of the node, 10.0.2.30 is harvester's VIP 10.0.0.30 and .31 are other VM / k8s based LB's that don't work
f
Should be the problem of the LB. The ping should work and only http(s) fails? Can you do 
iptables-save
to save to a file and post the file here?
s
sure..
And no ping to the VM LB's doesn't work
f
The command 
sysctl -a | grep bridge-nf-call
should show 
net.bridge.bridge-nf-call-iptables=0
in your configuration. If so, please 
sysctl -w net.bridge.bridge-nf-call-iptables=1
to check if the ping/curl back to work or not. Thanks
s
do I need to reboot or something then after?
f
No.
The sysctl should work at runtime.
s
did that now, still can't ping etc
f
Then I'll need the output file of the 
iptables-save
to check how the ping/http packet been redirected....
s
f
-A INPUT -d 10.0.0.31/32 -p tcp -m tcp --dport 80 -m comment --comment "default/webtest kube-vip load -A INPUT -d 10.0.0.30/32 -p tcp -m tcp --dport 6443 -m comment --comment "cluster-capi-mgmt-p-01/cluster-capi-mgmt-p-01-capi-mgmt-p-01-lb kube-vip load balancer IP" -j ACCEPT -A INPUT -d 10.0.0.30/32 -p udp -m udp --dport 68 -m comment --comment "cluster-capi-mgmt-p-01/cluster-capi-mgmt-p-01-capi-mgmt-p-01-lb kube-vip load balancer IP" -j ACCEPT It seems that you are able to 
curl
10.0.0.31 (the nginx). 
10.0.0.30
is only for 6443 (apiserver) but not allowed for port 80.
s
No I am not able to curl 10.0.0.31
Copy code
curl --connect-timeout 1 10.0.0.31:80
curl: (28) Failed to connect to 10.0.0.31 port 80 after 1001 ms: Timeout was reached
Copy code
curl --connect-timeout 1 10.0.0.30:6443
curl: (28) Failed to connect to 10.0.0.30 port 6443 after 1001 ms: Timeout was reached
that's from the 
bastion
on the same network as the LB
Also as a note if I set
Copy code
sysctl -w net.bridge.bridge-nf-call-iptables=0
Then the console / Harvester VIP is available from the Bastion if I set it to 1 then it doesn't ..
Copy code
sysctl -w net.bridge.bridge-nf-call-iptables=1
I do however reach the vip:6443 / kube api fine
r
Let me recap the current information: • The target VMs are on the 
10.0.0.0/24
subnet • The LB IP address for the target VMs is 
10.0.0.30
• The bastion VM is also on the 
10.0.0.0/24
subnet but outside the Harvester cluster • The Harvester nodes are on the 
10.0.2.0/24
subnet • The Harvester cluster VIP address is 
10.0.2.30
• The VM Network for the 
10.0.0.0/24
subnet is associated with the 
hypervisor
Cluster Network using a secondary NIC on each Harvester nodes And you’re unable to access the LB IP address with port 6443 from the bastion VM. Direct access to the target VMs’ port 6443 is okay. Is that correct?
If that’s the case, the Harvester Load Balancer currently does not support assigning LB IP addresses to secondary interfaces. It will always bind the LB IP addresses regardless of what subnets are to the management interface, i.e., 
mgmt-br
. So, for this kind of VM-type load balancer usage, it’s required to create LBs from the IP pool with the same subnet range as the Harvester nodes’ management network. The other way is to move the LB IP address inside the target VMs. If they are a guest cluster that has the Harvester Cloud Provider running, you can create an LB-type Service to announce the LB IP address on the 
10.0.0.0/24
subnet.
a
How could we get this kind of setup to work in Harvester? It is very limiting to only run LBs on the mgmt interface.
Thats correct
I think we need to get someone from Suse to answer us on this
p
@adventurous-portugal-91104 @red-king-19196 is the main developer of Load balancer feature.
👍 1
a
Oh I am sorry, he didnt have the SUSE employee tag on his name 🙂
@red-king-19196 So what me and Endre did
Let me summerize
• mgmt network: 10.0.2.0/24 (untagged) on first harvester nic, • vm-network (called hypervisor) tagged to vlanid 4 on secondary nic: 10.0.0.0/24, • Bastion host runnin on vm-network (10.0.0.5/24) • for vm network I have create also a cluster network and the network config ofcourse in harvester. • I have a workstation that can also contact all harvester from a different network, only Endre remotes into my environment through the Bastion host • Harvester node mgmt IP: 10.0.2.10 • Harvester mgmt Cluster VIP: 10.0.2.20
What we have done until now
We have tried to create multiple kubernetes clusters with @cool-thailand-26552 cluster api module. It creates the machines successfully and a LB in harvester. VMs are then places in the hypervisor network (10.0.0.0/24) and with a LB set to ip-pool with range 10.0.0.31-40, gateway 10.0.0.1. Machines boots and starts as expected though LB wont accept or forward any traffic to the vms in the hypervisor (vm-network/vlan id 4).
So we have been puzzled with this so I started to reverse my tracks and start over thinking. Well does it forward a traffic at all if I just create a vm in hypervisor network and install nginx and create a LB in hypervisor (vlan id4) with the same ip pool as we tried with the kubernetes clusters and no it dosent foward it
though here is the interesting part
so if I create another LB, but this LB is set to the mgmt network 10.0.2.0/24, THEN it transfers that traffic to the VM in the hypervisor network as expected though this LB is on the mgmt network and not hypervisor network.
If that’s the case, the Harvester Load Balancer currently does not support assigning LB IP addresses to secondary interfaces. It will always bind the LB IP addresses regardless of what subnets are to the management interface, i.e., 
mgmt-br
. So, for this kind of VM-type load balancer usage, it’s required to create LBs from the IP pool with the same subnet range as the Harvester nodes’ management network.
The other way is to move the LB IP address inside the target VMs. If they are a guest cluster that has the Harvester Cloud Provider running, you can create an LB-type Service to announce the LB IP address on the 
10.0.0.0/24
subnet.
So to understand you correctly, to get this to work we would need to use a Rancher server, connect it to Harvester with the Harvester Cloud Provider and use that path to provide a Kubernetes cluster to get a working with a LB in a different network than the MGMT network ?
r
so if I create another LB, but this LB is set to the mgmt network 10.0.2.0/24, THEN it transfers that traffic to the VM in the hypervisor network as expected though this LB is on the mgmt network and not hypervisor network.
This is the way to go. Traffic destined for 
10.0.0.30
will never be routed to the 
mgmt-br
interface.
Or like I mentioned, with Harvester Cloud Provider, you can bind the VIP address in the guest cluster. But I’m unsure how you provisioned the Kubernetes clusters. I guess you need to manually install it.
a
I could do this at my homelab, but if my workplace was to consider Harvester in our datacenters, this is something we would never accept when we need to change out vmware vsphere.
c
What exactly is not acceptable @adventurous-portugal-91104 ?
a
If we are forced to use the mgmt network of the physical nodes to LB any workload lets say that are not of type kubernetes guest LB into the same network as the nodes live. that would be exposing us to very high risk.
c
My understanding is that using ipPools would circumvent that ... am I right @red-king-19196 ?
a
my Network and Security Engineer at my current company would never accept that just saying
if you guys want to target all the ex-vmware customers to grab that business this needs to be fixed
though in a way for just kubernetes api it could be acceptable for my home lab to run this in mgmt network but a serous enterprise would never accept it. though I have one question if I then get this cluster up and I want to have a couple of services (ofcourse) for some workload to get a L2 vip for exposure, would this work as a guest kubernetes workload lb in say my vm-network (hypervisor: 10.0.0.0/24)?
or would this end up getting the same issue and has to have an endpoint also in the mgmt network?
s
What is the use of the feature to select alternative ip pools then that can be bound to alternate NICs then when creating a lb?
r
My understanding is that using ipPools would circumvent that
It wouldn’t. It’s a connectivity issue. Packets won’t be routed to the right interface where the LB IP address is bound.
What I suggested is like below:
c
Ok so it looks like it worked for me on Equinix Metal using Metal Gateways because, somehow, the 
mgmt-br
Interface was routed to the Metal gateway subnets....
r
Sorry, this one is more clear (not in dark mode)
Ok so it looks like it worked for me on Equinix Metal using Metal Gateways because, somehow, the 
mgmt-br
Interface was routed to the Metal gateway subnets....
If the VLANs and routing rules are configured appropriately, it would work 👍
a
What is needed to configure the VLANs and routing appropriately ? I run Ubiquiti network stack at home with decent vlan and networking configuration options.
btw your drawing is exactly what I want but it dosent work by using a LB IP on 10.0.0.30/32 I would have to put it on 10.0.2.x/32 to work
r
You’ll need to add a static route for network 10.0.0.30/32 to go to the first NIC of the Harvester node on the router. I don’t have a complex environment to check if that’s all you need to do.
a
As Endre also wrote, its weird we can create a vm-network on 10.0.0.x/24, vlan tag 4 on my secondary nic. and all VMs get correct IP config and routing works but when you try and add a LB like you do with a VM it dosent work. Could a idea here to be able to assign which NIC/Network this LB should be published to like a VM? and not only a Ip Pool
You’ll need to add a static route for network 10.0.0.30/32 to go to the first NIC of the Harvester node on the router. I don’t have a complex environment to check if that’s all you need to do.
I will check up on this.
r
The drawing is to use the “cluster-type” LB instead of the “VM-type” LB. The orange-dotted arrow means moving the LB IP address into the guest cluster.
a
aaaah
right
r
Is that acceptable?
a
sure, I will try and implement this with RKE2 and Rancher type deployment of kubernetes and see if it works 🙂
r
This should work out of the box if you use the Rancher Integration to spin up an RKE2 guest cluster.
s
What does s3tting the type really do under the hood?
r
Do you mean the LB type or the IPAM mode?
s
lb type
r
The difference is that “cluster-type” LB is for LB-type of Service objects on the guest cluster to work. The actual LB IP address lives in the guest cluster nodes (the VMs); The “VM-type” LB is more general-purpose. The LB IP address is bound to the Harvester nodes, not inside the VM.
s
Inside the VM network you mean?
Does that mean that cluster type lb needs to have kube-vip inside a given guest cluster,
r
Exactly
s
that is what rke2 does,
?
r
harvester cloud provider comes with kube-vip as dependency
Do you have the Harvester cluster imported into the external Rancher?
a
No not yet, that is the next step
one question we are able to use say Cilium as CNI in here ? I know for a standard rancher deployment not on harvester it is possible.
s
I sont get it why You need to have the CPI if you have cilium or k-vip or mlb running in l2 mode here?
c
CPI (in general, not only for Harvester) is also needed for node 
ProviderID
tagging and labelling with availability zones. For instance, without CPI on a workload cluster, CAPI is not able to recognize a 
Machine
as being 
Ready
(CAPI connects to the Workload cluster API, checks the 
Node
objects and matches them with 
Machine
object in the Management Cluster, and bubbles up the 
ProviderID
to the 
Machine
Object).
👍 1
r
one question we are able to use say Cilium as CNI in here ? I know for a standard rancher deployment not on harvester it is possible.
Yes, you can choose Cilium as the CNI plugin during cluster creation.
I sont get it why You need to have the CPI if you have cilium or k-vip or mlb running in l2 mode here?
From the LB point of view, it also allows you to allocate IP addresses for LB-type service from the IP pools managed by the underlying Harvester cluster. You can still use kube-vip or metallb in the guest Kubernetes cluster, it’s just that you’ll need to decide and manage what IP addresses to be allocated for LB services.
5 Views
Open in Slack
PreviousNext
Powered by