https://rancher.com/ logo
#harvester
Title
# harvester
a

adamant-kite-43734

03/25/2023, 2:51 AM
This message was deleted.
r

refined-analyst-8898

03/25/2023, 2:52 AM
I used the Tumbleweed minimal cloud image and chose the untagged VM network which I've used previously, successfully with the DHCP server in my lab LAN.
I opted-in to deprecated K8s version v1.24.9+rke2r2 because it was necessary to enable Harvester as cloud provider. I am using Rancher 2.7.1 with Harvester 1.1.1.
a

agreeable-oil-87482

03/25/2023, 8:46 AM
Looking at that screenshot it looks like eth0 doesn't have an IP address
r

refined-analyst-8898

03/25/2023, 2:13 PM
Now I see this intricate procedure for using the Harvester Node Driver. I didn't expect so many steps! I had naively selected the Harvester cloud provider and assumed that assigning the guest cluster's nodes to the untagged network bridged with the external LAN would allow them to receive IP leases from the external DHCP server. I reckon there is more preparation needed, and a more tailored netplan for those guest cluster node VMs. https://docs.harvesterhci.io/v1.1/rancher/cloud-provider/
a

agreeable-oil-87482

03/25/2023, 2:15 PM
Yes that's how it should work and does for my environment. My harvester VMs get addresses via DHCP from the respective bridge network
It's the same as simply spinning up a VM in harvester and attaching it to an appropriate network. I'd do that to troubleshoot why VMs aren't getting DHCP addresses
Also check with your harvester cluster config in rancher you're supplying an appropriate network-data config to set the VM interface to dhcp
r

refined-analyst-8898

03/25/2023, 2:18 PM
Perplexingly, I had no trouble with that experiment. I played with a few different images and was able to obtain leases on that same untagged VM network that is bridged with the cluster mgmt network. I believe you just suggested the key, missing piece for my case. I did not provide any network-data when creating the guest cluster.
The network-data didn't seem to be necessary when booting pfSense ISO nor the Ubuntu Cloud Minimal .img (qcow2?), but it may in fact be necessary when creating a cluster with for the Harvester cloud provider using the Tumbleweed Cloud Minimal qcow2 image.
To simplify, I reproduced the same (probable) cause with a single VM on the same untagged VM network with the same Tumbleweed Cloud Minimal image. Without any network-data,
eth0
does not receive an IP lease. I made a second VM to test supplying the network-data on first boot, but the result was the same as follows.
Copy code
version: 1
config:
  - type: physical
    name: eth0 # name is varies upon OS image
    subnets:
      - type: dhcp
I used
eth0
because it is shown in the console, so I surmise that is the correct adapter name for Tumbleweed. The DHCP server shows no leases given for the Ethernet addresses reported for these guests in Harvester.
This is the untagged VM network.
I began to suspect that Tumbleweed works differently and tried the same cloud-config directives with the Ubuntu Cloud Minimal image. I was able to configure the default user password and network devices normally. I'll need to figure out why Tumbleweed Cloud Minimal image did not self-configure with the provided cloud-config. That minor setback aside, I did discover that a VM network with VLAN 1 is able to receive DHCP leases from the external LAN, so that's an obvious good choice for creating cluster nodes through Harvester if I want them to be on that external network. I'm still trying to figure out how to isolate a VM network. When I tried to isolate by VLAN I made two unexpected observations: • layer 2 bleed from the external LAN caused my supposedly-isolated DHCP server to issue problematic IP leases to the external LAN (wrong subnet!) • a pcap of an ICMP echo request emitted by the
e1000
NIC on the VLAN showed a source IP of the cluster
mgmt
network bridge, i.e. the Harvester node's external IP
I'm at a loss. The anomalous behavior isn't limited to pfSense or those other NIC drivers. I can see the same with
virtio
on Ubuntu and LEAP. Using an Ubuntu VM as a router, the LAN client's packet is somehow mutated to have the source IP of the Harvester node's external interface. The router will forward the packet, and the response is received, but can not be routed to the LAN client because the destination of the response is incorrectly the node IP. 🤷
This might have been a wayward experiment from the beginning. I'm trying to use VMs as routers because I thought I wanted to use pfSense, which is powerful and familiar. Maybe I should be exploring ingresses instead, and keep trying to express the infrastructure as Kubernetes primitives.
b

bumpy-portugal-40754

03/27/2023, 11:03 PM
Rancher 2.7.1 is not (yet) supported with Harvester 1.1. Did you try 2.6.11?
r

refined-analyst-8898

03/27/2023, 11:03 PM
Ohhhhh. No, I naively installed latest. I will try latest 2.6. Thanks.
b

bumpy-portugal-40754

03/27/2023, 11:05 PM
Just an idea... I don't know if it makes a difference, but it's worth a try.
r

refined-analyst-8898

03/28/2023, 3:18 AM
It did make a difference. 2.6.11 rolled the nodes after a cluster config change without stuck pods left behind, and auto-selects the default Harvester storage class whereas 2.7.1 installed a node driver that would always pick the storage class named "longhorn" even though it's not the default. Now I just need to figure out how to select a particular Harvester storage class. The cloud provider chart adds one "harvester" class in the guest cluster, but there are several classes in Harvester I need to use, not just the default class.
👍 1
b

bumpy-portugal-40754

03/28/2023, 8:42 AM
Yeah, this is also not yet clear to me... but it's new in Harvester 1.1 and not very well documented. In a guest cluster the harvester storage class is configured and working fine.
132 Views