I opted-in to deprecated K8s version v1.24.9+rke2r2 because it was necessary to enable Harvester as cloud provider. I am using Rancher 2.7.1 with Harvester 1.1.1.

It's the same as simply spinning up a VM in harvester and attaching it to an appropriate network. I'd do that to troubleshoot why VMs aren't getting DHCP addresses

Also check with your harvester cluster config in rancher you're supplying an appropriate network-data config to set the VM interface to dhcp

The network-data didn't seem to be necessary when booting pfSense ISO nor the Ubuntu Cloud Minimal .img (qcow2?), but it may in fact be necessary when creating a cluster with for the Harvester cloud provider using the Tumbleweed Cloud Minimal qcow2 image.

To simplify, I reproduced the same (probable) cause with a single VM on the same untagged VM network with the same Tumbleweed Cloud Minimal image. Without any network-data,

eth0

does not receive an IP lease. I made a second VM to test supplying the network-data on first boot, but the result was the same as follows.

version: 1
config:
  - type: physical
    name: eth0 # name is varies upon OS image
    subnets:
      - type: dhcp

I used

eth0

because it is shown in the console, so I surmise that is the correct adapter name for Tumbleweed. The DHCP server shows no leases given for the Ethernet addresses reported for these guests in Harvester.

This is the untagged VM network.

I began to suspect that Tumbleweed works differently and tried the same cloud-config directives with the Ubuntu Cloud Minimal image. I was able to configure the default user password and network devices normally. I'll need to figure out why Tumbleweed Cloud Minimal image did not self-configure with the provided cloud-config. That minor setback aside, I did discover that a VM network with VLAN 1 is able to receive DHCP leases from the external LAN, so that's an obvious good choice for creating cluster nodes through Harvester if I want them to be on that external network. I'm still trying to figure out how to isolate a VM network. When I tried to isolate by VLAN I made two unexpected observations: • layer 2 bleed from the external LAN caused my supposedly-isolated DHCP server to issue problematic IP leases to the external LAN (wrong subnet!) • a pcap of an ICMP echo request emitted by the

e1000

NIC on the VLAN showed a source IP of the cluster

mgmt

network bridge, i.e. the Harvester node's external IP

I'm at a loss. The anomalous behavior isn't limited to pfSense or those other NIC drivers. I can see the same with

virtio

on Ubuntu and LEAP. Using an Ubuntu VM as a router, the LAN client's packet is somehow mutated to have the source IP of the Harvester node's external interface. The router will forward the packet, and the response is received, but can not be routed to the LAN client because the destination of the response is incorrectly the node IP. 🤷

This might have been a wayward experiment from the beginning. I'm trying to use VMs as routers because I thought I wanted to use pfSense, which is powerful and familiar. Maybe I should be exploring ingresses instead, and keep trying to express the infrastructure as Kubernetes primitives.