Hi guys, We have some customers that run mostly GPU heavy tasks and would like to monitor the usage of each NVidia GPU. Any software/tools recommendations for monitoring VMs (Ubuntu Server) that are using GPUs through PCI passthrough? I would like to avoid installing active agents (e.g. Zabbix active agent) into the VMs, if possible. Thanks

I'm not really my question should come to group harvester or rancher. But how can I force removing the clusters that I no longer need? They seemed to be stuck as below for a long time, after I click deleting them.

My RKE2 cluster targeting on Harvester is not ready. It's stuck on provisioning as in below logs Where can I first to look into?

[INFO ] waiting for viable init node
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, kubelet
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, kube-apiserver
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) cluster08-pool08-69c7d74894-776ff: waiting for probes: calico, kube-controller-manager, kube-scheduler

One of my VM's is stuck in a terminating state, any idea on how to fix it?

After rebooting my harvester node, I am unable to restart a VM. I see the following error, but not sure what caused this.

Events:
  Type     Reason              Age                    From                     Message
  ----     ------              ----                   ----                     -------
  Normal   Scheduled           5m57s                  default-scheduler        Successfully assigned default/virt-launcher-sle-15-sp4-base-9zjw2 to harvester-01
  Warning  FailedMount         3m54s                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[disk-0 disk-1], unattached volumes=[private disk-0 cloudinitdisk-ndata container-disks hotplug-disks sockets cloudinitdisk-udata public ephemeral-disks libvirt-runtime disk-1]: timed out waiting for the condition
  Warning  FailedAttachVolume  102s (x10 over 5m57s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-87fe9694-9c15-49fd-adc5-2bbe7337ce85" : PersistentVolume "pvc-87fe9694-9c15-49fd-adc5-2bbe7337ce85" is marked for deletion
  Warning  FailedAttachVolume  102s (x10 over 5m57s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-e9108404-e61d-402a-a553-62610bd987a9" : PersistentVolume "pvc-e9108404-e61d-402a-a553-62610bd987a9" is marked for deletion
  Warning  FailedMount         99s                    kubelet                  Unable to attach or mount volumes: unmounted volumes=[disk-0 disk-1], unattached volumes=[public hotplug-disks libvirt-runtime cloudinitdisk-ndata sockets disk-0 private ephemeral-disks container-disks cloudinitdisk-udata disk-1]: timed out waiting for the condition

Did a describe on the pvc:
Name:          sle-15-sp4-base-disk-0-6oyol
Namespace:     default
StorageClass:  longhorn-image-d48sr
Status:        Bound
Volume:        pvc-87fe9694-9c15-49fd-adc5-2bbe7337ce85
Labels:        <none>
Annotations:   <http://harvesterhci.io/imageId|harvesterhci.io/imageId>: default/image-d48sr
               <http://harvesterhci.io/owned-by|harvesterhci.io/owned-by>: [{"schema":"kubevirt.io.virtualmachine","refs":["default/sle-15-sp4-base"]}]
               <http://pv.kubernetes.io/bind-completed|pv.kubernetes.io/bind-completed>: yes
               <http://pv.kubernetes.io/bound-by-controller|pv.kubernetes.io/bound-by-controller>: yes
               <http://volume.beta.kubernetes.io/storage-provisioner|volume.beta.kubernetes.io/storage-provisioner>: <http://driver.longhorn.io|driver.longhorn.io>
               <http://volume.kubernetes.io/storage-provisioner|volume.kubernetes.io/storage-provisioner>: <http://driver.longhorn.io|driver.longhorn.io>
Finalizers:    [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection> <http://provisioner.storage.kubernetes.io/cloning-protection|provisioner.storage.kubernetes.io/cloning-protection>]
Capacity:      20Gi
Access Modes:  RWX
VolumeMode:    Block
Used By:       virt-launcher-sle-15-sp4-base-9zjw2
Events:        <none>


Name:          sle-15-sp4-base-disk-1-upd7u
Namespace:     default
StorageClass:  harvester-longhorn
Status:        Bound
Volume:        pvc-e9108404-e61d-402a-a553-62610bd987a9
Labels:        <none>
Annotations:   <http://harvesterhci.io/owned-by|harvesterhci.io/owned-by>: [{"schema":"kubevirt.io.virtualmachine","refs":["default/sle-15-sp4-base"]}]
               <http://pv.kubernetes.io/bind-completed|pv.kubernetes.io/bind-completed>: yes
               <http://pv.kubernetes.io/bound-by-controller|pv.kubernetes.io/bound-by-controller>: yes
               <http://volume.beta.kubernetes.io/storage-provisioner|volume.beta.kubernetes.io/storage-provisioner>: <http://driver.longhorn.io|driver.longhorn.io>
               <http://volume.kubernetes.io/storage-provisioner|volume.kubernetes.io/storage-provisioner>: <http://driver.longhorn.io|driver.longhorn.io>
Finalizers:    [<http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection> <http://provisioner.storage.kubernetes.io/cloning-protection|provisioner.storage.kubernetes.io/cloning-protection>]
Capacity:      60Gi
Access Modes:  RWX
VolumeMode:    Block
Used By:       virt-launcher-sle-15-sp4-base-9zjw2
Events:        <none>

I'm not having much luck with googling. I'm trying to install harvester, and it's just... freezing. No errors (in the installer). I used

CTRL + ALT + F2

to switch to the console TTY per the troubleshooting guide, https://docs.harvesterhci.io/v1.1/troubleshooting/installation. There is a default route, and working DNS, and I can ping and resolve things such as google.com. But it seems that K3S just isn't installed so everything related to docker is throwing an error, expecting k3s to exist? The troubleshooting doc has nada to say about this issue.

How can I persist the following iptables command across reboot on harvester?

iptables -t nat -A POSTROUTING -o mgmt-br -j MASQUERADE

Can someone explain to me what the "VM Data" is in the installer? It's not entirely clear. I have 1 SSD that I plan to use for the OS, and then 5 HDDs I plan to use for longhorn. It's not clear if the "VM Data" means "add this disk to longhorn" or if it's entirely seperate. I didn't plan my server layouts with enough disks to give 2 of the to the OS install

Hi, I need to reconfigure the DNS servers of my Harvester cluster's nodes (DNS servers are being migrated), is this possible without rebooting the nodes ? Those are old repurposed servers that do not have remote OOB management capability and I would really like to not have to go to the datacenter 🙂

Anyone know how to persist a second IP address on a given interface across harvester reboot. I tried adding the IP address to /etc/sysconfig/network/ifcfg-mgmt-br but that did not work.

What’s the difference between ‘cluster networks’ and ‘vm networks’ in Harvester UI?

set the channel description: https://github.com/harvester/harvester

set the channel topic: Latest release: v1.1.1 https://github.com/harvester/harvester/releases/tag/v1.1.1

in my flurry of frustration to get an external rancher manager provisioning rke2 nodes on harvester, combined with a brief power outage last night... I seem to have forgotten my harvester password. I still have ssh access to the nodes. I am trying to reset the password. Can someone tell me what user this resets the password for? hv01:/home/rancher # kubectl -n cattle-system exec $(kubectl --kubeconfig $KUBECONFIG -n cattle-system get pods -l app=rancher --no-headers | head -1 | awk '{ print $1 }') -c rancher -- reset-password W0301 013437.635364 64 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.New password for default admin user (user-wlmcg): j7PZznhNATc8LzzFPSf6 admin used to be the user, is user-wlmcg the user I should be using now? Is there a better way to reset the 'admin' account?

wait, is the harvester ISO not big enough to fit on a standard DVD?

I must be missing something obvious. I could not get a single new vm to work. Using the latest Harvester version with 3 powerful physical test nodes here. I created a new image in Harvester UI with URL from https://cloud-images.ubuntu.com/minimal/releases/focal/release/ubuntu-20.04-minimal-cloudimg-amd64.img, I think it is the same as Harvester doc. I created a new vm vlan (could not get dhcp to work, and went with manual/static, with CIDR and Gateway IP configured, with L2VlanNetwork on default mgmt cluster network). The new VM boots up, I could see login prompt in its UI serial console. But the VM did not get any IP assigned. Is this expected?

Hello, I'm trying to provision a single-node rke cluster with rancher using a bare-metal harvester, but I tried several times and I'm always stuck at

waiting for cluster agent to connect

... I was able to log into the VM, rke2-server and rancher-system-agent are both running and file, and I'm able to curl the rancher https endpoint (I'm using self-signed certificates). What am I missing and any idea on how I can debug this ?

I'm trying to get VM Import for VMWare working following the doc at https://docs.harvesterhci.io/v1.1/advanced/vmimport/ but I get this after setting up the source

harvester-management:/home/rancher # kubectl get vmwaresource.migration
NAME    STATUS
vcsim   clusterNotReady

Hello, my cluster has only two nodes named harv04 and harv05, it will fail when uploading image with large qcow2 file (greater than 50GB), the error message is "Pod spec node ID harv05 doesn't match the desired node ID harv04". How can I reslove this?

Hello, my cluster has only two nodes named harv04 and harv05, it will fail when uploading image with large qcow2 file (greater than 50GB), the error message is "Pod spec node ID harv05 doesn't match the desired node ID harv04". How can I resolve this?

ASK HW INFO: -Xeon with 500GB SSD 64GB RAM GOAL: 1. simulate vm motion between node 2. simulate trunk port without lacp. separate vm vlan and storage vlan STEPS: 1. install parent harvester 2. since I only have 1 server. create 3 harvester vms as child vm

k keep having issues w/ my cluster totally freaking out, vms locking up, and all of my replicas going nuts... i just took a support bundle after it happened, can someone take a look and lmk where to flippin start, it's driving me nuts! i dont know if it's a node, i dont know if it's a switch, i'm totally at a loss here.

ASK https://www.docker.com/pricing there it said Docker Desktop Business must paid $24/user/month but it didn't said Docker Server ? 1. so do I need to pay for Docker Server Business

ASK HW INFO: 3x Servers with Alpine Linux+Kubernetes Cluster+Podman 1x PC with Ubuntu LTS 22.10+Rancher+Docker I read somewhere Rancher need Docker If Docker Server Business must paid and I want to save budget ? 1. Can I use podman in K8 cluster and not using Docker at all but using Docker only for Rancher PC

Having a bit of an issue with a cluster/ storage network not coming up properly after rebooting a node here;

[   76.204319] A link change request failed with some changes committed already. Interface vm-br may have been left with an inconsistent configuration, please check.

controller:
time="2023-03-10T20:03:15Z" level=error msg="error syncing 'har-04': handler harvester-network-vlanconfig-controller: set up VLAN failed, vlanconfig: har-04, node: har-04, error: ensure bridge vm-br failed, error: set vlan filtering failed, error: invalid argument, iface: &{0xc0006de140}, requeuing"
I0310 20:03:15.252249       1 controller.go:75] vlan config har-04 has been changed, spec: {Description: ClusterNetwork:vm NodeSelector:map[<http://kubernetes.io/hostname:har-04|kubernetes.io/hostname:har-04>] Uplink:{NICs:[eno1 eno2] LinkAttrs:0xc001c85e30 BondOptions:0xc00052ad50}}

The associated bond is brought up just fine, any ideas what could be causing such a scenario?

anyone know of a way to map COM-1 serial port from the host to a vm? i cant seem to find it in PCI Devices

Howdy all! New Harvester user here (just a HomeLab, nothing huge). Just wanted to say hi…. super excited to test this beast out! Wondering if anyone has some “unofficial” harvester related docs or writeups you’d offer as quality material? I’m pretty comfortable in K8s and various cloud providers but haven’t self hosted beyond Rancher Desktop for local dev and my little k3d box over in the corner. Ideally my hope is to manage this whole setup in Terraform/packer/kubernetes native deployment tools, but right now i’m just figuring out the basic stuff - like where do i dig up schema on a harvester config file… Scratch that just found it Any Harvester newb suggestions welcome. I’m all ears!

anyone know how i can manually start a VM? my ui is not wanting to load.

also, how to reboot the ui! 🙂