rancher-users #harvester

bright-fireman-42144

12/27/2022, 6:26 PM

playing around with this spaghetti mess of rancher manager and harvester. I have managed to integrate another node to my overall harvester cluster but I am slimming down the requirements of that harvester cluster by installing rancher 'off cluster' and running into this when trying to provision both RKE2 and K33 nodes. /etc/rancher/ssl ls: cannot access 'docker-machine-driver-*': No such file or directory downloaded file failed sha256 checksum download of driver It gets as far as creating those nodes/VMs but then goes into deleting them. I am able to download the docker-machine-driver from the rancher URL. Self signed certs. Thoughts?

sticky-summer-13450

12/28/2022, 4:17 PM

Copy code

resource "harvester_virtualmachine" "ubuntu2204" {
  name                 = "ubuntu2204"
  namespace            = "default"
  restart_after_update = true

  description = "test ubuntu2203 raw image terraform"
  tags = {
    ssh-user = "ubuntu"
  }

  cpu    = 2
  memory = "2Gi"

  efi         = false
  secure_boot = false

  run_strategy = "RerunOnFailure"
  hostname     = "ubuntu2204"
  machine_type = "q35"

  network_interface {
    name           = "nic-1"
    network_name   = "vlan648"
    model          = "virtio"
    type           = "bridge"
    wait_for_lease = false
  }

  disk {
    name        = "rootdisk"
    size        = "10Gi"
    type        = "disk"
    bus         = "virtio"
    boot_order  = 1
    image       = "jammy-server-cloudimg-amd64.img"
    auto_delete = true
  }

  cloudinit {
    user_data    = <<-EOF
      #cloud-config
      EOF
...
    network_data = <<-EOF
      network:
        version: 1
        config:
...
      EOF
  }
}

best-hair-19862

12/28/2022, 9:53 PM

Hello everyone and Happy Holidays! I've just seen that there's a dedicated channel for Harvester so I share a previous message with you. TL:DR, I accidentally deleted the default PV for the Prometheus application and it doesn't automatically recreate. What should I do?

quaint-alarm-7893

12/30/2022, 6:05 PM

Any recommendations for monitoring Physical Disk Health in harvester? i MIGHT have a drive going bad, i'd like to run some checks on it if possible.

salmon-afternoon-72196

12/31/2022, 6:16 PM

Happy New Year to those on the other side of the world.....I recently install Harvester on my R630, I had to install using Ventoy and USB image....I have noticed that after I create a VM and I connect with WebVNC (Safari), caps lock is behaving backwards, in other words, once I connect, as soon as I touch the keyboard CAPS Lock turns on even though my keyboard (MacBook Pro) shoes Caps Lock off. I have to turn Caps Lock on if I want to type in lower case. Has anyone else seen this?

billions-diamond-53716

01/01/2023, 3:58 AM

Happy New Year All! I have a question, I have a 3 node cluster and one node has network interfaces named differently than the other two, so I cant attach these interfaces to Cluster Network configs. Ive tried renaming them on the host via the normal openSUSE methods, however none seem to work. Anyone had this issue? Cheers!

flat-evening-58664

01/02/2023, 12:25 PM

I've been playing with Harvester a bit on my home lab this weekend, and I have some questions based on that experience that I was hoping someone could help with: • Since I'm in a home lab the servers aren't SUPER beefy, but it seems like harvester has 50% of my 32 cores reserved without even running anything else. Is that expected? That seems really high. Is it not really worth running Harvester on lower-specced machines? I also have 14 of my 93GiB reserved (15%), but that at least leaves a lot more headroom. • I'm planning to run Rancher to manage the Harvester cluster, as well as my DO Kubernetes cluster, and it seems like there's at least some version of Rancher running "behind the scenes" of Harvester. Do I need to also create another set of VMs on the Harvester cluster to install Rancher on those as well? Is that a common pattern (managing the Harvester cluster Rancher is running on via the same Rancher)? It seems wasteful to run Rancher in order to run another Rancher. • Longhorn is running on the Harvester cluster, and I was planning to run Longhorn already myself. Is it typical for people to expand Harvester's Longhorn with more storage and use that same Longhorn cluster as a target from a Kubernetes cluster that's running in Harvester VMs? Should I instead be running another Longhorn cluster inside Harvester VMs to be the "Guest Longhorn"? If I were to set up a separate Longhorn cluster (say, bare metal or something) would I be able to use that for Harvester? • Is there any way for me to easily verify Harvester is using the correct NICs for its storage network? I was looking at the IPs to see if it was using the CIDR I provided, but it looks like it's using overlay network IPs? • I'm a big fan of config management and versioned config files for everything (I tend to forget how I set something up so having configs to redo it is great), can I use that with Harvester (and I guess by extension Rancher)? Can I write YAML manifests to define all the VMs/volumes/images/networks/etc and apply those to the cluster to be able to bring up any other Harvester cluster to the same state as another (assuming the same hardware)? I had to dig into the internals a bit in order to troubleshoot some of my Longhorn/Harvester issues when one of my nodes flaked out and it was nice to see Kubernetes "under the hood", but can I also use that and apply CRDs directly to the management cluster (possibly via CI/CD)? Thanks for any help/insight you can give! 😄

bright-fireman-42144

01/03/2023, 1:18 AM

due to resource constraints I have been finding any available (extra) CPU/mem/disk in my house to cobble together a working lab. I /was/ running rancher single node on an ubunutu VM in harvester but am pulling that out 1) recommended 2) it's the right thing to do and running into the same problem I had before running a single node (docker) on a ubuntu VM on vmware workstation on my laptop. I can perform the harvester integration but when I try to create a new cluster using rancher before doing anything or hitting [create] I get: clusters.management.cattle.io "c-m-sc8xdxj7" not found

little-dress-13576

01/03/2023, 4:01 PM

Happy new Year! While reading latest harvester doc, I am very surprised to find that harvester does not support zero downtime upgrade yet. In order to upgrade harvester, all the running VMs in the whole harvester cluster need to be powered off first per doc. For any enterprise looking to adopt harvester, this would be a major red-flag. Maybe it would be ok to do zero downtime upgrade starting from latest release v1.1.1? Any ETA for harvester to support zero-downtime upgrade later?

salmon-afternoon-72196

01/03/2023, 6:27 PM

What am I missing? I setup a K3S cluster and used Harvester as the Cloud Provider, when I create a LoadBalancer it using the 3 ip's of my work nodes. I would like to be able to route based on prefix i.e. myapp.app.domain.com. I setup CloudFlare to point wildcard *.app.domain.com to on of the worker node IP's and that works but if that particular worker node crashes and gets a new IP I can no longer get to my apps and I need to update the IP in CloudFlare. Is there a way to expose a single IP that can resolve the work node IP's? TIA

wonderful-pizza-30919

01/03/2023, 9:05 PM

Happy New Year to all! When is 1.1.2 with VXLAN support is expected to be released.

quaint-alarm-7893

01/03/2023, 9:25 PM

anyone having issues exporting 100gb volumes? i keep getting to 28% and then the manager pod for longhorn craps out.

Copy code

Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running	2.2 mins ago 
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running	2.2 mins ago 
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running	2.2 mins ago

quaint-alarm-7893

01/03/2023, 9:25 PM

in the longhorn ui, the backing image its trying to create gets to 28%, then dies, and starts over

big-judge-33880

01/04/2023, 1:45 PM

Spinning up a VM here with two disks - one based on an image with

<http://harvesterhci.io/storageClassName=fast-replicas-3|harvesterhci.io/storageClassName=fast-replicas-3>

(provider Longhorn, selects disks in Longhorn with the “fast” tag) and one new volume with the same storageclass. It gets scheduled to node 5, while the volumes get attached to node 5 and 2 respectively, and the VM then fails to mount the volume that is attached to a different node. Is this a bug, or am I doing something obviously stupid?

stale-painting-80203

01/05/2023, 5:02 PM

In an RKE2 cluster running on Harvester I would like to allow the cluster to create volumes on the longhorn storage created by harvester. I was told that I can use the CSI driver as per https://docs.harvesterhci.io/v1.1/rancher/csi-driver/#deploying-with-harvester-k3s-node-driver. Unfortunately I am unable to get this to work because I created the cluster using Rancher's "Use existing nodes and create a cluster using RKE2/K3s" . Apparently if you already have an existing cluster you can use the method described for "Harvester K3s Node Driver". I am able to deploy the driver, but when the cluster tries to create the volume it fails even though I can see the correct storage driver

Copy code

kubectl describe sc
Name:                  harvester
IsDefaultClass:        Yes
Annotations:           <http://meta.helm.sh/release-name=harvester-csi-driver,meta.helm.sh/release-namespace=kube-system,storageclass.kubernetes.io/is-default-class=true|meta.helm.sh/release-name=harvester-csi-driver,meta.helm.sh/release-namespace=kube-system,storageclass.kubernetes.io/is-default-class=true>
Provisioner:           <http://driver.harvesterhci.io|driver.harvesterhci.io>
Parameters:            <none>
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

cuddly-vegetable-29975

01/07/2023, 11:15 AM

Hi, noob here. I just started my very first single node install yesterday. I made a single virtual machine and went to bed. Today, when I login to Harvester, I am told that both volumes is degraded with "replica scheduling failed" Should I just ignore this until I get some more nodes online?

salmon-bear-45866

01/10/2023, 4:27 AM

Hello! I'm playing around with Harvester, rancher, and longhorn. I have two storage classes defined in Harvester - ssd, and hdd. When I'm creating an RKE cluster with Rancher in Harvester, there's no option to select which storage class to use anywhere that I can find. The storage class being selected for the node disk is the hdd class, and that is not what I want. How do I select a storage class for my node disk in this scenario?

salmon-bear-45866

01/10/2023, 4:42 AM

I think I figured it out - sort of. It uses the same storageclass as the cloud image the node template references. Is there a way to specify a label selector to choose a different storage class for the node?

adorable-exabyte-35533

01/11/2023, 4:40 AM

I have one query before installing the harvester ( with rancher ) if my harvester os will be corrupted by any scenario so how can I recover the os also my VMS will damage if os will pollute and how can I recover all my VMS drive if the rancher or harvester is got corrupted and are there any features wich can take auto backup of VM drive in OS thank you

full-crayon-745

01/11/2023, 10:36 AM

I'm providing a support bundle for my comment in the Longhorn volume in degraded state and keep on rebuilding replica #3347 issue.

👀 1

lively-zebra-61132

01/12/2023, 2:52 PM

Hi, I am having a problem with upgrading my harvester cluster. Everything went smoothly until the first node was in the

post-draining

status. At some point during that process a vm was spawned which got stuck in Containercreating because of a problem with the longhorn volume it was trying to mount. I deleted the vm launcher, the pvc and the post-drain job (because it was making it impossible to delete the vm + pvc). Now I want to restart the upgrade but when trying

kubectl delete <http://upgrade.harvesterhci.io/hvst-upgrade-gxc4t|upgrade.harvesterhci.io/hvst-upgrade-gxc4t> -n harvester-system

it fails with Error from server (BadRequest): admission webhook

"<http://validator.harvesterhci.io|validator.harvesterhci.io>" denied the request: cluster fleet-local/local status is provisioning, please wait for it to be provisioned

is there anything I can do to fix this?

quaint-alarm-7893

01/12/2023, 3:36 PM

sounds odd, but try rebooting the node, and give it like 20 minutes.

quaint-alarm-7893

01/12/2023, 3:42 PM

Hello Everyone. i'm still stuck trying to use storage classes. on my node, the parameters pane doesnt show up. this is a node that was directly upgraded from v1.0.3 to v1.1.1. ref: https://github.com/harvester/harvester/issues/3349 i can see everything in the yaml, so if anyone has a working storage class setup, send me some example yamls so i can see how to manually configure mine! 🙂

magnificent-vr-88571

01/16/2023, 3:13 AM

Kubernetesversion: v1.22.15+rke2r1 Longhorn version: 1.3.1 I have following persistent volume and in Longhorn dashboard actual size shows 658Gi. Initially volume created with 400Gi, latter expanded to 800Gi.

Copy code

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                                                                                                             STORAGECLASS   REASON   AGE
pvc-38d9351f-8f48-435e-bc92-e796031a92ef   800Gi      RWO            Delete           Bound    kubeflow/minio-pvc                                                                                                longhorn                108d

In actuals 800Gi as allocated and 415Gi is used being shown in the volume mounted server

Copy code

>> lsblk
sdc    8:32   0  800G  0 disk /var/lib/kubelet/pods/d4ea8ad2-6cc9-4e00-bd29-25ff5e8678bc/volume-subpaths/pvc-38d9351f-8f48-435e-bc92-e796031a92ef/minio/0

Copy code

>> /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-38d9351f-8f48-435e-bc92-e796031a92ef/globalmount/minio# du -sh
415G	.

on the replica server 658G in the replica created server.

Copy code

>>/var/lib/longhorn/replicas/pvc-38d9351f-8f48-435e-bc92-e796031a92ef-2dbe6c38# du -sh
658G	.

I would like to know how we can reduce the 658G in the replica server to the actual usage as 415Gi.

sparse-apple-20577

01/16/2023, 8:35 PM

good afternoon, I am trying to create a VM for OpenManage Enterprise (dell), they provide it in OVF format, and I used qemu-img converter to make it a qcow2 image, and uploaded it to the harvester, but when trying to create a VM it errors with only this shown: {"data":""} not sure how to troubleshoot it, or perhaps there is a different way of accomplishing this task?

important-cat-93045

01/17/2023, 2:16 AM

@important-cat-93045 has left the channel

full-crayon-745

01/17/2023, 11:06 AM

Hi guys, We have 1 host with 4 Nvidia 3090 GPUs with passthrough enabled and 1 VM with these GPUs attached that was working well. Recently, I switched off the VM and restarted the host. Once, the host returned from maintenance, I started the VM. But the VM did not start and Harvester returned an error message, complaining about “Insufficient nvidia.com/GA102_GEFORCE_RTX_3090”. I created a new VM and attached the GPUs to it, but the VM will not start and I’m getting this error. combined from similar events): server error. command SyncVMI failed: "LibvirtError(Code=1, Domain=10, Message='internal error: qemu unexpectedly closed the monitor: 2023-01-17T101836.433107Z qemu-system-x86_64: -device vfio-pci,host=00008100.0,id=ua-hostdevice-GA102_GEFORCE_RTX_30901,bus=pci.8,addr=0x0: vfio 00008100.0: group 30 is not viable\nPlease ensure all devices within the iommu_group are bound to their vfio bus driver.')" Not sure what I’m doing wrong here, since the GPUs worked once already. Anyone here worked with Nvidia GPUs and successfully managed to attach the GPUs to a VM? Thanks

rhythmic-jelly-81455

01/17/2023, 12:42 PM

Hello, I've got a setup on OVH dedicated server. The VIP (51.XX.XX.208) is from a block of floating IPs attached to the server (51.XX.XX.208/30). I am able to ping the IP from within the harvester node, but not from externally. Therefore the VIP is not reachable as a management address. Kindly assist

melodic-receptionist-94532

01/18/2023, 5:47 PM

👋 is there a documented way to deploy helm charts to harvester clusters? I don't seem to see it in the docs

wonderful-pizza-30919

01/19/2023, 2:34 AM

Hi, Is it possible to create virtual cluster in Rancher like Loft does? Thank you.