This message was deleted.

1. Yes, Harvester is production ready. 2. Harvester would take the place of vSphere. It is a hypervisor in itself. You could use Rancher to provision nodes dynamically, or create something yourself. 3. It is free, but has a paid support model if you want enterprise support.

@swift-eve-48927 great thanks for your help. Is there any video or docs about how harvester integrates with vsphere. thanks again.

To "just" provision on vsphere you don't need harvester. Rancher can do this with the vsphere cloud provider. But as Andrew has said, you could instead of vsphere use havester on your bare metal servers and than use rancher to directly provision your vms (either as kubernetes nodes or regular vm workloads) on harvester. To dynamically do clusters you can than again use rancher with the harvester cloud provider

@miniature-lock-53926 great thanks for your help. quite a lot of useful information. would nested visualization hurt performance significantly. I agree with you that harvester is quite a young project, so I decided to use it on top of the vsphere. one headache thing is that it seems quite hard to enable both nested virtualization and pci passthrough. esxi support much mature features for vm and network management so we could not drop it. e.g. BIOS tunes, vswitch configurations.

well actually pci passthrough is marked as experimental yet and that is an ongoing issue i personally have with using passthrough nvmes on our hardware https://github.com/harvester/pcidevices/issues/57

@miniature-lock-53926 so combine esxi with harvester could be a good idea for the current stage. e.g. minio cluster is outside the harvester and joined the rancher k8s cluster as a vm node directly on the esxi, and it could passthrough disks.

so you mean no vCenter in the mix?

yes, vCerter takes the control of managing virtual machines for installing harvester node and those who needs pci passthrough. while harvester is responsible for creating k8s nodes using the resources that vcenter gives him.

ah sorry I was confused... yes (big) harvester VMs on top of esxi nodes with drives passthrough to the harvester vms which then span the longhorn cluster between the harvester vms should be possible

vSAN is expensive, so in this way, we could take vSAN away, and use the local datastore of every esxi node.

ah yes that's what I was thinking about... because yes that is true

yes, performance loss will be a problem. esxi -> harvester -> k8s node. we are using kubevirt directly on top of rke2. it’s esxi -> k8s node -> virtual machine. all k8s nodes will suffer a performance loss because of a additional layer of virtualization. I don’t know if there will be a performance benchmark report about this.

if you want to do hci directly on vsphere but are not willing to pay for vSAN (that what we tried first) you can do that, but on big problem than is, that you can you the vsphere cloud provider but you have to again decide where to put your new nodes because you then cannot say to rancher when spinning up new nodes to use a shared storage system for the root vms and thus always have to device which local datastore you use and then also where the new vms have to run... so you actually dont get the automaticall scheduling an such

thanks for your reply. you mentioned that harvester is a k3s cluster, so is it possible to use this k3s cluster directly to deploy applications. I found it recommend that user should integrate with rancher and provision cluster on top of it.

Yeah it is possible, allthough not supported and discouraged to do that in the docs. I would not use the internal k3s-cluster for more than debugging harvester itself and MAYBE to host some small cluster-internal services that you might want to integrate like maybe external secrets provider or the like. You should setup a management HA-Rancher-Cluster ontop of the the harvester nodes and then import the Harvester Cluster into this Rancher and you that as your Cluster Orchestration. Everythin else would also not be supported AFAIK

thanks for your reply. setup a dedicated rancher cluster would be a great idea for achieving HA rather than set up it in docker. what I am confused is about the concept of

HCI

, which I think virtual machines should be combined more tightly with kubernetes pods. For example, in the usecase of

kubevirt

, virtual machines could be treated as a pod / service, so some old and near deprecated program could also be migrated into kubernetes cluster with less modification, as kubevirt makes virtual machines into pods and thus no matter virtual machines or pods, they could be managed in the same abstraction thus make the management easier. that’s what I think is the problem HCI want to deal with. (I am not sure if I understand it correctly) However, in harvester, virtual machines are outside the kubernetes cluster that in use. so I wonder how to use some virtual machines to serve some old programs that provide services for kubernetes pods.

I mean you could also just put the VMs in the same VMNetworks as you k8s Nodes and then use e.g. externalname services to provide a way to get to the services running on the kubevirt/harvester vms from inside downstream k8s-clusters, i'd think

I don’t know and exercise that practice. k8s makes operation much simpler, that is to say, pods could run and restart at any where and so as services, k8s will take care of that migration while I don’t if a vm goes down, will harvester brings up a new one on another host and keep that ip unchanged.

Yeah but Harvester will do that too for those VM Workloads that are running directly on the harvester/k3s/kubevirt stack

yep. to fulfill my needs,

vsphere <-> k8s <-> kubevirt vm

is better. I want to migrate this architecture to harvester, but I found it not fit my case well.