This message was deleted.

I think that the hint here is that n0 isn’t showing as control-plane as it was setup as a witness node on install, this I think is pointing toward an issue with that machine.

Hi Craig, I'm looking at your support bundle and I found a few of these error messages:

logs/kube-system/rke2-metrics-server-7f745dbddf-crckz/metrics-server.log:2024-09-12T13:11:05.801669145Z E0912 13:11:05.801609       1 scraper.go:149] "Failed to scrape node" err="Get \"<https://192.168.122.134:10250/metrics/resource>\": dial tcp 192.168.122.134:10250: connect: no route to host" node="n0"                                                                                                                          logs/kube-system/rke2-metrics-server-7f745dbddf-crckz/metrics-server.log:2024-09-12T13:11:20.777705961Z E0912 13:11:20.777630       1 scraper.go:149] "Failed to scrape node" err="Get \"<https://192.168.122.134:10250/metrics/resource>\": dial tcp 192.168.122.134:10250: connect: no route to host" node="n0"                                                                                                                          logs/kube-system/rke2-metrics-server-7f745dbddf-crckz/metrics-server.log:2024-09-12T13:11:35.817668323Z E0912 13:11:35.817610       1 scraper.go:149] "Failed to scrape node" err="Get \"<https://192.168.122.134:10250/metrics/resource>\": dial tcp 192.168.122.134:10250: connect: no route to host" node="n0"                                                                                                                          
[...]
logs/kube-system/rke2-metrics-server-7f745dbddf-crckz/metrics-server.log:2024-09-15T14:52:20.809677447Z E0915 14:52:20.809617       1 scraper.go:149] "Failed to scrape node" err="Get \"<https://192.168.122.134:10250/metrics/resource>\": dial tcp 192.168.122.134:10250: connect: no route to host" node="n0"                                                                                                                          logs/kube-system/rke2-metrics-server-7f745dbddf-crckz/metrics-server.log:2024-09-15T14:52:35.789668865Z E0915 14:52:35.789605       1 scraper.go:149] "Failed to scrape node" err="Get \"<https://192.168.122.134:10250/metrics/resource>\": dial tcp 192.168.122.134:10250: connect: no route to host" node="n0"                                                                                                                          logs/kube-system/rke2-metrics-server-7f745dbddf-crckz/metrics-server.log:2024-09-15T14:52:50.765663858Z E0915 14:52:50.765595       1 scraper.go:149] "Failed to scrape node" err="Get \"<https://192.168.122.134:10250/metrics/resource>\": dial tcp 192.168.122.134:10250: connect: no route to host" node="n0"

Seems like your node n0 has somehow disappeared at some point on September 12th? Could you please describe what the underlying hardware of your setup is, and how history with n0 being a witness node went down? I gathered from your earlier messages that you had a two-node setup with n0 as witness node. Did you then first try the upgrade or first add the other nodes?

n0 is the node we've removed from the cluster - it was a witness node used initially, I think that machine wasnt great and it clearly got demoted as part of the control-plane as 3 other machines were working as such. Is there a correct way to clean up an old node other than just deleting it from the UI?

I figured that, but I'm still trying to wrap my head around the order in which things happened. n0 was a witness node. I assume n4 and n5 were the two control planes at that point. Then you added more nodes and it seems n1 got promoted to control plane as well. Was this intended or an accident?

yes - basically n4/5 were taken from a previous 1.2 cluster and we added the witness node so we could run latest version alongside the old cluster which had failed to upgrade. Once all the workloads were moved over we then reset, upgraded the old hardware and pulled the other nodes across. We also updated the network so they all have a 10GSPF+ for management and one dedicated for storage to help improve the stability of big file operations.

So at one point you had two entirely separate v1.3.1 clusters and tried to join them into one cluster? I'm quite sure this operation isn't supported at all. Or did I misunderstand that?

no - you misunderstand - we had a 1.2 cluster and a 1.3 cluster - we decommissioned the 1.2 cluster and moved the workloads over by rebuilding the VM's on the new cluster and syncing as that 1.2 cluster was broken and we ran out of patience given we had enough machines to run 2 harvester clusters at the same time if we used a witness node with the new harvester cluster. Disabling metrics still fails to pass the admission web-hook btw. I guess I need to look at what it's checking for admission... not sure where to look atm.

The managed chart (

<http://ManagedChart.management.cattle.io/v3|ManagedChart.management.cattle.io/v3>

) named

harvester

in the

fleet-local

namespace is not ready:

status:
    conditions:
    - lastUpdateTime: "2024-09-05T01:42:17Z"
      message: Modified(1) [Cluster fleet-local/local]; <http://storageclass.storage.k8s.io|storageclass.storage.k8s.io>
        harvester-longhorn missing
      status: "False"
      type: Ready
    - lastUpdateTime: "2024-09-05T01:42:17Z"
      status: "True"
      type: Processed
    - lastUpdateTime: "2024-09-15T15:36:20Z"
      status: "True"
      type: Defined

Looks like it expects there to be a storage class named

harvester-longhorn

(which is there by default, I think), but in your setup that storage class has been removed. Perhaps you can move the upgrade along by just adding such a storage class?

i'll try that tonight - thanks for the spot!

Hey Craig, seems like something went wrong with adding that storage class. The managed helm chart is still showing the same status, even in the new support bundle:

status:
    conditions:
    - lastUpdateTime: "2024-09-05T01:42:17Z"
      message: Modified(1) [Cluster fleet-local/local]; <http://storageclass.storage.k8s.io|storageclass.storage.k8s.io>
        harvester-longhorn missing
      status: "False"
      type: Ready
    - lastUpdateTime: "2024-09-05T01:42:17Z"
      status: "True"
      type: Processed
    - lastUpdateTime: "2024-09-16T19:31:20Z"
      status: "True"
      type: Defined

The cluster

local

in Rancher is the cluster that Rancher runs on. This isn't the same cluster as Harvester. You can add the storage class from the Harvester UI directly. If you access Harvester through Rancher, you can navigate to the Harvester UI by

Virtualization Management

->

Harvester Clusters

->

select your Harvester cluster

.

Humm - harvester storage classes include the harvester-longhorn -

allowVolumeExpansion: true
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
kind: StorageClass
metadata:
  annotations:
    <http://field.cattle.io/description|field.cattle.io/description>: origional default storage
  creationTimestamp: '2024-09-16T16:01:11Z'
  managedFields:
    - apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
      fieldsType: FieldsV1
      fieldsV1:
        f:allowVolumeExpansion: {}
        f:metadata:
          f:annotations:
            .: {}
            f:<http://field.cattle.io/description|field.cattle.io/description>: {}
        f:parameters:
          .: {}
          f:diskSelector: {}
          f:migratable: {}
          f:numberOfReplicas: {}
          f:staleReplicaTimeout: {}
        f:provisioner: {}
        f:reclaimPolicy: {}
        f:volumeBindingMode: {}
      manager: harvester
      operation: Update
      time: '2024-09-16T16:01:11Z'
  name: harvester-longhorn
  resourceVersion: '52841563'
  uid: 1c9f79e3-9a75-44d6-bfca-eba4b7ad9d9b
parameters:
  diskSelector: hdd
  migratable: 'true'
  numberOfReplicas: '3'
  staleReplicaTimeout: '30'
provisioner: <http://driver.longhorn.io|driver.longhorn.io>
reclaimPolicy: Delete
volumeBindingMode: Immediate

I had the same issue, but got instructions by @clean-cpu-90380 https://github.com/harvester/harvester/issues/6561#issuecomment-2351938832 You have to fetch the manifest and apply it so that you get the correct annotations.

this sounds like its on the money - will try this later this evening and let you know the results!