Just trying this - https://docs.harvesterhci.io/v1.2/upgrade/v1-2-1-to-v1-2-2/

Following - Upgrade stuck in the

Upgrading System Service

state, from that doc - there is no node showing a certificate issue -

➜  Documents kubectl get <http://clusters.provisioning.cattle.io|clusters.provisioning.cattle.io> local -n fleet-local -o yaml
apiVersion: <http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>
kind: Cluster
metadata:
  annotations:
    <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
      {"apiVersion":"<http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>","kind":"Cluster","metadata":{"annotations":{},"name":"local","namespace":"fleet-local"},"spec":{"kubernetesVersion":"v1.25.9+rke2r1","rkeConfig":{"controlPlaneConfig":{"disable":["rke2-snapshot-controller","rke2-snapshot-controller-crd","rke2-snapshot-validation-webhook"]}}}}
    <http://objectset.rio.cattle.io/applied|objectset.rio.cattle.io/applied>: H4sIAAAAAAAA/4yQzU7DMBCEXwXt2Slt079Y4oAQ4sCVF9jYS2Ow15G9CYfK746SVqJC4udo78xovjlBIEGLgqBPgMxRUFzkPD1j+0ZGMskiubgwKOJp4eKts6ChT3F02UV2fKyMH7JQqkwiFAL1ozV+MKXqOL6DhoCMRwrEciUYa3Xz7NjePZwj/8xiDAQafDTo/yXOPZrJAUXB3NdFfnGBsmDoQfPgvQKPLflfR+gwd6Bhu9ztt3XdUGNwc7Crdr9u6jW1y/pg91vb2LXdbHarA6jzYpbSVwho6DCNNIMWBd9Yrtu+eiKpzpeiIPdkpnbzx2Wq+0G6R7Z9dCygT2WSCcpwwciURrJPxJRmZtDLUj4DAAD//5CVWGcAAgAA
    <http://objectset.rio.cattle.io/id|objectset.rio.cattle.io/id>: provisioning-cluster-create
    <http://objectset.rio.cattle.io/owner-gvk|objectset.rio.cattle.io/owner-gvk>: <http://management.cattle.io/v3|management.cattle.io/v3>, Kind=Cluster
    <http://objectset.rio.cattle.io/owner-name|objectset.rio.cattle.io/owner-name>: local
    <http://objectset.rio.cattle.io/owner-namespace|objectset.rio.cattle.io/owner-namespace>: ""
  creationTimestamp: "2023-10-27T14:18:54Z"
  finalizers:
  - <http://wrangler.cattle.io/provisioning-cluster-remove|wrangler.cattle.io/provisioning-cluster-remove>
  - <http://wrangler.cattle.io/rke-cluster-remove|wrangler.cattle.io/rke-cluster-remove>
  generation: 4
  labels:
    <http://objectset.rio.cattle.io/hash|objectset.rio.cattle.io/hash>: 50675339e9ca48d1b72932eb038d75d9d2d44618
    <http://provider.cattle.io|provider.cattle.io>: harvester
  name: local
  namespace: fleet-local
  resourceVersion: "490972466"
  uid: 7aec5041-e2e1-4469-909e-314a62837976
spec:
  kubernetesVersion: v1.25.9+rke2r1
  localClusterAuthEndpoint: {}
  rkeConfig:
    chartValues: null
    machineGlobalConfig: null
    provisionGeneration: 1
    upgradeStrategy:
      controlPlaneDrainOptions:
        deleteEmptyDirData: false
        disableEviction: false
        enabled: false
        force: false
        gracePeriod: 0
        ignoreDaemonSets: null
        postDrainHooks: null
        preDrainHooks: null
        skipWaitForDeleteTimeoutSeconds: 0
        timeout: 0
      workerDrainOptions:
        deleteEmptyDirData: false
        disableEviction: false
        enabled: false
        force: false
        gracePeriod: 0
        ignoreDaemonSets: null
        postDrainHooks: null
        preDrainHooks: null
        skipWaitForDeleteTimeoutSeconds: 0
        timeout: 0
status:
  clientSecretName: local-kubeconfig
  clusterName: local
  conditions:
  - lastUpdateTime: "2023-10-27T14:20:36Z"
    message: marking control plane as initialized and ready
    reason: Waiting
    status: Unknown
    type: Ready
  - lastUpdateTime: "2023-10-27T14:18:54Z"
    status: "False"
    type: Reconciling
  - lastUpdateTime: "2023-10-27T14:18:54Z"
    status: "False"
    type: Stalled
  - lastUpdateTime: "2023-11-11T11:30:32Z"
    status: "True"
    type: Created
  - lastUpdateTime: "2024-08-02T17:20:28Z"
    status: "True"
    type: RKECluster
  - status: Unknown
    type: DefaultProjectCreated
  - status: Unknown
    type: SystemProjectCreated
  - lastUpdateTime: "2023-10-27T14:19:09Z"
    status: "True"
    type: Connected
  - lastUpdateTime: "2024-08-02T10:09:32Z"
    status: "True"
    type: Updated
  - lastUpdateTime: "2024-08-02T10:09:32Z"
    status: "True"
    type: Provisioned
  observedGeneration: 4
  ready: true

The manifest log ends -

Wait for cluster settling down...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioned, current generation: 801608)...
CAPI cluster fleet-local/local is provisioned (current generation: 801610).
<http://cluster.fleet.cattle.io/local|cluster.fleet.cattle.io/local> patched
waiting for fleet-agent creation timestamp to be updated
waiting for fleet-agent creation timestamp to be updated
waiting for fleet-agent creation timestamp to be updated
waiting for fleet-agent creation timestamp to be updated
waiting for fleet-agent creation timestamp to be updated

The waiting for fleet-agent message repeats indefinately.

Checking certs on each of the nodes

n1:/ # (
> curl  --cacert /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt \
>   <https://127.0.0.1:10257/healthz> >/dev/null 2>&1 \
>   && echo "[OK] Kube Controller probe" \
>   || echo "[FAIL] Kube Controller probe";
> 
> curl --cacert /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt \
>   <https://127.0.0.1:10259/healthz> >/dev/null 2>&1  \
>   && echo "[OK] Scheduler probe" \
>   || echo "[FAIL] Scheduler probe";
> )
[OK] Kube Controller probe
[OK] Scheduler probe

However n4/n5 do fail!

n5:/ # (
> curl  --cacert /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt \
>   <https://127.0.0.1:10257/healthz> >/dev/null 2>&1 \
>   && echo "[OK] Kube Controller probe" \
>   || echo "[FAIL] Kube Controller probe";
> 
> curl --cacert /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt \
>   <https://127.0.0.1:10259/healthz> >/dev/null 2>&1  \
>   && echo "[OK] Scheduler probe" \
>   || echo "[FAIL] Scheduler probe";
> )
[FAIL] Kube Controller probe
[FAIL] Scheduler probe

So looks like this is the issue…

Ahh - but n4/n5 are workers not part of the control plane - don’t think this is an issue then

I’ve opened on the github tracker here - https://github.com/harvester/harvester/issues/6255

Support bundle attached - note cluster management lists a broken cluster currently - this is likely broken because the upgrade was part complete when I tried to remove it, its disposable. VM’s however we would like to retain save rebuilding/restoring from backups. Hoping this is something simple and can be solved by configuration? I’ve not yet tried rebooting the nodes, not sure if its advisable to do an upgrade from ISO instead - or if we should just start fro scratch and move direct to latest version and restore all vms from backup… obviously hoping that this can be ironed out with less effort since it will take many hours to get this restored if we have to reset and start again. Any pointers greatly appreciated - I’ve tried a few things but running out of ideas.

Posted some further info on the issue - though with my limited knowledge I'm not sure how useful this will be, woud be helpful to know if there is something simple a miss, whether restarts of the process, the nodes or re-install via ISO and or start again is the answer... I cant do anything major on the cluster until tonight so appreciate any feedback we can get before we go nuclear - think if we cant find a solution by Friday I'll be re-installing fresh, hoping that we can solve this before then :/

We're still stuck without a paddle at the moment on this install - @prehistoric-balloon-31801 do you know if its safe to re-start the upgrade at phase 3 (upgrading system services) since we are stuck here - we've removed our k8's cluster and deleted it now as it was disposable - but the suggestions from your colleague do not appear to resume the install. Appreciate any help here - I'm fast approaching re-install and restore conclusion but was hoping that wasn't needed... thanks in advance.

given everything a reboot - still no movement. Going to have to manually upgrade unless some light is shed.

Reboot made no difference, created an ISO and attempted an update from USB - but that option doesn't appear to be present in 1.2.2 - yet it's still in the documentation? Moving toward full reinstall which is frustrating.

Will update the issue too effect.

Honestly tho if it makes sense for us to restore from backup to new cluster we can do that now, I am however trying to understand the snags for continuity.