This message was deleted.

If you look at the loop's logic, it just wants that volume to be healthy, so you could start a bogus deployment that mounts that PVC, let Longhorn rebuild it and then afterwards delete all of it.

Thanks - may well give that a try - wont be until mid-week now...

Because it's not attached to any pods, LH won't rebuild it. Attach it to a dummy pod, wait for the rebuild and the loop will continue. Creating a new PVC/PV won't change the loop's "obsession" anymore, it's just a dumb script that doesn't update halfway through

Cool - I get you now... will try that tomorrow, held off reinstall as needed to add some capacity to do it smoothly. Issue is presently any vms that are stopped wont go up again, the k8s api service isn't talking, cant provision new vms - so shutting everything down isn't so much an option. Good think though is we do have backups, and there are 3 replicas of everything - so even if the migration continues live it'll just take longer...

If the kube-apiserver is not running, it won't let you schedule a pod either... you can go into the harvester node and check

systemctl status rke2-server

n2:/ # systemctl status rke2-server
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/etc/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/rke2-server.service.d
             └─override.conf
     Active: active (running) since Mon 2025-07-14 18:56:27 UTC; 2 months 23 days ago
       Docs: <https://github.com/rancher/rke2#readme>
    Process: 13630 ExecStartPost=/sbin/sysctl -w net.bridge.bridge-nf-call-iptables=0 (code=exited, status=0/SUCCESS)
   Main PID: 2570 (rke2)
      Tasks: 605
 
n3:/ # systemctl status rke2-server
● rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/etc/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/rke2-server.service.d
             └─override.conf
     Active: active (running) since Fri 2025-07-11 21:39:03 UTC; 2 months 26 days ago
       Docs: <https://github.com/rancher/rke2#readme>
   Main PID: 2676 (rke2)
      Tasks: 1147
     CGroup: /system.slice/rke2-server.service
             ├─   525 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 94c1541b1d593264a0982cb40925248c>
             ├─   972 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 56b2aa22f2243195410d2f10804eb501>
             ├─  1743 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 8443a91e561f382d17e26020f92556a8>
             ├─  1932 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id edd533c84e38cf5bd8cef815780474f6>
             ├─  2020 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 18b039c47000f4a26753e8fed18fa634>
             ├─  2168 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 520033f2a0b790a79c4ffb89c82a0ea7>
             ├─  2181 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id a1fdb9d0c44ee969fd7891afd9bfa4e3>
             ├─  2396 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 510f59a530927ce4fc6bbe2699d4fe8e>
             ├─  2676 "/opt/rke2/bin/rke2 server"
             ├─  2738 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml
             ├─  3246 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --anonymous-a>
             ├─  3392 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 220ecf020bb575f9470803dc954f11ee>
             ├─  3396 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 815dec47f383893ad238ccef48a1dda8>

 rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/etc/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/rke2-server.service.d
             └─override.conf
     Active: active (running) since Sat 2025-09-20 18:50:16 UTC; 2 weeks 2 days ago
       Docs: <https://github.com/rancher/rke2#readme>
    Process: 30148 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 30150 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 30151 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 30152 ExecStartPre=/usr/sbin/harv-update-rke2-server-url server (code=exited, status=0/SUCCESS)
    Process: 32403 ExecStartPost=/sbin/sysctl -w net.bridge.bridge-nf-call-iptables=0 (code=exited, status=0/SUCCESS)
   Main PID: 30154 (rke2)
      Tasks: 539
     CGroup: /system.slice/rke2-server.service
             ├─  1392 /var/lib/rancher/rke2/data/v1.32.4-rke2r1-2a30dd9c8583/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 023d5bbeedc99b9ad2675f2fb366a5f3>
             ├─  2953 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 03e20b57c38acac54cec13d59ebac5f4>
             ├─  2999 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id e37f6ba7db3552cee0724ef1f90b0b00>
             ├─  3533 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 6d29e4decf0c11d0df8724b0ebc130eb>
             ├─  5673 /var/lib/rancher/rke2/data/v1.31.7-rke2r1-7f85e977b85d/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 8b2e68265f92925dc8abf75b1d110be8>
             ├─  5796 /var/lib/rancher/rke2/data/v1.32.4-rke2r1-2a30dd9c8583/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 97b3cf352d35bf93607ad6843b9282e8>

Those are the two active control planes and show it running - N4 above is the one stuck post draining and is a control plane node. N1/N5 are workers so are not running the service - my guess is that because the upgrade has stalled the CP isn't functioning properly... I think we're probably in quite different timezones, its nearly midnight here, will give this another look tomorrow eve! Many thanks though - your support is much appreciated!

I'm in AU 😉 Yeah if node4 is the one on post-draining it makes sense for it to be not serving any endpoints, but that should not hurt the kube-apiserver itself with 2 other nodes having quorum. You can check in more detail also with

journalctl -xeu rke2-server

Anyway, purely based on my own experience with that dreaded loop. Focus on making LH rebuild that volume and your upgrade will continue without issues.

On N2 (the upgraded node -

Oct 07 11:03:37 n2 rke2[2570]: time="2025-10-07T11:03:37Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:46768: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:37 n2 rke2[2570]: time="2025-10-07T11:03:37Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:46770: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:37 n2 rke2[2570]: time="2025-10-07T11:03:37Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:46798: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:37 n2 rke2[2570]: time="2025-10-07T11:03:37Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:46800: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:37 n2 rke2[2570]: time="2025-10-07T11:03:37Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:46782: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:41 n2 rke2[2570]: time="2025-10-07T11:03:41Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49866: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:41 n2 rke2[2570]: time="2025-10-07T11:03:41Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49878: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:41 n2 rke2[2570]: time="2025-10-07T11:03:41Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49864: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:41 n2 rke2[2570]: time="2025-10-07T11:03:41Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49882: dial tcp 10.52.10.41:8443: connect: connection timed out"
Oct 07 11:03:41 n2 rke2[2570]: time="2025-10-07T11:03:41Z" level=error msg="Sending HTTP/1.1 502 response to 127.0.0.1:49894: dial tcp 10.52.10.41:8443: connect: connection timed out"

Lots of timeouts Same on the other CP nodes. Will give creating a deployment a go later tonight, new NVME's on the way should arrive this afternoon so we can get a new cluster started (plan is to run a single node, get key stuff moved over in parallel if the first cluster is available, then decommission and restore the rest), but fingers crossed it wont come to that... Not sure what these timeouts mean... Etcd logs show some errors - node 2's log - the upgraded

n2:/ # tail /var/log/pods/kube-system_etcd-n2_c4d9834e8f7ec8f6ac317ff390234b8d/etcd/0.log
2025-07-14T18:51:28.183449967Z stderr F {"level":"info","ts":"2025-07-14T18:51:28.183371Z","caller":"traceutil/trace.go:171","msg":"trace[798019453] linearizableReadLoop","detail":"{readStateIndex:477314862; appliedIndex:477314862; }","duration":"151.869082ms","start":"2025-07-14T18:51:28.031493Z","end":"2025-07-14T18:51:28.183362Z","steps":["trace[798019453] 'read index received'  (duration: 151.866677ms)","trace[798019453] 'applied index is now lower than readState.Index'  (duration: 2.074µs)"],"step_count":2}
2025-07-14T18:51:28.183470957Z stderr F {"level":"warn","ts":"2025-07-14T18:51:28.183437Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"151.935756ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/rke2\" limit:1 ","response":"range_response_count:1 size:520"}
2025-07-14T18:51:28.183477258Z stderr F {"level":"info","ts":"2025-07-14T18:51:28.183451Z","caller":"traceutil/trace.go:171","msg":"trace[883504774] range","detail":"{range_begin:/registry/leases/kube-system/rke2; range_end:; response_count:1; response_revision:459668105; }","duration":"151.966613ms","start":"2025-07-14T18:51:28.031481Z","end":"2025-07-14T18:51:28.183447Z","steps":["trace[883504774] 'agreement among raft nodes before linearized reading'  (duration: 151.935225ms)"],"step_count":1}
2025-07-14T18:51:28.183859059Z stderr F {"level":"warn","ts":"2025-07-14T18:51:28.183815Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"105.160344ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/leases/kube-system/rke2-cloud-controller-manager\" limit:1 ","response":"range_response_count:1 size:521"}
2025-07-14T18:51:28.183879337Z stderr F {"level":"info","ts":"2025-07-14T18:51:28.183847Z","caller":"traceutil/trace.go:171","msg":"trace[519652222] range","detail":"{range_begin:/registry/leases/kube-system/rke2-cloud-controller-manager; range_end:; response_count:1; response_revision:459668105; }","duration":"105.204827ms","start":"2025-07-14T18:51:28.078633Z","end":"2025-07-14T18:51:28.183838Z","steps":["trace[519652222] 'agreement among raft nodes before linearized reading'  (duration: 105.150767ms)"],"step_count":1}
2025-07-14T18:51:40.401792056Z stderr F {"level":"info","ts":"2025-07-14T18:51:40.401707Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":459663841}
2025-07-14T18:51:40.62442696Z stderr F {"level":"info","ts":"2025-07-14T18:51:40.624318Z","caller":"mvcc/kvstore_compaction.go:71","msg":"finished scheduled compaction","compact-revision":459663841,"took":"221.252662ms","hash":823256557,"current-db-size-bytes":157949952,"current-db-size":"158 MB","current-db-size-in-use-bytes":45027328,"current-db-size-in-use":"45 MB"}
2025-07-14T18:51:40.624459401Z stderr F {"level":"info","ts":"2025-07-14T18:51:40.624350Z","caller":"mvcc/hash.go:151","msg":"storing new hash","hash":823256557,"revision":459663841,"compact-revision":459657522}
2025-07-14T18:51:58.635902698Z stderr F {"level":"info","ts":"2025-07-14T18:51:58.635850Z","caller":"osutil/interrupt_unix.go:64","msg":"received signal; shutting down","signal":"terminated"}
2025-07-14T18:51:58.635919249Z stderr F {"level":"info","ts":"2025-07-14T18:51:58.635876Z","caller":"embed/etcd.go:408","msg":"closing etcd server","name":"n2-77276a18","data-dir":"/var/lib/rancher/rke2/server/db/etcd","advertise-peer-urls":["<http://localhost:2380>"],"advertise-client-urls":["<https://192.168.122.3:2379>"]}

I'm guessing as the upgrade is mid flow etcd is not getting quorum? n3's log tail - this is the active older node

n3:/ # tail /var/log/pods/kube-system_etcd-n3_21fe0b54bf605909d3da85285bd0eabd/etcd/0.log
2025-10-07T11:10:06.351248931Z stderr F {"level":"warn","ts":"2025-10-07T11:10:06.351174Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-10-07T11:10:05.990818Z","time spent":"360.349328ms","remote":"127.0.0.1:48906","response type":"/etcdserverpb.KV/Range","request count":0,"request size":53,"response count":0,"response size":31,"request content":"key:\"/registry/fleet.cattle.io/bundlenamespacemappings\" limit:1 "}
2025-10-07T11:10:06.351253121Z stderr F {"level":"warn","ts":"2025-10-07T11:10:06.351019Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2025-10-07T11:10:05.991111Z","time spent":"359.90182ms","remote":"127.0.0.1:40282","response type":"/etcdserverpb.KV/Range","request count":0,"request size":25,"response count":0,"response size":31,"request content":"key:\"/registry/controllers\" limit:1 "}
2025-10-07T11:10:06.351693659Z stderr F {"level":"warn","ts":"2025-10-07T11:10:06.351624Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"153.174833ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/health\" ","response":"range_response_count:0 size:8"}
2025-10-07T11:10:06.351709419Z stderr F {"level":"info","ts":"2025-10-07T11:10:06.351643Z","caller":"traceutil/trace.go:171","msg":"trace[312089974] range","detail":"{range_begin:/registry/health; range_end:; response_count:0; response_revision:567275528; }","duration":"153.213252ms","start":"2025-10-07T11:10:06.198424Z","end":"2025-10-07T11:10:06.351637Z","steps":["trace[312089974] 'agreement among raft nodes before linearized reading'  (duration: 153.179703ms)"],"step_count":1}
2025-10-07T11:10:06.351714469Z stderr F {"level":"warn","ts":"2025-10-07T11:10:06.351650Z","caller":"etcdserver/util.go:170","msg":"apply request took too long","took":"152.161357ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/node.harvesterhci.io/nodeconfigs/harvester-system/n4\" ","response":"range_response_count:1 size:1709"}
2025-10-07T11:10:06.351725649Z stderr F {"level":"info","ts":"2025-10-07T11:10:06.351673Z","caller":"traceutil/trace.go:171","msg":"trace[53243766] range","detail":"{range_begin:/registry/node.harvesterhci.io/nodeconfigs/harvester-system/n4; range_end:; response_count:1; response_revision:567275528; }","duration":"152.194666ms","start":"2025-10-07T11:10:06.199472Z","end":"2025-10-07T11:10:06.351666Z","steps":["trace[53243766] 'agreement among raft nodes before linearized reading'  (duration: 152.141107ms)"],"step_count":1}
2025-10-07T11:10:07.141012833Z stderr F {"level":"info","ts":"2025-10-07T11:10:07.140903Z","caller":"traceutil/trace.go:171","msg":"trace[529276496] transaction","detail":"{read_only:false; response_revision:567275534; number_of_response:1; }","duration":"124.399731ms","start":"2025-10-07T11:10:07.016490Z","end":"2025-10-07T11:10:07.140890Z","steps":["trace[529276496] 'process raft request'  (duration: 124.282681ms)"],"step_count":1}
2025-10-07T11:13:13.423338219Z stderr F {"level":"info","ts":"2025-10-07T11:13:13.423233Z","caller":"mvcc/index.go:214","msg":"compact tree index","revision":567273827}
2025-10-07T11:13:13.530656795Z stderr F {"level":"info","ts":"2025-10-07T11:13:13.530578Z","caller":"mvcc/kvstore_compaction.go:70","msg":"finished scheduled compaction","compact-revision":567273827,"took":"105.973003ms","hash":1700177557,"current-db-size-bytes":178892800,"current-db-size":"179 MB","current-db-size-in-use-bytes":46649344,"current-db-size-in-use":"47 MB"}
2025-10-07T11:13:13.530669665Z stderr F {"level":"info","ts":"2025-10-07T11:13:13.530603Z","caller":"mvcc/hash.go:151","msg":"storing new hash","hash":1700177557,"revision":567273827,"compact-revision":567269203}

You can remove the mutating/validating webhook temporarily. But likely you have more issues by now. It's been a long time since you left the cluster degraded.

Fresh install underway now! thanks for all the advice!