I had an upgrade hanging from 1.4.3 to 1.5.1 on an old test/dev cluster this week. I looked at various issues in trying to understand the problem. I stumbled on certs expiring. Indeed my cluster is ~450 days old. I managed to make things worse initially in the cli by, I believe, deleting the certs on a node. That node was kicked out of the cluster. But managed to recover by going to Advanced -> Settings -> auto-rotate-rke2-certs and and Enabling it. Is this the best solution going forward on a production cluster? Before hitting 365 days?

yep, I believe you can also start/stop the rke2 service on the nodes to get them to rotate when they're under 90 days.

Thanks! Since this was a test/dev cluster, I was free to try anything. I did try rebooting a node at a time. I also rebooted them all at once. That would start/stop the rke2 service? I'll read more, but I'm confused as to why the default configuration has this issue. And I'm concerned I'm working around a problem incorrectly.

Rebooting them all at once isn't a great idea because of etcd

Understood. I wasn't thinking that it was recommended. This has been a sort of test/dev/eval cluster, we have a different cluster that is lightly used for production. I've tried to recreate various failure modes on this eval cluster and I figured it was another failure I could test. It's been a while but, FYI: Harvester/Longhorn have been amazingly good during some previous tests. No issues recovering from pulling a disk out of a node. No issue pulling power from a node. No issue cutting power to all 3 nodes.

it looks like the message i sent to #C01GKHKAG0K expired, but i posted this a while back:

I have a harvester cluster running 1.5.1 (upgraded from 1.3.x over the last > year). I have new nodes installed from iso, and they aren't joining the cluster. I see an error on the new nodes:

rancher-system-agent[2796]: W0804 22:47:30.831275    2796 reflector.go:492] pkg/mod/k8s.io/client-go@v0.32.2/tools/cache/reflector.go:251: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: stream error: stream ID 29; INTERNAL_ERROR; received from peer") has prevented the request from succeeding

my solution was to run the following on each node:

for d in /var/lib/rancher/rke2/server/tls/kube-{scheduler,controller-manager}/* ; do mv -iv "$d"{,.$(date +%Y%m%d)} ; done
crictl rm -f $(crictl ps -q --name kube-scheduler)
crictl rm -f $(crictl ps -q --name kube-controller-manager)