Hi all! I started using longhorn a few days ago on k3s single node setup. I really like it so I decided to expand the cluster to a k3s etcd 3 master nodes setup. I installed ubuntu 22.04 and installed open-iscsi as described in the docs, but when the cluster starts to run the new longhorn pods on the new nodes, the

longhorn-manager

fails to get ready, unable to list/watch resources, showing various errors such as:

W1224 07:44:23.287997       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2022-12-24T07:44:23Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n, error exit status 2"
time="2022-12-24T07:44:24Z" level=info msg="Skip the leader election for the upgrade since the current Longhorn system is already up to date"
W1224 07:44:24.943618       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1224 07:44:24.961708       1 shared_informer.go:240] Waiting for caches to sync for longhorn datastore
W1224 07:44:25.003320       1 reflector.go:324] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: failed to list *v1beta1.CronJob: the server could not find the requested resource
E1224 07:44:25.003407       1 reflector.go:138] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: Failed to watch *v1beta1.CronJob: failed to list *v1beta1.CronJob: the server could not find the requested resource
W1224 07:44:25.020929       1 reflector.go:324] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1224 07:44:25.020957       1 reflector.go:138] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1224 07:44:25.027597       1 kubernetes_node_controller.go:227] couldn't get kubernetes node master1: node "master1" not found 
time="2022-12-24T07:44:25Z" level=warning msg="Can't find instance manager for pod instance-manager-e-e867fc5d, may be deleted"
time="2022-12-24T07:44:25Z" level=warning msg="Can't find instance manager for pod instance-manager-r-10c2a1ba, may be deleted"

Is it something I did wrong while installing? Any hints what may have gotten wrong?

Did you upgrade k3s to 1.25?