Hello, Background: 13 node bare-metal Raspberry Pi 4 based cluster, 1 master, 12 workers. Running K3s v1.24.8+k3s1 for compatibility with Rancher management UI. I’m doing this to learn Kubernetes. I’m trying to add 2 more master nodes. After working through various goofs of my own (failed to match K3s version and launching with the same flags as the original master), I believe I got all of that straightened out. The respective .service files indicate as such, anyway. However, when I attempt to add these 2 nodes as master, I get the following error in the logs:

2477 authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, [invalid bearer token, square/go-jose: error in cryptographic primitive]]"

And, they never show up in the output of

kubectl get nodes

I’ve Googled that error and find many bug reports, posts, etc. but no definitive answer or solution. I’ve wiped out these new nodes and started fresh, but still get the same error. Any advice would be appreciated.

That suggests that the other nodes have invalid certs, probably not shared with the first one. Are you using embedded etcd, or an external DB?

thanks for the reply! i am [attempting] to run embedded etcd i have uninstalled using the script and reinstalled numerous times to no avail. i have also verified the absence of

/var/lib/rancher/k3s

which is the only place I'm aware of that contains the certificates

something running somewhere in the cluster has a serviceaccount token that was signed by a different key. That’s all I can tell from that error. The most common cause of that is one of the control-plane nodes having a serviceaccount issuer key that is out of sync with the others.

right now there is just the one etcd/controlplane node. in my Googlings, about the only suggestion I found was to delete all service accounts and restart K3s, which I did, but again, no luck. If my install command has a flaw, that could explain why that didn't work. Is the command above good or should I remove one of the

server

flags? if so, which one?

there are not two server flags. You have

server

as in

k3s server

which is a command… and then you have

--server=<https://x:6443>

which is a flag that tells it where the existing server is.

ok, so the command is good? any advice how to proceed? again, I'm fairly new and trying to learn. Would rotating the certificates work maybe?