The apiserver serving cert is stored in the cluster, and is shared across all the servers. The kubelet certs are generated on startup every time so there’s no way it would persist across different hosts. I suspect that what you’re running into is that the serving cert isn’t valid in combination with some other setting you’re overriding.

ok yeah so in that case that’s the kubelet cert

That means you need to look at the node where the pod is running, not necessarily the server

The kubelet cert will be valid for the specified --node-ip and --node-external-ip addresses.

Did you install K3s from our install script or repo, or are you using some distro’s packaging of it? I think I remember seeing someone run into something similar using truenas’s modified install of K3s

I let it pick it because there is only one anyway

What is the output of

kubectl get node <NODENAME> -o yaml

for the node with the pod on it, and what is the output when you run

openssl x509 -noout -text -in /var/lib/rancher/k3s/agent/serving-kubelet.crt

on that node?

Is the

38.x.x.x

address in the error the old or current address of the node with the pod, or the server?

cert: https://gist.github.com/bhechinger/1672855ce76a32f6eac20767d7bab41a

that cert has the correct IP

looking at that node config I don't see anything obvious

Ok, now I am utterly confused. If I point my web browser at the kublet api on that host and look at the cert I get back it has the correct IP address. Where is that error coming from?

so, I'm not connected to that node. I'm using kubectl against a different node. Is that message coming from the node itself? Could it possibly have cached something and is confused? (just spitballing here)

yes

that's it

because I just switched my kubeconfig to the host in question and I can get logs now