I feel like I'm getting gaslighted, but I don't really know where to turn to next. I have three installs of Rancher with Elemental installed. Our dev cluster, suddenly, stopped working while I was PXE booting a test box to it (It had beeen working fine all week). When I register nodes to it, on the final reboot the elemental-system-agent fails to start and throws this error message endlessly:

Oct 24 20:07:10 .node systemd[1]: Started Elemental System Agent.
Oct 24 20:07:10 .node elemental-system-agent[26070]: time="2025-10-24T20:07:10Z" level=info msg="Rancher System Agent version dev (HEAD) is starting"
Oct 24 20:07:10 .node elemental-system-agent[26070]: time="2025-10-24T20:07:10Z" level=info msg="Using directory /var/lib/elemental/agent/work for work"
Oct 24 20:07:10 .node elemental-system-agent[26070]: time="2025-10-24T20:07:10Z" level=info msg="Starting remote watch of plans"
Oct 24 20:07:10 .node elemental-system-agent[26070]: time="2025-10-24T20:07:10Z" level=fatal msg="error while connecting to Kubernetes cluster: Get \"<https://dev-rancher.example.edu/k8s/clusters/local/version>\": tls: failed to verify certificate: x509: certificate signed by unknown authority"

The cert is signed by Let'sEncrypt and so are all the other instances.

curl

and

openssl

both work fine on the node and trust the cert that's presented, but for some reason this instance is just super messed up and I can't figure out why.

I had been getting that as well. I don't think the error message is necessarily a direct reflection of the issue.

In the Rancher helm values, I believe

agent-tls-mode

should be set to

strict

if using rancher self signed certificates, or

system-store

if using a well known authority. Have you checked to see if those are set correctly for your environment? If you are using a custom certificate, then you'll need to add the certificate authority to the system store.

It's system-store

checks out. Have you rotated certificates for your rancher nodes? You can do that by rebooting each node.

To be clear, it's multiple clusters, each signed by Let's Encrypt, and only one of them is throwing this error.

I hear you, I would try either rotating certificates in the manage cluster view or rebooting the rancher nodes to rotate them that way and see if that yields any helpful results.

The other thing I found was that curl (no

-k

) worked just fine from the console/ssh session.

yea I was about to ask about

-k

Yeah so the system trusts the cert, but for whatever reason the agent doesn't.

what OS/version is it? maybe somehow it's an older OS that doesn't have the updated letsencrypt CAs? (which is certainly a stretch, but I'm grasping at straws if this is working on other clusters outside of this one node)

It's the latest versions. We're actually using the same ISO for all the clusters with just different config yaml.

Curious, maybe you've stated this and I just missed it, have you tried with a freshly build seed image iso?

I hadn't because we're doing PXE and it works with all the other registration endpoints.

Gotcha, I only ask as a troubleshooting step

fwiw I have call with Support tomorrow morning related to this: https://github.com/rancher/elemental/issues/1736

if you have the ability to try it and see what it yields.

It just takes like an hour. 😅

heh, I feel you. I've done my fair share of hours on hours debugging lately lol

And it's not like it was never working... it was... then it stopped. And continues to work in other (almost identical) clusters

right, that is frustrating for sure.

For the install hooks?

ah I mis-understood, I thought you had a support call regarding this as well

Ah, yeah no. That other bug affects prod and this just stopped in a temp dev environment.

Just for sanity sake, are the ntp settings in sync between the two clusters?

yeah they're all in our data center and they should all be using the orgs ntp servers.

gotcha. Might be worth verifying just as a sanity check.

I'll likely wait until after the call tomorrow. It's very possible that I go back and try again and it'll all just be "working" ™️

ha, that's always how it goes

job security goes up, our sanity goes down.

And honestly, I do hope its that easy for you 😂

thing I hate about these damn SSL errors is that the logging sucks for them, and half the time it's not an issue with the cert/chain, and the failed cert just becomes a generic error message.

The thing that threw me is that curl validated it on the node no problem.

Yeah, I mean there are many factors that can cause it to fail, but yeah, better logging would be great.