Has anyone else seen the following errors when registering a Raspberry Pi 4 k3s cluster (using elemental-operator 1.1.0). The Raspberry Pi boots up and reaches out to Rancher MCM but it seems to fail to establish a websocket connection. elemental-operator pod logs

time="2023-01-05T22:41:38Z" level=info msg="Incoming HTTP request for /elemental/registration/9tb...9"
Thu, Jan 5 2023 5:41:39 pm	time="2023-01-05T22:41:39Z" level=info msg="Negotiated protocol version: 5"
Thu, Jan 5 2023 5:41:39 pm	time="2023-01-05T22:41:39Z" level=error msg="websocket communication interrupted: unknown message"

elemental client logs

Jan 05 22:46:04 rancher-31733 elemental[1417]: time="2023-01-05T22:46:04Z" level=info msg="Enable TPM emulation"
Jan 05 22:46:04 rancher-31733 elemental[1417]: time="2023-01-05T22:46:04Z" level=info msg="Connect to <https://rancher>.<redacted>"
Jan 05 22:46:05 rancher-31733 elemental[1417]: time="2023-01-05T22:46:05Z" level=info msg="Using TPMHash <redacted> to dial <wss://rancher>.<redacted>/elemental/registration/9tb...9"
Jan 05 22:46:05 rancher-31733 elemental[1417]: time="2023-01-05T22:46:05Z" level=debug msg="Start TPM attestation"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="TPM attestation successful"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=debug msg="elemental-register protocol version: 8"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="Negotiated protocol version: 5"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="Send SMBIOS data"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="Send system data"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=debug msg="system data:..."
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=error msg="failed to register machine inventory: failed to send system data: websocket: close 1006 (abnormal closure): unexpected EOF"

This appears to be the same error that was addressed by https://github.com/rancher/elemental/issues/346 and https://github.com/rancher/elemental-operator/issues/176 Any help is greatly appreciated! Thanks!

I assume you're on Rancher 2.7.0 stable version of the elemental operator from

<oci://registry.opensuse.org/isv/rancher/elemental/stable/charts/rancher/elemental-operator-chart>

Rancher 2.7.0 - yes Elemental Operator - I was on the stable (1.0.2 I think is what the chart pulled) but I upgraded to 1.1.0 to try as well.

ok, that's a possible scenario but yeah it opens questions about networking / filtering

It's v1.21.12+rke2r2 (needs updated)

"*The issue is in the operator when using kubernetes 1.24*."

I can try deploying latest from main branch of elemental-operator

I'm not saying it's the issue, just trying to guess here

And that should have been in both 1.0.2 and 1.1.0

if it's not too much a hassle, you can give a try, that will help narrow down the possibilities

If that doesn't work I could also try standing up a local Rancher MCM instance to rule out some networking funniness.

but, in general, Elemental being fairly recent, we mostly worked and tested with k8s > 1.24

Gotcha

but the issues at first sight look network related

We have PSP on cluster that prevents this.

I think it's worth a GitHub issue, it soes not look like you're missing something obvious

Thanks for your feed back! I'll poke around some more and update here.

ok, I'm currently in the process of installing on USFF format x86 systems, but when I'm done, I'll switch back to RPi

Thanks! Here's the info of the downloaded rpi.raw ISO I used to make the bootable USB

md5sum rpi.raw
a6181a673b70691dfc19a09dfd8c5a68  rpi.raw

not yet, sorry. Possibly Friday

No problem. Thanks again for the help.

All good now ?

Yes, I was able to register my Pi 4 with Rancher!

Yes I've done it with a local cluster without a public CA