rancher-users #elemental

strong-shoe-72392

01/05/2023, 10:52 PM

Has anyone else seen the following errors when registering a Raspberry Pi 4 k3s cluster (using elemental-operator 1.1.0). The Raspberry Pi boots up and reaches out to Rancher MCM but it seems to fail to establish a websocket connection. elemental-operator pod logs

Copy code

time="2023-01-05T22:41:38Z" level=info msg="Incoming HTTP request for /elemental/registration/9tb...9"
Thu, Jan 5 2023 5:41:39 pm	time="2023-01-05T22:41:39Z" level=info msg="Negotiated protocol version: 5"
Thu, Jan 5 2023 5:41:39 pm	time="2023-01-05T22:41:39Z" level=error msg="websocket communication interrupted: unknown message"

elemental client logs

Copy code

Jan 05 22:46:04 rancher-31733 elemental[1417]: time="2023-01-05T22:46:04Z" level=info msg="Enable TPM emulation"
Jan 05 22:46:04 rancher-31733 elemental[1417]: time="2023-01-05T22:46:04Z" level=info msg="Connect to <https://rancher>.<redacted>"
Jan 05 22:46:05 rancher-31733 elemental[1417]: time="2023-01-05T22:46:05Z" level=info msg="Using TPMHash <redacted> to dial <wss://rancher>.<redacted>/elemental/registration/9tb...9"
Jan 05 22:46:05 rancher-31733 elemental[1417]: time="2023-01-05T22:46:05Z" level=debug msg="Start TPM attestation"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="TPM attestation successful"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=debug msg="elemental-register protocol version: 8"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="Negotiated protocol version: 5"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="Send SMBIOS data"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=info msg="Send system data"
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=debug msg="system data:..."
Jan 05 22:46:06 rancher-31733 elemental[1417]: time="2023-01-05T22:46:06Z" level=error msg="failed to register machine inventory: failed to send system data: websocket: close 1006 (abnormal closure): unexpected EOF"

This appears to be the same error that was addressed by https://github.com/rancher/elemental/issues/346 and https://github.com/rancher/elemental-operator/issues/176 Any help is greatly appreciated! Thanks!

strong-shoe-72392

01/11/2023, 5:29 PM

On the latest version of the elemental-teal ISO x86, I'm seeing the following error

Copy code

time="2023-01-11T17:26:09Z" level=warning msg="Detected old elemental-register client: cloud-config data may not be applied correctly"

And it does not execute my cloud-config to set the username/password to be able to log into the instance after installation. Any ideas? I'm using Elemental-Operator 1.1.0.

mysterious-whale-87222

01/12/2023, 5:39 PM

Copy code

Hey Everybody, short question: How to set rke2/k3s command line params like `--kubelet-arg`by elemental? Didn't find anything in docs.

loud-sundown-69515

01/16/2023, 11:02 PM

On the Quickstart guide it says having rancher running is a requirement. Looking at ripping out K3os. 1. Does elemental provide that same experience? Specifically bringing up bare VMs into a multi node cluster? 2. Is cilium supported?

careful-piano-35019

01/16/2023, 11:10 PM

Hi Jesse. Yes (one of) Elemental purpose is to enable deploying and managing multiple multi nodes clusters from Rancher on bare metal (but also on virtual machines)

careful-piano-35019

01/16/2023, 11:12 PM

it leverages Rancher provisioning so it can deploy any configuration possible with Rancher

careful-piano-35019

01/16/2023, 11:15 PM

with K3s => https://ranchermanager.docs.rancher.com/reference-guides/cluster-configuration/rancher-server-configuration/k3s-cluster-configuration#clust[…]g-file

careful-piano-35019

01/16/2023, 11:15 PM

with RKE2 => https://ranchermanager.docs.rancher.com/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration#conta[…]vider

silly-airline-38945

01/17/2023, 12:19 PM

Hi, testing the elemental-iso-add-registration from the docs. Its failing on

<http://ftp.lysator.liu.se|ftp.lysator.liu.se> 404 not found

Is this the one to use or should the elemental-iso-build be used?

careful-piano-35019

01/17/2023, 12:29 PM

it looks like an issue related to openSUSE infrastructure

careful-piano-35019

01/17/2023, 12:33 PM

can you try

wget <https://download.opensuse.org/repositories/isv:/Rancher:/Elemental:/Stable:/Teal53/media/iso/elemental-teal.x86_64.iso>

careful-piano-35019

01/17/2023, 12:33 PM

(assuming you're trying Stable and x86_64)

loud-sundown-69515

01/17/2023, 1:09 PM

@careful-piano-35019 thanks for the reply, so I could use elemental like K3os. Stand up a multi node cluster on VMs with k3s or rke2 as the k distro and cilium as the networking and ingress controller. Would like to build this out locally before looking at use in a production setting.

careful-piano-35019

01/17/2023, 1:10 PM

In a similar way, yes. Main difference is that Rancher is required for the orchestration

careful-piano-35019

01/17/2023, 1:11 PM

Big advantage is that you'll get regular OS updates distributed as OCI image

loud-sundown-69515

01/17/2023, 1:24 PM

So I need to stand up an rke2 cluster with rancher to then provision elemental nodes?

careful-piano-35019

01/17/2023, 1:25 PM

yes, or one node k3s is fine

👍 1

careful-piano-35019

01/17/2023, 1:25 PM

it's a bit work at the beginning that you gain later 🙂 bc you can basically deploy without any access to the nodes

careful-piano-35019

01/17/2023, 2:23 PM

the ISO is customized with your rancher URL and registration endpoint, and when you start your VM, it will contact Rancher and OS setup and control plan (K3s or RKE2) will be setup and you'll get your kubeconfig from Rancher dashboard

sparse-monitor-30665

01/18/2023, 8:25 PM

Hi Everyone, sorry if this has been discussed before, but I don't see anything recent, and there's nothing about it in the docs.... What's the best way to go about formatting and mounting additional disks (with lvm?) for /var/lib/longhorn? I have tried adding commands in the Cloud Config stage, or with "Plans", but neither worked very well, and I could not get the disk to mount unless I did so manually. Perhaps I'm missing something, or is the general consensus to use the boot disk and increase it's size as needed? With an "Instructions" Plan I was able to get the LV created, but that only runs once so I can't mount it there. I've tried to create and mount it with the cloud-config: runcmd, but it just fails to ever execute.

INFO[2023-01-18T19:28:39Z] Applying 'Cloud init' for stage 'boot'. Total stages: 1

INFO[2023-01-18T19:28:39Z] Processing stage step ''. ( commands: 2, files: 0, ... )

ERRO[2023-01-18T19:28:39Z] Udev database has incomplete information about device /dev/sda.

/dev/sda: Failed to get external handle [udev].

Udev database has incomplete information about device /dev/sda1.

/dev/sda1: Failed to get external handle [udev].

Udev database has incomplete information about device /dev/sda2.

/dev/sda2: Failed to get external handle [udev].

Udev database has incomplete information about device /dev/sda3.

/dev/sda3: Failed to get external handle [udev].

Udev database has incomplete information about device /dev/sda4.

/dev/sda4: Failed to get external handle [udev].

Udev database has incomplete information about device /dev/sda5.

/dev/sda5: Failed to get external handle [udev].

Udev database has incomplete information about device /dev/sdb.

/dev/sdb: Failed to get external handle [udev].

Device /dev/sdb excluded by a filter.

: failed to run if test -z "$(blkid /dev/rancher/longhorn)"; then vgcreate rancher /dev/sdb && lvcreate -l 100%FREE -n rancher/longhorn && mkfs -t ext4 -L longhorn /dev/rancher/longhorn; fi: exit status 5

Same with the mount statement after it

if test "$(blkid /dev/rancher/longhorn)"; then mount ... fi

Cheers!

sparse-monitor-30665

01/24/2023, 2:58 PM

Expected behaviour??? In Rancher Manager, if I delete a cluster created from Elemental Machines, the MachineInventory gets removed. Ideally I would want to reuse these machines to either re-create the cluster, or be used in other clusters. Currently I have been reinstalling to get it back. Is there a better way to go about this? Is this expected or am I doing something wrong (maybe I'm not waiting long enough?) When the cluster gets removed, is there steps taken on the machines to clean up the kube install and reset it, or is there a way I can add this in?

orange-airplane-98016

01/24/2023, 10:35 PM

@orange-airplane-98016 has left the channel

strong-shoe-72392

01/31/2023, 9:13 PM

Hello again! Are there any other resources that need to be installed to use the Elemental OS Upgrade capability besides the prerequisites here - https://elemental.docs.rancher.com/next/quickstart/#what-is-the-rancher-elemental-stack- ? I tried to perform an Elemental Teal OS Upgrade on a Raspberry Pi 4 (that successfully provisioned k3s using the Quickstart and raw image for RPi) using the following:

Copy code

apiVersion: <http://elemental.cattle.io/v1beta1|elemental.cattle.io/v1beta1>
kind: ManagedOSImage
metadata:
  name: elemental-rpi-upgrade-demo
  namespace: fleet-default
spec:
  # Set to the new Elemental version you would like to upgrade to or track the latest tag
  osImage: "<http://registry.opensuse.org/isv/rancher/elemental/dev/teal53/15.4/rancher/elemental-teal/5.3:1.1.0-60.7|registry.opensuse.org/isv/rancher/elemental/dev/teal53/15.4/rancher/elemental-teal/5.3:1.1.0-60.7>"
  clusterTargets:
    - clusterName: elemental-rpi-cluster

My current version according to /etc/os-release is:

Copy code

IMAGE_REPO="<http://registry.opensuse.org/isv/rancher/elemental/dev/teal53/15.4/rancher/elemental-teal/5.3|registry.opensuse.org/isv/rancher/elemental/dev/teal53/15.4/rancher/elemental-teal/5.3>"
IMAGE_TAG="1.0.0-40.2"

When applied, I see that Fleet tries to make a new bundle but errors out on what appears to be missing CRDs:

Copy code

elemental-rpi-cluster-managed-system-agent
ErrApplied(1) [Cluster fleet-default/elemental-rpi-cluster: unable to build kubernetes objects from release manifest: [resource mapping not found for name: "system-agent-upgrader" namespace: "cattle-system" from "": no matches for kind "Plan" in version "<http://upgrade.cattle.io/v1|upgrade.cattle.io/v1>" ensure CRDs are installed first, resource mapping not found for name: "system-agent-upgrader-windows" namespace: "cattle-system" from "": no matches for kind "Plan" in version "<http://upgrade.cattle.io/v1|upgrade.cattle.io/v1>" ensure CRDs are installed first]]

I even attempted to install the Rancher System-Upgrade-Controller into our Rancher Server/MCM cluster as it looked like it had some of those CRDs, but that didn't fix anything. Did I miss a step? Is this not yet supported for Raspberry Pis (although it seems to be failing before the bundle is created anyway)? As usual, any help is greatly appreciated!

refined-analyst-8898

03/25/2023, 12:58 AM

Should I use Elemental for the node image when creating an RKE2 cluster in Rancher with Harvester as cloud provider? I was looking for a good node image anyway, and just heard about Elemental. I was using the Ubuntu Minimal Cloud image because it works with cloud-init and doesn't require an interactive installation step. I am also considering the Tumbleweed Minimal Cloud image which appears to have that same capability. My goal is to learn Rancher and Harvester and Longhorn and automate as much as practical.

few-jordan-3349

05/03/2023, 7:35 AM

hey, before elemental upgrade is applied there is a check for isHigherVersion. How to apply the FORCE flag befor upgrade to ignore this check ? https://github.com/rancher/elemental/blob/7ef045208a2eb054208ea03d19b4775d43eab99e/framework/files/usr/sbin/suc-upgrade#L71 i was not able to find it in the docs. Thanks for the help!:)

refined-analyst-8898

05/06/2023, 6:20 PM

@refined-analyst-8898 has left the channel

blue-kitchen-51801

05/16/2023, 8:39 PM

Any idea how can we install Elemental on a Software RAID? Thank you!

few-jordan-3349

05/26/2023, 1:47 PM

hey, i want to increase size setting of /tmp tmpfs. Therefor i am using the following approach to edit fstab during initramfs from the docs: https://elemental.docs.rancher.com/customizing#adding-extra-lvm-volume-group-disks-during-the-installation is this the correct approach? Because the setting is available in fstab file after boot (

tmpfs  /tmp  tmpfs  defaults,size=75%  0  0

) but it is not applied... i still have the default setting of 50%

acceptable-belgium-2684

05/30/2023, 9:02 PM

Being new to elemental I am confused on how to boot the teal ISO that is provided in the releases. Can i boot the iso using VirtualBox or VMWare?

acceptable-belgium-2684

05/30/2023, 9:31 PM

Just cannot seem to get it booting