https://rancher.com/ logo
Join Slack
Powered by
Hey, I have an rke2 issue (shipped from Harvester ...
# rke2
b
Hey, I have an rke2 issue (shipped from Harvester 1.8.2 ( I'm going to be opening a ticket there as well) but is there a known issue with the 
v1.35.7-rke2r1-74310658583d
release where the embedded binaries are x86_64? I'm stuck in an upgrade and I'm not sure how to get/patch the correct version. Also trying to figure out if I need to open a issue against rke2 as well or if it's been solved already.
Copy code
slc5:~ # file /opt/rke2/bin/rke2
/opt/rke2/bin/rke2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=fe4ee4d4272d234eb0261076d721b2bc4cdbb3b4, stripped

slc5:~ # rm -rf /var/lib/rancher/rke2/bin/*
slc5:~ #
slc5:~ # rm -rf /var/lib/rancher/rke2/data/*v1.35*
slc5:~ # systemctl start rke2-server
Job for rke2-server.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status rke2-server.service" and "journalctl -xeu rke2-server.service" for details.
slc5:~ # file /var/lib/rancher/rke2/bin/containerd
/var/lib/rancher/rke2/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=33098ab8baecd5b36899939762a6797189482784, stripped
c
that looks like a harvester issue
I can confirm that both the arm64 and amd64 rke2-runtime images for that release (
<http://docker.io/rancher/rke2-runtime:v1.35.7-rke2r1|docker.io/rancher/rke2-runtime:v1.35.7-rke2r1>
) contain binaries for the correct arch. I suspect that the Harvester ISO for aarch64 contains airgap image tarballs for amd64.
This is not an rke2 issue.
You should be able to work around it by deleting the data dir and replacing the tarball in /var/lib/rancher/rke2/agent/images with the airgap tarball for the correct arch, or just deleting it and letting it pull from a registry with the correct images loaded
b
I deleted the embedded images so it could pull from docker.io and verified that rke2 was the correct arch, but the extracted files inside that were incorrect.
c
are you sure?
b
That's what: `
Copy code
slc5:~ # file /opt/rke2/bin/rke2
/opt/rke2/bin/rke2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=fe4ee4d4272d234eb0261076d721b2bc4cdbb3b4, stripped

slc5:~ # rm -rf /var/lib/rancher/rke2/bin/*
slc5:~ #
slc5:~ # rm -rf /var/lib/rancher/rke2/data/*v1.35*
slc5:~ # systemctl start rke2-server
Job for rke2-server.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status rke2-server.service" and "journalctl -xeu rke2-server.service" for details.
slc5:~ # file /var/lib/rancher/rke2/bin/containerd
/var/lib/rancher/rke2/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=33098ab8baecd5b36899939762a6797189482784, stripped
was all about
c
I literally just dumped the files from the image and they are correct
like I said you need to delete the data dir
b
like 
lc5:~ # rm -rf /var/lib/rancher/rke2/data/*v1.35*
?
c
if the data dir exists it won’t extract the files again, even if they are for the wrong arch. It doesn’t check that.
yes
b
Yeah I did
It's right in the middle of that block
c
do you have a registry mirror with the wrong image pushed?
b
Yes, but I think that's a different issue?
c
…
so… it’s getting the wrong arch from there?
b
I removed all the x86_64 images from the node so it would pull the right ones (we're not airgapped) from docker.io
c
the image on docker hub is correct
b
I verified that at one point, let me find it
Sorry I've been debugging this until like 3am last night and then again all this morning it's a little spread out
Ok... Here's the image that's on the node NOW, after we deleted it. There's stuff for both arm and amd:
Copy code
slc1:~ # ctr -n <http://k8s.io|k8s.io> image inspect <http://docker.io/rancher/system-agent-installer-rke2:v1.35.7-rke2r1|docker.io/rancher/system-agent-installer-rke2:v1.35.7-rke2r1>
<http://docker.io/rancher/system-agent-installer-rke2:v1.35.7-rke2r1|docker.io/rancher/system-agent-installer-rke2:v1.35.7-rke2r1>
│    Created: 2026-08-25 16:33:12.460091853 +0000 UTC
│    Updated: 2026-08-25 16:33:12.460091853 +0000 UTC
│    Label "io.cri-containerd.image": "managed"
└── application/vnd.oci.image.index.v1+json @sha256:b41c2b53f3d083bfa406bd3aecbe9c3231710ecc9fe76a47db5555fe984c9ae7 (3197 bytes)
    ├── application/vnd.oci.image.manifest.v1+json @sha256:c39972cfbdae516048d4066a3f775c7371f79c1011ddd5b26526ff5f087b6ea6 (670 bytes)
    │   │    Platform: linux/amd64
    │   └──  Content does not exist locally, skipping
    ├── application/vnd.oci.image.manifest.v1+json @sha256:1116843ad4283286db4250b9b39d760422205eb70767fcc75bf7a989c11a6e25 (836 bytes)
    │   │    Platform: unknown/unknown
    │   └──  Content does not exist locally, skipping
    ├── application/vnd.oci.image.manifest.v1+json @sha256:b50606a700c930eb1730a905912a60c4d814cbe8fcfad681334b0a40b9c6080b (670 bytes)
    │   │    Platform: linux/arm64
    │   ├── application/vnd.oci.image.config.v1+json @sha256:8b38ebd2771d01b3f5a74f3212fdbd9d0de7a917ab55431f6d69eb2d2be3a5c8 (643 bytes)
    │   ├── application/vnd.oci.image.layer.v1.tar+gzip @sha256:71136e9afec353673d4fd8470b8a5fc7446b5fa9e92d0115f3f731a0e5f2e786 (1168 bytes)
    │   └── application/vnd.oci.image.layer.v1.tar+gzip @sha256:e3acbb1ef2882ada265ef999facfd9f8b7ddd792ae222c3268436390fb4e7dda (36153925 bytes)
    ├── application/vnd.oci.image.manifest.v1+json @sha256:353a4420e045a0ba558a323c76e8118d70cbbad3b7da1388c974f3baf49b31ba (836 bytes)
    │   │    Platform: unknown/unknown
    │   └──  Content does not exist locally, skipping
    ├── application/vnd.oci.image.manifest.v1+json @sha256:64478b9fbbb950de92e1918c64bb4cfc9490f1dac868651062adc82c71146a24 (864 bytes)
    │   │    Platform: windows/amd64
    │   └──  Content does not exist locally, skipping
    ├── application/vnd.oci.image.manifest.v1+json @sha256:c0ebdb636db348044c53c8c2309fad427872b80d9fc1f61f6a3076498c15f6f0 (838 bytes)
    │   │    Platform: unknown/unknown
    │   └──  Content does not exist locally, skipping
    ├── application/vnd.oci.image.manifest.v1+json @sha256:3aed4a67891334d7075411d9aa6bc980f47aa8b8efef7564c1a891aef82a467f (864 bytes)
    │   │    Platform: windows/amd64
    │   └──  Content does not exist locally, skipping
    └── application/vnd.oci.image.manifest.v1+json @sha256:e1f5c991e5b995524e90df4c470de0d6304b8fa4bf35a86358884869c88c252a (838 bytes)
        │    Platform: unknown/unknown
        └──  Content does not exist locally, skipping
c
Copy code
brandond@dev01:~$ go install <http://github.com/rancher/wharfie@latest|github.com/rancher/wharfie@latest>

brandond@dev01:~$ wharfie --arch arm64 <http://docker.io/rancher/rke2-runtime:v1.35.7-rke2r1|docker.io/rancher/rke2-runtime:v1.35.7-rke2r1> v1.35.7-rke2r1-linux-arm64
INFO[0000] Extract mapping / => /home/brandond/v1.35.7-rke2r1-linux-arm64
INFO[0000] Pulling image reference <http://index.docker.io/rancher/rke2-runtime:v1.35.7-rke2r1|index.docker.io/rancher/rke2-runtime:v1.35.7-rke2r1>
INFO[0000] Creating directory /home/brandond/v1.35.7-rke2r1-linux-arm64/bin
INFO[0000] Extracting file bin/containerd to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/containerd
INFO[0001] Extracting file bin/containerd-shim-runc-v2 to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/containerd-shim-runc-v2
INFO[0001] Extracting file bin/crictl to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/crictl
INFO[0001] Extracting file bin/ctr to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/ctr
INFO[0002] Extracting file bin/kubectl to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/kubectl
INFO[0002] Extracting file bin/kubelet to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/kubelet
INFO[0003] Extracting file bin/runc to /home/brandond/v1.35.7-rke2r1-linux-arm64/bin/runc
INFO[0003] Creating directory /home/brandond/v1.35.7-rke2r1-linux-arm64/charts
INFO[0003] Extracting file charts/harvester-cloud-provider.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/harvester-cloud-provider.yaml
INFO[0003] Extracting file charts/harvester-csi-driver.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/harvester-csi-driver.yaml
INFO[0003] Extracting file charts/rancher-vsphere-cpi.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rancher-vsphere-cpi.yaml
INFO[0003] Extracting file charts/rancher-vsphere-csi.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rancher-vsphere-csi.yaml
INFO[0003] Extracting file charts/rke2-calico-crd.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-calico-crd.yaml
INFO[0003] Extracting file charts/rke2-calico.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-calico.yaml
INFO[0003] Extracting file charts/rke2-canal.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-canal.yaml
INFO[0003] Extracting file charts/rke2-cilium.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-cilium.yaml
INFO[0003] Extracting file charts/rke2-coredns.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-coredns.yaml
INFO[0003] Extracting file charts/rke2-flannel.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-flannel.yaml
INFO[0003] Extracting file charts/rke2-ingress-nginx.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-ingress-nginx.yaml
INFO[0003] Extracting file charts/rke2-metrics-server.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-metrics-server.yaml
INFO[0003] Extracting file charts/rke2-multus.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-multus.yaml
INFO[0003] Extracting file charts/rke2-runtimeclasses.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-runtimeclasses.yaml
INFO[0003] Extracting file charts/rke2-snapshot-controller-crd.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-snapshot-controller-crd.yaml
INFO[0003] Extracting file charts/rke2-snapshot-controller.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-snapshot-controller.yaml
INFO[0003] Extracting file charts/rke2-snapshot-validation-webhook.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-snapshot-validation-webhook.yaml
INFO[0003] Extracting file charts/rke2-traefik-crd.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-traefik-crd.yaml
INFO[0003] Extracting file charts/rke2-traefik.yaml to /home/brandond/v1.35.7-rke2r1-linux-arm64/charts/rke2-traefik.yaml

brandond@dev01:~$ find v1.35.7-rke2r1-linux-arm64/ -type f -executable | xargs -n1 file
v1.35.7-rke2r1-linux-arm64/bin/kubectl: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=c008e7873c2468063324e7d611fa57121524e94f, stripped
v1.35.7-rke2r1-linux-arm64/bin/crictl: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=8c4ebca5eb69ad1c405e1b0018798cbdd40f783f, stripped
v1.35.7-rke2r1-linux-arm64/bin/containerd-shim-runc-v2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=bce62633630187fffa51cea705ead0c51ee57b90, stripped
v1.35.7-rke2r1-linux-arm64/bin/runc: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), static-pie linked, BuildID[sha1]=5b4688f32ca36278b5e7ae95cae552233c886049, stripped
v1.35.7-rke2r1-linux-arm64/bin/ctr: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=09ee4e15e976705fb4cbad2ad84bd7852a718bd0, stripped
v1.35.7-rke2r1-linux-arm64/bin/kubelet: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=31d89183914f68e2a7166aa5c1630ff7bf060a2a, stripped
v1.35.7-rke2r1-linux-arm64/bin/containerd: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=20387afd9890e41857a8ddbe7a5823e94720d70f, stripped
b
You mount that image locally:
Copy code
slc1:~ # mkdir -p /tmp/img-check
slc1:~ # ctr -n <http://k8s.io|k8s.io> image mount --platform linux/arm64 <http://docker.io/rancher/system-agent-installer-rke2:v1.35.7-rke2r1|docker.io/rancher/system-agent-installer-rke2:v1.35.7-rke2r1> /tmp/img-check
sha256:06a6ff1b772c1515c9039781c49802ac009e3ba6d7ea03255aa63d2a953a13f0
/tmp/img-chec
c
files in the image are all the correct arch. If you are still seeing the wrong arch then it is a problem with your registry mirror.
system-agent-installer-rke2 does not contain the files that go into the data dir… those all come from rke2-runtime and are extracted by rke2 itself during startup. Why are you looking at the system agent image?
b
Copy code
slc1:~ # file /tmp/bin/rke2
/tmp/bin/rke2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=fe4ee4d4272d234eb0261076d721b2bc4cdbb3b4, stripped
Right so the rke2 binary IN the image is aarch64 ☝️
c
yes. system-agent-installer-rke2 installs rke2. rke2 extracts the rke2-runtime image into the data dir during startup. If you have the wrong things in the data dir, then you have the wrong rke2-runtime image.
b
But when you run it, the files it puts in in /var/lib/rancher/rke2/bin/* are all x86_64
c
yes, as I said you have pushed the wrong rke2-runtime image to your registry
the one on docker hub has binaries for the correct arch, as I showed above
<http://docker.io/rancher/rke2-runtime:v1.35.7-rke2r1|docker.io/rancher/rke2-runtime:v1.35.7-rke2r1>
is multiarch, it supports linux-amd64, linux-arm64, and windows/amd64
I suspect that however you loaded images into your registry, you ended up with just amd64 in there, instead of a multiarch manifest list
Copy code
brandond@dev01:~$ docker buildx imagetools inspect <http://docker.io/rancher/rke2-runtime:v1.35.7-rke2r1|docker.io/rancher/rke2-runtime:v1.35.7-rke2r1> | grep Platform
  Platform:    linux/amd64
  Platform:    unknown/unknown
  Platform:    linux/arm64
  Platform:    unknown/unknown
  Platform:    windows/amd64
  Platform:    unknown/unknown
run that same command against your registry’s copy of that image
b
It's the local registry that is created by the version-arm64.yaml
c
that sounds like a harvester thing :/ if they are populating a registry mirror with the same incorrect content as they placed in the agent/images dir - which you already found to be amd64 - then yeah that’s gonna have the same wrong stuff.
b
Copy code
slc1:~ # ctr -n k8s.io image inspect docker.io/rancher/rke2-runtime:v1.35.7-rke2r1
docker.io/rancher/rke2-runtime:v1.35.7-rke2r1
│    Created: 2026-08-24 20:28:50.815927216 +0000 UTC
│    Updated: 2026-08-25 18:33:09.497501977 +0000 UTC
│    Label "io.cattle.rke2.pinned": "pinned"
│    Label "io.cri-containerd.image": "managed"
│    Label "io.cri-containerd.pinned": "pinned"
└── application/vnd.oci.image.manifest.v1+json @sha256:d93d8c9165d7f374efee87aa84215ad8362c0257f241de677cab7da7ee5386ce (402 bytes)
    ├── application/vnd.oci.image.config.v1+json @sha256:cede63decaea9e33874801d9b7f1227bf5b70f73211eaa3224d9d3cbd47e08a0 (962 bytes)
    └── application/vnd.oci.image.layer.v1.tar @sha256:753495bae826650e76e51802731de362ba7bd61b27b282a7b63eccb7dd14f8b1 (287066112 bytes)
It's not as clear cut as the system agent (which was also wrong)
c
thats just looking at the image in your local containerd image store. not the registry itself.
b
yeah
c
I would just open an issue in the harvester repo. Clearly they just put the wrong arch in the airgap tarball. Same tarballs get loaded by rke2, and pushed into the local registry.
b
yeah I have it started over there
I didn't know it called the other image, which I think was the missing piece.
Thank you Brandon
c
it’s not the binaries. It’s the airgap image tarballs. If you keep saying binaries and pointing at stuff in the data dir people are going to get confused.
Are you SURE you didn’t do anything weird here? I am looking at the 1.8.2 iso image and it contains the correct content.
b
c
the files in the iso are correct
Copy code
brandond@dev01:/mnt/loop$ mount | grep loop
/home/brandond/harvester-v1.8.2-arm64.iso on /mnt/loop type iso9660 (ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)

brandond@dev01:/mnt/loop$ mkdir /tmp/v1.35.7-rke2r1

brandond@dev01:/mnt/loop$ zstdcat bundle/harvester/images/rke2-images.linux-arm64-v1.35.7-rke2r1.tar.zst | tar -vxO blobs/sha256/753495bae826650e76e51802731de362ba7bd61b27b282a7b63eccb7dd14f8b1 | tar -xC /tmp/v1.35.7-rke2r1
blobs/sha256/753495bae826650e76e51802731de362ba7bd61b27b282a7b63eccb7dd14f8b1
brandond@dev01:/mnt/loop$ find /tmp/v1.35.7-rke2r1/ -type f -executable | xargs file
/tmp/v1.35.7-rke2r1/bin/runc:                    ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), static-pie linked, BuildID[sha1]=5b4688f32ca36278b5e7ae95cae552233c886049, stripped
/tmp/v1.35.7-rke2r1/bin/kubectl:                 ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=c008e7873c2468063324e7d611fa57121524e94f, stripped
/tmp/v1.35.7-rke2r1/bin/ctr:                     ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=09ee4e15e976705fb4cbad2ad84bd7852a718bd0, stripped
/tmp/v1.35.7-rke2r1/bin/kubelet:                 ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=31d89183914f68e2a7166aa5c1630ff7bf060a2a, stripped
/tmp/v1.35.7-rke2r1/bin/containerd-shim-runc-v2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=bce62633630187fffa51cea705ead0c51ee57b90, stripped
/tmp/v1.35.7-rke2r1/bin/crictl:                  ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=8c4ebca5eb69ad1c405e1b0018798cbdd40f783f, stripped
/tmp/v1.35.7-rke2r1/bin/containerd:              ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=20387afd9890e41857a8ddbe7a5823e94720d70f, stripped
Whatever is going on, it seems it is local to your environment. the files on docker hub are correct. the files in the iso are correct.
b
I'll mount the iso and copy them over manually
c
can you compare the image tarballs from the iso to what is on your node? How did you end up with the wrong ones?
b
There's def some amd tar there. We've restarted the upgrade multiple times for reasons ™️ but it it took a while for preloading these images so it SEEMed like it was validating checksums, but obviously at some point it wasn't right and there's the wrong files on here.
c
well the only one that provides the content you are having problems with is rke2-runtime and that one looks good. I checked the other tarballs and its not in any of them.
b
yeah. I was able to copy over the images from the iso and it started the service with the correct version.
Thanks for looking. 🙂
2 Views
Open in Slack
PreviousNext
Powered by