So we re rolling through some Harvester Uprades and it looks Rancher Users #harvester

So we're rolling through some Harvester Uprades an...

bland-article-62755

08/24/2026, 10:39 PM

So we're rolling through some Harvester Uprades and it looks like the amd64 iso is getting loaded/downloaded by default.

Copy code

slc7:~ # crictl images | grep harvester-upgrade
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/crictl.yaml"
docker.io/rancher/harvester-upgrade                                         v1.5.1                                                ca07e962bc3fc       150MB
docker.io/rancher/harvester-upgrade                                         v1.6.1                                                c67e8da79b88b       529MB
docker.io/rancher/harvester-upgrade                                         v1.7.1                                                09601d4b1cd6c       544MB
docker.io/rancher/harvester-upgrade                                         v1.8.2                                                5f5e1ce7388a2       553MB
slc7:~ # sudo crictl inspecti 5f5e1ce7388a2 | grep -i architecture
sudo: crictl: command not found
slc7:~ # crictl inspecti 5f5e1ce7388a2 | grep -i architecture
WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/crictl.yaml"
      "architecture": "amd64",

(solved... kinda... not really. )

bland-article-62755

08/24/2026, 10:49 PM

While I don't remember this from before, but it's been a long summer, there's different yaml for arm clusters: releases.rancher.com/harvester/v1.8.2/version-arm64.yaml

bland-article-62755

08/24/2026, 10:56 PM

Kinda because the logging has to be disabled.

bland-article-62755

08/24/2026, 10:57 PM

We tried it multiple times. The first time logging worked just fine... then after that, it consistently got the amd image instead of the arm image.

bland-article-62755

08/24/2026, 11:19 PM

Ugh the wrong image arch broke it

bland-article-62755

08/24/2026, 11:20 PM

Copy code

crictl images 2>/dev/null | tail -n +2 | while read -r repo tag id rest; do
  arch=$(crictl inspecti "$id" 2>/dev/null | grep -m 1 '"architecture"' | cut -d '"' -f 4)
  if [ "$arch" = "amd64" ]; then
    echo "Purging amd64 image: $repo:$tag"
    crictl rmi "$id" 2>/dev/null || echo "  -> Failed to delete"
  fi
done

bland-article-62755

08/25/2026, 3:13 PM

Copy code

Rke Service trying to restart is super failing

Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ']'"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stdout]: /usr/local is ro or a mount point"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: ++ cat /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: + PRIOR_RESTART_STAMP=94e214a014f79365c0075a449a89b62198c9f1998457327c24fb35900f982354"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: + '[' -n c1c3e577829fd744d59386bd88e1384b0437fb5031e439ac874ba08bbb688607 ']'"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: + '[' 94e214a014f79365c0075a449a89b62198c9f1998457327c24fb35900f982354 '!=' c1c3e577829fd744d59386bd88e1384b0437fb5031e439ac874ba08bbb688607 ']'"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: + RESTART=true"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20260825-093625/13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0 INSTALL_RKE2_TAR_PREFIX=/opt/rke2 installer.sh"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20260825-093625/13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0/sha256sum-arm64.txt"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0:stderr]: cp: cannot stat '/var/lib/rancher/agent/work/20260825-093625/13203543564eda0aa4b76188608f21fa24a0ef0d8a632b708c7ff1601012db24_0/sha256sum-arm64.txt': No such file or directory"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 1"
Aug 25 09:36:27 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:27Z" level=error msg="error executing instruction 0: <nil>"
Aug 25 09:36:28 slc5 rancher-system-agent[2240999]: time="2026-08-25T09:36:28Z" level=info msg="[K8s] updated plan secret fleet-local/custom-05ea8545ca94-machine-plan with feedback"

bland-article-62755

08/25/2026, 3:32 PM

There's somehow a bunch of arm64 images back on the nodes? I'm guessing those aren't suppose to be there.

bland-article-62755

08/25/2026, 6:11 PM

(I might be crazy)

👀 1

bland-article-62755

08/25/2026, 6:24 PM

See: https://rancher-users.slack.com/archives/C01PHNP149L/p1787682073428079?thread_ts=1787681745.793449&cid=C01PHNP149L

miniature-salesclerk-33951

08/25/2026, 6:45 PM

Copy code

slc5:~ # find /var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/ -type f | xargs -n1 file
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/runc: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked, BuildID[sha1]=393e71aed9aa8b8a93dcae1b9545ec6c190fef8d, stripped
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/ctr: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=2c7ded5e504c83a8ba6dd6c829c81726d13c64c3, stripped
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/kubelet: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=8166b66bec5766b15f9e107ad75a02d2451fcb83, stripped
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/kubectl: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=31fbef75e8ce1ece93b5ecdb4c6e57cf74be57cd, stripped
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/containerd-shim-runc-v2: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=a51f6e598bb53a006b053d1a1e9afaf6b54bc312, stripped
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/containerd: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=33098ab8baecd5b36899939762a6797189482784, stripped
/var/lib/rancher/rke2/data/v1.35.7-rke2r1-74310658583d/bin/crictl: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=e00d43b36a1f3ee1f793dc1a812b8df9191b5be8, stripped
slc5:~ # find /var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/ | xargs -n 1 file
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/: directory
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/runc: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), static-pie linked, BuildID[sha1]=fb93a5917535b7b649743cc97b956774eedeaeb3, stripped
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/ctr: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=9b6f86883fd6d0d05699be6e502edaa5f6f2ab9d, stripped
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/kubelet: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=86b562b20d3a442a45ef8b781d535d99be694e21, stripped
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/kubectl: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=hvd6ZQjC7jcEU2GnhkoT/ZHP8hK603etTQZncxu9O/GTOrrs65q3tfAd3PwDBo/FKdAQZ-Xshe167XPg8l6, BuildID[sha1]=51b5cd77c787b71990613cd9b969515e5caa4da9, stripped
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/containerd-shim-runc-v2: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=c40fb9bc751b4a04303b38c5ee4c3377776f94b8, stripped
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/containerd: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=5582f505f77a8c6311d73ab0bb1f4af39fe14532, stripped
/var/lib/rancher/rke2/data/v1.34.3-rke2r3-5b8349de68df/bin/crictl: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=c97233e0fd8dcbf307c40c959f44c5c86e0e19d9, stripped

Verifying that v1.34.3 is arm64 (correct), but version v1.35.7 is amd64/x86 (🔥 )

creamy-pencil-82913

08/25/2026, 6:55 PM

I can confirm that the published image for that version(

<http://docker.io/rancher/rke2-runtime:v1.35.7-rke2r1|docker.io/rancher/rke2-runtime:v1.35.7-rke2r1>

) contains binaries with the correct architecture, so I suspect the issue is with the airgap image tarballs bundled into the OS

👍 1

bland-article-62755

08/25/2026, 7:29 PM

github.com/harvester/harvester/issues/11507

miniature-salesclerk-33951

08/25/2026, 7:49 PM

I assume the ones staged here on the nodes:

Copy code

ls /var/lib/rancher/agent/images/rancherd-bootstrap-images-v1.
rancherd-bootstrap-images-v1.5.1.tar.zst  rancherd-bootstrap-images-v1.6.1.tar.zst  rancherd-bootstrap-images-v1.7.1.tar.zst  rancherd-bootstrap-images-v1.8.2.tar.zst
rancherd-bootstrap-images-v1.5.1.txt      rancherd-bootstrap-images-v1.6.1.txt      rancherd-bootstrap-images-v1.7.1.txt      rancherd-bootstrap-images-v1.8.2.txt

miniature-salesclerk-33951

08/25/2026, 7:52 PM

Yep, that checks out:

Copy code

grep -R amd64
grep: blobs/sha256/f63d2e943e35c0be8107fcc01dd0f95b210289536bde53dd8a818f759c8704c3: binary file matches
blobs/sha256/5682fb95f267bececee3c97defae8afdcc929afe793a99b0f4bd862736297457:{"id":"de522d91cba5d27aa9f96cb6b8fd019f975c4ad3dbcb2103101eaa1aaab4a386","parent":"0017849a9fc8aff7c6e0d81773746ce9247b80ace217f7156620c9702ae24c3b","created":"2026-08-04T20:46:03.921427858Z","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"/","Entrypoint":null,"OnBuild":null,"Labels":null},"architecture":"amd64","os":"linux"}
blobs/sha256/4302c8b05603b04558d9e793c0d5dae3e4f8e420cba2400ed1b5670432386bb9:{"id":"ca69e6738202487b21480c4efdbc5ba7cfc5f47526d85233486f0c200b588e52","parent":"7617b34e62b0479e4b775976d1b6c597a6cb75516a28ad1b9087a92c7852bdb7","created":"2026-06-29T20:19:42.922977999Z","container_config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":null,"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"","Entrypoint":null,"OnBuild":null,"Labels":null},"config":{"Hostname":"","Domainname":"","User":"","AttachStdin":false,"AttachStdout":false,"AttachStderr":false,"Tty":false,"OpenStdin":false,"StdinOnce":false,"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Cmd":null,"Image":"","Volumes":null,"WorkingDir":"/","Entrypoint":null,"OnBuild":null,"Labels":{"org.opencontainers.image.created":"2026-06-29T20:19:08.986Z","org.opencontainers.image.description":"Complete container management platform","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"4603afd276483411c06a2b716eca38931b564e48","org.opencontainers.image.source":"<https://github.com/rancher/rancher>","org.opencontainers.image.title":"rancher","org.opencontainers.image.url":"<https://github.com/rancher/rancher>","org.opencontainers.image.version":"v2.14.3"}},"architecture":"amd64","os":"linux"}
blobs/sha256/1fa1b290d435613f8dfafdd14f7ac0e7f5f78f608d56db7703acc799c9692911:{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"WorkingDir":"/","Labels":{"org.opencontainers.image.created":"2026-06-29T20:19:08.986Z","org.opencontainers.image.description":"Complete container management platform","org.opencontainers.image.licenses":"Apache-2.0","org.opencontainers.image.revision":"4603afd276483411c06a2b716eca38931b564e48","org.opencontainers.image.source":"<https://github.com/rancher/rancher>","org.opencontainers.image.title":"rancher","org.opencontainers.image.url":"<https://github.com/rancher/rancher>","org.opencontainers.image.version":"v2.14.3"}},"created":"2026-06-29T20:19:42.922977999Z","history":[{"created":"2026-06-29T20:19:42.762737128Z","created_by":"COPY /usr/bin/helm_v3 /helm # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-06-29T20:19:42.8804161Z","created_by":"COPY /src/bin/chart/*/*.tgz / # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-06-29T20:19:42.922977999Z","created_by":"COPY package/installer-run.sh /run.sh # buildkit","comment":"buildkit.dockerfile.v0"}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:1ad1d638fbe51f179335bcba693b3aae9b665f56880daa1df9bbd8a5aeb4f27b","sha256:879bcd8efb8fe27d348b5f40cf89f2c3d09271c84ab68355e6dfe60c42ee9206","sha256:4bbec84ce4182427469045ccc0fe3715a1995eafce115e966b6986a9242b5d3f"]}}
grep: blobs/sha256/1ad1d638fbe51f179335bcba693b3aae9b665f56880daa1df9bbd8a5aeb4f27b: binary file matches
blobs/sha256/017ae3a2514c7e8af85a57c3c20af354cce41755f1bcbcd4b62fbd7de80b94f4:{"architecture":"amd64","config":{"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"WorkingDir":"/"},"created":"2026-08-04T20:46:03.921427858Z","history":[{"created":"2026-08-04T20:46:03.531096749Z","created_by":"COPY package/run.sh /run.sh # buildkit","comment":"buildkit.dockerfile.v0"},{"created":"2026-08-04T20:46:03.921427858Z","created_by":"COPY artifacts/* / # buildkit","comment":"buildkit.dockerfile.v0"}],"os":"linux","rootfs":{"type":"layers","diff_ids":["sha256:1d84e2a65603f160d015a15a922ac890e0c26fcee9ef2920b4e6b83a49ce9335","sha256:f63d2e943e35c0be8107fcc01dd0f95b210289536bde53dd8a818f759c8704c3"]}}

miniature-salesclerk-33951

08/25/2026, 7:55 PM

I'm guessing replacing the /var/lib/rancher/agent/images/rancherd-bootstrap-images-v1.8.2.tar.zst with the arm64 version would get it to progress.

creamy-pencil-82913

08/25/2026, 8:33 PM

bundle/rancherd/images/rancherd-bootstrap-images-v1.8.2.tar.zst

on the ISO also has the correct arch in it. And that is just the system-agent-installer-rke2 and system-agent-installer-rancher images, NOT the rke2-runtime image.

creamy-pencil-82913

08/25/2026, 8:34 PM

idk how you ended up with all this amd64 stuff on there but it wasn’t from the arm64 iso

bland-article-62755

08/25/2026, 8:38 PM

yeah I'm baffled.

bland-article-62755

08/25/2026, 9:36 PM

purging all the images and placing the tar in the airgap folder allowed the service to start as expected, but some of the pods are now struggling to get created:

Copy code

slc5:/tmp # kubectl get pods -n harvester-system -o wide | grep slc5
harvester-network-controller-x8mh5                          0/1     CreateContainerError   0               21h     10.16.2.5      slc5   <none>           <none>
harvester-networkfs-manager-7tf2m                           0/1     CreateContainerError   0               21h     10.16.2.5      slc5   <none>           <none>
harvester-node-disk-manager-sw4cz                           0/1     CreateContainerError   0               21h     10.16.2.5      slc5   <none>           <none>
harvester-node-manager-drjx2                                0/1     CreateContainerError   0               21h     10.52.1.92     slc5   <none>           <none>
hvst-upgrade-7zqlp-post-drain-slc5-qb572                    0/1     ContainerCreating      0               7m23s   10.16.2.5      slc5   <none>           <none>
hvst-upgrade-7zqlp-upgradelog-agent-fluentbit-r4grg         0/1     CreateContainerError   0               14h     10.52.1.94     slc5   <none>           <none>
kube-vip-d49q7                                              0/1     CreateContainerError   0               21h     10.16.2.5      slc5   <none>           <none>
virt-handler-hgd56                                          0/1     Unknown                0               21h     10.52.1.95     slc5   <none>           <none>

bland-article-62755

08/25/2026, 10:45 PM

Had to delete the local containerd metadata

Copy code

systemctl stop rke2-server
rm -rf /var/lib/rancher/rke2/agent/containerd/io.containerd.metadata.v1.bolt/meta.db
systemctl start rke2-server

creamy-pencil-82913

08/25/2026, 11:21 PM

probably easier than using ctr to delete all the images yeah

creamy-pencil-82913

08/25/2026, 11:22 PM

I would probably just

rm -rf /var/lib/rancher/rke2/agent/containerd

though, now you have a db file that is out of sync with the on disk contents

bland-article-62755

08/25/2026, 11:23 PM

good call

miniature-salesclerk-33951

08/27/2026, 12:16 AM

I see this in the releases.rancher.com/harvester/v1.8.2/… iso I downloaded this afternoon, so it looks like there's at least x86 multus in there

creamy-pencil-82913

08/27/2026, 12:23 AM

the multus file is incorrectly named. If you look in it, it actually has arm64 images. we checked those too.

👍 2

7 Views

Open in Slack

Previous Next