https://rancher.com/ logo
#rke2
Title
b

blue-controller-9088

01/31/2023, 7:01 PM
Does anybody know what is the purpose of system-upgrade-controller in RKE2 spinned up as downstream cluster? I noticed it’s getting installed automatically. From what I can see in the docs https://docs.rke2.io/upgrade/automated_upgrade it can manage automated cluster upgrades. Do I really need it if I initiate upgrades manually, via terraform or UI?
c

creamy-pencil-82913

01/31/2023, 7:32 PM
if you’re using Rancher to provision the cluster, the SUC is used to manage version and configuration of the Rancher System Agent as installed on the nodes. The System Agent in turn manages the installation and configuration of RKE2 on the node, including Kubernetes version upgrades. This is done in order to avoid a dependency loop of having the System Agent manage its own updates.
This is more of a Rancher thing than a RKE2 thing.
b

blue-controller-9088

01/31/2023, 7:44 PM
The reason I asked is that the cluster is flooded with the pods apply-system-agent-upgrader-on-*, each of them with similar errors: ---
apply-system-agent-upgrader-on-phygatapp24-lab-with-2d753-zfhvx
+ TMPDIRBASE=/var/lib/rancher/agent/tmp
+ mkdir -p /host/var/lib/rancher/agent/tmp
mkdir: can't create directory '/host/var/lib/rancher/': No such file or directory
--- I thought it is something to do with cis-1.6 profile, but I disabled it and also have unresticted PSP. Same issue. Appreciate any suggestions on how to fix it.
image.png
c

creamy-pencil-82913

01/31/2023, 7:49 PM
Yeah that’s all managed by Rancher, I’m not sure what might be causing that. Have you looked at the pod logs to see if there are any prior errors? It looks like it’s expecting the host root filesystem to be mounted at /host, is that the case?
What OS are the nodes on?
b

blue-controller-9088

01/31/2023, 7:50 PM
ubuntu 21.04
Could it be because /var/lib/rancher is a link to /kubernetes/rancher?
c

creamy-pencil-82913

01/31/2023, 7:51 PM
yes that would probably do it
I am not sure there’s going to be an easy fix for that, other than re-rolling your nodes without symlinks in the paths
b

blue-controller-9088

01/31/2023, 7:53 PM
perhaps I’ll need to create a separate fs instead of doing link like I did in RKE1.
it’s POC cluster, expandable. Still looking at RKE2 as a replacement for RKE1
c

creamy-pencil-82913

01/31/2023, 7:57 PM
yeah, having /var/lib/rancher on another FS should be fine. It’s probably just getting confused by the symlink.
b

blue-controller-9088

01/31/2023, 8:02 PM
I’ll try hardlink. The philosophy behind is to have a separate FS for k8s stuff (pod logs, kubeliet, etcd, …)
so far was working in RKE1
c

creamy-pencil-82913

01/31/2023, 8:05 PM
you can’t hardlink across filesystems
b

blue-controller-9088

01/31/2023, 8:07 PM
yes, I know, meant other way around, bring kubelet and pods logs to /var/lib/rancher which is a dedicated FS
Thank you Brandon. That was it, symlink. Dedicated FS for /var/lib/rancher resolved the issue.
11 Views