https://rancher.com/ logo
Title
s

sparse-businessperson-74827

05/10/2023, 1:21 PM
hi, how do I add add
extra_arg
in RKE2 cluster? I'm running Rancher 2.7.1 with RKE2 1.24.9 I've found this link https://rke.docs.rancher.com/config-options/services/services-extras#extra-args but it is not clear where/how to add these options. I've check in
Cluster management
and went to edit YAML but that does not look like a right place. Any pointers? Thank you
b

bulky-sunset-52084

05/10/2023, 1:34 PM
You set your settings in the
config.yaml
at
/etc/rancher/rke2/config.yaml
. you can also set the config options for clusters via the cluster crd under the
rkeConfig
section https://docs.rke2.io/install/configuration
s

sparse-businessperson-74827

05/10/2023, 1:36 PM
Thank you, can I set it of existing cluster?
b

bulky-sunset-52084

05/10/2023, 1:37 PM
Yup - update the file then restart the rke2 service in the node
Or just deploy that file to all your nodes with something like Ansible then restart them all ;)
s

sparse-businessperson-74827

05/10/2023, 1:53 PM
so there is no
config.yaml
at
/etc/rancher/rke2
but there is
/etc/rancher/rke2/config.yaml.d/50-rancher.yaml
but it it's not yaml file and I'm not sure it is correct one. I just need to modify
eviction-hard
to modify disk pressure thresholds on the nodes
b

bulky-sunset-52084

05/10/2023, 1:56 PM
Oh yea its different when rancher provisions it - sorry forgot about that. Yes that 50-rancher.yaml is the correct file. It's just in json instead of yaml.
s

sparse-businessperson-74827

05/10/2023, 2:40 PM
hm, it does not take my setting. I've added below but then then the service would not start
"kubelet": [
    "extra_args": [
      "eviction-hard: "imagefs.available<5%,nodefs.available<5%" ]]
b

bulky-sunset-52084

05/10/2023, 3:05 PM
yea that's not quite exactly the correct syntax - so how the config.yaml file is set up is its based on the flags you pass the rke2 binary when it starts - say for example you are starting a server node - these are going to be all the flags: https://docs.rke2.io/reference/server_config most important to this is probably
--kubelet-arg
so take that flag and convert it to JSON and put it in that 50-rancher.yaml file.
"kubelet-args": [
  "eviction-hard: "imagefs.available<5%,nodefs.available<5%"
]
I'll test this too - just give me a few - its possible I messed up the syntax here
s

sparse-businessperson-74827

05/10/2023, 3:28 PM
tried it but service would not start
b

bulky-sunset-52084

05/10/2023, 3:30 PM
I realized that i missed a close comma after
eviction-hard
that might have something to do with it
also it should tell you in the rke2 logs as to what part of the config it doesn't like
s

sparse-businessperson-74827

05/10/2023, 3:33 PM
ok, the
"
was missing, here is that worked so far:
"kubelet-args": [
    "eviction-hard": "imagefs.available<5%,nodefs.available<5%"
  ],
now need to test it
df -h
it does not make any difference, in the logs I'm getting this:
Flag --eviction-minimum-reclaim has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
b

bulky-sunset-52084

05/10/2023, 3:49 PM
ahh dang yea the kubelet switched to a config file for it's configurations... yea let me look it up
s

sparse-businessperson-74827

05/10/2023, 3:49 PM
I've made yaml with
kind: KubeletConfiguration
but when trying to apply getting
no matches for kind "KubeletConfiguration" in version "<http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>"
b

bulky-sunset-52084

05/10/2023, 3:51 PM
yea that wont be applied to the cluster it will be referenced by kubelet. Much like setting max pods. https://github.com/rancher/rke/issues/1298#issuecomment-1482008422
s

sparse-businessperson-74827

05/10/2023, 4:23 PM
I'm losing will to live 🙂 so I have my kube-config.yaml
apiVersion: <http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>
kind: KubeletConfiguration
evictionHard:
  nodefs.available: "1Gi"
  imagefs.available: "100Gi"
  nodefs.inodesFree: "1%"
I made a change in 50-rancher.yaml file to load my file:
"kubelet-arg": [
    "config=/tmp/kube-config.yaml"
  ],
the service restarts the logs do not show problems loading file
time="2023-05-10T16:16:00Z" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --config=/tmp/kube-config.yaml --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=dev-master-3 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=10.0.0.13 --node-labels=<http://cattle.io/os=linux,rke.cattle.io/machine=af095b89-819a-4320-9e60-9ac1d3d2f824,topology.kubernetes.io/region=hel1,topology.kubernetes.io/zone=hel1-dc2|cattle.io/os=linux,rke.cattle.io/machine=af095b89-819a-4320-9e60-9ac1d3d2f824,topology.kubernetes.io/region=hel1,topology.kubernetes.io/zone=hel1-dc2> --pod-infra-container-image=<http://index.docker.io/rancher/pause:3.6|index.docker.io/rancher/pause:3.6> --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --register-with-taints=<http://node-role.kubernetes.io/control-plane:NoSchedule,node-role.kubernetes.io/etcd:NoExecute|node-role.kubernetes.io/control-plane:NoSchedule,node-role.kubernetes.io/etcd:NoExecute> --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key"
but from above I still see that it doe not take my config as
--eviction-hard=imagefs.available<5%,nodefs.available<5%
and I still seeing
Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
Any ideas what is missing?