https://rancher.com/ logo
Join Slack
Powered by
:wave: Hi, I'm working on certification of Rancher...
# general
b
đź‘‹ Hi, I'm working on certification of Rancher/k3s for the ECK operator for Elastic and I'm running into a bit of an issue with cgroups. It seems to be this user's exact issue reported here: https://github.com/k3s-io/k3s/issues/2500. The environment that we are running within is k3d/k3s and appears to be "hybrid" (link) . This 
/sys/fs/cgroup/cpu,cpuacct/cpu.cfs_quota_us
is set to 
-1
in our environment, so cpu limits simply aren't supported. We have a nearly identical set of e2e tests that we have been running for a long period of time using 
kind
that doesn't present this issue. Are you aware of any known issues with this type of setup? I'm happy to provide additional information as requested. Thanks in advance for the assist!
Here's what the 
check-config
tool shows:
Copy code
INFO[0000] Acquiring lock file /opt/buildkite-agent/.rancher/k3s/data/.lock
INFO[0000] Preparing data dir /opt/buildkite-agent/.rancher/k3s/data/8460fc4df8e204d507e141d109ea776ef78412e65121eeb3cf7c67ee581d699f
cat: /sys/kernel/security/apparmor/profiles: Permission denied
Verifying binaries in /opt/buildkite-agent/.rancher/k3s/data/8460fc4df8e204d507e141d109ea776ef78412e65121eeb3cf7c67ee581d699f/bin:
- sha256sum: good
- links: good
System:
- /usr/sbin iptables v1.8.4 (legacy): ok
- swap: disabled
- routes: ok
Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000
modprobe: FATAL: Module configs not found in directory /lib/modules/5.15.0-1083-gcp
info: reading kernel config from /boot/config-5.15.0-1083-gcp ...
Generally Necessary:
- cgroup hierarchy: cgroups Hybrid mounted, cpuset|memory controllers status: good
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
c
check-config just checks your kernel config, it does not look at how cgroups are actually managed
Also, the fact that you are using 
/opt/buildkite-agent/.rancher/k3s/data
suggests that you are running rootless?
in which case cgroup v2 is a hard requirement
b
Well this is great to know. Thank you. I'll see if I can figure out how to get a pure v2 environment.
c
this is covered in the rootless docs:
Cgroup v1 and Hybrid v1/v2 are not supported; only pure Cgroup v2 is supported. If K3s fails to start due to missing cgroups when running rootless, it is likely that your node is in Hybrid mode, and the “missing” cgroups are still bound to a v1 controller.
Just out of curiosity, why are you running rootless? It has a LOT of odd behavior compared to normal k3s or k8s.
b
this is how things are currently set up in our CI environment unfortunately. I'll see about finding another path.
c
Are you running this in a container? or somehow running rootless k3s in as a systemd user service in your CI pipeline?
b
again, thanks for the quick assist!
2 Views
Open in Slack
PreviousNext
Powered by