https://rancher.com/ logo
Join Slack
Powered by
Hello, I'm having trouble running RKE2 using with ...
# rke2
w
Hello, I'm having trouble running RKE2 using with 
cni: cilium
. The pods 
helm-install-*
(except for 
cilium
and 
coredns
) as well as 
rke2-coredns-*
are stuck at 
ContainerCreating
with the following error message:
Copy code
Warning  FailedCreatePodSandBox  2m40s                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4f288707dd56dc2bd0e6c511b3cadfaf21d310eb63401c272956bee0ee68c838": plugin type="portmap" failed (add): failed to find plugin "portmap" in path [/opt/cni/bin]
I have verified that 
portmap
is indeed not present in 
/opt/cni/bin
. Other binaries (
loopback
, 
cilium-cni
) have, however, successfully been installed. None of my configuration should be out of the ordinary (I have tried both with and without 
kubeProxyReplacement
). SELinux is enabled on he host system.
c
these should be installed by the CNI. Are you sure that all the containers for the CNI pod ran successfully? Do you have something that might have deleted the files off the node after they were created by the CNI install init container?
w
Hi Brad, thank you for your answer! I think you have pointed in the right direction: The cilium pod has started meaning that all init containers exited with code 0. However, while 
install-cni-binaries
has not reported any issues,:
Copy code
Installing loopback to /host/opt/cni/bin/loopback ...
Wrote /host/opt/cni/bin/loopback
Installing cilium-cni to /host/opt/cni/bin/cilium-cni ...
Wrote /host/opt/cni/bin/cilium-cni
the init container 
install-portmap-cni-plugin
logged the following:
Copy code
2025-02-24T08:41:24.796598281-05:00 bandwidth is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.797785437-05:00 bridge is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.799744289-05:00 dhcp is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.803007076-05:00 cp: can't create '/host/opt/cni/bin/dummy': Permission denied
2025-02-24T08:41:24.803469043-05:00 ./install-cnis.sh: line 30: exit_with_error: not found
2025-02-24T08:41:24.805267202-05:00 firewall is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.806858841-05:00 flannel is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.808328237-05:00 host-device is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.810601965-05:00 host-local is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.812120978-05:00 ipvlan is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.813898048-05:00 loopback is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.815476626-05:00 macvlan is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.818599541-05:00 cp: can't create '/host/opt/cni/bin/portmap': Permission denied
2025-02-24T08:41:24.818834834-05:00 ./install-cnis.sh: line 30: exit_with_error: not found
2025-02-24T08:41:24.820595644-05:00 ptp is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.822765582-05:00 sbr is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.824294903-05:00 static is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.827745681-05:00 cp: can't create '/host/opt/cni/bin/tap': Permission denied
2025-02-24T08:41:24.828224071-05:00 ./install-cnis.sh: line 30: exit_with_error: not found
2025-02-24T08:41:24.830959110-05:00 tuning is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.834755651-05:00 vlan is in SKIP_CNI_BINARIES, skipping
2025-02-24T08:41:24.836475640-05:00 vrf is in SKIP_CNI_BINARIES, skipping
I can relate this issue to an audit denial:
Copy code
Feb 24 08:41:24 rke-03 kernel: audit: type=1400 audit(1740404484.800:6): avc:  denied  { write } for  pid=7856 comm="cp" name="bin" dev="vda3" ino=54690595 scontext=system_u:system_r:container_t:s0:c192,c991 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0
Feb 24 08:41:24 rke-03 kernel: audit: type=1400 audit(1740404484.816:7): avc:  denied  { write } for  pid=7866 comm="cp" name="bin" dev="vda3" ino=54690595 scontext=system_u:system_r:container_t:s0:c192,c991 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0
Feb 24 08:41:24 rke-03 kernel: audit: type=1400 audit(1740404484.825:8): avc:  denied  { write } for  pid=7871 comm="cp" name="bin" dev="vda3" ino=54690595 scontext=system_u:system_r:container_t:s0:c192,c991 tcontext=system_u:object_r:var_t:s0 tclass=dir permissive=0
So it seems that while 
install-cni-binaries
can successfully install the binaries to 
/opt/cni/bin/
(even with SELinux enabled), 
install-portmap-cni-plugin
can, for some reason, not.
This is a comparison of the init containers' manifests:
Copy code
env:
- name: SKIP_CNI_BINARIES
  value: bandwidth,bridge,dhcp,firewall,flannel,host-device,host-local,ipvlan,loopback,macvlan,ptp,sbr,static,tuning,vlan,vrf
image: rancher/hardened-cni-plugins:v1.6.0-build20241022
imagePullPolicy: IfNotPresent
name: install-portmap-cni-plugin
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /host/opt/cni/bin
  name: cni-path
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: kube-api-access-8mb98
  readOnly: true
vs.
Copy code
command:
- /install-plugin.sh
image: rancher/mirrored-cilium-cilium:v1.16.5
imagePullPolicy: IfNotPresent
name: install-cni-binaries
resources:
  requests:
    cpu: 100m
    memory: 10Mi
securityContext:
  capabilities:
    drop:
    - ALL
  seLinuxOptions:
    level: s0
    type: spc_t
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: FallbackToLogsOnError
volumeMounts:
- mountPath: /host/opt/cni/bin
  name: cni-path
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  name: kube-api-access-8mb98
  readOnly: true
Strikingly, 
securityContext.seLinuxOptions
is configured for 
install-cni-binaries
but not for 
install-portmap-cni-plugin
.
c
did you install the rke2-selinux package? Is rke2 running with selinux enabled?
w
Yes and yes
c
Are you using any nonstandard paths? What distro is this on?
w
/opt/cni
is a symlink to 
/var/cni
. This is AlmaLinux 9.5
So I'll guess I'll have to change the filesystem layout (given 
/opt
is read-only in our case) unless there is some way to not break SELinux policies
c
yeah could be related? If you can reproduce without symlinks then I’d suggest opening an issue with Cilium… but in general selinux policies only apply to the default paths. If you go moving stuff around you will have to maintain your own rules for your custom paths.
w
I see. Thank you!
I could in fact not reproduce it with 
/opt/cni
residing on a separate disk without any symlinking
23 Views
Open in Slack
PreviousNext
Powered by