in case anyone is wondering... I did a fresh cluster install using rancher manager but still choosing the cilium cni. Then I tried to manually install cilium using their cilium binary and one by one deleting exisiting resources it was complaining about like clusterroles, daemonsets and clusterrolebindings until it worked. Probably not the most elegant way but it worked.

rogue@TUMBLER:~/.kube$ cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode) \__/¯¯\__/ Hubble Relay: OK \__/ ClusterMesh: disabled DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3 Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1 Deployment hubble-relay Desired: 1, Ready: 1/1, Available: 1/1 Containers: cilium Running: 3 cilium-operator Running: 1 hubble-relay Running: 1 Cluster Pods: 21/21 managed by Cilium Helm chart version: 1.13.4 Image versions cilium-operator quay.io/cilium/operator-generic:v1.13.4@sha256:09ab77d324ef4d31f7d341f97ec5a2a4860910076046d57a2d61494d426c6301: 1 hubble-relay quay.io/cilium/hubble-relay:v1.13.4@sha256:bac057a5130cf75adf5bc363292b1f2642c0c460ac9ff018fcae3daf64873871: 1 cilium quay.io/cilium/cilium:v1.13.4@sha256:bde8800d61aaad8b8451b10e247ac7bdeb7af187bb698f83d40ad75a38c1ee6b: 3

that hubble relay was enabled after the fact with a cilium hubble enable, which was initially a problem but working now.