Hi I ve deployed a RKE2 cluster with Cilium as the CNI and i rancher-users #rke2

Title

proud-salesmen-12221

11/14/2022, 12:45 AM

Hi, I've deployed a RKE2 cluster with Cilium as the CNI and installed the Rancher Management app and after that it works great, I'm able to access the Rancher Management UI via a browser. But, once I enable IPSec in Cilium, I'm not longer able to hit the UI. Do you have any tips on how I can debug this?

bland-account-99790

11/14/2022, 2:33 PM

How did you enable IPSec in Cilium?

proud-salesmen-12221

11/14/2022, 3:55 PM

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-cilium
  namespace: kube-system
spec:
  valuesContent: |-
    k8sServiceHost: rke2-server1
    k8sServicePort: 6443
    operator:
      replicas: 1
    encryption:
      enabled: true
      type: ipsec
    hubble:
      enabled: true
    ...
    ...

I applied this

bland-account-99790

11/14/2022, 3:56 PM

You first deployed rke2+cilium and after that applied that?

proud-salesmen-12221

11/14/2022, 3:57 PM

yes

Now Cilium reports this:

root@rke2-server1:/home/cilium# cilium status
KVStore:                 Ok   Disabled
Kubernetes:              Ok   1.24 (v1.24.4+rke2r1) [linux/amd64]
Kubernetes APIs:         ["cilium/v2::CiliumClusterwideNetworkPolicy", "cilium/v2::CiliumEndpoint", "cilium/v2::CiliumNetworkPolicy", "cilium/v2::CiliumNode", "core/v1::Namespace", "core/v1::Node", "core/v1::Pods", "core/v1::Service", "discovery/v1::EndpointSlice", "<http://networking.k8s.io/v1::NetworkPolicy|networking.k8s.io/v1::NetworkPolicy>"]
KubeProxyReplacement:    Probe
Host firewall:           Disabled
CNI Chaining:            none
Cilium:                  Ok   1.12.0 (v1.12.0-9447cd1)
NodeMonitor:             Listening for events on 4 CPUs with 64x4096 of shared memory
Cilium health daemon:    Ok
IPAM:                    IPv4: 3/254 allocated from 10.42.0.0/24,
BandwidthManager:        Disabled
Host Routing:            Legacy
Masquerading:            IPTables [IPv4: Enabled, IPv6: Disabled]
Controller Status:       25/25 healthy
Proxy Status:            OK, ip 10.42.0.118, 0 redirects active on ports 10000-20000
Global Identity Range:   min 256, max 65535
Hubble:                  Ok   Current/Max Flows: 4095/4095 (100.00%), Flows/s: 6.88   Metrics: Disabled
Encryption:              IPsec
Cluster health:          3/3 reachable   (2022-11-14T15:54:59Z)

Hubble is active as well and I'm able to observe traffic using it

bland-account-99790

11/14/2022, 3:58 PM

So Cilium was deployed with one config and then "redeployed" using ipsec. I wonder if that's supported 🤔. Any reason why you did not deploy directly with ipsec?

What error you see when accessing the UI?

proud-salesmen-12221

11/14/2022, 3:59 PM

I didn't have a secret setup on the original deployment

I just get a Failed to Connect. Connection Refused.

$ kctl get pods -n cattle-system
NAME                               READY   STATUS    RESTARTS   AGE
rancher-7c676f75c-fc4dj            1/1     Running   0          16h
rancher-webhook-66dcd7db66-75cxj   1/1     Running   0          15h

$ kctl get svc -n cattle-system
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
rancher           ClusterIP   10.43.220.187   <none>        80/TCP,443/TCP   16h
rancher-webhook   ClusterIP   10.43.242.115   <none>        443/TCP          15h
webhook-service   ClusterIP   10.43.215.246   <none>        443/TCP          15h

I'm still relatively new to this so even basic debugging suggestions could be helpful. Rancher doesn't appear to report any errors.

bland-account-99790

11/14/2022, 4:05 PM

you were connecting to

10.43.220.187

proud-salesmen-12221

11/14/2022, 4:10 PM

No, not to the clusterip. I was opening a browser to the node that the pod was running on.

bland-account-99790

11/14/2022, 4:12 PM

Could you try using the clusterIP

10.43.220.187

in the browser?

Let's try to disect the problem 😛

👍 1

proud-salesmen-12221

11/14/2022, 4:19 PM

I should describe my environment. I'm using Vagrant on an Ubuntu Host. My vagrant cluster stands up 3 nodes for RKE2. One server and two agents. I used to be able to open a browser on my ubuntu host to the Rancher UI whose pod was on agent1. And then, with ipsec enabled, opening a browser on the host with the same url, I get the Failed to Connect error. So now trying to open a browser on my host to the ClusterIP 10.43.220.187 I'm get 'Unable to Connect'

bland-account-99790

11/15/2022, 3:03 PM

ok, can you show me

kubectl get endpoints -A

10 Views

#rke2 Join Slack