enough-toddler-31145
10/21/2022, 3:28 PMenough-toddler-31145
10/21/2022, 3:29 PMWarning FailedCreatePodSandBox 16m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "488aaaff6ba2bdab7b5c1ce46065b25734329b60b5c2ac40bbf1207e6579a2b7": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
Warning FailedCreatePodSandBox 15m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "66cbf3810675b54dbcaf37e1e00879cca698c8b64a970debaf275d02ad202572": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
Warning FailedCreatePodSandBox 15m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c13ee0a7667be018d6d89a6e7341957443f9f8f22a30c7443d897318fe10af6a": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
Warning FailedCreatePodSandBox 2m25s (x60 over 15m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "30cf6e2b3ddf13280927d19b746c09d5c95cc13a0b0c4129638ba169a8eb1b62": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
enough-toddler-31145
10/21/2022, 3:33 PMkubectl get pods -n kube-system
i get
rke2-canal-2f5fg 1/2 CrashLoopBackOff 709 (3m13s ago) 43h
rke2-canal-2gmx5 1/2 Running 1684 (49s ago) 43h
rke2-canal-5lmmd 1/2 CrashLoopBackOff 727 (38s ago) 43h
rke2-canal-9lb7j 1/2 Running 729 (56s ago) 43h
rke2-canal-qkgts 1/2 CrashLoopBackOff 1650 (44s ago) 43h
rke2-canal-r9hvg 1/2 Running 1522 (2m17s ago) 43h
rke2-canal-xnqnx 1/2 CrashLoopBackOff 711 (3m38s ago) 43h
rke2-canal-zhc42 1/2 CrashLoopBackOff 726 (117s ago) 43h
and this is the output for one of the pods
Normal Created 13m (x2 over 14m) kubelet Created container calico-node
Normal Started 13m (x2 over 14m) kubelet Started container calico-node
Normal Pulled 13m (x2 over 14m) kubelet Container image "rancher/hardened-calico:v3.23.3-build20220810" already present on machine
Normal Killing 13m kubelet Container calico-node failed liveness probe, will be restarted
Warning Unhealthy 13m (x11 over 14m) kubelet Readiness probe failed: Get "<http://localhost:9099/readiness>": dial tcp: lookup localhost on 8.8.8.8:53: no such host
Warning BackOff 9m29s (x6 over 10m) kubelet Back-off restarting failed container
Warning FailedMount 5m20s kubelet MountVolume.SetUp failed for volume "flannel-cfg" : failed to sync configmap cache: timed out waiting for the condition
Normal Pulled 4m19s (x2 over 5m19s) kubelet Container image "rancher/hardened-calico:v3.23.3-build20220810" already present on machine
Normal Created 4m19s (x2 over 5m19s) kubelet Created container calico-node
Normal Started 4m19s (x2 over 5m19s) kubelet Started container calico-node
Warning Unhealthy 4m19s (x6 over 5m9s) kubelet Liveness probe failed: calico/node is not ready: Felix is not live: Get "<http://localhost:9099/liveness>": dial tcp: lookup localhost on 8.8.8.8:53: no such host
Normal Killing 4m19s kubelet Container calico-node failed liveness probe, will be restarted
Warning Unhealthy 4m17s (x11 over 5m11s) kubelet Readiness probe failed: Get "<http://localhost:9099/readiness>": dial tcp: lookup localhost on 8.8.8.8:53: no such host
Warning BackOff 15s (x6 over 69s) kubelet Back-off restarting failed container
I'm assuming the certificates expired for calico no?
Any help would be greatproud-ram-62490
10/24/2022, 3:16 PMvictorious-analyst-3332
10/25/2022, 3:41 PM<http://rke2.io/node-args|rke2.io/node-args>
annotation, but editing that directly doesn’t appear to have any impact on the /etc/rancher/rke2/config.yaml.d/50-rancher.yaml
file on the node itself. Thanks a lot.stocky-article-82001
10/25/2022, 8:44 PMkubelet:
extra_args:
max-pods: 250
limited-motherboard-41807
10/26/2022, 1:47 PMbored-rain-98291
10/26/2022, 6:48 PMrough-ocean-41843
10/27/2022, 2:16 PMrough-ocean-41843
10/27/2022, 2:18 PMprehistoric-solstice-99854
10/27/2022, 7:41 PMcool-pillow-1781
10/28/2022, 1:43 PMhundreds-hairdresser-46043
10/31/2022, 12:53 PMcool-pillow-1781
10/31/2022, 2:25 PMhundreds-hairdresser-46043
10/31/2022, 2:30 PMcool-pillow-1781
10/31/2022, 2:34 PMnarrow-noon-75604
10/31/2022, 4:58 PMbored-rain-98291
10/31/2022, 7:05 PMbored-rain-98291
10/31/2022, 7:10 PMhundreds-hairdresser-46043
11/01/2022, 9:32 AMmirrors:
<http://europe-west4-docker.pkg.dev|europe-west4-docker.pkg.dev>:
endpoint:
- "<https://europe-west4-docker.pkg.dev>"
<http://docker.io|docker.io>:
endpoint:
- "<https://europe-west4-docker.pkg.dev>"
rewrite:
"^rancher/(.*)": "/rke2/rke2-core-cluster/$1"
<http://index.docker.io|index.docker.io>:
endpoint:
- "<https://europe-west4-docker.pkg.dev>"
rewrite:
"^rancher/(.*)": "/rke2/rke2-core-cluster/$1"
configs:
<http://europe-west4-docker.pkg.dev|europe-west4-docker.pkg.dev>:
auth:
username: _json_key_base64
password: |
ICasasaAgIblah...
red-waitress-37932
11/01/2022, 11:04 AMcool-pillow-1781
11/03/2022, 4:42 PMcareful-table-97595
11/04/2022, 12:07 PMcool-pillow-1781
11/07/2022, 2:52 PM/var/lib/rancher > /var/log > /
With / still being 500G~1T
Expecting the vast majority of data to be in /var/lib/rancher
creamy-pencil-82913
11/08/2022, 5:45 PMhundreds-hairdresser-46043
11/11/2022, 1:07 PMproud-salesmen-12221
11/14/2022, 12:45 AMrefined-scientist-20236
11/15/2022, 1:29 PM"advertise-address": "10.10.255.103"
"cni": "none"
"disable":
- "rke2-ingress-nginx"
"disable-cloud-controller": true
"disable-kube-proxy": false
"disable-network-policy": false
"flannel-iface": "eth1"
"kube-controller-manager-arg": "flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins"
"kubelet-arg":
- "cloud-provider=external"
- "volume-plugin-dir=/var/lib/kubelet/volumeplugins"
"node-ip": "10.10.255.103"
"node-name": "k8s-test-control-plane-3"
"node-taint":
- "<http://node-role.kubernetes.io/control-plane:NoSchedule|node-role.kubernetes.io/control-plane:NoSchedule>"
"server": "<https://10.10.255.101:9345>"
"tls-san":
- "195.201.20.2"
"token": "xxxx"
"write-kubeconfig-mode": "0644"
Info: Currently no LB is between Nodes - Server is the first node and CNI: node is because I setup calico typha operator after cluster init (on setup time the first node has the default cni)proud-ram-62490
11/16/2022, 4:39 PMambitious-plastic-3551
11/16/2022, 5:44 PM