proud-salesmen-12221
09/13/2022, 1:53 AM[vagrant@rke2-server1 ~]$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system cloud-controller-manager-rke2-server1 1/1 Running 2 (174m ago) 3h14m 10.0.2.15 rke2-server1 <none> <none>
kube-system etcd-rke2-server1 1/1 Running 1 (175m ago) 3h14m 192.168.33.101 rke2-server1 <none> <none>
kube-system helm-install-rke2-canal-c48pq 0/1 Completed 0 3h14m 10.0.2.15 rke2-server1 <none> <none>
kube-system helm-install-rke2-coredns-znhbd 0/1 Completed 0 3h14m 10.0.2.15 rke2-server1 <none> <none>
kube-system helm-install-rke2-ingress-nginx-fbp8s 0/1 Completed 0 3h14m 10.42.0.4 rke2-server1 <none> <none>
kube-system helm-install-rke2-metrics-server-4kcj7 0/1 Completed 0 3h14m 10.42.0.2 rke2-server1 <none> <none>
kube-system kube-apiserver-rke2-server1 1/1 Running 1 (175m ago) 3h14m 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-controller-manager-rke2-server1 1/1 Running 2 (174m ago) 3h14m 10.0.2.15 rke2-server1 <none> <none>
kube-system kube-proxy-rke2-agent1 1/1 Running 0 173m 10.0.2.15 rke2-agent1 <none> <none>
kube-system kube-proxy-rke2-agent2 1/1 Running 0 173m 10.0.2.15 rke2-agent2 <none> <none>
kube-system kube-proxy-rke2-server1 1/1 Running 1 (175m ago) 3h14m 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-scheduler-rke2-server1 1/1 Running 1 (175m ago) 3h14m 192.168.33.101 rke2-server1 <none> <none>
kube-system rke2-canal-m8vb6 2/2 Running 2 (175m ago) 3h14m 10.0.2.15 rke2-server1 <none> <none>
kube-system rke2-canal-ptnk8 0/2 CrashLoopBackOff 110 (87s ago) 3h10m 10.0.2.15 rke2-agent2 <none> <none>
kube-system rke2-canal-rmnv6 0/2 CrashLoopBackOff 110 (2m2s ago) 3h13m 10.0.2.15 rke2-agent1 <none> <none>
kube-system rke2-coredns-rke2-coredns-76cb76d66-mkv2c 1/1 Running 1 (175m ago) 3h14m 10.42.0.12 rke2-server1 <none> <none>
kube-system rke2-coredns-rke2-coredns-76cb76d66-t569h 0/1 ContainerCreating 0 3h13m <none> rke2-agent1 <none> <none>
kube-system rke2-coredns-rke2-coredns-autoscaler-58867f8fc5-8n589 1/1 Running 1 (175m ago) 3h14m 10.42.0.11 rke2-server1 <none> <none>
kube-system rke2-ingress-nginx-controller-c8vmd 0/1 ContainerCreating 0 3h9m <none> rke2-agent2 <none> <none>
kube-system rke2-ingress-nginx-controller-n9st4 1/1 Running 1 (175m ago) 3h14m 10.42.0.13 rke2-server1 <none> <none>
kube-system rke2-ingress-nginx-controller-rdml6 0/1 ContainerCreating 0 3h12m <none> rke2-agent1 <none> <none>
kube-system rke2-metrics-server-6979d95f95-4z57b 1/1 Running 1 (175m ago) 3h14m 10.42.0.10 rke2-server1 <none> <none>
[vagrant@rke2-server1 ~]$ kubectl -n kube-system get events
LAST SEEN TYPE REASON OBJECT MESSAGE
52m Normal Pulled pod/rke2-canal-ptnk8 Container image "rancher/hardened-calico:v3.22.2-build20220509" already present on machine
37m Warning BackOff pod/rke2-canal-ptnk8 Back-off restarting failed container
7m23s Warning Unhealthy pod/rke2-canal-ptnk8 Liveness probe failed: calico/node is not ready: Felix is not live: Get "http://localhost:9099/liveness": dial tcp 127.0.0.19099 connect: connection refused
2m12s Warning BackOff pod/rke2-canal-ptnk8 Back-off restarting failed container
7m23s Warning Unhealthy pod/rke2-canal-rmnv6 Readiness probe failed: Get "http://localhost:9099/readiness": dial tcp 127.0.0.19099 connect: connection refused
2m13s Warning BackOff pod/rke2-canal-rmnv6 Back-off restarting failed container
2m16s Warning FailedCreatePodSandBox pod/rke2-coredns-rke2-coredns-76cb76d66-t569h (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "73f957aa6c9bfa7ab3930dc7ca03c0bf3ffeb8e59cda674e73f25207b9c0c5a8": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
2m14s Warning FailedCreatePodSandBox pod/rke2-ingress-nginx-controller-c8vmd (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "4bd0c31f8656f12d8e23bcb3eac70f7f917ea1083f57890d641a0f3f6b8b85bd": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/
2m10s Warning FailedCreatePodSandBox pod/rke2-ingress-nginx-controller-rdml6 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "9013c62529b9776bc1332121fa101ae82b91d83fcc161a43523ca8bef605bbb6": plugin type="calico" failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calicoplain-byte-79620
09/13/2022, 6:58 AMproud-salesmen-12221
09/13/2022, 3:52 PM[vagrant@rke2-server1 ~]$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system cilium-69b5p 1/1 Running 0 25m 10.0.2.15 rke2-server1 <none> <none>
kube-system cilium-mrxhv 0/1 CrashLoopBackOff 8 (80s ago) 24m 10.0.2.15 rke2-agent1 <none> <none>
kube-system cilium-node-init-rg4t5 1/1 Running 0 24m 10.0.2.15 rke2-agent1 <none> <none>
kube-system cilium-node-init-tgf8l 1/1 Running 0 25m 10.0.2.15 rke2-server1 <none> <none>
kube-system cilium-operator-6d77cf7469-pbxch 1/1 Running 0 25m 10.0.2.15 rke2-server1 <none> <none>
kube-system cilium-operator-6d77cf7469-s9cp6 0/1 CrashLoopBackOff 7 (4m29s ago) 25m 10.0.2.15 rke2-agent1 <none> <none>
kube-system cloud-controller-manager-rke2-server1 1/1 Running 0 25m 192.168.33.101 rke2-server1 <none> <none>
kube-system etcd-rke2-server1 1/1 Running 0 25m 192.168.33.101 rke2-server1 <none> <none>
kube-system helm-install-rke2-cilium-cfcps 0/1 Completed 0 25m 10.0.2.15 rke2-server1 <none> <none>
kube-system helm-install-rke2-coredns-w2wb4 0/1 Completed 0 25m 10.0.2.15 rke2-server1 <none> <none>
kube-system helm-install-rke2-ingress-nginx-b9b5f 0/1 Completed 0 25m 10.42.0.196 rke2-server1 <none> <none>
kube-system helm-install-rke2-metrics-server-zndxv 0/1 Completed 0 25m 10.42.0.127 rke2-server1 <none> <none>
kube-system kube-apiserver-rke2-server1 1/1 Running 0 25m 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-controller-manager-rke2-server1 1/1 Running 0 25m 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-proxy-rke2-server1 1/1 Running 0 25m 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-scheduler-rke2-server1 1/1 Running 0 25m 192.168.33.101 rke2-server1 <none> <none>
kube-system rke2-coredns-rke2-coredns-76cb76d66-l2vs9 1/1 Running 0 25m 10.42.0.93 rke2-server1 <none> <none>
kube-system rke2-coredns-rke2-coredns-76cb76d66-xvt89 0/1 ContainerCreating 0 23m <none> rke2-agent1 <none> <none>
kube-system rke2-coredns-rke2-coredns-autoscaler-58867f8fc5-jpgb6 1/1 Running 0 25m 10.42.0.27 rke2-server1 <none> <none>
kube-system rke2-ingress-nginx-controller-clh78 1/1 Running 0 22m 10.42.0.249 rke2-server1 <none> <none>
kube-system rke2-ingress-nginx-controller-p2lkd 0/1 ContainerCreating 0 20m <none> rke2-agent1 <none> <none>
kube-system rke2-metrics-server-6979d95f95-88pw9 1/1 Running 0 23m 10.42.0.47 rke2-server1 <none> <none>
plain-byte-79620
09/13/2022, 3:54 PMkubectl -n kube-system logs cilium-mrxhv
proud-salesmen-12221
09/13/2022, 3:56 PM[vagrant@rke2-server1 ~]$ kubectl -n kube-system logs cilium-mrxhv
Defaulted container "cilium-agent" out of: cilium-agent, mount-cgroup (init), apply-sysctl-overwrites (init), mount-bpf-fs (init), wait-for-node-init (init), clean-cilium-state (init)
Error from server (NotFound): the server could not find the requested resource ( pods/log cilium-mrxhv)
[vagrant@rke2-server1 ~]$
plain-byte-79620
09/13/2022, 3:59 PMkubectl -n kube-system describe pod cilium-mrxhv
proud-salesmen-12221
09/13/2022, 4:05 PM[vagrant@rke2-server1 ~]$ kubectl -n kube-system describe pod cilium-mrxhv
Name: cilium-mrxhv
Namespace: kube-system
Priority: 2000001000
Priority Class Name: system-node-critical
Node: rke2-agent1/10.0.2.15
Start Time: Tue, 13 Sep 2022 15:27:27 +0000
Labels: controller-revision-hash=78cf4df44d
k8s-app=cilium
pod-template-generation=1
Annotations: <http://container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites|container.apparmor.security.beta.kubernetes.io/apply-sysctl-overwrites>: unconfined
<http://container.apparmor.security.beta.kubernetes.io/cilium-agent|container.apparmor.security.beta.kubernetes.io/cilium-agent>: unconfined
<http://container.apparmor.security.beta.kubernetes.io/clean-cilium-state|container.apparmor.security.beta.kubernetes.io/clean-cilium-state>: unconfined
<http://container.apparmor.security.beta.kubernetes.io/mount-cgroup|container.apparmor.security.beta.kubernetes.io/mount-cgroup>: unconfined
<http://kubernetes.io/psp|kubernetes.io/psp>: global-unrestricted-psp
<http://prometheus.io/port|prometheus.io/port>: 9962
<http://prometheus.io/scrape|prometheus.io/scrape>: true
Status: Running
IP: 10.0.2.15
IPs:
IP: 10.0.2.15
Controlled By: DaemonSet/cilium
Init Containers:
mount-cgroup:
Container ID: <containerd://312b54397879591f51f5c3ee2d895a9e30290c055bc1c5e8ded1a67df68dbba>6
Image: rancher/mirrored-cilium-cilium:v1.12.0
Image ID: <http://docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade|docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>
Port: <none>
Host Port: <none>
Command:
sh
-ec
cp /usr/bin/cilium-mount /hostbin/cilium-mount;
nsenter --cgroup=/hostproc/1/ns/cgroup --mount=/hostproc/1/ns/mnt "${BIN_PATH}/cilium-mount" $CGROUP_ROOT;
rm /hostbin/cilium-mount
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 13 Sep 2022 15:31:36 +0000
Finished: Tue, 13 Sep 2022 15:31:36 +0000
Ready: True
Restart Count: 0
Environment:
CGROUP_ROOT: /run/cilium/cgroupv2
BIN_PATH: /opt/cni/bin
Mounts:
/hostbin from cni-path (rw)
/hostproc from hostproc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tcxz8 (ro)
apply-sysctl-overwrites:
Container ID: <containerd://d5bb1b9af9de598c9497091365775420920539f2dde0682b2b8d5be9b9f4ecb>d
Image: rancher/mirrored-cilium-cilium:v1.12.0
Image ID: <http://docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade|docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>
Port: <none>
Host Port: <none>
Command:
sh
-ec
cp /usr/bin/cilium-sysctlfix /hostbin/cilium-sysctlfix;
nsenter --mount=/hostproc/1/ns/mnt "${BIN_PATH}/cilium-sysctlfix";
rm /hostbin/cilium-sysctlfix
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 13 Sep 2022 15:31:38 +0000
Finished: Tue, 13 Sep 2022 15:31:38 +0000
Ready: True
Restart Count: 0
Environment:
BIN_PATH: /opt/cni/bin
Mounts:
/hostbin from cni-path (rw)
/hostproc from hostproc (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tcxz8 (ro)
mount-bpf-fs:
Container ID: <containerd://e7d4bbdce1c57626819d1927e4779c4d1548a9d380df1b20a88f618e298abf3>3
Image: rancher/mirrored-cilium-cilium:v1.12.0
Image ID: <http://docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade|docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>
Port: <none>
Host Port: <none>
Command:
/bin/bash
-c
--
Args:
mount | grep "/sys/fs/bpf type bpf" || mount -t bpf bpf /sys/fs/bpf
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 13 Sep 2022 15:31:39 +0000
Finished: Tue, 13 Sep 2022 15:31:39 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/sys/fs/bpf from bpf-maps (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tcxz8 (ro)
wait-for-node-init:
Container ID: <containerd://96ca17d9bd8ebab56ac6d579530b8cd9210da4e88e8157cd26b46049d23f96f>6
Image: rancher/mirrored-cilium-cilium:v1.12.0
Image ID: <http://docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade|docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>
Port: <none>
Host Port: <none>
Command:
sh
-c
until test -s "/tmp/cilium-bootstrap.d/cilium-bootstrap-time"; do
echo "Waiting on node-init to run...";
sleep 1;
done
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 13 Sep 2022 15:31:40 +0000
Finished: Tue, 13 Sep 2022 15:31:40 +0000
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/tmp/cilium-bootstrap.d from cilium-bootstrap-file-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tcxz8 (ro)
clean-cilium-state:
Container ID: <containerd://23517402f6814bdf97ef245ab095f89a31d0b203cbe31ffd8c24a521d8680c5>a
Image: rancher/mirrored-cilium-cilium:v1.12.0
Image ID: <http://docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade|docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>
Port: <none>
Host Port: <none>
Command:
/init-container.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Tue, 13 Sep 2022 15:31:41 +0000
Finished: Tue, 13 Sep 2022 15:31:41 +0000
Ready: True
Restart Count: 0
Requests:
cpu: 100m
memory: 100Mi
Environment:
CILIUM_ALL_STATE: <set to the key 'clean-cilium-state' of config map 'cilium-config'> Optional: true
CILIUM_BPF_STATE: <set to the key 'clean-cilium-bpf-state' of config map 'cilium-config'> Optional: true
Mounts:
/run/cilium/cgroupv2 from cilium-cgroup (rw)
/sys/fs/bpf from bpf-maps (rw)
/var/run/cilium from cilium-run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tcxz8 (ro)
Containers:
cilium-agent:
Container ID: <containerd://c03f1fcc1c6375c3b3f072de54a9e550f6d25f439444221e5f81a45468f89a4>f
Image: rancher/mirrored-cilium-cilium:v1.12.0
Image ID: <http://docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade|docker.io/rancher/mirrored-cilium-cilium@sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>
Ports: 4244/TCP, 9962/TCP, 9964/TCP
Host Ports: 4244/TCP, 9962/TCP, 9964/TCP
Command:
cilium-agent
Args:
--config-dir=/tmp/cilium/config-map
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Tue, 13 Sep 2022 16:02:01 +0000
Finished: Tue, 13 Sep 2022 16:03:09 +0000
Ready: False
Restart Count: 10
Liveness: http-get <http://127.0.0.1:9879/healthz> delay=0s timeout=5s period=30s #success=1 #failure=10
Readiness: http-get <http://127.0.0.1:9879/healthz> delay=0s timeout=5s period=30s #success=1 #failure=3
Startup: http-get <http://127.0.0.1:9879/healthz> delay=0s timeout=1s period=2s #success=1 #failure=105
Environment:
K8S_NODE_NAME: (v1:spec.nodeName)
CILIUM_K8S_NAMESPACE: kube-system (v1:metadata.namespace)
CILIUM_CLUSTERMESH_CONFIG: /var/lib/cilium/clustermesh/
CILIUM_CNI_CHAINING_MODE: <set to the key 'cni-chaining-mode' of config map 'cilium-config'> Optional: true
CILIUM_CUSTOM_CNI_CONF: <set to the key 'custom-cni-conf' of config map 'cilium-config'> Optional: true
Mounts:
/host/etc/cni/net.d from etc-cni-netd (rw)
/host/opt/cni/bin from cni-path (rw)
/host/proc/sys/kernel from host-proc-sys-kernel (rw)
/host/proc/sys/net from host-proc-sys-net (rw)
/lib/modules from lib-modules (ro)
/run/xtables.lock from xtables-lock (rw)
/sys/fs/bpf from bpf-maps (rw)
/tmp/cilium/config-map from cilium-config-path (ro)
/var/lib/cilium/clustermesh from clustermesh-secrets (ro)
/var/run/cilium from cilium-run (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tcxz8 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
cilium-run:
Type: HostPath (bare host directory volume)
Path: /var/run/cilium
HostPathType: DirectoryOrCreate
bpf-maps:
Type: HostPath (bare host directory volume)
Path: /sys/fs/bpf
HostPathType: DirectoryOrCreate
hostproc:
Type: HostPath (bare host directory volume)
Path: /proc
HostPathType: Directory
cilium-cgroup:
Type: HostPath (bare host directory volume)
Path: /run/cilium/cgroupv2
HostPathType: DirectoryOrCreate
cni-path:
Type: HostPath (bare host directory volume)
Path: /opt/cni/bin
HostPathType: DirectoryOrCreate
etc-cni-netd:
Type: HostPath (bare host directory volume)
Path: /etc/cni/net.d
HostPathType: DirectoryOrCreate
lib-modules:
Type: HostPath (bare host directory volume)
Path: /lib/modules
HostPathType:
xtables-lock:
Type: HostPath (bare host directory volume)
Path: /run/xtables.lock
HostPathType: FileOrCreate
cilium-bootstrap-file-dir:
Type: HostPath (bare host directory volume)
Path: /tmp/cilium-bootstrap.d
HostPathType: DirectoryOrCreate
clustermesh-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: cilium-clustermesh
Optional: true
cilium-config-path:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: cilium-config
Optional: false
host-proc-sys-net:
Type: HostPath (bare host directory volume)
Path: /proc/sys/net
HostPathType: Directory
host-proc-sys-kernel:
Type: HostPath (bare host directory volume)
Path: /proc/sys/kernel
HostPathType: Directory
kube-api-access-tcxz8:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <http://kubernetes.io/os=linux|kubernetes.io/os=linux>
Tolerations: op=Exists
<http://node.kubernetes.io/disk-pressure:NoSchedule|node.kubernetes.io/disk-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/memory-pressure:NoSchedule|node.kubernetes.io/memory-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/network-unavailable:NoSchedule|node.kubernetes.io/network-unavailable:NoSchedule> op=Exists
<http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists
<http://node.kubernetes.io/pid-pressure:NoSchedule|node.kubernetes.io/pid-pressure:NoSchedule> op=Exists
<http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists
<http://node.kubernetes.io/unschedulable:NoSchedule|node.kubernetes.io/unschedulable:NoSchedule> op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 36m default-scheduler Successfully assigned kube-system/cilium-mrxhv to rke2-agent1
Warning Failed 35m kubelet Failed to pull image "rancher/mirrored-cilium-cilium:v1.12.0": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/mirrored-cilium-cilium:v1.12.0|docker.io/rancher/mirrored-cilium-cilium:v1.12.0>": failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://registry-1.docker.io/v2/rancher/mirrored-cilium-cilium/blobs/sha256:534f4b5bc2dac3237b7fcc4f7b32bb5e87ab1caa0e9ebac82a56b1b1ef255a41>": dial tcp 127.0.0.1:443: connect: connection refused
Warning Failed 34m kubelet Failed to pull image "rancher/mirrored-cilium-cilium:v1.12.0": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/mirrored-cilium-cilium:v1.12.0|docker.io/rancher/mirrored-cilium-cilium:v1.12.0>": failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://registry-1.docker.io/v2/rancher/mirrored-cilium-cilium/blobs/sha256:75d8c87dfd66cb5f2f6c5958aefda6765cd97b953eeb237de55065ac343c33b9>": dial tcp: i/o timeout
Warning Failed 33m (x3 over 35m) kubelet Error: ErrImagePull
Warning Failed 33m kubelet Failed to pull image "rancher/mirrored-cilium-cilium:v1.12.0": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/mirrored-cilium-cilium:v1.12.0|docker.io/rancher/mirrored-cilium-cilium:v1.12.0>": failed to copy: httpReadSeeker: failed open: failed to do request: Get "<https://registry-1.docker.io/v2/rancher/mirrored-cilium-cilium/manifests/sha256:079baa4fa1b9fe638f96084f4e0297c84dd4fb215d29d2321dcbe54273f63ade>": dial tcp: lookup <http://registry-1.docker.io|registry-1.docker.io> on 10.0.2.3:53: read udp 10.0.2.15:45774->10.0.2.3:53: i/o timeout
Warning Failed 33m (x5 over 35m) kubelet Error: ImagePullBackOff
Normal BackOff 33m (x5 over 35m) kubelet Back-off pulling image "rancher/mirrored-cilium-cilium:v1.12.0"
Normal Pulling 33m (x4 over 36m) kubelet Pulling image "rancher/mirrored-cilium-cilium:v1.12.0"
Normal Pulled 32m kubelet Successfully pulled image "rancher/mirrored-cilium-cilium:v1.12.0" in 47.723868398s
Normal Created 32m kubelet Created container mount-cgroup
Normal Started 32m kubelet Started container mount-cgroup
Normal Pulled 32m kubelet Container image "rancher/mirrored-cilium-cilium:v1.12.0" already present on machine
Normal Created 32m kubelet Created container apply-sysctl-overwrites
Warning BackOff 6m12s (x90 over 29m) kubelet Back-off restarting failed container
Warning Unhealthy 80s (x292 over 32m) kubelet Startup probe failed: Get "<http://127.0.0.1:9879/healthz>": dial tcp 127.0.0.1:9879: connect: connection refused
[vagrant@rke2-server1 ~]$
plain-byte-79620
09/13/2022, 4:08 PMproud-salesmen-12221
09/13/2022, 4:14 PM[vagrant@rke2-server1 ~]$ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system cilium-dcwvt 0/1 CrashLoopBackOff 4 (78s ago) 8m22s 10.0.2.15 rke2-agent1 <none> <none>
kube-system cilium-mpv6m 1/1 Running 0 9m38s 10.0.2.15 rke2-server1 <none> <none>
kube-system cilium-node-init-24n8v 1/1 Running 0 9m38s 10.0.2.15 rke2-server1 <none> <none>
kube-system cilium-node-init-bh9zs 1/1 Running 0 8m22s 10.0.2.15 rke2-agent1 <none> <none>
kube-system cilium-operator-6d77cf7469-rjvdb 1/1 Running 0 9m38s 10.0.2.15 rke2-server1 <none> <none>
kube-system cilium-operator-6d77cf7469-wlnmb 0/1 CrashLoopBackOff 4 (60s ago) 9m38s 10.0.2.15 rke2-agent1 <none> <none>
kube-system cloud-controller-manager-rke2-server1 1/1 Running 0 9m1s 192.168.33.101 rke2-server1 <none> <none>
kube-system etcd-rke2-server1 1/1 Running 0 9m31s 192.168.33.101 rke2-server1 <none> <none>
kube-system helm-install-rke2-cilium-4df5f 0/1 Completed 0 9m44s 10.0.2.15 rke2-server1 <none> <none>
kube-system helm-install-rke2-coredns-6f8ld 0/1 Completed 0 9m44s 10.0.2.15 rke2-server1 <none> <none>
kube-system helm-install-rke2-ingress-nginx-hw99x 0/1 Completed 0 9m44s 10.42.0.219 rke2-server1 <none> <none>
kube-system helm-install-rke2-metrics-server-nhml9 0/1 Completed 0 9m44s 10.42.0.177 rke2-server1 <none> <none>
kube-system kube-apiserver-rke2-server1 1/1 Running 0 9m12s 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-controller-manager-rke2-server1 1/1 Running 0 9m7s 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-proxy-rke2-server1 1/1 Running 0 8m59s 192.168.33.101 rke2-server1 <none> <none>
kube-system kube-scheduler-rke2-server1 1/1 Running 0 9m7s 192.168.33.101 rke2-server1 <none> <none>
kube-system rke2-coredns-rke2-coredns-76cb76d66-nnvgw 0/1 ContainerCreating 0 8m17s <none> rke2-agent1 <none> <none>
kube-system rke2-coredns-rke2-coredns-76cb76d66-qvmpz 1/1 Running 0 9m38s 10.42.0.206 rke2-server1 <none> <none>
kube-system rke2-coredns-rke2-coredns-autoscaler-58867f8fc5-7sd2r 1/1 Running 0 9m38s 10.42.0.50 rke2-server1 <none> <none>
kube-system rke2-ingress-nginx-controller-92796 1/1 Running 0 8m20s 10.42.0.1 rke2-server1 <none> <none>
kube-system rke2-ingress-nginx-controller-9hxrg 0/1 ContainerCreating 0 7m11s <none> rke2-agent1 <none> <none>
kube-system rke2-metrics-server-6979d95f95-hk56w 1/1 Running 0 8m26s 10.42.0.149 rke2-server1 <none> <none>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 8m35s default-scheduler Successfully assigned kube-system/cilium-dcwvt to rke2-agent1
Normal Pulling 8m33s kubelet Pulling image "rancher/mirrored-cilium-cilium:v1.12.0"
Normal Pulled 7m39s kubelet Successfully pulled image "rancher/mirrored-cilium-cilium:v1.12.0" in 54.219246539s
Normal Created 7m39s kubelet Created container mount-cgroup
Normal Started 7m39s kubelet Started container mount-cgroup
Normal Pulled 7m37s kubelet Container image "rancher/mirrored-cilium-cilium:v1.12.0" already present on machine
Normal Created 7m37s kubelet Created container apply-sysctl-overwrites
Normal Started 7m37s kubelet Started container apply-sysctl-overwrites
Normal Pulled 7m36s kubelet Container image "rancher/mirrored-cilium-cilium:v1.12.0" already present on machine
Normal Created 7m36s kubelet Created container mount-bpf-fs
Normal Started 7m36s kubelet Started container mount-bpf-fs
Normal Pulled 7m35s kubelet Container image "rancher/mirrored-cilium-cilium:v1.12.0" already present on machine
Normal Created 7m35s kubelet Created container wait-for-node-init
Normal Started 7m35s kubelet Started container wait-for-node-init
Normal Pulled 7m34s kubelet Container image "rancher/mirrored-cilium-cilium:v1.12.0" already present on machine
Normal Created 7m34s kubelet Created container clean-cilium-state
Normal Started 7m34s kubelet Started container clean-cilium-state
Normal Pulled 7m33s kubelet Container image "rancher/mirrored-cilium-cilium:v1.12.0" already present on machine
Normal Created 7m33s kubelet Created container cilium-agent
Normal Started 7m33s kubelet Started container cilium-agent
Warning Unhealthy 3m31s (x100 over 7m31s) kubelet Startup probe failed: Get "<http://127.0.0.1:9879/healthz>": dial tcp 127.0.0.1:9879: connect: connection refused
[vagrant@rke2-agent1 ~]$ sudo systemctl status rke2-agent
● rke2-agent.service - Rancher Kubernetes Engine v2 (agent)
Loaded: loaded (/usr/local/lib/systemd/system/rke2-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2022-09-13 16:23:59 UTC; 10min ago
Docs: <https://github.com/rancher/rke2#readme>
Process: 8050 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 8045 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 8042 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Main PID: 8055 (rke2)
Tasks: 82
Memory: 2.1G
CGroup: /system.slice/rke2-agent.service
├─ 8055 /usr/local/bin/rke2 agent
├─ 8067 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd
├─ 8082 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/r>
├─ 8186 /var/lib/rancher/rke2/data/v1.24.4-rke2r1-85bc7d7bec85/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 6083c2c7a2ca38ec38549ae0113e1814888e67afbcca1e1f0f9a01bbeeefe2b7 -address /run/k3s/containerd/containerd.sock
├─ 8196 /var/lib/rancher/rke2/data/v1.24.4-rke2r1-85bc7d7bec85/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id a95ec6ce3ed2290395a52213432d21110b565725769ce7b1f4a29994f666155f -address /run/k3s/containerd/containerd.sock
├─ 8261 /var/lib/rancher/rke2/data/v1.24.4-rke2r1-85bc7d7bec85/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 6ce42cf5411ba2ed3cb8c94bab1b15095e74a32f51ea92a6355f8f4080be3b82 -address /run/k3s/containerd/containerd.sock
├─10296 /opt/cni/bin/cilium-cni
└─10310 /opt/cni/bin/cilium-cni
Sep 13 16:34:19 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:19Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:9345: connect: connection refused"
Sep 13 16:34:19 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:19Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:9345: connect: connection refused"
Sep 13 16:34:23 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:23Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:44974->127.0.0.1:6444: read: connection reset by peer"
Sep 13 16:34:24 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:24Z" level=info msg="Connecting to proxy" url="<wss://10.0.2.15:9345/v1-rke2/connect>"
Sep 13 16:34:24 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:24Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:9345: connect: connection refused"
Sep 13 16:34:24 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:24Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:9345: connect: connection refused"
Sep 13 16:34:28 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:28Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: CA cert validation failed: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:45004->127.0.0.1:6444: read: connection reset by peer"
Sep 13 16:34:29 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:29Z" level=info msg="Connecting to proxy" url="<wss://10.0.2.15:9345/v1-rke2/connect>"
Sep 13 16:34:29 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:29Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.2.15:9345: connect: connection refused"
Sep 13 16:34:29 rke2-agent1 rke2[8055]: time="2022-09-13T16:34:29Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.2.15:9345: connect: connection refused"
[vagrant@rke2-agent1 ~]$ date
Tue Sep 13 16:34:37 UTC 2022
[vagrant@rke2-agent1 ~]$
plain-byte-79620
09/14/2022, 8:22 AMproud-salesmen-12221
09/14/2022, 10:02 PM[vagrant@rke2-server1 ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:fc:e9:96 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
valid_lft 86280sec preferred_lft 86280sec
inet6 fe80::a00:27ff:fefc:e996/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 08:00:27:56:23:b3 brd ff:ff:ff:ff:ff:ff
inet 10.11.0.101/24 brd 10.11.0.255 scope global noprefixroute eth1
valid_lft forever preferred_lft forever
inet6 fe80::a00:27ff:fe56:23b3/64 scope link
valid_lft forever preferred_lft forever
### VagrantFile
# rke2 SERVER NODES
SERVER_N = 3
(1..SERVER_N).each do |i|
config.vm.define "server#{i}", primary: true do |server|
server.vm.hostname = "rke2-server#{i}#{DOMAIN}"
server.vm.box = VBOX
server.vm.network "private_network", ip: "10.11.0.#{100+i}"
server.vm.provider "virtualbox" do |vb|
vb.memory = "8192"
vb.cpus = "4"
end
plain-byte-79620
09/15/2022, 8:01 AMnode-ip
config? On the pods 192.168.33.101
is still configured when you removed it do you uninstall RKE2?proud-salesmen-12221
09/15/2022, 6:36 PM