Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
m
magnificent-vr-88571
05/15/2023, 10:54 PMhi,
I have 3 master node + 6 worker node as a cluster.
I need to release 3 master node and convert existing 3 nodes in worker to master.
To do it I have killed 2agent node and drained 2 master.
Then started rke2-server services in the old agent nodes.
Facing following error
Normal SandboxChanged 6h18m (x26 over 6h23m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedMount 6h15m (x7 over 6h16m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : object "kube-system"/"rke2-ingress-nginx-admission" not registered
Warning NetworkNotReady 6h15m (x18 over 6h16m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Warning FailedMount 6h2m (x7 over 6h2m) kubelet MountVolume.SetUp failed for volume "webhook-cert" : object "kube-system"/"rke2-ingress-nginx-admission" not registered
Warning NetworkNotReady 5h52m (x302 over 6h2m) kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Warning FailedMount 5h46m kubelet MountVolume.SetUp failed for volume "webhook-cert" : failed to sync secret cache: timed out waiting for the condition
Normal SandboxChanged 5h39m (x25 over 5h46m) kubelet Pod sandbox changed, it will be killed and re-created.
Warning FailedMount 5h34m kubelet MountVolume.SetUp failed for volume "webhook-cert" : failed to sync secret cache: timed out waiting for the condition
Normal SandboxChanged 5h32m (x12 over 5h34m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal SandboxChanged 38m (x10 over 40m) kubelet Pod sandbox changed, it will be killed and re-created.
Normal SandboxChanged 4m (x28 over 9m41s) kubelet Pod sandbox changed, it will be killed and re-created.I see above error in many pods
any recommended steps to recover
I am using cilium as CNi, but cilium itself not coming up 😞
Mistake I made in the environment. Missed CNI update in the agent node config file to be converted to server node. This lead to default cni helm installed and entire environment got trouble.
Recovered entire environment by following steps.
1) restored last required etcd in the server.
2) added all server node after cleaning up etcd eb and snapshot
3) added all agents
It’s back to normal and entire environment is stable