Hi I m experiencing a rke2 server which is stuck during star Rancher Users #rke2

Hi, I'm experiencing a rke2-server which is stuck ...

kind-air-74358

06/08/2026, 7:38 AM

Hi, I'm experiencing a rke2-server which is stuck during starting.. Any idea what could prevent is starting rke2-server?

Copy code

Jun 08 07:29:29 control-plane-01 systemd[1]: Failed to start rke2-server.service - Rancher Kubernetes Engine v2 (server).
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 3951.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Found left-over process 2433 (containerd-shim) in control group while starting unit. Ignoring.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Found left-over process 2558 (containerd-shim) in control group while starting unit. Ignoring.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Found left-over process 2651 (containerd-shim) in control group while starting unit. Ignoring.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
... Skipping
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Found left-over process 508541 (containerd-shim) in control group while starting unit. Ignoring.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 08 07:29:34 control-plane-01 systemd[1]: Starting rke2-server.service - Rancher Kubernetes Engine v2 (server)...
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Found left-over process 2433 (containerd-shim) in control group while starting unit. Ignoring.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
... Skipping
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: Found left-over process 508541 (containerd-shim) in control group while starting unit. Ignoring.
Jun 08 07:29:34 control-plane-01 systemd[1]: rke2-server.service: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 08 07:29:34 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:34Z" level=warning msg="not running in CIS mode"
Jun 08 07:29:34 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:34Z" level=info msg="Applying Pod Security Admission Configuration"
Jun 08 07:29:34 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:34Z" level=info msg="Starting rke2 v1.33.10+rke2r3 (2dcec00118382207a15bd0d219b8e45cd8b02b2f)"
Jun 08 07:29:34 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:34Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
Jun 08 07:29:34 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:34Z" level=info msg="Reconciling bootstrap data between datastore and disk"
Jun 08 07:29:34 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:34Z" level=info msg="Opening etcd client connection with endpoints [<https://127.0.0.1:2379>]"
Jun 08 07:29:44 control-plane-01 rke2[3563050]: {"level":"warn","ts":"2026-06-08T07:29:44.728981Z","logger":"k3s.remotestore","caller":"v3@v3.5.26-k3s1/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-e>
Jun 08 07:29:44 control-plane-01 rke2[3563050]: time="2026-06-08T07:29:44Z" level=fatal msg="Error: preparing server: failed to bootstrap cluster data: failed to reconcile with local datastore: context deadline exceeded"
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 2433 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 2558 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 2651 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 2661 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 2938 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 2947 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 3011 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 3049 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 3103 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 3144 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 8667 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 8761 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 8832 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 8866 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: rke2-server.service: Unit process 508541 (containerd-shim) remains running after unit stopped.
Jun 08 07:29:44 control-plane-01 systemd[1]: Failed to start rke2-server.service - Rancher Kubernetes Engine v2 (server).

kind-air-74358

06/08/2026, 7:46 AM

Some more information; there is no containerd process running, looking into the containerd logs since this issue started these are the last containerd logs;

Copy code

ime="2026-06-07T14:44:37.871861917Z" level=info msg="StartContainer for \"697069933adcb691d06b370583851f823c273fb11de48602b746081aee183446\""
time="2026-06-07T14:44:37.871941439Z" level=info msg="StartContainer for \"2962222bde4e7d84ef30d3cdd77dedfca7989d76fcacd56d2c5177c82b2c594d\""
time="2026-06-07T14:44:37.873652837Z" level=info msg="StartContainer for \"8e4783fee0ceb131b0d02d17621c7f28ec5dfb8870923daa338e2ca8e0742b06\""
time="2026-06-07T14:44:37.876570036Z" level=info msg="StartContainer for \"c5bae2696403d7c36c3beca13aa2a9d125522c403ac532ab3619b04b16d4c82f\""
time="2026-06-07T14:44:37.908734354Z" level=info msg="connecting to shim c5bae2696403d7c36c3beca13aa2a9d125522c403ac532ab3619b04b16d4c82f" address="unix:///run/containerd/s/aa80ecd235600de04a0aa3ee703e15b267e751d55b0e7bd24eee79726baf4918" protocol=ttrpc version=3
time="2026-06-07T14:44:37.908818206Z" level=info msg="connecting to shim 697069933adcb691d06b370583851f823c273fb11de48602b746081aee183446" address="unix:///run/containerd/s/0ce24dc019d6777099b50b528c88c7f57fe7f81f78237ad4f66ab8e6f43b6b38" protocol=ttrpc version=3
time="2026-06-07T14:44:37.908875037Z" level=info msg="connecting to shim 8e4783fee0ceb131b0d02d17621c7f28ec5dfb8870923daa338e2ca8e0742b06" address="unix:///run/containerd/s/6f2e6fbd6e9a4b38bc5e3005a39a230aff43e574763f45f93d7fd8e6fc8d3829" protocol=ttrpc version=3
time="2026-06-07T14:44:37.913401583Z" level=info msg="connecting to shim 2962222bde4e7d84ef30d3cdd77dedfca7989d76fcacd56d2c5177c82b2c594d" address="unix:///run/containerd/s/e94691a68e73c93f982714425982d14fb52aea6a3fe743abd87526bed41c3a5a" protocol=ttrpc version=3
time="2026-06-07T14:44:38.315821950Z" level=info msg="StartContainer for \"c5bae2696403d7c36c3beca13aa2a9d125522c403ac532ab3619b04b16d4c82f\" returns successfully"
time="2026-06-07T14:44:38.315890662Z" level=info msg="StartContainer for \"2962222bde4e7d84ef30d3cdd77dedfca7989d76fcacd56d2c5177c82b2c594d\" returns successfully"
time="2026-06-07T14:44:38.315978073Z" level=info msg="StartContainer for \"697069933adcb691d06b370583851f823c273fb11de48602b746081aee183446\" returns successfully"
time="2026-06-07T14:44:38.316735856Z" level=info msg="StartContainer for \"8e4783fee0ceb131b0d02d17621c7f28ec5dfb8870923daa338e2ca8e0742b06\" returns successfully"
time="2026-06-07T14:44:58.102039937Z" level=info msg="received container exit event container_id:\"aa73b58227073ccb27d074e405fa0556af316ef754e9f5604dcc5f55f6ca9fa7\" id:\"aa73b58227073ccb27d074e405fa0556af316ef754e9f5604dcc5f55f6ca9fa7\" exit_status:1 exited_at:{secon
ds:1780843497 nanos:770116186}"
time="2026-06-07T14:44:58.204021406Z" level=info msg="received container exit event container_id:\"cdc41b18306c73748170fa84474c3c473cbb241247ed1fea5f35aab91cb40fbf\" id:\"cdc41b18306c73748170fa84474c3c473cbb241247ed1fea5f35aab91cb40fbf\" exit_status:1 exited_at:{secon
ds:1780843497 nanos:770270568}"
time="2026-06-07T14:45:13.617483372Z" level=error msg="ttrpc: received message on inactive stream" stream=1535561
time="2026-06-07T14:45:14.335516001Z" level=error msg="ttrpc: received message on inactive stream" stream=1535563
time="2026-06-07T14:45:14.695272403Z" level=error msg="failed to handle container TaskExit event container_id:\"aa73b58227073ccb27d074e405fa0556af316ef754e9f5604dcc5f55f6ca9fa7\" id:\"aa73b58227073ccb27d074e405fa0556af316ef754e9f5604dcc5f55f6ca9fa7\" exit_status:1 exi
ted_at:{seconds:1780843497 nanos:770116186}" error="failed to stop container: context deadline exceeded"
time="2026-06-07T14:45:15.667726817Z" level=error msg="ttrpc: received message on inactive stream" stream=1535563
time="2026-06-07T14:45:15.828559284Z" level=error msg="failed to handle container TaskExit event container_id:\"cdc41b18306c73748170fa84474c3c473cbb241247ed1fea5f35aab91cb40fbf\" id:\"cdc41b18306c73748170fa84474c3c473cbb241247ed1fea5f35aab91cb40fbf\" exit_status:1 exi
ted_at:{seconds:1780843497 nanos:770270568}" error="failed to stop container: context deadline exceeded"
time="2026-06-07T14:45:15.981075102Z" level=error msg="ttrpc: received message on inactive stream" stream=1535561
time="2026-06-07T14:45:17.334277092Z" level=info msg="TaskExit event container_id:\"aa73b58227073ccb27d074e405fa0556af316ef754e9f5604dcc5f55f6ca9fa7\" id:\"aa73b58227073ccb27d074e405fa0556af316ef754e9f5604dcc5f55f6ca9fa7\" exit_status:1 exited_at:{seconds:1780843497 n
anos:770116186}"
time="2026-06-07T14:45:18.893666080Z" level=error msg="ExecSync for \"7621ec3c256f82a2e032fb90f4f52f8f5b28d57def0e06fb2ff8b8c4c7757f89\" failed" error="rpc error: code = DeadlineExceeded desc = failed to exec in container: timeout 15s exceeded: context deadline exceed
ed"
time="2026-06-07T14:45:20.575991683Z" level=info msg="received container exit event container_id:\"d3c07457aaf8020a87e0a10afd1c817f777b290d9d12eea5c8f60577c9669ae0\" id:\"d3c07457aaf8020a87e0a10afd1c817f777b290d9d12eea5c8f60577c9669ae0\" pid:2568614 exit_status:1 exit
ed_at:{seconds:1780843519 nanos:432111340}"
time="2026-06-07T14:45:22.152292058Z" level=error msg="ExecSync for \"7621ec3c256f82a2e032fb90f4f52f8f5b28d57def0e06fb2ff8b8c4c7757f89\" failed" error="rpc error: code = DeadlineExceeded desc = failed to exec in container: timeout 15s exceeded: context deadline exceeded"

156 Views

Open in Slack

Previous Next