bitter-tailor-6977
03/22/2023, 5:43 AMicy-agency-38675
03/22/2023, 1:35 PMbitter-tailor-6977
03/23/2023, 4:24 AMicy-agency-38675
03/23/2023, 4:24 AMbitter-tailor-6977
03/23/2023, 4:41 AMicy-agency-38675
03/23/2023, 4:46 AMbitter-tailor-6977
03/23/2023, 4:48 AMunable to delete node: could not delete node ip-172-31-9-232.ec2.internal with node ready condition is False, reason is KubernetesNodeNotReady, node schedulable false, and 0 replica, 0 engine running on it
i hve deleted unknown pod and by using longhorn ui tried to delete the node above message was happening..
my cluster is in k3sicy-agency-38675
03/23/2023, 4:57 AMbitter-tailor-6977
03/23/2023, 4:59 AMicy-agency-38675
03/23/2023, 5:08 AMbitter-tailor-6977
03/23/2023, 5:11 AMicy-agency-38675
03/23/2023, 5:17 AMbitter-tailor-6977
03/23/2023, 5:19 AMicy-agency-38675
03/23/2023, 5:22 AMtar file
from the UI page.bitter-tailor-6977
03/23/2023, 5:23 AMicy-agency-38675
03/23/2023, 5:24 AMbitter-tailor-6977
03/23/2023, 5:25 AMicy-agency-38675
03/23/2023, 5:26 AMbitter-tailor-6977
03/23/2023, 5:26 AMicy-agency-38675
03/23/2023, 5:35 AMbitter-tailor-6977
03/23/2023, 5:38 AMicy-agency-38675
03/23/2023, 5:39 AMbitter-tailor-6977
03/23/2023, 5:40 AMicy-agency-38675
03/23/2023, 5:42 AMbitter-tailor-6977
03/23/2023, 5:47 AMicy-agency-38675
03/23/2023, 6:14 AM2023-03-23T05:13:54Z
The engine is running at 2023-03-23T05:13:51.887241937Z
The volume was just running right before you started the down nodes…bitter-tailor-6977
03/23/2023, 6:29 AMicy-agency-38675
03/23/2023, 6:45 AMbitter-tailor-6977
03/23/2023, 6:45 AMicy-agency-38675
03/23/2023, 6:45 AMbitter-tailor-6977
03/23/2023, 6:49 AMicy-agency-38675
03/23/2023, 6:49 AMkubectl -n longhorn-system get instancemanagers
bitter-tailor-6977
03/23/2023, 6:52 AMicy-agency-38675
03/23/2023, 6:53 AMbitter-tailor-6977
03/23/2023, 6:53 AMxyz ~ % k delete pod instance-manager-e-b9b279de634f2725487a4ceee253c408 instance-manager-r-b9b279de634f2725487a4ceee253c408 --force -n longhorn-system
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "instance-manager-e-b9b279de634f2725487a4ceee253c408" force deleted
pod "instance-manager-r-b9b279de634f2725487a4ceee253c408" force deleted
xyz ~ % kubectl -n longhorn-system get instancemanagers
NAME STATE TYPE NODE AGE
instance-manager-e-b9b279de634f2725487a4ceee253c408 unknown engine ip-172-31-8-245.ec2.internal 15h
instance-manager-r-b9b279de634f2725487a4ceee253c408 unknown replica ip-172-31-8-245.ec2.internal 15h
icy-agency-38675
03/23/2023, 6:56 AMbitter-tailor-6977
03/23/2023, 6:56 AMicy-agency-38675
03/23/2023, 6:58 AMroot@rancher60-master:~/longhorn# kl get instancemanager
NAME STATE TYPE NODE AGE
instance-manager-r-89ccb5ff5acd0803dc91e2b6355464d5 running replica rancher60-worker3 160m
instance-manager-e-89ccb5ff5acd0803dc91e2b6355464d5 running engine rancher60-worker3 160m
instance-manager-e-f4731def1cefda4f51631c82ea14ad12 running engine rancher60-worker1 160m
instance-manager-r-f4731def1cefda4f51631c82ea14ad12 running replica rancher60-worker1 160m
instance-manager-r-21850b3359743fcfc290fc3abf9cae85 unknown replica rancher60-worker2 160m
instance-manager-e-21850b3359743fcfc290fc3abf9cae85 unknown engine rancher60-worker2 160m
root@rancher60-master:~/longhorn# kl delete instancemanager instance-manager-e-21850b3359743fcfc290fc3abf9cae85
<http://instancemanager.longhorn.io|instancemanager.longhorn.io> "instance-manager-e-21850b3359743fcfc290fc3abf9cae85" deleted
root@rancher60-master:~/longhorn# kl get instancemanager
NAME STATE TYPE NODE AGE
instance-manager-r-89ccb5ff5acd0803dc91e2b6355464d5 running replica rancher60-worker3 160m
instance-manager-e-89ccb5ff5acd0803dc91e2b6355464d5 running engine rancher60-worker3 160m
instance-manager-e-f4731def1cefda4f51631c82ea14ad12 running engine rancher60-worker1 160m
instance-manager-r-f4731def1cefda4f51631c82ea14ad12 running replica rancher60-worker1 160m
instance-manager-r-21850b3359743fcfc290fc3abf9cae85 unknown replica rancher60-worker2 160m
bitter-tailor-6977
03/23/2023, 7:17 AMicy-agency-38675
03/23/2023, 7:17 AMbitter-tailor-6977
03/23/2023, 7:20 AMicy-agency-38675
03/23/2023, 11:29 AMbitter-tailor-6977
03/23/2023, 11:30 AM