Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
s
sticky-truck-78998
05/12/2022, 5:42 AMHas this burned anyone yet? We just found this setting. I hope it solves the issue for us. Lost a node (1of3)today and manually fixing stuck pods sucks. Luckily our admin was around when it happened. 3 hours later restored.
I feel like this should be highlighted a bit more. We have had no issues with longhorn unless a node had failed previously. We started testing LH a year ago and has been great. This issue(node fail)has caught us off guard a few times before but, thought it was related to something else. I'm curious what version this setting was implemented?
f
famous-journalist-11332
05/14/2022, 2:16 AMDoes turning on the setting help your case?
s
sticky-truck-78998
05/14/2022, 2:21 AMWe just finished testing today with VM longhorn environment with 80 pods. Seems to have fixed the problem. Takes a minimum of 5 minutes for it to restore. Working through how to handle pods that don't allow replica sets. I guess live with the downtime in the event of node failure. Don't expect that to happen much.
f
famous-journalist-11332
05/14/2022, 2:22 AMThank you for the feedback
Working through how to handle pods that don’t allow replica setsYeah, Longhorn cannot auto delete these pods because if Longhorn does, no one will recreate them
Why cann’t you putting those pods in a statefulset or deployment?
w
white-battery-15789
05/14/2022, 2:35 AMhello, our problems is more related to ReadWriteMany, some pods that need to share some volume, and we have to set affinity to them in order to have both on same node
as longhorn doesnt support it we have to use RWO
f
famous-journalist-11332
05/14/2022, 2:36 AMI see
w
white-battery-15789
05/14/2022, 2:36 AMbut the setting helped us, still testing with more containers, to make sure
but back when we had that problem, enabling the setting didnt solved, i had to enable it and force kubernetes to delete the pod
only after delete all pods, longhorn recovered and started to attach new volumes
also i had some timeout attaching pods, i had to do some attachment manually
f
famous-journalist-11332
05/14/2022, 2:38 AMyou might want to take a look at the experiment feature readwritemany https://longhorn.io/docs/1.2.4/advanced-resources/rwx-workloads/
w
white-battery-15789
05/14/2022, 2:38 AMi had read about that, im planning to try it out
do you guys have some set of setting that is recommended?
f
famous-journalist-11332
05/25/2022, 10:53 PMDo you mean is there any way to delete the pods are not inside any deployment or statefulset by Longhorn?