blue-painting-33432
03/03/2023, 12:23 AMblue-painting-33432
03/03/2023, 12:23 AMUnable to attach or mount volumes: unmounted volumes=[alertmanager-kube-prometheus-stack-alertmanager-db], unattached volumes=[web-config kube-api-access-8lfhh config-volume config-out tls-assets alertmanager-kube-prometheus-stack-alertmanager-db]: timed out waiting for the condition
blue-painting-33432
03/03/2023, 12:24 AMblue-painting-33432
03/03/2023, 12:30 AMblue-painting-33432
03/03/2023, 1:05 AMacceptable-soccer-28720
03/03/2023, 2:04 AMrancher kubectl describe pod gitlab-postgresql-0 -n gitlab
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned gitlab/gitlab-postgresql-0 to vik8scases-w-2
Warning FailedMount 10m kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-g28wp custom-init-scripts postgresql-password dshm data]: timed out waiting for the condition
Warning FailedMount 7m49s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[postgresql-password dshm data kube-api-access-g28wp custom-init-scripts]: timed out waiting for the condition
Warning FailedMount 5m34s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[dshm data kube-api-access-g28wp custom-init-scripts postgresql-password]: timed out waiting for the condition
Warning FailedMount 3m18s (x2 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-g28wp custom-init-scripts postgresql-password dshm]: timed out waiting for the condition
Warning FailedAttachVolume 2m9s (x6 over 12m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-b59dafa1-3efa-44fc-92ba-e2be23e5d4a4" : timed out waiting for external-attacher of <http://driver.longhorn.io|driver.longhorn.io> CSI driver to attach volume pvc-b59dafa1-3efa-44fc-92ba-e2be23e5d4a4
Warning FailedMount 64s kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[custom-init-scripts postgresql-password dshm data kube-api-access-g28wp]: timed out waiting for the condition
plain-breakfast-5576
03/03/2023, 6:02 PMrich-shoe-36510
03/08/2023, 11:03 AMrich-shoe-36510
03/08/2023, 11:14 AMable-library-94454
03/09/2023, 4:08 AMable-library-94454
03/09/2023, 5:31 AMbig-judge-33880
03/09/2023, 9:18 AM[Thu Mar 9 09:13:02 2023] Buffer I/O error on dev dm-2, logical block 2441200, async page read
[Thu Mar 9 09:13:32 2023] buffer_io_error: 23 callbacks suppressed
[Thu Mar 9 09:13:32 2023] Buffer I/O error on dev dm-0, logical block 25837552, async page read
crooked-cat-21365
03/13/2023, 7:21 AMMar 12 00:00:02 srvl062 kernel: [713683.338101] nfs: server 10.43.170.7 not responding, timed out
Mar 12 00:00:03 srvl062 kernel: [713683.914078] nfs: server 10.43.187.251 not responding, timed out
Mar 12 00:00:03 srvl062 kernel: [713684.170058] nfs: server 10.43.28.105 not responding, timed out
Mar 12 00:00:05 srvl062 kernel: [713685.453982] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:05 srvl062 kernel: [713685.610363] nfs: server 10.43.133.91 not responding, timed out
Mar 12 00:00:05 srvl062 kernel: [713686.217960] nfs: server 10.43.28.71 not responding, timed out
Mar 12 00:00:06 srvl062 kernel: [713687.245956] nfs: server 10.43.43.8 not responding, timed out
Mar 12 00:00:08 srvl062 kernel: [713689.289855] nfs: server 10.43.131.82 not responding, timed out
Mar 12 00:00:10 srvl062 kernel: [713690.501785] nfs: server 10.43.170.7 not responding, timed out
Mar 12 00:00:10 srvl062 kernel: [713690.569795] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:13 srvl062 kernel: [713693.385742] nfs: server 10.43.187.251 not responding, timed out
Mar 12 00:00:13 srvl062 kernel: [713694.185659] nfs: server 10.43.43.8 not responding, timed out
Mar 12 00:00:14 srvl062 kernel: [713695.013664] nfs: server 10.43.28.105 not responding, timed out
Mar 12 00:00:15 srvl062 kernel: [713695.693620] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:16 srvl062 kernel: [713696.521595] nfs: server 10.43.133.91 not responding, timed out
Mar 12 00:00:20 srvl062 kernel: [713700.557466] nfs: server 10.43.170.165 not responding, timed out
Mar 12 00:00:20 srvl062 kernel: [713700.617405] nfs: server 10.43.86.90 not responding, timed out
Mar 12 00:00:20 srvl062 kernel: [713700.813461] nfs: server 10.43.131.82 not responding, timed out
Mar 12 00:00:21 srvl062 kernel: [713701.641479] nfs: server 10.43.187.251 not responding, timed out
Mar 12 00:00:22 srvl062 kernel: [713703.241337] nfs: server 10.43.70.182 not responding, timed out
Mar 12 00:00:24 srvl062 kernel: [713704.717343] nfs: server 10.43.170.7 not responding, timed out
Mar 12 00:00:25 srvl062 kernel: [713705.485281] nfs: server 10.43.202.97 not responding, timed out
What is the recommended procedure to get out of this?aloof-hair-13897
03/13/2023, 1:25 PMbland-painting-61617
03/13/2023, 3:50 PMcrooked-cat-21365
03/14/2023, 8:42 AMbusy-judge-4614
03/15/2023, 6:40 PMbland-painting-61617
03/15/2023, 8:07 PMfailed to set the setting priority-class with invalid value rancher-critical: cannot modify priority class setting before all volumes are detached
The change should be put in place and applied when possible, without causing downtime.
There is no easy way to detach all volumes without scaling all workload to 0, which is painful.
Maybe a kill/maintenance switch in longhorn can be implemented to facilitate such changes in an easier manner.creamy-pencil-82913
03/15/2023, 9:28 PMnumerous-lighter-90852
03/17/2023, 2:56 PMlate-needle-80860
03/17/2023, 3:12 PMlate-needle-80860
03/17/2023, 3:32 PMlate-needle-80860
03/17/2023, 4:03 PMlate-needle-80860
03/17/2023, 4:08 PMred-printer-88208
03/18/2023, 4:49 AMlate-needle-80860
03/18/2023, 6:36 PMlate-needle-80860
03/18/2023, 6:37 PMlate-needle-80860
03/18/2023, 8:26 PMbusy-judge-4614
03/19/2023, 12:34 PMbitter-tailor-6977
03/22/2023, 5:43 AMbitter-tailor-6977
03/22/2023, 5:43 AMicy-agency-38675
03/22/2023, 1:35 PM