Hi I m receinvg a events with message `Message EXT4 fs error rancher-users #longhorn-storage

Title

flaky-coat-75909

08/19/2022, 11:22 AM

Hi, I'm receinvg a events with message

Message: EXT4-fs error (device sda): ext4_find_entry:1446: inode #12: comm grafana-server: reading directory lblock 0

And my Node Problem detector is showing

{*reason*="Ext4Error"}

more than 1 thousand appears on one node but in UI everything works fine (all is on green) Grafana also working as well I have 5 nodes let's say node1, node2, node3, node4, node5 grafana is working on node3 and the data replicas is on node4 and node5 Mayby something is missing?

I have found https://longhorn.io/kb/troubleshooting-volume-readonly-or-io-error/

Root causes

2. The network bandwidth is not sufficient. Normally 1Gbps network will only able to serve 3 volumes if all of those volumes are running a high intensive workload

My Bandwith is

930 Mbits/sec

between nodes And node1 have 0 replicas (master) node2 have 5 replicas of diffrent pvc node3 have 6 replicas node4 have 3 replicas node5 have 3 replicas and number of volume is 6 (one of the biggest pvc volume is 15Gi) It can make a issue in my case?

icy-agency-38675

08/22/2022, 2:59 AM

The io error might be probably caused by the insufficient network bandwidth or short network outage.

flaky-coat-75909

08/22/2022, 11:05 AM

@icy-agency-38675 thanks for response hmm I suspect dns resoultion in my cluster (short network outage, or time resolution is sometimes 3sec instead of instant ) how longhorn-managers and other components will communicate to each other? they are using service name or something? (for example

instance-manager.svc...

) if not the second is short network outage in random periods

icy-agency-38675

08/23/2022, 12:47 AM

Use the IP address mostly.

6 Views

#longhorn-storage Join Slack