https://rancher.com/ logo
Title
w

wonderful-kangaroo-15590

05/11/2022, 8:45 AM
hi, my longhorn volume failed to attach to pod/ node. Checking the dmesg log, I'm seeing error like this, do you have any idea what can go wrong here? kubernetes v1.20.15, longhorn v1.2.3, OS RHEL8.4
[Wed May 11 08:35:15 2022] iscsi: registered transport (tcp)
[Wed May 11 08:35:15 2022] scsi host4: iSCSI Initiator over TCP/IP
[Wed May 11 08:35:15 2022] scsi 4:0:0:0: RAID              IET      Controller       0001 PQ: 0 ANSI: 5
[Wed May 11 08:35:15 2022] scsi 4:0:0:0: Attached scsi generic sg2 type 12
[Wed May 11 08:35:15 2022] scsi 4:0:0:1: Direct-Access     IET      VIRTUAL-DISK     0001 PQ: 0 ANSI: 5
[Wed May 11 08:35:15 2022] sd 4:0:0:1: Power-on or device reset occurred
[Wed May 11 08:35:15 2022] sd 4:0:0:1: Attached scsi generic sg3 type 0
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] Write Protect is off
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] Mode Sense: 69 00 10 08
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] Write cache: enabled, read cache: enabled, supports DPO and FUA
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#17 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#17 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#17 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#17 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#18 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#18 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#18 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#19 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#19 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#19 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#19 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#20 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#20 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#20 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#20 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#21 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#21 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#21 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#22 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#22 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#22 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#22 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] Dev sdc: unable to read RDB block 0
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#23 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#23 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#23 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#23 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#24 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#24 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#24 CDB: Read(10) 28 00 00 00 00 00 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 0, async page read
[Wed May 11 08:35:15 2022]  sdc: unable to read partition table
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] Attached SCSI disk
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#61 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#61 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#61 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#61 CDB: Read(10) 28 00 00 1f ff 80 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 2097024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#62 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=0s
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#62 Sense Key : Medium Error [current]
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#62 Add. Sense: Unrecovered read error
[Wed May 11 08:35:15 2022] sd 4:0:0:1: [sdc] tag#62 CDB: Read(10) 28 00 00 1f ff 80 00 00 08 00
[Wed May 11 08:35:15 2022] blk_update_request: critical medium error, dev sdc, sector 2097024 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Wed May 11 08:35:15 2022] Buffer I/O error on dev sdc, logical block 262128, async page read
[Wed May 11 08:35:21 2022] sd 4:0:0:1: [sdc] Synchronizing SCSI cache
[root@pe-lab2-worker-vm-5 ~]#
I stop firewalld and thiss suddenly working again. Do we need to open any port here?
another note in here that, after re-enabling firewalld service, the already attached volume continue working fine. However, still for new volume, I cannot attach it to node
I install the old longhorn version 1.2.2 and 1.1.2, they are both working fine. 1.2.3, 1.2.4 and 1.1.3 are not working
wonder if it has anything to do with this https://github.com/longhorn/longhorn/issues/3419
­čĹŹ 1
f

famous-journalist-11332

05/31/2022, 11:55 PM