https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
a
to add sdb I had to do 
kubectl.exe --kubeconfig .\local.yaml -n longhorn-system edit <http://node.longhorn.io|node.longhorn.io> harvester-02
and add a disk entrie + mount the disk
the issue is it's a kind of hack, and I have to mount myself to /var/lib/harvester/extra/sdb because fstab is overwritted at reboot
btw adding any sdx from UI result in a error
s
Hi @abundant-plastic-29088 Hmm, that’s weired… What is your harvester version? and Could you provide the support bundle or simply give the following output
Copy code
# kubectl get blockdevices -A -o yaml
a
Hi, I 'll reinstall from scratch, it was 1.2 rc6
@salmon-city-57654
Copy code
kubectl.exe --kubeconfig .\local.yaml get blockdevices -A -o yaml
apiVersion: v1
items:
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:35Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 1
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: disk
    name: 3f707a1d978d3eca880e16b4441ee4b5
    namespace: longhorn-system
    resourceVersion: "4530"
    uid: f6baa0cd-ac37-4a90-9794-ec381cd3b7a6
  spec:
    devPath: /dev/sdg
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 15938355200
      details:
        busPath: pci-0000:00:1d.0-usb-0:1.1.1:1.0-scsi-0:0:0:0
        deviceType: disk
        driveType: HDD
        isRemovable: true
        model: Innostor
        ptUUID: b3036c9d
        serialNumber: "148397037665904"
        storageController: SCSI
        vendor: Innostor
        wwn: unknown
      devPath: /dev/sdg
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      partitioned: true
    provisionPhase: Unprovisioned
    state: Active
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:35Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 1
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: disk
    name: 591e66c592678b93954b28e641838f88
    namespace: longhorn-system
    resourceVersion: "4511"
    uid: 0f23f8e8-2041-46a4-a850-72bc41a5cf0c
  spec:
    devPath: /dev/sde
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 1000204886016
      details:
        busPath: pci-0000:01:00.0-sas-phy1-lun-0
        deviceType: disk
        driveType: SSD
        model: PNY_1TB_SATA_SSD
        ptUUID: "60481770"
        serialNumber: PNF14230008880900695
        storageController: SCSI
        vendor: ATA
        wwn: "0x5f8db4c142300888"
      devPath: /dev/sde
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      partitioned: false
    provisionPhase: Unprovisioned
    state: Active
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:35Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 1
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: disk
    name: 6a74d0d99297c02900d194dccbf1ecba
    namespace: longhorn-system
    resourceVersion: "4515"
    uid: e0bdeace-9ae7-4518-9203-81254dd94afb
  spec:
    devPath: /dev/sdf
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 250059350016
      details:
        busPath: pci-0000:00:1f.2-ata-2.0
        deviceType: disk
        driveType: SSD
        model: PNY_CS900_250GB_SSD
        ptUUID: ccf46bdf
        serialNumber: PNY2242221020010C68F
        storageController: SCSI
        vendor: ATA
        wwn: "0x5f8db4c22420c68f"
      devPath: /dev/sdf
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      partitioned: false
    provisionPhase: Unprovisioned
    state: Active
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:36Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 1
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: part
      <http://ndm.harvesterhci.io/parent-device|ndm.harvesterhci.io/parent-device>: 3f707a1d978d3eca880e16b4441ee4b5
    name: 7dda37a25763aca12729123e521d55f4
    namespace: longhorn-system
    resourceVersion: "4533"
    uid: ca9c3cca-299c-43a8-8547-fe9278be150b
  spec:
    devPath: /dev/sdg2
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 33554432
      details:
        deviceType: part
        driveType: HDD
        label: VTOYEFI
        partUUID: b3036c9d-02
        storageController: SCSI
        uuid: 66F6-D57E
      devPath: /dev/sdg2
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      parentDevice: /dev/sdg
      partitioned: false
    provisionPhase: Unprovisioned
    state: Active
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:36Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 1
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: part
      <http://ndm.harvesterhci.io/parent-device|ndm.harvesterhci.io/parent-device>: 3f707a1d978d3eca880e16b4441ee4b5
    name: 94239507204ed26a4077a70f318ffde3
    namespace: longhorn-system
    resourceVersion: "4540"
    uid: c42c951d-adc3-4345-9f20-5f2a647cc010
  spec:
    devPath: /dev/sdg1
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 15903752192
      details:
        deviceType: part
        driveType: HDD
        label: Ventoy
        partUUID: b3036c9d-01
        storageController: SCSI
        uuid: 4E21-0000
      devPath: /dev/sdg1
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      parentDevice: /dev/sdg
      partitioned: false
    provisionPhase: Unprovisioned
    state: Active
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:35Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 1
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: disk
    name: e4b81e8cc1ea30d9ab63387d65d47b1e
    namespace: longhorn-system
    resourceVersion: "4535"
    uid: 8ab2290f-9c43-4042-b2b9-38f4bd6992a5
  spec:
    devPath: /dev/sdc
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 1000204886016
      details:
        busPath: pci-0000:01:00.0-sas-phy0-lun-0
        deviceType: disk
        driveType: SSD
        model: PNY_CS900_1TB_SSD
        ptUUID: 5bad403d
        serialNumber: PNY2316230420010016C
        storageController: SCSI
        vendor: ATA
        wwn: "0x5f8db4c23160016c"
      devPath: /dev/sdc
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      partitioned: false
    provisionPhase: Unprovisioned
    state: Active
- apiVersion: <http://harvesterhci.io/v1beta1|harvesterhci.io/v1beta1>
  kind: BlockDevice
  metadata:
    creationTimestamp: "2023-09-04T22:33:35Z"
    finalizers:
    - <http://wrangler.cattle.io/harvester-block-device-handler|wrangler.cattle.io/harvester-block-device-handler>
    generation: 2
    labels:
      <http://kubernetes.io/hostname|kubernetes.io/hostname>: harvester-01
      <http://ndm.harvesterhci.io/device-type|ndm.harvesterhci.io/device-type>: disk
    name: fe8eacdf009a23135a7ee8cf315f8bbc
    namespace: longhorn-system
    resourceVersion: "4545"
    uid: 1b79046a-bb1c-443b-9406-bd82cc9c3bfa
  spec:
    devPath: /dev/sdd
    fileSystem:
      mountPoint: ""
    nodeName: harvester-01
  status:
    deviceStatus:
      capacity:
        physicalBlockSizeBytes: 512
        sizeBytes: 1000204886016
      details:
        busPath: pci-0000:01:00.0-sas-phy1-lun-0
        deviceType: disk
        driveType: SSD
        model: PNY_1TB_SATA_SSD
        ptUUID: "60481770"
        serialNumber: PNF14230008880900695
        storageController: SCSI
        vendor: ATA
        wwn: "0x5f8db4c142300888"
      devPath: /dev/sde
      fileSystem:
        isReadOnly: true
        mountPoint: ""
        type: ""
      partitioned: false
    provisionPhase: Unprovisioned
    state: Active
kind: List
metadata:
  resourceVersion: ""
Copy code
harvester-01:/home/rancher # blkid
/dev/sdf: PTUUID="ccf46bdf" PTTYPE="dos"
/dev/sdd: PTUUID="6bb48608" PTTYPE="dos"
/dev/sdb: LABEL="HARV_LH_DEFAULT" UUID="3f601817-5869-4f75-b635-3fccc6280fca" BLOCK_SIZE="4096" TYPE="ext4" 
/dev/sdg1: LABEL="Ventoy" UUID="4E21-0000" BLOCK_SIZE="512" TYPE="exfat" PTTYPE="dos" PARTUUID="b3036c9d-01"
/dev/sdg2: SEC_TYPE="msdos" LABEL_FATBOOT="VTOYEFI" LABEL="VTOYEFI" UUID="66F6-D57E" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="b3036c9d-02"
/dev/loop0: LABEL="COS_ACTIVE" UUID="2542670c-9f6e-4240-a564-384a956e60ed" BLOCK_SIZE="4096" TYPE="ext2"
/dev/sde: PTUUID="60481770" PTTYPE="dos"
/dev/sdc: PTUUID="5bad403d" PTTYPE="dos"
/dev/sda4: LABEL="COS_STATE" UUID="1e1d464f-8eb7-4bbf-ba1b-f9674b19a040" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="state" PARTUUID="96ef48e9-a6a4-4eba-a06e-71f3453b3359"
/dev/sda2: LABEL="COS_OEM" UUID="ca82615c-0a06-456f-afaf-396a171383f3" BLOCK_SIZE="1024" TYPE="ext4" PARTLABEL="oem" PARTUUID="ff4c1b43-cb0a-4e43-9b95-bc387e3e65ed"    
/dev/sda5: LABEL="COS_PERSISTENT" UUID="9b9ec913-6f00-4645-bd58-a9eef7939214" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="persistent" PARTUUID="eec915fa-b6a2-44c6-a388-d48d465d5b29"
/dev/sda3: LABEL="COS_RECOVERY" UUID="9a6da12b-f96a-4ea3-a926-f3b7393bc279" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="recovery" PARTUUID="7d9b12b4-a263-4d5f-8e91-6dd7d5fe913b"
/dev/sda1: SEC_TYPE="msdos" LABEL_FATBOOT="COS_GRUB" LABEL="COS_GRUB" UUID="677B-50AE" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="efi" PARTUUID="fc97cf46-81cb-4ace-a1dd-14be4b19249e"
Copy code
harvester-01:/home/rancher # udevadm info --query=all --name=/dev/sda | grep ID_SERIAL
E: ID_SERIAL=PNY_CS900_250GB_SSD_PNY22422210200109AFD
E: ID_SERIAL_SHORT=PNY22422210200109AFD
harvester-01:/home/rancher # udevadm info --query=all --name=/dev/sdb | grep ID_SERIAL
E: ID_SERIAL=PNY_CS900_1TB_SSD_PNY2316230420010018E
E: ID_SERIAL_SHORT=PNY2316230420010018E
harvester-01:/home/rancher # udevadm info --query=all --name=/dev/sdc | grep ID_SERIAL
E: ID_SERIAL=PNY_CS900_1TB_SSD_PNY2316230420010016C
E: ID_SERIAL_SHORT=PNY2316230420010016C
harvester-01:/home/rancher # udevadm info --query=all --name=/dev/sdd | grep ID_SERIAL
E: ID_SERIAL=PNY_1TB_SATA_SSD_PNF14230008880901141
E: ID_SERIAL_SHORT=PNF14230008880901141
harvester-01:/home/rancher # udevadm info --query=all --name=/dev/sde | grep ID_SERIAL
E: ID_SERIAL=PNY_1TB_SATA_SSD_PNF14230008880900695
E: ID_SERIAL_SHORT=PNF14230008880900695
harvester-01:/home/rancher # udevadm info --query=all --name=/dev/sdf | grep ID_SERIAL
E: ID_SERIAL=PNY_CS900_250GB_SSD_PNY2242221020010C68F
E: ID_SERIAL_SHORT=PNY2242221020010C68F
Copy code
lshw -class disk
  *-disk:1
       description: ATA Disk
       product: PNY CS900 1TB SS
       physical id: 0.1.0
       bus info: scsi@0:0.1.0
       logical name: /dev/sdc
       version: 06D5
       serial: PNY2316230420010016C
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=5bad403d
  *-disk:2
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.2.0
       bus info: scsi@0:0.2.0
       logical name: /dev/sde
       version: 916a
       serial: PNF14230008880900695
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=60481770
  *-disk:3
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.3.0
       bus info: scsi@0:0.3.0
       logical name: /dev/sdd
       version: 916a
       serial: PNF14230008880901141
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=6bb48608
  *-disk
       description: SCSI Disk
       product: Innostor
       vendor: Innostor
       physical id: 0.0.0
       bus info: scsi@7:0.0.0
       logical name: /dev/sdg
       version: 1.00
       serial: 148397037665904
       size: 14GiB (15GB)
       capabilities: removable
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512
     *-medium
          physical id: 0
          logical name: /dev/sdg
          size: 14GiB (15GB)
          capabilities: partitioned partitioned:dos
          configuration: signature=b3036c9d
  *-disk:0
       description: ATA Disk
       product: PNY CS900 250GB
       physical id: 0
       bus info: scsi@1:0.0.0
       logical name: /dev/sda
       version: 0615
       serial: PNY22422210200109AFD
       size: 232GiB (250GB)
       capabilities: gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=5 guid=60336376-9f77-42fb-94bd-aca062205535 logicalsectorsize=512 sectorsize=512
  *-disk:1
       description: ATA Disk
       product: PNY CS900 250GB
       physical id: 1
       bus info: scsi@2:0.0.0
       logical name: /dev/sdf
       version: 0615
       serial: PNY2242221020010C68F
       size: 232GiB (250GB)
       configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512 signature=ccf46bdf
I don't know what more I can provide 😅
(now it's /dev/sde which is duplicated)
s
That’s Ok, I will use the support bundle for checking. This time we got the 
/dev/sde
duplicated?
@abundant-plastic-29088 ^^^
a
Yes
👍 1
s
Hi @abundant-plastic-29088, sorry for the late update. Could you use the sas device with dual port attached to your host? I saw the following log
Copy code
2023-09-04T22:33:35.458406526Z time="2023-09-04T22:33:35Z" level=info msg="Create new device fe8eacdf009a23135a7ee8cf315f8bbc with wwn: 0x5f8db4c14230
0888"
2023-09-04T22:33:35.460543540Z time="2023-09-04T22:33:35Z" level=info msg="Add new block device fe8eacdf009a23135a7ee8cf315f8bbc with device: /dev/sdd"
2023-09-04T22:33:35.470381595Z time="2023-09-04T22:33:35Z" level=info msg="Create new device 591e66c592678b93954b28e641838f88 with wwn: 0x5f8db4c142300888"
2023-09-04T22:33:35.472632631Z time="2023-09-04T22:33:35Z" level=info msg="Add new block device 591e66c592678b93954b28e641838f88 with device: /dev/sde"
Means you have the same device but exposed as 
/dev/sdd
and 
/dev/sde
. Could I know what is your hardware setup (Disks only)? BTW, so on your environment, you will have two devices`/dev/sdd` and 
/dev/sde
?
I opened the issue for it https://github.com/harvester/harvester/issues/4494. This should not happened.
a
Hi
Could you use the sas device with dual port attached to your host?
I'm not sure I got the question 😅 I'm already using the sas card for data disks (4x1Tb SSD) and 2 sata port directly from the host for os (2x250Gb), (I only use 1 if this disk, I'm too lazy to unplug the second 240Gb disk 😅
Could I know what is your hardware setup (Disks only)?
Copy code
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) // flashed in IT mode, work without issue on ceph as example
Copy code
sudo lshw -class disk
  *-disk:1
       description: ATA Disk
       product: PNY CS900 1TB SS
       physical id: 0.1.0
       bus info: scsi@0:0.1.0
       logical name: /dev/sdc
       version: 06D5
       serial: PNY2316230420010016C
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=5bad403d
  *-disk:2
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.2.0
       bus info: scsi@0:0.2.0
       logical name: /dev/sde
       version: 916a
       serial: PNF14230008880900695
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=60481770
  *-disk:3
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.3.0
       bus info: scsi@0:0.3.0
       logical name: /dev/sdd
       version: 916a
       serial: PNF14230008880901141
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=6bb48608
  *-disk:0
       description: ATA Disk
       product: PNY CS900 250GB
       physical id: 0
       bus info: scsi@1:0.0.0
       logical name: /dev/sda
       version: 0615
       serial: PNY22422210200109AFD
       size: 232GiB (250GB)
       capabilities: gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=5 guid=60336376-9f77-42fb-94bd-aca062205535 logicalsectorsize=512 sectorsize=512
  *-disk:1
       description: ATA Disk
       product: PNY CS900 250GB
       physical id: 1
       bus info: scsi@2:0.0.0
       logical name: /dev/sdf
       version: 0615
       serial: PNY2242221020010C68F
       size: 232GiB (250GB)
       configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512 signature=ccf46bdf
BTW, so on your environment, you will have two devices/dev/sdd and /dev/sde ?
you mean in the ui or in linux directly ?
s
Is this result also the same as the support bundle you generated?
because I only saw the 
PNF14230008880900695
, 
PNY2316230420010016C
, 
PNY2242221020010C68F
I saw you mentioned you use the LSI 2308 with IT mode. Does it means you use it for the external devices? (4 1TB ssd)
a
I saw you mentioned you use the LSI 2308 with IT mode. Does it means you use it for the external devices? (4 1TB ssd)
yes, data disk (1Gb) are on it, while os is on classic sata directly plugged to the motherboard of the server
👍 1
idk why lshw doesn't show the 4th disks
Copy code
sudo lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT
NAME   FSTYPE   SIZE MOUNTPOINT
loop0  ext2     1.4G /
sda           232.9G 
|-sda1 vfat      64M 
|-sda2 ext4      64M /oem
|-sda3 ext4       4G 
|-sda4 ext4       8G /run/initramfs/cos-state
`-sda5 ext4   220.8G /usr/local
sdb    ext4   931.5G /var/lib/harvester/defaultdisk
sdc           931.5G 
sdd           931.5G 
sde           931.5G 
sdf           232.9G 
sdh               2G /var/lib/kubelet/pods/b7b1504c-642b-4dda-b652-7fdc42118bf4/
sdi               5G /var/lib/kubelet/pods/f4d2833e-679b-425e-8805-f753f79b642b/
sdj              50G /var/lib/kubelet/pods/12144a4f-48a4-4a5d-8c05-bbb778305ba8/
s
You have 4 data disks 
sd[b-e]
, the one is chosen when you install right? The default data disk would not be present on the block devices. Could you give me the result of following command? The 
/dev/sdd
should be aware.
Copy code
# kubectl get blockdevices -A
I think the 4th 1T data disk is present as 
/dev/sdb
not sure why lshw doesn’t show this
a
currently installing kubectl on the mac 💤
brew is so long 😅
Copy code
kubectl --kubeconfig local.yaml get blockdevices -A
NAMESPACE         NAME                               TYPE   DEVPATH     MOUNTPOINT   NODENAME       PROVISIONPHASE   AGE
longhorn-system   07ffd6ef4fef104cc35f335803cd94f7   part   /dev/sdg2                harvester-02   Unprovisioned    37h
longhorn-system   3f707a1d978d3eca880e16b4441ee4b5   disk   /dev/sdg                 harvester-01   Unprovisioned    2d9h
longhorn-system   53a7fa6878c9add26f38fb00333a6225   disk   /dev/sdc                 harvester-02   Unprovisioned    37h
longhorn-system   591e66c592678b93954b28e641838f88   disk   /dev/sde                 harvester-01   Unprovisioned    2d9h
longhorn-system   6395c2a171dca8312044f9e79d307b4d   part   /dev/sdg1                harvester-02   Unprovisioned    37h
longhorn-system   6a74d0d99297c02900d194dccbf1ecba   disk   /dev/sdf                 harvester-01   Unprovisioned    2d9h
longhorn-system   7dda37a25763aca12729123e521d55f4   part   /dev/sdg2                harvester-01   Unprovisioned    2d9h
longhorn-system   8ecaeca76e369fd92f30d0a6574b15f7   disk   /dev/sdg                 harvester-02   Unprovisioned    37h
longhorn-system   94239507204ed26a4077a70f318ffde3   part   /dev/sdg1                harvester-01   Unprovisioned    2d9h
longhorn-system   97ed8a7576f9f483889a35267a8b7624   disk   /dev/sdc                 harvester-02   Unprovisioned    37h
longhorn-system   ae19bed8c76d3bb3f0d94d1e211eb94d   disk   /dev/sdc                 harvester-02   Unprovisioned    37h
longhorn-system   b14cf344ff7adac747015d4e8344881d   disk   /dev/sdf                 harvester-02   Unprovisioned    37h
longhorn-system   e4b81e8cc1ea30d9ab63387d65d47b1e   disk   /dev/sdc                 harvester-01   Unprovisioned    2d9h
longhorn-system   fe8eacdf009a23135a7ee8cf315f8bbc   disk   /dev/sde                 harvester-01   Unprovisioned    2d9h
I did not yet installed harvester on the 3th server
s
Did you ever reboot these nodes?
a
should I reboot the node 01 ?
I don't remember having rebooted
(also on harvester-02 I still have the usbkey with harvester iso)
s
Could you get the logs of harvester-node-disk-manager pod on the harvester-02?
a
kubectl --kubeconfig local.yaml logs harvester-node-disk-manager-dwzsw -n harvester-system
log-harvester.txt
s
It’s very weird for these logs
Copy code
time="2023-09-05T18:24:09Z" level=info msg="Create new device 53a7fa6878c9add26f38fb00333a6225 with wwn: 0x5f8db4c142300888"
time="2023-09-05T18:24:09Z" level=info msg="Add new block device 53a7fa6878c9add26f38fb00333a6225 with device: /dev/sdb"
time="2023-09-05T18:24:09Z" level=info msg="Create new device ae19bed8c76d3bb3f0d94d1e211eb94d with wwn: 0x5f8db4c142300888"
time="2023-09-05T18:24:09Z" level=info msg="Add new block device ae19bed8c76d3bb3f0d94d1e211eb94d with device: /dev/sdc"
time="2023-09-05T18:24:09Z" level=info msg="Create new device 97ed8a7576f9f483889a35267a8b7624 with wwn: 0x5f8db4c142300888"
time="2023-09-05T18:24:09Z" level=info msg="Add new block device 97ed8a7576f9f483889a35267a8b7624 with device: /dev/sde"
From your lshw info, these three devices should be different. Can you get lshw info again on harvester-02?
a
Copy code
rancher@harvester-02:~> sudo lshw -class disk
  *-disk:1                  
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.1.0
       bus info: scsi@0:0.1.0
       logical name: /dev/sdc
       version: 916a
       serial: PNF14230008880900682
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=cb6c9a01
  *-disk:2
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.2.0
       bus info: scsi@0:0.2.0
       logical name: /dev/sdb
       version: 916a
       serial: PNF14230008880901153
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=c941ce9e
  *-disk:3
       description: ATA Disk
       product: PNY 1TB SATA SSD
       physical id: 0.3.0
       bus info: scsi@0:0.3.0
       logical name: /dev/sde
       version: 916a
       serial: PNF14230008880901134
       size: 931GiB (1TB)
       capacity: 931GiB (1TB)
       capabilities: 15000rpm
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512 signature=1985e398
  *-disk
       description: SCSI Disk
       product: Innostor
       vendor: Innostor
       physical id: 0.0.0
       bus info: scsi@7:0.0.0
       logical name: /dev/sdg
       version: 1.00
       serial: 148397037665904
       size: 14GiB (15GB)
       capabilities: removable
       configuration: ansiversion=6 logicalsectorsize=512 sectorsize=512
     *-medium
          physical id: 0
          logical name: /dev/sdg
          size: 14GiB (15GB)
          capabilities: partitioned partitioned:dos
          configuration: signature=b3036c9d
  *-disk:0
       description: ATA Disk
       product: PNY CS900 250GB
       physical id: 0
       bus info: scsi@1:0.0.0
       logical name: /dev/sdd
       version: 0615
       serial: PNY22432210260101833
       size: 232GiB (250GB)
       capabilities: gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=5 guid=b6e54799-0604-4bf8-8703-ee35a0080e76 logicalsectorsize=512 sectorsize=512
  *-disk:1
       description: ATA Disk
       product: PNY CS900 250GB
       physical id: 1
       bus info: scsi@2:0.0.0
       logical name: /dev/sdf
       version: 0615
       serial: PNY2242221020010C690
       size: 232GiB (250GB)
       configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512 signature=15a044de
  *-disk
       description: SCSI Disk
       product: VIRTUAL-DISK
       vendor: IET
       physical id: 0.0.1
       bus info: scsi@8:0.0.1
       logical name: /dev/sdh
       version: 0001
       serial: beaf11
       size: 30GiB (32GB)
       capabilities: gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=5 guid=8a49268c-6948-4d42-a28b-c234361988b6 logicalsectorsize=512 sectorsize=512
👀 1
s
IIRC, if you use the SATA disk through the SAS card, the SAS card should distribute the wwn to make each one different. Looks like they are all the same. Could you try to upgrade your sas card firmware? Also, I will check if we could consider more like serial number to see if we could make it more flexible.
a
I believe I'm already on the latest firmware
yeah, 20.00.07 is the current firmware
s
For now, we could not handle this well in these cases, the wwn is the unique key we use to identify the disk.
Actually, the wwn should not be the same in the real world, I could see what we could improve in this case. I also checked the 2308 firmware, looks the 20.00.07 is the latest as you mentioned. Did you have other disks to test for the wwn?
a
unfortunally I have not other 2.5" disk
maybe I could use a SAS to 4 sata "octopus" cable and try with 1Tb hdd
s
Hmm, what a means for the sata 
octopus
?
a
at work we call it like that 😅
Hi, do you recommand any sas card affordable for an homelab ?
s
Hi @abundant-plastic-29088, sorry for the late reply. Actually, I do not use SAS cards in my local environment. But I tried to use “sas card” “homelab” as keywords, it will have some recommendations. 😅
a
Open in Slack
PreviousNext
Powered by