https://rancher.com/ logo
Join SlackCommunities
Powered by
This message was deleted.
# academy
a
This message was deleted.
l
Thank you for the feedback! Any particular error you're getting? cc: @happy-wire-88980
b
Hi, it just sits there trying to pull the image "rancherlabs/swiss-army-knife" and gets nowhere...
h
Can you post a kubectl describe pod Longhorn-simple-pod
b
Copy code
kubectl describe pod longhorn-simple-pod
Name:             longhorn-simple-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             longhorn3/192.168.10.132
Start Time:       Sun, 10 Mar 2024 09:13:11 -0500
Labels:           <none>
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Containers:
  swiss-army-knife:
    Container ID:   
    Image:          rancherlabs/swiss-army-knife
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from volv (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9z282 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  volv:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  longhorn-simple-pvc
    ReadOnly:   false
  kube-api-access-9z282:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  42s   default-scheduler  Successfully assigned default/longhorn-simple-pod to longhorn3
Some more event output;
Copy code
Warning  FailedAttachVolume  15m (x2 over 17m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-9d24c52b-886d-4308-95e3-b19bde787788" : rpc error: code = Aborted desc = volume pvc-9d24c52b-886d-4308-95e3-b19bde787788 is not ready for workloads
  Warning  FailedAttachVolume  43s (x15 over 19m)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-9d24c52b-886d-4308-95e3-b19bde787788" : rpc error: code = DeadlineExceeded desc = volume pvc-9d24c52b-886d-4308-95e3-b19bde787788 failed to attach to node longhorn3 with attachmentID csi-874491a268d846c14e98cddea8a14ad729d76e756c63efbd67982232a8bf6bb1
  Warning  FailedMount         41s (x9 over 18m)   kubelet                  Unable to attach or mount volumes: unmounted volumes=[volv], unattached volumes=[volv], failed to process volumes=[]: timed out waiting for the condition
h
The PVC isn’t ready. Do you have the iSCSI client installed on the node?
b
yes, let me check if it's running
iscsid.service - Open-iSCSI running on all three nodes
h
b
Copy code
kubectl apply -f <https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-iscsi-installation.yaml>
error: error parsing <https://github.com/longhorn/longhorn/blob/master/deploy/prerequisite/longhorn-iscsi-installation.yaml>: error converting YAML to JSON: yaml: line 203: mapping values are not allowed in this context
h
O you need to grab the RAW link, I’m on my phone and can’t grab it right now
b
ok, it applied after I downloaded it, I see three instances of Init:CrashLoopBackoff
h
What OS are you running on the nodes?
b
Ahh sudo command not found, I'm on Leap 15.5
Copy code
bash: sudo: command not found                                                                                                                                                               │
│ Stream closed EOF for default/longhorn-iscsi-installation-wznzz (iscsi-installation)
Copy code
modprobe: FATAL: Module iscsi_tcp not found in directory /lib/modules/5.14.21-150500.55.49-default                                                                                          │
│ Stream closed EOF for default/longhorn-iscsi-installation-7x72z (iscsi-installation)
h
If you run "ls -l /dev/longhorn/" on node longhorn3/192.168.10.132 Do you see the scsi device?
b
No
h
OK, that means the "client" isn't pickup the volume. Can you run a support bundle from the Longhorn UI and grab a log collect from the node in-question? https://github.com/rancherlabs/support-tools/tree/master/collection/rancher/v2.x/logs-collector
b
Will do, what do I put for the URL?
h
Anything is fine
thank you, let me take a look
b
Many thanks for your help 😄
h
Here is the problem
Copy code
024-03-10T10:04:45.059252131-05:00 [pvc-9d24c52b-886d-4308-95e3-b19bde787788-e-0] time="2024-03-10T15:04:45Z" level=warning msg="Failed to discover" func="iscsidev.(*Device).StartInitator" file="iscsi.go:161" error="failed to execute: /usr/bin/nsenter [nsenter --mount=/host/proc/1042/ns/mnt --net=/host/proc/1042/ns/net iscsiadm -m discovery -t sendtargets -p 10.42.2.24], output , stderr libkmod: kmod_module_insert_module: could not find module by name='iscsi_tcp'\niscsiadm: Could not insert module tcp. Kmod error -2\niscsiadm: iSCSI driver tcp is not loaded. Load the module then retry the command.\niscsiadm: Could not perform SendTargets discovery: iSCSI driver not found. Please make sure it is loaded, and retry the operation\n: exit status 12"
2024-03-10T10:04:45.069324130-05:00 [pvc-9d24c52b-886d-4308-95e3-b19bde787788-e-0] time="2024-03-10T15:04:45Z" level=warning msg="Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-9d24c52b-886d-4308-95e3-b19bde787788" func="iscsidev.(*Device).StartInitator" file="iscsi.go:168"
try running modprobe iscsi_tcp
If that fixes it, run echo "iscsi_tcp" | sudo tee /etc/modules-load.d/iscsi.conf to make the change persistent
b
yes, the module is not present in kernel-default, need to locate it...
OK, so the qcow image I used is the base kernel, had to swap for kernel-default, also needed to add sudo and hostname, deleted the old pod, and now all good... Maybe the check script needs some tweaking?
oh, kernel-default loaded the iscsi_tcp module ok
I also wonder if I needed to apply the longhorn-iscsi-installation?
h
So that runs a DS so it should just auto start
b
ahh ok, well many thanks for the help, I need to go back and do it all again now... make sure my notes are all good!
h
your welcome
b
@limited-pizza-33551 we go it sorted 😄
31 Views
Open in Slack
PreviousNext
Powered by