I don't have any volumes in use, I think I'm going to go through the checklist for removing it and making sure I follow all the best practices (perhaps properly this time? LOL) on longhorn.io

What’s you longhorn version?

1.3

Hello, can you check if the pods under longhorn-system are removed? after successful installation, you can install v1.3.1. I think you don't need to taint any node. Longhorn will be deployed on each worker node.

I'm on GKE and would like to dedicate nodes to longhorn so require nodes that can install the CSI driver. So I very much need to taint nodes.

I see. Shue. you can taint some nodes according to your design.

only thing 'running' in that namespace is the longhorn-uninstall batch job

Can you check the log?

which log? helm, that job, cluster...

ah, sorry. How did you uninstall LH?

probably a combination of multiple things. As this is not production I was not taking detailed notes. And my mistake, the longhorn-uninstall is in the default namespace.

I see. But I'd like to know how did you install and uninstall LH? Using kubelet or app catalog?

I think the rancher UI is just not in synch with a what I might have done via kubectl. It shows the longhorn-uninstall run was successful but it's still scheduled for deletion and a kubectl for all jobs in the default namespace does not show anything.

Let me summarize it. You installed the LH using app catalog and uninstalled it using kubectl. Correct?

well, the tainttolerations for the driver, UI, etc. components I mean. Then I applied the taints on the nodes 'manually'.

I see. Typically, if you install by app catalog, you need to uninstall it by the same tool. But it seems you use the different tools...Let me think how to deal with the case. I'll give you update if I find a solution.

currently the only thing in the longhorn-system namespace is a batch job in error and catalog app in transitioning

Got it. Looks you uninstall by kubectl. Can you finish the all step in https://longhorn.io/docs/1.3.0/deploy/uninstall/#uninstalling-longhorn-using-kubectl for removing v1.3.0? Then, delete the LH from rancher app catalog.

sure... I'll give it a shot. Thanks for the poking and prodding help. Appreciate it!

Don't mention it. 🙂

running through all the kubectl steps again.

Can you show

kubectl get pods

?

kubectl create -f https://raw.githubusercontent.com/longhorn/longhorn/v1.3.0/uninstall/uninstall.yaml

Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ job.batch/longhorn-uninstall created Error from server (AlreadyExists): error when creating "https://raw.githubusercontent.com/longhorn/longhorn/v1.3.0/uninstall/uninstall.yaml": podsecuritypolicies.policy "longhorn-uninstall-psp" already exists Error from server (AlreadyExists): error when creating "https://raw.githubusercontent.com/longhorn/longhorn/v1.3.0/uninstall/uninstall.yaml": serviceaccounts "longhorn-uninstall-service-account" already exists Error from server (AlreadyExists): error when creating "https://raw.githubusercontent.com/longhorn/longhorn/v1.3.0/uninstall/uninstall.yaml": clusterroles.rbac.authorization.k8s.io "longhorn-uninstall-role" already exists Error from server (AlreadyExists): error when creating "https://raw.githubusercontent.com/longhorn/longhorn/v1.3.0/uninstall/uninstall.yaml": clusterrolebindings.rbac.authorization.k8s.io "longhorn-uninstall-bind" already exists

kubectl get job/longhorn-uninstall -n default -w

NAME COMPLETIONS DURATION AGE longhorn-uninstall 0/1 29s 30s

Then, show

kubectl logs longhorn-uninstall-sc8b7

lots of 'failed to find' which is to be expected. Anything I'm looking for specifically?

Can you some whole error messages?

the end tail of the log:

Then, can you show

kubectl -n longhorn-system get pods
kubectl -n longhorn-system get crds

kubectl -n longhorn-system get pods NAME READY STATUS RESTARTS AGE longhorn-ui-768cf55d4d-m9nzw 1/1 Running 0 6d19h

Looks the uninstallation is completed.

cool, I'll try this but from the catalog and go through the steps to change the tolerations on the longhorn.io docs site. Thanks Derek. Sometimes just having someone there really helps.

Cool😀 Just try to install v1.3.1.

that and the stuck batch job.. have I mentioned I hate helm? 😉

Yes. You can check v1.3.0's webinar or introduction. In v1.3.1, we just fixed the bugs found in v1.3.0. So, it's much better.

🎉

kubectl -n longhorn-system logs <pod name>

kubectl get pods -n longhorn-system NAME READY STATUS RESTARTS AGE longhorn-admission-webhook-5985df5688-4cqhj 1/1 Running 0 59m longhorn-admission-webhook-5985df5688-dh4d5 1/1 Running 0 59m longhorn-conversion-webhook-7b74768796-94kvx 1/1 Running 0 59m longhorn-conversion-webhook-7b74768796-gfzv5 1/1 Running 0 59m longhorn-driver-deployer-5655d57559-5l78r 0/1 Init:0/1 0 59m longhorn-manager-kt29w 0/1 CrashLoopBackOff 16 (114s ago) 59m longhorn-manager-n6t7n 0/1 CrashLoopBackOff 16 (2m12s ago) 59m longhorn-ui-675b96b9f7-7wpmr 1/1 Running 0 59m

kubectl logs ^C

kubectl logs longhorn-driver-deployer-5655d57559-5l78r -n longhorn-system

Error from server (BadRequest): container "longhorn-driver-deployer" in pod "longhorn-driver-deployer-5655d57559-5l78r" is waiting to start: PodInitializing

kubectl logs longhorn-manager-kt29w -n longhorn-system

time="2022-08-24T173842Z" level=error msg="Failed environment check, please make sure you have iscsiadm/open-iscsi installed on the host" time="2022-08-24T173842Z" level=fatal msg="Error starting manager: environment check failed: Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net iscsiadm --version], output , stderr, nsenter: failed to execute iscsiadm: No such file or directory\n, error exit status 127"

Ah

annotations: {} csi: attacherReplicaCount: null kubeletRootDir: null provisionerReplicaCount: null resizerReplicaCount: null snapshotterReplicaCount: null defaultSettings: allowNodeDrainWithLastHealthyReplica: null allowRecurringJobWhileVolumeDetached: null allowVolumeCreationWithDegradedAvailability: null autoCleanupSystemGeneratedSnapshot: null autoDeletePodWhenVolumeDetachedUnexpectedly: null autoSalvage: null backingImageCleanupWaitInterval: null backingImageRecoveryWaitInterval: null backupTarget: null backupTargetCredentialSecret: null backupstorePollInterval: null concurrentAutomaticEngineUpgradePerNodeLimit: null concurrentReplicaRebuildPerNodeLimit: null createDefaultDiskLabeledNodes: null defaultDataLocality: null defaultDataPath: null defaultLonghornStaticStorageClass: null defaultReplicaCount: null disableReplicaRebuild: null disableRevisionCounter: null disableSchedulingOnCordonedNode: null guaranteedEngineManagerCPU: null guaranteedReplicaManagerCPU: null kubernetesClusterAutoscalerEnabled: null mkfsExt4Parameters: null nodeDownPodDeletionPolicy: null orphanAutoDeletion: null priorityClass: null replicaAutoBalance: null replicaReplenishmentWaitInterval: null replicaSoftAntiAffinity: null replicaZoneSoftAntiAffinity: null storageMinimalAvailablePercentage: null storageNetwork: null storageOverProvisioningPercentage: null systemManagedComponentsNodeSelector: null systemManagedPodsImagePullPolicy: null taintToleration: key=value:NoSchedule upgradeChecker: null enablePSP: true global: cattle: systemDefaultRegistry: '' windowsCluster: defaultSetting: systemManagedComponentsNodeSelector: kubernetes.io/os:linux taintToleration: cattle.io/os=linux:NoSchedule enabled: false nodeSelector: kubernetes.io/os: linux tolerations: - effect: NoSchedule key: cattle.io/os operator: Equal value: linux systemProjectId: p-s67s8 image: csi: attacher: repository: rancher/mirrored-longhornio-csi-attacher tag: v3.4.0 nodeDriverRegistrar: repository: rancher/mirrored-longhornio-csi-node-driver-registrar tag: v2.5.0 provisioner: repository: rancher/mirrored-longhornio-csi-provisioner tag: v2.1.2 resizer: repository: rancher/mirrored-longhornio-csi-resizer tag: v1.2.0 snapshotter: repository: rancher/mirrored-longhornio-csi-snapshotter tag: v3.0.3 longhorn: backingImageManager: repository: rancher/mirrored-longhornio-backing-image-manager tag: v3_20220808 engine: repository: rancher/mirrored-longhornio-longhorn-engine tag: v1.3.1 instanceManager: repository: rancher/mirrored-longhornio-longhorn-instance-manager tag: v1_20220808 manager: repository: rancher/mirrored-longhornio-longhorn-manager tag: v1.3.1 shareManager: repository: rancher/mirrored-longhornio-longhorn-share-manager tag: v1_20220808 ui: repository: rancher/mirrored-longhornio-longhorn-ui tag: v1.3.1 pullPolicy: IfNotPresent defaultImage: true ingress: annotations: null enabled: false host: sslip.io ingressClassName: null path: / secrets: null secureBackends: false tls: false tlsSecret: longhorn.local-tls longhornDriver: nodeSelector: {} priorityClass: null tolerations: null longhornManager: log: format: plain nodeSelector: {} priorityClass: null serviceAnnotations: {} tolerations: null longhornUI: nodeSelector: {} priorityClass: null replicas: 1 tolerations: null namespaceOverride: '' persistence: backingImage: dataSourceParameters: null dataSourceType: null enable: false expectedChecksum: null name: null defaultClass: true defaultClassReplicaCount: 3 defaultDataLocality: disabled defaultFsType: ext4 migratable: false reclaimPolicy: Delete recurringJobSelector: enable: false jobList: [] privateRegistry: createSecret: null registryPasswd: null registrySecret: null registryUrl: null registryUser: null resources: {} service: manager: loadBalancerIP: '' loadBalancerSourceRanges: '' nodePort: '' type: ClusterIP ui: nodePort: null type: ClusterIP serviceAccount: annotations: {} longhorn: default_setting: false

You have to install iscsi on each node

3 nodes can't have iscsi, 3 nodes are ubuntu specifically so it can hold the drivers

Check all prerequisites (like open-iscsi) are installed

was working before, just had some scheduling of resource issues that 'I think' were also related to having dedicated nodes just for longhorn.

Suggest to check all nodes again. Looks the log complains iscsi is not installed

I'll try... the other nodes are 'Container-Optimized OS from Google' so shrug, at least that gives me some things to literally 'google' LOL

I see…not sure they are supported🤔I will check tomorrow

definetly not for the nodes that actually have the csi driver for sure. Hence my taints and tolerations.

Awesome! Thanks for the update!

and as much as I dislike Rancher, LOL. this was easy to replace those worker nodes with the new ubuntu ones.

BTW, if you want longhorn supports

Container-Optimized

OS, you can create a ticket in https://github.com/longhorn/longhorn/issues. Then, we can investigate it.

not sure if google is interested in supporting issci but I'll see if anyone else has mentioned it in open issues. Thanks!