mammoth-manchester-45520
11/28/2022, 11:14 PMWarning FailedAttachVolume 88s (x10 over 5m43s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-5545180c-95e1-48d1-a905-f51380353016" : rpc error: code = DeadlineExceeded desc = volume pvc-5545180c-95e1-48d1-a905-f51380353016 failed to attach to node ip-10-0-3-184
yet otherwise, things generally look healthy. I am not really sure where to start troubleshooting things. I saw one issue that pointed at Linux firewalls causing problems, so I stopped ufw
(Ubuntu 20.04 system), but that didn't seem to help.
I looked at various longhorn pod logs for that node, and didn't notice anything, but any pointers to things to investigate, would be much appreciated.
Thanks!acceptable-printer-7134
11/29/2022, 4:17 AMlonghorn
for our storage needs. Currently we are dealing with this error , any help appreciated?
Warning ExternalExpanding 81s volume_expand Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC
acceptable-printer-7134
12/02/2022, 6:05 PMearly-magazine-12054
12/05/2022, 10:04 AMgentle-island-98110
12/05/2022, 5:21 PMgentle-island-98110
12/05/2022, 5:22 PMgentle-island-98110
12/05/2022, 5:23 PMchilly-salesmen-35649
12/07/2022, 9:50 AMchilly-salesmen-35649
12/07/2022, 9:52 AMchilly-salesmen-35649
12/07/2022, 9:52 AMbig-judge-33880
12/07/2022, 3:23 PMloud-daybreak-83328
12/15/2022, 1:05 PMmysterious-rose-43856
12/15/2022, 4:24 PMmysterious-rose-43856
12/15/2022, 4:29 PMgifted-stone-19912
12/15/2022, 9:12 PMquaint-book-15301
12/16/2022, 3:01 PMsparse-monitor-30665
12/19/2022, 4:07 AMmsg="Can't find backing image manager for pod, may be deleted" controller=longhorn-backing-image-manager node=harvester-gtsgd pod=backing-image-manager-d7ad-bfcf
Backing Image Manager: msg="SyncingFile: failed to get the checksum from a valid config during processing wrap-up, will directly calculated it then"
I'm still fairly novice in this eco-system, but I'll to my best to get more details if pointed in the right direction.wooden-planet-93714
12/20/2022, 9:47 AMplain-breakfast-5576
12/21/2022, 3:19 PMwooden-planet-93714
12/22/2022, 12:13 PMwonderful-appointment-6480
12/24/2022, 8:07 AMlonghorn-manager
fails to get ready, unable to list/watch resources, showing various errors such as:
W1224 07:44:23.287997 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2022-12-24T07:44:23Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n, error exit status 2"
time="2022-12-24T07:44:24Z" level=info msg="Skip the leader election for the upgrade since the current Longhorn system is already up to date"
W1224 07:44:24.943618 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1224 07:44:24.961708 1 shared_informer.go:240] Waiting for caches to sync for longhorn datastore
W1224 07:44:25.003320 1 reflector.go:324] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: failed to list *v1beta1.CronJob: the server could not find the requested resource
E1224 07:44:25.003407 1 reflector.go:138] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: Failed to watch *v1beta1.CronJob: failed to list *v1beta1.CronJob: the server could not find the requested resource
W1224 07:44:25.020929 1 reflector.go:324] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1224 07:44:25.020957 1 reflector.go:138] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1224 07:44:25.027597 1 kubernetes_node_controller.go:227] couldn't get kubernetes node master1: node "master1" not found
time="2022-12-24T07:44:25Z" level=warning msg="Can't find instance manager for pod instance-manager-e-e867fc5d, may be deleted"
time="2022-12-24T07:44:25Z" level=warning msg="Can't find instance manager for pod instance-manager-r-10c2a1ba, may be deleted"
Is it something I did wrong while installing? Any hints what may have gotten wrong?adventurous-mechanic-95332
01/02/2023, 3:16 AMquaint-alarm-7893
01/05/2023, 1:08 AMNode harvester-01 is ready 2.2 mins ago
Node harvester-01 is ready 2.2 mins ago
Node harvester-01 is ready 2.2 mins ago
Node harvester-01 is ready 2.2 mins ago
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running 2.2 mins ago
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running 2.2 mins ago
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running 2.2 mins ago
in the longhorn ui, the backing image its trying to create gets to 28%, then dies, and starts overstale-painting-80203
01/05/2023, 5:04 PMkubectl describe sc
Name: harvester
IsDefaultClass: Yes
Annotations: <http://meta.helm.sh/release-name=harvester-csi-driver,meta.helm.sh/release-namespace=kube-system,storageclass.kubernetes.io/is-default-class=true|meta.helm.sh/release-name=harvester-csi-driver,meta.helm.sh/release-namespace=kube-system,storageclass.kubernetes.io/is-default-class=true>
Provisioner: <http://driver.harvesterhci.io|driver.harvesterhci.io>
Parameters: <none>
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Delete
VolumeBindingMode: Immediate
Events: <none>
bumpy-accountant-48077
01/08/2023, 1:35 PMbitter-garage-45274
01/09/2023, 7:31 AMbitter-garage-45274
01/09/2023, 7:32 AMbitter-garage-45274
01/09/2023, 7:32 AMbitter-garage-45274
01/09/2023, 7:32 AMbitter-garage-45274
01/09/2023, 7:32 AM