white-battery-15789
08/10/2022, 9:50 PMbig-judge-33880
08/11/2022, 8:59 AMvolumeMode=Block
, I get a volume with volumeMode=Filesystem
, which appears to cause issues getting these volumes mounted by a downstream rke2 cluster that uses longhorn csilively-hospital-24301
08/12/2022, 3:24 PMlevel=warn ts=2022-08-11T20:36:59.034Z caller=main.go:849 fs_type=NFS_SUPER_MAGIC msg="This filesystem is not supported and may lead to data corruption and data loss. Please carefully read <https://prometheus.io/docs/prometheus/latest/storage/> to learn more about supported filesystems."
level=error ts=2022-08-12T11:00:13.904Z caller=db.go:821 component=tsdb msg="compaction failed" err="WAL truncation in Compact: create checkpoint: read segments: corruption in segment /prometheus/wal/00000016 at 60187438: unexpected full record"
Could anyone point me to the right direction of troubleshooting this issue?straight-businessperson-27680
08/13/2022, 12:52 PMstraight-businessperson-27680
08/13/2022, 1:00 PMstocky-article-82001
08/17/2022, 11:50 AMshy-megabyte-75492
08/17/2022, 11:54 AMgifted-stone-19912
08/17/2022, 2:55 PMflaky-coat-75909
08/19/2022, 11:22 AMMessage: EXT4-fs error (device sda): ext4_find_entry:1446: inode #12: comm grafana-server: reading directory lblock 0
And my Node Problem detector is showing {*reason*="Ext4Error"}
more than 1 thousand appears on one node
but in UI everything works fine (all is on green)
Grafana also working as well
I have 5 nodes let's say
node1, node2, node3, node4, node5
grafana is working on node3 and the data replicas is on node4 and node5
Mayby something is missing?bright-fireman-42144
08/23/2022, 11:04 PMbright-fireman-42144
08/23/2022, 11:25 PMaloof-hair-13897
08/31/2022, 2:22 PMswift-zebra-42479
09/01/2022, 5:46 AMbored-apple-66429
09/08/2022, 9:30 AMmagnificent-vr-88571
09/11/2022, 8:30 PME0911 20:16:38.965933 17195 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[data kube-api-access-ztp4j dshm]: timed out waiting for the condition" pod="cvat/cvat-postgresql-0"
E0911 20:23:07.393663 16782 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"container\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=container pod=metadata-grpc-deployment-f8d68f687-5fvbs_kubeflow(d72591f7-e2c4-475f-ad83-fc59c996219a)\"" pod="kubeflow/metadata-grpc-deployment-f8d68f687-5fvbs" podUID=d72591f7-e2c4-475f-ad83-fc59c996219a
I0911 20:23:08.718940 16782 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-62552b22-3e99-4b63-8a56-69519573ae1d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\|kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\>") pod \"loki-0\" (UID: \"8aef7574-fb66-415f-a130-6b8ec9091672\") "
E0911 20:23:08.724147 16782 nestedpendingoperations.go:335] Operation for "{volumeName:<http://kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d|kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d> podName: nodeName:}" failed. No retries permitted until 2022-09-11 20:25:10.724134581 +0000 UTC m=+21624.816950484 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-62552b22-3e99-4b63-8a56-69519573ae1d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\|kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\>") pod \"loki-0\" (UID: \"8aef7574-fb66-415f-a130-6b8ec9091672\") "
I0911 20:23:09.829046 16782 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\|kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\>") pod \"harbor-redis-0\" (UID: \"912226dd-12cf-4cb5-a54b-fb831b4e7e73\") "
E0911 20:23:09.831850 16782 nestedpendingoperations.go:335] Operation for "{volumeName:<http://kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d|kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d> podName: nodeName:}" failed. No retries permitted until 2022-09-11 20:25:11.831837052 +0000 UTC m=+21625.924652956 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\|kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\>") pod \"harbor-redis-0\" (UID: \"912226dd-12cf-4cb5-a54b-fb831b4e7e73\") "
Any solution to recover?creamy-mechanic-63134
09/19/2022, 12:26 PM\"<http://backingimagemanagers.longhorn.io|backingimagemanagers.longhorn.io>\" is invalid: status.storedVersions[1]: Invalid value: \"v1beta2\": must appear in spec.versions && cannot patch \"<http://backingimagedatasources.longhorn.io|backingimagedatasources.longhorn.io>\" with kind CustomResourceDefinition: <http://CustomResourceDefinition.apiextensions.k8s.io|CustomResourceDefinition.apiextensions.k8s.io> \"<http://backingimagedatasources.longhorn.io|backingimagedatasources.longhorn.io>\" is invalid: status.storedVersions[1]: Invalid value: \"v1beta2\": must appear in spec.versions && cannot patch \"<http://backuptargets.longhorn.io|backuptargets.longhorn.io>\" with kind CustomResourceDefinition: <http://CustomResourceDefinition.apiextensions.k8s.io|CustomResourceDefinition.apiextensions.k8s.io> \"<http://backuptargets.longhorn.io|backuptargets.longhorn.io>\" is invalid: status.storedVersions[1]: Invalid value: \"v1beta2\": must appear in spec.versions && cannot patch \"<http://backupvolumes.longhorn.io|backupvolumes.longhorn.io>\" with kind CustomResourceDefinition: <http://CustomResourceDefinition.apiextensions.k8s.io|CustomResourceDefinition.apiextensions.k8s.io> \"<http://backupvolumes.longhorn.io|backupvolumes.longhorn.io>\" is invalid: status.storedVersions[1]: Invalid value: \"v1beta2\": must appear in spec.versions && cannot patch \"<http://backups.longhorn.io|backups.longhorn.io>\" with kind CustomResourceDefinition: <http://CustomResourceDefinition.apiextensions.k8s.io|CustomResourceDefinition.apiextensions.k8s.io>
helpful-beard-54962
09/19/2022, 1:34 PMloud-daybreak-83328
09/19/2022, 2:56 PMloud-daybreak-83328
09/20/2022, 12:07 PMhigh-butcher-71851
09/20/2022, 10:35 PMnarrow-noon-75604
09/22/2022, 11:05 AM[pvc-28f57305-2b5b-44c8-9447-549101dea147-e-ded1735d] time="2022-09-22T10:59:13Z" level=warning msg="FAIL to discover due to Failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net iscsiadm -m discovery -t sendtargets -p 10.42.50.26], output , stderr, iscsiadm: Cannot perform discovery. Invalid Initiatorname.\niscsiadm: Could not perform SendTargets discovery: invalid parameter\n, error exit status 7"
[pvc-28f57305-2b5b-44c8-9447-549101dea147-e-ded1735d] time="2022-09-22T10:59:13Z" level=warning msg="Nodes cleaned up for iqn.2019-10.io.longhorn:pvc-28f57305-2b5b-44c8-9447-549101dea147"
I am not sure what I am missing. Any suggestions would be appreciated.big-judge-33880
09/26/2022, 3:07 PMproud-salesmen-12221
09/29/2022, 11:32 PM---
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
kind: StorageClass
metadata:
name: longhorn-crypto-v1-volume
provisioner: <http://driver.longhorn.io|driver.longhorn.io>
allowVolumeExpansion: true
parameters:
numberOfReplicas: "3"
staleReplicaTimeout: "2880" # 48 hours in minutes
fromBackup: ""
encrypted: "true"
<http://csi.storage.k8s.io/provisioner-secret-name|csi.storage.k8s.io/provisioner-secret-name>: ${pvc.name}
<http://csi.storage.k8s.io/provisioner-secret-namespace|csi.storage.k8s.io/provisioner-secret-namespace>: ${pvc.namespace}
<http://csi.storage.k8s.io/node-publish-secret-name|csi.storage.k8s.io/node-publish-secret-name>: ${pvc.name}
<http://csi.storage.k8s.io/node-publish-secret-namespace|csi.storage.k8s.io/node-publish-secret-namespace>: ${pvc.namespace}
<http://csi.storage.k8s.io/node-stage-secret-name|csi.storage.k8s.io/node-stage-secret-name>: ${pvc.name}
<http://csi.storage.k8s.io/node-stage-secret-namespace|csi.storage.k8s.io/node-stage-secret-namespace>: ${pvc.namespace}
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nginx-v1-pvc
namespace: v1-ns
spec:
accessModes:
- ReadWriteOnce
storageClassName: longhorn
resources:
requests:
storage: 2Gi
proud-salesmen-12221
09/29/2022, 11:34 PMsteep-furniture-72588
09/30/2022, 1:04 PMbland-painting-61617
10/02/2022, 9:00 AMAllow Node Drain with the Last Healthy Replica
is on but kured is unable to drain the node - there is one volume left on the node but it should be rebooted. Yesterday another node was stuck in this state with no volumes on the node itself - a bug?
evicting pod longhorn-system/instance-manager-e-9ca60819
error when evicting pods/"instance-manager-e-9ca60819" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"instance-manager-r-850386c0" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
bland-painting-61617
10/03/2022, 1:14 PMfull-toddler-53694
10/06/2022, 10:14 AMfull-toddler-53694
10/06/2022, 12:44 PMbig-judge-33880
10/07/2022, 6:54 AM