rancher-users #general

rhythmic-angle-26667

01/30/2023, 7:28 PM

It looks like there’s a replica still in the

running

state:

Copy code

NAME                                                  STATE     NODE        DISK                                   INSTANCEMANAGER                                       IMAGE                                    AGE
pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b-r-02dff950   stopped   k3s-sd-02   7b8edb0b-be57-445a-a46a-b51f9d4108fc                                                                                                  27h
pvc-e98b8f1f-be78-4f4c-ad89-c5dacff9b944-r-7802ebd8   stopped   k3s-sd-02   7b8edb0b-be57-445a-a46a-b51f9d4108fc                                                                                                  27h
pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b-r-e8f3a25f   stopped   k3s-sd-03   2a123a1a-e5d5-4ff2-9792-47681f0de058                                                                                                  27h
pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b-r-b7cd708f   stopped   k3s-sd-02   7b8edb0b-be57-445a-a46a-b51f9d4108fc                                                                                                  19h
pvc-e98b8f1f-be78-4f4c-ad89-c5dacff9b944-r-1b84d377   stopped   k3s-sd-03   2a123a1a-e5d5-4ff2-9792-47681f0de058                                                                                                  27h
pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b-r-87243b29   running   k3s-sd-03   2a123a1a-e5d5-4ff2-9792-47681f0de058   instance-manager-r-a12ca5e95c05a4540f1c4927edf6e26c   longhornio/longhorn-engine:master-head   19h
pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b-r-63c3c5b6   unknown   k3s-sd-01   ed2f0682-4140-41a6-b96d-f2d5e0eb4562   instance-manager-r-9a6b35fa67bbad2fe7f184ae87ea6659   longhornio/longhorn-engine:master-head   23h

But again, there’s no instance-manager running.

early-vegetable-32238

01/30/2023, 7:30 PM

ah. ok. So those pvc are linked to a PV. Can you first delete the PVC’s, once done, delete the pv’s related to this.

early-vegetable-32238

01/30/2023, 7:32 PM

detach those volumes, once the pvc’s are deleted.

rhythmic-angle-26667

01/30/2023, 7:33 PM

how do you detach the volume?

early-vegetable-32238

01/30/2023, 7:34 PM

in the longhorn ui: Volume > Name > Volume Details > Instance Manager

early-vegetable-32238

01/30/2023, 7:34 PM

there should be an option to detach the 2 volumes attached to the pvc above.

rhythmic-angle-26667

01/30/2023, 7:34 PM

Yeah there is no longhorn UI anymore. No pods are running in the

longhorn-system

namespace

early-vegetable-32238

01/30/2023, 7:35 PM

ah ok. what do you get by: kubectl get pvc,pv -n longhorn-system

victorious-kite-35099

01/30/2023, 7:37 PM

Hi everybody. I am reviewing the limits and resources of my Rancher-managed RKE2 cluster and, found out that the kube-apiserver and coredns pods inside the kube-system namespace have no resource limits and requests. Wouldn't it be a good idea to set some for them, as they are using quite substantial amount of memory? If so, how would I do that in Rancher?

rhythmic-angle-26667

01/30/2023, 7:37 PM

I cannot delete the PVs, it just hangs. Here’s the output you requested, minus unrelated PVs:

Copy code

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS        CLAIM                               STORAGECLASS   REASON   AGE
persistentvolume/pvc-e98b8f1f-be78-4f4c-ad89-c5dacff9b944   90Gi       RWO            Delete           Terminating   foo/bar               longhorn                27h
persistentvolume/pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b   1000Mi     RWO            Delete           Terminating   baz/bif   longhorn                28h

early-vegetable-32238

01/30/2023, 7:37 PM

is there no pvc linked to those pv’s?

early-vegetable-32238

01/30/2023, 7:37 PM

from your 1st output, I see 2 pvc’s

rhythmic-angle-26667

01/30/2023, 7:38 PM

Correct, the PVCs deleted fine. The PVs remain.

early-vegetable-32238

01/30/2023, 7:38 PM

and even deleting the pv’s with --force doesn’t do anything?

rhythmic-angle-26667

01/30/2023, 7:39 PM

Correct

early-vegetable-32238

01/30/2023, 7:40 PM

and the pv’s don’t have a finalizer? what’s the output if you do a get pv <pvname> -o yaml

rhythmic-angle-26667

01/30/2023, 7:41 PM

Copy code

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    <http://longhorn.io/volume-scheduling-error|longhorn.io/volume-scheduling-error>: ""
    <http://pv.kubernetes.io/provisioned-by|pv.kubernetes.io/provisioned-by>: <http://driver.longhorn.io|driver.longhorn.io>
  creationTimestamp: "2023-01-29T15:33:19Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2023-01-30T16:35:13Z"
  finalizers:
  - external-attacher/driver-longhorn-io
  name: pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b
  resourceVersion: "32232463"
  uid: c7555b24-d221-4a3b-9e46-d4039a34a76f
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1000Mi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: diaspora-images-longhorn
    namespace: diaspora
    resourceVersion: "32066620"
    uid: f04941e1-6181-40ec-9dd4-a0da45337b0b
  csi:
    driver: <http://driver.longhorn.io|driver.longhorn.io>
    fsType: ext4
    volumeAttributes:
      dataLocality: disabled
      fromBackup: ""
      fsType: ext4
      numberOfReplicas: "3"
      staleReplicaTimeout: "30"
      <http://storage.kubernetes.io/csiProvisionerIdentity|storage.kubernetes.io/csiProvisionerIdentity>: <http://1675000522421-8081-driver.longhorn.io|1675000522421-8081-driver.longhorn.io>
    volumeHandle: pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b
  persistentVolumeReclaimPolicy: Delete
  storageClassName: longhorn
  volumeMode: Filesystem
status:
  phase: Released

early-vegetable-32238

01/30/2023, 7:42 PM

wait a minute…. kubectl edit pv <pvname> and remove the finalizer line and the line below it

rhythmic-angle-26667

01/30/2023, 7:42 PM

Deleted the finalizers helped

early-vegetable-32238

01/30/2023, 7:42 PM

nice one Brad

early-vegetable-32238

01/30/2023, 7:43 PM

ok, you also mentioned that a replicaset is holding these pv’s. is there also a deployment?

early-vegetable-32238

01/30/2023, 7:43 PM

replicasets are created by a deployment. So removing the deployment, should delete de replicaset, but not the pv

rhythmic-angle-26667

01/30/2023, 7:44 PM

The deployments no longer have any reference to the volumes

early-vegetable-32238

01/30/2023, 7:46 PM

awesome Brad. Glad it worked. Remember….those finalizer bastard are a pain in the ass when troubleshooting. It took me quite some time to figure this out. I know it’s there for a reason, but sometimes it can become frustrating during investigations 🤣

rhythmic-angle-26667

01/30/2023, 7:46 PM

Trying to delete the finalizers from the longhorn volumes results in:

Copy code

error: <http://volumes.longhorn.io|volumes.longhorn.io> "pvc-f04941e1-6181-40ec-9dd4-a0da45337b0b" could not be patched: Internal error occurred: failed calling webhook "<http://validator.longhorn.io|validator.longhorn.io>": failed to call webhook: Post "<https://longhorn-admission-webhook.longhorn-system.svc:9443/v1/webhook/validaton?timeout=10s>": service "longhorn-admission-webhook" not found

early-vegetable-32238

01/30/2023, 7:46 PM

mmmm…..where is this webhook coming from. So there’s still some minor thing that’s preventing the deletion

rhythmic-angle-26667

01/30/2023, 7:47 PM

Likewise for

lhn

resources

rhythmic-angle-26667

01/30/2023, 7:47 PM

I deleted

lhei

and

lhe

ok tho

rhythmic-angle-26667

01/30/2023, 7:47 PM

after removing their finalizers

early-vegetable-32238

01/30/2023, 7:49 PM

nice one. yeah all those relations interconnecting with each other. there must be an easier way