https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
check the details for error info?
h
where do I have to look into? etcd pods?
c
three dots on the right side of the row, I believe?
or just click on the name perhaps?
I can’t remember
they are 0 bytes because they failed
h
It's just that.
Looks like it's a bug in GUI, as I can see the snapshot created on related node
Rancher GUI
c
The snapshot info is stored in a configmap, I'm curious what it has in it
Are there any errors in the rke2-server logs around the time those snapshots were taken?
h
I did an on-demand snapshot. There seemed no error
Copy code
lv-nodes@longvan-node1:~$ sudo journalctl -f -u rke2-server
-- Logs begin at Tue 2023-01-03 15:18:57 UTC. --
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.010Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400.part"}
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.013Z","logger":"client","caller":"v3@v3.5.4-k3s1/maintenance.go:211","msg":"opened snapshot stream; downloading"}
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.013Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"<https://127.0.0.1:2379>"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.043Z","logger":"client","caller":"v3@v3.5.4-k3s1/maintenance.go:219","msg":"completed snapshot read; closing"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.298Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"<https://127.0.0.1:2379>","size":"83 MB","took":"1 second ago"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.298Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Applying local snapshot retention policy: retention: 5, snapshotPrefix: etcd-snapshot-longvan-node1, directory: /var/lib/rancher/rke2/server/db/snapshots"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673172000"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete"
The snapshot is created:
Copy code
lv-nodes@longvan-node1:~$ sudo ls -alh /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
-rw------- 1 root root 79M Jan  9 10:00 /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
In Cluster Management, Snapshots tab, it's being shown up as zero bytes
But if I click on the snapshot name, it is displayed as Active in ETCDSnapsho t list.
Full snapshot template:
Copy code
apiVersion: <http://rke.cattle.io/v1|rke.cattle.io/v1>
kind: ETCDSnapshot
metadata:
  annotations:
    <http://etcdsnapshot.rke.io/snapshot-file-name|etcdsnapshot.rke.io/snapshot-file-name>: etcd-snapshot-longvan-node1-1673258400
    <http://etcdsnapshot.rke.io/storage|etcdsnapshot.rke.io/storage>: local
  creationTimestamp: "2023-01-09T10:09:40Z"
  generation: 1
  labels:
    <http://rke.cattle.io/cluster-name|rke.cattle.io/cluster-name>: longvan
    <http://rke.cattle.io/machine-id|rke.cattle.io/machine-id>: da9fd3b82e0da646cdea34c9cac24a1c04c3ab4f24751a9f57a96cabdba7734
  managedFields:
  - apiVersion: <http://rke.cattle.io/v1|rke.cattle.io/v1>
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:<http://etcdsnapshot.rke.io/snapshot-file-name|etcdsnapshot.rke.io/snapshot-file-name>: {}
          f:<http://etcdsnapshot.rke.io/storage|etcdsnapshot.rke.io/storage>: {}
        f:labels:
          .: {}
          f:<http://rke.cattle.io/cluster-name|rke.cattle.io/cluster-name>: {}
          f:<http://rke.cattle.io/machine-id|rke.cattle.io/machine-id>: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"a63f64d9-40eb-4804-8e16-631ff8163f90"}: {}
      f:snapshotFile:
        .: {}
        f:location: {}
        f:name: {}
        f:nodeName: {}
      f:spec:
        .: {}
        f:clusterName: {}
    manager: rancher
    operation: Update
    time: "2023-01-09T10:09:40Z"
  name: longvan-etcd-snapshot-longvan-node1-1673258400-local
  namespace: fleet-default
  ownerReferences:
  - apiVersion: <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: custom-b35f992eb53c
    uid: a63f64d9-40eb-4804-8e16-631ff8163f90
  resourceVersion: "36849052"
  uid: 7d2852ec-8434-4c4f-b88d-1e3def5b7231
snapshotFile:
  location: file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
  name: etcd-snapshot-longvan-node1-1673258400
  nodeName: longvan-node1
spec:
  clusterName: longvan
If you can tell me what config map is holding the snapshot info, I will try to find it.
c
there is a job on the Rancher side that imports snapshot data from the config map on the downstream cluster, it sounds like maybe that is erroring out?
on the downstream cluster, it is rke2-etcd-snapshots in the kube-system namespace
165 Views
Open in Slack
PreviousNext
Powered by