check the details for error info rancher-users #rke2

Title

creamy-pencil-82913

01/09/2023, 5:22 AM

check the details for error info?

handsome-monitor-68857

01/09/2023, 5:24 AM

where do I have to look into? etcd pods?

creamy-pencil-82913

01/09/2023, 5:25 AM

three dots on the right side of the row, I believe?

or just click on the name perhaps?

I can’t remember

they are 0 bytes because they failed

The snapshot info is stored in a configmap, I'm curious what it has in it

Are there any errors in the rke2-server logs around the time those snapshots were taken?

handsome-monitor-68857

01/09/2023, 10:38 AM

I did an on-demand snapshot. There seemed no error

lv-nodes@longvan-node1:~$ sudo journalctl -f -u rke2-server
-- Logs begin at Tue 2023-01-03 15:18:57 UTC. --
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.010Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400.part"}
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.013Z","logger":"client","caller":"v3@v3.5.4-k3s1/maintenance.go:211","msg":"opened snapshot stream; downloading"}
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.013Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"<https://127.0.0.1:2379>"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.043Z","logger":"client","caller":"v3@v3.5.4-k3s1/maintenance.go:219","msg":"completed snapshot read; closing"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.298Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"<https://127.0.0.1:2379>","size":"83 MB","took":"1 second ago"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.298Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Applying local snapshot retention policy: retention: 5, snapshotPrefix: etcd-snapshot-longvan-node1, directory: /var/lib/rancher/rke2/server/db/snapshots"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673172000"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete"

The snapshot is created:

lv-nodes@longvan-node1:~$ sudo ls -alh /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
-rw------- 1 root root 79M Jan  9 10:00 /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400

Full snapshot template:

apiVersion: <http://rke.cattle.io/v1|rke.cattle.io/v1>
kind: ETCDSnapshot
metadata:
  annotations:
    <http://etcdsnapshot.rke.io/snapshot-file-name|etcdsnapshot.rke.io/snapshot-file-name>: etcd-snapshot-longvan-node1-1673258400
    <http://etcdsnapshot.rke.io/storage|etcdsnapshot.rke.io/storage>: local
  creationTimestamp: "2023-01-09T10:09:40Z"
  generation: 1
  labels:
    <http://rke.cattle.io/cluster-name|rke.cattle.io/cluster-name>: longvan
    <http://rke.cattle.io/machine-id|rke.cattle.io/machine-id>: da9fd3b82e0da646cdea34c9cac24a1c04c3ab4f24751a9f57a96cabdba7734
  managedFields:
  - apiVersion: <http://rke.cattle.io/v1|rke.cattle.io/v1>
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:<http://etcdsnapshot.rke.io/snapshot-file-name|etcdsnapshot.rke.io/snapshot-file-name>: {}
          f:<http://etcdsnapshot.rke.io/storage|etcdsnapshot.rke.io/storage>: {}
        f:labels:
          .: {}
          f:<http://rke.cattle.io/cluster-name|rke.cattle.io/cluster-name>: {}
          f:<http://rke.cattle.io/machine-id|rke.cattle.io/machine-id>: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"a63f64d9-40eb-4804-8e16-631ff8163f90"}: {}
      f:snapshotFile:
        .: {}
        f:location: {}
        f:name: {}
        f:nodeName: {}
      f:spec:
        .: {}
        f:clusterName: {}
    manager: rancher
    operation: Update
    time: "2023-01-09T10:09:40Z"
  name: longvan-etcd-snapshot-longvan-node1-1673258400-local
  namespace: fleet-default
  ownerReferences:
  - apiVersion: <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: custom-b35f992eb53c
    uid: a63f64d9-40eb-4804-8e16-631ff8163f90
  resourceVersion: "36849052"
  uid: 7d2852ec-8434-4c4f-b88d-1e3def5b7231
snapshotFile:
  location: file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
  name: etcd-snapshot-longvan-node1-1673258400
  nodeName: longvan-node1
spec:
  clusterName: longvan

If you can tell me what config map is holding the snapshot info, I will try to find it.

creamy-pencil-82913

01/09/2023, 4:26 PM

there is a job on the Rancher side that imports snapshot data from the config map on the downstream cluster, it sounds like maybe that is erroring out?

on the downstream cluster, it is rke2-etcd-snapshots in the kube-system namespace

52 Views

#rke2 Join Slack