https://rancher.com/ logo
Powered by
#rke2
Title
# rke2
c

creamy-pencil-82913

01/09/2023, 5:22 AM
check the details for error info?
h

handsome-monitor-68857

01/09/2023, 5:24 AM
where do I have to look into? etcd pods?
c

creamy-pencil-82913

01/09/2023, 5:25 AM
three dots on the right side of the row, I believe?
or just click on the name perhaps?
I can’t remember
they are 0 bytes because they failed
The snapshot info is stored in a configmap, I'm curious what it has in it
Are there any errors in the rke2-server logs around the time those snapshots were taken?
h

handsome-monitor-68857

01/09/2023, 10:38 AM
I did an on-demand snapshot. There seemed no error
Copy code
lv-nodes@longvan-node1:~$ sudo journalctl -f -u rke2-server
-- Logs begin at Tue 2023-01-03 15:18:57 UTC. --
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.010Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400.part"}
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.013Z","logger":"client","caller":"v3@v3.5.4-k3s1/maintenance.go:211","msg":"opened snapshot stream; downloading"}
Jan 09 10:00:00 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:00.013Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"<https://127.0.0.1:2379>"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.043Z","logger":"client","caller":"v3@v3.5.4-k3s1/maintenance.go:219","msg":"completed snapshot read; closing"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.298Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"<https://127.0.0.1:2379>","size":"83 MB","took":"1 second ago"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: {"level":"info","ts":"2023-01-09T10:00:01.298Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400"}
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Applying local snapshot retention policy: retention: 5, snapshotPrefix: etcd-snapshot-longvan-node1, directory: /var/lib/rancher/rke2/server/db/snapshots"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673172000"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap"
Jan 09 10:00:01 longvan-node1 rke2[935083]: time="2023-01-09T10:00:01Z" level=info msg="Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete"
The snapshot is created:
Copy code
lv-nodes@longvan-node1:~$ sudo ls -alh /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
-rw------- 1 root root 79M Jan  9 10:00 /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
Full snapshot template:
Copy code
apiVersion: <http://rke.cattle.io/v1|rke.cattle.io/v1>
kind: ETCDSnapshot
metadata:
  annotations:
    <http://etcdsnapshot.rke.io/snapshot-file-name|etcdsnapshot.rke.io/snapshot-file-name>: etcd-snapshot-longvan-node1-1673258400
    <http://etcdsnapshot.rke.io/storage|etcdsnapshot.rke.io/storage>: local
  creationTimestamp: "2023-01-09T10:09:40Z"
  generation: 1
  labels:
    <http://rke.cattle.io/cluster-name|rke.cattle.io/cluster-name>: longvan
    <http://rke.cattle.io/machine-id|rke.cattle.io/machine-id>: da9fd3b82e0da646cdea34c9cac24a1c04c3ab4f24751a9f57a96cabdba7734
  managedFields:
  - apiVersion: <http://rke.cattle.io/v1|rke.cattle.io/v1>
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:<http://etcdsnapshot.rke.io/snapshot-file-name|etcdsnapshot.rke.io/snapshot-file-name>: {}
          f:<http://etcdsnapshot.rke.io/storage|etcdsnapshot.rke.io/storage>: {}
        f:labels:
          .: {}
          f:<http://rke.cattle.io/cluster-name|rke.cattle.io/cluster-name>: {}
          f:<http://rke.cattle.io/machine-id|rke.cattle.io/machine-id>: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"a63f64d9-40eb-4804-8e16-631ff8163f90"}: {}
      f:snapshotFile:
        .: {}
        f:location: {}
        f:name: {}
        f:nodeName: {}
      f:spec:
        .: {}
        f:clusterName: {}
    manager: rancher
    operation: Update
    time: "2023-01-09T10:09:40Z"
  name: longvan-etcd-snapshot-longvan-node1-1673258400-local
  namespace: fleet-default
  ownerReferences:
  - apiVersion: <http://cluster.x-k8s.io/v1beta1|cluster.x-k8s.io/v1beta1>
    blockOwnerDeletion: true
    controller: true
    kind: Machine
    name: custom-b35f992eb53c
    uid: a63f64d9-40eb-4804-8e16-631ff8163f90
  resourceVersion: "36849052"
  uid: 7d2852ec-8434-4c4f-b88d-1e3def5b7231
snapshotFile:
  location: file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-longvan-node1-1673258400
  name: etcd-snapshot-longvan-node1-1673258400
  nodeName: longvan-node1
spec:
  clusterName: longvan
If you can tell me what config map is holding the snapshot info, I will try to find it.
c

creamy-pencil-82913

01/09/2023, 4:26 PM
there is a job on the Rancher side that imports snapshot data from the config map on the downstream cluster, it sounds like maybe that is erroring out?
on the downstream cluster, it is rke2-etcd-snapshots in the kube-system namespace
111 Views
Powered by