adventurous-engine-93026
11/22/2022, 1:34 AMmagnificent-vr-88571
11/22/2022, 8:54 AMjolly-electrician-10061
11/22/2022, 3:34 PMjolly-camera-40096
11/23/2022, 10:34 PMchilly-jewelry-23407
11/24/2022, 12:45 AMbig-judge-33880
11/25/2022, 12:15 PMwonderful-pizza-30919
11/27/2022, 1:04 PMbig-judge-33880
11/27/2022, 4:25 PMFailed to pull image "rancher/harvester-cluster-repo:v1.1.0": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/rancher/harvester-cluster-repo:v1.1.0|docker.io/rancher/harvester-cluster-repo:v1.1.0>": failed to resolve reference "<http://docker.io/rancher/harvester-cluster-repo:v1.1.0|docker.io/rancher/harvester-cluster-repo:v1.1.0>": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
- did someone just nuke the repo?big-judge-33880
11/28/2022, 2:03 PMlively-translator-30710
11/29/2022, 3:40 PM[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: calico
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler, kubelet
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-controller-manager, kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-scheduler
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for probes: kube-apiserver
[INFO ] configuring bootstrap node(s) pax-pool1-cb9c56f68-l9fss: waiting for cluster agent to connect
alert-leather-47297
12/01/2022, 12:42 PMprehistoric-balloon-31801
12/02/2022, 6:28 AMkind-waitress-15815
12/02/2022, 9:24 AMkind-waitress-15815
12/02/2022, 9:25 AMlimited-magazine-85005
12/03/2022, 12:03 AMlimited-magazine-85005
12/03/2022, 12:06 AM[ 1061.946668] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-c3aa5aaa98e869f97a404b8a1bc27cbdfbdcd85960dcb021930bceffe70c6061.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568dc473_4171_4b3f_9e8f_61ad43186fa1.slice/cri-containerd-c3aa5aaa98e869f97a404b8a1bc27cbdfbdcd85960dcb021930bceffe70c6061.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568dc473_4171_4b3f_9e8f_61ad43186fa1.slice/cri-containerd-c3aa5aaa98e869f97a404b8a1bc27cbdfbdcd85960dcb021930bceffe70c6061.scope,task=ruby,pid=11345,uid=100
[ 1061.946678] Memory cgroup out of memory: Killed process 11345 (ruby) total-vm:955300kB, anon-rss:771480kB, file-rss:6780kB, shmem-rss:0kB
limited-magazine-85005
12/03/2022, 12:14 AMfast-oyster-13791
12/03/2022, 8:44 PMfast-oyster-13791
12/03/2022, 8:44 PMfast-oyster-13791
12/03/2022, 8:55 PMechoing-easter-77539
12/04/2022, 3:12 AMwonderful-pizza-30919
12/04/2022, 4:52 AMsticky-summer-13450
12/04/2022, 12:56 PM$ kubectl logs deployment/rancher -n cattle-system --context=harvester --since=1m --follow
...
evicting pod longhorn-system/instance-manager-r-1503169c
evicting pod longhorn-system/instance-manager-e-029f5eba
error when evicting pods/"instance-manager-r-1503169c" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"instance-manager-e-029f5eba" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-r-1503169c
evicting pod longhorn-system/instance-manager-e-029f5eba
error when evicting pods/"instance-manager-e-029f5eba" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
error when evicting pods/"instance-manager-r-1503169c" -n "longhorn-system" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod longhorn-system/instance-manager-r-1503169c
evicting pod longhorn-system/instance-manager-e-029f5eba
FWIW I got bored after watching them for around 15 minutes and killed the pods myself:
$ kubectl delete pod instance-manager-r-1503169c -n "longhorn-system" --context=harvester; kubectl delete pod instance-manager-e-029f5eba -n "longhorn-system" --context=harvester
Which got the upgrade moving again.
This happened at the post-drained phase of the 2nd and 3rd nodes of a 3 node Harvester cluster.wonderful-pizza-30919
12/04/2022, 11:36 PMwonderful-pizza-30919
12/04/2022, 11:39 PMquaint-alarm-7893
12/05/2022, 6:16 AMwonderful-pizza-30919
12/06/2022, 3:11 PMquaint-alarm-7893
12/06/2022, 9:46 PMcrooked-scooter-58172
12/07/2022, 7:47 PMcrooked-scooter-58172
12/08/2022, 2:33 PM