acoustic-addition-45641
04/27/2023, 5:55 PMbetter-garage-30620
04/27/2023, 8:49 PMacoustic-addition-45641
04/27/2023, 8:57 PMprehistoric-balloon-31801
04/28/2023, 3:56 AMapply-manifests
job shows what it’s waiting for, pease see
https://docs.harvesterhci.io/v1.1/upgrade/troubleshooting#phase-3-upgrade-system-servicessticky-summer-13450
04/28/2023, 12:36 PMapply-manifests
job shows that it’s waiting for this:
Waiting for CAPI cluster fleet-local/local to be provisioned (current phase: Provisioning, current generation: 225721)...
as it has for the last 15 hours?acoustic-addition-45641
04/28/2023, 12:59 PMwitty-jelly-95845
05/06/2023, 6:35 PMsticky-summer-13450
05/06/2023, 7:44 PM<http://upgrade.harvesterhci.io/hvst-upgrade-*|upgrade.harvesterhci.io/hvst-upgrade-*>
failes with the error Error from server (BadRequest): admission webhook "<http://validator.harvesterhci.io|validator.harvesterhci.io>" denied the request: cluster fleet-local/local status is provisioning, please wait for it to be provisioned
.prehistoric-balloon-31801
05/08/2023, 12:52 AMsticky-summer-13450
05/08/2023, 4:57 PMprehistoric-balloon-31801
05/09/2023, 3:28 AMdobby
Mark’s is waiting on node harvester003
crictl ps | grep kube-controller-manager
crictl ps | grep kube-scheduler
sticky-summer-13450
05/09/2023, 7:38 AMrancher@harvester003:~> uptime
07:34:18 up 2 days 16:38, 1 user, load average: 1.96, 1.87, 1.63
rancher@harvester003:~> sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock ps | grep kube-controller-manager
28e5a38a53d92 38df782a74380 2 days ago Running kube-controller-manager 6 0608ea1d9c747 kube-controller-manager-harvester003
rancher@harvester003:~> sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock ps | grep kube-scheduler
2279ccabaca22 38df782a74380 2 days ago Running kube-scheduler 4 fe906f7163aa3 kube-scheduler-harvester003
Created: *471 days ago*
Running the suggested test:
(
curl --cacert /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt \
<https://127.0.0.1:10257/healthz> >/dev/null 2>&1 \
&& echo "[OK] Kube Controller probe" \
|| echo "[FAIL] Kube Controller probe";
curl --cacert /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt \
<https://127.0.0.1:10259/healthz> >/dev/null 2>&1 \
&& echo "[OK] Scheduler probe" \
|| echo "[FAIL] Scheduler probe";
)
results in:
[FAIL] Kube Controller probe
[FAIL] Scheduler probe
So I have run something similar to the suggested fix (I need sudo and the full path to crictl
and the full path to the unix socket):
echo "Rotating kube-controller-manager certificate"
sudo rm /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.{crt,key}
sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock rm -f $(sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock ps -q --name kube-controller-manager)
echo "Rotating kube-scheduler certificate"
sudo rm /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.{crt,key}
sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock rm -f $(sudo /var/lib/rancher/rke2/bin/crictl -r unix:///var/run/k3s/containerd/containerd.sock ps -q --name kube-scheduler)
witty-jelly-95845
05/09/2023, 10:50 AM