This message was deleted.
# harvester
a
This message was deleted.
Doesn't look like the exact issue, but the pod names/deployments are probably relevant.
I'd check any outputs/logs from jobs objects from the system namespaces as well.
s
In my rancher dashboard I keep seeing FailedCreatePodSandbox with an error that states failed to allocate for range 0; no IP addresses available in the range set
p
By default Canal keeps track of pod IPs by creating a lock file for each IP in /var/lib/cni/networks/k8s-pod-network. Each IP belongs to a single pod and will be deleted as soon as the pod is removed. However, in the unlikely event that containerd loses track of the running pods, lock files may be leaked and Canal will not be able to reuse those IPs anymore. If this occurs, you may experience IP exhaustion errors, for example: failed to allocate for range 0: no IP addresses available in range set There are two ways to resolve this. You can either manually remove unused IPs from that directory or drain the node, run rke2-killall.sh, start the rke2 systemd service and uncordon the node. If you need to undertake any of these actions, please report the problem via GitHub, making sure to specify how it was triggered.
s
I restarted each node one at a time and that cleared it up. After restarting node 2, which the upgrade volume was attached to, it fixed the problem and is currently finishing the upgrade