https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
p
How do you determine if an update is safe or not? For example what if a patch comes out that breaks something majorly, it'll just continue to roll it out. Sure you can have cluster health checks but they would have to be pretty extensive to cover every type of 'this update broke the cluster' case. Fwiw I gave up on sles micro because it feels a bit half baked and as you described it needs a ton of extras and wrappers just to make it stable/viable
b
We do something similar for patching rke2 outside of rancher with ansible 1. check all workloads (deployments, statefulsets, k8s jobs etc) are healthy (ready pods = spec replicas) 2. cordon and drain a node 3. apply updates and reboot 4. wait for node to become ready 5. uncordon 6. wait for all workloads to become healthy 7. error if they don't, move on if they do We should replace step 1 & 6 with PodDisruptionBudgets but haven't had time yet to create those across all our workloads
eventually we would also like to add a step that queries alarms from https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack but haven't had time to do that yet
w
@polite-piano-74233 this is security updates only, as we're going node by node and checking post update (this includes allowing for a soak time and checking workloads) my feeling is the risks should be low. Ultimately risk should be one node. Out of interest what distro are you using? We tried a few and in the end SLES Micro came up much faster, played nicely with harvester/rancher and the basics such as NFS worked out of the gate - we've so far found it to be pretty good. My thought also is Its this also a good test to ensure that your architecture isn't brittle, if you've got your services structured in the right way and that services transition without disruption then its all good preparation for if and when things do go wrong... thats my theory anyway...
b
As I meant to say, is Kured worth looking at?
w
Haven't heard of this, I shall check it out - https://github.com/kubereboot/kured sounds like it might be ideal. In our case we have some specific considerations over some workloads that have to be left until last to migrate (in cluster container registry). I'll give this a good look over! Will see how it fits with what we have done 🙂
It actually doesn't quite cover our use case and adds additional overhead (daemonset) and I think this would be a different story with a non "tansactional" os. Our approach has the some advantages - 1. Control-plane quorum safety — our prepare waits for 
EtcdIsVoter=True
on all CP nodes before proceeding. kured has no concept of etcd quorum. 2. Uncordon timing — kured uncordons as soon as the node is back and passes the node health check. Our approach holds the maintenance taint until the next node's prepare script has verified full readiness, which is a stricter gate. 3. Wave ordering — workers can only start after all CP nodes are patched. kured treats all nodes equally. 4. Transactional snapshot model — 
transactional-update
creates a new BTRFS snapshot and activates it on reboot. kured has no visibility into whether the snapshot is clean or whether the reboot-required state is actually from a new snapshot vs. an old pending one. Kured is the right choice for "safe Kubernetes reboots" — but that's on distributions where the OS does the patching independently (Debian/Ubuntu etc). SUSE Leap with transactional-update is a different model where the patch orchestration and reboot are tightly coupled by design.
👍 1
Actually gone full circle now... yet to test this fully, but basic problem is the upstream base images are behind the updates - so manual update is needed. On this basis we took a step back and realised that we have harvester, so its simple enough to create a VM, update it then export the image to then use from our mirror. This takes less time and gives us a stable image to pull from, and removed the complexity since rancher can then handle the transition of the nodes.
2 Views
Open in Slack
PreviousNext
Powered by