On an embedded device, what is the fastest way to safely shutdown k3s, if the device is about to loose power. And/or what is the harm in killing it all uncleanly? Have any of you seen corruption on the k3s side? (single node, super super small k3s ‘cluster’)

How much notice do you have that power is going? We have a reboot function in our cluster, and we basically just execute the reboot command (which shuts down services cleanly - we think!)

I am not certain on that yet. Its still under design, but I am guessing very little, like seconds…

If your running K3s as a service, you can always just stop the service.

Do you see much a performance difference when you use the etcd version @nutritious-tomato-14686? We are right at the very very edge on CPU performance. And do you find one to be less corruptible than the other? We actually dont care about the ‘state’ of any of our deployments, as long as they come back up on next restart.

Performance?) The etcd version will use more CPU at scale compared to sqlite. Corruption?) I have not heard or seen a major difference between the two when it comes to corruption. If your fine with sqlite and that recovery method, then you should be good to go. Killing K3s 10K times?) Yes this likely will cause issues over time. If you have an active cluster, with pods writing to DB and scheduling things, and traffic coming across, killing K3s hard (via either a

k3s-killall.sh

or a power loss) will cause corruption in your DB. You might get lucky and those lost writes are not needed, but do it a few thousand times and I guarantee you'll run into issues. I'm not sure what others are doing to "ensure k3s stays happy", there isn't a way AFAIK to verify the integrity of the K3s DB. At the end of the day, this is why High Availability K8s is so valuable, its very resilient when you have 10s of clusters made up of 3 nodes vs 100s of single node clusters.

We have certainly had sqlite database corruption in places where folks just yanked the power cords out of the backs of machines 😬

Thank you both for the replies! So our setup is very basic and only a single node with 4 tiny deployments that all work together. Their state never changes, I am thinking maybe I can nuke the whole setup if anything goes wrong and just re-deploy on startup? Is there a database health check I can run on k3s? Like at startup? And if it doesn’t show healthy can I just copy the old original database back in place and re-deploy?

Hey @quiet-memory-19288 I don't think that will helps with corruption on the DB. It just reports whether the DB is alive and reachable. AFAIK we don't have any translation support in kine for a

PRAGMA quick_check

Any chance there is a k3s document on like IoT deployments? Where we always come up in a stable state? Like lets say I install k3s and my deployments now 5+ years old. I dont care about anything that happens (ever) as long as every time it restarts we go back to that one known state? Maybe we upgrade every other year? But besides that, I just want a black box setup where the end users never knows it’s there or has to ‘do’ anything?