Morning everyone, I'm looking for emergency shutd...
# general
w
Morning everyone, I'm looking for emergency shutdown advice - we've an APC UPS that I can configure to shutdown servers and start them in an emergency, we've about 40 mins power coverage for our cluster - 3 node rancher management cluster managing a 5 node harvester setup and the clusters deployed on it. My question is - is this something we simply have to install on each host manually, or is there a "Rancher" way to hook up the UPS? Basically - when the power just goes we've seen drive fails and data corruption, even if we add a generator to the building we need to deal with the edge case of power not being available - hence a controlled shutdown may be needed so looking at how best to achieve this. Any advice greatly appreciated!
OK - figuring this is something you'd have to install manually on the bare-metal. However - first problem I've hit is installer wants to put binaries in /usr/bin
Copy code
Startup script=/usr/bin/PowerChute
cp: cannot create regular file '/usr/bin/PowerChute': Read-only file system
chmod: cannot access '/usr/bin/PowerChute': No such file or directory
I could customise the installer, ultimately harvester manages the installation of a node - so "patching" these nodes to allow the UPS to trigger an emergency shutdown is my train of thought - but looking like this might be tricky.. will post again if I work it out...
Copy code
n1:~/power # echo $PATH
/sbin:/usr/sbin:/usr/local/sbin:/root/bin:/usr/local/bin:/usr/bin:/bin:/var/lib/rancher/rke2/bin
/usr/local/bin is not a folder but is in the path - so will mod the installer to use this instead!
Nailed it.... to make this work you need to patch the installer as above - once configured the servers will hook up with the APC. Only gotacha I can see at the moment is when Harvester upgrades itself your changes are likely to be lost and you may need to patch the nodes again. I'll try to remember to post back when 1.4.2 is applied to see if this is the case or not - not sure if its possible to run a "post install" hook to trigger a script as that would make it easy to auto-patch the nodes on update 🤔
The above was tested with PowerChuteâ„¢ Network Shutdown v4.2