I noticed that it's possible to rollout various ssh-keys to the Harvester nodes at installation time. To follow the normal lifecycle it is necessary to be able to add or remove keys later at runtime. How can this be achieved?

ssh-copy-id

?

ssh rancher@<node>

then

vi ~/.ssh/authorized_keys

?

Really? Thanks! Didn't think it's that easy and "normal". I was told that this is not going to work because the key is lost after every reboot... I think I need to take a deeper look into the node setup.

SSH keys in VMs (whether provisioned via cloud-init using key attached during VM creation or subsequently added after) should remain post-restart of VM. Which OS/image are you using for VM?

I think the OP was asking about the Harvester nodes, not the VMs running on the nodes - although it's the same answer for both 🙂

Note IP addresses of VMs can change after reboot so might be giving appearance of SSH keys disappearing - they're not, just that you're authenticating to what's now the wrong IP

Yes, I meant Harvester nodes only. I have multiple admins here and need to decide wether to use one shared ssh key for the rancher user or deploy separate user keys for each admin. When somebody is leaving, I need to deploy a new single key vs. I just need to adjust the authorizedkeys file. I prefer multiple keys but I was also told that this is not working because the authorizedkeys file is reset after a reboot or update. As per your description this doesn't seem to be true.

I'm not SUSE but that's poop - after installation /home/rancher/.ssh/authorized_keys doesn't exist on a Harvester node (unless you choose to import keys during install - I don't*). I create and add my desktop one post-install to both my nodes. I can then edit that file later and the changes stick post-reboot as I've just tested. Even if whoever told you that has confused with VMs it's still not true as you can change those too (though cloud-init can confuse things a bit). *it's possible importing during install starts a process rewriting file but that would be nuts for this exact reason!

Ok, thanks. I'm currently trying to install Harvester in a VM (no baremetal here) and play around a bit myself. I was wondering why Rancher would introduce such a "feature"... it would be a really bad design decision... even though the infrastructure should be immutable in general.

4 CPU cores should just be enough but 8GB RAM is not. I've yet to try upgrading my nodes as don't want to break them right now!

Nope. 4 cores are not enough because at some point the overall CPU reservations cannot be satisfied and some pods don't start anymore. Easily visible in the cluster events.