This message was deleted.

...
Sep 24 13:24:08 prod-worker-b-1 sshd[3524]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:08 prod-worker-b-1 systemd-logind[811]: New session 17 of user root.
Sep 24 13:24:09 prod-worker-b-1 sshd[3524]: Received disconnect from 10.10.0.31 port 58128:11: disconnected by user
Sep 24 13:24:09 prod-worker-b-1 sshd[3524]: Disconnected from user root 10.10.0.31 port 58128
Sep 24 13:24:09 prod-worker-b-1 sshd[3524]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:09 prod-worker-b-1 systemd-logind[811]: Session 17 logged out. Waiting for processes to exit.
Sep 24 13:24:09 prod-worker-b-1 systemd-logind[811]: Removed session 17.
Sep 24 13:24:10 prod-worker-b-1 sshd[3707]: Accepted publickey for root from 10.10.0.31 port 32298 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:10 prod-worker-b-1 sshd[3707]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:10 prod-worker-b-1 systemd-logind[811]: New session 18 of user root.
Sep 24 13:24:11 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/mkdir -p /etc/systemd/system/docker.service.d
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:11 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/tee /etc/systemd/system/docker.service.d/10-machine.conf
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:11 prod-worker-b-1 sshd[3707]: Received disconnect from 10.10.0.31 port 32298:11: disconnected by user
Sep 24 13:24:11 prod-worker-b-1 sshd[3707]: Disconnected from user root 10.10.0.31 port 32298
Sep 24 13:24:11 prod-worker-b-1 sshd[3707]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:11 prod-worker-b-1 systemd-logind[811]: Session 18 logged out. Waiting for processes to exit.
Sep 24 13:24:11 prod-worker-b-1 systemd-logind[811]: Removed session 18.
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: Accepted publickey for root from 10.10.0.31 port 29613 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:12 prod-worker-b-1 systemd-logind[811]: New session 19 of user root.
Sep 24 13:24:12 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl daemon-reload
Sep 24 13:24:12 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:12 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: Received disconnect from 10.10.0.31 port 29613:11: disconnected by user
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: Disconnected from user root 10.10.0.31 port 29613
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:12 prod-worker-b-1 systemd-logind[811]: Session 19 logged out. Waiting for processes to exit.
Sep 24 13:24:12 prod-worker-b-1 systemd-logind[811]: Removed session 19.
Sep 24 13:24:14 prod-worker-b-1 sshd[3839]: Accepted publickey for root from 10.10.0.31 port 56568 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:14 prod-worker-b-1 sshd[3839]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:14 prod-worker-b-1 systemd-logind[811]: New session 20 of user root.
Sep 24 13:24:14 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl -f restart docker
Sep 24 13:24:14 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:15 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:15 prod-worker-b-1 sshd[3839]: Received disconnect from 10.10.0.31 port 56568:11: disconnected by user
Sep 24 13:24:15 prod-worker-b-1 sshd[3839]: Disconnected from user root 10.10.0.31 port 56568
Sep 24 13:24:15 prod-worker-b-1 sshd[3839]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:15 prod-worker-b-1 systemd-logind[811]: Session 20 logged out. Waiting for processes to exit.
Sep 24 13:24:15 prod-worker-b-1 systemd-logind[811]: Removed session 20.
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: Accepted publickey for root from 10.10.0.31 port 22440 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:16 prod-worker-b-1 systemd-logind[811]: New session 21 of user root.
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: Received disconnect from 10.10.0.31 port 22440:11: disconnected by user
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: Disconnected from user root 10.10.0.31 port 22440
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:16 prod-worker-b-1 systemd-logind[811]: Session 21 logged out. Waiting for processes to exit.
Sep 24 13:24:16 prod-worker-b-1 systemd-logind[811]: Removed session 21.
Sep 24 13:24:17 prod-worker-b-1 sshd[4059]: Accepted publickey for root from 10.10.0.31 port 35597 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:17 prod-worker-b-1 sshd[4059]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:17 prod-worker-b-1 systemd-logind[811]: New session 22 of user root.
Sep 24 13:24:17 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl -f enable docker
Sep 24 13:24:17 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:19 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:19 prod-worker-b-1 sshd[4059]: Received disconnect from 10.10.0.31 port 35597:11: disconnected by user
Sep 24 13:24:19 prod-worker-b-1 sshd[4059]: Disconnected from user root 10.10.0.31 port 35597
Sep 24 13:24:19 prod-worker-b-1 sshd[4059]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:19 prod-worker-b-1 systemd-logind[811]: Session 22 logged out. Waiting for processes to exit.
Sep 24 13:24:19 prod-worker-b-1 systemd-logind[811]: Removed session 22.
Sep 24 13:24:19 prod-worker-b-1 sshd[4209]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 24 13:24:19 prod-worker-b-1 sshd[4209]: Connection closed by authenticating user root 10.10.0.31 port 23482 [preauth]

Please note the various

COMMAND

executions until the login suddenly fails.

You're using too new of an image to work w/old RSA keys. SSH on the $node is not allowing ssh-rsa keys.

userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]

either go down to a lower version of your image that allows ssh-rsa or change your image templates to allow ssh-rsa is it's ssh algos

Yeah, that's what I thought, too. But the keys I am downloading from Rancher UI are RSA keys, I can successfully login to one of the failing nodes with it and also Rancher is initially somehow able to login with it.

2.5 has been EOL since Jan 2023. it's really nice that it's been working until now, but for security's sake it's definitely worth upgrading https://www.suse.com/lifecycle/#suse-rancher-prime

Im aware of that and in the progress of updating old infrastructure I somehow inherited and neglected too long.