https://rancher.com/ logo
Join Slack
Powered by
Hi. I am having a strange issue with an older ver...
# general
q
Hi. I am having a strange issue with an older version of Rancher 
v2.5.17
and I cannot figure it out. I provision nodes at Hetzner and at some point during the bootstrap of new nodes I receive the following error message in Rancher UI: 
Ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain : exit status 1
. Observing 
/var/log/auth.log
at the new machine I see various successful logins and command executions by Rancher, but at some point they fail with 
userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
(full log in 🧵). I have a very similar setup on a test cluster using the same OS image where the same works without issues. Any ideas why that happens? Rancher doesn't switch SSH login keys during bootstrap, does it?
Copy code
...
Sep 24 13:24:08 prod-worker-b-1 sshd[3524]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:08 prod-worker-b-1 systemd-logind[811]: New session 17 of user root.
Sep 24 13:24:09 prod-worker-b-1 sshd[3524]: Received disconnect from 10.10.0.31 port 58128:11: disconnected by user
Sep 24 13:24:09 prod-worker-b-1 sshd[3524]: Disconnected from user root 10.10.0.31 port 58128
Sep 24 13:24:09 prod-worker-b-1 sshd[3524]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:09 prod-worker-b-1 systemd-logind[811]: Session 17 logged out. Waiting for processes to exit.
Sep 24 13:24:09 prod-worker-b-1 systemd-logind[811]: Removed session 17.
Sep 24 13:24:10 prod-worker-b-1 sshd[3707]: Accepted publickey for root from 10.10.0.31 port 32298 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:10 prod-worker-b-1 sshd[3707]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:10 prod-worker-b-1 systemd-logind[811]: New session 18 of user root.
Sep 24 13:24:11 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/mkdir -p /etc/systemd/system/docker.service.d
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:11 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/tee /etc/systemd/system/docker.service.d/10-machine.conf
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:11 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:11 prod-worker-b-1 sshd[3707]: Received disconnect from 10.10.0.31 port 32298:11: disconnected by user
Sep 24 13:24:11 prod-worker-b-1 sshd[3707]: Disconnected from user root 10.10.0.31 port 32298
Sep 24 13:24:11 prod-worker-b-1 sshd[3707]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:11 prod-worker-b-1 systemd-logind[811]: Session 18 logged out. Waiting for processes to exit.
Sep 24 13:24:11 prod-worker-b-1 systemd-logind[811]: Removed session 18.
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: Accepted publickey for root from 10.10.0.31 port 29613 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:12 prod-worker-b-1 systemd-logind[811]: New session 19 of user root.
Sep 24 13:24:12 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl daemon-reload
Sep 24 13:24:12 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:12 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: Received disconnect from 10.10.0.31 port 29613:11: disconnected by user
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: Disconnected from user root 10.10.0.31 port 29613
Sep 24 13:24:12 prod-worker-b-1 sshd[3759]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:12 prod-worker-b-1 systemd-logind[811]: Session 19 logged out. Waiting for processes to exit.
Sep 24 13:24:12 prod-worker-b-1 systemd-logind[811]: Removed session 19.
Sep 24 13:24:14 prod-worker-b-1 sshd[3839]: Accepted publickey for root from 10.10.0.31 port 56568 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:14 prod-worker-b-1 sshd[3839]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:14 prod-worker-b-1 systemd-logind[811]: New session 20 of user root.
Sep 24 13:24:14 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl -f restart docker
Sep 24 13:24:14 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:15 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:15 prod-worker-b-1 sshd[3839]: Received disconnect from 10.10.0.31 port 56568:11: disconnected by user
Sep 24 13:24:15 prod-worker-b-1 sshd[3839]: Disconnected from user root 10.10.0.31 port 56568
Sep 24 13:24:15 prod-worker-b-1 sshd[3839]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:15 prod-worker-b-1 systemd-logind[811]: Session 20 logged out. Waiting for processes to exit.
Sep 24 13:24:15 prod-worker-b-1 systemd-logind[811]: Removed session 20.
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: Accepted publickey for root from 10.10.0.31 port 22440 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:16 prod-worker-b-1 systemd-logind[811]: New session 21 of user root.
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: Received disconnect from 10.10.0.31 port 22440:11: disconnected by user
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: Disconnected from user root 10.10.0.31 port 22440
Sep 24 13:24:16 prod-worker-b-1 sshd[4011]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:16 prod-worker-b-1 systemd-logind[811]: Session 21 logged out. Waiting for processes to exit.
Sep 24 13:24:16 prod-worker-b-1 systemd-logind[811]: Removed session 21.
Sep 24 13:24:17 prod-worker-b-1 sshd[4059]: Accepted publickey for root from 10.10.0.31 port 35597 ssh2: RSA SHA256:RLdutecJfZeWc+83QnDfifz8z2JCR7UbsJKt4YBCTqE
Sep 24 13:24:17 prod-worker-b-1 sshd[4059]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:17 prod-worker-b-1 systemd-logind[811]: New session 22 of user root.
Sep 24 13:24:17 prod-worker-b-1 sudo:     root : PWD=/root ; USER=root ; COMMAND=/usr/bin/systemctl -f enable docker
Sep 24 13:24:17 prod-worker-b-1 sudo: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Sep 24 13:24:19 prod-worker-b-1 sudo: pam_unix(sudo:session): session closed for user root
Sep 24 13:24:19 prod-worker-b-1 sshd[4059]: Received disconnect from 10.10.0.31 port 35597:11: disconnected by user
Sep 24 13:24:19 prod-worker-b-1 sshd[4059]: Disconnected from user root 10.10.0.31 port 35597
Sep 24 13:24:19 prod-worker-b-1 sshd[4059]: pam_unix(sshd:session): session closed for user root
Sep 24 13:24:19 prod-worker-b-1 systemd-logind[811]: Session 22 logged out. Waiting for processes to exit.
Sep 24 13:24:19 prod-worker-b-1 systemd-logind[811]: Removed session 22.
Sep 24 13:24:19 prod-worker-b-1 sshd[4209]: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Sep 24 13:24:19 prod-worker-b-1 sshd[4209]: Connection closed by authenticating user root 10.10.0.31 port 23482 [preauth]
Please note the various 
COMMAND
executions until the login suddenly fails.
w
You're using too new of an image to work w/old RSA keys. SSH on the $node is not allowing ssh-rsa keys.
Copy code
userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
either go down to a lower version of your image that allows ssh-rsa or change your image templates to allow ssh-rsa is it's ssh algos
q
Yeah, that's what I thought, too. But the keys I am downloading from Rancher UI are RSA keys, I can successfully login to one of the failing nodes with it and also Rancher is initially somehow able to login with it.
s
2.5 has been EOL since Jan 2023. it's really nice that it's been working until now, but for security's sake it's definitely worth upgrading https://www.suse.com/lifecycle/#suse-rancher-prime
q
Im aware of that and in the progress of updating old infrastructure I somehow inherited and neglected too long.
Having said that, any idea what might go wrong?
The (working) test env I mentioned runs on v2.6 (and I am aware that it’s probably also EOL) and I will upgrade if that could fix the problem. However, being able to provision capacity is a more pressing problem to me.
Just for the record: updating to 
v2.6.11
fixed this issue. Next steps: more updates! Yay!
Open in Slack
PreviousNext
Powered by