I've playing with it for few hours now to no avail. I'm sure, that something is missing, but i have no clue what

Is there a load balancer in front of the Rancher server? Is Rancher running in Docker here?

no lb yes rancher is running in docker

So you are using an IP address to access the Rancher UI?

yes i’m

No, that is not where I am going. If you you were using a tool like

ngrok

or a load balancer to access the Rancher server, then I would know a potential solution. In this case, there would be a lot more debugging that would need to happen. The certificate hash is sent in cloud-init when setting up new nodes and the system-agent install script (which produces the logs that you posted) verifies the cert it gets from Rancher. For some reason that is not happening as expected for you.

adding a cert solved the issue

Great to hear!

Turns out to be the rancher instance sg not the tls after all i was sure that it had 0.0.0.0/0 on there. iv’e created this rule when creating the lb without thinking about it too much, that made everything work. now with this rule there even when i’m using the ip i can still create clusters very easily

A wise person once told me, “If you think it isn't a problem with security groups, then you should check the security groups.”

This wise person, is very wise indeed