well the cluster agent crashed...

time="2024-10-29T134055Z" level=info msg="Connecting to wss://rancher-test.blah.io/v3/connect/register with token starting with xxxxxxxxxxxxxxxxxxxxx │ │ time="2024-10-29T134055Z" level=info msg="Connecting to proxy" url="wss://rancher-test.blah.io/v3/connect/register" │ │ time="2024-10-29T134055Z" level=error msg="Failed to connect to proxy. Response status: 400 - 400 Bad Request. Response body: cluster not found" error=" │ │ time="2024-10-29T134055Z" level=error msg="Remotedialer proxy error" error="websocket: bad handshake"

rotate certs see what happens.

Rookie Lesson from Rancher Rodeo for Newbies 101: Clusters should be considered ephemeral (and unreliable) and you need your devops working. Unfortunately getting our devops working is never a priority and always too complex and just out of reach. From our perspective clusters need to be always available and reliable and never updated 🙂 just like hot and cold tap water, and need to run our legacy stateful monolithic suboptimal deployments of applications with embedded ElasticSearch. We're always just doing it wrong and wronger 😭

I think I can takeaway a strong position against Documenting and Gold-Plating anything to do with infrastructure -- why document and test and gold-plate that which is always ephemeral and unreliable? I would push back any enthusiasm for infrastructure as code and say "get your devops working first". Be able to push production to any set of standard endpoints in 5 minutes without service interruption.