This message was deleted.

Huh, this pod terminating is just a consequence of the rather clunky (in my opinion) kubernetes disaster recovery system. As soon as your node will be seen as failed by kubes, prometheus should restart somewhere else.

That's exactly what I thought would happen but no. The node was shut down and all the pods got stuck either terminating or a blue filler 'ready' in Rancher. when I tried to check the logs they said no logs existed in Rancher and if I tried to get the logs using kubectl logs {pod} -n {namespace} it said the pod didn't exist? Although the pod was listed when listing all pods. I have fed this back to SUSE but I wonder if there is a better way to monitor the cluster using the Rancher API and whether anyone had done it?