https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
a
from the Rancher docker container I see
Copy code
2023/05/16 21:32:07 [ERROR] Failed to handle tunnel request from remote address :40636: response 400: <http://nodes.management.cattle.io|nodes.management.cattle.io> "c-x8lmc/m-308799fb7fde" not found
2023/05/16 21:32:07 [ERROR] Failed to handle tunnel request from remote address :39390: response 400: <http://nodes.management.cattle.io|nodes.management.cattle.io> "c-x8lmc/m-8e45b2978781" not found
nobody? my whole cluster is inaccessible and can't even back my stuff up
p
i got same message on cattle-node-agent as well when upgrade rancher to 2.6.12. Wed, May 17 2023 11558 pm time=“2023-05-17T011558Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 11758 pm time=“2023-05-17T011758Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 11959 pm time=“2023-05-17T011959Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 12159 pm time=“2023-05-17T012159Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 12359 pm time=“2023-05-17T012359Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 12559 pm time=“2023-05-17T012559Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 12759 pm time=“2023-05-17T012759Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 12959 pm time=“2023-05-17T012959Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 13200 pm time=“2023-05-17T013200Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 13400 pm time=“2023-05-17T013400Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 13600 pm time=“2023-05-17T013600Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 13800 pm time=“2023-05-17T013800Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 14000 pm time=“2023-05-17T014000Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 14201 pm time=“2023-05-17T014201Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 14401 pm time=“2023-05-17T014401Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 14601 pm time=“2023-05-17T014601Z” level=info msg=“Requesting kubelet certificate regeneration”
i killed the pod and the logs in new pods look good
the last two logs time=“2023-05-17T014846Z” level=info msg=“Requesting kubelet certificate regeneration” Wed, May 17 2023 14846 pm time=“2023-05-17T014846Z” level=info msg=“Starting plan monitor, checking every 120 seconds”
a
did your node get registered to rancher?
mine are stuck at "Registering with Kubernetes"
p
yes. the new nodes registered in rancher; i checked the rancher logs, there is no msg msg=“Requesting kubelet certificate regeneration
a
oh. mine never get registered, no matter what I do
p
actually the msg=“Requesting kubelet certificate regeneration” still keep happening
time=“2023-05-17T014717Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T014917Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T015118Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T015318Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T015518Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T015718Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T015918Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T020118Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T020318Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T020518Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T020718Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T020918Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T021118Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T021318Z” level=info msg=“Requesting kubelet certificate regeneration” time=“2023-05-17T021518Z” level=info msg=“Requesting kubelet certificate regeneration”
a
I tried upgrading to 2.7.5-rc1
I suppose that one has the fix
but the same
p
it is a false alarm
a
I mean, my nodes are still trying to register 2 hours laters, clearly there is something wrong with my installation
p
yes, but will be different cause. nothing to do with “they get stuck at 
Requesting kubelet certificate regeneration"
Copy code
msg="Connecting to proxy" url="<wss://REDACTED/v3/connect>"
you may have a look if REDCATED is dns resolved and reachable? if not, it mean websocket to it will fail to connect
a
yes, it can resolve that, I think that has something to do with
Copy code
2023/05/16 21:32:07 [ERROR] Failed to handle tunnel request from remote address :40636: response 400: <http://nodes.management.cattle.io|nodes.management.cattle.io> "c-x8lmc/m-308799fb7fde" not found
that IP is from one of those control plane nodes
332 Views
Open in Slack
PreviousNext
Powered by