Hey just upgrade to 2 6 8 and rancher is not coming up anymo rancher-users #general

Title

# general

many-area-51777

08/31/2022, 12:19 PM

Hey, just upgrade to 2.6.8 and rancher is not coming up anymore. First I had to extend the liveness settings as mentioned in https://github.com/rancher/rancher/issues/38177 but even though the logs show that something is happening, at the end the pod dies and tries to restart.

~~Started seeing these after 2.6.8 upgrade:~~

~~nauthorized~~

~~W0903 221123.709526 32 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Unauthorized~~

E0903 221123.709553 32 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Unauthorized

~~W0903 221123.726334 32 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Unauthorized~~

E0903 221123.726360 32 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Unauthorized

~~2022/09/03 221124 [ERROR] failed to call leader func: failed to add management data: Unauthorized~~

~~2022/09/03 221124 [ERROR] failed to call leader func: Unauthorized~~

~~W0903 221124.358226 32 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Unauthorized~~

E0903 221124.358258 32 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Unauthorized

~~I0903 221124.474577 32 leaderelection.go:283] failed to renew lease kube-system/cattle-controllers: timed out waiting for the condition~~

~~E0903 221124.474647 32 leaderelection.go:306] Failed to release lock: resource name may not be empty~~

~~2022/09/03 221124 [FATAL] leaderelection lost for cattle-controllers~~

~~Any known solution for that? I can't start rancher pod anymore.~~

Started seeing these after 2.6.8 upgrade:

nauthorized

W0903 221123.709526 32 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Unauthorized

E0903 221123.709553 32 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Unauthorized

W0903 221123.726334 32 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Unauthorized

E0903 221123.726360 32 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Unauthorized

2022/09/03 221124 [ERROR] failed to call leader func: failed to add management data: Unauthorized

2022/09/03 221124 [ERROR] failed to call leader func: Unauthorized

W0903 221124.358226 32 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Unauthorized

E0903 221124.358258 32 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Unauthorized

I0903 221124.474577 32 leaderelection.go:283] failed to renew lease kube-system/cattle-controllers: timed out waiting for the condition

E0903 221124.474647 32 leaderelection.go:306] Failed to release lock: resource name may not be empty

2022/09/03 221124 [FATAL] leaderelection lost for cattle-controllers

Any known solution for that? I can't start rancher pod anymore.

22 Views