https://rancher.com/ logo
#rke
Title
# rke
a

adamant-kite-43734

08/20/2022, 7:21 AM
This message was deleted.
f

freezing-wolf-83208

08/20/2022, 7:51 AM
we are currently in k8s 1.19.3 and we like to go to 1.23.7
f

fast-piano-59234

08/27/2022, 12:36 PM
Please always share full logs, but usually this means a missing
cluster.rkestate
file. After you backed up your local files you can retrieve it using
rke util
and compare it with the one you have.
f

freezing-wolf-83208

09/02/2022, 12:14 PM
@fast-piano-59234 I removed the certs and rke up generated new certs but the rancher instances did not come up. I see the error 0902 050727.435772 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.435816 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.470105 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.470153 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.868705 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.868769 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1 I suspect this might be due to cert-manager used, our local cluster still runs 1.0.4. I saw a forum where it was mentioned k8s 1.22 requires a min of cert-manager 1.5
the complete logs are -sh-4.2$ kubectl -n cattle-system logs -l app=rancher W0902 050727.435772 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.435816 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.470105 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.470153 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.868705 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.868769 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050728.145523 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cert-manager.io/v1/issuers?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050728.145574 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cert-manager.io/v1/issuers?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050728.613687 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/events.k8s.io/v1/events?resourceVersion=910043819&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050728.613737 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/events.k8s.io/v1/events?resourceVersion=910043819&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.131451 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/projectalerts?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.131483 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/projectalerts?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.656405 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.656441 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050727.756945 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/certmanager.k8s.io/v1alpha1/issuers?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050727.757022 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/certmanager.k8s.io/v1alpha1/issuers?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050728.474378 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/apps/v1/replicasets?resourceVersion=906146727&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050728.474485 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/apps/v1/replicasets?resourceVersion=906146727&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 050728.627930 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/multiclusterapps?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 050728.627966 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go168 Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/multiclusterapps?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") 2022/09/02 050708 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050710 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050712 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050714 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050716 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050718 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050720 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050722 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050724 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 050726 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials -sh-4.2$
@fast-piano-59234 actually you were right - it was the rkestate file. though I still would have had to upgrade cert-manager. Thanks!!
👍 1
153 Views