https://rancher.com/ logo
Title
f

freezing-wolf-83208

08/20/2022, 7:21 AM
trying to upgrade kubernetes on my local cluster - I first downloaded rke 1.3.12, we were using 1.2.2 before. and then on running the rke up command I run into WARN[0196] [etcd] host [161.211.xx.xx] failed to check etcd health: failed to get /health for host [161.211.xx.xx]: Get "https://161.211.xx.xx:2379/health": remote error: tls: bad certificate FATA[0196] [etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts [161.211.xx.xx] failed to report healthy. Check etcd container logs on each host for more information anyone run into the same?
we are currently in k8s 1.19.3 and we like to go to 1.23.7
f

fast-piano-59234

08/27/2022, 12:36 PM
Please always share full logs, but usually this means a missing
cluster.rkestate
file. After you backed up your local files you can retrieve it using
rke util
and compare it with the one you have.
f

freezing-wolf-83208

09/02/2022, 12:14 PM
@fast-piano-59234 I removed the certs and rke up generated new certs but the rancher instances did not come up. I see the error 0902 05:07:27.435772 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.435816 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.470105 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.470153 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.868705 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.868769 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1 I suspect this might be due to cert-manager used, our local cluster still runs 1.0.4. I saw a forum where it was mentioned k8s 1.22 requires a min of cert-manager 1.5
the complete logs are -sh-4.2$ kubectl -n cattle-system logs -l app=rancher W0902 05:07:27.435772 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.435816 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.470105 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.470153 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.868705 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.868769 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:28.145523 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cert-manager.io/v1/issuers?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:28.145574 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cert-manager.io/v1/issuers?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:28.613687 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/events.k8s.io/v1/events?resourceVersion=910043819&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:28.613737 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/events.k8s.io/v1/events?resourceVersion=910043819&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.131451 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/projectalerts?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.131483 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/projectalerts?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.656405 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.656441 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:27.756945 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/certmanager.k8s.io/v1alpha1/issuers?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:27.757022 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/certmanager.k8s.io/v1alpha1/issuers?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:28.474378 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/apps/v1/replicasets?resourceVersion=906146727&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:28.474485 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/apps/v1/replicasets?resourceVersion=906146727&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") W0902 05:07:28.627930 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/multiclusterapps?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") E0902 05:07:28.627966 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/multiclusterapps?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca") 2022/09/02 05:07:08 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:10 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:12 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:14 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:16 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:18 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:20 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:22 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:24 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials 2022/09/02 05:07:26 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials -sh-4.2$
@fast-piano-59234 actually you were right - it was the rkestate file. though I still would have had to upgrade cert-manager. Thanks!!
👍 1