Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
f
freezing-wolf-83208
08/20/2022, 7:21 AMtrying to upgrade kubernetes on my local cluster - I first downloaded rke 1.3.12, we were using 1.2.2 before. and then on running the rke up command I run into
WARN[0196] [etcd] host [161.211.xx.xx] failed to check etcd health: failed to get /health for host [161.211.xx.xx]: Get "https://161.211.xx.xx:2379/health": remote error: tls: bad certificate
FATA[0196] [etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts [161.211.xx.xx] failed to report healthy. Check etcd container logs on each host for more information
anyone run into the same?
we are currently in k8s 1.19.3 and we like to go to 1.23.7
f
fast-piano-59234
08/27/2022, 12:36 PMPlease always share full logs, but usually this means a missing
cluster.rkestaterke utilf
freezing-wolf-83208
09/02/2022, 12:14 PM@fast-piano-59234 I removed the certs and rke up generated new certs but the rancher instances did not come up. I see the error
0902 05:07:27.435772 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.435816 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.470105 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.470153 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.868705 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.868769 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1
I suspect this might be due to cert-manager used, our local cluster still runs 1.0.4. I saw a forum where it was mentioned k8s 1.22 requires a min of cert-manager 1.5
the complete logs are
-sh-4.2$ kubectl -n cattle-system logs -l app=rancher
W0902 05:07:27.435772 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.435816 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/rke.cattle.io/v1/rkebootstraptemplates?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.470105 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.470153 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cluster.x-k8s.io/v1beta1/clusters?resourceVersion=906146238&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.868705 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.868769 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/node.k8s.io/v1beta1/runtimeclasses?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:28.145523 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cert-manager.io/v1/issuers?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:28.145574 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/cert-manager.io/v1/issuers?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:28.613687 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/events.k8s.io/v1/events?resourceVersion=910043819&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:28.613737 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/events.k8s.io/v1/events?resourceVersion=910043819&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.131451 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/projectalerts?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.131483 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/projectalerts?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.656405 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.656441 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:27.756945 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/certmanager.k8s.io/v1alpha1/issuers?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:27.757022 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/certmanager.k8s.io/v1alpha1/issuers?resourceVersion=906146214&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:28.474378 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/apps/v1/replicasets?resourceVersion=906146727&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:28.474485 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/apps/v1/replicasets?resourceVersion=906146727&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
W0902 05:07:28.627930 34 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/multiclusterapps?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
E0902 05:07:28.627966 34 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.23.3-rancher2/tools/cache/reflector.go:168: Failed to watch *summary.SummarizedObject: failed to list *summary.SummarizedObject: Get "https://10.43.0.1:443/apis/management.cattle.io/v3/multiclusterapps?resourceVersion=906146237&timeout=15m0s": x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kube-ca")
2022/09/02 05:07:08 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:10 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:12 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:14 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:16 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:18 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:20 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:22 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:24 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
2022/09/02 05:07:26 [INFO] Waiting for server to become available: the server has asked for the client to provide credentials
-sh-4.2$
@fast-piano-59234 actually you were right - it was the rkestate file. though I still would have had to upgrade cert-manager. Thanks!!
👍 1