This message was deleted Rancher Users #general

Join Slack

This message was deleted.

# general

adamant-kite-43734

09/19/2023, 12:35 PM

This message was deleted.

agreeable-oil-87482

09/19/2023, 3:00 PM

Can you post the full log? What's the location of your client / rancher server / downstream cluster?

bumpy-queen-38421

09/19/2023, 8:04 PM

i used rancher 2.6.5 and i got cluster in k8s 1.20.15. I downloaded kubeconfig by rancher UI and i copy ./kube/config

bumpy-queen-38421

09/19/2023, 8:10 PM

Copy code

I0919 20:06:25.934619    8845 loader.go:379] Config loaded from file:  kubeconfig-test
I0919 20:06:26.002349    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/admissionregistration.k8s.io/v1?timeout=32s> 200 OK in 65 milliseconds
I0919 20:06:26.002490    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/apiregistration.k8s.io/v1beta1?timeout=32s> 200 OK in 66 milliseconds
I0919 20:06:26.002635    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/management.cattle.io/v3?timeout=32s> 200 OK in 66 milliseconds
I0919 20:06:26.002644    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/batch/v1beta1?timeout=32s> 200 OK in 66 milliseconds
I0919 20:06:26.006741    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/networking.k8s.io/v1?timeout=32s> 200 OK in 71 milliseconds
I0919 20:06:26.008525    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/objectbucket.io/v1alpha1?timeout=32s> 200 OK in 71 milliseconds
I0919 20:06:26.008661    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/openebs.io/v1alpha1?timeout=32s> 200 OK in 72 milliseconds
I0919 20:06:26.010863    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/extensions/v1beta1?timeout=32s> 200 OK in 74 milliseconds
I0919 20:06:26.012138    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/catalog.kubedb.com/v1alpha1?timeout=32s> 200 OK in 76 milliseconds
I0919 20:06:26.012843    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/apiregistration.k8s.io/v1?timeout=32s> 200 OK in 76 milliseconds
I0919 20:06:26.014240    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/crd.projectcalico.org/v1?timeout=32s> 200 OK in 77 milliseconds
I0919 20:06:26.014711    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/catalog.cattle.io/v1?timeout=32s> 200 OK in 79 milliseconds
I0919 20:06:57.937549    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/api/v1?timeout=32s>  in 32001 milliseconds
I0919 20:06:57.937633    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/discovery.k8s.io/v1beta1?timeout=32s>  in 32001 milliseconds
I0919 20:06:57.937761    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/api/v1?timeout=32s>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I0919 20:06:57.937789    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/discovery.k8s.io/v1beta1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:06:57.938003    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>  in 32001 milliseconds
I0919 20:06:57.938147    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
I0919 20:06:57.938302    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/batch/v1?timeout=32s>  in 32002 milliseconds
I0919 20:06:57.938426    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/batch/v1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:06:57.938575    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/rook.io/v1alpha2?timeout=32s>  in 32002 milliseconds
I0919 20:06:57.938664    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/rook.io/v1alpha2?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:06:57.938830    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/snapshot.storage.k8s.io/v1alpha1?timeout=32s>  in 32002 milliseconds
I0919 20:06:57.938919    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/snapshot.storage.k8s.io/v1alpha1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:06:57.982514    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/snapshot.storage.k8s.io/v1alpha1?timeout=32s> 200 OK in 40 milliseconds
I0919 20:06:57.982591    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/batch/v1?timeout=32s> 200 OK in 40 milliseconds
I0919 20:06:57.982594    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/rook.io/v1alpha2?timeout=32s> 200 OK in 40 milliseconds
I0919 20:06:57.983521    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/api/v1?timeout=32s> 200 OK in 41 milliseconds
I0919 20:07:29.942289    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>  in 32000 milliseconds
I0919 20:07:29.942382    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/discovery.k8s.io/v1beta1?timeout=32s>  in 32000 milliseconds
I0919 20:07:29.942428    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:07:29.942490    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/discovery.k8s.io/v1beta1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:07:29.942588    8845 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: <http://discovery.k8s.io/v1beta1|discovery.k8s.io/v1beta1>: Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/discovery.k8s.io/v1beta1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers), <http://monitoring.coreos.com/v1|monitoring.coreos.com/v1>: Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:07:30.001897    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/discovery.k8s.io/v1beta1?timeout=32s> 200 OK in 57 milliseconds
I0919 20:08:01.945624    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>  in 32000 milliseconds
I0919 20:08:01.945781    8845 cached_discovery.go:78] skipped caching discovery info due to Get "<https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/apis/monitoring.coreos.com/v1?timeout=32s>": net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0919 20:08:02.012130    8845 round_trippers.go:445] GET <https://xxxxxxxxxxxxxxxxxx/k8s/clusters/c-4t67g/api/v1/nodes?limit=500> 200 OK in 61 milliseconds

bumpy-queen-38421

09/19/2023, 8:48 PM

never show cluster metrics

agreeable-oil-87482

09/20/2023, 6:47 AM

What's the connectivity like between rancher and these clusters? What's the state of the cluster agent in these clusters (ie a lot of restarts?)

bumpy-queen-38421

09/20/2023, 7:18 AM

the connectivity between rancher and all clusters is ok

bumpy-queen-38421

09/21/2023, 12:56 PM

kubectl api-resources

error: unable to retrieve the complete list of server APIs: <http://apiextensions.k8s.io/v1beta1|apiextensions.k8s.io/v1beta1>:

bumpy-queen-38421

09/21/2023, 12:57 PM

i got this, do you know @agreeable-oil-87482?

agreeable-oil-87482

09/21/2023, 1:25 PM

It generally sounds like you have intermittent connectivity issues between rancher and the downstream cluster. Have you enabled authorised endpoint on these clusters? If so, have you tried testing by hitting the downstream k8s API directly?

bumpy-queen-38421

09/22/2023, 12:47 PM

@agreeable-oil-87482 i restarted docker in my cluster local rancher and now its working in my cluster cls-01

Open in Slack

Previous Next