rough-exabyte-8533
06/21/2023, 3:00 PMwonderful-terabyte-28236
06/21/2023, 7:46 PMFailed Mount
Unable to attach or mount volumes: unmounted volumes=[data-0], unattached volumes=[ready-files kube-api-access-8jhkt data-0 app-name-tmp cluster-ca broker-certs client-ca-cert kafka-metrics-and-logging]: timed out waiting for the condition
brash-cpu-62691
06/26/2023, 7:46 AMk3s[1510]: {"level":"warn","ts":"2023-06-26T04:33:34.490Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000d8a000/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
k3s[1510]: time="2023-06-26T04:33:34Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
k3s[1510]: time="2023-06-26T04:33:34Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Tried few solutions from online but nothing helped
https://github.com/etcd-io/etcd/issues/12234
https://github.com/mmumshad/kubernetes-the-hard-way/blob/master/docs/07-bootstrapping-etcd.md#configure-the-etcd-server
https://stackoverflow.com/questions/55571566/unable-to-bring-up-kubernetes-api-server
Can anyone help or tell how to bring etcd service back??
when I try to manually run etcd it starts as expected
but k3s service doesnt start because port 2379 is already in use
Also port 2379 and 2380 are open on all nodestall-lizard-48428
07/03/2023, 9:45 AM<http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>
at this point the annotation is so big that updates fail with the following error. is invalid: metadata.annotations: Too long: must have at most 262144 bytes
.
Is it possible to disable the use of this annotation for rancher?adorable-lunch-23568
07/06/2023, 2:31 AMelegant-candle-74808
07/06/2023, 1:02 PMelegant-candle-74808
07/06/2023, 1:03 PMelegant-candle-74808
07/06/2023, 1:20 PMhelpful-tailor-39774
07/08/2023, 10:34 AMancient-car-21816
07/10/2023, 7:33 PMaloof-pencil-74759
07/27/2023, 10:16 AMhallowed-breakfast-56871
07/31/2023, 7:54 PMcareful-appointment-9763
08/03/2023, 5:37 PMfaint-shampoo-17603
08/04/2023, 9:19 AMbetter-elephant-49838
08/13/2023, 12:43 PMmagnificent-king-20947
08/17/2023, 10:44 PMkubectl top nodes
shows unknown
status for 3/4 nodes. kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary
says it can't find the node, but if I run kubectl get --raw /api/v1/nodes
, it returns info on the Rancher nodes, not the worker nodes as listed in the top command. What could be going on here?magnificent-midnight-15169
08/23/2023, 4:30 PMfull-train-34126
08/24/2023, 7:59 PMambitious-lamp-19735
08/29/2023, 2:25 PMenough-jordan-10522
08/30/2023, 5:43 PMastonishing-toddler-94422
08/31/2023, 10:37 AMhandsome-jewelry-27358
09/01/2023, 6:54 AMeager-byte-23295
09/04/2023, 3:37 PMAdmission webhook "<http://rancher.cattle.io.globalrolebindings.management.cattle.io|rancher.cattle.io.globalrolebindings.management.cattle.io>" denied the request: user "system:serviceaccount:cattle-system:rancher-infra" (groups=["system:serviceaccounts" "system:serviceaccounts:cattle-system" "system:authenticated"]) is attempting to grant RBAC permissions not currently held: {APIGroups:[""], Resources:["secrets"], Verbs:["create"]} {APIGroups:["<http://catalog.cattle.io|catalog.cattle.io>"], Resources:["clusterrepos"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["catalogs"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["cisbenchmarkversions"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["cisconfigs"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["clusters"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["clustertemplaterevisions"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["features"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["fleetworkspaces"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["globaldnses"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["globaldnsproviders"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["kontainerdrivers"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["multiclusterapps"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["nodedrivers"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["nodetemplates"], Verbs:["create"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["podsecurityadmissionconfigurationtemplates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["podsecuritypolicytemplates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["preferences"], Verbs:["*"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["principals"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rancherusernotifications"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rkeaddons"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rkek8sserviceoptions"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["rkek8ssystemimages"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["roletemplates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["settings"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["templates"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://management.cattle.io|management.cattle.io>"], Resources:["templateversions"], Verbs:["get" "list" "watch"]} {APIGroups:["<http://project.cattle.io|project.cattle.io>"], Resources:["sourcecodecredentials"], Verbs:["*"]} {APIGroups:["<http://project.cattle.io|project.cattle.io>"], Resources:["sourcecoderepositories"], Verbs:["*"]} {APIGroups:["<http://provisioning.cattle.io|provisioning.cattle.io>"], Resources:["clusters"], Verbs:["create"]} {APIGroups:["<http://rke-machine-config.cattle.io|rke-machine-config.cattle.io>"], Resources:["*"], Verbs:["create"]}; resolution errors: [[<http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> "cluster-admin" not found, <http://clusterroles.rbac.authorization.k8s.io|clusterroles.rbac.authorization.k8s.io> "system:discovery" not found]]
I can both of the cluster-admin
and the system:discoveryy
roles in the target cluster.
So what's going on why can't we connect?
Technical info:
• Rancher is installed on AKS cluster version 1.26
• I tried to connect the following types of clusters:
◦ AKS cluster with no RBAC
◦ AKS cluster with RBAC
◦ Linode cluster with RBACwitty-fall-54583
09/05/2023, 9:28 PM<CHART_REPO>
witty-fall-54583
09/05/2023, 9:28 PM--set hostname=master1.ods.vuw.ac.nz--set bootstrapPassword=xxxxSo this failed.
witty-fall-54583
09/05/2023, 9:28 PMancient-florist-59155
09/06/2023, 8:01 PMlittle-autumn-44758
09/14/2023, 4:41 PMworried-whale-5883
09/19/2023, 7:52 AMnice-ambulance-66339
09/20/2023, 6:09 AM