billowy-apple-60989
01/09/2023, 3:35 PM1.21.7
which uses chart version 9.18.201
by default which in turn deployed traefik 2.4.8
bland-area-30120
01/09/2023, 3:35 PMbumpy-printer-21267
01/09/2023, 4:20 PMproud-shampoo-40692
01/09/2023, 5:00 PM"The application cannot reach the general internet for updated kubernetes versions and other components, but can still operate."
My system is Windows 11 Pro.wonderful-judge-81095
01/09/2023, 9:27 PMwonderful-judge-81095
01/09/2023, 9:47 PMminiature-midnight-96801
01/10/2023, 1:17 AMblue-tomato-62407
01/10/2023, 2:59 AMdial tcp 127.0.0.1:6443: connect: connection refused
on Ubuntu 22.04.1? Just upgraded a server from 20.04 LTS and can't seem to find much to go on so far.kind-vase-40458
01/10/2023, 5:45 AM/etc/resolv.conf
which indeeds contains nameserver 127.0.0.1
. If i manually removed the "loop" from config map, the core-dns pod would start fine, but after config map changes, it would crash soon after..
• Why can't i remove the localhost loopback? The localhost is actually intended, since I want to forward to localhost where my upstream DNS process is running on the machine node.
• Question: Is there a way to distinguish between pod's localhost from node's localhost?
• What i tried: adding 127.0.0.2 localhost
in it's /etc/hosts equivalent, and then i'd forward resolution to /etc/resolv.conf
with nameserver 127.0.0.2
seems like there's no DNS resolution happeningadorable-exabyte-35533
01/10/2023, 6:11 AMbillions-lifeguard-63461
01/10/2023, 1:09 PMred-waitress-37932
01/10/2023, 2:56 PM$ kubectl apply --all --namespace mynamespace --prune -f resources.yml
secret/config created
W0110 15:15:02.543661 25197 prune.go:71] Deprecated: kubectl apply will no longer prune non-namespaced resources by default when used with the --namespace flag in a future release. To preserve the current behaviour, list the resources you want to target explicitly in the --prune-allowlist flag.
namespace/cattle-system pruned
namespace/ingress-nginx pruned
namespace/kubernetes-dashboard pruned
careful-mouse-48712
01/10/2023, 5:20 PMminiature-notebook-6405
01/10/2023, 5:40 PMorange-barista-66154
01/10/2023, 5:49 PMsilly-airline-38945
01/10/2023, 8:03 PMrancher-system-agent[7696]: time="2023-01-10T19:32:18Z" level=error msg="error syncing 'fleet-default/custom-4a5e8a9fa273-machine-plan': handler secret-watch: Unauthorized, requeuing"
The fleet agent running on the RKE2 cluster is reporting:
level=error msg="failed to report cluster node status: Unauthorized"
Is there a way of fixing this issue?bland-painting-61617
01/10/2023, 8:57 PMgorgeous-cat-50570
01/11/2023, 10:07 AMWaiting on project-precan-alert-controller_c-vfm92
Inspecting container logs :
[ERROR] error syncing 'c-vfm92/creator-cluster-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/u-mxcjeyd7a6-admin': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
[ERROR] error syncing 'c-vfm92/c-vfm92-fleet-default-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
melodic-terabyte-2284
01/11/2023, 11:55 AMcrooked-cat-21365
01/11/2023, 12:28 PMJan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> sh[2138]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> sh[2139]: /bin/sh: 1: /usr/bin/systemctl: not found
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=warning msg="not running in CIS mode"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Starting rke2 v1.24.8+rke2r1 (b061438ce1f04518d980aac842b62f92095d793d)"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Managed etcd cluster initializing"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Starting etcd for new cluster"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Tunnel server egress proxy mode: agent"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,rke2 --authorization-mode=Node,RBAC --bin>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/se>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Server node token is available at /var/lib/rancher/rke2/server/token"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="To join server node to cluster: rke2 server -s <https://172.19.96.128:9345> -t ${SERVER_NODE_TOKEN}"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Agent node token is available at /var/lib/rancher/rke2/server/agent-token"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="To join agent node to cluster: rke2 agent -s <https://172.19.96.128:9345> -t ${AGENT_NODE_TOKEN}"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or>
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Run: rke2 kubectl"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="certificate CN=<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> signed by CN=rke2-server-ca@1673437584: notBefore=2023-01-11 11:46:24 +0000 UTC notAfter=2024-01-11 12:02:46 +0000 UTC"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="certificate CN=system:node:<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de>,O=system:nodes signed by CN=rke2-client-ca@1673437584: notBefore=2023-01-11 11:46:24 +0000 UTC notAfter=2024-01-11 12:02:46 +0000 UTC"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Module overlay was already loaded"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Module br_netfilter was already loaded"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 3670016"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Runtime image <http://index.docker.io/rancher/rke2-runtime:v1.24.8-rke2r1|index.docker.io/rancher/rke2-runtime:v1.24.8-rke2r1> bin and charts directories already exist; skipping extract"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Using private registry config file at /etc/rancher/rke2/registries.yaml"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"
Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Containerd is now running"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 12.972685ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 1.097175ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 946.039<C2><B5>s"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 1.136069ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 1.041789ms"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/li>
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Connecting to proxy" url="<wss://127.0.0.1:9345/v1-rke2/connect>"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Handling backend connection request [<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de>]"
Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:02:48 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2281]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
:
:
crooked-cat-21365
01/11/2023, 12:29 PMJan 11 13:02:48 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2281]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
Jan 11 13:02:51 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:51+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:02:52 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:52+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:02:56 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:56+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:02:58 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:58+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:01 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:01+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:03 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:03+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:06 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:06+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:06 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:06+01:00" level=info msg="Container for etcd not found (no matching container found), retrying"
Jan 11 13:03:08 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:08+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:11 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:11+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:13 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:13+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: {"level":"warn","ts":"2023-01-11T13:03:16.760+0100","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005ff340/127.0.0.1:2379>","attempt":0,"error":">
Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Waiting for etcd server to become available"
Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Waiting for API server to become available"
Jan 11 13:03:18 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: {"level":"warn","ts":"2023-01-11T13:03:18.136+0100","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005ff340/127.0.0.1:2379>","attempt":0,"error":">
Jan 11 13:03:18 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:18+01:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jan 11 13:03:18 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:18+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:21 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:21+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:23 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:23+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:26 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:26+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:26 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:26+01:00" level=info msg="Container for etcd not found (no matching container found), retrying"
Jan 11 13:03:28 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:28+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:31 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:31+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Jan 11 13:03:33 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: {"level":"warn","ts":"2023-01-11T13:03:33.136+0100","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005ff340/127.0.0.1:2379>","attempt":0,"error":">
Jan 11 13:03:33 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:33+01:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jan 11 13:03:33 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:33+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jan 11 13:03:36 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:36+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Any helpful hint is highly appreciated.adventurous-thailand-75978
01/11/2023, 3:23 PMstocky-fall-82040
01/11/2023, 3:50 PMbig-jordan-45387
01/11/2023, 4:30 PMlevel=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such file or directory"
My question is, what could prevent this kube-scheduler.crt
from being in the node?creamy-pencil-82913
01/11/2023, 4:33 PMgorgeous-cat-50570
01/11/2023, 5:06 PM[ERROR] failed on subscribe prometheus: NotFound 404: the server could not find the requested resource (get <http://prometheuses.meta.k8s.io|prometheuses.meta.k8s.io>)
2023/01/11 17:04:39 [ERROR] failed on subscribe serviceMonitor: NotFound 404: the server could not find the requested resource (get <http://servicemonitors.meta.k8s.io|servicemonitors.meta.k8s.io>)
2023/01/11 17:04:39 [ERROR] failed on subscribe prometheusRule: NotFound 404: the server could not find the requested resource (get <http://prometheusrules.meta.k8s.io|prometheusrules.meta.k8s.io>)
2023/01/11 17:04:39 [ERROR] failed on subscribe alertmanager: NotFound 404: the server could not find the requested resource (get <http://alertmanagers.meta.k8s.io|alertmanagers.meta.k8s.io>)
Sometimes I make api request to rancher but I get
status code was 500 and not [201]: HTTP Error 500: Internal Server Error
Any help?delightful-jordan-42115
01/11/2023, 6:39 PMminiature-midnight-96801
01/11/2023, 11:42 PMminiature-midnight-96801
01/12/2023, 1:51 AMshy-tent-66642
01/12/2023, 6:12 AMapiVersion: <http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>
kind: KubeletConfiguration
maxPods: 250
edit /etc/systemd/system/k3s.service
to change the k3s server args:
ExecStart=/usr/local/bin/k3s \
server \
'--disable' \
'servicelb' \
'--disable' \
'traefik' \
'--kubelet-arg=config=/etc/rancher/k3s/kubelet.config'
reload systemctl to pick up the service change:
sudo systemctl daemon-reload
restart k3s:
sudo systemctl restart k3s