cuddly-egg-57762
10/14/2022, 10:45 AMott 14 10:24:13 node1 k3s[23642]: E1014 10:24:13.249106 23642 leaderelection.go:330] error retrieving resource lock kube-system/cloud-controller-manager: Get "<https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cloud-controller-manager?timeou>>
ott 14 10:24:13 node1 k3s[23642]: I1014 10:24:13.249252 23642 leaderelection.go:283] failed to renew lease kube-system/cloud-controller-manager: timed out waiting for the condition
ott 14 10:24:13 node1 k3s[23642]: F1014 10:24:13.249332 23642 controllermanager.go:237] leaderelection lost
ott 14 10:24:13 node1 k3s[23642]: E1014 10:24:13.249629 23642 runtime.go:76] Observed a panic: F1014 10:24:13.249332 23642 controllermanager.go:237] leaderelection lost
ott 14 10:24:13 node1 k3s[23642]: goroutine 22078 [running]:
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3f70c00|k8s.io/apimachinery/pkg/util/runtime.logPanic({0x3f70c00>, 0xc0230f6340})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/apimachinery@v1.23.6-k3s1/pkg/util/runtime/runtime.go:74 +0x7d
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0|k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0>, 0x0, 0x1})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/apimachinery@v1.23.6-k3s1/pkg/util/runtime/runtime.go:48 +0x75
ott 14 10:24:13 node1 k3s[23642]: panic({0x3f70c00, 0xc0230f6340})
ott 14 10:24:13 node1 k3s[23642]: /usr/local/go/src/runtime/panic.go:1038 +0x215
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/klog/v2.(*loggingT).output(0x7c06200|k8s.io/klog/v2.(*loggingT).output(0x7c06200>, 0x3, 0x0, 0xc02be58700, 0x0, {0x6018774, 0x0}, 0xc00eabd1c0, 0x0)
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/klog/v2@v2.30.0-k3s1/klog.go:982 +0x625
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/klog/v2.(*loggingT).printf(0x0|k8s.io/klog/v2.(*loggingT).printf(0x0>, 0x0, 0x0, {0x0, 0x0}, {0x4811943, 0x13}, {0x0, 0x0, 0x0})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/klog/v2@v2.30.0-k3s1/klog.go:753 +0x1c5
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/klog/v2.Fatalf(...)|k8s.io/klog/v2.Fatalf(...)>
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/klog/v2@v2.30.0-k3s1/klog.go:1513
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/cloud-provider/app.Run.func3()|k8s.io/cloud-provider/app.Run.func3()>
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/cloud-provider@v1.23.6-k3s1/app/controllermanager.go:237 +0x55
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1()|k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run.func1()>
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/client-go@v1.23.6-k3s1/tools/leaderelection/leaderelection.go:203 +0x1f
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc00991f440|k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0xc00991f440>, {0x51d17b0, 0xc00012a008})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/client-go@v1.23.6-k3s1/tools/leaderelection/leaderelection.go:213 +0x189
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/client-go/tools/leaderelection.RunOrDie({0x51d17b0|k8s.io/client-go/tools/leaderelection.RunOrDie({0x51d17b0>, 0xc00012a008}, {{0x5223e48, 0xc00eba1cc0}, 0x37e11d600, 0x2540be400, 0x77359400, {0xc0076699e0, 0x4bacd10, 0x0}, ...})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/client-go@v1.23.6-k3s1/tools/leaderelection/leaderelection.go:226 +0x94
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/cloud-provider/app.leaderElectAndRun(0xc0072e29a8|k8s.io/cloud-provider/app.leaderElectAndRun(0xc0072e29a8>, {0xc00c4fd410, 0x2a}, 0xc007319308, {0x47dabc2, 0x6}, {0x4829eef, 0x18}, {0xc0076699e0, 0x4bacd10, ...})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/cloud-provider@v1.23.6-k3s1/app/controllermanager.go:516 +0x2c5
ott 14 10:24:13 node1 k3s[23642]: created by <http://k8s.io/cloud-provider/app.Run|k8s.io/cloud-provider/app.Run>
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/cloud-provider@v1.23.6-k3s1/app/controllermanager.go:220 +0x7a5
ott 14 10:24:13 node1 k3s[23642]: panic: unreachable
ott 14 10:24:13 node1 k3s[23642]: goroutine 22078 [running]:
ott 14 10:24:13 node1 k3s[23642]: <http://k8s.io/cloud-provider/app.leaderElectAndRun(0xc0072e29a8|k8s.io/cloud-provider/app.leaderElectAndRun(0xc0072e29a8>, {0xc00c4fd410, 0x2a}, 0xc007319308, {0x47dabc2, 0x6}, {0x4829eef, 0x18}, {0xc0076699e0, 0x4bacd10, ...})
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/cloud-provider@v1.23.6-k3s1/app/controllermanager.go:526 +0x2d8
ott 14 10:24:13 node1 k3s[23642]: created by <http://k8s.io/cloud-provider/app.Run|k8s.io/cloud-provider/app.Run>
ott 14 10:24:13 node1 k3s[23642]: /go/pkg/mod/github.com/k3s-io/kubernetes/staging/src/k8s.io/cloud-provider@v1.23.6-k3s1/app/controllermanager.go:220 +0x7a5
ott 14 10:24:17 node1 systemd[1]: k3s.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Just for curiosity I check also the kernel messages and I saw this strange stack related to eBPF:
[ 6381.435296] WARNING: CPU: 3 PID: 66903 at mm/vmalloc.c:330 vmalloc_to_page+0x219/0x220
[ 6381.436034] Modules linked in: tcp_diag inet_diag fuse sunrpc cls_bpf sch_ingress xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_nat xt_set xt_CT veth ip_set_hash_ip xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_filter ip6table_raw ip6table_mangle ip6_tables iptable_filter iptable_raw iptable_mangle ip6t_MASQUERADE ipt_MASQUERADE xt_conntrack xt_comment nft_counter xt_mark nft_compat iptable_nat ip_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_snapshot dm_bufio br_netfilter bridge stp llc overlay nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink bochs_drm drm_vram_helper drm_ttm_helper ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm joydev pcspkr virtio_balloon i2c_piix4 xfs libcrc32c sr_mod cdrom sd_mod t10_pi sg ata_generic ata_piix libata virtio_net serio_raw net_failover virtio_console failover virtio_scsi dm_mirror dm_region_hash dm_log dm_mod
[ 6381.436234] [last unloaded: nft_fib]
[ 6381.442676] Red Hat flags: eBPF/cls eBPF/cgroup
[ 6381.443442] CPU: 3 PID: 66903 Comm: tc Kdump: loaded Not tainted 4.18.0-372.9.1.el8.x86_64 #1
[ 6381.444220] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS <http://rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org|rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org> 04/01/2014
[ 6381.445856] RIP: 0010:vmalloc_to_page+0x219/0x220
[ 6381.446707] Code: 50 ec 00 c3 48 81 e7 00 00 00 c0 e9 1a ff ff ff 48 8b 3d 32 bf ed 00 48 81 e7 00 f0 ff ff 48 89 fa eb 8d 0f 0b e9 15 fe ff ff <0f> 0b 31 c0 c3 66 90 0f 1f 44 00 00 e8 d6 fd ff ff 48 2b 05 ff 4f
[ 6381.448372] RSP: 0018:ffffa3a70de3bca0 EFLAGS: 00010293
[ 6381.449215] RAX: 0000000000000063 RBX: fffff440c41877c0 RCX: 0000000000000000
[ 6381.450091] RDX: 0000000000000000 RSI: ffffffffc0c008bb RDI: 0000000000000000
[ 6381.450970] RBP: ffffffffc0bff8bb R08: 0000000000000000 R09: 0000000000000001
[ 6381.451854] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
[ 6381.452774] R13: ffffa3a70de3bcef R14: ffffffffc0c008bb R15: 0000000000000001
[ 6381.453647] FS: 00007f61cc987740(0000) GS:ffff96fe73d80000(0000) knlGS:0000000000000000
[ 6381.454571] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6381.455437] CR2: 000055c5f2725df0 CR3: 0000000106106000 CR4: 00000000000006e0
[ 6381.456367] Call Trace:
[ 6381.457321] __text_poke+0x203/0x260
[ 6381.458295] text_poke_bp_batch+0x67/0x160
[ 6381.459135] ? bpf_prog_c13c61c429dc020a_handle_policy+0x25c3/0x2d08
[ 6381.460008] text_poke_bp+0x3a/0x54
[ 6381.460830] ? bpf_prog_c13c61c429dc020a_handle_policy+0x25c3/0x2d08
[ 6381.461658] __bpf_arch_text_poke+0x19e/0x1b0
[ 6381.462551] prog_array_map_poke_run+0xe2/0x1d0
[ 6381.463447] bpf_fd_array_map_update_elem+0x8b/0xe0
[ 6381.464299] map_update_elem+0x1cf/0x1f0
[ 6381.465148] __do_sys_bpf+0x92b/0x1da0
[ 6381.465987] ? audit_log_exit+0x2b2/0xd60
[ 6381.466788] do_syscall_64+0x5b/0x1a0
[ 6381.467656] entry_SYSCALL_64_after_hwframe+0x65/0xca
[ 6381.468496] RIP: 0033:0x7f61ccac073d
[ 6381.469318] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 23 37 0d 00 f7 d8 64 89 01 48
[ 6381.471074] RSP: 002b:00007fff72060128 EFLAGS: 00000246 ORIG_RAX: 0000000000000141
[ 6381.471970] RAX: ffffffffffffffda RBX: 00007fff72062530 RCX: 00007f61ccac073d
[ 6381.472845] RDX: 0000000000000080 RSI: 00007fff72060130 RDI: 0000000000000002
[ 6381.473702] RBP: 0000564de994f1f0 R08: 00007fff7206020c R09: 0000000000000008
[ 6381.474554] R10: 0000000000000000 R11: 0000000000000246 R12: 0000564de9901a80
[ 6381.475530] R13: 0000564de994f700 R14: 00007fff72060260 R15: 0000564de9991938
[ 6381.476413] ---[ end trace b293578addd05c82 ]---
It's the first time after months I see this problem: do you know why this is happening? Could the problem be caused by the kernel version (4.18.0-372.9.1.el8.x86_64)?enough-carpet-20915
10/15/2022, 9:55 AMInsecureSkipVerify = true
to the built in traefik in k3s?enough-carpet-20915
10/15/2022, 10:11 AMechoing-ability-7881
10/15/2022, 11:47 AMflat-waiter-75025
10/15/2022, 12:30 PMmelodic-hamburger-23329
10/17/2022, 1:00 AMmelodic-hamburger-23329
10/17/2022, 5:43 AMresource mapping not found for name: "" namespace: "" from "kube-tracing.yaml": no matches for kind "TracingConfiguration" in version "<http://apiserver.config.k8s.io/v1alpha1|apiserver.config.k8s.io/v1alpha1>"
ensure CRDs are installed first
resource mapping not found for name: "" namespace: "" from "kube-tracing.yaml": no matches for kind "KubeletConfiguration" in version "<http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>"
ensure CRDs are installed first
kube-tracing.yaml:
apiVersion: <http://apiserver.config.k8s.io/v1alpha1|apiserver.config.k8s.io/v1alpha1>
kind: TracingConfiguration
endpoint: <...>:4317
---
apiVersion: <http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>
kind: KubeletConfiguration
featureGates:
KubeletTracing: true
tracing:
endpoint: <...>:4317
I enabled the feature gates (feature-gates=APIServerTracing=true,KubeletTracing=true
), but it seems the CRDs are missing. How do I get the CRDs..?great-monitor-14716
10/17/2022, 7:55 AMlevel=info msg="Initializing daemon" subsys=daemon
level=info msg="Establishing connection to apiserver" host="<https://127.0.0.1:6443>" subsys=k8s
level=info msg="Establishing connection to apiserver" host="<https://127.0.0.1:6443>" subsys=k8s
level=info msg="Establishing connection to apiserver" host="<https://127.0.0.1:6443>" subsys=k8s
level=info msg="Establishing connection to apiserver" host="<https://127.0.0.1:6443>" subsys=k8s
I've also tried using cilium CLI and set
cilium config set k8sServiceHost 192.168.250.11
cilium config set k8sServicePort 6443
but it keeps on trying to connect to 127.0.0.1:6443
I've disabled kube-proxy, network policy, servicelb and traefik during k3s install.
kubectl get nodes -A -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
k3s-agent1 Ready <none> 8h v1.24.6+k3s1 192.168.250.14 <none> Ubuntu 20.04.5 LTS 5.4.0-128-generic <containerd://1.6.8-k3s1>
k3s-server1 Ready control-plane,etcd,master 9h v1.24.6+k3s1 192.168.250.11 <none> Ubuntu 20.04.5 LTS 5.4.0-128-generic <containerd://1.6.8-k3s1>
k3s-server2 Ready control-plane,etcd,master 9h v1.24.6+k3s1 192.168.250.12 <none> Ubuntu 20.04.5 LTS 5.4.0-128-generic <containerd://1.6.8-k3s1>
k3s-server3 Ready control-plane,etcd,master 8h v1.24.6+k3s1 192.168.250.13 <none> Ubuntu 20.04.5 LTS 5.4.0-128-generic <containerd://1.6.8-k3s1>
root@k3s-server1:~# cilium status
/¯¯\
/¯¯\__/¯¯\ Cilium: 2 errors
\__/¯¯\__/ Operator: OK
/¯¯\__/¯¯\ Hubble: disabled
\__/¯¯\__/ ClusterMesh: disabled
\__/
DaemonSet cilium Desired: 4, Ready: 3/4, Available: 3/4, Unavailable: 1/4
Deployment cilium-operator Desired: 1, Ready: 1/1, Available: 1/1
Containers: cilium Running: 4
cilium-operator Running: 1
Cluster Pods: 3/3 managed by Cilium
Image versions cilium <http://quay.io/cilium/cilium:v1.12.2@sha256:986f8b04cfdb35cf714701e58e35da0ee63da2b8a048ab596ccb49de58d5ba36|quay.io/cilium/cilium:v1.12.2@sha256:986f8b04cfdb35cf714701e58e35da0ee63da2b8a048ab596ccb49de58d5ba36>: 4
cilium-operator <http://quay.io/cilium/operator-generic:v1.12.2@sha256:00508f78dae5412161fa40ee30069c2802aef20f7bdd20e91423103ba8c0df6e|quay.io/cilium/operator-generic:v1.12.2@sha256:00508f78dae5412161fa40ee30069c2802aef20f7bdd20e91423103ba8c0df6e>: 1
Errors: cilium cilium 1 pods of DaemonSet cilium are not ready
cilium cilium-4spdt unable to retrieve cilium status: container cilium-agent is in CrashLoopBackOff, exited with code 1: level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get \"<https://127.0.0.1:6443/api/v1/namespaces/kube-system>\": dial tcp 127.0.0.1:6443: connect: connection refused" subsys=daemon
root@k3s-server1:~# kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system cilium-4spdt 0/1 CrashLoopBackOff 99 (2m25s ago) 7h15m 192.168.250.14 k3s-agent1 <none> <none>
kube-system cilium-6mjjd 1/1 Running 1 (26m ago) 7h15m 192.168.250.13 k3s-server3 <none> <none>
kube-system cilium-9xcpk 1/1 Running 1 (26m ago) 7h15m 192.168.250.11 k3s-server1 <none> <none>
kube-system cilium-b7grx 1/1 Running 1 (25m ago) 7h15m 192.168.250.12 k3s-server2 <none> <none>
kube-system cilium-operator-7b5b55f786-4nfmd 1/1 Running 1 (25m ago) 9h 192.168.250.12 k3s-server2 <none> <none>
kube-system coredns-b96499967-96fsc 1/1 Running 1 (25m ago) 9h 10.0.1.76 k3s-server2 <none> <none>
kube-system local-path-provisioner-7b7dc8d6f5-gbxjc 1/1 Running 1 (25m ago) 9h 10.0.1.20 k3s-server2 <none> <none>
kube-system metrics-server-668d979685-j6xzf 1/1 Running 1 (25m ago) 9h 10.0.1.177 k3s-server2 <none> <none>
rapid-jelly-9995
10/17/2022, 12:12 PMError from server: Get "<https://192.168.1.100:10351/containerLogs/kubeedge/edgemesh-agent-q78bp/edgemesh-agent>": x509: cannot validate certificate for 192.168.1.100 because it doesn't contain any IP SANs
Now, my question: is it somehow possible to accept nodes blindly without validating their certificates? One thing to mention is that this is a local only dev cluster.great-monitor-14716
10/17/2022, 2:28 PMcilium install \
--kube-proxy-replacement=strict \
--helm-set ingressController.enabled=true
Getting error:
Error: Unable to install Cilium: unable to create secret kube-system/hubble-server-certs: secrets "hubble-server-certs" already exists
Should I throw it out and install cilium through helm instead? or is there a way to use the CLI to get the same results?enough-carpet-20915
10/17/2022, 5:45 PMancient-raincoat-46356
10/17/2022, 9:07 PMCOPY_CONTENTS=( "/var/lib/rancher/k3s/server/db/snapshots" "/etc/rancher/k3s/k3s.yaml" "/etc/default/k3s" )
The docs really only say to backup the ETCD snapshots but I'm finding a few other things that seem important. Does anyone have any other suggestions of things I should be including?agreeable-garden-91801
10/18/2022, 8:26 AMfaint-pillow-41543
10/18/2022, 2:26 PMfaint-pillow-41543
10/18/2022, 2:40 PMstocky-controller-51764
10/20/2022, 6:38 AMfaint-pillow-41543
10/20/2022, 1:30 PMprehistoric-judge-25958
10/20/2022, 8:06 PM--feature-gates=StatefulSetAutoDeletePVC=true
tall-belgium-91376
10/21/2022, 12:45 PMinvalid bearer token, Token has been invalidated
all over the master’s logs. Do I need to do something on agent side other than restarting the k3s-agent
?clever-air-65544
10/21/2022, 1:59 PMcareful-piano-35019
10/21/2022, 3:55 PMmelodic-hamburger-23329
10/23/2022, 4:49 AMable-mechanic-45652
10/24/2022, 9:07 AMsquare-engine-61315
10/24/2022, 11:44 AMerror killing pod: failed to "KillPodSandbox" for "26335a48-9ee2-422c-8909-c8c8db421e85" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \"...\": failed to get network \"cbr0\" cached result: decoding version from network config: unexpected end of JSON input"I'm not sure whether this is a bug, nor where to file it if it is! I have tried restarting the node, but it makes no difference. Any ideas?
enough-carpet-20915
10/24/2022, 5:47 PMenough-carpet-20915
10/24/2022, 5:48 PMenough-carpet-20915
10/24/2022, 5:49 PMfierce-horse-50681
10/25/2022, 3:21 AMenough-carpet-20915
10/25/2022, 11:39 AM--flannel-iface
all I need?white-garden-41931
10/26/2022, 12:24 PMs/Centos/Fedora/
)