10/17/2022, 12:12 PM
Hi! I have a question regarding node registration for k3s clusters. I am running k3s v1.22.15-k3s1 (started using k3d v5.3.0) in combination with [KubeEdge]( So there are nodes created that aren't managed via k3d and are "unknown" to k3s. Joining nodes work and rolling out workload as well. However, the kubectl exec and logs feature don't work, because it says:
Error from server: Get "<>": x509: cannot validate certificate for because it doesn't contain any IP SANs
Now, my question: is it somehow possible to accept nodes blindly without validating their certificates? One thing to mention is that this is a local only dev cluster.


10/17/2022, 12:49 PM
Sounds like a very bad idea to blindly accept nodes .. basically a glaring security issue to do so.
Reading e.g. this > I get the impression that you should try with one K3s control-plane/master node and then join the kube-edge workers by following that guide.
And see if that’s possible.
Where that brings you.


10/18/2022, 6:23 AM
Hi! I'm quite familiar with the process of joining kubeedge nodes. It brings me to the point where the edge node is present and accepting workload just like standard nodes in the cluster. There you see my Raspberry PI edge node called
$ kubectl get nodes -o wide                                                                                
NAME                             STATUS   ROLES                  AGE   VERSION                                            INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
k3d-edgefarm-core-dev-agent-1    Ready    <none>                 17h   v1.22.15+k3s1                                <none>        K3s dev              5.17.0-1016-oem     <containerd://1.5.13-k3s1>
k3d-edgefarm-core-dev-server-0   Ready    control-plane,master   17h   v1.22.15+k3s1                                <none>        K3s dev              5.17.0-1016-oem     <containerd://1.5.13-k3s1>
k3d-edgefarm-core-dev-agent-0    Ready    <none>                 17h   v1.22.15+k3s1                                <none>        K3s dev              5.17.0-1016-oem     <containerd://1.5.13-k3s1>
clownfish                        Ready    agent,edge             10h   v1.22.6-kubeedge-v1.11.1-12+0d66acb85546eb-dirty   <none>        Ubuntu 22.04.1 LTS   5.15.0-1012-raspi   <docker://20.10.18>
Here are my pods running on my edge node.
$ kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=clownfish
NAMESPACE   NAME                                    READY   STATUS    RESTARTS   AGE     IP              NODE        NOMINATED NODE   READINESS GATES
kubeedge    edgemesh-agent-lcttg                    1/1     Running   0          3m29s   clownfish   <none>           <none>
nodegroup   example-app-clownfish-7c4f5654c-x5cqb   2/2     Running   0          60s      clownfish   <none>           <none>
kubectl exec and logs calls try to directly talk to the node to execute the request there. This cannot work for edge devices, because they are very likely located in a completely different network, or may be connected via a LTE modem. Thus, these
calls from kubectl are forwarded using iptables to the cloudcore service running on a cloud node. So, whats happening is that the request targets to`` but gets redirected to the control plane nodes (via the iptables-manager pod).
kubectl get pods -o wide -n kubeedge 
Found existing alias for "kubectl get pods". You should use: "kgp"
NAME                                           READY   STATUS    RESTARTS      AGE     IP              NODE                             NOMINATED NODE   READINESS GATES
edgemesh-agent-rhnzj                           1/1     Running   1 (11h ago)   17h    k3d-edgefarm-core-dev-server-0   <none>           <none>
kubeedge-controller-manager-85b5d8f6fb-q94x8   1/1     Running   1 (11h ago)   17h      k3d-edgefarm-core-dev-agent-1    <none>           <none>
edgemesh-agent-56t9b                           1/1     Running   2 (11h ago)   17h    k3d-edgefarm-core-dev-agent-1    <none>           <none>
edgemesh-agent-t5b4v                           1/1     Running   1 (11h ago)   17h    k3d-edgefarm-core-dev-agent-0    <none>           <none>
iptables-manager-x6bqt                         1/1     Running   0             10h    k3d-edgefarm-core-dev-server-0   <none>           <none>
cloudcore-59bbfcf5c7-lmxtx                     2/2     Running   0             10h    k3d-edgefarm-core-dev-agent-0    <none>           <none>
edgemesh-agent-lcttg                           1/1     Running   0             9m43s   clownfish                        <none>           <none>
See the iptables here that placed on the control-plane node.
$ kubectl exec -it iptables-manager-x6bqt sh
/ # iptables -t nat -L TUNNEL-PORT
Chain TUNNEL-PORT (2 references)
target     prot opt source               destination         
DNAT       tcp  --  anywhere             anywhere             tcp dpt:10353 to:
DNAT       tcp  --  anywhere             anywhere             tcp dpt:10351 to:
DNAT       tcp  --  anywhere             anywhere             tcp dpt:10352 to:
So, when i exec or log to some pod that is running on the edge node, it calls it with
. The request gets redirected by the iptables rule to the node cloudcore is running (
) to the port 10003 where some cloudcore port is opened. Cloudcore acts as a proxy for API calls to the edge node. Now, the problem is that the kubectl request sent to the API Server still has the
in it. The cloudcore server certificate the request gets redirected to, does not have the edge node's IP address in its
IP Sans
. Therefore the request gets rejected. I added
to the IP Sans of the cloudcore server certificate just to verify that this is the point of failure. When i retrieve logs from a edge node's pod i get the following error:
$ kubectl logs edgemesh-agent-lcttg
Error from server: Get "<>": x509: certificate is valid for, not
So, this is definitly the issue.
kubectl logs
allow the feature
--insecure-skip-tls-verify-backend=false: Skip verifying the identity of the kubelet that logs are requested from.
In theory, an attacker could provide invalid log content back. You might want to use this if your kubelet serving
certificates have expired
So the same logs request with the insecure-skip-tls-verify-backend enabled allows me to see the pods logs
$ kubectl logs edgemesh-agent-lcttg --insecure-skip-tls-verify-backend
I1018 05:56:26.085654       1 server.go:55] Version: v1.12.0-dirty
I1018 05:56:26.085891       1 server.go:89] [1] Prepare agent to run
I1018 05:56:26.088853       1 netif.go:96] bridge device edgemesh0 already exists
I1018 05:56:26.089349       1 server.go:93] edgemesh-agent running on EdgeMode
I could live with that fact, but the
kubectl exec
feature doesn't allow the
flag or anything that does the same. I hope you can understand my problem. If not, please let me know and i try to elaborate a little bit deeper. The only possible solution i might think of right now Predefine a range of IP addresses my edge node can get in my local dev environment.