I have a weird issue when bootstrapping a new cluster There rancher-users #k3s

Title

rich-cartoon-70161

01/10/2023, 3:40 PM

I have a weird issue when bootstrapping a new cluster. There seems to be an issue with tls certificates. I’ll post more details in this 🧵

Launch config (Systemd):

ExecStart=/usr/local/bin/k3s \
    server \
        '--cluster-init' \
        '--cluster-cidr' \
        '10.3.0.0/16' \
        '--service-cidr' \
        '10.4.0.0/16' \
        '--cluster-dns' \
        '10.4.0.10' \
        '--disable' \
        'local-storage' \
        '--disable-cloud-controller' \
        '--disable' \
        'traefik' \
        '--disable' \
        'servicelb' \
        '--flannel-backend=wireguard-native' \
        '--kubelet-arg=cloud-provider=external' \
        '--advertise-address' \
        '10.1.1.1' \
        '--node-ip' \
        '10.1.1.1' \
        '--node-external-ip' \
        'THE_EXTERNAL_IP' \

The error loop is this:

Jan 10 15:40:59 test-control-plane-0 k3s[2248]: time="2023-01-10T15:40:59Z" level=info msg="Waiting for control-plane node agent startup"
Jan 10 15:41:00 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:00Z" level=info msg="Waiting for control-plane node agent startup"
Jan 10 15:41:01 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:01Z" level=error msg="dynamiclistener [::]:6443: failed to update cert with connection local address: EOF"
Jan 10 15:41:01 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:01Z" level=error msg="dynamiclistener [::]:6443: failed to update cert with connection local address: EOF"
Jan 10 15:41:01 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:01Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6443/cacerts>\": remote error: tls: internal error"
Jan 10 15:41:01 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:01Z" level=info msg="Waiting for control-plane node agent startup"
Jan 10 15:41:02 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:02Z" level=info msg="Waiting for control-plane node agent startup"
Jan 10 15:41:03 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:03Z" level=error msg="dynamiclistener [::]:6443: failed to update cert with connection local address: EOF"
Jan 10 15:41:03 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:03Z" level=error msg="dynamiclistener [::]:6443: failed to update cert with connection local address: EOF"
Jan 10 15:41:03 test-control-plane-0 k3s[2248]: time="2023-01-10T15:41:03Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6443/cacerts>\": remote error: tls: internal error"

root@test-control-plane-0:~# curl <https://127.0.0.1:6443/cacerts>
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

k3s version v1.23.15+k3s1 (50cab3b3)
go version go1.17.13

But I’ll try with 1.24 now too…

okay apparently it worked with 1.24 now. or maybe I did something else weirdly…

will try again tomorrow

well okay it is a timing/startup/network issue

if I move the server fast enough into the private network in which it can assign the 10.1.1.1 to itself it works

if I don’t, or do it later, it gets into this loop

There’s a chicken-egg-problem in my terraform code. It will only add the server to the private network once it’s provisioned… but it can’t be “ready” before it’s in the private network. Therefore I have to do it via provider console UI.

Okay indeed. In 1.24 this does not happen. So I discovered a 1.23 bug 😄 well I won’t be creating new 1.23 clusters anymore so this thread can be ignored.

51 Views

#k3s Join Slack