https://rancher.com/ logo
Join Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
Title
c

chilly-telephone-51989

09/11/2022, 1:11 AM
trying to setup a k3s cluster or 3 nodes on aws machines. on master k3s installed successfully but on worker node it is stuck at starting agent message. i used external ip of master as url and its toke. this is the log of service and it cant get certificates 
k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: activating (start) since Sun 2022-09-11 00:57:46 UTC; 2min 17s ago
       Docs: <https://k3s.io>
    Process: 2842576 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 2842578 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 2842579 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 2842580 (k3s-agent)
      Tasks: 9
     Memory: 15.5M
        CPU: 146ms
     CGroup: /system.slice/k3s-agent.service
             └─2842580 "/usr/local/bin/k3s agent"

Sep 11 00:57:46 ip-172-31-41-97 sh[2842576]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Sep 11 00:57:46 ip-172-31-41-97 sh[2842577]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Sep 11 00:57:46 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:57:46Z" level=info msg="Starting k3s agent v1.24.4+k3s1 (c3f830e9)"
Sep 11 00:57:46 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:57:46Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [3.128.3.142:6443]"
Sep 11 00:58:06 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:58:06Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:58:28 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:58:28Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:58:50 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:58:50Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:59:12 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:59:12Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:59:34 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:59:34Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
Sep 11 00:59:56 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:59:56Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": context deadline >
~
i tried using curl but it wont work.
c

creamy-pencil-82913

09/11/2022, 1:23 AM
If curl won't work then the agent won't either. Sounds like you need to open some ports on the server?
You curled the server url, correct?
c

chilly-telephone-51989

09/11/2022, 1:24 AM
I curl for cacerts on localhost as well as VLAN ip of worker . doesnt work
$ curl <http://172.31.41.97:6444/cacerts>
curl: (7) Failed to connect to 172.31.41.97 port 6444 after 0 ms: Connection refused
$ curl <http://127.0.0.1:6444/cacerts>
curl: (56) Recv failure: Connection reset by peer
c

creamy-pencil-82913

09/11/2022, 1:26 AM
6444 is not the right port. Should be 6443 on the server IP.
c

chilly-telephone-51989

09/11/2022, 1:27 AM
this i used to install agent 
curl -sfL <https://get.k3s.io> | K3S_URL=<https://172.31.46.55:6443> K3S_TOKEN=<token> sh -
c

creamy-pencil-82913

09/11/2022, 1:29 AM
You need to be able to curl the K3S_URL address. If you're can't then you need to check the server IP and/or open the correct ports.
c

chilly-telephone-51989

09/11/2022, 1:31 AM
i'm using ubuntu, disabled the ufw firewall. i'm trying but curl doesnt work. it just hangs 
$ curl <https://172.31.46.55:6443/cacerts>
where do i need to look? is it some configuration file i need changed on server?
k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-09-10 19:09:11 UTC; 6h ago
       Docs: <https://k3s.io>
    Process: 169774 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 169776 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 169777 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 169778 (k3s-server)
      Tasks: 108
     Memory: 1.1G
        CPU: 22min 1.441s
     CGroup: /system.slice/k3s.service
             ├─169778 "/usr/local/bin/k3s server"
             ├─169883 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib>
             ├─171029 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 7e75a1f48>
             ├─171071 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id e19ef81c2>
             ├─171214 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id aa12f560d>
             ├─172321 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 78657bc84>
             └─172352 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 6dbbbaa64>

Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: E0910 19:10:11.590025  169778 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFo>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: I0910 19:10:11.590090  169778 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="0c2cbd5a84703>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: E0910 19:10:11.590892  169778 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFo>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: I0910 19:10:11.591111  169778 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="72dcddebde696>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: E0910 19:10:11.591894  169778 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFo>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: I0910 19:10:11.591975  169778 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="3d3b976e33d45>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: E0910 19:10:11.592433  169778 remote_runtime.go:604] "ContainerStatus from runtime service failed" err="rpc error: code = NotFo>
Sep 10 19:10:11 ip-172-31-46-55 k3s[169778]: I0910 19:10:11.592465  169778 kuberuntime_gc.go:361] "Error getting ContainerStatus for containerID" containerID="bc744a234a1ae>
Sep 10 21:25:35 ip-172-31-46-55 k3s[169778]: time="2022-09-10T21:25:35Z" level=warning msg="Proxy error: write failed: write tcp 127.0.0.1:6443->127.0.0.1:54670: write: con>
Sep 10 23:40:26 ip-172-31-46-55 k3s[169778]: time="2022-09-10T23:40:26Z" level=warning msg="Proxy error: write failed: write tcp 127.0.0.1:6443->127.0.0.1:49300: write: bro>
~
c

creamy-pencil-82913

09/11/2022, 3:46 AM
disable ufw, disable firewalld, make sure the ports are open on the security groups if you’re running on a cloud provider.
c

chilly-telephone-51989

09/11/2022, 3:51 AM
these are aws machines running ubuntu 20.10. i already disabled the firewall ufw. not sure what firewalld is. im not using aws cloud framework as such. these are simple EC2 machines. previously i had good experience with setting up k3s on two node machine cluster which was by a relatively unknown provider. it was very smooth. now on aws machines things seem amess. and I'm not even through the basic setup. i'm not sure what you mean by security groups but if I'm guided i think this can be resolved.
c

creamy-pencil-82913

09/11/2022, 5:27 AM
AWS has a built-in firewall that is outside the VM called security groups. They control what sort of traffic can get to the node. I suspect that’s where you’re being blocked.
c

chilly-telephone-51989

09/11/2022, 5:28 AM
how do I access that?
i mean is it possible thru command line?
c

creamy-pencil-82913

09/11/2022, 7:13 AM
No, you manage it through the aws control panel or aws cli
c

chilly-telephone-51989

09/13/2022, 6:23 PM
i just had cleaned machines along with the opened ports and restarted the process of installation on AWS EC2 machines. this time I was able to install agent but the agent after successful instalation stayed in the loaded state. this is the output from the journalctl and systemctl this time: 
$ sudo systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Tue 2022-09-13 18:18:54 UTC; 1s ago
       Docs: <https://k3s.io>
    Process: 1368106 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 1368108 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 1368109 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
    Process: 1368110 ExecStart=/usr/local/bin/k3s agent (code=exited, status=1/FAILURE)
   Main PID: 1368110 (code=exited, status=1/FAILURE)
        CPU: 557ms

Sep 13 18:18:54 ip-172-31-41-97 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 18:18:54 ip-172-31-41-97 systemd[1]: k3s-agent.service: Failed with result 'exit-code'.
$ journalctl -xe
░░ The job identifier is 19227.
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.881020 1368328 server.go:395] "Kubelet version" kubeletVersion="v1.24.4+k3s1"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.881053 1368328 server.go:397] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: W0913 18:19:21.883573 1368328 manager.go:159] Cannot detect current cgroup on cgroup v2
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.883986 1368328 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/var/lib/rancher/k3s/agent/client-ca.crt"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.886066 1368328 server.go:644] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.886447 1368328 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.886636 1368328 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroups>
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.886834 1368328 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.886961 1368328 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.887091 1368328 state_mem.go:36] "Initialized new in-memory state store"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.892131 1368328 kubelet.go:376] "Attempting to sync node with API server"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.892400 1368328 kubelet.go:267] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.892597 1368328 kubelet.go:278] "Adding apiserver pod source"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.892796 1368328 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.893836 1368328 kuberuntime_manager.go:239] "Container runtime initialized" containerRuntime="containerd" version="v1.6.6-k3s1" apiVersion="v1"
Sep 13 18:19:21 ip-172-31-41-97 systemd[1]: run-rc488dc88be974c37af8525a45620ac98.scope: Deactivated successfully.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: <http://www.ubuntu.com/support>
░░ 
░░ The unit run-rc488dc88be974c37af8525a45620ac98.scope has successfully entered the 'dead' state.
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.894515 1368328 server.go:1177] "Started kubelet"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.895424 1368328 server.go:150] "Starting to listen" address="0.0.0.0" port=10250
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: I0913 18:19:21.896198 1368328 server.go:410] "Adding debug handlers to kubelet server"
Sep 13 18:19:21 ip-172-31-41-97 k3s[1368328]: E0913 18:19:21.896811 1368328 server.go:166] "Failed to listen and serve" err="listen tcp 0.0.0.0:10250: bind: address already in use"
Sep 13 18:19:21 ip-172-31-41-97 microk8s.daemon-kubelite[1249831]: E0913 18:19:21.934045 1249831 kubelet.go:2424] "Error getting node" err="node \"ip-172-31-41-97\" not found"
Sep 13 18:19:21 ip-172-31-41-97 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: <http://www.ubuntu.com/support>
░░ 
░░ An ExecStart= process belonging to unit k3s-agent.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Sep 13 18:19:21 ip-172-31-41-97 systemd[1]: k3s-agent.service: Failed with result 'exit-code'.
FIXED, had traces of some old program microk8s.. after removing and restarting, this is the latest status 
$ sudo systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2022-09-13 18:25:02 UTC; 35s ago
       Docs: <https://k3s.io>
   Main PID: 1371914 (k3s-agent)
      Tasks: 43
     Memory: 80.0M
        CPU: 2.583s
     CGroup: /system.slice/k3s-agent.service
             ├─1371914 "/usr/local/bin/k3s agent"
             ├─1371928 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
             └─1372208 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 206042638c2e1d9b44189a36d4de2d2adb726ea6e0bd41>

Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.056267 1371914 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.068475 1371914 kube.go:357] Skip setting NodeNetworkUnavailable
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.073387 1371914 apiserver.go:52] "Watching apiserver"
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: time="2022-09-13T18:25:03Z" level=info msg="Wrote flannel subnet file to /run/flannel/subnet.env"
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: time="2022-09-13T18:25:03Z" level=info msg="Running flannel backend."
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.076088 1371914 vxlan_network.go:61] watching for new subnet leases
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.080718 1371914 topology_manager.go:200] "Topology Admit Handler"
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.085814 1371914 reconciler.go:159] "Reconciler: start to sync state"
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.092278 1371914 iptables.go:177] bootstrap done
Sep 13 18:25:03 ip-172-31-41-97 k3s[1371914]: I0913 18:25:03.096937 1371914 iptables.go:177] bootstrap done
agent can't get certificate: 
Sep 13 18:51:44 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:51:44Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:51848->127.0.0.1:6444: read: connection>
Sep 13 18:51:56 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:51:56Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:46952->127.0.0.1:6444: read: connection>
Sep 13 18:52:08 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:52:08Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:47310->127.0.0.1:6444: read: connection>
Sep 13 18:52:20 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:52:20Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:48554->127.0.0.1:6444: read: connection>
Sep 13 18:52:32 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:52:32Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:34618->127.0.0.1:6444: read: connection>
Sep 13 18:52:45 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:52:45Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:43442->127.0.0.1:6444: read: connection>
Sep 13 18:52:57 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:52:57Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:33678->127.0.0.1:6444: read: connection>
Sep 13 18:53:09 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:53:09Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:47584->127.0.0.1:6444: read: connection>
Sep 13 18:53:21 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:53:21Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:34560->127.0.0.1:6444: read: connection>
Sep 13 18:53:33 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:53:33Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:52804->127.0.0.1:6444: read: connection>
Sep 13 18:53:45 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:53:45Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:38864->127.0.0.1:6444: read: connection>
Sep 13 18:53:57 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:53:57Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:49322->127.0.0.1:6444: read: connection>
Sep 13 18:54:09 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:54:09Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:49990->127.0.0.1:6444: read: connection>
Sep 13 18:54:21 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:54:21Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:42560->127.0.0.1:6444: read: connection>
Sep 13 18:54:32 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:54:32Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:43896->127.0.0.1:6444: read: connection>
Sep 13 18:54:44 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:54:44Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:50946->127.0.0.1:6444: read: connection>
Sep 13 18:54:56 ip-172-31-34-0 k3s[11609]: time="2022-09-13T18:54:56Z" level=error msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:52860->127.0.0.1:6444: read: connection>
Sep
from https://github.com/k3s-io/k3s/issues/2852 I tried the curl 
curl -vk <https://127.0.0.1:6444/cacerts>
*   Trying 127.0.0.1:6444...
* Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444 
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444
c

creamy-pencil-82913

09/13/2022, 7:04 PM
Don’t curl the agent tunnel port (127.0.0.1:6444), curl the server URL
That’s just a tunnel to the server, as you can see from this log line: 
Sep 11 00:57:46 ip-172-31-41-97 k3s[2842580]: time="2022-09-11T00:57:46Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [3.128.3.142:6443]"
So curl that URL directly, instead of trying to curl the local load-balancer port
c

chilly-telephone-51989

09/13/2022, 7:10 PM
~$ curl -vk <https://3.128.3.142:6443>
*   Trying 3.128.3.142:6443...
* Connected to 3.128.3.142 (3.128.3.142) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
c

creamy-pencil-82913

09/13/2022, 7:12 PM
is that the end of it? I don’t see that you actually get a response.
c

chilly-telephone-51989

09/13/2022, 7:13 PM
no i truncated it. this is the end 
5Gq9ygUwCgYIKoZIzj0EAwIDSQAwRgIhAJ6cFBI+o9tU8K2p0HhOH9Vh/d6l0p6N
hZByi9VeHIVKAiEAoYvANyaaZorxBrP3nSRtfJcI8yIAqrkXaRjWTNdH0jI=
-----END CERTIFICATE-----
* Connection #0 to host 3.128.3.142 left intact
should I paste all?
c

creamy-pencil-82913

09/13/2022, 7:24 PM
Yes, I am curious if you are able to actually get a response from the server. You should be able to do 
curl -vks <https://3.128.3.142:6443/ping>
and get a 
pong
response
c

chilly-telephone-51989

09/13/2022, 7:26 PM
$ curl -vks <https://3.128.3.142:6443/ping> 
*   Trying 3.128.3.142:6443...
* Connected to 3.128.3.142 (3.128.3.142) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Sep 10 19:09:06 2022 GMT
*  expire date: Sep 13 19:09:54 2023 GMT
*  issuer: CN=k3s-server-ca@1662836946
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55b3fee32010)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /ping HTTP/2
> Host: 3.128.3.142:6443
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200 
< content-type: text/plain
< content-length: 4
< date: Tue, 13 Sep 2022 19:26:06 GMT
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host 3.128.3.142 left intact
pong
thiis is what i get from the k3s agent node 
curl -vk <https://3.128.3.142:6443/ping> | more
*   Trying 3.128.3.142:6443...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 3.128.3.142 (3.128.3.142) port 6443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS header, Certificate Status (22):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.2 (IN), TLS header, Finished (20):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [15 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
{ [45 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [963 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [79 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.2 (OUT), TLS header, Finished (20):
} [5 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
} [8 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=k3s; CN=k3s
*  start date: Sep 10 19:09:06 2022 GMT
*  expire date: Sep 13 19:09:54 2023 GMT
*  issuer: CN=k3s-server-ca@1662836946
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5632688f4550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
> GET /ping HTTP/2
> Host: 3.128.3.142:6443
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [130 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
} [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
< HTTP/2 200 
< content-type: text/plain
< content-length: 4
< date: Tue, 13 Sep 2022 19:32:04 GMT
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
{ [5 bytes data]
100     4  100     4    0     0    504      0 --:--:-- --:--:-- --:--:--   571
* Connection #0 to host 3.128.3.142 left intact
pong
c

creamy-pencil-82913

09/13/2022, 7:34 PM
ok, that looks good
Can you attach (not paste in) the complete k3s-agent log?
c

chilly-telephone-51989

09/13/2022, 7:35 PM
where can i find it
c

creamy-pencil-82913

09/13/2022, 7:40 PM
in the systemd journal?
journalctl -u k3s-agent --no-pager > k3s-agent.log
what does the server say when all of this is happening?
Do you have something blocking port 6444 on the agent? iptables rules left over from something else?
c

chilly-telephone-51989

09/14/2022, 2:14 AM
$ sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      541/systemd-resolve 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      870/sshd: /usr/sbin 
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      12440/k3s agent     
tcp6       0      0 :::22                   :::*                    LISTEN      870/sshd: /usr/sbin 
udp        0      0 0.0.0.0:40987           0.0.0.0:*                           12440/k3s agent     
udp        0      0 127.0.0.53:53           0.0.0.0:*                           541/systemd-resolve 
udp        0      0 172.31.34.0:68          0.0.0.0:*                           539/systemd-network 
udp        0      0 127.0.0.1:323           0.0.0.0:*                           717/chronyd         
udp6       0      0 ::1:323                 :::*                                717/chronyd
$ curl <https://127.0.0.1:6444/cacerts/ping>
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444
the port was blocked by AWS security group. i had it opened (some client side person takes care of that) so after opening it i was able to curl it from the node machine here is the output 
curl -vk <https://127.0.0.1:6444/cacerts>
*   Trying 127.0.0.1:6444...
* Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444 
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 127.0.0.1:6444
the connection reset by peer error is gone now. I got the latest log of the agent too but the problem remains the same there and no change is there. Should I reinstall it or something? UFW is already off. (but after every boot with 
sudo reboot
it is enabled magically so I have to disable it manually. I did restart the service 
sudo systemctl restart k3s-agent
but i'm still getting same error though the curl as you can see has started working for https://127.0.0.1:6444/cacerts
I used the public IP for the server instead of LAN Ip 
$ sudo systemctl status k3s-agent
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2022-09-14 11:51:30 UTC; 45s ago
       Docs: <https://k3s.io>
    Process: 17418 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 17420 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 17421 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 17422 (k3s-agent)
      Tasks: 42
     Memory: 293.5M
        CPU: 5.056s
     CGroup: /system.slice/k3s-agent.service
             ├─17422 "/usr/local/bin/k3s agent"
             ├─17441 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
             └─17810 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id e5fa2f35fc2d8b44bcf5fcb92ae2e6644eeab37602306248>

Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.558658   17422 reconciler.go:159] "Reconciler: start to sync state"
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.824386   17422 kube.go:128] Node controller sync successful
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.824503   17422 vxlan.go:138] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.846396   17422 kube.go:357] Skip setting NodeNetworkUnavailable
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: time="2022-09-14T11:51:31Z" level=info msg="Wrote flannel subnet file to /run/flannel/subnet.env"
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: time="2022-09-14T11:51:31Z" level=info msg="Running flannel backend."
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.849661   17422 vxlan_network.go:61] watching for new subnet leases
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.859232   17422 iptables.go:177] bootstrap done
Sep 14 11:51:31 ip-172-31-34-0 k3s[17422]: I0914 11:51:31.862995   17422 iptables.go:177] bootstrap done
Sep 14 11:51:41 ip-172-31-34-0 k3s[17422]: I0914 11:51:41.235876   17422 topology_manager.go:200] "Topology Admit Handler"
this is the latest: fixes: 1. opened ports 6444, 16443 on AWS Security Group 2. used public ip of server instead of private VLAN ip 
$ k get nodes
NAME              STATUS   ROLES                  AGE   VERSION
ip-172-31-46-55   Ready    control-plane,master   20m   v1.24.4+k3s1
ip-172-31-34-0    Ready    <none>                 17m   v1.24.4+k3s1
ip-172-31-41-97   Ready    <none>                 32s   v1.24.4+k3s1
😃
View count: 587
Back to #k3sJoin thread in Slack