adamant-kite-43734
03/31/2023, 2:37 PMhundreds-evening-84071
03/31/2023, 3:27 PMjournalctl -u k3s
on additional nodes that you are trying to join?breezy-autumn-81048
04/03/2023, 5:54 AM: desc = \"transport: Error while dialing dial tcp 10.154.146.193:2379: connect: no route to host\""}
ERRO[0023] Failed to get member list from etcd cluster. Will assume this member is already added
INFO[0023] Starting etcd to join cluster with members [node1.example.net-e0ce15fe=<https://10.154.146.193:2380> node3.example.net-f5c8229b=<https://10.158.146.17:2380>]
{"level":"info","ts":"2023-04-03T07:49:14.530+0200","caller":"embed/etcd.go:124","msg":"configuring peer listeners","listen-peer-urls":["<https://10.158.146.17:2380>","<https://127.0.0.1:2380>"]}
{"level":"info","ts":"2023-04-03T07:49:14.530+0200","caller":"embed/etcd.go:482","msg":"starting with peer TLS","tls-info":"cert = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt, key = /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key, client-cert=, client-key=, trusted-ca = /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]}
{"level":"info","ts":"2023-04-03T07:49:14.530+0200","caller":"embed/etcd.go:132","msg":"configuring client listeners","listen-client-urls":["<https://10.158.146.17:2379>","<https://127.0.0.1:2379>"]}
{"level":"info","ts":"2023-04-03T07:49:14.531+0200","caller":"embed/etcd.go:306","msg":"starting an etcd server","etcd-version":"3.5.5","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.6","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":false,"name":"node3.example.net-f5c8229b","data-dir":"/var/lib/rancher/k3s/server/db/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/k3s/server/db/etcd/member","force-new-cluster":false,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://localhost:2380>"],"listen-peer-urls":["<https://10.158.146.17:2380>","<https://127.0.0.1:2380>"],"advertise-client-urls":["<https://10.158.146.17:2379>"],"listen-client-urls":["<https://10.158.146.17:2379>","<https://127.0.0.1:2379>"],"listen-metrics-urls":["<http://127.0.0.1:2381>"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"node1.example.net-e0ce15fe=<https://10.154.146.193:2380>,node3.example.net-f5c8229b=<https://10.158.146.17:2380>","initial-cluster-state":"existing","initial-cluster-token":"etcd-cluster","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-04-03T07:49:14.532+0200","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","took":"1.098286ms"}
INFO[0023] Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error
{"level":"warn","ts":"2023-04-03T07:49:14.536+0200","caller":"etcdserver/cluster_util.go:79","msg":"failed to get cluster response","address":"<https://10.154.146.193:2380/members>","error":"Get \"<https://10.154.146.193:2380/members>\": dial tcp 10.154.146.193:2380: connect: no route to host"}
{"level":"info","ts":"2023-04-03T07:49:14.544+0200","caller":"embed/etcd.go:371","msg":"closing etcd server","name":"node3.example.net-f5c8229b","data-dir":"/var/lib/rancher/k3s/server/db/etcd","advertise-peer-urls":["<http://localhost:2380>"],"advertise-client-urls":["<https://10.158.146.17:2379>"]}
{"level":"info","ts":"2023-04-03T07:49:14.545+0200","caller":"embed/etcd.go:373","msg":"closed etcd server","name":"node3.example.net-f5c8229b","data-dir":"/var/lib/rancher/k3s/server/db/etcd","advertise-peer-urls":["<http://localhost:2380>"],"advertise-client-urls":["<https://10.158.146.17:2379>"]}
FATA[0023] ETCD join failed: cannot fetch cluster info from peer urls: could not retrieve cluster information from the given URLs
breezy-autumn-81048
04/03/2023, 6:16 AM# nc 10.154.146.193 2380
Ncat: No route to host.
# nc 10.154.146.193 2379
Ncat: No route to host.
But what does that really mean? On the K3S version 1.24, I could add nodes to the cluster without any problems at all.breezy-autumn-81048
04/03/2023, 6:25 AMnc -v 10.154.146.193 6443
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: Connected to 10.154.146.193:6443.
nc -vu 10.154.146.193 8472
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: Connected to 10.154.146.193:8472.
nc -vu 10.154.146.193 10250
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: Connected to 10.154.146.193:10250.
nc -vu 10.154.146.193 2380 #UDP
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: Connected to 10.154.146.193:2380.
nc -v 10.154.146.193 2380
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: No route to host.
nc -v 10.154.146.193 2379
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: No route to host.
nc -vu 10.154.146.193 2379 #UDP
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: Connected to 10.154.146.193:2379.
51820 - is not in use.breezy-autumn-81048
04/03/2023, 6:31 AMnc -v 10.154.146.193 10250
Ncat: Version 7.70 ( <https://nmap.org/ncat> )
Ncat: No route to host.
breezy-autumn-81048
04/03/2023, 7:20 AMcreamy-pencil-82913
04/03/2023, 7:57 AMcreamy-pencil-82913
04/03/2023, 7:58 AMbreezy-autumn-81048
04/03/2023, 8:05 AMbreezy-autumn-81048
04/03/2023, 8:06 AMcreamy-pencil-82913
04/03/2023, 9:18 AMbreezy-autumn-81048
04/03/2023, 2:04 PM"ERROR runner Failed to get new registration token {"runner": "github-actions-runner-small-001-f6qfz-49wvb", "error": "failed to create registration token: Post "<https://test-github.example.net/api/v3/orgs/myorg/actions/runners/registration-token/>": could not refresh installation id 5's token: could not get access_tokens from GitHub API for installation ID 5: dial tcp: lookup <http://test-github.example.net|test-github.example.net>: i/o timeout"}"
The same thing with cert-manager:
E0403 13:54:46.663175 1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=49236>": dial tcp 10.43.0.1:443: i/o timeout
On version 1.26 cert-manager and ARC work fine, without any issues. The only problem there is that I can't add nodes to the cluster.
iptables and firewalld are not installed.creamy-pencil-82913
04/03/2023, 9:30 PMsomehow I can’t add nodes to the master node, I get “*no route to host*”.I don’t think it has anything to do with the version. It has to do with the nodes you’re trying to add not being able to reach the node you’re trying to add them to
creamy-pencil-82913
04/03/2023, 9:31 PMbreezy-autumn-81048
04/06/2023, 11:42 AM