This message was deleted.
# rke2a
adamant-kite-43734
02/09/2023, 12:43 PMThis message was deleted.
e
eager-london-83975
02/09/2023, 12:54 PMare the logs coming from the node agent you are trying to add or the master ?
s
stocky-article-82001
02/09/2023, 12:54 PMThe node I’m trying to add.
stocky-article-82001
02/09/2023, 12:55 PM Copy code
{"level":"warn","ts":"2023-02-09T12:33:59.607Z","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc001504000/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp 172.30.12.3:2379: connect: no route to host\""}
Feb 09 12:33:59 <MASTER HOSTNAME> rke2[13805]: time="2023-02-09T12:33:59Z" level=warning msg="Learner <HOSTNAME OF NEW NODE>-9ad37932 stalled at RaftAppliedIndex=0 for 5m0.607218804s"
Feb 09 12:33:59 <MASTER HOSTNAME> rke2[13805]: time="2023-02-09T12:33:59Z" level=warning msg="Removed learner <HOSTNAME OF NEW NODE>-9ad37932 from etcd cluster"stocky-article-82001
02/09/2023, 12:55 PMThese are some logs from the master which is weird.
e
eager-london-83975
02/09/2023, 12:55 PMYour network is not routed correctly
eager-london-83975
02/09/2023, 12:56 PMYour masternodes cannot communicate with each other
s
stocky-article-82001
02/09/2023, 12:57 PMThat’s bizarre, this new one has the exact same config as the other 2 masters.
e
eager-london-83975
02/09/2023, 12:57 PM Copy code
transport: Error while dialing dial tcp 172.30.12.3:2379: connect: no route to hosteager-london-83975
02/09/2023, 12:57 PMIt's easy to miss, but check your security groups/firewall
eager-london-83975
02/09/2023, 12:57 PMOr that your subnet is routed properly, or even if they are in the same network/vpc
s
stocky-article-82001
02/09/2023, 12:58 PMYeah ok, I’ll have a dig around. Thanks!
e
eager-london-83975
02/09/2023, 1:00 PMPlease don't forget to notify if you manage to solve it, might help others!
s
stocky-article-82001
02/09/2023, 3:16 PMThe nodes can communicate (I’ve confirmed) but it is still not working.
stocky-article-82001
02/09/2023, 3:17 PMI can curl the :6443 on the master from the new node fine
e
eager-london-83975
02/09/2023, 3:17 PMbut is 2379 open, namely ETCD running
s
stocky-article-82001
02/09/2023, 3:18 PMHmm, seems I’ve spoken too soon. It is working now, however there were some 500 errors at the start
stocky-article-82001
02/09/2023, 3:18 PMit has joined to the cluster successfully now, let me wait for it to fully reconcile.
stocky-article-82001
02/09/2023, 3:22 PMYeah it still seems to be fucking up with networking.
stocky-article-82001
02/09/2023, 3:23 PM~~~scratch that~~~
e
eager-london-83975
02/09/2023, 5:31 PMSo it's working good now ?
s
stocky-article-82001
02/09/2023, 5:36 PMNo, we’ve found some networking issues that we’re currently working through.
e
eager-london-83975
02/09/2023, 5:36 PMIs it on-premises or what cloud are you using to manage your network and nodes ?
1864 Views