https://rancher.com/ logo
Powered by
Title
n

narrow-article-96388

02/13/2023, 3:26 PM
Hello guys, I want to ask to fix my problem. I have one baremetal server and 2 VPC on AWS, i want create K3S cluster with master in AWS and worker/ nodes in baremetal. First, I installed K3S (default pod CIDR 10.42.0.0/16 and service CIDR 10.43.0.0/16) without wireguard (using vxlan), i have problem the baremetal01 cannot telnet 10.43.0.1 port 443. on master: 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker
on worker: 
curl -sfL <https://get.k3s.io> | K3S_URL=<https://10.10.1.60:6443> K3S_TOKEN={{ Token }} sh -s - --docker --node-ip 10.116.212.4 --node-external-ip 10.116.212.4 --flannel-iface eth1
- I installed K3S using wireguard with following this guideline https://www.inovex.de/de/blog/how-to-set-up-a-k3s-cluster-on-wireguard/ and K3S running well: on master: 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
on worker: 
curl -sfL <https://get.k3s.io> | K3S_URL=<https://10.10.1.60:6443> K3S_TOKEN={{ Token }} sh -s - --docker --node-ip 10.116.212.4 --node-external-ip 10.116.212.4 --flannel-iface eth1
- Baremetal01 (10.116.1.2) can connect to VPC A & VPC B via IPSec Site to Site - All instances in VPC A & B can connect to baremetal01 But i have one problem, the pods in baremetal01, cannot connect to VPC A and VPC B, only connect to IP internal baremetal01. Any advice would be appreciated. Thank you
Revision: First (without wireguard): on master: 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker
on worker: 
curl -sfL <https://get.k3s.io> | K3S_URL=<https://10.10.1.60:6443> K3S_TOKEN={{ Token }} sh -s - --docker --node-ip 10.222.0.2 --node-external-ip 10.116.1.2 --flannel-iface eth1
Second (with wireguard): on master: 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
on worker: 
curl -sfL <https://get.k3s.io> | K3S_URL=<https://10.10.1.60:6443> K3S_TOKEN={{ Token }} sh -s - --docker --node-ip 10.222.0.2 --node-external-ip 10.116.1.2 --flannel-iface eth1
p

plain-byte-79620

02/13/2023, 3:55 PM
Hi, are you trying to contact 
10.10.1.60
from a pod on baremetal01? Could it be related to the MTU? As I understand you are doing a wireguard tunnel to connect all the nodes and another wireguard tunnel used by K3s for the pods.
n

narrow-article-96388

02/13/2023, 5:00 PM
Thanks for your response, From a pod on baremetal cannot contact 
10.10.1.60
, this is my wireguard setting: on master: 
[Interface]
Address = 10.222.0.1
ListenPort = 51871
PrivateKey = xxxxxxxxx/Dq1wHNBrG7Efx3U=
MTU = 1420

[Peer]
PublicKey = xxxxxxxxxx/ukusTxPnKAq1mGyc5T8uDeWixw=
Endpoint = 10.116.1.2:51871
AllowedIPs = 10.222.0.2/32
PersistentKeepalive = 29
on node: 
[Interface]
Address = 10.222.0.2/32
ListenPort = 51871
PrivateKey = xxxxxxxxxDCUjdpCiojksD9l8IbUyjw9IO34=

[Peer]
PublicKey = xxxxxxxxxxx+xwraRsunUXlz43m9HID4M7x3k=
Endpoint = 10.10.1.60:51871
AllowedIPs = 10.222.0.1/32
PersistentKeepalive = 29
Is there anything that needs to be changed from the wireguard configuration?
p

plain-byte-79620

02/13/2023, 5:04 PM
from the master can you ping the pod?
n

narrow-article-96388

02/13/2023, 5:26 PM
From master (node) cannot ping pod IP
p

plain-byte-79620

02/13/2023, 5:43 PM
but you can ping 10.222.0.2
n

narrow-article-96388

02/14/2023, 1:53 AM
Yes can
p

plain-byte-79620

02/14/2023, 7:59 AM
with 
wg show
what do you get?
n

narrow-article-96388

02/14/2023, 8:02 AM
on master 
root@master-baremetal01:~# wg show
interface: wg0
  public key: xxxxx+cJmA+xwraRsunUXlz43m9HID4M7x3k=
  private key: (hidden)
  listening port: 51871

peer: xxxxxD2kd10zmdI/ukusTxPnKAq1mGyc5T8uDeWixw=
  endpoint: 10.116.1.2:51871
  allowed ips: 10.222.0.2/32
  latest handshake: 29 seconds ago
  transfer: 40.92 MiB received, 150.46 MiB sent
  persistent keepalive: every 29 seconds

interface: flannel-wg
  public key: xxxxxf9hYbAMltlGWD9UlKR1bD2yvABrHnu964i6lM=
  private key: (hidden)
  listening port: 51820

peer: xxxxx+T2hwyaWfdlh8BJOeVqBJM0wNkN1S5A8MhHk=
  endpoint: 10.116.1.2:51820
  allowed ips: 10.42.1.0/24
  latest handshake: 1 minute, 9 seconds ago
  transfer: 201.19 KiB received, 116.75 KiB sent
  persistent keepalive: every 25 seconds
on nodes/ worker: 
interface: flannel-wg
  public key: xxxxxx+T2hwyaWfdlh8BJOeVqBJM0wNkN1S5A8MhHk=
  private key: (hidden)
  listening port: 51820

peer: xxxxxxx9hYbAMltlGWD9UlKR1bD2yvABrHnu964i6lM=
  endpoint: 10.10.1.60:51820
  allowed ips: 10.42.0.0/24
  latest handshake: 52 seconds ago
  transfer: 22.51 KiB received, 46.35 KiB sent
  persistent keepalive: every 25 seconds

interface: wg0
  public key: xxxxx2kd10zmdI/ukusTxPnKAq1mGyc5T8uDeWixw=
  private key: (hidden)
  listening port: 51871

peer: xxxxxxx+cJmA+xwraRsunUXlz43m9HID4M7x3k=
  endpoint: 10.10.1.60:51871
  allowed ips: 10.222.0.1/32
  latest handshake: 12 seconds ago
  transfer: 150.32 MiB received, 40.93 MiB sent
  persistent keepalive: every 29 seconds
p

plain-byte-79620

02/14/2023, 9:05 AM
ok maybe you should use 
10.222.0
IPs as node-ip on K3s
n

narrow-article-96388

02/14/2023, 9:59 AM
do you mean like this? 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --cluster-cidr 10.222.0.0/24 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
p

plain-byte-79620

02/14/2023, 10:00 AM
--node-ip 10.222.0.1
n

narrow-article-96388

02/14/2023, 10:01 AM
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-ip 10.222.0.1 --cluster-cidr 10.222.0.0/24 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
p

plain-byte-79620

02/14/2023, 10:02 AM
yes and on the other node 
10.222.0.2
n

narrow-article-96388

02/14/2023, 10:02 AM
Ok, i will try first
I try to create new master and k3s not running with this command: 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-ip 10.222.0.1 --cluster-cidr 10.222.0.0/24 --node-external-ip 10.10.3.224 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
Error: 
Feb 14 10:55:35 ip-10-10-3-224 k3s[32292]: I0214 10:55:35.351317   32292 reconciler.go:169] "Reconciler: start to sync state"
Feb 14 10:55:35 ip-10-10-3-224 k3s[32292]: time="2023-02-14T10:55:35Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Feb 14 10:55:35 ip-10-10-3-224 systemd-networkd[416]: flannel-wg: Link UP
Feb 14 10:55:35 ip-10-10-3-224 k3s[32292]: time="2023-02-14T10:55:35Z" level=fatal msg="flannel exited: failed to set up the route: failed to add route flannel-wg: file exists"
Feb 14 10:55:35 ip-10-10-3-224 systemd-networkd[416]: flannel-wg: Gained carrier
Feb 14 10:55:35 ip-10-10-3-224 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 10:55:35 ip-10-10-3-224 systemd[1]: k3s.service: Failed with result 'exit-code'.
Feb 14 10:55:35 ip-10-10-3-224 systemd[1]: k3s.service: Consumed 6.944s CPU time.
p

plain-byte-79620

02/14/2023, 11:07 AM
you didn't stop the other 
k3s
instance
n

narrow-article-96388

02/14/2023, 11:14 AM
I have stopped the other k3s and the server for this new master I have setup using ip 
10.222.0.1
p

plain-byte-79620

02/14/2023, 11:15 AM
did you run 
k3s-uninstall.sh
? The error seems related to the flannel interface created from the previous setup
n

narrow-article-96388

02/14/2023, 11:19 AM
Screenshot 2023-02-14 at 18.18.02.png
k3s-error.log
k3s-error.log
Screenshot 2023-02-14 at 18.20.53.png
p

plain-byte-79620

02/14/2023, 11:27 AM
why 
flannel-wg
has 
10.222.0.0
as address? Why do you specify 
10.222.0.0/24
as 
cluster-cidr
? you are creating an IP overlapping. Maybe I didn't understand correctly what you want to do.
n

narrow-article-96388

02/15/2023, 1:50 AM
flannel-wg auto-generated after I installed k3s.
Before install k3s
After install k3s
p

plain-byte-79620

02/15/2023, 9:47 AM
flannel-wg
is installed by K3s and gets an IP from the 
cluster-cidr
that it's overlapping with the IP that you are using on your wireguard tunnel. If you want that the PODs traffic be forwarded on the wireguard tunnel that you created you should use 
--flannel-backend=host-gw
and configures the routes manually if you use 
--flannel-backend=wireguard-native
K3s will always create an additional wireguard tunnel.
n

narrow-article-96388

02/15/2023, 10:02 AM
Oh ok, thanks for the explanation, so how can the pods in the baremetal be connected to all the instances in VPC A and VPC B?
p

plain-byte-79620

02/15/2023, 10:05 AM
Do the pods need to talk with other pods on VPC A and B or other services?
n

narrow-article-96388

02/15/2023, 10:07 AM
I have some services on instances in VPC A and VPC B such as database and others, and I want the pods in baremetal to be able to talk to the instances in VPC A and VPC B.
p

plain-byte-79620

02/15/2023, 10:18 AM
you should add a route that specify 
10.222.0.1
as default gateway for all the IPs that the PODs need to contact. With that you'll force all the traffic to the wireguard tunnel.
n

narrow-article-96388

02/15/2023, 12:12 PM
Like this in baremetal? 
ip route add 10.10.0.0/21 via 10.222.0.1
it still doesn’t work 😞
p

plain-byte-79620

02/15/2023, 2:03 PM
did you remove 
cluster-cidr
from the K3s config?
n

narrow-article-96388

02/15/2023, 4:06 PM
Yup, because if i use 
--cluster-cidr 10.222.0.0/24
, it will overlapping.
could you give me some advice?
p

plain-byte-79620

02/15/2023, 6:11 PM
don't use 
--cluster-cidr 10.222.0.0/24
use another cidr and use the 
wireguard-native
backend. I think that if you create an additional routing table with 10.222.0.1 as default gateway and then an ip rule that matches the IPs of the pods as src and 
10.10.0.0/21
as destination to use that table it should work.
n

narrow-article-96388

02/17/2023, 6:38 AM
I already created additional routing in baremetal, but still doesnt work. FYI, my baremetal have 2 interfaces, 1 interface (eth0/bond0) for private IP (10.116.1.2) and 1 interface (eth1/bond1) for public IP (xx.xx.xx.xx). This is when i run ip route table main and table 220 in baremetal: 
root@baremetal01:~# ip route
default via 160.202.190.49 dev bond1 proto static
10.0.0.0/8 via 10.116.1.1 dev bond0 proto static
10.42.0.0/16 dev flannel-wg scope link
10.42.1.0/24 dev cni0 proto kernel scope link src 10.42.1.1
10.116.1.0/26 dev bond0 proto kernel scope link src 10.116.1.2
10.222.0.1 dev wg0 scope link
160.26.0.0/16 via 10.116.1.1 dev bond0 proto static
160.202.190.48/28 dev bond1 proto kernel scope link src 160.202.190.54
166.8.0.0/14 via 10.116.1.1 dev bond0 proto static
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
table 220: 
root@baremetal01:~# ip route list table 220
10.10.0.0/21 via 160.202.190.49 dev bond1 proto static src 10.116.1.2
172.16.0.0/19 via 160.202.190.49 dev bond1 proto static src 10.116.1.2
p

plain-byte-79620

02/17/2023, 9:08 AM
do you add a rule for that table? 
ip rule add from 10.42.0.0/16 to 10.10.0.0/21 table new_table
the new table should route the traffic to 10.10.0.0/21 to 10.222.0.1 because all the traffic from the pods to those addresses should be forwarded inside the wireguard tunnel. The route that you have doesn't do that.
n

narrow-article-96388

02/20/2023, 6:19 AM
In baremetal
Still destination host unreachable
p

plain-byte-79620

02/20/2023, 9:22 AM
Could you try to inspect the traffic? I think that the traffic is rightly forwarded inside the wireguard tunnel but is not able to reach the other node. You can try to use 
tcpdump
on the node with the 
10.10.0.83
address filtering the ICMP packets.
n

narrow-article-96388

02/20/2023, 3:21 PM
its WORKS! thank you @plain-byte-79620 for your help!!!
👍 1
67 Views
#k3sJoin Slack
Powered by