https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
n
Revision: First (without wireguard): on master:
Copy code
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker
on worker:
Copy code
curl -sfL <https://get.k3s.io> | K3S_URL=<https://10.10.1.60:6443> K3S_TOKEN={{ Token }} sh -s - --docker --node-ip 10.222.0.2 --node-external-ip 10.116.1.2 --flannel-iface eth1
Second (with wireguard): on master:
Copy code
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
on worker:
Copy code
curl -sfL <https://get.k3s.io> | K3S_URL=<https://10.10.1.60:6443> K3S_TOKEN={{ Token }} sh -s - --docker --node-ip 10.222.0.2 --node-external-ip 10.116.1.2 --flannel-iface eth1
p
Hi, are you trying to contact 
10.10.1.60
from a pod on baremetal01? Could it be related to the MTU? As I understand you are doing a wireguard tunnel to connect all the nodes and another wireguard tunnel used by K3s for the pods.
n
Thanks for your response, From a pod on baremetal cannot contact 
10.10.1.60
, this is my wireguard setting: on master:
Copy code
[Interface]
Address = 10.222.0.1
ListenPort = 51871
PrivateKey = xxxxxxxxx/Dq1wHNBrG7Efx3U=
MTU = 1420

[Peer]
PublicKey = xxxxxxxxxx/ukusTxPnKAq1mGyc5T8uDeWixw=
Endpoint = 10.116.1.2:51871
AllowedIPs = 10.222.0.2/32
PersistentKeepalive = 29
on node:
Copy code
[Interface]
Address = 10.222.0.2/32
ListenPort = 51871
PrivateKey = xxxxxxxxxDCUjdpCiojksD9l8IbUyjw9IO34=

[Peer]
PublicKey = xxxxxxxxxxx+xwraRsunUXlz43m9HID4M7x3k=
Endpoint = 10.10.1.60:51871
AllowedIPs = 10.222.0.1/32
PersistentKeepalive = 29
Is there anything that needs to be changed from the wireguard configuration?
p
from the master can you ping the pod?
n
From master (node) cannot ping pod IP
p
but you can ping 10.222.0.2
n
Yes can
p
with 
wg show
what do you get?
n
on master
Copy code
root@master-baremetal01:~# wg show
interface: wg0
  public key: xxxxx+cJmA+xwraRsunUXlz43m9HID4M7x3k=
  private key: (hidden)
  listening port: 51871

peer: xxxxxD2kd10zmdI/ukusTxPnKAq1mGyc5T8uDeWixw=
  endpoint: 10.116.1.2:51871
  allowed ips: 10.222.0.2/32
  latest handshake: 29 seconds ago
  transfer: 40.92 MiB received, 150.46 MiB sent
  persistent keepalive: every 29 seconds

interface: flannel-wg
  public key: xxxxxf9hYbAMltlGWD9UlKR1bD2yvABrHnu964i6lM=
  private key: (hidden)
  listening port: 51820

peer: xxxxx+T2hwyaWfdlh8BJOeVqBJM0wNkN1S5A8MhHk=
  endpoint: 10.116.1.2:51820
  allowed ips: 10.42.1.0/24
  latest handshake: 1 minute, 9 seconds ago
  transfer: 201.19 KiB received, 116.75 KiB sent
  persistent keepalive: every 25 seconds
on nodes/ worker:
Copy code
interface: flannel-wg
  public key: xxxxxx+T2hwyaWfdlh8BJOeVqBJM0wNkN1S5A8MhHk=
  private key: (hidden)
  listening port: 51820

peer: xxxxxxx9hYbAMltlGWD9UlKR1bD2yvABrHnu964i6lM=
  endpoint: 10.10.1.60:51820
  allowed ips: 10.42.0.0/24
  latest handshake: 52 seconds ago
  transfer: 22.51 KiB received, 46.35 KiB sent
  persistent keepalive: every 25 seconds

interface: wg0
  public key: xxxxx2kd10zmdI/ukusTxPnKAq1mGyc5T8uDeWixw=
  private key: (hidden)
  listening port: 51871

peer: xxxxxxx+cJmA+xwraRsunUXlz43m9HID4M7x3k=
  endpoint: 10.10.1.60:51871
  allowed ips: 10.222.0.1/32
  latest handshake: 12 seconds ago
  transfer: 150.32 MiB received, 40.93 MiB sent
  persistent keepalive: every 29 seconds
p
ok maybe you should use 
10.222.0
IPs as node-ip on K3s
n
do you mean like this?
Copy code
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --cluster-cidr 10.222.0.0/24 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
p
--node-ip 10.222.0.1
n
Copy code
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-ip 10.222.0.1 --cluster-cidr 10.222.0.0/24 --node-external-ip 10.10.1.60 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
p
yes and on the other node 
10.222.0.2
n
Ok, i will try first
I try to create new master and k3s not running with this command:
Copy code
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="server --disable traefik" sh -s - --docker --advertise-address 10.222.0.1 --node-ip 10.222.0.1 --cluster-cidr 10.222.0.0/24 --node-external-ip 10.10.3.224 --flannel-iface=wg0 --flannel-backend=wireguard-native --flannel-external-ip
Error:
Copy code
Feb 14 10:55:35 ip-10-10-3-224 k3s[32292]: I0214 10:55:35.351317   32292 reconciler.go:169] "Reconciler: start to sync state"
Feb 14 10:55:35 ip-10-10-3-224 k3s[32292]: time="2023-02-14T10:55:35Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Feb 14 10:55:35 ip-10-10-3-224 systemd-networkd[416]: flannel-wg: Link UP
Feb 14 10:55:35 ip-10-10-3-224 k3s[32292]: time="2023-02-14T10:55:35Z" level=fatal msg="flannel exited: failed to set up the route: failed to add route flannel-wg: file exists"
Feb 14 10:55:35 ip-10-10-3-224 systemd-networkd[416]: flannel-wg: Gained carrier
Feb 14 10:55:35 ip-10-10-3-224 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 10:55:35 ip-10-10-3-224 systemd[1]: k3s.service: Failed with result 'exit-code'.
Feb 14 10:55:35 ip-10-10-3-224 systemd[1]: k3s.service: Consumed 6.944s CPU time.
p
you didn't stop the other 
k3s
instance
n
I have stopped the other k3s and the server for this new master I have setup using ip 
10.222.0.1
p
did you run 
k3s-uninstall.sh
? The error seems related to the flannel interface created from the previous setup
why 
flannel-wg
has 
10.222.0.0
as address? Why do you specify 
10.222.0.0/24
as 
cluster-cidr
? you are creating an IP overlapping. Maybe I didn't understand correctly what you want to do.
n
flannel-wg auto-generated after I installed k3s.
Before install k3s
After install k3s
p
flannel-wg
is installed by K3s and gets an IP from the 
cluster-cidr
that it's overlapping with the IP that you are using on your wireguard tunnel. If you want that the PODs traffic be forwarded on the wireguard tunnel that you created you should use 
--flannel-backend=host-gw
and configures the routes manually if you use 
--flannel-backend=wireguard-native
K3s will always create an additional wireguard tunnel.
n
Oh ok, thanks for the explanation, so how can the pods in the baremetal be connected to all the instances in VPC A and VPC B?
p
Do the pods need to talk with other pods on VPC A and B or other services?
n
I have some services on instances in VPC A and VPC B such as database and others, and I want the pods in baremetal to be able to talk to the instances in VPC A and VPC B.
p
you should add a route that specify 
10.222.0.1
as default gateway for all the IPs that the PODs need to contact. With that you'll force all the traffic to the wireguard tunnel.
n
Like this in baremetal?
Copy code
ip route add 10.10.0.0/21 via 10.222.0.1
it still doesn’t work 😞
p
did you remove 
cluster-cidr
from the K3s config?
n
Yup, because if i use 
--cluster-cidr 10.222.0.0/24
, it will overlapping.
could you give me some advice?
p
don't use 
--cluster-cidr 10.222.0.0/24
use another cidr and use the 
wireguard-native
backend. I think that if you create an additional routing table with 10.222.0.1 as default gateway and then an ip rule that matches the IPs of the pods as src and 
10.10.0.0/21
as destination to use that table it should work.
n
I already created additional routing in baremetal, but still doesnt work. FYI, my baremetal have 2 interfaces, 1 interface (eth0/bond0) for private IP (10.116.1.2) and 1 interface (eth1/bond1) for public IP (xx.xx.xx.xx). This is when i run ip route table main and table 220 in baremetal:
Copy code
root@baremetal01:~# ip route
default via 160.202.190.49 dev bond1 proto static
10.0.0.0/8 via 10.116.1.1 dev bond0 proto static
10.42.0.0/16 dev flannel-wg scope link
10.42.1.0/24 dev cni0 proto kernel scope link src 10.42.1.1
10.116.1.0/26 dev bond0 proto kernel scope link src 10.116.1.2
10.222.0.1 dev wg0 scope link
160.26.0.0/16 via 10.116.1.1 dev bond0 proto static
160.202.190.48/28 dev bond1 proto kernel scope link src 160.202.190.54
166.8.0.0/14 via 10.116.1.1 dev bond0 proto static
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
table 220:
Copy code
root@baremetal01:~# ip route list table 220
10.10.0.0/21 via 160.202.190.49 dev bond1 proto static src 10.116.1.2
172.16.0.0/19 via 160.202.190.49 dev bond1 proto static src 10.116.1.2
p
do you add a rule for that table?
Copy code
ip rule add from 10.42.0.0/16 to 10.10.0.0/21 table new_table
the new table should route the traffic to 10.10.0.0/21 to 10.222.0.1 because all the traffic from the pods to those addresses should be forwarded inside the wireguard tunnel. The route that you have doesn't do that.
n
In baremetal
Still destination host unreachable
p
Could you try to inspect the traffic? I think that the traffic is rightly forwarded inside the wireguard tunnel but is not able to reach the other node. You can try to use 
tcpdump
on the node with the 
10.10.0.83
address filtering the ICMP packets.
n
its WORKS! thank you @plain-byte-79620 for your help!!!
👍 1
367 Views
Open in Slack
PreviousNext
Powered by