This message was deleted.

Have you looked into enabling proxy protocol between haproxy and Traefik?

Looks like I hit an old bug at traefik helm chart and upgrading the whole thing fixed the proxy working. That's probably the best solution :o) For the records:

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    ports:
      websecure:
        proxyProtocol:
          insecure: true
      web:
        proxyProtocol:
          insecure: true

as

/var/lib/rancher/k3s/server/manifests/traefik-config.yaml

on one of the master nodes. In my case, I can use

insecure: true

, because no other host except the LBs is able to connect anyway. Other people might want to use

trustedIPs: [ "10.0.0.0/8" ]

or similar instead.

There are also options for solving the virtual ip problem. I would look into purelb or metallb. You could also build your own using things like keepalived and haproxy. A former company I worked at built their own combination LB/ingress controller using keepalived + haproxy + some python scripts.

@quick-dentist-45681 I only have a bunch on VMs from a hosting provider who doesn't offer anything like LB. I don't know which of the VMs are on the same hypervisor or even DC. I don't think HA is possible with this setup, or is it?

So the idea of purelb and metallb is that they manage one or more virtual ips, and one of your designated vms holds that ip (in addition to its real ip). If that vm goes down, purelb/metallb arranges for one of the other vms to take over the virtual ip. For this to work on public internet, you need all your designated vms to be publicly accessible on the internet, and you need a range of public ips that you can select virtual ips from, that get routed into your network. I've never done the network side of that, so don't know how difficult that part is, but the purelb docs have some details about what you need I believe. Of course, if all your vms are really on the same bare-metal server, and that server goes down, then you'll be offline no matter what, but there are a range of situations where one vm can be down, while the others are up, and these solutions work well in those cases. I recommend checking out the purelb docs.

you need a range of public ips that you can select virtual ips from, that get routed into your network

And that is exactly what I don't have. I don't have any virtual IPs, just 1 fixed IP per server. It could work with IPv6, but I'm not sure I want to get involved with this right now. There's also no way to use BGP or anything like that. Sadly