Hello i have 3 node HA cluster in home and i want to create nginx based load balancer for kube api server. I am looking for a way to automatically reload nginx configuration if new nodes are added, and i wonder if there's a simple way to ask my cluster for all

NodeHosts

? I know they are in

coredns

configmap, so maybe using RBAC and calling kube API directly is the simplest way and then reloading the nginx config?

I have a K3S cluster using Traefik as the ingress controller. I have an ingress deployed that routes correctly to a sample nginx pod. I then added a tls secret to the ingress that contains a server certificate that I created with the cluster IP (192.168.1.6) as the CN (IP is also in the SAN). I also included some dummy domain using (k3s.test.192.168.1.6.nip.io) in the SAN section of the cert. In Chrome, when I use k3s.test.192.168.1.6.nip.io I can see that the server certificate is the one I created. However, when I just use the IP in Chrome - 192.168.1.6 - the server cert found is TRAEFIK DEFAULT CERT.

Ok, installed the first master with this command:

curl -sfL <https://get.k3s.io> | INSTALL_K3S_VERSION=v1.25.2+k3s1 K3S_TOKEN="REDACTED" sh -s - server --cluster-init --cluster-cidr "10.44.0.0/16" --flannel-iface "enp35s0.4000" --node-ip "10.45.0.1" --node-external-ip "95.217.198.219"

It’s up and running fine.

But when I tried to start the second master to join it to this once it can’t connect to 10.45.0.1:2308

looking on the first master I see this from lsof:

k3s-serve 26328            root   11u  IPv4 475400      0t0  TCP localhost.localdomain:2380 (LISTEN)

why localhost? that doesn’t seem right? It should be listening on the private vlan interface.

Ok, so this is weird. On the node I can use

kubectl

just fine but as soon as I scp

/etc/rancher/k3s/k3s.yaml

to my desktop (and edit the server setting to point to the server name instead of localhost) I get this error:

Unable to connect to the server: x509: certificate signed by unknown authority

DNS, nevermind. 😬

What's the best linux distro for k3s (not much configuration required, low memory/disk space footprint)?

Any idea why

kubectl exec

hangs and timeouts when trying to connect k3s API server via Traefik TCP ingress route? Other commands, including

port-forward

work.

> kubectl exec -i -t <...> -n <...> --v=9 -- /bin/sh
...
028 09:26:17.416564    9632 round_trippers.go:553] POST https://<...>:443/api/v1/namespaces/<...>/pods/<...>/exec?command=%2Fbin%2Fsh&container=<...>&stdin=true&stdout=true&tty=true 101 Switching Protocols in 104 milliseconds
                                                                                                  I1028 09:26:17.416585    9632 round_trippers.go:570] HTTP Statistics: DNSLookup 15 ms Dial 19 ms TLSHandshake 0 ms Duration 104 ms
                            I1028 09:26:17.416596    9632 round_trippers.go:577] Response Headers:
                                                                                                  I1028 09:26:17.416603    9632 round_trippers.go:580]     Connection: Upgrade
                                                                                                                                                                              I1028 09:26:17.416608    9632 round_trippers.go:580]     Upgrade: SPDY/3.1
                                                I1028 09:26:17.416613    9632 round_trippers.go:580]     X-Stream-Protocol-Version: <http://v4.channel.k8s.io|v4.channel.k8s.io>
                                                                                                                                                     error: Timeout occurred

Does k3s differ from k8s in terms of ingress? I always have trouble getting ingress to work in k3s, tried with haproxy and some traefik (But not familiar with traefik) Its a bare metal server, if I expose the deployment with LB it works to access.. Or if anyone are aware of any good writeups/docs for this please let me know

so I am trying to upgrade traefik to the latest version, and it seems to be working okay locally and i can curl from the host to the public IP on port 80, and it works. However if i try to connect remotely, (from my PC) it doesn't respond at all. With the older version of traefik it works okay and i CAN connect remotely. What is going on? How can I event debug this?

k3s weekly report! https://github.com/k3s-io/k3s/discussions/6382

just upgraded from 1.24 to 1.25 and now it seems that (depends on what depends on what, but i assume tigera would need to come up first) tiger/calico does not come up and the ingress. I use the vanilla/public helm charts for both, so i boostrap k3s with:

curl -fL <https://get.k3s.io> | K3S_KUBECONFIG_MODE="640" INSTALL_K3S_EXEC="server --disable-kube-proxy --disable=servicelb --disable=servicelb --disable-network-policy --flannel-backend=none --disable traefik" sh

And then install the tigera helm chart https://artifacthub.io/packages/helm/projectcalico/tigera-operator (using v3.24.3 right now) and the nginx ingress helm chart https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx (using 4.2.5).

Hi! I noticed that when installing k3s it deployes the traefik ingress with just one replica. This can lead to some downtime if the node running traefik goes down, until the pod is created on another node. I can manually scale the traefik deployment to more replicas but somehow this gets reset automatically to 1 (probably by helm). Is there a way to configure the traefik replicas count at k3s install time (or later)?

I am new to Kubernetes and directly jumping to K3s (no theory good and have done hands on at cloud level). My use case is to get streaming data (like IoT device or IP camera feed) to worker node. My understanding is, NGNIX or other ingress controller cant be used for this scenario. The L3s ecosystem will be like listener of RTSP stream, not being contacted with HTTP req. Has anynone worked or knowledgeable about this scenario and provide some details? Thanks in advance.

Hi you all, in case you are interested my book “Edge Computing with Kubernetes” was officially released by Packt publisher. With the book you can learn about how to create your own edge computing system using K3s and ARM devices such a Raspberry Pi devices. Available in Amazon https://www.amazon.com/Sergio-Mendez-ebook/dp/B0B2PSZ1BK … just in case that someone is interested 🙂

Hi,everyone!

How many nodes can a k3s cluster support?

I can not find any similar pages to kubernetes.io

Hasving some issue with rancher and k3s latest versions. 33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 failed to list *v1beta1.CronJob: the ser ││││ │ │ │ │ │ │ │ │ │ │ │ │ │ │││││││ │││││││ E1102 142220.708719 33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go168 Failed to watch *v1beta1.CronJob: failed

Perphaps someone could help me with this issue

this is the error in one of the rancher pods

On an embedded device, what is the fastest way to safely shutdown k3s, if the device is about to loose power. And/or what is the harm in killing it all uncleanly? Have any of you seen corruption on the k3s side? (single node, super super small k3s ‘cluster’)

Oh! Another question. This is probably already out there, but my googling is failing me today. What is the smallest possible install of k3s? Right now we use

INSTALL_K3S_CHANNEL=v1.2xxx sh -s - --disable=traefik --write-kubeconfig-mode 644

Can I get leaner? We only use host network, can I rip out flannel and turn off logging (pod logging too?) and the metrics server? Does anyone already have a study on this I can read? I want just enough to run like 5 very little pods. But I need nothing else…

I have some questions about k3s configuration und how some parts work together. After trying different solutions und reading a lot of tutorials and documentations in the last weeks I hope someone here in the community can help me out to clarify what is going on and why and how to get a working solution. The Recent Setup is build as HA with external Datastore. Therefor 3 Master/Server, 3 Worker/Agent and 3 Datastore (etcd) nodes are deployed with a HA IP and DNS name. Each node has a public and a local IP. The public IP on the first interface and the local IP on the second interface. The first and most important problem is that the internal cluster overwrites headers like x-forwarded-for, x-real-ip but we have one application that needs to know the IP of the client and gets only an internal cluster IP atm. I already tried a lot of solution e.g. setting

externalTrafficPolicy: Local

and many more. Is there any solution at all with my recent setup? Do I have to exchange Traefik and/or KlipperLB with another product like MetalLB, NGINX-Ingress or something else to get this working on-prem? Do I have to disable NAT Masquerading to preserve the IP and is it mandatory to configure routes myself without NAT or can k3s handle that by itself? Or do I have to change something in flannel? Another Problem is that randomly some nodes rapidly consume a lot of RAM and freeze. Only a reboot can fix that. Happens with with master and worker nodes. Could that be a side effect of forwarding traffic inside the cluster? Is that a known issue and is there a solution to fix that? For Installation the following commands were used

# First Master
curl -sfL <https://get.k3s.io> |sh -s - server --datastore-endpoint ${K3S_DATASTORE_ENDPOINT} --node-taint CriticalAddonsOnly=true:NoExecute --node-ip <LOCAL IP MASTER01> --node-external-ip <PUBLIC IP MASTER01> --tls-san <LOCAL IP MASTER01> --tls-san <LOCAL IP MASTER02> --tls-san <LOCAL IP MASTER03> --tls-san <PUBLIC IP MASTER01> --tls-san <PUBLIC IP MASTER02> --tls-san <PUBLIC IP MASTER03> --tls-san <http://master.example.com|master.example.com> --tls-san master01.example.xom --tls-san <http://master02.example.com|master02.example.com> --tls-san <http://master03.example.com|master03.example.com> --flannel-iface=eth1

# Second/Third Master (same command with exchanged IPs plus token and serverURL)

# Worker Nodes
curl -sfL <https://get.k3s.io> |sh -s - agent --server ${K3S_URL} --token ${K3S_NODE_TOKEN} --node-ip <NODE LOCAL IP> --node-external-ip <NODE PUBLIC IP> --flannel-iface eth1

👋 k3s folk! A bit confused with best way to install a Helm chart on k3s… Reading https://docs.k3s.io/helm and from that am I right that if I put a manifest with

kind: HelmChart

in

/var/lib/rancher/k3s/server/manifests

them it will automatically deploy it? Or does it go in

/var/lib/rancher/k3s/server/static/charts

Help me. after "rke up" i cannot login rancher, i get pod and check log almost relate to certificate "x509: certificate is valid for 10.32.0.81, 10.32.0.82, 10.32.0.83, 127.0.0.1, 10.43.0.1, "

Hi, I am seeing some aux/iptables errors when check the K3s config

k3s check-config

what does these errors exactly mean should is solve this and if so how? I am running k3s on Debian 11 bullseye with 3 masters (etcd).

root:# k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/2ef87ff954adbb390309ce4dc07500f29c319f84feec1719bfb5059c8808ec6a/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)
....
STATUS: 6 (fail)