https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
k3s
  • c

    chilly-exabyte-62143

    10/26/2022, 6:19 PM
    Hello i have 3 node HA cluster in home and i want to create nginx based load balancer for kube api server. I am looking for a way to automatically reload nginx configuration if new nodes are added, and i wonder if there's a simple way to ask my cluster for all
    NodeHosts
    ? I know they are in
    coredns
    configmap, so maybe using RBAC and calling kube API directly is the simplest way and then reloading the nginx config?
  • l

    lively-battery-54332

    10/26/2022, 10:54 PM
    I have a K3S cluster using Traefik as the ingress controller. I have an ingress deployed that routes correctly to a sample nginx pod. I then added a tls secret to the ingress that contains a server certificate that I created with the cluster IP (192.168.1.6) as the CN (IP is also in the SAN). I also included some dummy domain using (k3s.test.192.168.1.6.nip.io) in the SAN section of the cert. In Chrome, when I use k3s.test.192.168.1.6.nip.io I can see that the server certificate is the one I created. However, when I just use the IP in Chrome - 192.168.1.6 - the server cert found is TRAEFIK DEFAULT CERT.
    c
    • 2
    • 5
  • e

    enough-carpet-20915

    10/26/2022, 11:52 PM
    Ok, installed the first master with this command:
    curl -sfL <https://get.k3s.io> | INSTALL_K3S_VERSION=v1.25.2+k3s1 K3S_TOKEN="REDACTED" sh -s - server --cluster-init --cluster-cidr "10.44.0.0/16" --flannel-iface "enp35s0.4000" --node-ip "10.45.0.1" --node-external-ip "95.217.198.219"
    It’s up and running fine.
  • e

    enough-carpet-20915

    10/26/2022, 11:53 PM
    But when I tried to start the second master to join it to this once it can’t connect to 10.45.0.1:2308
  • e

    enough-carpet-20915

    10/26/2022, 11:54 PM
    looking on the first master I see this from lsof:
    k3s-serve 26328            root   11u  IPv4 475400      0t0  TCP localhost.localdomain:2380 (LISTEN)
  • e

    enough-carpet-20915

    10/26/2022, 11:54 PM
    why localhost? that doesn’t seem right? It should be listening on the private vlan interface.
    c
    • 2
    • 19
  • e

    enough-carpet-20915

    10/27/2022, 5:54 PM
    Ok, so this is weird. On the node I can use
    kubectl
    just fine but as soon as I scp
    /etc/rancher/k3s/k3s.yaml
    to my desktop (and edit the server setting to point to the server name instead of localhost) I get this error:
    Unable to connect to the server: x509: certificate signed by unknown authority
  • e

    enough-carpet-20915

    10/27/2022, 6:03 PM
    DNS, nevermind. 😬
    s
    • 2
    • 2
  • c

    chilly-exabyte-62143

    10/27/2022, 9:27 PM
    What's the best linux distro for k3s (not much configuration required, low memory/disk space footprint)?
    w
    w
    • 3
    • 3
  • m

    melodic-hamburger-23329

    10/28/2022, 3:39 AM
    Any idea why
    kubectl exec
    hangs and timeouts when trying to connect k3s API server via Traefik TCP ingress route? Other commands, including
    port-forward
    work.
    > kubectl exec -i -t <...> -n <...> --v=9 -- /bin/sh
    ...
    028 09:26:17.416564    9632 round_trippers.go:553] POST https://<...>:443/api/v1/namespaces/<...>/pods/<...>/exec?command=%2Fbin%2Fsh&container=<...>&stdin=true&stdout=true&tty=true 101 Switching Protocols in 104 milliseconds
                                                                                                      I1028 09:26:17.416585    9632 round_trippers.go:570] HTTP Statistics: DNSLookup 15 ms Dial 19 ms TLSHandshake 0 ms Duration 104 ms
                                I1028 09:26:17.416596    9632 round_trippers.go:577] Response Headers:
                                                                                                      I1028 09:26:17.416603    9632 round_trippers.go:580]     Connection: Upgrade
                                                                                                                                                                                  I1028 09:26:17.416608    9632 round_trippers.go:580]     Upgrade: SPDY/3.1
                                                    I1028 09:26:17.416613    9632 round_trippers.go:580]     X-Stream-Protocol-Version: <http://v4.channel.k8s.io|v4.channel.k8s.io>
                                                                                                                                                         error: Timeout occurred
    c
    • 2
    • 4
  • f

    fast-agency-28891

    10/28/2022, 6:18 AM
    Does k3s differ from k8s in terms of ingress? I always have trouble getting ingress to work in k3s, tried with haproxy and some traefik (But not familiar with traefik) Its a bare metal server, if I expose the deployment with LB it works to access.. Or if anyone are aware of any good writeups/docs for this please let me know
    c
    • 2
    • 8
  • g

    great-winter-35080

    10/28/2022, 12:15 PM
    so I am trying to upgrade traefik to the latest version, and it seems to be working okay locally and i can curl from the host to the public IP on port 80, and it works. However if i try to connect remotely, (from my PC) it doesn't respond at all. With the older version of traefik it works okay and i CAN connect remotely. What is going on? How can I event debug this?
    • 1
    • 2
  • c

    clever-air-65544

    10/28/2022, 5:29 PM
    k3s weekly report! https://github.com/k3s-io/k3s/discussions/6382
    🚀 1
  • n

    numerous-country-20400

    10/31/2022, 10:10 AM
    just upgraded from 1.24 to 1.25 and now it seems that (depends on what depends on what, but i assume tigera would need to come up first) tiger/calico does not come up and the ingress. I use the vanilla/public helm charts for both, so i boostrap k3s with:
    curl -fL <https://get.k3s.io> | K3S_KUBECONFIG_MODE="640" INSTALL_K3S_EXEC="server --disable-kube-proxy --disable=servicelb --disable=servicelb --disable-network-policy --flannel-backend=none --disable traefik" sh
    And then install the tigera helm chart https://artifacthub.io/packages/helm/projectcalico/tigera-operator (using v3.24.3 right now) and the nginx ingress helm chart https://artifacthub.io/packages/helm/ingress-nginx/ingress-nginx (using 4.2.5).
    c
    • 2
    • 9
  • b

    brainy-ram-25474

    11/01/2022, 8:19 AM
    Hi! I noticed that when installing k3s it deployes the traefik ingress with just one replica. This can lead to some downtime if the node running traefik goes down, until the pod is created on another node. I can manually scale the traefik deployment to more replicas but somehow this gets reset automatically to 1 (probably by helm). Is there a way to configure the traefik replicas count at k3s install time (or later)?
  • c

    cuddly-fountain-93707

    11/01/2022, 8:09 PM
    I am new to Kubernetes and directly jumping to K3s (no theory good and have done hands on at cloud level). My use case is to get streaming data (like IoT device or IP camera feed) to worker node. My understanding is, NGNIX or other ingress controller cant be used for this scenario. The L3s ecosystem will be like listener of RTSP stream, not being contacted with HTTP req. Has anynone worked or knowledgeable about this scenario and provide some details? Thanks in advance.
  • d

    damp-xylophone-94549

    11/01/2022, 10:19 PM
    Hi you all, in case you are interested my book “Edge Computing with Kubernetes” was officially released by Packt publisher. With the book you can learn about how to create your own edge computing system using K3s and ARM devices such a Raspberry Pi devices. Available in Amazon https://www.amazon.com/Sergio-Mendez-ebook/dp/B0B2PSZ1BK … just in case that someone is interested 🙂
    🚀 1
    👍 1
  • b

    bland-summer-47692

    11/02/2022, 6:27 AM
    Hi,everyone!
  • b

    bland-summer-47692

    11/02/2022, 6:28 AM
    How many nodes can a k3s cluster support?
    n
    • 2
    • 1
  • b

    bland-summer-47692

    11/02/2022, 6:30 AM
    I can not find any similar pages to kubernetes.io
  • b

    bland-summer-47692

    11/02/2022, 6:30 AM
    https://kubernetes.io/docs/setup/best-practices/cluster-large/
  • q

    quaint-library-7108

    11/02/2022, 2:23 PM
    Hasving some issue with rancher and k3s latest versions. 33 reflector.go:325] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: failed to list *v1beta1.CronJob: the ser ││││ │ │ │ │ │ │ │ │ │ │ │ │ │ │││││││ │││││││ E1102 14:22:20.708719 33 reflector.go:139] pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168: Failed to watch *v1beta1.CronJob: failed
  • q

    quaint-library-7108

    11/02/2022, 2:23 PM
    Perphaps someone could help me with this issue
  • q

    quaint-library-7108

    11/02/2022, 2:23 PM
    this is the error in one of the rancher pods
  • q

    quiet-memory-19288

    11/02/2022, 9:20 PM
    On an embedded device, what is the fastest way to safely shutdown k3s, if the device is about to loose power. And/or what is the harm in killing it all uncleanly? Have any of you seen corruption on the k3s side? (single node, super super small k3s ‘cluster’)
    c
    n
    • 3
    • 17
  • q

    quiet-memory-19288

    11/02/2022, 10:03 PM
    Oh! Another question. This is probably already out there, but my googling is failing me today. What is the smallest possible install of k3s? Right now we use
    INSTALL_K3S_CHANNEL=v1.2xxx sh -s - --disable=traefik --write-kubeconfig-mode 644
    Can I get leaner? We only use host network, can I rip out flannel and turn off logging (pod logging too?) and the metrics server? Does anyone already have a study on this I can read? I want just enough to run like 5 very little pods. But I need nothing else…
  • a

    able-traffic-85986

    11/03/2022, 3:22 PM
    I have some questions about k3s configuration und how some parts work together. After trying different solutions und reading a lot of tutorials and documentations in the last weeks I hope someone here in the community can help me out to clarify what is going on and why and how to get a working solution. The Recent Setup is build as HA with external Datastore. Therefor 3 Master/Server, 3 Worker/Agent and 3 Datastore (etcd) nodes are deployed with a HA IP and DNS name. Each node has a public and a local IP. The public IP on the first interface and the local IP on the second interface. The first and most important problem is that the internal cluster overwrites headers like x-forwarded-for, x-real-ip but we have one application that needs to know the IP of the client and gets only an internal cluster IP atm. I already tried a lot of solution e.g. setting
    externalTrafficPolicy: Local
    and many more. Is there any solution at all with my recent setup? Do I have to exchange Traefik and/or KlipperLB with another product like MetalLB, NGINX-Ingress or something else to get this working on-prem? Do I have to disable NAT Masquerading to preserve the IP and is it mandatory to configure routes myself without NAT or can k3s handle that by itself? Or do I have to change something in flannel? Another Problem is that randomly some nodes rapidly consume a lot of RAM and freeze. Only a reboot can fix that. Happens with with master and worker nodes. Could that be a side effect of forwarding traffic inside the cluster? Is that a known issue and is there a solution to fix that? For Installation the following commands were used
    # First Master
    curl -sfL <https://get.k3s.io> |sh -s - server --datastore-endpoint ${K3S_DATASTORE_ENDPOINT} --node-taint CriticalAddonsOnly=true:NoExecute --node-ip <LOCAL IP MASTER01> --node-external-ip <PUBLIC IP MASTER01> --tls-san <LOCAL IP MASTER01> --tls-san <LOCAL IP MASTER02> --tls-san <LOCAL IP MASTER03> --tls-san <PUBLIC IP MASTER01> --tls-san <PUBLIC IP MASTER02> --tls-san <PUBLIC IP MASTER03> --tls-san <http://master.example.com|master.example.com> --tls-san master01.example.xom --tls-san <http://master02.example.com|master02.example.com> --tls-san <http://master03.example.com|master03.example.com> --flannel-iface=eth1
    # Second/Third Master (same command with exchanged IPs plus token and serverURL)
    # Worker Nodes
    curl -sfL <https://get.k3s.io> |sh -s - agent --server ${K3S_URL} --token ${K3S_NODE_TOKEN} --node-ip <NODE LOCAL IP> --node-external-ip <NODE PUBLIC IP> --flannel-iface eth1
    b
    c
    • 3
    • 6
  • w

    wide-author-88664

    11/03/2022, 8:57 PM
    👋 k3s folk! A bit confused with best way to install a Helm chart on k3s… Reading https://docs.k3s.io/helm and from that am I right that if I put a manifest with
    kind: HelmChart
    in
    /var/lib/rancher/k3s/server/manifests
    them it will automatically deploy it? Or does it go in
    /var/lib/rancher/k3s/server/static/charts
    c
    n
    • 3
    • 35
  • b

    boundless-smartphone-66270

    11/04/2022, 6:12 PM
    Help me. after "rke up" i cannot login rancher, i get pod and check log almost relate to certificate "x509: certificate is valid for 10.32.0.81, 10.32.0.82, 10.32.0.83, 127.0.0.1, 10.43.0.1, "
  • p

    prehistoric-judge-25958

    11/06/2022, 10:33 PM
    Hi, I am seeing some aux/iptables errors when check the K3s config
    k3s check-config
    what does these errors exactly mean should is solve this and if so how? I am running k3s on Debian 11 bullseye with 3 masters (etcd).
    root:# k3s check-config
    
    Verifying binaries in /var/lib/rancher/k3s/data/2ef87ff954adbb390309ce4dc07500f29c319f84feec1719bfb5059c8808ec6a/bin:
    - sha256sum: good
    - links: aux/ip6tables should link to iptables-detect.sh (fail)
    - links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
    - links: aux/ip6tables-save should link to iptables-detect.sh (fail)
    - links: aux/iptables should link to iptables-detect.sh (fail)
    - links: aux/iptables-restore should link to iptables-detect.sh (fail)
    - links: aux/iptables-save should link to iptables-detect.sh (fail)
    ....
    STATUS: 6 (fail)
    n
    • 2
    • 8
Powered by Linen
Title
p

prehistoric-judge-25958

11/06/2022, 10:33 PM
Hi, I am seeing some aux/iptables errors when check the K3s config
k3s check-config
what does these errors exactly mean should is solve this and if so how? I am running k3s on Debian 11 bullseye with 3 masters (etcd).
root:# k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/2ef87ff954adbb390309ce4dc07500f29c319f84feec1719bfb5059c8808ec6a/bin:
- sha256sum: good
- links: aux/ip6tables should link to iptables-detect.sh (fail)
- links: aux/ip6tables-restore should link to iptables-detect.sh (fail)
- links: aux/ip6tables-save should link to iptables-detect.sh (fail)
- links: aux/iptables should link to iptables-detect.sh (fail)
- links: aux/iptables-restore should link to iptables-detect.sh (fail)
- links: aux/iptables-save should link to iptables-detect.sh (fail)
....
STATUS: 6 (fail)
n

nutritious-tomato-14686

11/07/2022, 5:32 PM
See https://docs.k3s.io/advanced#additional-preparation-for-debian-buster-based-distributions
p

prehistoric-judge-25958

11/07/2022, 6:16 PM
Thanks, but in the docs I read its for "Debian Buster" I am running Debian Bullseye
n

nutritious-tomato-14686

11/07/2022, 6:17 PM
What version of the iptables do you have though? It applies to every OS that is running version 1.8.0-1.8.4
p

prehistoric-judge-25958

11/07/2022, 6:24 PM
I did a clean installation of Debian 11 and looks like there is no iptables packages installed at all.
which iptables
gives me no result. Is Iptables required for K3s?
n

nutritious-tomato-14686

11/07/2022, 6:56 PM
Then you are fine, K3s bundles its own version of iptables, its just using that
We typically expect the host to have some version of iptables, and k3s defaults to using the host binary. That's why we have those checks in the
k3s check-config
p

prehistoric-judge-25958

11/07/2022, 6:58 PM
Ok thanks! so then I just ignore these 6 FAIL messages
n

nutritious-tomato-14686

11/07/2022, 7:25 PM
Yeah you can ignore those 6 FAILS. If you want to see all the iptables rules that k3s makes, you can always use
sudo /var/lib/rancher/k3s/data/current/bin/aux/iptables -L
. This would be a good thing to improve IMO about the k3s check-config. I think a warning about "k3s is using embedded iptables" would be better than 6 FAILS.
👍 1
View count: 30