Hello, k3s team! I recently submitted a PR to klipper-lb to address the "second issue" mentioned in post below, would anyone in the team be able to review this PR? Or is there something I am missing when making contributions? https://github.com/k3s-io/klipper-lb/pull/54 Thank you in advance! https://rancher-users.slack.com/archives/CGGQEHPPW/p1680014208965239

Hello Guys, i have a Problem with my K3S Server. On my Controll Plane Node i have 100% CPU Load. The K3s Server Version is 1.26.3. How to debug this Issue ? I allready restartet the Node

Hello folks, Is creating a symlink like https://github.com/k3s-io/k3s/issues/2068#issuecomment-1374672584, a fool proof solution for this issue? I've created symlinks specifically for

/var/lib/rancher/k3s

and

/run/k3s

and the installation works fine. Just want to confirm if there wouldn't be any other edge cases or any other problems that we could run into later due to this solution.

Good morning everyone! I’m confused because I created 2 different Load balancer services in a namespace, 1 load balancer is for UDP and the other is for another protocol. But when I check to see what pods are running in the “Kube-System” namespace I can see the following: “svclb-service1UDP” “svclb-service2DifferentProtocol” “svclb-traefik” Now after a little research, I found K3s automatically comes with traefik load balancer along with the default Kubernetes load balancer. But why does K3s by default come with both? I think the answer is because traefik is an ingress component while K3s LB is just a default svclb….. I found using the traefik ingress just has more features and is specifically better for security purposes compared to the default K3s load balancer service. But are both using the cloud? I don’t think the K3s LB is….

Once in a while, we do see that too many contained-shim child processes are running even if we run only 1 pod with 3 containers. We could not figure out what could go wrong in this situation. Any ideas are recommendations? k3s version we are running is 1.25+

Hi guys. K3s is working great for me on Ubuntu 22.04, but every time I do a k3s-killall.sh the entire connection to the VM goes down and I have to phisically reboot the VM. Is there something I am not considering?

this

Hi, i got a question about pre-release states: https://github.com/k3s-io/k3s/releases it shows a release like v1.27.2+k3s1 is a pre-release but it's a non-rc so it should be a "real" release right?

What’s the limitation when it comes to migrating a k3s cluster from ipv4 only to dual-stack? I’d naively expect to simply be able to assign additional cluster/service cidrs + node-ip, dns ip and carve out some pod cidrs for my CNI plugin, but feel like I’m missing something here.

Hello everyone, I am using traefik as the ingress controller for K3S cluster having two master nodes and three worker nodes. When I do kubectl get ingress, the address field has multiple IPs instead of one and don’t know why or how to fix it

Good morning everyone! I have my applications deployed using k3s deployment component But when I turn off the node with the application on it, it takes a lot longer then I expected k3s to reschedule the app onto a different node And also it keeps the old pod that was on the node that I turned off it keeps the old pod in terminating state even after it finally reschedule the pod and it’s running in a different node… so I have the old pod in terminating state and the new pod on different node running…. Does anyone have any recommended fields they added to increase k3s deployments pods reschedules? Or to stop the terminating state of the old pod? Thanks everybody -Fareed

Is it possible to migrate sqlite to postgres ?

How to push configuration data needed by the proxy to consul for service delivery in k3s? Is there any documentation to achieve this. Thanks

I'm looking at the Nvidia container runtime setup, and wondering if its possible to use multiple runtimes on a single node. I'd like to use the nvidia runtime for specific jobs, but use the containerd runtime for the rest of the workloads.

hey folks, i'm currently configuring a local private registry, and was able to successfully pull with

k3s ctr images pull

. However, when I have my image defined in a pod, I see the successful message at first in the pods events, but it ultimately ends up in a failed state with

CreateContainerError

. This is because after pulling from my private registry, it seemingly is still trying to then pull the image from

<http://docker.io/library/myimage:latest|docker.io/library/myimage:latest>

. Which obviously, it shouldn't need to do if I've already successfully grabbed the image. Not sure what config needs to change to fix this behavior

I'm deploying k3s with the following:

curl -sfL <https://get.k3s.io> | \
K3S_KUBECONFIG_OUTPUT="/root/.kube/config" \
K3S_KUBECONFIG_MODE="0600" \
sh -s - \
--data-dir /foo \
--cluster-cidr 241.80.0.0/16 \
--service-cidr 241.90.0.0/16

but flannel keeps failing with this error from the log:

level=fatal msg="flannel exited: failed to acquire lease: subnet \"241.80.0.0/16\" specified in the flannel net config doesn't contain \"10.42.0.0/24\" PodCIDR of the \"bar\" node"

Is there a way that I should force flannel to use the new cidr instead of assuming that I am using the default one?

Hello! I have problem doing air gapped install of k3s (v1.26.3+k3s1) on RHEL 9.1 INSTALL_K3S_SKIP_DOWNLOAD=true install_k3s.sh I get error:

[ERROR]  Failed to find the k3s-selinux policy, please install:

dnf install -y container-selinux

dnf install -y <https://rpm.rancher.io/k3s/stable/common/centos/9/noarch/>

Installing of container-selinux works, but the latter one does not:

[root@localhost uus]# dnf install -y <https://rpm.rancher.io/k3s/stable/common/centos/9/noarch/>

Updating Subscription Management repositories.

Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                 49 kB/s | 4.5 kB     00:00

Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs)                                 38 kB/s | 4.1 kB     00:00

repo-zabbix-50-rhel-9-x86_64                                                             24 kB/s | 2.0 kB     00:00

Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                    42 kB/s | 4.1 kB     00:00

repo-elasticsearch-rhel-9-x86_64                                                         23 kB/s | 2.0 kB     00:00

Red Hat Enterprise Linux 9 for x86_64 - Supplementary (RPMs)                             41 kB/s | 3.7 kB     00:00

repo-postgresql_common-rhel-9-x86_64                                                     26 kB/s | 2.0 kB     00:00

repo-postgresql_11-rhel-9-x86_64                                                         30 kB/s | 2.3 kB     00:00

repo-cassandra-rhel-9-x86_64                                                             26 kB/s | 2.0 kB     00:00

Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                      40 kB/s | 3.8 kB     00:00

repo-epel-rhel-9-x86_64                                                                  31 kB/s | 2.3 kB     00:00

Cannot open /var/cache/dnf/commandline-5fd3bd7dde2b1c5e/packages/.: Is a directory

Can you help?

I’ve got a rather annoying issue, that I’m not sure how to solve. I’m asking here as it’s related and gets more traffic. A bit of backstory first: I have a software I maintain that is currently running on RancherOS. It has been running fine for years. I cannot for the life of me get the IT department that owns the VM/Software to actually install a VM with K3S. What’s worse, is this server was previously used with Rancher 1.6.x. It wasn’t built to be manually maintained, it was built to be maintained by Rancher. Well, the client decided to go with a different product a year or two ago. Fast forward a few months, and I deleted the rancher server, as this client was the last software running on it. Well, fast-forward an additional couple of months, and the client decided to keep the software. The problem was now I no longer had a way to properly orchestrate the server. This is further aggrivated by the fact that everytime this server goes down, I have to manually ‘restore’ the images, and it’s a pain in the rear. So, ultimately, I’m in a position where I want to “somehow” install and use K3S on this server, so I can go ahead and install the software on my rancher 2.x install. I just don’t know how to go about doing this. At this rate, I feel like I’m just going to have to use KinD, which is a bit stupid. If all possible, I simply want to run K3S on the RancherOS server, and be done with it.

I found this. I’ll try it.

Hey guys, I'm having a little bit of trouble with a k3s external datastore. Here are the details It seems that every day or so k3s is generating a new authentication key and is trying to write it into my external mysql database. The problem is that there is apparently a lot of writings when he does that because my sql provider indicate that there is a cpu Outage and the mysql server becomes unavalable during theses writes. This cause all my nodes to become unavailble because they don't have the new k3s key to authenticate until I physically reboot one of the nodes. Therefore I was trying migrate my external mysql datastore to an etcd datastore but I don't have any clue on how to do that and even if it's possible, at the end of the k3s documentation for ha-embedded there is a line indicating that it should be possible, but the trys I've made have miserably failed and created new empty k3s clusters 🙂 (source : https://docs.k3s.io/datastore/ha-embedded) If any of you have some ideas on how I could resolve the issue I would really be glad because for now i'm forced to restart my nodes every day and my cluster is not reliable in that state 😕 here is the cluster creation script i've been using :

curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="--node-external-ip 192.168.0.10 --service-node-port-range 7777-32767" \
 sh -s - --disable=traefik --disable=local-storage --kubelet-arg system-reserved=cpu=500m,memory=1Gi,ephemeral-storage=1Gi \
 --kubelet-arg system-reserved=cpu=300m,memory=500Mi --kubelet-arg eviction-hard='memory.available<500Mi,nodefs.available<10%' \
 --token ${NODE_TOKEN} \
 --datastore-endpoint="mysql://${MYSQL_USER}:${MYSQL_PWD}/tcp(<${MYSQL_URI}:${MYSQL_PORT}>)/${MYSQL_DATABASE}"

Greetings! Does k3s need a special configuration if a host has several network interfaces (e.g., ens3, ens4, etc.)? I got an error in coredns pod saying:

[INFO] plugin/ready: Still waiting on "kubernetes"
[WARNING] plugin/kubernetes: starting server with unsynced Kubernetes API
.53
...
[INFO] plugin/ready: Still waiting on "kubernetes"

Does it mean that coredns cannot reach Kubernetes API?

Any hits guys? ^^^

hi. thanks for the work that you do to anyone that’s working on this project. for the first time in 5 years I’m actually looking forward to working with kubernetes. can anyone tell me how bad of an idea it is to start running k3s along ‘regular baremetal services’ and slowly starting to migrate everything to k3s provided I have enough CPU/memory resources for both?

question regarding flannel -- should k3s configure flannel such that both PodCIDR AND ServiceCIDR ranges are accessible via flannel? I have a dual-stack k3s cluster provisioned but flannel only appears to be adding routes for the PodCIDR network, so pods attempting to talk to other pods' services are failing. Is there a trick to debugging this?

Hi, I have been running a k3s cluster for years and I was having some traefik issues (duplicate names or something) so I --disabled traefik which is fine, traefik got removed, removing the above to reinstall traefik the crd got installed but not the pod, water disabling again it seems the traefik HelmChart crd is stuck, been there over 2 years and I can't seem to manually delete it either, just hangs, --force ---grace-period 0 and --now don't help the deletion. It's like there are some resources left behind so can't finish the delete, but I can't find them

Hello, Anyone know how to fix this issue. Getting this error on worker nodes of k3s cluster

Hi I'm new to K3s. I have follow https://www.rancher.com/quick-start. How can I create k3s cluster?

Hi. In following the directions on installing an air-gapped rancher environment I'm at the point of setting up an nginx load balancer for the front end, and the sample config file does not work at all: https://ranchermanager.docs.rancher.com/how-to-guides/new-user-guides/infrastructure-setup/nginx-load-balancer I get two errors:

1 - the first proxy_pass in the http stanza complains that it's not allowed here.
2 - I get this error: host not found in upstream "rancher_servers_https" in /etc/nginx/nginx.conf

The Rancher 2.5 documentation doesn't have that http section in it at all. I'm running this on RedHat8 any ideas?

Looks like someone tried asking the nginx folks about a few months ago and they said the config looks broken, so you might want to review the settings you have on the web page: https://trac.nginx.org/nginx/ticket/2451

I wanted to get opinions on this: I saw the K3s.service messages mentioning- “Raft protocol warning High clock drift” And I noticed all my nodes system clocks were not exact. And 1 was even like 8 minutes ahead. How important is it to ensure all nodes have the exact sync for their time? I solved the issue/synced all my nodes clocks…..but I wasn’t having to bad of a problem at all working with K3s when the issue was happening…. Cheers