Anyone have thoughts on why the hostname of my k3s cluster node change from the FQDN it started with to it’s shortname after an outage? Or vice versa?

Good afternoon everyone,

I connected a new node to my cluster as an agent node, and I noticed the agent node k3s version is v1.27. All the rest of my nodes that have been in the cluster for some time now, including my embedded etcd server control plane nodes are all version v1.26 . is it a problem if my agent node is a newer version of k3s then my control plane nodes ? thankyou everyone and have a nice day and weekend.

has anyone managed to use the helm operator with private s3 buckets?

Hello everybody!

My question is regarding k3s binary functionality: 1. It looks like the binary doesn’t come with a k3s.service, so when I turned off the node it couldn’t reconnect back to the cluster. 2. I went to the K3s-io / k3s git hub page and I found the the default K3s service yaml. I then ensured the service was active, and I turned off the node…. But the node still did not rejoin the cluster. My question, is there a default K3s.service somewhere i can use for server and agent nodes installed through binary method? If so, does anyone know where I could find it? Thankyou - Fareed

https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik#comment-32065 I am folowing this article to test the pod networking of my cluster. It creates a simple hello world application using deployment , service and ingress and it used traefik for ingress . Somehow I am not able to do curl localhost:80 ( ingress is configured to work on :80 ) can anyone please suggest me what has gone wrong here ( service , pods , endpoints all are responding on port 80 )

Is there a reason other than possibly choosing releases unsupported by rancher that the automated upgrades doc states you should not use your own system-upgrade-controller plans when you’re dealing with clusters imported into rancher?

I've been using k3s for a month or so now. When I originally set it up, I set up 3 masters using HA embedded etcd. I'd now like to migrate to an external postgres DB. I can't seem to find any information on migrating from etcd to postgres. Does anyone have any instructions on how to do that, or could point me in a direction?

Hi Everyone had a question regarding certificates generated and its allowed IPs Current Enviroment I have create a CLI tools which can create HA k3s clusters where the controlplanes are joined and are loadbalanced by a HAproxy loadbalancer all works great but after creation the first time it gets a x509 cert error

connection error: Get <https://74.220.22.141:6443/api?timeout=32s>: tls: failed to verify certificate: x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.1.3, 192.168.1.4, 192.168.1.5, 192.168.1.6, ::1, not 74.220.22.141
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for 10.43.0.1, 127.0.0.1, 192.168.1.3, 192.168.1.4, 192.168.1.5, 192.168.1.6, ::1, not 74.220.22.141

as These are running a VM in cloud platforms so public IP are dynamic here the 74.220.22.141 is the publicIP of loadbalancer So the issue gets resolved when I pass in this command

kubectl cluster-info --insecure-skip-tls-verify

after that all the kubectl commands function normally Here is the SHELL command I used to create the control plane with known loadbalancer privateIP

curl -sfL <https://get.k3s.io> | sh -s - server --node-taint CriticalAddonsOnly=true:NoExecute --tls-san 192.168.1.8

Here 192.168.1.8 is provateIP of lb and to add additional controlplane nodes I used these commands

curl -sfL <https://get.k3s.io> | sh -s - server --token=$SECRET --node-taint CriticalAddonsOnly=true:NoExecute --tls-san 192.168.1.8

Here is the HA proxy config

cat <<EOF > /etc/haproxy/haproxy.cfg
frontend kubernetes-frontend
  bind *:6443
  mode tcp
  option tcplog
  timeout client 10s
  default_backend kubernetes-backend

backend kubernetes-backend
  timeout connect 10s
  timeout server 10s
  mode tcp
  option tcp-check
  balance roundrobin
  server k3sserver1 <privateip>:6443 
  server k3sserver2 <privateip>:6443
EOF

once i got the kubeconfig I replaced the server address

newKubeconfig := strings.Replace(kubeconfig, "127.0.0.1", loadBalancer.PublicIP, 1)

does

--tls-san

or

--advertise-address

will be useful and what should be its value?

So the issue gets resolved when I pass in this command but I need to not have to use this command what is the possible solution??

through this page we should consider as latest https://github.com/k3s-io/k3s/releases 1.27.1 or 1.26.4

1.27.1 is the latest release or 1.26.4 is the latest

they are both the latest in their respective release channels

Hi, I have an air gapped Rancher install and K3s cluster, I'm looking at driving K3s upgrades using the system-upgrade-controller and ideally kick an upgrade off from the Rancher UI. Being air gapped I do not get a drop down to select the target K3s version, can a local repo be specified anywhere to populate the list?

Good morning everyone! I was going through more kubernetes documentation today and I came across a component called “Leases” and after reading the page it seems like leases are a good way to have multiple replicas of a deployment for example…. And the “Lease” is configured so all replicas elect only 1 leader to be running I/O at one time and the rest of the replicas just act as standby instances ready to take leadership role if the leader replica pod dies. Is my thinking correct here regarding the usage? Obviously you have to implement code for your applications to use the lease appropriately. Cheers guys!

Hey all I’m having a super weird issue after upgrading my cluster to HA embedded etcd… I still just connect to a single server node for kubectl but it seems pretty often that I’ll get no response or server not responding errors when I do normal operations like listing pods. I’m not sure how to debug this… any ideas? I’ve got a 6 node cluster, 3 of which are server nodes. Centos stream 9 vms on proxmox

Hey all, I am having problem setup k3s node with gpu access. I tried some configs and it works with v1.21.4 but it seems to not work with version 1.25.8 the drivers and nvidia-container-runtime are installed properly and the cluster works but cant get access to gpu resources. I have modified a config.toml.tmpl in /var/lib/rancher/k3s/agent/etc/containerd/ but it seems I'm probably missing something. I have tried with a sample config file in k3d docs for cuda workloads but as I said it only works in older versions. Any tips or idea what I'm missing ?

Hey all! how do you taint control plane nodes for an embeded etcd cluster?

hello there I added a new disk to my server, it is using ZFS I won't be using this disk on kubernetes but k3s is trying to use it somehow and won't start

Hi，k3s master node will access itself's 6443 with Public IP(I setup the master with option --node-external-ip 1.2.3.4 as I has some cross-network nodes) ➜ ~ ss -antp dst 1.2.3.4 State Recv-Q Send-Q Local Address:Port Peer Address:Port Process ESTAB 0 0 192.168.0.153:52288 1.2.3.4:6443 users:(("k3s-server",pid=2027176,fd=191)) ESTAB 0 1702 [:ffff192.168.0.153]:6443 [:ffff1.2.3.4]:11179 users:(("k3s-server",pid=2027176,fd=157)) ESTAB 0 0 [:ffff192.168.0.153]:6443 [:ffff1.2.3.4]:52288 users:(("k3s-server",pid=2027176,fd=268)) How do I avoid this? Thank you :)

Hi All, @agreeable-laptop-45252

Ignore above msg Hi All, We have deployed k3s 1.25 version with longhorn in Ubuntu 18.04 i can see below kernal.logs

May 24 15:18:18 Demo-Box kernel: [1670655.742173] cni0: port 52(vethfa98589d) entered disabled state
May 24 15:18:18 Demo-Box kernel: [1670655.753874] device vethfa98589d left promiscuous mode
May 24 15:18:18 Demo-Box kernel: [1670655.753884] cni0: port 52(vethfa98589d) entered disabled state
May 24 15:18:21 Demo-Box kernel: [1670658.647720] IPv6: ADDRCONF(NETDEV_UP): veth6e07cfbf: link is not ready
May 24 15:18:21 Demo-Box kernel: [1670658.648223] cni0: port 52(veth6e07cfbf) entered blocking state
May 24 15:18:21 Demo-Box kernel: [1670658.648226] cni0: port 52(veth6e07cfbf) entered disabled state
May 24 15:18:21 Demo-Box kernel: [1670658.648402] device veth6e07cfbf entered promiscuous mode
May 24 15:18:21 Demo-Box kernel: [1670658.653907] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
May 24 15:18:21 Demo-Box kernel: [1670658.653935] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
May 24 15:18:21 Demo-Box kernel: [1670658.653997] IPv6: ADDRCONF(NETDEV_CHANGE): veth6e07cfbf: link becomes ready
May 24 15:18:21 Demo-Box kernel: [1670658.654068] cni0: port 52(veth6e07cfbf) entered blocking state
May 24 15:18:21 Demo-Box kernel: [1670658.654070] cni0: port 52(veth6e07cfbf) entered forwarding state
May 24 15:21:19 Demo-Box kernel: [1670836.700175] nfs: server 10.43.252.141 not responding, still trying
May 24 15:21:35 Demo-Box kernel: [1670852.252148] nfs: server 10.43.65.88 not responding, still trying
May 24 15:21:39 Demo-Box kernel: [1670856.860416] nfs: server 10.43.252.141 not responding, still trying
May 24 15:21:55 Demo-Box kernel: [1670872.284298] nfs: server 10.43.174.157 not responding, timed out
May 24 15:21:55 Demo-Box kernel: [1670872.408142] nfs: server 10.43.65.88 not responding, timed out
May 24 15:21:56 Demo-Box kernel: [1670872.988271] nfs: server 10.43.252.141 not responding, timed out
May 24 15:22:03 Demo-Box kernel: [1670880.476079] nfs: server 10.43.65.88 not responding, still trying
May 24 15:22:56 Demo-Box kernel: [1670933.724423] nfs: server 10.43.174.157 not responding, still trying
May 24 15:23:49 Demo-Box kernel: [1670986.717866] INFO: task influxd:27866 blocked for more than 120 seconds.

on the same time we found k3s logs as below, please anyone help on this, much appreciate

Hi team 👋 I was redirected from #rancher (kubernetes) channel. I have a specific question regarding how to access etcd data in a k3s cluster, any help would be appreciated! I set up k3s with High Availability Embedded etcd, and now trying to access etcd data following this instruction. However I get inconsistent results between etcd returned results and kubectl. What I’m trying to do is to get pods count in etcd using

etcdctl get --prefix --keys-only /registry/pods

, and it returns me 7 pods. When I get pods via kubectl across all namespaces, it gives me 8 pods. And the pods’ names don’t match.

Anyone help me on this ? Notes: NFS server issues which is there in logs has been clarified with Longhorn team and it should be fixed by 1.4.2 stable release, but even though k3s cluster went NodeNotReady state, i suspect flannel may be the reason… but not sure. what may be the issue ?

Hey all! is there a way to migrate from embeded etcd back to non HA sqlite?

sad day