Hey all! Does anyone know if it's possible to install rootless K3s with Docker?

hi I have SUSE Leap(firewalld/ntfables)/k3s (v1.26.7+k3s1) clean install and non of the containers can reach off-box hosts; eg coredns can't connect to dns servers. If I stop firewalld connectivity works so have I missed something from the doc re firewalls cause I only saw replacing fwd with iptables for RHEL

@echoing-tomato-53055 has left the channel

Hi, I have an issue with pods not being able to communicate across nodes. My environment is like this: • server: WSL2, with all necessary ports open on Windows and NAT-ed to WSL instance • agent: a Gentoo Linux, with all necessary ports open After I installed the server and the agent on their respective hosts. I can see the agent in my k3s environment, and I am able to create pods in both nodes. However, the pods on the agent cannot access the pods on the server node. Pods on the agent cannot access the DNS server and cannot resolve the internal pods' service name. I monitored the firewall log on both nodes, and no packages dropped between the two nodes. I checked the k3s service logs on both nodes but could not find any obvious error.

I just installed a vanilla k3s install (using

--local

) and metrics server is failing with "no route to host localhost.localdomain" I wonder what the best way to troubleshoot this is... ? I tried editing the

k3s.service

unit file and then realized that it's probably a simple problem (k3s n Fedora 38 server)

Ahhh, spankin' new:

NAME                    STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                           KERNEL-VERSION           CONTAINER-RUNTIME
localhost.localdomain   Ready    control-plane,master   86s   v1.27.4+k3s1   192.168.86.41   <none>        Fedora Linux 38 (Server Edition)   6.4.13-200.fc38.x86_64   <containerd://1.7.1-k3s1>

I feel like I should have a better domain, any suggestions?

homelab

.myhomelabdomain.io with A records using my private address?

Hello, k3s environment is behind a corp http/https proxy. The VM where k3s is installed has been set up with http/https/no_proxy and HTTP/HTTPS/NO_PROXY environment variables (via /etc/environment). However, the pods seem to fail to connect to the internet.

cto@vm01:~$ kubectl run cluster-tester -it --rm --restart=Never --image=busybox:latest
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ # wget <http://google.com|google.com>
Connecting to <http://google.com|google.com> (142.250.113.101:80)
wget: can't connect to remote host (142.250.113.101): Connection timed out

When I manually set the http/https proxy for the pod, it works.

/ # export http_proxy=<http://proxy_ip>:port
/ # export https_proxy=<http://proxy_ip>:port
/ # wget <http://google.com|google.com>
Connecting to proxy_ip:port (proxy_ip:port)
saving to 'index.html'
index.html           100% |*********************************************************************************************************************************| 19265  0:00:00 ETA
'index.html' saved
/ #

How do I set proxy details so that, it's available for every pod that gets created in this environment?

I have an installation of k3s (version: v1.27.4+k3s1) on proxmox VMs (ubuntu 22.04). I installed cilium 1.14.1 with default config (only non default is

ipam:.operator.clusterPoolIPv4PodCIDRList={{ pod_cidr }}

. After that each time I run install script (

curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC=agent INSTALL_K3S_SKIP_START=false INSTALL_K3S_SKIP_ENABLE=false K3S_TOKEN=REDACTED K3S_URL="<https://192.168.189.108:6443>" sh -

)on a worker node, this node gets networking broken. To fix it I have to reset VM. It doesn't happen when k3s' default flannel is used. It doesn't affect also control-plane nodes. Last entries reported using journalctl before networking is frozen:

Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: NETFILTER_CFG table=mangle:305 family=2 entries=18 op=nft_register_chain pid=13896 subj=unconfined comm="iptables-restor"
Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: NETFILTER_CFG table=mangle:305 family=2 entries=13 op=nft_unregister_table pid=13896 subj=unconfined comm="iptables-restor"
Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: SYSCALL arch=c000003e syscall=46 success=yes exit=4236 a0=3 a1=7ffcc5b52480 a2=0 a3=7ffcc5b5246c items=0 ppid=13725 pid=13896 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=8 comm="iptables-restor" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
Sep 05 22:03:45 k3s-dev-worker-0 audit: PROCTITLE proctitle="iptables-restore"
Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: NETFILTER_CFG table=raw:306 family=2 entries=14 op=nft_register_chain pid=13896 subj=unconfined comm="iptables-restor"
Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: NETFILTER_CFG table=raw:306 family=2 entries=12 op=nft_unregister_table pid=13896 subj=unconfined comm="iptables-restor"
Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: SYSCALL arch=c000003e syscall=46 success=yes exit=4820 a0=3 a1=7ffcc5b52480 a2=0 a3=5f183649a000 items=0 ppid=13725 pid=13896 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts5 ses=8 comm="iptables-restor" exe="/usr/sbin/xtables-nft-multi" subj=unconfined key=(null)
Sep 05 22:03:45 k3s-dev-worker-0 audit: PROCTITLE proctitle="iptables-restore"
Sep 05 22:03:45 k3s-dev-worker-0 audit[13896]: NETFILTER_CFG table=filter:307 family=2 entries=21 op=nft_register_chain pid=13896 subj=unconfined comm="iptables-restor"

Seems like the same happens when I run on worker node

/usr/local/bin/k3s-killall.sh

Article https://docs.k3s.io/upgrades/killall says that standard upgrade doesn't terminate the containers, but from my observations this is only true for master nodes. Worker nodes get pods killed.

To monitor PVC on aks I was able to monitor as kubelet_volum_stats_capacity_bytes was visible in metrics, but when are using local-path storageclass on onprem k3s this metric is not visible and not able to monitor pvc, can anyone suggest how we can achieve it

started poking at k3s not long ago to support a new project and absolutely loving the experience so far

I’m currently trying to install k3s across two different networks, with a nat in the middle but i’m running into problems with advertise address. I have the servers on one network, behind the nat and the agents on the other side. the connection to :6443 works initially, but then it breaks and the agent tries to connect to the wrong ip

Hi,all I install k3s at the time

2023-09-07

.But I discovered that the system time was incorrect later.So,I used

date -s 2023-08-20

to adjust the system time to past. Finally,I found the k3s is not working properly,like this:

Unable to connect to the server: tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2023-08-20T00:00:47Z is before 2023-09-07T09:08:43Z

So,how to solve this error?Is there any better way to solve this error besides reinstalling it.

Dears, I have one question about k3s native embed loadbalancer, just want to know what's the difference with other non cloud provider LB? And how it loadbalance traffic to differ nodes from certain daemonset svrlb?

Oops, I meant to post the question here: https://rancher-users.slack.com/archives/C3ASABBD1/p1694118386545619 instead of #general

Hi everyone! I need to setup an authorization webhook for the apiserver - where can I find any examples on how to achieve this use case with k3s? Thanks for any help!

Hello Everyone, I am using CIS scan chart by Rancher over my k3s cluster. I can see some of the scan's failed . For example Ensure that the --audit-log-path argument is set (Automated) . Is there a way to implement the fix via Rancher? . Any help would be appreciated. Thanks

I am but a humble homelabber using my

k3s

cluster to practice for my CKA and I'm looking for advice on the best traefik Ingress configuration.... When I look at https://docs.k3s.io/networking#traefik-ingress-controller it mentions

/var/lib/rancher/k3s/server/manifests/traefik.yaml

but that directory doesn't exist. I only have

/var/lib/rancher/k3s/server

I'm trying to get this hello-world tutorial to run on my IPv6-only VPS. The last step,

curl localhost:80

, times out. What could be wrong?

*   Trying 127.0.0.1:80...
* connect to 127.0.0.1 port 80 failed: Connection refused
*   Trying ::1:80...

k3s is such a sweet disto... thank you. Good night.

Hi k3s community, my team is running k3s v1.24.14+k3s1 with flannel vxlan backend and experiencing issues following an upgrade from Ubuntu bionic to Ubuntu jammy Jellyfish: we observe DNS resolution issues when trying to reach the core dns pod from pods running on a different host: UDP 53 packets seem to reach the core dns pod (as we see from strace) but responses are never received by the client. ICMP traffic across the client pod and coredns pod is ok when testing though

ping

. We suspected an iptable-related issue with nft backend being used by default in ubuntu jammy. However, inspecting iptable counters we are not observing any count for DROP or REJECT rules (using

iptables -L -v

). We also tried passing the k3s flag

--prefer-bundled-bin

without more luck. Any other idea on how to troubleshoot this issue ? Thanks in advance for your help.

what could be the reason for not being able to resolve any DNS names from newly joined worker node? DNS resolution works fine from master node and old worker node. All works fine when I add

dnsConfig

stanza to the pod, without it, it doesn't.

Hello! I set up a k3s cluster over tailscale because I'm planning to add nodes from other networks in the future and I just saw that pods cannot connect to other pods that are not on the same node, does anyone know what the issue might be?

So I’m reaching out to know what the best practice is in regards to the use of

—tls-san

parameter. • should I configure / set it on all server nodes? • And if I upgrade control-plane/server nodes by replacing them one by one ( will end up with the same name and ip ) should I set the parameter in that scenario as well? —- With the v1.28.1 release ( and others ) this parameter have come into focus because of the need to set it to patch the cert. CVE on k3s … Thank you

dear k3s users, i'm trying to debug some service connection issues. when i use the kubectl proxy

kubectl port-forward

i can get any agent node to connect to a k3s service just fine. but if i start a pod, the pod can't seem to access the service. is there a good way to debug this issue? im using default flannel config, maybe it's an iptables issue?

Hi community, i'm observing some warning trace message about slow database like below: ---"Write to database call finished" len:1034400,err:<nil> 629ms (09:04:31.877) but when i checked stats from database, i couldn't find any thing similar to above log k3s=# select max_exec_time from pg_stat_statements order by max_exec_time desc limit 5; max_exec_time -------------------- 185.297388 71.854067 67.12399400000001 28.610996999999998 24.494908 (5 rows) k3s=# am I doing anything wrong here? thanks a lot

Dear community, could you please help me here to give me some confident to file a bug ticket? 😄 i’m observing some warning trace message about slow database like below: ---“Write to database call finished” len1034400,err<nil> 629ms (090431.877) but when i checked stats from database, i couldn’t find any thing similar to above log k3s=# select max_exec_time from pg_stat_statements order by max_exec_time desc limit 5; max_exec_time -------------------- 185.297388 71.854067 67.12399400000001 28.610996999999998 24.494908 (5 rows) k3s=# am I doing anything wrong here? thanks a lot

Hi Team, do you know which parameter will influence the numbers of database client connection initiated from k3s api-server? on my node, i can’t only see 3 connections, no matter what i did. ps -ef | grep “postgres: k3s” postgres 7053 20352 4 10:21 ? 000006 postgres: k3s k3s 10.10.202.1(38498) idle postgres 7494 20352 3 10:21 ? 000006 postgres: k3s k3s 10.10.202.1(38526) idle postgres 20898 20352 0 10:01 ? 000000 postgres: k3s k3s 10.10.202.1(42816) idle

Dear community, I installed K3s with ingress-nginx and I need to expose a TCP service of mongo which is on 27017. I created the ConfigMap for tcp-services by following this doc and enabled "use-proxy-headers". But as soon as I turn it on, the other normal services on http ports began to crash with "TCP 400 Broken header" errors. If I turn it off, the mongo server starts to crash with "TCP 400 Broken header while Proxy" and other services begin to work

Hi, what's the best way to modify an existing (rancher provisioned) k3s cluster to split control-plane / etc roles from worker? I had 3 all in one nodes and 1 worker only - then I scaled up the worker nodes to 5 so I would now like to remove the worker role from the 3 control plane nodes and also make sure that in the future new workloads are not deployed on the control plane / etcd nodes?

facing issue with prefer bundled bin These are documented here: https://docs.k3s.io/known-issues#iptables And a solution have applied the prefer-bundled-bin to use in config.yaml file passed to k3s server

rootless: true
prefer-bundled-bin: true

OS config

ubuntu@ip-172-31-34-14:~$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

k3s check-config is still utilizing host ip tables:

ubuntu@ip-172-31-34-14:~$ k3s check-config

Verifying binaries in /var/lib/rancher/k3s/data/42d7f04c5f9849f0016361b8fd9226a044204a3bca9f576ac8aabe93d2560386/bin:
- sha256sum: good
- links: good

System:
- /usr/sbin iptables v1.8.7 (nf_tables): ok
- swap: disabled
- routes: ok

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

...
...

STATUS: pass

Can anyone please explain why the prefer-bundled-bin not getting enforced?