Hi folks! Is there some authoritative source for the base runtime cost of running a k3s server node?

I found a comparison from MicroK8s here but of course there must be some bias there 🙂

Ah, after a lot more digging I found https://docs.k3s.io/reference/resource-profiling - might be good to cross-link that in a more visible way, since all the banner claims about k3s are about size 🙂

Hello everyone! I may be having a very edge-casey issue. I tried setting up k3s on Asahi Linux on an M1 Mac for the fun of it, and I seem to be getting the following issue across the board:

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "<http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>": failed to extract layer sha256:c640e628658788773e4478ae837822c9bc7db5b512442f54286a98ad50f88fd4: mount callback failed on /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount3367908043: signal: segmentation fault: : unknown

Segmentation fault always seems like there is quite something going wrong, and I couldn't find anything related to this when googling (additionally, I have no idea what I'm doing when it comes to k8s), so I thought maybe you people can give me some guidance here. Or is this more of an issue with containerd itself?

Hi community, I have deployed a K3S cluster using Rancher and on top of it have installed cert-manager v1.11.0. All pods are running, however, the cert-manager-webhook pod is logging some errors:

Trace[1068908304]: [30.003276269s] [30.003276269s] END
E0314 15:02:02.236947       1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
W0314 15:03:28.953687       1 reflector.go:424] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
I0314 15:03:28.953816       1 trace.go:219] Trace[516939538]: "Reflector ListAndWatch" name:<http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169> (14-Mar-2023 15:02:58.949) (total time: 30004ms):
Trace[516939538]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout 30004ms (15:03:28.953)
Trace[516939538]: [30.004226263s] [30.004226263s] END
E0314 15:03:28.953837       1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
W0314 15:04:44.919380       1 reflector.go:424] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
I0314 15:04:44.919458       1 trace.go:219] Trace[430405071]: "Reflector ListAndWatch" name:<http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169> (14-Mar-2023 15:04:14.918) (total time: 30000ms):
Trace[430405071]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (15:04:44.919)
Trace[430405071]: [30.000964846s] [30.000964846s] END
E0314 15:04:44.919472       1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout

Can someone explain what's wrong? It feels that I can't fully install a helm chart because of this issue. ( I noticed this issue when was trying to install a helm chart of actions-runner-controller, and the error I got:

Error: Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": failed to call webhook: Post "<https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s>": context deadline exceeded )

As well, here are some logs from the pod of actions-runner-controller:

Warning  FailedMount  17m                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-v48zj secret tmp cert]: timed out waiting for the condition
  Warning  FailedMount  8m32s                kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[tmp cert kube-api-access-v48zj secret]: timed out waiting for the condition
  Warning  FailedMount  6m18s (x5 over 19m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-v48zj]: timed out waiting for the condition
  Warning  FailedMount  103s (x2 over 4m1s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-v48zj secret tmp]: timed out waiting for the condition
  Warning  FailedMount  86s (x18 over 21m)   kubelet            MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found

Thanks in advance,

I seem to have some directories within containers within pods with /etc/pki (and subdirectories) as read-only, so much so that when I try:

root# touch /etc/pki/file
touch: cannot touch '/etc/pki/file': No such file or directory

Which is odd as I'm logged into root, do not have any special mounts on this directory, or any pv/pvc claiming this. Any ideas on where to look? I have considered doing something along the lines of:

$ kubectl -n <namespace> get <pod> -o yaml > config.yml
# <create a new kustomization.yml which patches the current pod with (emptDir: {} and securityContext)
$ kustomize build . | kubectl -n <namespace> apply -f -

But that seems egregious and unnecessary.

@flat-whale-67864 has left the channel

@here: is anyone faced/facing the below issue when spinning up kubernetes cluster using rancher 2.7.

level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/

Hey, I had a k3s that's been running on an ubuntu VM for a couple years now. (single node) I noticed that it recently stopped forwarding egress traffic. I tried switching the flannel mode from the default vxlan to the wireguard-native type. It is definitely using wireguard, but the egress traffic still times out. It's a proxmox VM connected to a bridge device on the proxmox host. The weird thing is that the packets leaving the ubuntu VM still have the container's address set as the source ip. I tried restarting k3s and also tried restarting the whole host. I saw a few similar issues on the github issue tracker. I'll include my iptables-save output in a thread.

Hi all, I am new to k3s. I am trying to set it up in sysbox. • I setup an ubuntu 22.04 vm in Azure • Installed https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository on the VM • Installed https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md#installing-sysbox on the VM • On the VM I ran

docker run --runtime=sysbox-runc -it --rm -P --hostname=syscont nestybox/ubuntu-bionic-systemd-docker:latest

• From with in the System container I ran

docker run --name k3s-server-1 --hostname k3s-server-1 -p 6443:6443 -d rancher/k3s:v1.24.10-k3s1 server

I can see the k3s-server container is running, but all pods are on pending and it doesn’t show the server node. In the logs of the k3s container I see this message/ error:

Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: operation not permitted

Anyone who has experience in setting up k3s in Sysbox? Haven’t been able to find where I can adjust the snapshotter to native for example.

Hi! does anyone have experience with a

k3s cluster

with a k3s server with

public ip

and k3s agents in a private network behind nat (like home net for example)?. • I opened the udp ports

8472,51820,51821

and tcp ports

6443,10250

in my router for allowing connections to the private ip where the agents are located. • I also started the agents with the dynamic ip address given from my isp and the server with the public ip address. but somehow the

traefik ingress controller

or the

Ingress

is not able to forward the incoming requests from the public url

<http://staging.company.org|staging.company.org>

to the agents in my private net. I also created other agents with

public ips

and they are able to serve a

whoami

application though

<http://staging.company.org|staging.company.org>

but when the load balancer selects the pods running inside the nodes on private net, then it just hangs and any answer comes from the pods.

v1.24.10+k3s1

Hi All, I have longhorn cluster with 10 nodes and 10 volume mounts in it with 3 replicas each… now one of my node is unavailable…. now the mount point created and pointed by volumes to that particular node is trying to detach and keep on trying…but as it is unable to reach it cannot make it up…. is this a expected behaviour from longhorn ….if so what is the use of replicas and longhorn will be always depends on mount-point locations , there is no auto recovery strategy from replicas ??

there is no much audience and response in longhorn-logging channel so tried here ? anyone have the quick and smart answer is much appreciated !!!

Does anyone have any pointers on best ways to run a devcontainer in a k3s/k3d cluster?

Hi

I'm using rancher and had a k3s cluster using Postgress and kine. For reasons i lost that cluster, but i have the DB. How do I restore the cluster using that same DB?

i get an error

bootstrap data already found and encrypted with different token

Would you expect k3s to install & run on an AWS ec2 RHEL9 instance? I'm hitting a cryptic (to me) error when the service goes to start:

Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
[root@jb-rhel9 ~]# systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2023-03-23 15:43:07 UTC; 1s ago
       Docs: <https://k3s.io>
    Process: 52199 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=1/FAILURE)
        CPU: 8ms
[root@jb-rhel9 ~]# systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2023-03-23 15:43:23 UTC; 3s ago
       Docs: <https://k3s.io>
    Process: 52207 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=1/FAILURE)
        CPU: 7ms

my troubleshooting:

[root@jb-rhel9 ~]# /bin/sh -xc ! /usr/bin/systemctl is-enabled  nm-cloud-setup.service
[root@jb-rhel9 ~]# echo $?
1
[root@jb-rhel9 ~]# /usr/bin/systemctl is-enabled  nm-cloud-setup.service
enabled
[root@jb-rhel9 ~]# echo $?
0

I'm thinking k3s is refusing to start because the cloud setup service is enabled? This is a poorly handled error if that is the case (a human readable message would be helpful, or doc or other check during prereq checks). If I remove this check install and startup seems to work. I have other installs on rhel 7.8 ec2 instances that work fine but from the doc it appears there may be no official support for rhel 7?

~Hi This is related to setting up multus and accessing machine using network interfaces. Setup Two Machines both in same subnet 10.2.0.0 Machine1 - Ip 10.2.0.4 Machine2 - IP 10.2.0.8 Machine1 with eth0 is installed with k3s along with multus, macvlan. From the sampepod am not able to ping machin2 via net1 (network interface) attached. Could you please help. Details are in the document attached. Let me know any other details needed~

Hi This is related to setting up multus and accessing machine using network interfaces. Setup Two Machines both in same subnet 10.2.0.0 Machine1 - Ip 10.2.0.4 Machine2 - IP 10.2.0.8 Machine1 with eth0 is installed with k3s along with multus, macvlan. From the sampepod am not able to ping machin2 via net1 (network interface) attached. Could you please help. Details are in the document attached. Let me know any other details needed.

Is there a way to disable auto upgrading of a k3s cluster that is imported in rancher?

Hello! I am running k3s in docker and as part of my setup, I am using a private docker registry where I am following this documentation and have declared the following YAML file in

/etc/ranger/k3s/

mirrors:
  <http://itregistry.io|itregistry.io>:
    endpoint:
    - <http://it:5000>

However, when I try to create Deployment (that contains an image located at

<http://it:5000>

I get the following error:

failed to resolve reference "<http://itregistry.io/fix-server:0.1.0|itregistry.io/fix-server:0.1.0>": failed to do request: Head "<https://itregistry.io/v2/fix-server/manifests/0.1.0>": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host

More context: • my docker registry is itself running in docker. My k3s and docker registry are in the same docker network. Things I have tried: • I am able to "ping" the registry from the k3s container e.g.

/ # wget --spider <http://it:5000>
Connecting to it:5000 (192.168.32.2:5000)
remote file exists

//...
/ # wget <http://it:5000/v2/fix-server/manifests/0.1.0>
Connecting to it:5000 (192.168.80.2:5000)
saving to '0.1.0'
0.1.0                100% |********************************************************************| 18278  0:00:00 ETA
'0.1.0' saved
/ #

Does anyone see anything immediately wrong or have any suggestions in how to troubleshoot this issue? Thanks in advance!

I feel like I'm missing something-- I can't reliably bring up a cluster.

The rancher profile for k3s doesn't have a CSI, anyone have the rancher helm repo url for the one with the CSI? I don't really want to use 3rd party

Hi guys, any idea where I can look for DNS issues? Deployd awx but can't resolve our internal gitlab. I can resolve external addresses and also internal/local IPs - but not internal names. (for example our domain is

domain-xx.local

and

10.2.17.21

is the vm (

server1

running k3s.))

kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup 10.2.17.21
Server:    192.168.246.10
Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local

Name:      10.2.17.21
Address 1: 10.2.17.21 server1.domain-xx.local

Lookup with IP is working ✅ Now I'm trying to resolve the same system via name:

kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup server1.domain-xx.local
Server:    192.168.246.10
Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local

nslookup: can't resolve 'server1.domain-xx.local'

Lookup with name is not working. ❌ I'm new to k3s, so bear with me. It's a single k3s node, fresh installed. The host can resolve in both ways. When I'm running:

kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"dnsPolicy":"Default"}}' -- sh -c 'cat /etc/resolv.conf'

to check for the /etc/resolv.conf it shows me the correct entries for the search domain and our dns server. kube-dns (coredns) is runnig and dns server is

192.168.246.10

pod:
coredns-597584b69b-9hspl   1/1     Running   1 (65m ago)   87m

svc:
kube-dns   ClusterIP   192.168.246.10   <none>        53/UDP,53/TCP,9153/TCP   87m

Hi community, What is the best way to apply

--flannel-backend=host-gw

to the existing K3S cluster? Should the file

/etc/systemd/system/multi-user.target.wants/k3s.service

be edited by simply adding

ExecStart=/usr/local/bin/k3s \
  server \
  '--cluster-init' \
  '--flannel-backend=host-gw' \

or is there any other way how that has to be done? The goal is to let hosts access cluster pods. As I understood, that should help. Thanks in advance,

My metal lb stopped working all the sudden for some reason with that new cluster lol

Hi to all, I have an issue with k3s update controller`apply-server-plan-on-k3os-5272-with-3df70977393d76cb31ced-9npqp upgrade sha256sum: can't open '/hostsupervise-daemon': No such file or directory` I manually updated the server but the server-upgrade plan still fails I think there is an `/`after host missing

Anyone used the inbuilt helm-controller to deploy cilium? Since I disabled flannel/netpolicies, the job won't deploy it on the NotReady node. Everything in valuesContent seems to be ignored 😞

Hi, I have a couple of issues since upgrading to 1.23.17 or 1.24/1.25: • first seems be related to https://github.com/k3s-io/k3s/pull/5657 and https://github.com/k3s-io/klipper-lb/issues/38 - after the upgrade svclb daemonset is created in kube-system, but existing svclb is still running in service’s namespace so node-port is not releases and new svclb remains in pending state. Was existing svclb supposed to be removed automatically? • and second issue is with upgrade of klipper-lb from v0.3.5 to v0.4.0 - with v0.3.5 it was possible to connect to nodeport on localhost i.e. curl http://localhost, but in v0.4.0 it only works with host’s IP but not localhost. Could be related to the following line that was added?

iptables -t filter -A FORWARD -d ${dest_ip}/32 -p ${DEST_PROTO} --dport ${DEST_PORT} -j DROP