https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
k3s
  • i

    important-kitchen-32874

    03/15/2023, 1:45 PM
    Hi folks! Is there some authoritative source for the base runtime cost of running a k3s server node?
  • i

    important-kitchen-32874

    03/15/2023, 1:51 PM
    I found a comparison from MicroK8s here but of course there must be some bias there 🙂
    n
    • 2
    • 4
  • i

    important-kitchen-32874

    03/15/2023, 1:57 PM
    Ah, after a lot more digging I found https://docs.k3s.io/reference/resource-profiling - might be good to cross-link that in a more visible way, since all the banner claims about k3s are about size 🙂
  • d

    delightful-author-23241

    03/15/2023, 11:04 PM
    Hello everyone! I may be having a very edge-casey issue. I tried setting up k3s on Asahi Linux on an M1 Mac for the fun of it, and I seem to be getting the following issue across the board:
    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "rancher/mirrored-pause:3.6": failed to pull image "rancher/mirrored-pause:3.6": failed to pull and unpack image "<http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>": failed to extract layer sha256:c640e628658788773e4478ae837822c9bc7db5b512442f54286a98ad50f88fd4: mount callback failed on /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount3367908043: signal: segmentation fault: : unknown
    Segmentation fault always seems like there is quite something going wrong, and I couldn't find anything related to this when googling (additionally, I have no idea what I'm doing when it comes to k8s), so I thought maybe you people can give me some guidance here. Or is this more of an issue with containerd itself?
    c
    • 2
    • 9
  • b

    breezy-autumn-81048

    03/16/2023, 11:44 AM
    Hi community, I have deployed a K3S cluster using Rancher and on top of it have installed cert-manager v1.11.0. All pods are running, however, the cert-manager-webhook pod is logging some errors:
    Trace[1068908304]: [30.003276269s] [30.003276269s] END
    E0314 15:02:02.236947       1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
    W0314 15:03:28.953687       1 reflector.go:424] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
    I0314 15:03:28.953816       1 trace.go:219] Trace[516939538]: "Reflector ListAndWatch" name:<http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169> (14-Mar-2023 15:02:58.949) (total time: 30004ms):
    Trace[516939538]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout 30004ms (15:03:28.953)
    Trace[516939538]: [30.004226263s] [30.004226263s] END
    E0314 15:03:28.953837       1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
    W0314 15:04:44.919380       1 reflector.go:424] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
    I0314 15:04:44.919458       1 trace.go:219] Trace[430405071]: "Reflector ListAndWatch" name:<http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169> (14-Mar-2023 15:04:14.918) (total time: 30000ms):
    Trace[430405071]: ---"Objects listed" error:Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout 30000ms (15:04:44.919)
    Trace[430405071]: [30.000964846s] [30.000964846s] END
    E0314 15:04:44.919472       1 reflector.go:140] <http://k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169|k8s.io/client-go@v0.26.0/tools/cache/reflector.go:169>: Failed to watch *v1.Secret: failed to list *v1.Secret: Get "<https://10.43.0.1:443/api/v1/namespaces/cert-manager/secrets?fieldSelector=metadata.name%3Dcert-manager-webhook-ca&resourceVersion=360915>": dial tcp 10.43.0.1:443: i/o timeout
    Can someone explain what's wrong? It feels that I can't fully install a helm chart because of this issue. ( I noticed this issue when was trying to install a helm chart of actions-runner-controller, and the error I got:
    Error: Internal error occurred: failed calling webhook "<http://webhook.cert-manager.io|webhook.cert-manager.io>": failed to call webhook: Post "<https://cert-manager-webhook.cert-manager.svc:443/mutate?timeout=10s>": context deadline exceeded )
    As well, here are some logs from the pod of actions-runner-controller:
    Warning  FailedMount  17m                  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[kube-api-access-v48zj secret tmp cert]: timed out waiting for the condition
      Warning  FailedMount  8m32s                kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[tmp cert kube-api-access-v48zj secret]: timed out waiting for the condition
      Warning  FailedMount  6m18s (x5 over 19m)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[secret tmp cert kube-api-access-v48zj]: timed out waiting for the condition
      Warning  FailedMount  103s (x2 over 4m1s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-v48zj secret tmp]: timed out waiting for the condition
      Warning  FailedMount  86s (x18 over 21m)   kubelet            MountVolume.SetUp failed for volume "cert" : secret "actions-runner-controller-serving-cert" not found
    Thanks in advance,
    r
    • 2
    • 12
  • w

    wonderful-crayon-55427

    03/16/2023, 2:19 PM
    I seem to have some directories within containers within pods with /etc/pki (and subdirectories) as read-only, so much so that when I try:
    root# touch /etc/pki/file
    touch: cannot touch '/etc/pki/file': No such file or directory
    Which is odd as I'm logged into root, do not have any special mounts on this directory, or any pv/pvc claiming this. Any ideas on where to look? I have considered doing something along the lines of:
    $ kubectl -n <namespace> get <pod> -o yaml > config.yml
    # <create a new kustomization.yml which patches the current pod with (emptDir: {} and securityContext)
    $ kustomize build . | kubectl -n <namespace> apply -f -
    But that seems egregious and unnecessary.
  • f

    flat-whale-67864

    03/16/2023, 7:56 PM
    @flat-whale-67864 has left the channel
  • e

    echoing-tomato-53055

    03/17/2023, 5:30 PM
    @here: is anyone faced/facing the below issue when spinning up kubernetes cluster using rancher 2.7.
    level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/
  • b

    bored-horse-3670

    03/20/2023, 2:28 PM
    Hey, I had a k3s that's been running on an ubuntu VM for a couple years now. (single node) I noticed that it recently stopped forwarding egress traffic. I tried switching the flannel mode from the default vxlan to the wireguard-native type. It is definitely using wireguard, but the egress traffic still times out. It's a proxmox VM connected to a bridge device on the proxmox host. The weird thing is that the packets leaving the ubuntu VM still have the container's address set as the source ip. I tried restarting k3s and also tried restarting the whole host. I saw a few similar issues on the github issue tracker. I'll include my iptables-save output in a thread.
    c
    • 2
    • 9
  • c

    careful-honey-96496

    03/20/2023, 7:38 PM
    Hi all, I am new to k3s. I am trying to set it up in sysbox. • I setup an ubuntu 22.04 vm in Azure • Installed https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository on the VM • Installed https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-package.md#installing-sysbox on the VM • On the VM I ran
    docker run --runtime=sysbox-runc -it --rm -P --hostname=syscont nestybox/ubuntu-bionic-systemd-docker:latest
    • From with in the System container I ran
    docker run --name k3s-server-1 --hostname k3s-server-1 -p 6443:6443 -d rancher/k3s:v1.24.10-k3s1 server
    I can see the k3s-server container is running, but all pods are on pending and it doesn’t show the server node. In the logs of the k3s container I see this message/ error:
    Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: operation not permitted
    Anyone who has experience in setting up k3s in Sysbox? Haven’t been able to find where I can adjust the snapshotter to native for example.
    c
    • 2
    • 2
  • b

    brash-controller-15153

    03/21/2023, 5:04 PM
    Hi! does anyone have experience with a
    k3s cluster
    with a k3s server with
    public ip
    and k3s agents in a private network behind nat (like home net for example)?. • I opened the udp ports
    8472,51820,51821
    and tcp ports
    6443,10250
    in my router for allowing connections to the private ip where the agents are located. • I also started the agents with the dynamic ip address given from my isp and the server with the public ip address. but somehow the
    traefik ingress controller
    or the
    Ingress
    is not able to forward the incoming requests from the public url
    <http://staging.company.org|staging.company.org>
    to the agents in my private net. I also created other agents with
    public ips
    and they are able to serve a
    whoami
    application though
    <http://staging.company.org|staging.company.org>
    but when the load balancer selects the pods running inside the nodes on private net, then it just hangs and any answer comes from the pods.
    v1.24.10+k3s1
    c
    p
    • 3
    • 25
  • b

    bitter-tailor-6977

    03/22/2023, 12:30 PM
    Hi All, I have longhorn cluster with 10 nodes and 10 volume mounts in it with 3 replicas each… now one of my node is unavailable…. now the mount point created and pointed by volumes to that particular node is trying to detach and keep on trying…but as it is unable to reach it cannot make it up…. is this a expected behaviour from longhorn ….if so what is the use of replicas and longhorn will be always depends on mount-point locations , there is no auto recovery strategy from replicas ??
  • b

    bitter-tailor-6977

    03/22/2023, 12:32 PM
    there is no much audience and response in longhorn-logging channel so tried here ? anyone have the quick and smart answer is much appreciated !!!
  • h

    handsome-salesclerk-54324

    03/22/2023, 1:18 PM
    Does anyone have any pointers on best ways to run a devcontainer in a k3s/k3d cluster?
    r
    r
    • 3
    • 4
  • w

    wonderful-rain-13345

    03/23/2023, 12:32 AM
    Hi
  • w

    wonderful-rain-13345

    03/23/2023, 12:34 AM
    I'm using rancher and had a k3s cluster using Postgress and kine. For reasons i lost that cluster, but i have the DB. How do I restore the cluster using that same DB?
  • w

    wonderful-rain-13345

    03/23/2023, 1:19 AM
    i get an error
    bootstrap data already found and encrypted with different token
    c
    • 2
    • 12
  • j

    jolly-state-39751

    03/23/2023, 4:08 PM
    Would you expect k3s to install & run on an AWS ec2 RHEL9 instance? I'm hitting a cryptic (to me) error when the service goes to start:
    Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
    [INFO]  systemd: Starting k3s
    Job for k3s.service failed because the control process exited with error code.
    See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.
    [root@jb-rhel9 ~]# systemctl status k3s.service
    ● k3s.service - Lightweight Kubernetes
         Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
         Active: activating (auto-restart) (Result: exit-code) since Thu 2023-03-23 15:43:07 UTC; 1s ago
           Docs: <https://k3s.io>
        Process: 52199 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=1/FAILURE)
            CPU: 8ms
    [root@jb-rhel9 ~]# systemctl status k3s.service
    ● k3s.service - Lightweight Kubernetes
         Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: disabled)
         Active: activating (auto-restart) (Result: exit-code) since Thu 2023-03-23 15:43:23 UTC; 3s ago
           Docs: <https://k3s.io>
        Process: 52207 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=1/FAILURE)
            CPU: 7ms
    my troubleshooting:
    [root@jb-rhel9 ~]# /bin/sh -xc ! /usr/bin/systemctl is-enabled  nm-cloud-setup.service
    [root@jb-rhel9 ~]# echo $?
    1
    [root@jb-rhel9 ~]# /usr/bin/systemctl is-enabled  nm-cloud-setup.service
    enabled
    [root@jb-rhel9 ~]# echo $?
    0
    I'm thinking k3s is refusing to start because the cloud setup service is enabled? This is a poorly handled error if that is the case (a human readable message would be helpful, or doc or other check during prereq checks). If I remove this check install and startup seems to work. I have other installs on rhel 7.8 ec2 instances that work fine but from the doc it appears there may be no official support for rhel 7?
    c
    • 2
    • 2
  • n

    nutritious-vase-53845

    03/23/2023, 4:19 PM
    ~Hi This is related to setting up multus and accessing machine using network interfaces. Setup Two Machines both in same subnet 10.2.0.0 Machine1 - Ip 10.2.0.4 Machine2 - IP 10.2.0.8 Machine1 with eth0 is installed with k3s along with multus, macvlan. From the sampepod am not able to ping machin2 via net1 (network interface) attached. Could you please help. Details are in the document attached. Let me know any other details needed~
  • n

    nutritious-vase-53845

    03/23/2023, 4:20 PM
    Hi This is related to setting up multus and accessing machine using network interfaces. Setup Two Machines both in same subnet 10.2.0.0 Machine1 - Ip 10.2.0.4 Machine2 - IP 10.2.0.8 Machine1 with eth0 is installed with k3s along with multus, macvlan. From the sampepod am not able to ping machin2 via net1 (network interface) attached. Could you please help. Details are in the document attached. Let me know any other details needed.
    k3s-multus.txt
  • m

    many-evening-49066

    03/23/2023, 5:11 PM
    Is there a way to disable auto upgrading of a k3s cluster that is imported in rancher?
  • m

    miniature-honey-15162

    03/23/2023, 8:15 PM
    Hello! I am running k3s in docker and as part of my setup, I am using a private docker registry where I am following this documentation and have declared the following YAML file in
    /etc/ranger/k3s/
    mirrors:
      <http://itregistry.io|itregistry.io>:
        endpoint:
        - <http://it:5000>
    However, when I try to create Deployment (that contains an image located at
    <http://it:5000>
    I get the following error:
    failed to resolve reference "<http://itregistry.io/fix-server:0.1.0|itregistry.io/fix-server:0.1.0>": failed to do request: Head "<https://itregistry.io/v2/fix-server/manifests/0.1.0>": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host
    More context: • my docker registry is itself running in docker. My k3s and docker registry are in the same docker network. Things I have tried: • I am able to "ping" the registry from the k3s container e.g.
    / # wget --spider <http://it:5000>
    Connecting to it:5000 (192.168.32.2:5000)
    remote file exists
    
    //...
    / # wget <http://it:5000/v2/fix-server/manifests/0.1.0>
    Connecting to it:5000 (192.168.80.2:5000)
    saving to '0.1.0'
    0.1.0                100% |********************************************************************| 18278  0:00:00 ETA
    '0.1.0' saved
    / #
    Does anyone see anything immediately wrong or have any suggestions in how to troubleshoot this issue? Thanks in advance!
    c
    • 2
    • 19
  • w

    wonderful-rain-13345

    03/24/2023, 12:01 AM
    I feel like I'm missing something-- I can't reliably bring up a cluster.
    h
    c
    p
    • 4
    • 120
  • w

    wonderful-rain-13345

    03/24/2023, 3:21 PM
    The rancher profile for k3s doesn't have a CSI, anyone have the rancher helm repo url for the one with the CSI? I don't really want to use 3rd party
  • f

    flaky-branch-26433

    03/24/2023, 7:52 PM
    Hi guys, any idea where I can look for DNS issues? Deployd awx but can't resolve our internal gitlab. I can resolve external addresses and also internal/local IPs - but not internal names. (for example our domain is
    domain-xx.local
    and
    10.2.17.21
    is the vm (
    server1
    running k3s.))
    kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup 10.2.17.21
    Server:    192.168.246.10
    Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local
    
    Name:      10.2.17.21
    Address 1: 10.2.17.21 server1.domain-xx.local
    Lookup with IP is working ✅ Now I'm trying to resolve the same system via name:
    kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup server1.domain-xx.local
    Server:    192.168.246.10
    Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local
    
    nslookup: can't resolve 'server1.domain-xx.local'
    Lookup with name is not working. ❌ I'm new to k3s, so bear with me. It's a single k3s node, fresh installed. The host can resolve in both ways. When I'm running:
    kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"dnsPolicy":"Default"}}' -- sh -c 'cat /etc/resolv.conf'
    to check for the /etc/resolv.conf it shows me the correct entries for the search domain and our dns server. kube-dns (coredns) is runnig and dns server is
    192.168.246.10
    pod:
    coredns-597584b69b-9hspl   1/1     Running   1 (65m ago)   87m
    
    svc:
    kube-dns   ClusterIP   192.168.246.10   <none>        53/UDP,53/TCP,9153/TCP   87m
    p
    c
    • 3
    • 35
  • b

    breezy-autumn-81048

    03/24/2023, 10:05 PM
    Hi community, What is the best way to apply
    --flannel-backend=host-gw
    to the existing K3S cluster? Should the file
    /etc/systemd/system/multi-user.target.wants/k3s.service
    be edited by simply adding
    ExecStart=/usr/local/bin/k3s \
      server \
      '--cluster-init' \
      '--flannel-backend=host-gw' \
    or is there any other way how that has to be done? The goal is to let hosts access cluster pods. As I understood, that should help. Thanks in advance,
    c
    w
    • 3
    • 19
  • w

    wonderful-rain-13345

    03/25/2023, 12:16 AM
    My metal lb stopped working all the sudden for some reason with that new cluster lol
  • a

    aloof-dog-75479

    03/25/2023, 9:16 AM
    Hi to all, I have an issue with k3s update controller`apply-server-plan-on-k3os-5272-with-3df70977393d76cb31ced-9npqp upgrade sha256sum: can't open '/hostsupervise-daemon': No such file or directory` I manually updated the server but the server-upgrade plan still fails I think there is an `/`after host missing
    r
    c
    • 3
    • 10
  • a

    adorable-engine-54231

    03/27/2023, 3:12 PM
    Anyone used the inbuilt helm-controller to deploy cilium? Since I disabled flannel/netpolicies, the job won't deploy it on the NotReady node. Everything in valuesContent seems to be ignored 😞
    • 1
    • 1
  • g

    gorgeous-belgium-64567

    03/28/2023, 2:36 PM
    Hi, I have a couple of issues since upgrading to 1.23.17 or 1.24/1.25: • first seems be related to https://github.com/k3s-io/k3s/pull/5657 and https://github.com/k3s-io/klipper-lb/issues/38 - after the upgrade svclb daemonset is created in kube-system, but existing svclb is still running in service’s namespace so node-port is not releases and new svclb remains in pending state. Was existing svclb supposed to be removed automatically? • and second issue is with upgrade of klipper-lb from v0.3.5 to v0.4.0 - with v0.3.5 it was possible to connect to nodeport on localhost i.e. curl http://localhost, but in v0.4.0 it only works with host’s IP but not localhost. Could be related to the following line that was added?
    iptables -t filter -A FORWARD -d ${dest_ip}/32 -p ${DEST_PROTO} --dport ${DEST_PORT} -j DROP
Powered by Linen
Title
g

gorgeous-belgium-64567

03/28/2023, 2:36 PM
Hi, I have a couple of issues since upgrading to 1.23.17 or 1.24/1.25: • first seems be related to https://github.com/k3s-io/k3s/pull/5657 and https://github.com/k3s-io/klipper-lb/issues/38 - after the upgrade svclb daemonset is created in kube-system, but existing svclb is still running in service’s namespace so node-port is not releases and new svclb remains in pending state. Was existing svclb supposed to be removed automatically? • and second issue is with upgrade of klipper-lb from v0.3.5 to v0.4.0 - with v0.3.5 it was possible to connect to nodeport on localhost i.e. curl http://localhost, but in v0.4.0 it only works with host’s IP but not localhost. Could be related to the following line that was added?
iptables -t filter -A FORWARD -d ${dest_ip}/32 -p ${DEST_PROTO} --dport ${DEST_PORT} -j DROP
View count: 4