https://rancher.com/ logo
Docs
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
k3s
  • c

    creamy-waiter-66684

    09/29/2022, 6:16 PM
    Hello team! I'm investigating k3s as a potential solution for storing a large amount of configuration information for various service teams in my organization. We're considering using Custom Resource Definitions (CRDs) in the API server. K8s was not viable due to the 8GB limit etcd imposes. While the external database feature of k3s solves the size limit, I noticed while inserting a large number of very large Customer Resources (CRs), the k3s server begins to eat a large amount of the systems RAM. I'm inserting around 1 million of these large CRs on an EC2 instance with over 128GB of RAM. At around 140,000 objects inserted, the k3s server is eating over 70% of the systems RAM. Is this intended? I assume k3s server is caching the inserted objects? Is there anyway to alleviate the large amount of memory usage? Even after stopping the insertions, the RAM utilization still sits around 60 or 70 percent (goes does 20-30% from peak).
    c
    • 2
    • 11
  • m

    melodic-hamburger-23329

    09/30/2022, 7:34 AM
    Containerd still fails to boot after manual k3s upgrade… Any way to avoid this happening?
    time="2022-09-30T16:30:47.729342645+09:00" level=fatal msg="failed to create new snapshotter" error="failed to restore remote snapshot: failed to prepare remote snapshot: sha256:08b10ee4e4d584086d7203095776335fc5f3a541402bb19e89e908096b30df2e: failed to resolve layer: failed to resolve layer \"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\" from \"...\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"...\", ref:\"...\", digest:\"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\"): failed to access to the registry with code 404: failed to resolve: failed to resolve target"
    Kind of impossible to do upgrades if need to basically recreate cluster every time :S Am I possibly doing something wrong? Steps: • download latest k3s binary and put to /usr/local/bin • systemctl stop k3s (or k3s-killall.sh; not sure which one recommended?) • systemctl start k3s (or rerun install script; same result)
    c
    b
    • 3
    • 16
  • m

    melodic-hamburger-23329

    09/30/2022, 8:04 AM
    Where can I find definition of /etc/rancher/k3s/config.yaml? Particularly, it seems disable syntax is ambiguous.
    disable:
      - "etcd"
    or
    disable-etcd: true
    https://rancher.com/docs/k3s/latest/en/installation/disable-flags/ https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/
    c
    • 2
    • 10
  • c

    clever-air-65544

    09/30/2022, 4:50 PM
    Weekly report o'clock! https://github.com/k3s-io/k3s/discussions/6202
    👍 4
  • e

    elegant-article-67113

    10/01/2022, 4:03 PM
    Are there any recommended CNI optimizations to avoid getting out of order UDP packets? They always show up in order on bare metal, but can get messed up in containers (this is just on a LAN with a single switch).
  • e

    eager-cartoon-94692

    10/01/2022, 11:39 PM
    Hi all, sorry if this isn't the place to ask a k3d specific question, but it seemed like the best option. Here it goes: Is there a way to specify a cacert that I create to be used for mutual TLS authentication?
    c
    • 2
    • 4
  • p

    prehistoric-diamond-4224

    10/03/2022, 10:31 AM
    Hi there! I cordoned, drained, upgraded the resources of a worker node and restarted it, but k3s still doesn't schedule pods on the upgraded node, whenever i restart some workload they are still assigned to one of the other smaller nodes. To force-schedule pods to the new node I have to cordon all the other ones and delete some pods, only then after some time they are re-scheduled to the under-utilized node, but every time they are restarted they are placed on the other smaller nodes. So now I have one of my nodes sitting at 91% RAM and another at 30%. Apparently resource requests are not affecting the scheduling. Any idea of what could be happening?
  • g

    green-energy-38738

    10/03/2022, 2:15 PM
    1. any one runs k3s on docker ?
    r
    • 2
    • 1
  • c

    chilly-telephone-51989

    10/03/2022, 2:37 PM
    when i have images in a custom registry i have to tag the image and also refer to this custom repo in my yaml files for instance image: 172.16.0.2:5000/myapp:0.3 is there a way to not specify the registry server and k3s by default looks to that address? so it becomes image: myapp:0.3
    k
    q
    • 3
    • 2
  • b

    bright-jordan-61721

    10/03/2022, 3:28 PM
    Recently built a k3s cluster with version
    v1.24.6+k3s1
    and I have some pods configured with
    dnsPolicy: ClusterFirst
    (which is the default) and noticing weird DNS resolution problems. When I shell into a pod with this dns policy and
    cat /etc/resolv.conf
    this is what I see:
    bash-5.1# cat /etc/resolv.conf
    search default.svc.cluster.local svc.cluster.local cluster.local [home search domains redacted]
    nameserver 10.43.0.10
    options ndots:5
    I believe ndots:5 is causing the problem, as
    ping <http://github.com|github.com>
    fails due to dns resolution, but
    ping <http://github.com|github.com>.
    works instead. Why is k3s setting the ndots:5 option by default? I’m not setting this with the pod’s dnsConfig at all. If this option were removed or reduced to ndots:1 it would likely solve my issue.
    b
    k
    • 3
    • 14
  • p

    prehistoric-judge-25958

    10/03/2022, 5:59 PM
    Hi, i have setup a private registry (harbor) inside my k3s cluster. I have pushed images succesfully to this registry and now i want to pull images from the registry. When i put this yaml config below in
    /etc/rancher/k3s/registries.yaml
    my node become in a NotReady state after "systemcl restart k3s"
    mirrors:
      harbor.k8s.lan:
        endpoint:
          - "<https://harbor.k8s.lan:443>"
    configs:
      "harbor.k8s.lan:443":
        tls:
          cert_file: /etc/rancher/k3s/certs/cert.pem
          key_file:  /etc/rancher/k3s/certs/cert-key.pem
          ca_file:   /etc/rancher/k3s/certs/k8s-lan.crt
          insecure_skip_verify: "true"
    I am using self-signed certificates for my k8s.lan domain and put them in the directory /etc/rancher/k3s/certs/ describe node k3s-master-01 output:
    Normal   Starting                 23m                kubelet          Starting kubelet.
      Warning  InvalidDiskCapacity      23m                kubelet          invalid capacity 0 on image filesystem
      Normal   NodeAllocatableEnforced  23m                kubelet          Updated Node Allocatable limit across pods
      Normal   NodeHasSufficientMemory  23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasSufficientMemory
      Normal   NodeHasNoDiskPressure    23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasNoDiskPressure
      Normal   NodeHasSufficientPID     23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasSufficientPID
      Normal   NodeReady                23m                kubelet          Node k3s-master-01 status is now: NodeReady
      Normal   NodeNotReady             20m (x3 over 63m)  node-controller  Node k3s-master-01 status is now: NodeNotReady
    ✅ 1
    c
    • 2
    • 7
  • l

    late-needle-80860

    10/04/2022, 2:04 PM
    I’m looking into adding
    --egress-selector-mode=disabled
    to an already running cluster ( server/control-plane side of course). Is that possible or do one need to redeploy the cluster anew? The reason for this is that I’m see
    failed: error dialing backend: EOF
    when e.g. running the
    connectivity test
    provided by the
    Cilium
    cli
    . When I tried introducing it to a running test cluster I’m getting the infamous
    failed to validate server configuration critical configuration value mismatch
    …. Is other there a workaround to get this in on a live/already running cluster? Thank you very much
    c
    • 2
    • 2
  • a

    adamant-waiter-35487

    10/05/2022, 8:30 AM
    In k3s HA with embedded db, is it possible to have server and agent on the same node? The quick start command (
    curl -sfL <https://get.k3s.io> | sh -
    ) installs both server and agent, but the tutorial on embedded ha ask us to start server on 3 nodes, and then join agent later. I am not sure if this means I need 3 nodes just for ha control plan, and need more nodes to behave as agent (worker) node?
    c
    • 2
    • 3
  • l

    late-needle-80860

    10/05/2022, 6:23 PM
    I managed to get the
    --egress-selector-mode=disabled
    set on the
    servers
    of X K3s cluster I have running - on v1.24.4+k3s1. However, when running the
    cilium connectivity test …
    command for the
    Cilium
    CNI
    I know get the following err:
    unable to start container process: open /dev/pts/0: operation not permitted: unknown
    - troubleshooting that err leads me to: • https://github.com/opencontainers/runc/pull/3554 • and this release: https://github.com/opencontainers/runc/releases/tag/v1.1.4 What
    K3s
    release is that part of - if any?
    g
    • 2
    • 5
  • q

    quiet-chef-27276

    10/06/2022, 1:55 AM
    Cross-posting as I don't seem to be able to delete the original - probably should have been posted here.
    l
    e
    • 3
    • 6
  • l

    late-needle-80860

    10/07/2022, 8:05 AM
    I was wondering - after having used
    --egress-selector-mode=disabled
    on nodes in a cluster. What is one not getting? What’s the downside? In the docs it says:
    The apiserver does not use agent tunnels to communicate with nodes. Requires that servers run agents, and have direct connectivity to the kubelet on agents, or the apiserver will not be able to function access service endpoints or perform kubectl exec and kubectl logs.
    So sounds fine to me. I wasn’t disabling the agent on
    servers
    anyway. So not loosing anything there. Are there any downsides or considerations one should have? Thank you very much
    c
    • 2
    • 6
  • l

    late-needle-80860

    10/07/2022, 10:40 AM
    Someone on here having experience with the joining of K3s worker nodes in cordoned fashion … for different reasons we wish that X joining
    worker
    node joins cordoned in order to different processes to complete in due time before regular workloads starts piling in on the new
    worker
    . Some of these processes might be/are: • the configuration of
    containerd
    for a private self-hosted registry • Longhorn bootstrapping … and storage space setup … which needs to be fully up and ready before potential persistent storage needing regular workloads starts appearing
    c
    • 2
    • 14
  • h

    handsome-painter-48813

    10/07/2022, 10:48 AM
    Hello, I upgraded debian from 10 to 11 and I can't start k3s server:
    failed to find cpuset cgroup (v2)
    k3s check-config:
    Generally Necessary:
    - cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: bad (fail)
        (for cgroups V1/Hybrid on non-Systemd init see <https://github.com/tianon/cgroupfs-mount>)
    - /usr/sbin/apparmor_parser
    apparmor: enabled and tools installed
    I already set
    GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0"
    And it does not work 😕
    l
    c
    • 3
    • 10
  • c

    clever-air-65544

    10/07/2022, 4:45 PM
    k3s weekly is up! https://github.com/k3s-io/k3s/discussions/6229
    👀 2
    👍 2
  • r

    red-boots-23091

    10/08/2022, 11:50 AM
    Hey all, I have a cluster that i created on a docker install within v2.5.7. I have used this for testing for around 1 year now. However at the end of August I started getting problems with it (kubectl connection refused and various crash loops). I cant seem to resolve these. I assume it is to do with updates I made to my server packages. Long story short I am trying to upgrade my Rancher server. However no matter what I do I cant reliably connect to rancher UI and when I can i still have trouble with kubectl. I am running on Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-125-generic x86_64). Docker Engine Version: 20.10.12 and I am trying to spin up rancher v2.6.8. These are my server logs https://pastebin.com/cDgYiJCb. Does anyone have any guidance
    c
    • 2
    • 20
  • e

    enough-carpet-20915

    10/08/2022, 5:58 PM
    I've replaced 3 of my k3s nodes. They have new IPs but the same names as before. I'm now getting this error:
    Error from server: error dialing backend: x509: certificate is valid for 127.0.0.1, 45.x.x.x, 2a02:c206:xxxx:xxxx::1, not 38.x.x.x
    I tried doing
    k3s certificate rotate
    (from https://github.com/k3s-io/k3s/wiki/K3s-Cert-Rotation) which seems to have rotated certs, but I'm still getting the same error.
    c
    • 2
    • 27
  • e

    enough-carpet-20915

    10/08/2022, 6:56 PM
    It doesn't look like it actually changed the certs.
  • e

    enough-carpet-20915

    10/08/2022, 6:56 PM
    admin@marge:~$ sudo k3s certificate rotate
    INFO[0000] Server detected, rotating server certificates 
    INFO[0000] Rotating certificates for admin service      
    INFO[0000] Rotating certificates for etcd service       
    INFO[0000] Rotating certificates for api-server service 
    INFO[0000] Rotating certificates for controller-manager service 
    INFO[0000] Rotating certificates for cloud-controller service 
    INFO[0000] Rotating certificates for scheduler service  
    INFO[0000] Rotating certificates for k3s-server service 
    INFO[0000] Rotating dynamic listener certificate        
    INFO[0000] Rotating certificates for k3s-controller service 
    INFO[0000] Rotating certificates for auth-proxy service 
    INFO[0000] Rotating certificates for kubelet service    
    INFO[0000] Rotating certificates for kube-proxy service 
    INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/k3s/server/tls-1665255335, please restart k3s server or agent to rotate certificates 
    
    admin@marge:~$ sudo diff -sr /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/tls-1665255335/ | grep -i identical | awk '{print $2}' | xargs basename -a | awk 'BEGIN{print "Identical Files:  "}; {print $1}'
    Identical Files:  
    client-ca.crt
    client-ca.key
    dynamic-cert.json
    peer-ca.crt
    peer-ca.key
    server-ca.crt
    server-ca.key
    request-header-ca.crt
    request-header-ca.key
    server-ca.crt
    server-ca.key
    service.key
    apiserver-loopback-client__.crt
    apiserver-loopback-client__.key
    c
    • 2
    • 4
  • g

    gifted-branch-26934

    10/10/2022, 12:12 PM
    hello guys, i have an app deployed in k8s with 2 replicas, the problem is each time i increase the number of replicas, the new replica is directly marked as available and operating. but the real thing that is happening that at the startup it builds a model that takes about 5 mins. for this issue i want the traffic to be routed to the first replica that has been operating for long time, and once the new replica builds the model, then i need to make it receive traffic as well. any idea how to do that?
    a
    • 2
    • 8
  • a

    average-arm-20932

    10/11/2022, 6:36 PM
    Hello Team, During the scanning I found 'weak SSL ciphers', in k3s server, I'm using K3S version 'v1.21.1+k3s1'. I found an official WEB-URL to fix the issue and seems it is for 'Rancher-manager', but do we have any fix for K3S. For Rancher-Manager. https://www.suse.com/c/resolve-cipher-and-ssl-threats-security-scans/ Thanks & Regards.
    c
    • 2
    • 9
  • f

    famous-flag-15098

    10/12/2022, 3:38 PM
    I have a few questions regarding multi-master k3s. I currently have a 6 node home cluster that currently has 2 etcd/masters. When I setup the second master I used the K3S_URL of the first master. Now I would like to turn the original master into an agent node, and then turn two of the existing agents into masters. How should I approach this?
    b
    • 2
    • 2
  • f

    famous-flag-15098

    10/12/2022, 3:38 PM
    I am especially wondering about how the K3S_URL works.
    b
    • 2
    • 2
  • b

    billowy-bird-32869

    10/13/2022, 9:40 AM
    Hello. I am using K3D for K3S and I would like to parametrize from the command line (or even as a variable) the value of a chart manifest used with the autodeploy feature. More explicitly, I have created a HelmChart resource in
    /var/lib/rancher/k3s/server/manifests/
    and in that resource, I have used spec.set to change the value of some property. The value to the property is delivered via an environment variable at the moment as I cannot think of any other way after looking at the options from k3s. Would someone have an idea, how I could parametrize my chart or if the way I have done it is correct, how do I supply the value. Thanks. Sample:
    spec:
      helmVersion: v3
      repo: <https://charts.gitlab.io>
      chart: gitlab-runner
      targetNamespace: gitlab-runners
      set:
        runners.tags: "$my_tag"
  • s

    stale-vegetable-37217

    10/13/2022, 1:52 PM
    Hi, Can someone help me out? I am trying to run a k3s for the first time behind a corporate proxy. The proxy allows the use of ipv4 as well as ipv6 but on different hostnames. After installing awx on the 1 node cluster and checking with kubectl logs I get an proxyconnect tcp error code 503. The full error is printed below. The ipv6 address printed is the one from the server, and port 6443 is running. It seems that k3s is trying to connect to localhost using the proxy (which is not needed). NO_PROXY var is also set. Anyone has an idea as to why that is happening? Full error: 'Error from server: Get "https😕/[2a02:xxxxx]:10250/containerLogs/awx/awx-operator-controller-manager-fcf6db67c-h9s9q/awx-manager?follow=true": proxyconnect tcp: proxy error from [::1]:6443 while dialing proxy.xxx.be:8080, code 503: 503 Service Unavailable'
    c
    • 2
    • 2
  • m

    melodic-hamburger-23329

    10/14/2022, 2:44 AM
    I noticed some discrepancy in the docs. Here
    disable-etcd
    is mentioned but not here. Are there other components missing from the latter list?
    c
    g
    • 3
    • 2
Powered by Linen
Title
m

melodic-hamburger-23329

10/14/2022, 2:44 AM
I noticed some discrepancy in the docs. Here
disable-etcd
is mentioned but not here. Are there other components missing from the latter list?
c

creamy-pencil-82913

10/14/2022, 3:24 AM
The args list on that page may need a re-sync with the code. PR welcome.
✅ 1
g

gray-lawyer-73831

10/14/2022, 3:50 PM
For some history on that, there are some flags (like this
disable-etcd
and similarly below
disable-apiserver
and a few others) that are hidden but available. It may be worth including them in the docs, but they’re generally either not recommended or less supported. In the case of most of these
disable-*
flags though, it probably makes sense for us to unhide them and show them in that doc 🙂
View count: 22