rancher-users #k3s

creamy-waiter-66684

09/29/2022, 6:16 PM

Hello team! I'm investigating k3s as a potential solution for storing a large amount of configuration information for various service teams in my organization. We're considering using Custom Resource Definitions (CRDs) in the API server. K8s was not viable due to the 8GB limit etcd imposes. While the external database feature of k3s solves the size limit, I noticed while inserting a large number of very large Customer Resources (CRs), the k3s server begins to eat a large amount of the systems RAM. I'm inserting around 1 million of these large CRs on an EC2 instance with over 128GB of RAM. At around 140,000 objects inserted, the k3s server is eating over 70% of the systems RAM. Is this intended? I assume k3s server is caching the inserted objects? Is there anyway to alleviate the large amount of memory usage? Even after stopping the insertions, the RAM utilization still sits around 60 or 70 percent (goes does 20-30% from peak).

melodic-hamburger-23329

09/30/2022, 7:34 AM

Containerd still fails to boot after manual k3s upgrade… Any way to avoid this happening?

Copy code

time="2022-09-30T16:30:47.729342645+09:00" level=fatal msg="failed to create new snapshotter" error="failed to restore remote snapshot: failed to prepare remote snapshot: sha256:08b10ee4e4d584086d7203095776335fc5f3a541402bb19e89e908096b30df2e: failed to resolve layer: failed to resolve layer \"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\" from \"...\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"...\", ref:\"...\", digest:\"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\"): failed to access to the registry with code 404: failed to resolve: failed to resolve target"

Kind of impossible to do upgrades if need to basically recreate cluster every time :S Am I possibly doing something wrong? Steps: • download latest k3s binary and put to /usr/local/bin • systemctl stop k3s (or k3s-killall.sh; not sure which one recommended?) • systemctl start k3s (or rerun install script; same result)

melodic-hamburger-23329

09/30/2022, 8:04 AM

Where can I find definition of /etc/rancher/k3s/config.yaml? Particularly, it seems disable syntax is ambiguous.

Copy code

disable:
  - "etcd"

Copy code

disable-etcd: true

https://rancher.com/docs/k3s/latest/en/installation/disable-flags/ https://rancher.com/docs/k3s/latest/en/installation/install-options/server-config/

clever-air-65544

09/30/2022, 4:50 PM

Weekly report o'clock! https://github.com/k3s-io/k3s/discussions/6202

👍 4

elegant-article-67113

10/01/2022, 4:03 PM

Are there any recommended CNI optimizations to avoid getting out of order UDP packets? They always show up in order on bare metal, but can get messed up in containers (this is just on a LAN with a single switch).

eager-cartoon-94692

10/01/2022, 11:39 PM

Hi all, sorry if this isn't the place to ask a k3d specific question, but it seemed like the best option. Here it goes: Is there a way to specify a cacert that I create to be used for mutual TLS authentication?

prehistoric-diamond-4224

10/03/2022, 10:31 AM

Hi there! I cordoned, drained, upgraded the resources of a worker node and restarted it, but k3s still doesn't schedule pods on the upgraded node, whenever i restart some workload they are still assigned to one of the other smaller nodes. To force-schedule pods to the new node I have to cordon all the other ones and delete some pods, only then after some time they are re-scheduled to the under-utilized node, but every time they are restarted they are placed on the other smaller nodes. So now I have one of my nodes sitting at 91% RAM and another at 30%. Apparently resource requests are not affecting the scheduling. Any idea of what could be happening?

green-energy-38738

10/03/2022, 2:15 PM

1. any one runs k3s on docker ?

chilly-telephone-51989

10/03/2022, 2:37 PM

when i have images in a custom registry i have to tag the image and also refer to this custom repo in my yaml files for instance image: 172.16.0.25000/myapp0.3 is there a way to not specify the registry server and k3s by default looks to that address? so it becomes image: myapp:0.3

bright-jordan-61721

10/03/2022, 3:28 PM

Recently built a k3s cluster with version

v1.24.6+k3s1

and I have some pods configured with

dnsPolicy: ClusterFirst

(which is the default) and noticing weird DNS resolution problems. When I shell into a pod with this dns policy and

cat /etc/resolv.conf

this is what I see:

Copy code

bash-5.1# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local [home search domains redacted]
nameserver 10.43.0.10
options ndots:5

I believe ndots:5 is causing the problem, as

ping <http://github.com|github.com>

fails due to dns resolution, but

ping <http://github.com|github.com>.

works instead. Why is k3s setting the ndots:5 option by default? I’m not setting this with the pod’s dnsConfig at all. If this option were removed or reduced to ndots:1 it would likely solve my issue.

prehistoric-judge-25958

10/03/2022, 5:59 PM

Hi, i have setup a private registry (harbor) inside my k3s cluster. I have pushed images succesfully to this registry and now i want to pull images from the registry. When i put this yaml config below in

/etc/rancher/k3s/registries.yaml

my node become in a NotReady state after "systemcl restart k3s"

Copy code

mirrors:
  harbor.k8s.lan:
    endpoint:
      - "<https://harbor.k8s.lan:443>"
configs:
  "harbor.k8s.lan:443":
    tls:
      cert_file: /etc/rancher/k3s/certs/cert.pem
      key_file:  /etc/rancher/k3s/certs/cert-key.pem
      ca_file:   /etc/rancher/k3s/certs/k8s-lan.crt
      insecure_skip_verify: "true"

I am using self-signed certificates for my k8s.lan domain and put them in the directory /etc/rancher/k3s/certs/ describe node k3s-master-01 output:

Copy code

Normal   Starting                 23m                kubelet          Starting kubelet.
  Warning  InvalidDiskCapacity      23m                kubelet          invalid capacity 0 on image filesystem
  Normal   NodeAllocatableEnforced  23m                kubelet          Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     23m (x2 over 23m)  kubelet          Node k3s-master-01 status is now: NodeHasSufficientPID
  Normal   NodeReady                23m                kubelet          Node k3s-master-01 status is now: NodeReady
  Normal   NodeNotReady             20m (x3 over 63m)  node-controller  Node k3s-master-01 status is now: NodeNotReady

✅ 1

late-needle-80860

10/04/2022, 2:04 PM

I’m looking into adding

--egress-selector-mode=disabled

to an already running cluster ( server/control-plane side of course). Is that possible or do one need to redeploy the cluster anew? The reason for this is that I’m see

failed: error dialing backend: EOF

when e.g. running the

connectivity test

provided by the

Cilium

cli

. When I tried introducing it to a running test cluster I’m getting the infamous

failed to validate server configuration critical configuration value mismatch

…. Is other there a workaround to get this in on a live/already running cluster? Thank you very much

adamant-waiter-35487

10/05/2022, 8:30 AM

In k3s HA with embedded db, is it possible to have server and agent on the same node? The quick start command (

curl -sfL <https://get.k3s.io> | sh -

) installs both server and agent, but the tutorial on embedded ha ask us to start server on 3 nodes, and then join agent later. I am not sure if this means I need 3 nodes just for ha control plan, and need more nodes to behave as agent (worker) node?

late-needle-80860

10/05/2022, 6:23 PM

I managed to get the

--egress-selector-mode=disabled

set on the

servers

of X K3s cluster I have running - on v1.24.4+k3s1. However, when running the

cilium connectivity test …

command for the

Cilium

CNI

I know get the following err:

unable to start container process: open /dev/pts/0: operation not permitted: unknown

- troubleshooting that err leads me to: • https://github.com/opencontainers/runc/pull/3554 • and this release: https://github.com/opencontainers/runc/releases/tag/v1.1.4 What

K3s

release is that part of - if any?

quiet-chef-27276

10/06/2022, 1:55 AM

Cross-posting as I don't seem to be able to delete the original - probably should have been posted here.

late-needle-80860

10/07/2022, 8:05 AM

I was wondering - after having used

--egress-selector-mode=disabled

on nodes in a cluster. What is one not getting? What’s the downside? In the docs it says:

Copy code

The apiserver does not use agent tunnels to communicate with nodes. Requires that servers run agents, and have direct connectivity to the kubelet on agents, or the apiserver will not be able to function access service endpoints or perform kubectl exec and kubectl logs.

So sounds fine to me. I wasn’t disabling the agent on

servers

anyway. So not loosing anything there. Are there any downsides or considerations one should have? Thank you very much

late-needle-80860

10/07/2022, 10:40 AM

Someone on here having experience with the joining of K3s worker nodes in cordoned fashion … for different reasons we wish that X joining

worker

node joins cordoned in order to different processes to complete in due time before regular workloads starts piling in on the new

worker

. Some of these processes might be/are: • the configuration of

containerd

for a private self-hosted registry • Longhorn bootstrapping … and storage space setup … which needs to be fully up and ready before potential persistent storage needing regular workloads starts appearing

handsome-painter-48813

10/07/2022, 10:48 AM

Hello, I upgraded debian from 10 to 11 and I can't start k3s server:

Copy code

failed to find cpuset cgroup (v2)

k3s check-config:

Copy code

Generally Necessary:
- cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: bad (fail)
    (for cgroups V1/Hybrid on non-Systemd init see <https://github.com/tianon/cgroupfs-mount>)
- /usr/sbin/apparmor_parser
apparmor: enabled and tools installed

I already set

Copy code

GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=0"

And it does not work 😕

clever-air-65544

10/07/2022, 4:45 PM

k3s weekly is up! https://github.com/k3s-io/k3s/discussions/6229

👀 2

👍 2

red-boots-23091

10/08/2022, 11:50 AM

Hey all, I have a cluster that i created on a docker install within v2.5.7. I have used this for testing for around 1 year now. However at the end of August I started getting problems with it (kubectl connection refused and various crash loops). I cant seem to resolve these. I assume it is to do with updates I made to my server packages. Long story short I am trying to upgrade my Rancher server. However no matter what I do I cant reliably connect to rancher UI and when I can i still have trouble with kubectl. I am running on Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-125-generic x86_64). Docker Engine Version: 20.10.12 and I am trying to spin up rancher v2.6.8. These are my server logs https://pastebin.com/cDgYiJCb. Does anyone have any guidance

enough-carpet-20915

10/08/2022, 5:58 PM

I've replaced 3 of my k3s nodes. They have new IPs but the same names as before. I'm now getting this error:

Copy code

Error from server: error dialing backend: x509: certificate is valid for 127.0.0.1, 45.x.x.x, 2a02:c206:xxxx:xxxx::1, not 38.x.x.x

I tried doing

k3s certificate rotate

(from https://github.com/k3s-io/k3s/wiki/K3s-Cert-Rotation) which seems to have rotated certs, but I'm still getting the same error.

enough-carpet-20915

10/08/2022, 6:56 PM

It doesn't look like it actually changed the certs.

enough-carpet-20915

10/08/2022, 6:56 PM

Copy code

admin@marge:~$ sudo k3s certificate rotate
INFO[0000] Server detected, rotating server certificates 
INFO[0000] Rotating certificates for admin service      
INFO[0000] Rotating certificates for etcd service       
INFO[0000] Rotating certificates for api-server service 
INFO[0000] Rotating certificates for controller-manager service 
INFO[0000] Rotating certificates for cloud-controller service 
INFO[0000] Rotating certificates for scheduler service  
INFO[0000] Rotating certificates for k3s-server service 
INFO[0000] Rotating dynamic listener certificate        
INFO[0000] Rotating certificates for k3s-controller service 
INFO[0000] Rotating certificates for auth-proxy service 
INFO[0000] Rotating certificates for kubelet service    
INFO[0000] Rotating certificates for kube-proxy service 
INFO[0000] Successfully backed up certificates for all services to path /var/lib/rancher/k3s/server/tls-1665255335, please restart k3s server or agent to rotate certificates 

admin@marge:~$ sudo diff -sr /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/tls-1665255335/ | grep -i identical | awk '{print $2}' | xargs basename -a | awk 'BEGIN{print "Identical Files:  "}; {print $1}'
Identical Files:  
client-ca.crt
client-ca.key
dynamic-cert.json
peer-ca.crt
peer-ca.key
server-ca.crt
server-ca.key
request-header-ca.crt
request-header-ca.key
server-ca.crt
server-ca.key
service.key
apiserver-loopback-client__.crt
apiserver-loopback-client__.key

gifted-branch-26934

10/10/2022, 12:12 PM

hello guys, i have an app deployed in k8s with 2 replicas, the problem is each time i increase the number of replicas, the new replica is directly marked as available and operating. but the real thing that is happening that at the startup it builds a model that takes about 5 mins. for this issue i want the traffic to be routed to the first replica that has been operating for long time, and once the new replica builds the model, then i need to make it receive traffic as well. any idea how to do that?

average-arm-20932

10/11/2022, 6:36 PM

Hello Team, During the scanning I found 'weak SSL ciphers', in k3s server, I'm using K3S version 'v1.21.1+k3s1'. I found an official WEB-URL to fix the issue and seems it is for 'Rancher-manager', but do we have any fix for K3S. For Rancher-Manager. https://www.suse.com/c/resolve-cipher-and-ssl-threats-security-scans/ Thanks & Regards.

famous-flag-15098

10/12/2022, 3:38 PM

I have a few questions regarding multi-master k3s. I currently have a 6 node home cluster that currently has 2 etcd/masters. When I setup the second master I used the K3S_URL of the first master. Now I would like to turn the original master into an agent node, and then turn two of the existing agents into masters. How should I approach this?

famous-flag-15098

10/12/2022, 3:38 PM

I am especially wondering about how the K3S_URL works.

billowy-bird-32869

10/13/2022, 9:40 AM

Hello. I am using K3D for K3S and I would like to parametrize from the command line (or even as a variable) the value of a chart manifest used with the autodeploy feature. More explicitly, I have created a HelmChart resource in

/var/lib/rancher/k3s/server/manifests/

and in that resource, I have used spec.set to change the value of some property. The value to the property is delivered via an environment variable at the moment as I cannot think of any other way after looking at the options from k3s. Would someone have an idea, how I could parametrize my chart or if the way I have done it is correct, how do I supply the value. Thanks. Sample:

Copy code

spec:
  helmVersion: v3
  repo: <https://charts.gitlab.io>
  chart: gitlab-runner
  targetNamespace: gitlab-runners
  set:
    runners.tags: "$my_tag"

stale-vegetable-37217

10/13/2022, 1:52 PM

Hi, Can someone help me out? I am trying to run a k3s for the first time behind a corporate proxy. The proxy allows the use of ipv4 as well as ipv6 but on different hostnames. After installing awx on the 1 node cluster and checking with kubectl logs I get an proxyconnect tcp error code 503. The full error is printed below. The ipv6 address printed is the one from the server, and port 6443 is running. It seems that k3s is trying to connect to localhost using the proxy (which is not needed). NO_PROXY var is also set. Anyone has an idea as to why that is happening? Full error: 'Error from server: Get "https//[2a02xxxxx]:10250/containerLogs/awx/awx-operator-controller-manager-fcf6db67c-h9s9q/awx-manager?follow=true": proxyconnect tcp: proxy error from [:1]6443 while dialing proxy.xxx.be:8080, code 503: 503 Service Unavailable'

melodic-hamburger-23329

10/14/2022, 2:44 AM

I noticed some discrepancy in the docs. Here

disable-etcd

is mentioned but not here. Are there other components missing from the latter list?