Hi community, Quick question. In case I get

curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here: <https://curl.haxx.se/docs/sslcerts.html>

when pod is trying to access my GitHub Enterprise Server, is it enough to add a self-signed certificate to

/var/lib/rancher/k3s/server/tls

? Only the server cert should be added or the rootCA and intermediate certs as well? Thanks in advance,

Welcome @modern-island-11920!

👋

Is k3os dead?

Hello, I was trying to upgrade my cluster with the following command:

sudo curl -sfL <https://get.k3s.io> | INSTALL_K3S_VERSION=v1.24.11+k3s1 INSTALL_K3S_EXEC="--write-kubeconfig-mode "0644" --kubelet-arg=config=/etc/rancher/k3s/kubelet-config.yaml --disable=traefik --cluster-cidr=172.16.0.0/16 --flannel-backend=none --disable-network-policy --datastore-endpoint=etcd --default-local-storage-path=/mnt/storage" sh -

However I didn't have the kubelet config file in place

/etc/rancher/k3s/kubelet-config.yaml

at that moment making the k3s not starting properly

...
[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
[INFO]  systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO]  systemd: Starting k3s
Job for k3s.service failed because the control process exited with error code.
See "systemctl status k3s.service" and "journalctl -xeu k3s.service" for details.

Once the file was in place, k3s started but now I see two nodes:

kubectl get no
NAME             STATUS     ROLES                       AGE    VERSION
old-desired      Ready      control-plane,etcd,master   323d   v1.23.6+k3s1
new-undesired    NotReady   control-plane,etcd,master   118s   v1.24.11+k3s1

Is there a way to remove that

undesired

node without affecting the

desired

node?

Is there a way to get k3s to stop setting

local-path

as a default storage class? I try removing the default but k3s keeps putting it back.

I'm running high load jobs on k3s (embedded etcd, single-node cluster), and I get:

$ kubectl describe nodes

  Warning  ContainerGCFailed  51s (x621 over 10h)  kubelet  rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (16782381 vs. 16777216)

And somehow, my node automatically gets placed into a

NotReady

state. Has anyone seen this before?

I'm setting up some service accounts and associated token secrets by dropping a manifest at

/var/lib/rancher/k3s/server/manifests

, but I'm only getting the last defined secret. In the manifest I use

generateName

to set the name of the secrets, so

name

is not set. That seems to result in k3s only creating the last one defined in the file. If I explicitly set

name

, all secrets are created. I have a feeling this worked before in an older cluster of mine (which has since been scrapped, so can't confirm), has there been any changes with regards to this in the last couple releases? (I'd say after 1.23 maybe?)

Is it possible to change the

cluster-cidr

of an existing cluster? I have a cluster created with --cluster-cidr=192.168.0.0/16 but due to some conflicts I want it to be --cluster-cidr=172.16.0.0/16 but that makes the k3s service to restart frequently due to a panic:

Mar 31 05:04:34 my-node k3s[1059813]: panic: F0331 05:04:34.864500 1059813 controllermanager.go:222] error starting controllers: failed to mark cidr[192.168.0.0/24] at idx [0] as occupied for node: my-node: cidr 192.168.0.0/24 is out the range of cluster cidr 172.16.0.0/16

Hi. How can I configure the traefik logs on k3s ? I created a file /var/lib/rancher/k3s/server/manifests/traefik-config.yaml with the content :

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>

kind: HelmChartConfig metadata: name: traefik namespace: kube-system spec: log: filePath: "/var/log/traefik/traefik.log" level: DEBUG

but no file is created

Hi all, I installed a K3S (k3s version v1.26.2+k3s1 (ea094d1d)) on the node1. After that was trying to add additional master and agent nodes. However, unsuccessfully. It shows that k3s service is active and running, but: on the agent node, I run:

INSTALL_K3S_SKIP_DOWNLOAD=true K3S_TOKEN=SECRET INSTALL_K3S_EXEC="--resolv-conf=/etc/rancher/k3s/resolv.conf --selinux" sh install.sh agent --server <https://node1.example.net:6443>

and it seems that it didn't configure it as an agent node, but as a master:

k3s kubectl get nodes
NAME                   STATUS   ROLES                  AGE     VERSION
<http://node4.example.net|node4.example.net>   Ready    control-plane,master   2m20s   v1.26.2+k3s1

It should be added to the node1 and the role should be an "agent" On the node1 (master):

k3s kubectl get nodes
NAME                   STATUS   ROLES                       AGE    VERSION
<http://node1.example.net|node1.example.net>   Ready    control-plane,etcd,master   3d4h   v1.26.2+k3s1

on the rest 2 master nodes I run:

INSTALL_K3S_SKIP_DOWNLOAD=true K3S_TOKEN=SECRET INSTALL_K3S_EXEC="--resolv-conf=/etc/rancher/k3s/resolv.conf --selinux" sh install.sh server --server <https://node1.example.net:6443>

But again, these nodes were not added to the main node1. Is it a bug?

@creamy-pencil-82913 - 👋 do yall know of any funky race conditions (or straight up bugs) with the combination of: external sql DB backend + HA + secrets encryption wherein if you cycle the nodes (like by nuking a vm and letting an ASG replace it) and getting a new encryption key each time?

On a single node k3s install that doesnt have any network interfaces (at times) is there a best set of steps for the install (to keeps things running)?

good news! turns out rancher 2.7.1 can actually import a k3s cluster

i was originally told rancher 2.7.1 couldnt import 1.25.x clusters 🤔

Hey all, I have issues with my DNS and I follow [this](https://ranchermanager.docs.rancher.com/v2.5/troubleshooting/other-troubleshooting-tips/dns) Troubleshooting guide. This is the output of the loop to test FQDN resolution. There are three manager nodes in my cluster and two of them cannot resolve google.com. Unfortunately the docs only say

had the UDP ports blocked

but I opened all the ports mentioned [here](https://ranchermanager.docs.rancher.com/v2.5/getting-started/installation-and-upgrade/installation-requirements/port-requirements#downstr[…]er-nodes)

=> Start DNS resolve test
E0404 08:32:58.066969  105106 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:32:58.153873  105106 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:32:58.181868  105106 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:32:58.202611  105106 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:32:58.377407  105191 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:32:58.439348  105191 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:32:58.459940  105191 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
command terminated with exit code 1
<http://XXX.XXX.XXX.XXX|XXX.XXX.XXX.XXX> cannot resolve <http://www.google.com|www.google.com>
E0404 08:33:58.886092  107876 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:33:58.941918  107876 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:33:58.959819  107876 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:33:59.759025  107929 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:33:59.839680  107929 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0404 08:33:59.857799  107929 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
command terminated with exit code 1
YYY.YYY.YYY.YYY cannot resolve <http://www.google.com|www.google.com>
=> End DNS resolve test

Hi all! I would like to monitor a single node k3s cluster with rancher monitoring to have local dashboards available on the edge even if internet connection breaks. As the default rancher monitoring app is relatively resource hungry, does anyone have usable helm values available that I can use as a starting point for a single node k3s cluster?

Hi, I deployed a HA-cluster with two master and a mysql instance. I noticed that there has a huge throughout from mysql to one of masters as shown in the page below all the time. Is this normal or how can I debug it? Thanks.

Has anything changed with the k3s

install.sh

recently? Since today when I try to execute it with

INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_EXEC="server --cluster-init --token <token> --tls-san <ipaddress> --disable local-storage --config /etc/rancher/k3s/config.yaml" ./install.sh

it just exits without error message

My node has multiple network interfaces. Is there any way to make the Pod receive and send traffic from multiple network interfaces in K3s?

Hi, I just install k3s, with default options. I try to forward traefik dashboard port,

kubectl forward-port -n kube-system service/traefik 9000:9000

but I cannot access to traefik dashboard, from localhost. I miss something ?

(I try to access from another device from my local Network)

@glamorous-accountant-39131 What about `kubectl -n kube-system port-forward

kubectl get pods -n kube-system -o

custom-columns=:metadata.name | grep '^traefik-'

9000:9000`

Hi. Since a few hours ago, my DNS in k3s stopped working. Nobody was doing anything on the cluster as far as I can see. I'll put more details in the thread

I successfully installed K3S. In the

control-plane

node, when I run

sudo k3s kubectl config view

I get the following kube config:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: <https://127.0.0.1:6443>
  name: default
contexts:
- context:
    cluster: default
    user: default
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: default
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

Above configuration won't allow me to connect from a remote machine. Is there any way that I can get a kube config setting that allow me to do

kubectl

from a remote machine (means it include the user keys) ?

Is there a way to get notified (e.g. via e-mail) when

system-upgrade-controller

performs an update? I need some kind of audit trail + notification system.

I'm trying to screw around with https://kubevirt.io/user-guide/operations/installation/ It says:

Kubernetes apiserver must have --allow-privileged=true in order to run KubeVirt's privileged DaemonSet.

How would I verify that on k3s? Is that just stuffed into the unit file like normal k3s config flags?

Hi all, I've been trying to add a new master to an existing cluster by installing from ISO download and pointing it at a config.yaml for cloudinit that looks like this:

hostname: k3os-master-node
k3os:
  data_sources: []
  token: ...
  k3s_args:
    - server
    - "--write-kubeconfig-mode"
    - "644"
  modules:
    - kvm
    - rbd
    - openvswitch
  sysctl:
    kernel.printk: "4 4 1 7"
    net.ipv4.ip_forward: 1
  ntp_servers:
    - <http://0.us.pool.ntp.org|0.us.pool.ntp.org>
    - <http://1.us.pool.ntp.org|1.us.pool.ntp.org>
  kubernetes_url: <https://existing-master:6443>
  dns_nameservers:
    - 8.8.8.8
    - 8.8.4.4
  ntp_servers:
    - <http://0.pool.ntp.org|0.pool.ntp.org>
    - <http://1.pool.ntp.org|1.pool.ntp.org>
ssh_authorized_keys:
  - ssh-rsa ....

Is it actually possible to add a new master agent to an existing cluster like this?

when I install a k3s cluster with n nodes all nodes IPs shoud respond to the port 80 and 443 of traefik ?