Having an issue, I think.... I setup k3s on Harvester via Rancher, everything works fine except it seems like my networking is backwards. On my load balancer I am getting multiple external ip's for each node that is accessible externally, the ips are from dhcp from my external router. What I am trying to do is have a external facing load balancer so that it can do routing based on prefix. I am also using Cloudlare. Any recommendations?

I first wanted to post some giant question here, but I figured I’ll write a github issue because it’ll be more discoverable by other people and >90 days if somebody else has such issues: https://github.com/k3s-io/k3s/issues/6679

Is it possible to downgrade a HA control-plane setup to a single server one?

I'm trying to use K3c with K3s. any pointers or helpful resources people know? k3c is pretty bare on youtube.

If I have 4 nodes. what is the optimal setup for a K3s cluster?

from what I've gathered, 3 master nodes and 1 worker node would be optimal, but why? And should at least 1 of the master nodes be combined in this setup as a master/worker? thanks.

Hello I'm facing an issue very recently. A k3s cloud with one master and 2 worker nodes has stopped responding. I restarted k3s and here is the log

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-01-05 10:01:20 UTC; 9min ago
       Docs: <https://k3s.io>
    Process: 963939 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 963941 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 963942 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 963943 (k3s-server)
      Tasks: 88
     Memory: 679.2M
        CPU: 46.703s
     CGroup: /system.slice/k3s.service
             ├─  1626 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 2a9aca2d3>
             ├─  1809 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 4034e2e93>
             ├─  2269 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 31f1caa57>
             ├─963482 /var/lib/rancher/k3s/data/577968fa3d58539cc4265245941b7be688833e6bf5ad7869fa2afe02f15f1cd2/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 76b5f8348>
             ├─963943 "/usr/local/bin/k3s server"
             └─963959 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib>

Jan 05 10:01:34 ip-172-31-46-55 k3s[963943]: I0105 10:01:34.384632  963943 shared_informer.go:262] Caches are synced for garbage collector
Jan 05 10:01:34 ip-172-31-46-55 k3s[963943]: I0105 10:01:34.384666  963943 garbagecollector.go:158] Garbage collector: all resource monitors have synced. Proceeding to coll>
Jan 05 10:01:34 ip-172-31-46-55 k3s[963943]: I0105 10:01:34.454332  963943 shared_informer.go:262] Caches are synced for garbage collector
Jan 05 10:06:31 ip-172-31-46-55 k3s[963943]: I0105 10:06:31.679514  963943 trace.go:205] Trace[110892778]: "Get" url:/api/v1/namespaces/xplorie/pods/gateway-86c6cc8bf4-fjnr>
Jan 05 10:06:31 ip-172-31-46-55 k3s[963943]: Trace[110892778]: ---"Writing http response done" 6201ms (10:06:31.679)
Jan 05 10:06:31 ip-172-31-46-55 k3s[963943]: Trace[110892778]: [6.204322629s] [6.204322629s] END
Jan 05 10:07:27 ip-172-31-46-55 k3s[963943]: I0105 10:07:27.025536  963943 trace.go:205] Trace[1243742719]: "Get" url:/api/v1/namespaces/xplorie/pods/web-7df799b896-kmjwl/l>
Jan 05 10:07:27 ip-172-31-46-55 k3s[963943]: Trace[1243742719]: ---"Writing http response done" 35215ms (10:07:27.025)
Jan 05 10:07:27 ip-172-31-46-55 k3s[963943]: Trace[1243742719]: [35.217361472s] [35.217361472s] END
Jan 05 10:09:32 ip-172-31-46-55 k3s[963943]: I0105 10:09:32.845634  963943 log.go:195] http: TLS handshake error from 127.0.0.1:39796: read tcp 127.0.0.1:10250->127.0.0.1:3

These servers are AWS EC machines

Hello! I foolishly upgraded from 1.24 to 1.26 without reading the changelog and now I'm running into issues where helm charts that try to install PodSecurityPolicies are failing because kubernetes is refusing the api. Is there a k3s command-line-flag that might allow the creation of the objects with the understanding that they'll be no-ops?

It is not listening on ipv6 only. That is a dual-stack bind.

Hi all,

I havent changed anything on the machine running K3S at all

checking journalctl shows this:

Jan 09 01:18:01 facio k3s[29667]: E0109 01:18:01.542615   29667 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2023-01-09T01:18:01Z is after 2023-01-08T22:37:57Z, verifying certificate SN=2555825581207100645, SKID=,AKID=8F:E9:C8:2E:7F:91:B7:01:BC:7A:48:DD:77:8C:6A:EF:92:DA:E5:58 failed: x509: certificate has expired or is not yet valid: current time 2023-01-09T01:18:01Z is after 2023-01-08T22:37:57Z]"

can’t understand how this happened

current time 2023-01-09T01:18:01Z is after 2023-01-08T22:37:57Z

one last comment, I can run

kubectl

commands in the master node (facio), I see pods running, but the service shows the errors described before

Have you restarted it... ever?

The cers expire after a year. They are automatically renewed if they are within 90 days of expiring but for that to work you have to actually be patching the server and restarting the service periodically.

I have a number of clusters that are still running Traefik v1 which we are planning to migrate to v2.. however we are also looking into using Nginx instead of Traefik. Is there any pros/cons to using one over the other in k3s clusters managed by Rancher? Have very limited experience working with either Proxy other then deploying ingresses i guess..

Clusters are internal only btw and hosting a few applications that are not seeing a ton of request so high-performance/speed isn’t that high on list of priorities

Hello 👋 I recently updated my k3s installation and my TLS certificates are no longer being signed by my root CA. I injected my certificates based on the instructions here: https://github.com/k3s-io/k3s/issues/1868#issuecomment-639690634 Did the certificate handling change? Is there a better documented way to use my private root CA to sign the CAs used by k3s? My goal is to be able to validate the certs when pushing changes with kubectl.

I have a weird issue when bootstrapping a new cluster. There seems to be an issue with tls certificates. I’ll post more details in this 🧵

Is there support for moving a k3s master from a machine to another? (Looking at moving it from cloud to local LXD vm) No HA

I have a single node k3s instance that's kinda freaking out 😄

k3s server

is taking like 200% CPU and the kubernetes API is becoming unreachable which is causing a bunch of stuff to crash which is making the issue worse. I've poured through

journalctl

and don't really see anything notable. Any ideas how to track down the root of this?

I believe rancher and k3s was developed by same organizations but rancher installation with helm isn't working because of kubeversion specification on rancher helm chart. It requires version to be v1.25.4 but actual node version using k3s is v1.25.4+k3s1. What is the fix here?

Rancher can't be installed on K8s versions above 1.24 mostly a consequence of PSP being removed, and replaced by PSAs

I’m able to connect to my redis deployment, from a custom subdomain HostSNI, outside the cluster after I configured an IngressRouteTCP along with tls secret, like so

---
apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: IngressRouteTCP
metadata:
  name: redis-service-tcp

spec:
  entryPoints:
    - redis
  routes:
  - match: HostSNI(`<http://redis.example.net|redis.example.net>`)
    services:
    - name: redis-service
      port: 6379
  tls:
    secretName: wildcard-secret

(After applying this, I’m able to connect to

<http://redis.example.net|redis.example.net>

using redis-cli) However, I’m not able to do the same with postgres, even though the certificate etc is valid. Doesn’t postgres support SNI? Has anyone here exposed their postgres deployments via Traefik IngressRouteTCP with custom domain and TLS?

Is Rocky Linux 8 and/or 9 good RHEL distro's to run K3s on, or are there known issues with those distro's?

I have installed k3s on 2 nodes.

I'm trying to install k3s on a raspberry pi running dietpi (https://dietpi.com), using an external mysql database for datastorage. k3s is installed and looks ok, but for some reason it can't resolve the name of the mysql server. The network uses mdns for local names, so the mysql server is identified by

emrys.local

, which (after a few hickups with missing nss libraries and configuration) seems to resolve fine for everything else on the system, but k3s refuses to resolve it. Does k3s do direct DNS lookups? Is there a way to get it to resolve mdns names?

Is updating k3s multiple minor versions supported? For ex if i’m running 1.21 and want to update to 1.24 can i do so directly or do i need to update to .22 and .23 first?

We run some Openshift Cluster and there it’s not supported to jump between multiple minor version… think it was the same in EKS when we ran that a long time ago hence my question 🙂

Hi, I have two questions to ask: 1. If I just need kube-apiserver (to create CRDs), not other K8s functionalities (e.g. Pods, Services, etc.), could I customize K3s to start only kube-apiserver (no controller-manager, scheduler, etc.)? 2. Which control plane datastore choice can provider better performance and scale for a large number of CRDs (e.g. several millions of CRs)?