👋 Hi, I'm new here. I'm trying to upgrade a k3s cluster, and I'm reading things like this in the docs:

Please note that upgrades from experimental Dqlite to embedded etcd are not supported. If you attempt an upgrade it will not succeed and data will be lost.

How do I know which one I'm using?

--cluster-init seems to be a one time deal? Is there a way to just --cluster-init and quit? How harmful is it to keep the option all the time? What happens if cluster-init is on multiple servers? Also after initial bootstrapping can you specifiy a list of servers so that you aren't always trying to connect to a single server?

Hello, Does anyone use calico instead of flannel with k3s? I can observe the node is taking far more time to start with calico than with flannel (pods are running after 200s instead of 30s). Any clue on how to fix that?

Did anyone try to create more than 3000 K3s clusters (preferably on Raspberry Pi) and able to manage from Rancher? Wanted to know the scalability for K3s with Rancher.. any suggestions on how to manage clusters at this scale would help.

I'm trying to diagnose, why a readiness probe mysteriously fails on my k3os cluster. is there a log file or something where that would show up?

Hey @creamy-pencil-82913 Following from this https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ My aim is to create a k3s cluster where the control-plane nodes are not accessible with

kubectl get no

Research shows me that I should start the kubernetes process with

--register-node=false

Is this possible with k3s or do you have a better suggestion on how I could achieve this?

This is what really brought me here . . .

Quick question, I see in the release notes for 1.24.3 that etcd snapshot compression has been added and is used “if enabled” but I can’t find anything in the docs on this new option and it’s not entirely clear from these commits whether it is on by default, or what key I’d use to configure it

No. It's not a real thing that you can ping. Services are just iptables port forwarding managed by kube-proxy.

Hello Everyone, I am trying to add new master node in the highly available k3s setup (I am using postgres as database to store the state of k3s) and here is the command

curl -sfL <https://get.k3s.io> | sh -s - server --datastore-endpoint='<postgres://k3s:password@192.168.1013:5432/k3s>'  --token='K102xxxxx''

I'm using to add new master node, but it is giving me following error so can you someone look into it and help ?? FYI, This highly available k3s cluster is configured one year back. `

systemd[1]: k3s.service: Service hold-off time over, scheduling restart.

systemd[1]: k3s.service: Scheduled restart job, restart counter is at 97783.

systemd[1]: Stopped Lightweight Kubernetes.

systemd[1]: Starting Lightweight Kubernetes...

sh[23500]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service

sh[23500]: /bin/sh: 1: /usr/bin/systemctl: not found

k3s[23512]: time="2022-08-11T07:19:00Z" level=info msg="Starting k3s v1.24.3+k3s1 (990ba0e8)"

k3s[23512]: time="2022-08-11T07:19:00Z" level=info msg="Configuring postgres database connection pooling: maxIdleConns=2, maxOpenConns=

k3s[23512]: time="2022-08-11T07:19:00Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."

k3s[23512]: time="2022-08-11T07:19:00Z" level=info msg="Database tables and indexes are up to date"

k3s[23512]: time="2022-08-11T07:19:00Z" level=info msg="Kine available at <unix://kine.sock>"

k3s[23512]: time="2022-08-11T07:19:00Z" level=fatal msg="starting kubernetes: preparing server: 
bootstrap data already found and encryp

"
systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE

systemd[1]: k3s.service: Failed with result 'exit-code'.

`

I'm trying to make the database service available inside kubernetes for which I created the following service along with the endpoint

apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: xp
  labels:
    service: postgres
    type: database
spec:
  type: ClusterIP
  selector:
    service: postgres
    type: database
  ports:
    - name: client
      protocol: TCP
      port: 5432
      targetPort: 5432
---
apiVersion: v1
kind: Endpoints
metadata:
  name: postgres
  namespace: xp
  labels:
    service: postgres
    type: database
subsets:
  - addresses:
      - ip: 172.19.0.2
    ports:
      - name: client
        port: 5432
        protocol: TCP

I tried verifying the service using busybox but nslookup and ping both fail for postgres.xp however pinging the IP 172.19.0.2 runs just fine. how do I expose the IP of an external service inside?

1-node setup like https://github.com/drewp/infra/blob/main/multikube.py I run a hello service; can't curl it from the one node: details, config, and logs here: https://gist.github.com/drewp/e381498de3b1cf8c51f26e74bc08625d (line 44 is what i think should work) A setup like this 1) works on blank ubuntu VMs at digitalocean, and 2) used to work at home. Did i mess with some subtle ipv6 setting or something?

I'm running k3s locally and i need to have ingress from which the traffic should divert to "gateway" service this is my very simple ingress file. I'm unable to access it even with my local machine ip. is there something wrong with the file?

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: ingress
  namespace: xplore
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: traefik
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: gateway
                port:
                  number: 80

I'm looking for best practises for caching inside containers. I have a .NET 6 backend which uses MemoryCache to store certain keys and values to improve the overall performance and lessen the load on the 3rd party backend. Turns out that MemoryCache is pod specific (duh), which sadly took me way too long to figure out why I had 5 different cache responses (I had 5 active pods). How have people been resolving such an issue? Do you divert to a volume / database?

Hi Guys, any documentation on how to deploy k3 cluster using terraform?

Is there a way to view what install options were used on k3s after the fact?

k3s seems to be still using 1.5 series containerd. Is there something preventing versionup to 1.6? I noticed there’s 1.6.6 tag in the fork repo, but there doesn’t seem to be k3s release using this one. Also, is there some documentation regarding the differences between official containerd and the fork?

Greetings.. using K3s (

1.24.3+k3s1

) on Ubuntu 22.04.1 with ZFS (

2.1.4

) and external containerd (

1.5.9

). Works GREAT, no day-to-day issues. What I've been struggling with is when K3s uses an external containerd -- either kubelet / cadvisor is a bit wonky with metrics in that the

image=

and

container=

are missing which breaks many dashboards. I can't tell if this is a K3s / kubelet issue, cadvisor issue, containerd issue.. not sure where to seek advise. Kube-Prometheus-Stack is deployed, works well as long as a dashboard doesn't try to use something like

container_cpu_usage_seconds_total{image!=""}

returns an empty set as the image reference is missing, but that does return data when K3s uses bundled containerd. Suggestions welcomed.

☝️ For more context on this issue : https://github.com/dotdc/grafana-dashboards-kubernetes/issues/18 Seems that the problem comes from using an external containerd. @refined-toddler-64572 is doing this because he's using ZFS and the embedded containerd in k3s doesn't seems to support it (maybe due to the overlay FS). Is there a way to make k3s work with ZFS? If yes, how? If not, is there any plan to support ZFS in the future?

If I want to do

nerdctl build

inside a container running in k3s, what should I do? k3s doesn’t bundle buildkitd I think, so I guess I need to set up buildkitd manually. k3s bundles containerd, so I’m not quite sure how this manual should be applied: https://github.com/containerd/nerdctl/blob/master/docs/build.md

hi, we started seeing this issue https://github.com/kubernetes/kubernetes/issues/97129 in recent versions of k3s v1.22 and 1.24. Can anyone please help resolve this?

while true; do kubectl exec -it <pod_name> /bin/ls; done

Running above command will give intermittent error:

Error from server: error dialing backend: EOF

Latest k3s weekly report is up! https://github.com/k3s-io/k3s/discussions/6039

Hi, I am able to boot of k3s on yocto without an issues, that i know of, but coredns just keeps printing `still waiting on: kubernetes`` and eventually has a timeout on the api server

Kubernetes API connection failure: Get "<https://10.43.0.1:443/version>": dial tcp 10.43.0.1:443: connect: network is unreachable

. Any ideas how to approached this problem? DNS cache issue, flannel?

Hi there! I have a bit of a problem at hand, one of our less eperienced fellows upgraded k3s by several versions at once, from before 1.22 to 1.24, this of course caused mayhem for a ot of workloads and clients that relied on older APIs. Traefik ingress is one of those, as of now all of our ingresses are down. Since there was an instance of traefik v1.81 present on the cluster at the time of upgrade, k3s did not touch it, but of course, now traefik complains that it cannot find

*v1beta1.Ingress

which was deprecated in 1.22. Is there a traefik v1 version that i can upgrade to that supports

v1.Ingress

? I'd rather not update traefik to v2 since i'd have to migrate all of our ingresses, many of which are rmanaged by helm charts. Would nginx be compatibe with standard Ingress resources? Is a migration from traefiik v1.81 to nginx ingress feasible without modifying existing ingress resources? Thank you

How do I upgrade from v1.24.3+k3s1 to v1.24.4+k3s1? Is just

stop k3s => replace k3s binary => start k3s

enough?

I got this error this morning from `pod/helm-install-traefik-q76r7`:

+ echo 'Installing helm_v3 chart'
+ helm_v3 install --set-string global.systemDefaultRegistry= traefik <https://10.43.0.1:443/static/charts/traefik-10.19.300.tgz> --values /config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml
Error: INSTALLATION FAILED: cannot re-use a name that is still in use

k3s-1.24.3

kube-system    pod/helm-install-traefik-crd-f8qps                      0/1     Completed          0                  18m
kube-system    pod/helm-install-traefik-q76r7                          0/1     CrashLoopBackOff   5 (63s ago)        4m32s
kube-system   job.batch/helm-install-traefik-crd                   1/1           14m        18m
kube-system   job.batch/helm-install-traefik                       0/1           4m35s      4m39s

#k3s have anybody made the k3s cluster thru terraform on vsphere? any documentation would be really great?

I'm having an issue where k3s v1.20.11+k3s2 with Rancher 2.5.x in a 2 cluster of 2 VM nodes with MySQL as its external database is using all available cpu on the host and showing anywhere from 1500 - 12000 sockets in time_wait with the mysql server port varying by sometimes 200+ connections per second. Rancher pods are showing 100%+ cpu, Traefix is using 50%, and mysql is using 50% and k3s-server is using at times > 600% cpu. Rancher only manages 1 3-node baremetal RKE cluster. I'm unable to determine what is causing the massive number of TCP connections and/or the high CPU usage within the K3S cluster. Both clusters were running fine until Saturday when the Rancher K3S cluster pretty much blew up and hasn't been able to recover since. both k3s nodes are complaining about possible SYN flood due to the number of connections, but traffic is all internal.

Any idea why I’m not seeing any resources in kubernetes-dashboard running in k3s even though logging in with a token generated for a cluster-admin account? I set up the dashboard using the helm chart.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin
  namespace: kube-system
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRoleBinding
metadata:
  name: admin
roleRef:
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin
  namespace: kube-system

`kubectl get clusterrole cluster-admin -oyaml`:

apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  annotations:
    <http://rbac.authorization.kubernetes.io/autoupdate|rbac.authorization.kubernetes.io/autoupdate>: "true"
  creationTimestamp: "2022-08-29T05:57:20Z"
  labels:
    <http://kubernetes.io/bootstrapping|kubernetes.io/bootstrapping>: rbac-defaults
  name: cluster-admin
  resourceVersion: "72"
  uid: f381f7e3-9f54-4da5-bcc4-39745a2e8bbd
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
- nonResourceURLs:
  - '*'
  verbs:
  - '*'