Hi Guys. I am running a 4-node k3s cluster v1.24.4+k3s1 - and last night my cluster blew up because of a failed traefik helm update. I had to upgrade every node to restart the helm install job ( could probably have done it smarter ) 🙂 The question is : What has triggered the helm upgrade ? Can that somehow be disabled ? Thanks a lot !

@nutritious-house-62299 has left the channel

Hi people nice to meet you all, one quick question. I've deployed rancher on Hetzner following this guide https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud It has one master and one worker node, my question now is how to add additional node? I'm confident with Terraform, but nowhere is the number of worker nodes that should be spin up defined?

hello friends, QQ - I am deploying PROD Ha k2s cluster - we want to use external mysql DB - it is already setup, together with HA HAprosy deployment - now everywhere I read, I come across - 3 master nodes if internal DB is used or “only” two if one uses external DB. But now questions is - if we use external DB, what about using 3 master nodes anyway? Is that not optimal or is it? Any idea on this? Or some document?

Hi everyone, is there a way to access the kube-proxy pod in k3s?

@miniature-agent-31894 Hi you mean shell access? If so see <https://kubernetes.io/docs/tasks/debug/debug-application/get-shell-running-container/>

hi mates, I'm trying to start etcd cluster with 5 nodes. I was working fine last week. All of a sudden, I found that etcd db size grew over 2GB then etcd cluster stopped work. So the first action was to increase the size by adding --etcd-arg into /etc/systemd/system/k3s.service.

--etcd-arg=quota-backend-bytes=6442450944

I added the above parameter in k3s.service file on the k3s host where the cluster is initialized. However, after systemctl daemon-reload, systemctl restart k3s, etcd was still reporting that there is no space. So my second action was to restore the etcd server from snapshot from last known good state. I did the restore with guidance posted on k3s.io website (HA restore).

hi mates, I'm trying to start etcd cluster with 5 nodes. I was working fine last week. All of a sudden, I found that etcd db size grew over 2GB then etcd cluster stopped work. So the first action was to increase the size by adding --etcd-arg into /etc/systemd/system/k3s.service.

--etcd-arg=quota-backend-bytes=6442450944

I added the above parameter in k3s.service file on the k3s host where the cluster is initialized. However, after systemctl daemon-reload, systemctl restart k3s, etcd was still reporting that there is no space. So my second action was to restore the etcd server from snapshot from last known good state. I did the restore with guidance posted on k3s.io website (HA restore). Now what I got is : • ETCD cluster is not running, no active member. • From the k3s log I can see that etcd server is trying to start with a new db size of 6GB. But also it tries to start a instance with 2GB with data dir /<k3s-dir>/etcd-tmp. I'm not sure if its by design or just error. • k3s service failed to start on all 5 nodes. The k3s status is 'activating' • after 3 restore process, I don't see any error related to etcd db size. But etcd still doesn't work at all. Do you guys have any recommendations?

quick question; Is Rancher 2.7.1 compatible with Kubernetes 1.25? I tried searching github repo readme and everything but I can’t see clear statement up to what version is compatible. I’m in the need to use cluster autoscaler function and it says it only from version Kubernetes 1.25 but it’s confusing now if I can use AS function with Rancher 2.71 and K3s... Anybody who can shine a light here please? Thanks!

2.7.2 has 1.25 support. it has been released last week

@straight-midnight-66298 ^^

Iam using k3s with

--datastore-endpoint="mysql..."

on a galera cluster. This is using a lot of ressources.

My version is

v1.24.11+k3s1

and its fireing these crazy statements

| SELECT *
		FROM (
			SELECT (
		SELECT MAX(rkv.id) AS id
		FROM kine AS rkv), (
		SELECT MAX(crkv.prev_revision) AS prev_revision
		FROM kine AS crkv
		WHERE crkv.name = 'compact_rev_key'), kv.id AS theid, kv.name, kv.created, kv.deleted, kv.create_revision, kv.prev_revision, kv.lease, kv.value, kv.old_value
			FROM kine AS kv
			JOIN (
				SELECT MAX(mkv.id) AS id
				FROM kine AS mkv
				WHERE
					mkv.name LIKE '/registry/leases/kube-system/k3s-cloud-controller-manager'
					
				GROUP BY mkv.name) AS maxkv
				ON maxkv.id = kv.id
			WHERE
				kv.deleted = 0 OR
				0
		) AS lkv
		ORDER BY lkv.theid ASC

is this a know problem?

anyone using v1.26.4-rc1+k3s1 with mysql in HA? Can't get it to bootstrap, getting "preparing server: invalid cipher text, not : delimited" - Filled an issue for that: https://github.com/k3s-io/k3s/issues/7313

@sparse-fireman-14239 has left the channel

doesn't even bootstrap with a single server..

anyone found a nice pattern to retrieve the kubeconfig from a server in a HA setup that is tf created? I feel like the right way is probably setting up all tls stuff locally but I feel then I can right away go with vanilla k8s..

I have a 2 node cluster. A single replica of http image running from one of the nodes. I exposed it with a loadbalancer service. The LB service got 2 external IPs, same as the IPs of the nodes. I could reach externally to the http pod, but I have to use the external IP of the node that runs the pod. CANNOT reach with external-ip of the second node. Why is that ? should I be able to reach with the IPs of any nodes in the cluster?

Is it true, that we cannot have any service running port 80 or 443 , when we have traefik?

Creating an ingress resource with traefik. Do we create an ingress for each service or only create just 1 ingress and add rules to it ?

Hi Everyone! We were running a k3s cluster on flatcar on top of a VM, during the upgrade, we seem to have messed up our first master. The other 2 masters are up and running fine it seems. When starting the k3s service, we keep getting this logs. We reset the 1st master and now are trying to add it using 2nd master as primary.. Appreciate if someone can help.. Is there a way we can join this node to one of the other masters? Thanks!

Apr 19 16:28:55 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:28:55Z" level=info msg="Waiting for etcd server to become available"
Apr 19 16:28:55 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:28:55Z" level=info msg="Waiting for API server to become available"
Apr 19 16:28:57 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:28:57Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:00 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:00Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:02 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:02Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:05 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:05Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:07 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:07Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:10 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:10Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:10 ng-10-51-52-101.lab k3s[989]: {"level":"warn","ts":"2023-04-19T16:29:10.926Z","caller":"etcdserver/server.go:2065","msg":"failed to publish local member to cluster through raft","local-member-id":"7dff6a3be78a5ee3","local-member-attributes":"{Name:ng-10-51-52-101.lab-accc9f42 ClientURLs:[<https://10.51.52.101:2379>]}","request-path":"/0/members/7dff6a3be78a5ee3/attributes","publish-timeout":"15s","error":"etcdserver: request timed out"}
Apr 19 16:29:11 ng-10-51-52-101.lab k3s[989]: {"level":"warn","ts":"2023-04-19T16:29:11.746Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005428c0/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Apr 19 16:29:11 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:11Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Apr 19 16:29:12 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:12Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:15 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:15Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:17 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:17Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:20 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:20Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:22 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:22Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:25 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:25Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"

Hello Everyone, I did the ranchers k3s setup. And whenever I tried to create a new deployment its failing. Please find below link. I have already asked question in general channel. I f any one have any solution please let me know. Thanks in advance https://rancher-users.slack.com/archives/C3ASABBD1/p1681983250118589

I'm trying to install a helm chart that is stored in a private OCI repository using the helm controller feature of k3s. Does anyone know how I can provide the necessary credentials to the helm controller?

If I start an agent with the

--resolv-conf

flag, how can I verify, that this configuration is actually used?

It seems way easier to just use longhorn then manually setting up a db to provide persistent storage across node failures. Is Longhorn recommended for k3s?

I am running Rancher Desktop on Mac ( Apple M1 Pro ) and trying to install Kube-prometheus-stack via helm chart.

I am frequently getting shailmis@shailmis-mac rancher-prac % kubectl get pods -n prom

Unable to connect to the server: net/http: TLS handshake timeout

Hi @creamy-pencil-82913 and All, I have posted a issue in https://github.com/k3s-io/k3s/issues/7343 , Please do the needful