rancher-users #k3s

rapid-businessperson-28408

11/07/2022, 7:23 PM

Do I have something misconfigured to where K3s isn't showing me the pods in kube-system? Things like etcd, the api, and the controllers? I assume those are just running as crictl containers, do I need to enable something for the k8s api to see them?

billions-airline-85860

11/08/2022, 10:05 AM

Hey everyone. I'm getting back into k8s and I've come up to some documentation I wrote myself and I'm a bit baffled. I have a k8s cluster running with k3s and am using Emissary as the ingress controller. I've successfully gotten some decent apps going like Rancher and Keycloak, all being exposed with proper URLs too, like https://rancher.my-url.com and https://auth.my-url.com. I have an Nginx LB in front of the cluster and this is where I'm a bit baffled. It simply streams to the agent/ worker nodes (4 of them) with config like the following.

Copy code

server {
    listen 443;
    proxy_pass my_agent_nodes_443;
  }

  upstream my_agent_nodes_443 {
    server 185.230.138.110:30679;
    server 185.239.208.173:30679;
    server 38.242.195.7:30679;
    server 38.242.134.42:30679;
  }

And, everything is working hunky dory (which is why I'm flabbergasted). My missing understanding is, how did I get the port numbers for the worker nodes i.e. 30679? I do all this for a hobby and did this work like 6 months ago, but didn't think to also document how I got to the port number and I can't find any mention of those ports in the emissary configs, only listeners for ports 80 and 443. 😮

billions-airline-85860

11/08/2022, 10:06 AM

Any tips on this would be greatly appreciated. 🙂

billions-airline-85860

11/08/2022, 10:14 AM

Hehe... two minutes after I wrote that, I figured it out. It's the "node port" set by k8s.

jolly-waitress-71272

11/09/2022, 7:27 PM

I'm running postgres on the same host as a k3s server. How do I determine generically what CIDR I need to plop in pg_hba.conf in order for containers to connect to the db? I can see containers are being assigned from 10.42.0.0/16, is that always the default in K3s with flannel? Is there somewhere I can read from to get the exact CIDR?

clever-air-65544

11/11/2022, 2:57 PM

Weekly report is up! https://github.com/k3s-io/k3s/discussions/6483

👍 1

🚀 5

shy-shampoo-22224

11/14/2022, 3:41 PM

Has anyone run into issues installing linkerd CNI with k3s flannel where it won’t assign an IP address?

gray-florist-76480

11/14/2022, 8:44 PM

Hey! I wanted to run some TCP and UDP service in my home cluster and expose one of it's port to LAN. For HTTP services I'm using

Ingress

but not sure what about raw service like that which I need to access on predictable address.

faint-airport-39912

11/15/2022, 8:30 AM

Hi All, Recently i have updated the k3s cluster version but after that i start getting below error in the logs and timeout response for the

kubectl

command so please suggest the solution to this problem ??

Unable to connect to the server: net/http: TLS handshake timeout

whenever i am getting timeout response for

kubectl

command the application exposed via ingress stop working and getting

response. These are the error logs from

/var/log/syslog

limited-potato-16824

11/16/2022, 2:41 PM

G'day, we are using terraform for k3s upgrades and today we did a upgrade from v1.22.12 -> v1.22.15 but after that I can see a pod/job running:

apply-k3s-master-plan-on-ip-1-2-3-444-with-37285dc25da-spg7r

Inspecting the jobs it looks like it want to downgrade to the v1.22.12 version again since that is the version I can see in Rancher: I have read the page https://docs.k3s.io/upgrades/automated, our k3s clusters is managed by Rancher but we want to be in control of the upgrades. Is there any way to prevent Rancher from trying to run the upgrade plans since I suspect the upgrade plans is installed through the cattle-cluster-agent? I see this in our k3s cluster:

Copy code

$ kubectl get <http://plans.upgrade.cattle.io|plans.upgrade.cattle.io> -n cattle-system
NAME              AGE
k3s-worker-plan   7d1h
k3s-master-plan   7d1h

few-minister-97494

11/16/2022, 4:48 PM

Folks - regarding the k3s upgrade controller - there's an air-gapped k3s install scenario but - no corresponding air-gapped upgrade using the upgrade controller that I can see. E.g. if the Plan CRD offered a way to locate the binary/binaries on a local machine via SSH, or maybe S3 bucket running on a local Minio, or a persistent volume, etc. - that would support an air-gapped upgrade. Has anyone at Rancher given consideration to this?

future-fountain-82544

11/17/2022, 10:31 PM

I’m working on some orchestration to spin up a K3s cluster. Is there a sane way to pre-generate the the K3S_TOKEN file? For a really sillly reason out of the scope of this channel, it’s a big pain to copy a file off of the first K3s server onto other nodes

future-fountain-82544

11/17/2022, 10:33 PM

It looks like the token file is in the format

K[33 random bytes hex-encoded]::server:[16 random bytes hex-encoded]

. The thing I’m not sure of is if the values are related (IE if they’re a public/private keypair or something)

famous-flag-15098

11/18/2022, 4:13 PM

Hi all! I need some support with an issue that has been plaguing me for months. ServiceLB, though disabled, continues to reinstall and break my LB configuration for certain pods. • I have a 6 node k3s cluster, half the nodes are ARM64, and the other half are Intel based. • 3 of the nodes are masters and the other 3 are agents.I am using metallb for a load balancer, along with a helm install of traefik. I have servicelb disabled in all node startup scripts. master

Copy code

ExecStart=/usr/local/bin/k3s \
server \
        ' --disable' \
        'traefik' \
        '--disable' \
        'servicelb'

agent

Copy code

ExecStart=/usr/local/bin/k3s \
    agent \
      ' --disable=traefik,servicelb'

When I first encountered this, I added the above to the systemd scripts, did a daemon-reload and restart. All the servicelb nodes terminated and all seemed well. Approximately 2 weeks later, some of my services were not accessible and sure enough, I see that servicelb has started up again. What gives?

bright-london-1095

11/21/2022, 10:13 AM

Hi All, Is there a procedure/steps to follow to downgrade the cluster version? Wanted to test how cleanly we can downgrade cluster without messing up things... 🧵 TIA

jolly-waitress-71272

11/22/2022, 4:56 PM

I'm trying to follow some instructions to set up some pods. What does this command do?

Copy code

k3s kubectl create secret generic kubeconfig --from-file=/etc/rancher/k3s/k3s.yaml

I'm looking at https://jamesdefabia.github.io/docs/user-guide/kubectl/kubectl_create_secret_generic/ and what I think happens is the k3s.yaml gets stored in essentially a protected kubernetes var. Is this something people normally do for some reason? I don't see it referenced at any other point in the instructions.

bright-london-1095

11/23/2022, 1:06 AM

Hi All, has any faced this error? I have installed

k3s

version

1.22.8

and the traffic pod is in crashLoopBackOff and not accessible. I tried deleting pod but didn't help!

Copy code

+ helm_v3 install --set-string global.systemDefaultRegistry= traefik <https://10.49.0.22:443/static/charts/traefik-10.14.100.tgz> --value/config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml
Error: INSTALLATION FAILED: cannot re-use a name that is still in use

bright-london-1095

11/23/2022, 1:15 AM

Copy code

kube-system    pod/helm-install-traefik-q76r7                          0/1     CrashLoopBackOff   5 (63s ago)        4m32s

bright-london-1095

11/23/2022, 1:26 AM

I got this from

k3s

logs:

Copy code

Nov 23 01:19:01 nestle-test k3s[1186]: I1123 01:19:01.070591    1186 scope.go:110] "RemoveContainer" containerID="e87d34e8fb2f4d3e07ab9ee9e8656ee73f0cd1dd00f7bce0ee9956d3bc091c67"
Nov 23 01:19:01 nestle-test k3s[1186]: E1123 01:19:01.070873    1186 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik--1-8htfg_kube-system(9833a11e-3065-48d7-908f-baf37fef0168)\"" pod="kube-system/helm-install-traefik--1-8htfg" podUID=9833a11e-3065-48d7-908f-baf37fef0168
Nov 23 01:19:03 nestle-test k3s[1186]: E1123 01:19:03.366518    1186 remote_runtime.go:164] "RemovePodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to remove volatile sandbox root directory \"/run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/345c21c66d2e341863fb9b95eb829a477f2c6e799016674d3b2a80fe113601a1\": unlinkat /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/345c21c66d2e341863fb9b95eb829a477f2c6e799016674d3b2a80fe113601a1/shm: device or resource busy" podSandboxID="345c21c66d2e341863fb9b95eb829a477f2c6e799016674d3b2a80fe113601a1"

bright-london-1095

11/23/2022, 1:31 AM

traefik

shipped with

k3s

(v1.22.8+k3s1)

early-solstice-46134

11/23/2022, 8:04 AM

good morning from Germany, how can I get the logs of the embedded

containerd

daemon inside of K3s? I need to debug some stuff regarding our private registry, but I can't dig deep enough.

journalctl -u k3s | grep <imagenaname>

does not show anything related to

containerd

pulling/pushing said image, even when I start k3s with the

--debug

flag. The logs for k3s show that it picks up the

/etc/rancher/k3s/registries.yaml

file correctly.

kind-helmet-90731

11/24/2022, 6:39 AM

evening, I was attempting to upgrade my k3s cluster to v1.25.4+k3s1 and it seems to have gone ok, except that all my ingresses are now not functional. Reviewing the logs in the traefik pod reveals

kind-helmet-90731

11/24/2022, 6:39 AM

Copy code

time="2022-11-24T05:48:21Z" level=info msg="Configuration loaded from flags."
time="2022-11-24T05:51:04Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=prometheus-server serviceName=prometheus-server namespace=prometheus servicePort="&ServiceBackendPort{Name:,Number:80,}"
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=prometheus serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=prometheus-server
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" namespace=opensearch providerName=kubernetes ingress=opensearch-dashboards
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=prometheus servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=prometheus-server providerName=kubernetes ingress=prometheus-server
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=opensearch providerName=kubernetes serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" ingress=opensearch-dashboards
time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=prometheus-server namespace=prometheus

kind-helmet-90731

11/24/2022, 6:42 AM

I performed a downgrade back to v1.25.3+k3s1 and still getting the same behavior above. Any recommendations on how to clear this and get the cluster ingresses back online? Not sure what to do next. I can provide an ingress definition of one of my services if it will help diagnose.

bright-postman-91926

11/24/2022, 7:41 AM

Hello K3s community. Hopefully I am in the right place for my enquiries. I have been charged with verifying a K3s deployment against the CIS benchmark controls. For this I am using the CIS benchmark v1.6 from the CIS community coupled with the Hardening guide from the K3s website (https://docs.k3s.io/security/hardening-guide) and the self assessment guide on the same site (https://docs.k3s.io/security/self-assessment). I understand that those guides on the K3s site are intended to match a slightly later revision of the CIS benchmark, however there is very little difference and I have successfully run through all the checks except for the following: CIS Control 4.2.9 - Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture CIS Control 4.2.13 - Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers For both these checks CIS recommends running "ps -ef |grep kubelet" on each worker node - which as I understand is not suitable for a k3s deployment. However the recommended alternative from the K3s self assessment guide seems equally unsuitable "/bin/ps -fC containerd" So my question(s) is, am I missing something or is the K3s approach to that control miss-documented? Can we even pass those parameters to the K3s binary on the worker nodes? if so, is there another way I can satisfy these checks? be it pass or fail. All and any help would be greatly appreciated. Even if its just to point me to another channel/slack I should be enquiring on.

full-park-34540

11/24/2022, 3:36 PM

https://rancher-users.slack.com/archives/C3ASABBD1/p1669301884658589

creamy-room-58344

11/25/2022, 3:21 PM

Hello there! I build a k3s cluster with 3 nodes and installed Rancher on it via Helm Can't Rancher deploy a new cluster with k3s? The only options available are those on the image ps.: I had to install k3s v1.24.8+k3s1 cause Rancher helm char requires < v1.25.0)

handsome-painter-48813

11/25/2022, 4:24 PM

Hello, I got something strage

Copy code

overlay         468G   15G  430G   4% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5e03898951de490bdc8e32ca67c24a1c691cf9efa8629ee494460f3fc5a6bc9b/rootfs

It goes up by 1 GB each second, how can I find container that does this?

red-boots-23091

11/25/2022, 5:36 PM

Hey All, I am using rancher v.2.5.7, single node docker install. on top of ubuntu server 20.04 lTS. The rancher/rancher:v2.5.7 container will not stay up longer than a few minutes generally it gives a k3s fatal 255 (this is a recent problem I have been running this server for 19 months) my logs are in this pastbin can anyone help me diagnose this issue? https://pastebin.com/0zwxEiJ5

flat-continent-80260

11/26/2022, 12:35 AM

New flatcar deployment. Installed k3s without an issue. However, I can't get past the bootstrap login. I suspect this is due to k3s using kubernetes 1.25. I was able to get rancher to install by manually pulling the helm chart and removing the <1.25 requirement. I also tried versions 2.7.0. just curious if there is an ETA for 1.25 support. Or if there is a current work around? Or maybe I just need to use an older version of k3s?