https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
k3s
  • r

    rapid-businessperson-28408

    11/07/2022, 7:23 PM
    Do I have something misconfigured to where K3s isn't showing me the pods in kube-system? Things like etcd, the api, and the controllers? I assume those are just running as crictl containers, do I need to enable something for the k8s api to see them?
    c
    • 2
    • 27
  • b

    billions-airline-85860

    11/08/2022, 10:05 AM
    Hey everyone. I'm getting back into k8s and I've come up to some documentation I wrote myself and I'm a bit baffled. I have a k8s cluster running with k3s and am using Emissary as the ingress controller. I've successfully gotten some decent apps going like Rancher and Keycloak, all being exposed with proper URLs too, like https://rancher.my-url.com and https://auth.my-url.com. I have an Nginx LB in front of the cluster and this is where I'm a bit baffled. It simply streams to the agent/ worker nodes (4 of them) with config like the following.
    server {
        listen 443;
        proxy_pass my_agent_nodes_443;
      }
    
      upstream my_agent_nodes_443 {
        server 185.230.138.110:30679;
        server 185.239.208.173:30679;
        server 38.242.195.7:30679;
        server 38.242.134.42:30679;
      }
    And, everything is working hunky dory (which is why I'm flabbergasted). My missing understanding is, how did I get the port numbers for the worker nodes i.e. 30679? I do all this for a hobby and did this work like 6 months ago, but didn't think to also document how I got to the port number and I can't find any mention of those ports in the emissary configs, only listeners for ports 80 and 443. 😮
  • b

    billions-airline-85860

    11/08/2022, 10:06 AM
    Any tips on this would be greatly appreciated. 🙂
  • b

    billions-airline-85860

    11/08/2022, 10:14 AM
    Hehe... two minutes after I wrote that, I figured it out. It's the "node port" set by k8s.
  • j

    jolly-waitress-71272

    11/09/2022, 7:27 PM
    I'm running postgres on the same host as a k3s server. How do I determine generically what CIDR I need to plop in pg_hba.conf in order for containers to connect to the db? I can see containers are being assigned from 10.42.0.0/16, is that always the default in K3s with flannel? Is there somewhere I can read from to get the exact CIDR?
    b
    • 2
    • 2
  • c

    clever-air-65544

    11/11/2022, 2:57 PM
    Weekly report is up! https://github.com/k3s-io/k3s/discussions/6483
    👍 1
    🚀 5
  • s

    shy-shampoo-22224

    11/14/2022, 3:41 PM
    Has anyone run into issues installing linkerd CNI with k3s flannel where it won’t assign an IP address?
    • 1
    • 1
  • g

    gray-florist-76480

    11/14/2022, 8:44 PM
    Hey! I wanted to run some TCP and UDP service in my home cluster and expose one of it's port to LAN. For HTTP services I'm using
    Ingress
    but not sure what about raw service like that which I need to access on predictable address.
    b
    c
    • 3
    • 5
  • f

    faint-airport-39912

    11/15/2022, 8:30 AM
    Hi All, Recently i have updated the k3s cluster version but after that i start getting below error in the logs and timeout response for the
    kubectl
    command so please suggest the solution to this problem ??
    Unable to connect to the server: net/http: TLS handshake timeout
    whenever i am getting timeout response for
    kubectl
    command the application exposed via ingress stop working and getting
    502
    response. These are the error logs from
    /var/log/syslog
    c
    • 2
    • 5
  • l

    limited-potato-16824

    11/16/2022, 2:41 PM
    G'day, we are using terraform for k3s upgrades and today we did a upgrade from v1.22.12 -> v1.22.15 but after that I can see a pod/job running:
    apply-k3s-master-plan-on-ip-1-2-3-444-with-37285dc25da-spg7r
    Inspecting the jobs it looks like it want to downgrade to the v1.22.12 version again since that is the version I can see in Rancher: I have read the page https://docs.k3s.io/upgrades/automated, our k3s clusters is managed by Rancher but we want to be in control of the upgrades. Is there any way to prevent Rancher from trying to run the upgrade plans since I suspect the upgrade plans is installed through the cattle-cluster-agent? I see this in our k3s cluster:
    $ kubectl get <http://plans.upgrade.cattle.io|plans.upgrade.cattle.io> -n cattle-system
    NAME              AGE
    k3s-worker-plan   7d1h
    k3s-master-plan   7d1h
    • 1
    • 1
  • f

    few-minister-97494

    11/16/2022, 4:48 PM
    Folks - regarding the k3s upgrade controller - there's an air-gapped k3s install scenario but - no corresponding air-gapped upgrade using the upgrade controller that I can see. E.g. if the Plan CRD offered a way to locate the binary/binaries on a local machine via SSH, or maybe S3 bucket running on a local Minio, or a persistent volume, etc. - that would support an air-gapped upgrade. Has anyone at Rancher given consideration to this?
    c
    • 2
    • 10
  • f

    future-fountain-82544

    11/17/2022, 10:31 PM
    I’m working on some orchestration to spin up a K3s cluster. Is there a sane way to pre-generate the the K3S_TOKEN file? For a really sillly reason out of the scope of this channel, it’s a big pain to copy a file off of the first K3s server onto other nodes
  • f

    future-fountain-82544

    11/17/2022, 10:33 PM
    It looks like the token file is in the format
    K[33 random bytes hex-encoded]::server:[16 random bytes hex-encoded]
    . The thing I’m not sure of is if the values are related (IE if they’re a public/private keypair or something)
    c
    • 2
    • 3
  • f

    famous-flag-15098

    11/18/2022, 4:13 PM
    Hi all! I need some support with an issue that has been plaguing me for months. ServiceLB, though disabled, continues to reinstall and break my LB configuration for certain pods. • I have a 6 node k3s cluster, half the nodes are ARM64, and the other half are Intel based. • 3 of the nodes are masters and the other 3 are agents.I am using metallb for a load balancer, along with a helm install of traefik. I have servicelb disabled in all node startup scripts. master
    ExecStart=/usr/local/bin/k3s \
    server \
            ' --disable' \
            'traefik' \
            '--disable' \
            'servicelb'
    agent
    ExecStart=/usr/local/bin/k3s \
        agent \
          ' --disable=traefik,servicelb'
    When I first encountered this, I added the above to the systemd scripts, did a daemon-reload and restart. All the servicelb nodes terminated and all seemed well. Approximately 2 weeks later, some of my services were not accessible and sure enough, I see that servicelb has started up again. What gives?
    • 1
    • 2
  • b

    bright-london-1095

    11/21/2022, 10:13 AM
    Hi All, Is there a procedure/steps to follow to downgrade the cluster version? Wanted to test how cleanly we can downgrade cluster without messing up things... 🧵 TIA
    c
    • 2
    • 5
  • j

    jolly-waitress-71272

    11/22/2022, 4:56 PM
    I'm trying to follow some instructions to set up some pods. What does this command do?
    k3s kubectl create secret generic kubeconfig --from-file=/etc/rancher/k3s/k3s.yaml
    I'm looking at https://jamesdefabia.github.io/docs/user-guide/kubectl/kubectl_create_secret_generic/ and what I think happens is the k3s.yaml gets stored in essentially a protected kubernetes var. Is this something people normally do for some reason? I don't see it referenced at any other point in the instructions.
    s
    q
    • 3
    • 3
  • b

    bright-london-1095

    11/23/2022, 1:06 AM
    Hi All, has any faced this error? I have installed
    k3s
    version
    1.22.8
    and the traffic pod is in crashLoopBackOff and not accessible. I tried deleting pod but didn't help!
    + helm_v3 install --set-string global.systemDefaultRegistry= traefik <https://10.49.0.22:443/static/charts/traefik-10.14.100.tgz> --value/config/values-01_HelmChart.yaml --values /config/values-10_HelmChartConfig.yaml
    Error: INSTALLATION FAILED: cannot re-use a name that is still in use
  • b

    bright-london-1095

    11/23/2022, 1:15 AM
    kube-system    pod/helm-install-traefik-q76r7                          0/1     CrashLoopBackOff   5 (63s ago)        4m32s
  • b

    bright-london-1095

    11/23/2022, 1:26 AM
    I got this from
    k3s
    logs:
    Nov 23 01:19:01 nestle-test k3s[1186]: I1123 01:19:01.070591    1186 scope.go:110] "RemoveContainer" containerID="e87d34e8fb2f4d3e07ab9ee9e8656ee73f0cd1dd00f7bce0ee9956d3bc091c67"
    Nov 23 01:19:01 nestle-test k3s[1186]: E1123 01:19:01.070873    1186 pod_workers.go:949] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"helm\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=helm pod=helm-install-traefik--1-8htfg_kube-system(9833a11e-3065-48d7-908f-baf37fef0168)\"" pod="kube-system/helm-install-traefik--1-8htfg" podUID=9833a11e-3065-48d7-908f-baf37fef0168
    Nov 23 01:19:03 nestle-test k3s[1186]: E1123 01:19:03.366518    1186 remote_runtime.go:164] "RemovePodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to remove volatile sandbox root directory \"/run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/345c21c66d2e341863fb9b95eb829a477f2c6e799016674d3b2a80fe113601a1\": unlinkat /run/k3s/containerd/io.containerd.grpc.v1.cri/sandboxes/345c21c66d2e341863fb9b95eb829a477f2c6e799016674d3b2a80fe113601a1/shm: device or resource busy" podSandboxID="345c21c66d2e341863fb9b95eb829a477f2c6e799016674d3b2a80fe113601a1"
  • b

    bright-london-1095

    11/23/2022, 1:31 AM
    traefik
    shipped with
    k3s
    (v1.22.8+k3s1)
  • e

    early-solstice-46134

    11/23/2022, 8:04 AM
    good morning from Germany, how can I get the logs of the embedded
    containerd
    daemon inside of K3s? I need to debug some stuff regarding our private registry, but I can't dig deep enough.
    journalctl -u k3s | grep <imagenaname>
    does not show anything related to
    containerd
    pulling/pushing said image, even when I start k3s with the
    --debug
    flag. The logs for k3s show that it picks up the
    /etc/rancher/k3s/registries.yaml
    file correctly.
    • 1
    • 1
  • k

    kind-helmet-90731

    11/24/2022, 6:39 AM
    evening, I was attempting to upgrade my k3s cluster to v1.25.4+k3s1 and it seems to have gone ok, except that all my ingresses are now not functional. Reviewing the logs in the traefik pod reveals
  • k

    kind-helmet-90731

    11/24/2022, 6:39 AM
    time="2022-11-24T05:48:21Z" level=info msg="Configuration loaded from flags."
    time="2022-11-24T05:51:04Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes ingress=prometheus-server serviceName=prometheus-server namespace=prometheus servicePort="&ServiceBackendPort{Name:,Number:80,}"
    time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=prometheus serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}" providerName=kubernetes ingress=prometheus-server
    time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" namespace=opensearch providerName=kubernetes ingress=opensearch-dashboards
    time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=prometheus servicePort="&ServiceBackendPort{Name:,Number:80,}" serviceName=prometheus-server providerName=kubernetes ingress=prometheus-server
    time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" namespace=opensearch providerName=kubernetes serviceName=opensearch-dashboards servicePort="&ServiceBackendPort{Name:,Number:5601,}" ingress=opensearch-dashboards
    time="2022-11-24T05:51:05Z" level=error msg="Skipping service: no endpoints found" providerName=kubernetes serviceName=prometheus-server servicePort="&ServiceBackendPort{Name:,Number:80,}" ingress=prometheus-server namespace=prometheus
    c
    • 2
    • 10
  • k

    kind-helmet-90731

    11/24/2022, 6:42 AM
    I performed a downgrade back to v1.25.3+k3s1 and still getting the same behavior above. Any recommendations on how to clear this and get the cluster ingresses back online? Not sure what to do next. I can provide an ingress definition of one of my services if it will help diagnose.
  • b

    bright-postman-91926

    11/24/2022, 7:41 AM
    Hello K3s community. Hopefully I am in the right place for my enquiries. I have been charged with verifying a K3s deployment against the CIS benchmark controls. For this I am using the CIS benchmark v1.6 from the CIS community coupled with the Hardening guide from the K3s website (https://docs.k3s.io/security/hardening-guide) and the self assessment guide on the same site (https://docs.k3s.io/security/self-assessment). I understand that those guides on the K3s site are intended to match a slightly later revision of the CIS benchmark, however there is very little difference and I have successfully run through all the checks except for the following: CIS Control 4.2.9 - Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture CIS Control 4.2.13 - Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers For both these checks CIS recommends running "ps -ef |grep kubelet" on each worker node - which as I understand is not suitable for a k3s deployment. However the recommended alternative from the K3s self assessment guide seems equally unsuitable "/bin/ps -fC containerd" So my question(s) is, am I missing something or is the K3s approach to that control miss-documented? Can we even pass those parameters to the K3s binary on the worker nodes? if so, is there another way I can satisfy these checks? be it pass or fail. All and any help would be greatly appreciated. Even if its just to point me to another channel/slack I should be enquiring on.
    l
    • 2
    • 7
  • f

    full-park-34540

    11/24/2022, 3:36 PM
    https://rancher-users.slack.com/archives/C3ASABBD1/p1669301884658589
  • c

    creamy-room-58344

    11/25/2022, 3:21 PM
    Hello there! I build a k3s cluster with 3 nodes and installed Rancher on it via Helm Can't Rancher deploy a new cluster with k3s? The only options available are those on the image ps.: I had to install k3s v1.24.8+k3s1 cause Rancher helm char requires < v1.25.0)
    • 1
    • 2
  • h

    handsome-painter-48813

    11/25/2022, 4:24 PM
    Hello, I got something strage
    overlay         468G   15G  430G   4% /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/5e03898951de490bdc8e32ca67c24a1c691cf9efa8629ee494460f3fc5a6bc9b/rootfs
    It goes up by 1 GB each second, how can I find container that does this?
    • 1
    • 1
  • r

    red-boots-23091

    11/25/2022, 5:36 PM
    Hey All, I am using rancher v.2.5.7, single node docker install. on top of ubuntu server 20.04 lTS. The rancher/rancher:v2.5.7 container will not stay up longer than a few minutes generally it gives a k3s fatal 255 (this is a recent problem I have been running this server for 19 months) my logs are in this pastbin can anyone help me diagnose this issue? https://pastebin.com/0zwxEiJ5
  • f

    flat-continent-80260

    11/26/2022, 12:35 AM
    New flatcar deployment. Installed k3s without an issue. However, I can't get past the bootstrap login. I suspect this is due to k3s using kubernetes 1.25. I was able to get rancher to install by manually pulling the helm chart and removing the <1.25 requirement. I also tried versions 2.7.0. just curious if there is an ETA for 1.25 support. Or if there is a current work around? Or maybe I just need to use an older version of k3s?
    • 1
    • 1
Powered by Linen
Title
f

flat-continent-80260

11/26/2022, 12:35 AM
New flatcar deployment. Installed k3s without an issue. However, I can't get past the bootstrap login. I suspect this is due to k3s using kubernetes 1.25. I was able to get rancher to install by manually pulling the helm chart and removing the <1.25 requirement. I also tried versions 2.7.0. just curious if there is an ETA for 1.25 support. Or if there is a current work around? Or maybe I just need to use an older version of k3s?
Just gave an older version of k3s a whirl and used 1.24.8.
curl -sfL <https://get.k3s.io|https://get.k3s.io> | INSTALL_K3S_VERSION=v1.24.8+k3s1 INSTALL_K3S_BIN_DIR=/opt/bin sh -
Everything deployed without issue. But still feels strange using an older version
View count: 14