https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
k3s
  • g

    gifted-morning-94496

    09/11/2022, 10:31 AM
    #k3s how do I create the k3s cluster with 3 master and 3 worker nodes with a single command line? do we have any configuration file (having all the node details) and pass it as argument to k3s .. please let me know !!
    c
    • 2
    • 3
  • l

    lively-tailor-38572

    09/12/2022, 11:38 PM
    hello. I'm trying to link k3s running rootless together with nerdctl. I can get k3s rootless going pretty easily and I have nerdctl working rootless using their containerd-rootless-setuptool.sh. That's where the success ends.
  • l

    lively-tailor-38572

    09/12/2022, 11:42 PM
    If I don't do the nerctl rootless stuff and use nsenter to run nerdctl I can connect to the k3s containerd and use nerdctl ps and images. I can get buildkit started with nsenter buildkitd and other settings but when it starts building I get errors when the Dockerfile does a COPY
  • l

    lively-tailor-38572

    09/12/2022, 11:44 PM
    anyone have any experience with this or can give me some pointers? end goal is to be able to use nerdctl to start and build containers while I transition my teams development environment from docker to k8s but also be able to use k3s.
  • m

    melodic-hamburger-23329

    09/14/2022, 1:05 AM
    Is it possible to use the upgrade image for upgrading single-node cluster? Tried with the approach described here and here, but it doesn’t seem to work as expected. Upgrade image is executed, but
    kubectl version
    still shows old server version.
    c
    • 2
    • 9
  • c

    chilly-telephone-51989

    09/15/2022, 7:57 AM
    (message removed)
  • c

    cuddly-egg-57762

    09/16/2022, 10:01 AM
    Hi people! I need a hand to understand why calico installation job fails when I install it via helm-controller. I'm trying to deploy this helmchart resource:
    apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
    kind: HelmChart
    metadata:
      name: cilium
      namespace: kube-system
    spec:
      bootstrap: True
      chart: cilium/cilium
      version: 1.12.1
      targetNamespace: kube-system
      valuesContent: |-
        operator:
          replicas: 2
          image:
            useDigest: false
        tunnel: disabled
        autoDirectNodeRoutes: true
        kubeProxyReplacement: strict
        loadBalancer:
          standalone: false
          mode: dsr
        k8sServiceHost: 10.130.42.39
        k8sServicePort: 6443
        nativeRoutingCIDR: 10.0.0.0/16
        image: 
          useDigest: false
          pullPolicy: IfNotPresent
    but the job pod fails with this error:
    + echo 'Installing helm_v3 chart'
    + helm_v3 install --namespace kube-system --version 1.12.1 cilium cilium/cilium --values /config/values-01_HelmChart.yaml
    Error: INSTALLATION FAILED: failed to download "cilium/cilium" at version "1.12.1"
    if I try do install the chart manually (
    helm install cilium cilium/cilium ...
    ), it works
    c
    • 2
    • 1
  • c

    clever-air-65544

    09/16/2022, 2:29 PM
    Hey friends, here's what the k3s team worked on this week: https://github.com/k3s-io/k3s/discussions/6149
  • j

    jolly-waitress-71272

    09/16/2022, 5:12 PM
    What would the process look like if I had 1 master+worker, 3 worker nodes and I wanted to swap to 3 master+worker, 1 worker? Would there be anything to it more than just draining the two existing nodes, deleting them with
    k delete node exampleNode
    , uninstalling k3s on them, then following https://rancher.com/docs/k3s/latest/en/installation/ha/#2-launch-server-nodes
    c
    • 2
    • 19
  • b

    bitter-furniture-26042

    09/16/2022, 11:38 PM
    Hey there folks! I'd like to ask a question about creating service accounts in k3s and their related access tokens - I'm using Argo Workflows locally (for development purposes) and somehow I can't seem to get the token I need for accessing my local Argo programatically. More on this thread.
    c
    • 2
    • 9
  • w

    wooden-angle-771

    09/17/2022, 10:15 AM
    message has been deleted
    c
    • 2
    • 2
  • j

    jolly-waitress-71272

    09/20/2022, 5:34 PM
    k3s doesn’t have masters/workers, instead has servers/agents
    k
    • 2
    • 2
  • g

    gifted-morning-94496

    09/22/2022, 5:42 AM
    #k3s have anybody installed elastic stack on K3s in airgap environment ?
  • c

    cuddly-jordan-17092

    09/22/2022, 6:17 AM
    Hi All Pen test reported that weak & invalid ssl communication is in use by one k3s server. Literally, i am a very new to this k3's. Could some one point me on how to fix the below issue
    openssl s_client -connect kube1001:6443 2>&1 | grep issuer
    verify error:num=20:unable to get local issuer certificate
    issuer=/CN=k3s-server-ca@1649270763
    
    openssl s_client -connect kube1001:443 2>&1 | grep issuer
    verify error:num=20:unable to get local issuer certificate
    issuer=/CN=k3s-server-ca@1649270763
    
    openssl s_client -connect kube1001:80 2>&1 | grep issuer
    verify error:num=20:unable to get local issuer certificate
    issuer=/CN=k3s-server-ca@1649270763
    b
    • 2
    • 22
  • b

    bland-account-99790

    09/22/2022, 3:16 PM
    Hey! We would like to understand what k3s users are choosing as flannel backend. We created a one-question survey and we would be very thankful if you could answer it: https://github.com/k3s-io/k3s/discussions/6170
  • c

    clever-analyst-23771

    09/22/2022, 3:28 PM
    Hello <!channel> we rarely use the
    channel
    but this survey helps us drive direction and is critically important to the k3s team, sorry and thank you for your support.
    ✅ 6
    🚀 4
  • f

    fancy-monitor-61453

    09/22/2022, 3:30 PM
    @bland-account-99790 and @clever-analyst-23771 we dont use flannel but are working on a much faster fabric for gpus from cloud to iot. ML use cases are very diff. we are working towards something like the AWS elastic fabric adapter.
    👍 1
    s
    b
    +3
    • 6
    • 17
  • b

    bulky-computer-31499

    09/22/2022, 3:31 PM
    I’ve never had to stray from the default vxlan
    👍 3
  • h

    helpful-stone-91643

    09/22/2022, 3:51 PM
    been using Wireguard as I have servers in the cloud and agents in the edge
    👌 1
  • a

    adamant-vegetable-68940

    09/23/2022, 2:12 AM
    IIUC K3s uses traefik by default. I knew that there is an annotation to redirect to SSL ports on the ingress. Did this change recently? The annoation used to be
    '<http://traefik.ingress.kubernetes.io/redirect-entry-point|traefik.ingress.kubernetes.io/redirect-entry-point>': 'https',
    c
    • 2
    • 1
  • n

    narrow-area-44893

    09/23/2022, 2:55 AM
    Howdy, I am working on getting my first k3s cluster up and running and was wanted to expose some services externally with metallb. Has anyone been able to get the latest version of metallb running with a rancher deployed k3s cluster? I found a few references to metallb and k3s in the search, but nothing yet that seemed to be hitting my particular issue. Thanks for any help,
    h
    b
    • 3
    • 5
  • b

    bitter-nightfall-76021

    09/23/2022, 2:21 PM
    Hi team, all my dev environments are k3s based. I use vcluster to spin up k3s clusters. Now I am testing istio-cni and I can see the following error with Calico and Cilium
    Error: failed to create ambient informer service: error getting host IP: <nil>
    I do not expect anybody to understand that, but at least to point me what could be te reason the the process cannot get the host IP. Thanks!
  • b

    bland-painting-61617

    09/24/2022, 11:56 AM
    I'm experimenting with an idea of hosting the control plane of my K3s based cluster away from home - in another Kubernetes cluster hosted in the cloud. I've created the needed manifests and got it all working using the
    --disable-agent
    flag so that the control plane pod is not registering itself as a node. The control plane environment is accessed by a public IP which works well, I can get pod logs and shell from my test workstation and from a side container on the control plane node I can curl to services inside the cluster which confirms the built in proxying is working - however, when I deploy gatekeeper, the control plane is not able to execute the webhook (which is strange because I can curl that webhook from the control plane pod sidecar just fine).
    Error from server (InternalError): error when creating "<https://raw.githubusercontent.com/metallb/metallb/v0.12.1/manifests/namespace.yaml>": Internal error occurred: failed calling webhook "check-ignore-label.gatekeeper.sh": failed to call webhook: Post "<https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admitlabel?timeout=3s>": context deadline exceeded
    The above is the error and below is the curl from the CP side car:
    / # curl "<https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/admit?timeout=3s>" -Ivk
    *   Trying 10.0.17.2:443...
    * Connected to gatekeeper-webhook-service.gatekeeper-system.svc (10.0.17.2) port 443 (#0)
    * ALPN: offers h2
    * ALPN: offers http/1.1
    * TLSv1.3 (OUT), TLS handshake, Client hello (1):
    * TLSv1.3 (IN), TLS handshake, Server hello (2):
    * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    * TLSv1.3 (IN), TLS handshake, Certificate (11):
    * TLSv1.3 (IN), TLS handshake, CERT verify (15):
    * TLSv1.3 (IN), TLS handshake, Finished (20):
    * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
    * TLSv1.3 (OUT), TLS handshake, Finished (20):
    * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
    * ALPN: server accepted h2
    * Server certificate:
    *  subject: CN=gatekeeper-webhook-service.gatekeeper-system.svc
    *  start date: Aug 23 11:45:37 2022 GMT
    *  expire date: Aug 23 11:55:37 2024 GMT
    *  issuer: CN=gatekeeper-ca
    *  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
    The connectivity is definitely there, but somehow something isn't connecting. The k3s cp process logs:
    k3s I0924 11:53:04.418026      46 trace.go:205] Trace[729582201]: "Proxy via http_connect protocol over tcp" address:10.42.0.8:8443 (24-Sep-2022 11:50:54.455) (total time: 129962ms):
    k3s Trace[729582201]: [2m9.962952731s] [2m9.962952731s] END
    k3s I0924 11:53:04.418106      46 trace.go:205] Trace[975991258]: "Proxy via http_connect protocol over tcp" address:10.42.0.7:8443 (24-Sep-2022 11:50:53.454) (total time: 130963ms):
    k3s Trace[975991258]: [2m10.963777266s] [2m10.963777266s] END
    k3s I0924 11:53:04.418387      46 trace.go:205] Trace[1687549248]: "Proxy via http_connect protocol over tcp" address:10.42.0.7:8443 (24-Sep-2022 11:50:53.454) (total time: 130964ms):
    k3s Trace[1687549248]: [2m10.964212472s] [2m10.964212472s] END
    k3s I0924 11:53:04.418121      46 trace.go:205] Trace[1695746900]: "Proxy via http_connect protocol over tcp" address:10.42.0.9:8443 (24-Sep-2022 11:50:54.944) (total time: 129473ms):
    k3s Trace[1695746900]: [2m9.473126709s] [2m9.473126709s] END
    k3s I0924 11:53:04.418155      46 trace.go:205] Trace[1578279874]: "Proxy via http_connect protocol over tcp" address:10.42.0.8:8443 (24-Sep-2022 11:50:54.457) (total time: 129961ms):
    k3s Trace[1578279874]: [2m9.961131206s] [2m9.961131206s] END
    k3s I0924 11:53:04.418160      46 trace.go:205] Trace[406850929]: "Proxy via http_connect protocol over tcp" address:10.42.0.7:8443 (24-Sep-2022 11:50:53.944) (total time: 130473ms):
    k3s Trace[406850929]: [2m10.473654339s] [2m10.473654339s] END
    k3s I0924 11:53:04.418208      46 trace.go:205] Trace[1213933259]: "Proxy via http_connect protocol over tcp" address:10.42.0.9:8443 (24-Sep-2022 11:50:54.455) (total time: 129962ms):
    k3s Trace[1213933259]: [2m9.962336223s] [2m9.962336223s] END
    k3s I0924 11:53:04.418239      46 trace.go:205] Trace[1402600249]: "Proxy via http_connect protocol over tcp" address:10.42.0.7:8443 (24-Sep-2022 11:50:55.121) (total time: 129297ms):
    k3s Trace[1402600249]: [2m9.297126293s] [2m9.297126293s] END
    k3s I0924 11:53:04.418302      46 trace.go:205] Trace[1277249521]: "Proxy via http_connect protocol over tcp" address:10.42.0.8:8443 (24-Sep-2022 11:50:54.119) (total time: 130298ms):
    k3s Trace[1277249521]: [2m10.298506336s] [2m10.298506336s] END
    k3s I0924 11:53:04.418305      46 trace.go:205] Trace[688495020]: "Proxy via http_connect protocol over tcp" address:10.42.0.8:8443 (24-Sep-2022 11:50:53.454) (total time: 130964ms):
    k3s Trace[688495020]: [2m10.964143771s] [2m10.964143771s] END
    k3s I0924 11:53:05.376142      46 trace.go:205] Trace[1922244545]: "Call mutating webhook" configuration:gatekeeper-mutating-webhook-configuration,webhook:mutation.gatekeeper.sh,resource:/v1, Resource=configmaps,subresource:,operation:UPDATE,UID:bfb30054-8598-496e-9cd7-cbb4765fa8e1 (24-Sep-2022 11:53:04.375) (total time: 1000ms):
    k3s Trace[1922244545]: [1.000712635s] [1.000712635s] END
    k3s W0924 11:53:05.376198      46 dispatcher.go:180] Failed calling webhook, failing open mutation.gatekeeper.sh: failed calling webhook "mutation.gatekeeper.sh": failed to call webhook: Post "<https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/mutate?timeout=1s>": context deadline exceeded
    k3s E0924 11:53:05.376218      46 dispatcher.go:184] failed calling webhook "mutation.gatekeeper.sh": failed to call webhook: Post "<https://gatekeeper-webhook-service.gatekeeper-system.svc:443/v1/mutate?timeout=1s>": context deadline exceeded
    Any ideas why the proxying wouldn't work for the webhook but works for logs and curl?
    c
    • 2
    • 14
  • l

    limited-potato-16824

    09/26/2022, 5:30 AM
    Hi, I'm using k3s and have mysql as backend server and have now run into a issue where I have reached the maximum number for the
    id
    field. I receive the error
    Duplicate entry '2147483647' for key 'PRIMARY'
    from the database. Any hints on how to fix this?
    c
    • 2
    • 9
  • c

    careful-alarm-5840

    09/26/2022, 6:58 AM
    Does rancher/suse provide addon license for k3s ?
  • a

    aloof-article-40011

    09/26/2022, 4:23 PM
    Hi everyone, I posted this in #general but I think this is more appropriate. I'm hoping I can get some help standing up a k3s cluster on bare metal: I'm using k3sup, and I've setup 3 servers and 3 workers, they all show as ready.
    flannel-backend=host-gw
    All nodes are reachable between each other and from workers, I see the cni0 interface and I can ping pods from the host. On server I cannot ping pods form the host, and there is no cni0 interface. Servers have a
    --node-taint='k3s-controlplane=true:NoSchedule'
    taint, which I assumed meant that non system workloads aren't scheduled, but looks like no workloads are scheduled on workers at all, including coredns, etc. I ran out of places to look for a cause, does anyone have any insight? Thanks in advance
    b
    c
    • 3
    • 28
  • s

    stale-author-3655

    09/27/2022, 6:36 AM
    Is Postgres a commonly used backend? Does anyone have any performance data vs etcd?
    c
    n
    a
    • 4
    • 8
  • w

    wooden-library-55047

    09/27/2022, 3:19 PM
    I have a question about the "pending" status for my cluster, am I able to re-register a cluster with the same name as one I deleted? I attempted to test system upgrade controller on the cluster but it became unreachable so I had to delete it and launch a new cluster with the same name. The new cluster is stuck in the pending state and I am not sure how to get it active.
    c
    • 2
    • 1
  • n

    narrow-area-44893

    09/28/2022, 4:11 AM
    I know there is an open issue out there with a little bit of action every once in a while https://github.com/rancher/rancher/issues/36463 But just curious if any one out here has any info on if arm64 is going to be supported from the rancher UI when creating clusters.
  • n

    nutritious-apartment-10061

    09/28/2022, 5:16 PM
    where does k3s store the client-ca and in how much pain would I be if I rotate it?
    n
    • 2
    • 18
Powered by Linen
Title
n

nutritious-apartment-10061

09/28/2022, 5:16 PM
where does k3s store the client-ca and in how much pain would I be if I rotate it?
n

nutritious-tomato-14686

09/29/2022, 9:01 PM
K3s can auto rotate the certs https://docs.k3s.io/advanced#certificate-rotation
The cert is located in
var/lib/rancher/k3s/server/tls/client-ca.crt
if you want to go messing around with it, but I wouldn't recommend it
You can also use the
k3s certificate
subcommand to do the rotation on demand
Process is not documented (TODO for me), but you 1. stop the k3s server 2. run
k3s certificate rotate
3. start the K3s server
n

nutritious-apartment-10061

09/30/2022, 5:56 AM
k3s certificate rotate specifically doesn’t rotate that one, that's why I’m here :)
literally removing that file makes k3s recreate it
n

nutritious-tomato-14686

09/30/2022, 7:11 PM
Yeah I chatted with @fancy-guitar-13855 (he wrote the cert rotation command) and he mentioned that you can just delete that file and K3s will make a new one.
n

nutritious-apartment-10061

09/30/2022, 7:12 PM
https://github.com/k3s-io/k3s/blob/master/pkg/cli/cert/cert.go#L125-L184 I can't find the place in the code where it removes the client ca though
and in here https://github.com/k3s-io/k3s/issues/5147#issuecomment-1048269049 it's stated that the client ca is indeed not rotated
I'm fine if I need to go thorugh the hoops to rotate it; I just want to know which those involve 🙂
n

nutritious-tomato-14686

09/30/2022, 7:14 PM
So I got more info and we specifically don't rotate any ca* certs because its a Back-Compat thing with RKE1. When you removed the file, K3s regenerated a new one?
n

nutritious-apartment-10061

09/30/2022, 7:16 PM
nope; it restored the old cert from someplace
I stopped k3s, rm-rf'ed the client-ca*, started it and got the same client-ca back
that's still confusing as to how that works
n

nutritious-tomato-14686

09/30/2022, 8:36 PM
Okay I did some more digging. Currently, there is no easy way to regenerate the client or server -ca keys/crts. This is because we cache the keys in the etcd/db. They are static for the life of that cluster. If you kill the cluster and remove the files, on restart, k3s will simple pull down the cached version from the DB and regenerate them.
The only way to get new keys is to completely wipe out the K3s cluster...
k3s server --cluster-reset
or a k3s-uninstall.sh.... but that defeats the point of trying to "rotate" the ca keys.
n

nutritious-apartment-10061

09/30/2022, 9:57 PM
aha, I thought they'd be in etcd; that explains it.
a bit unfortunate; but yeah; wiped the cluster, learned my lesson. No more CA auth.
View count: 37