Hi Everybody, short question: Is it possible to join nodes by individual tokens with expiration or something like this? Or is there another way to securely remove a (maybe) compromised node?

Hi all, having problems with my 5 node k3s cluster (1.22) on k3os (1 node control plane with sqllite). Not quite sure what happened, but basically, I’m in a state after some restarts where I can’t even get apiserver to start. TLDR of the issue is k3s takes forever to start, once it does and starts to kick off containerd, all that happens in the log is a bunch of TLS errors with bad handshakes and a bunch of

authentication.go:63] "Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"

. … I’m basically unable to run kubectl or anything and my pods never come online .. eventually k3s-service just crashes and restarts. Not quite sure where to begin troubleshooting.

Though the more I am looking ,it seems like k3os isnt really a supported install method anymore 😛

Just to sanity check, since Rancher 2.7.0 lists Kubernetes v1.24.6 (Default), that means it will NOT work w/ K3s v1.25+?

Hi everyone I'm trying to set a node taint but it doesn't seem to be working I'm currently passing the following argument:

--node-taint <http://node-role.kubernetes.io/master=true:NoSchedule|node-role.kubernetes.io/master=true:NoSchedule>

but my master nodes still show

Taints: <none>

when I check the node annotations

<http://k3s.io/node-args|k3s.io/node-args>

contains

[..., "--node-taint", "<http://node-role.kubernetes.io/master=true:NoSchedule|node-role.kubernetes.io/master=true:NoSchedule>"]

Hi! X-posting here, hoping to get some input. 🙂 Summary: We are trying to track down a flake test issue in our operator end-to-end tests. The operator under test is installed into a single node ephemeral k3s (k3d) cluster running on hosted GitHub runners (with limited resources). More details in the Kubernetes Slack post: https://kubernetes.slack.com/archives/C18NZM5K9/p1674144533725179

Is there any way to use aws lb with k3s? I do not want to manually create lb and let k3s handle it.

Hi! I’m struggling with an issue related to rke v1.4.1, particularly there I’m getting errors like:

Failed to upgrade hosts: 192.168.199.237 with error [host 192.168.199.237 not ready]

. On the nodes I’m seeing errors like this in `kubelet`:

E0120 20:34:36.159187   28876 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
E0120 20:34:41.170900   28876 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
E0120 20:34:46.183912   28876 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
E0120 20:34:51.195207   28876 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"
E0120 20:34:56.207977   28876 kubelet.go:2349] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized"

Only the base containers are started:

ubuntu@worker-1:~$ docker ps
CONTAINER ID   IMAGE                                COMMAND                  CREATED          STATUS          PORTS     NAMES
185b536f265f   rancher/hyperkube:v1.24.8-rancher1   "/opt/rke-tools/entr…"   17 minutes ago   Up 16 minutes             kube-proxy
bc10253b52f5   rancher/hyperkube:v1.24.8-rancher1   "/opt/rke-tools/entr…"   17 minutes ago   Up 17 minutes             kubelet
9065c75bf9f4   rancher/rke-tools:v0.1.88            "nginx-proxy CP_HOST…"   17 minutes ago   Up 17 minutes             nginx-proxy

We have the following options set:

ignore_docker_version: false
enable_cri_dockerd: true

Do you have ay pointers on how to proceed with debugging ? A few days ago we upgraded to K8S 1.24 and also added a few nodes. Everything was fine until a physical reboot.

For the control plane we have 3 nodes with controlplane and etcd roles. Two of them are operating without errors and one of them with exactly the same error as the workers

Hi folks, I'm curious how are you guys using k3s to build an IoT/Edge solution? Connecting various IoT devices to the cluster is still a huge pain... I found EdgeXFoundry but it's not k8s native. Rancher has previously built octopus but it got shutdown. [cnrancher/octopus: Lightweight device management system for Kubernetes/k3s (github.com) ]

hi all, recently I faced an error after restarting k3s.service on my on-premise environment. Since I was not able to find out the root cause, immediately restored from snapshot. My environment consists of 2 master servers with local etcd database. The first server came up and running after the restore. But another one is not able to join the cluster since restore. I read the github documentation and issues, and found that "restore from snapshot" option restores cluster with only one server and others will stay outside of the cluster. So I deleted -data-dir on the second server and tried to rejoin, but with no luck. It says the below, do you have any suggestions on this? Thanks.

INFO[0005] Tunnel server egress proxy waiting for runtime core to become available
INFO[0005] Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error
INFO[0010] Tunnel server egress proxy waiting for runtime core to become available
INFO[0010] Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error
{"level":"info","ts":"2023-01-23T15:28:45.167+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"ef75232eb2491d72 switched to configuration voters=(5985293894695459910 17254736230806527346)"}
{"level":"info","ts":"2023-01-23T15:28:45.167+0100","caller":"membership/cluster.go:535","msg":"promote member","cluster-id":"96831dd7fb68f941","local-member-id":"ef75232eb2491d72"}
INFO[0015] Tunnel server egress proxy waiting for runtime core to become available
INFO[0015] Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error
{"level":"warn","ts":"2023-01-23T15:28:50.015+0100","caller":"etcdserver/server.go:2065","msg":"failed to publish local member to cluster through raft","local-member-id":"ef75232eb2491d72","local-member-attributes":"{Name:server-002.e636bb1a ClientURLs:[<https://10.31.194.2:2379>]}","request-path":"/0/members/ef75232eb2491d72/attributes","publish-timeout":"15s","error":"etcdserver: request timed out, possibly due to connection lost"}
{"level":"info","ts":"2023-01-23T15:28:50.015+0100","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"ef75232eb2491d72","local-member-attributes":"{Name:aserver-002.e636bb1a ClientURLs:[<https://10.31.194.2:2379>]}","request-path":"/0/members/ef75232eb2491d72/attributes","cluster-id":"96831dd7fb68f941","publish-timeout":"15s"}

You can’t have a two-node etcd cluster.

You should have 1/3/5 or so on. If you have two nodes, neither one can function without the other. That is just how quorum works.

@orange-airplane-98016 has left the channel

Hi, two dumb questions about DB: 1. I am using

k3s server

to start the server. According to the documentation, the default datastore should be sqlite. I did see one log about sqlite:

time="2023-01-25T003034Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"

But I also saw logs about etcd:

time="2023-01-25T003037Z" level=info msg="ETCD server is now running"

So, is

sqlite

or

etcd

running in my case? 2. In any case, I did not see the

db

directory in my

data-dir

. Where can I find the DB files then?

Hi! We're new to k3s, and hoping to be able to verify downloaded binaries (from https://github.com/k3s-io/k3s/releases and https://hub.docker.com/r/rancher/klipper-lb etc.) ideally via a detached GPG signature with a previously published/well-known GPG key from the k3s maintainers. I've looked over the docs/FAQ, github issues and searched slack so far. It it the case that no such signatures are available? Thank you for any thoughts on this!

Question on customizing containerd. I’ve read: • https://docs.k3s.io/advanced#configuring-containerd and • https://github.com/k3s-io/k3s/blob/master/pkg/agent/templates/templates_linux.go Is that approach the recommended way of doing things in regards to customizing root = "/var/lib/containerd" and   state = "/run/containerd" As mentioned in: https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration ??? —- The goal is to have container images and other non runtime related containerd data on a non system disk to avoid disk space usages on system disks. Thank you

I apologize if this question has already been answered. I just recently joined this slack channel. I am using Windows Rancher Desktop (v1.7.0) with WSL to develop a local Kubernetes cluster (v1.25.6) with pods the need to federate a projected service account to a managed identity in Azure. I have generated RSA public/private key files, which will be used by the Kubernetes API server to sign the projected service account token and by Azure to verify the token's signature. For this, I use a provisioning script that • generates a

config.yaml

file in the

/etc/rancher/k3s

directory to configure the k3s kube-apiserver service, • copies the RSA public/private key files to the

/etc/rancher/k3s/pki

directory, • change the mode of the RSA key files to 644. When I start Rancher Desktop and review the k3s.log file, the kube-apiserver service tries to start with the updated service-account-* command line arguments. However, the kube-apiserver service writes this error to the log, "`Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory`". I have opened the rancher-desktop WSL distro and verified that I can read both key files at their specified locations. I have also tried placing the keys files in a

/var/lib/rancher/k3s/server/pki

directory and updated the config.yaml to point to the new directory. However, the kube-apiserver service throws the same error. Unfortunately, I have not found much on the internet for a go-forward path. Should I write the RSA keys to a different directory in order for k3s to find the RSA key files? Is there another setting that I need to configure for k3s to have?

Rancher Desktop 1.7.0
Kubernetes 1.25.6

provisioning script:

#!/bin/sh

mkdir -p /etc/rancher/k3s
cat > /etc/rancher/k3s/config.yaml <<EOF
kube-apiserver-arg:
- service-account-signing-key-file="/etc/rancher/k3s/pki/sa.key"
- service-account-key-file="/etc/rancher/k3s/pki/sa.pub"
- service-account-issuer="<https://www.example.net/oidc-test/>"
EOF

mkdir -p /etc/rancher/k3s/pki
cat > /etc/rancher/k3s/pki/sa.key <<EOF2
-----BEGIN PRIVATE KEY-----
MII...
-----END PRIVATE KEY-----
EOF2
chmod 644 /etc/rancher/k3s/pki/sa.key

cat > /etc/rancher/k3s/pki/sa.pub <<EOF3
-----BEGIN PUBLIC KEY-----
MII...
-----END PUBLIC KEY-----
EOF3
chmod 644 /etc/rancher/k3s/pki/sa.pub

k3s.log

time="2023-01-27T16:40:54Z" level=info msg="Starting k3s v1.25.6+k3s1 (9176e03c)"
time="2023-01-27T16:40:54Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2023-01-27T16:40:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2023-01-27T16:40:55Z" level=info msg="Database tables and indexes are up to date"
time="2023-01-27T16:40:55Z" level=info msg="Kine available at <unix://kine.sock>"
time="2023-01-27T16:40:55Z" level=info msg="Reconciling bootstrap data between datastore and disk"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy mode: agent"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=<unix://kine.sock> --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=\"<https://csgcorpitaaioidcissuer01.blob.core.windows.net/oidc-test/>\" --service-account-key-file=\"/etc/rancher/k3s/pki/sa.pub\" --service-account-signing-key-file=\"/etc/rancher/k3s/pki/sa.key\" --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-01-27T16:40:55Z" level=info msg="Waiting for API server to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
I0127 16:40:55.053432     436 server.go:581] external host was not specified, using 192.168.67.3
Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory
time="2023-01-27T16:40:55Z" level=fatal msg="apiserver exited: failed to parse service-account-issuer-key-file: open \"/etc/rancher/k3s/pki/sa.key\": no such file or directory"
time="2023-01-27T16:40:55Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"

Hello everyone, my name is Hergy

i am new to k3s slack channel

i would like to contribute

Hi, using multipass and k3s, is it possible to push local docker image to k3s easily? The end goal is to try tilt development environment. Anyone using tilt with k3s on mac?

The following error occurred during startup，What caused it

I0131 14:25:43.970982    9050 shared_informer.go:262] Caches are synced for certificate-csrsigning-kubelet-client
I0131 14:25:43.970965    9050 shared_informer.go:262] Caches are synced for certificate-csrsigning-kubelet-serving
I0131 14:25:44.022011    9050 shared_informer.go:262] Caches are synced for certificate-csrapproving
time="2023-01-31T14:25:44+08:00" level=info msg="Stop pulling image rancher/mirrored-metrics-server:v0.6.2: Status: Downloaded newer image for rancher/mirrored-metrics-server:v0.6.2"
I0131 14:25:44.350653    9050 shared_informer.go:262] Caches are synced for garbage collector
I0131 14:25:44.350669    9050 garbagecollector.go:163] Garbage collector: all resource monitors have synced. Proceeding to collect garbage
I0131 14:25:44.386944    9050 shared_informer.go:262] Caches are synced for garbage collector
W0131 14:25:44.864126    9050 handler_proxy.go:105] no RequestInfo found in the context
W0131 14:25:44.864125    9050 handler_proxy.go:105] no RequestInfo found in the context
E0131 14:25:44.864185    9050 controller.go:116] loading OpenAPI spec for "<http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io>" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0131 14:25:44.864191    9050 controller.go:113] loading OpenAPI spec for "<http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io>" failed with: Error, could not get list of group versions for APIService
I0131 14:25:44.864196    9050 controller.go:129] OpenAPI AggregationController: action for item <http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io>: Rate Limited Requeue.
I0131 14:25:44.865275    9050 controller.go:126] OpenAPI AggregationController: action for item <http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io>: Rate Limited Requeue.
time="2023-01-31T14:25:45+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"

Is this error because the host IP is not added to the tls-san

msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
time="2023-02-01T15:11:34+08:00" level=info msg="Using CNI configuration file /xj-data/k3s/data/agent/etc/cni/net.d/10-flannel.conflist"
time="2023-02-01T15:11:37+08:00" level=info msg="certificate CN=id-test-it-kubernetes-10.7.4.4-master02 signed by CN=k3s-server-ca@1675144488: notBefore=2023-01-31 05:54:48 +0000 UTC notAfter=2024-02-01 07:11:37 +0000 UTC"
time="2023-02-01T15:11:37+08:00" level=info msg="certificate CN=system:node:id-test-it-kubernetes-10.7.4.4-master02,O=system:nodes signed by CN=k3s-client-ca@1675144488: notBefore=2023-01-31 05:54:48 +0000 UTC notAfter=2024-02-01 07:11:37 +0000 UTC"

Has anyone found a way to run istio in k3s?

Hey folks, I am new to k3s. I was trying to understand how klipper-lb/svc-lb works. I deployed a Single Node cluster with traefik disabled. I don't see any difference between NodePort service v/s klipper lb. I saw the klipper-lb entry script on Github and it seems like it's just opening traffic on Node's IPTables to Cluster IP. Both are opening ports on the Nodes to direct the incoming traffic to the Cluster IP. I am sure there is something more to it which I can't understand. I would love to understand it more in depth. Can any expert please help me in understanding what benefit I am getting on using it in a private cluster? And what extra does it offer in comparison to the NodePort service as in my understanding both will redirect the traffic to Cluster IP and Cluster IP will do the internal Load Balancing among the pods. Thanks!

Hi! I'm running Rancher on-top of three

k3s

nodes,

k3s

is been started using a couple of flags via

curl -sfL <https://get.k3s.io> | sh -s -

. I would like to create a

config.yaml

and add everything there and restart

k3s

with only the

--config

flag. AFAICT i can change the systemd unit-file and

ExecStart

to add

--config

and do a restart. Any caveats making a change like this?

I have released a new version of https://github.com/vitobotta/hetzner-k3s - for those not familiar with it, it's the easiest way to create fully functional k3s clusters in Hetzner Cloud (an awesome but cheap provider with locations in USA and Europe) in just a few minutes