Ladies and gentlemen, after the agent node deployment, my k3s prompts such an error. What is the cause

server：

INSTALL_K3S_MIRROR=cn K3S_NODE_NAME=cubmaster01     K3S_KUBECONFIG_OUTPUT=/home/escape/.kube/config     INSTALL_K3S_EXEC="--docker" | sh install.sh

agent：

INSTALL_K3S_MIRROR=cn K3S_NODE_NAME=cubnode01     K3S_KUBECONFIG_OUTPUT=/home/escape/.kube/config     K3S_URL=<https://192.168.71.130:6443>     K3S_TOKEN=K100....  sh install.sh

If I run this:

kubectl create secret generic kubeconfig --from-file=/etc/rancher/k3s/k3s.yaml

Is there a manifest (or literally any file at all) created somewhere I can reference? I'm trying to ansibilize it and I just want to run it once, with a

creates

directive. So if there was a manifest, that'd be super easy. I need to do this for some argo shenanigans if you're curious why I would yo dawg my kubeconfig.

I have a small 3 node cluster running on NUC's. All 3 are masters with embedded etcd. I recently shut down one of them, as I had enough with two, and could spare on electricity. Running on two of three masters is in theory not ideal, but it works enough for this homelab. Things evolved and I might consider running all on 1, but would like to keep a second offline. Is it possible to convert a master node to a worker node, without having to fully reinstall? The node is running k3os, and I'd like to avoid having to reinstall it, for now.

I'm occasionally finding pods stuck in

Terminating

state and I want to know whether it's k3s, k8s, or me. Example: I have cluster with 1

server

node and several

worker

nodes, and I have workloads spread across the workers. Lets say a worker node dies - maybe it's never going to return.

$ kubectl get pods --context kube001 --all-namespaces -o=wide |grep Terminating 
kube-system        traefik-9c6dc6686-jdt9f                            1/1     Terminating   0              24d    10.42.1.4     kube002   <none>           <none>
active-mq          active-mq-6665f5d8b9-ztwnq                         1/1     Terminating   0              15d    10.42.1.82    kube002   <none>           <none>

Some of the pods get stuck in the terminating state and don't get replaced on other worker nodes. This means the cluster is no-longer respecting the declarative state. Is this a problem specific to me, a problem specific to k3s, a problem with k8s, or something else?

Hi all, I'm trying the multi-master setup and have a few questions: 1. I've setup m1 (initial) and m2 connected to m1, if m1 goes down and completely got terminated, when spinning up another master like m3, should I use the m2 ip as the server address? 2. Do i need to connect all agent nodes to all master/server nodes? Or connecting it to a single master node is all I have to do? Cheers Thank you

Having this weird problem on my 1.23.10 k3s cluster where browsers in my network cannot connect to http://myinternaldns.com insecurely (ie: http not https). I can curl the endpoints without an issue, but with the browsers they will timeout connecting to https://myinternaldns.com which is not what I asked for. Sometimes in-fact it works, and other times not. I have looked into Metallb as the potential issue, but I am not seeing anything there and don't know where to look next. I do have Traefik configured for external ingressroutes only. Any help much appreciated.

I'm not able to deploy a k3s cluster via rancher on vSphere I keep getting

waiting for cluster agent to connect

Rancher v2.7.0 Machine Provider: VMware vSphere Kubernetes version: v1.24.8+k3s+1 pool1: 3 master+etcd nodes pool2: 4 worker nodes Nodes image: Ubuntu Cloud 22.04 (same as used in RKE1 and RKE2) Conditions shows:

Ready	False	2.7 hours ago 	[Disconnected] Cluster agent is not connected
Reconciling	True	2.7 hours ago 	[Reconciling]
RKECluster	True	2.6 hours ago 	—
SecretsMigrated	True	2.7 hours ago 	—
ServiceAccountSecretsMigrated	True	2.7 hours ago 	—
Stalled	False	2.7 hours ago 	—
SystemProjectCreated	True	2.7 hours ago 	—
Updated	Unknown	2.6 hours ago 	[Waiting] configuring bootstrap node(s) k3s-lab01-master-789d65b648-fxft7: waiting for cluster agent to connect

I'm able to deploy RKE1 and RKE2 clusters What other info can I provide to help solve this issue?

hey all, I'm having some issues installing k3s, can anyone help?

Hello! I need to migrate a k3s cluster from a set of 3 machines down to a bigger one. I know it's not ideal and that we lose redundancy, but for our workloads and uptime requirements it should be no problem. As of right now we have 1 master+worker and 2 workers, I was wondering if it would be possible to just add the new machine to the cluster as a new Master, migrate all the workloads and volumes there and then simply terminate the old ones. This way everything should be kept as-is, but i'm wondering if having only two master nodes and then removing one would be a problem for the cluster, since ETCD wants an odd number of masters. Is this kind of operation possible? Do you forsee there being any issue in the migration? Thanks

So I'm trying to add entrypoints to the automatically installed traefik in k3s. I'm having issues, however. If I apply this:

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    additionalArguments:
      - "--entrypoints.minecraft.address=:25565/tcp"
    ports:
    - containerPort: 25565
      name: minecraft
      protocol: TCP

I get this error from the helm-install-traefik-$foo pod:

Error: UPGRADE FAILED: template: traefik/templates/service.yaml:10:27: executing "traefik/templates/service.yaml" at <$name>: wrong type for value; expected string; got int

What am I doing wrong here?

Hello all, We would like to upgrade the cluster version from

1.22 to 1.24

. and i have few questions in my mind 1. Is it fine to upgrade from 1.22 to 1.24 ? 2. Which minor version of

1.24.x

is a stable one to use it production

k3s

cluster ? 3. I believe it is

traefik-v2.x

will be shipped along with 1.24.x ? TIA

i this a good one to install k3s? https://github.com/k3s-io/k3s-ansible

i also use terraform but i assume this step is for a configuration manager

Has anyone noticed k3s shutdown times (on containerd) are super fast if you remove the metrics-server? I tried all kinds of combinations and versions of k3s…. All with same results. Any ideas why?

Hi, I recently re-installed k3s cluster to change the db to embedded etcd. I've applied all configs from our git repo to the new cluster but for some reason when applying our application configs the pod's get stuck.

I am trying to install an agent and get the error... [ERROR] Only https:// URLs are supported for K3S_URL. When the server/master is installed is SSL auto provisioned?

👋 K3S folk! Have a question, can one move a K3S cluster (nodes) on one network, to another network?

Or is it basically easier to destroy cluster (a new one, nothing running on it yet) & reinstall when nodes on new network?

When adding an agent to a k3s server do you need to specify the data store details too? ie I m using mysql so i pass the details as an env variable when setting up the server.

Looking at the architecture on https://docs.k3s.io/architecture#single-server-setup-with-an-embedded-db to try and understand really where the load balancer sits and operates from. My understanding is that

servers

actually run Traefik and this acts as the load balancer. I am planning to use a single server so I would like to understand if i actually need to spin my own load balancer solution, how and if not if indeed the loadbalancer is actually withing the Server Node

I run this installation

curl -sfL <https://get.k3s.io> |  INSTALL_K3S_VERSION=v1.25.4+k3s1 sh -s - server --datastore-endpoint=\"<mysql://user:pass@tcp>(host:3306)/my_k3s_test

STD out shows

"[INFO]  Using v1.25.4+k3s1 as release",
        "[INFO]  Downloading hash <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt>",
        "[INFO]  Downloading binary <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s>",
        "[INFO]  Verifying binary download",
        "[INFO]  Installing k3s to /usr/local/bin/k3s",
        "[INFO]  Skipping installation of SELinux RPM",
        "[INFO]  Creating /usr/local/bin/kubectl symlink to k3s",
        "[INFO]  Creating /usr/local/bin/crictl symlink to k3s",
        "[INFO]  Creating /usr/local/bin/ctr symlink to k3s",
        "[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh",
        "[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh",
        "[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env",
        "[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service",
        "[INFO]  systemd: Enabling k3s unit",
        "[INFO]  systemd: Starting k3s"

However in std_err

"Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.",
        "Job for k3s.service failed because the control process exited with error code.",
        "See \"systemctl status k3s.service\" and \"journalctl -xe\" for details."

If i go to the server and run

ubuntu@my-k3s-server:~$ crictl version
FATA[0000] load config file: stat /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml: no such file or directory

From the journal

ubuntu@my-k3s-server:~$ journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.
ubuntu@myreviews-k3s-server:~$ sudo journalctl -xe
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
-- 
-- Automatic restarting of the unit k3s.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
-- Subject: Unit k3s.service has finished shutting down
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
-- 
-- Unit k3s.service has finished shutting down.
Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
-- Subject: Unit k3s.service has begun start-up
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
-- 
-- Unit k3s.service has begun starting up.
Dec 08 07:24:52 myreviews-k3s-server sh[4543]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 08 07:24:52 myreviews-k3s-server sh[4543]: /bin/sh: 1: /usr/bin/systemctl: not found
Dec 08 07:24:52 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:52Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
Dec 08 07:24:56 myreviews-k3s-server sudo[4573]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 08 07:24:56 myreviews-k3s-server sudo[4573]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)

From system CTL

systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2022-12-08 07:24:18 UTC; 3s ago
     Docs: <https://k3s.io>
  Process: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
  Process: 4372 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 4370 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 4363 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
 Main PID: 4373 (code=exited, status=1/FAILURE)

What am I doing wrong?

Is there a value in the configuration that i can provide to ensure there is no certificate mismatch?

Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, localhost, myapp-k3s-server, not <http://k3s.domain.space|k3s.domain.space>

hello I have been trying to install k3s on a xilinx board PYNQ Z1 and I keep getting status notready when I run Kubectl get nodes. below is the output. Can someone help me? I am new to kubernetes root@pynq:/home/xilinx# kubectl get nodes NAME STATUS ROLES AGE VERSION pynq NotReady control-plane,master 8d v1.25.4+k3s1 root@pynq:/home/xilinx# kubectl describe node pynq Name: pynq Roles: control-plane,master Labels: beta.kubernetes.io/arch=arm beta.kubernetes.io/instance-type=k3s beta.kubernetes.io/os=linux egress.k3s.io/cluster=true kubernetes.io/arch=arm kubernetes.io/hostname=pynq kubernetes.io/os=linux node-role.kubernetes.io/control-plane=true node-role.kubernetes.io/master=true node.kubernetes.io/instance-type=k3s Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"86e8b12660:bc"} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 192.168.0.115 k3s.io/hostname: pynq k3s.io/internal-ip: 192.168.0.115 k3s.io/node-args: ["server","--flannel-backend","none"] k3s.io/node-config-hash: 447SKVE725DJE7IL6YFZKSK35XQASDGUESLF6EHAW2PGVRUPL5OQ==== k3s.io/node-env: {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/bf78b680b175415f467303cca6d37bb8d7eac8bf0b6bd0a65a72cc26d01efbdb","K3S_KUBECONFIG_MODE":"644"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 30 Nov 2022 082356 +0000 Taints: node.kubernetes.io/not-ready:NoSchedule Unschedulable: false Lease: HolderIdentity: pynq AcquireTime: <unset> RenewTime: Thu, 08 Dec 2022 181257 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 08 Dec 2022 180934 +0000 Wed, 30 Nov 2022 083442 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 08 Dec 2022 180934 +0000 Wed, 30 Nov 2022 083442 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 08 Dec 2022 180934 +0000 Wed, 30 Nov 2022 083442 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Thu, 08 Dec 2022 180934 +0000 Thu, 01 Dec 2022 062339 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized Addresses: InternalIP: 192.168.0.115 Hostname: pynq Capacity: cpu: 2 ephemeral-storage: 15089632Ki memory: 505428Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 14679193999 memory: 505428Ki pods: 110 System Info: Machine ID: 5a5ef3f8a3034b4c8b00d42b6daec8fb System UUID: 5a5ef3f8a3034b4c8b00d42b6daec8fb Boot ID: 63304d8c-9a1e-4d60-9f29-105755b27543 Kernel Version: 5.15.19-xilinx-v2022.1 OS Image: PynqLinux, based on Ubuntu 22.04 Operating System: linux Architecture: arm Container Runtime Version: containerd://1.6.8-k3s1 Kubelet Version: v1.25.4+k3s1 Kube-Proxy Version: v1.25.4+k3s1 PodCIDR: 10.42.0.0/24 PodCIDRs: 10.42.0.0/24 ProviderID: k3s://pynq Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system helm-install-traefik-6l4hc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system helm-install-traefik-crd-lbmbv 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system local-path-provisioner-79f67d76f8-rbtxb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system metrics-server-5c8978b444-9z4dk 100m (5%) 0 (0%) 70Mi (14%) 0 (0%) 8d kube-system coredns-597584b69b-2sd6j 100m (5%) 0 (0%) 70Mi (14%) 170Mi (34%) 8d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 200m (10%) 0 (0%) memory 140Mi (28%) 170Mi (34%) ephemeral-storage 0 (0%) 0 (0%) Events: <none>

Can anyone explain to me why when using ipv6 --service-cluster-ip-range has to be a /108 or smaller? It neither makes sense to me and especially when we can specify a /16 for ipv4, we should at least be able to specify a /48 😕... I wanted to use one /48 for pods and another /48 for services 😕

Hey all - what is the best way to get kube logs from within a kube pod? (and maybe there is a better way to do this). I want to periodically export logs from various pods as well as system metrics and send them to a location on a filesystem for eventual download (consider this an IoT-like environment where there is sporadic data access)

im using k3s on amazom. restarted the master node and it started giving 504 gateway timed out. here is the log

k -n kube-system logs traefik-7cd4fcff68-49cn2 -f

time="2022-12-09T09:42:28Z" level=info msg="Configuration loaded from flags."
time="2022-12-09T09:42:59Z" level=error msg="Error watching kubernetes events: could not retrieve server version: Get \"<https://10.43.0.1:443/version?timeout=32s>\": dial tcp 10.43.0.1:443: i/o timeout" providerName=kubernetes
I1209 09:42:59.042242       1 trace.go:205] Trace[1457098512]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.039) (total time: 30002ms):
Trace[1457098512]: [30.002607632s] [30.002607632s] END
E1209 09:42:59.042279       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IngressRoute: failed to list *v1alpha1.IngressRoute: Get "<https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/ingressroutes?limit=500&resourceVersion=0>": dial tcp 10.43.0.1:443: i/o timeout
I1209 09:42:59.042377       1 trace.go:205] Trace[1530918296]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.041) (total time: 30000ms):
Trace[1530918296]: [30.00042368s] [30.00042368s] END
E1209 09:42:59.042389       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.MiddlewareTCP: failed to list *v1alpha1.MiddlewareTCP: Get "<https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/middlewaretcps?limit=500&resourceVersion=0>": dial tcp 10.43.0.1:443: i/o timeout
I1209 09:42:59.042468       1 trace.go:205] Trace[649799410]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.042) (total time: 30000ms):
Trace[649799410]: [30.000317588s] [30.000317588s] END

[3:00 PM] please note that requests are not being passed to the gateway or any other pod. here is my ingress file:

ingress.middleware.yaml

apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: Middleware
metadata:
  name: strip-path
  namespace: xplorie
spec:
  stripPrefix:
    prefixes:
      - /api
[3:00 PM] ingress.yaml

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: ingress
  namespace: xplorie
  annotations:
    <http://traefik.ingress.kubernetes.io/router.middlewares|traefik.ingress.kubernetes.io/router.middlewares>: xplorie-strip-path@kubernetescrd
spec:
  rules:
    - http:
        paths:
          - path: "/api"
            pathType: Prefix
            backend:
              service:
                name: gateway
                port:
                  number: 80
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: portal
                port:
                  number: 80