https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
k3s
  • m

    many-journalist-64069

    11/26/2022, 4:18 AM
    Ladies and gentlemen, after the agent node deployment, my k3s prompts such an error. What is the cause
    f
    • 2
    • 24
  • m

    many-journalist-64069

    11/26/2022, 4:18 AM
  • m

    many-journalist-64069

    11/26/2022, 4:20 AM
    server:
    INSTALL_K3S_MIRROR=cn K3S_NODE_NAME=cubmaster01     K3S_KUBECONFIG_OUTPUT=/home/escape/.kube/config     INSTALL_K3S_EXEC="--docker" | sh install.sh
    agent:
    INSTALL_K3S_MIRROR=cn K3S_NODE_NAME=cubnode01     K3S_KUBECONFIG_OUTPUT=/home/escape/.kube/config     K3S_URL=<https://192.168.71.130:6443>     K3S_TOKEN=K100....  sh install.sh
  • m

    many-journalist-64069

    11/26/2022, 4:24 AM
    https://github.com/k3s-io/k3s/issues/6562
  • j

    jolly-waitress-71272

    11/28/2022, 5:27 PM
    If I run this:
    kubectl create secret generic kubeconfig --from-file=/etc/rancher/k3s/k3s.yaml
    Is there a manifest (or literally any file at all) created somewhere I can reference? I'm trying to ansibilize it and I just want to run it once, with a
    creates
    directive. So if there was a manifest, that'd be super easy. I need to do this for some argo shenanigans if you're curious why I would yo dawg my kubeconfig.
    s
    c
    • 3
    • 5
  • b

    best-accountant-61831

    11/28/2022, 8:02 PM
    I have a small 3 node cluster running on NUC's. All 3 are masters with embedded etcd. I recently shut down one of them, as I had enough with two, and could spare on electricity. Running on two of three masters is in theory not ideal, but it works enough for this homelab. Things evolved and I might consider running all on 1, but would like to keep a second offline. Is it possible to convert a master node to a worker node, without having to fully reinstall? The node is running k3os, and I'd like to avoid having to reinstall it, for now.
    s
    n
    • 3
    • 6
  • s

    some-petabyte-4846

    11/29/2022, 4:58 AM

    https://youtu.be/vv_eIJgbFNc▾

  • s

    sticky-summer-13450

    11/29/2022, 3:08 PM
    I'm occasionally finding pods stuck in
    Terminating
    state and I want to know whether it's k3s, k8s, or me. Example: I have cluster with 1
    server
    node and several
    worker
    nodes, and I have workloads spread across the workers. Lets say a worker node dies - maybe it's never going to return.
    $ kubectl get pods --context kube001 --all-namespaces -o=wide |grep Terminating 
    kube-system        traefik-9c6dc6686-jdt9f                            1/1     Terminating   0              24d    10.42.1.4     kube002   <none>           <none>
    active-mq          active-mq-6665f5d8b9-ztwnq                         1/1     Terminating   0              15d    10.42.1.82    kube002   <none>           <none>
    Some of the pods get stuck in the terminating state and don't get replaced on other worker nodes. This means the cluster is no-longer respecting the declarative state. Is this a problem specific to me, a problem specific to k3s, a problem with k8s, or something else?
    c
    • 2
    • 9
  • f

    faint-tomato-17392

    11/30/2022, 7:36 AM
    Hi all, I'm trying the multi-master setup and have a few questions: 1. I've setup m1 (initial) and m2 connected to m1, if m1 goes down and completely got terminated, when spinning up another master like m3, should I use the m2 ip as the server address? 2. Do i need to connect all agent nodes to all master/server nodes? Or connecting it to a single master node is all I have to do? Cheers Thank you
    c
    • 2
    • 7
  • f

    famous-flag-15098

    11/30/2022, 10:32 PM
    Having this weird problem on my 1.23.10 k3s cluster where browsers in my network cannot connect to http://myinternaldns.com insecurely (ie: http not https). I can curl the endpoints without an issue, but with the browsers they will timeout connecting to https://myinternaldns.com which is not what I asked for. Sometimes in-fact it works, and other times not. I have looked into Metallb as the potential issue, but I am not seeing anything there and don't know where to look next. I do have Traefik configured for external ingressroutes only. Any help much appreciated.
    • 1
    • 2
  • c

    creamy-room-58344

    12/02/2022, 9:07 PM
    I'm not able to deploy a k3s cluster via rancher on vSphere I keep getting
    waiting for cluster agent to connect
    Rancher v2.7.0 Machine Provider: VMware vSphere Kubernetes version: v1.24.8+k3s+1 pool1: 3 master+etcd nodes pool2: 4 worker nodes Nodes image: Ubuntu Cloud 22.04 (same as used in RKE1 and RKE2) Conditions shows:
    Ready	False	2.7 hours ago 	[Disconnected] Cluster agent is not connected
    Reconciling	True	2.7 hours ago 	[Reconciling]
    RKECluster	True	2.6 hours ago 	—
    SecretsMigrated	True	2.7 hours ago 	—
    ServiceAccountSecretsMigrated	True	2.7 hours ago 	—
    Stalled	False	2.7 hours ago 	—
    SystemProjectCreated	True	2.7 hours ago 	—
    Updated	Unknown	2.6 hours ago 	[Waiting] configuring bootstrap node(s) k3s-lab01-master-789d65b648-fxft7: waiting for cluster agent to connect
    I'm able to deploy RKE1 and RKE2 clusters What other info can I provide to help solve this issue?
    c
    • 2
    • 3
  • g

    gentle-advantage-38637

    12/03/2022, 2:06 PM
    hey all, I'm having some issues installing k3s, can anyone help?
    l
    c
    • 3
    • 3
  • p

    prehistoric-diamond-4224

    12/04/2022, 3:13 PM
    Hello! I need to migrate a k3s cluster from a set of 3 machines down to a bigger one. I know it's not ideal and that we lose redundancy, but for our workloads and uptime requirements it should be no problem. As of right now we have 1 master+worker and 2 workers, I was wondering if it would be possible to just add the new machine to the cluster as a new Master, migrate all the workloads and volumes there and then simply terminate the old ones. This way everything should be kept as-is, but i'm wondering if having only two master nodes and then removing one would be a problem for the cluster, since ETCD wants an odd number of masters. Is this kind of operation possible? Do you forsee there being any issue in the migration? Thanks
    g
    • 2
    • 8
  • e

    enough-carpet-20915

    12/04/2022, 6:44 PM
    So I'm trying to add entrypoints to the automatically installed traefik in k3s. I'm having issues, however. If I apply this:
    apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
    kind: HelmChartConfig
    metadata:
      name: traefik
      namespace: kube-system
    spec:
      valuesContent: |-
        additionalArguments:
          - "--entrypoints.minecraft.address=:25565/tcp"
        ports:
        - containerPort: 25565
          name: minecraft
          protocol: TCP
    I get this error from the helm-install-traefik-$foo pod:
    Error: UPGRADE FAILED: template: traefik/templates/service.yaml:10:27: executing "traefik/templates/service.yaml" at <$name>: wrong type for value; expected string; got int
    What am I doing wrong here?
    c
    • 2
    • 5
  • b

    bright-london-1095

    12/06/2022, 8:55 AM
    Hello all, We would like to upgrade the cluster version from
    1.22 to 1.24
    . and i have few questions in my mind 1. Is it fine to upgrade from 1.22 to 1.24 ? 2. Which minor version of
    1.24.x
    is a stable one to use it production
    k3s
    cluster ? 3. I believe it is
    traefik-v2.x
    will be shipped along with 1.24.x ? TIA
    c
    • 2
    • 1
  • l

    lively-night-78214

    12/06/2022, 1:51 PM
    i this a good one to install k3s? https://github.com/k3s-io/k3s-ansible
  • l

    lively-night-78214

    12/06/2022, 1:51 PM
    i also use terraform but i assume this step is for a configuration manager
    r
    q
    l
    • 4
    • 7
  • q

    quiet-memory-19288

    12/06/2022, 5:37 PM
    Has anyone noticed k3s shutdown times (on containerd) are super fast if you remove the metrics-server? I tried all kinds of combinations and versions of k3s…. All with same results. Any ideas why?
    c
    • 2
    • 8
  • a

    able-mechanic-45652

    12/07/2022, 6:55 AM
    Hi, I recently re-installed k3s cluster to change the db to embedded etcd. I've applied all configs from our git repo to the new cluster but for some reason when applying our application configs the pod's get stuck.
    c
    • 2
    • 40
  • l

    lively-night-78214

    12/07/2022, 1:51 PM
    I am trying to install an agent and get the error... [ERROR] Only https:// URLs are supported for K3S_URL. When the server/master is installed is SSL auto provisioned?
    c
    • 2
    • 8
  • w

    wide-author-88664

    12/07/2022, 4:10 PM
    👋 K3S folk! Have a question, can one move a K3S cluster (nodes) on one network, to another network?
    c
    • 2
    • 6
  • w

    wide-author-88664

    12/07/2022, 4:11 PM
    Or is it basically easier to destroy cluster (a new one, nothing running on it yet) & reinstall when nodes on new network?
  • l

    lively-night-78214

    12/07/2022, 5:56 PM
    When adding an agent to a k3s server do you need to specify the data store details too? ie I m using mysql so i pass the details as an env variable when setting up the server.
    c
    • 2
    • 2
  • l

    lively-night-78214

    12/08/2022, 5:09 AM
    Looking at the architecture on https://docs.k3s.io/architecture#single-server-setup-with-an-embedded-db to try and understand really where the load balancer sits and operates from. My understanding is that
    servers
    actually run Traefik and this acts as the load balancer. I am planning to use a single server so I would like to understand if i actually need to spin my own load balancer solution, how and if not if indeed the loadbalancer is actually withing the Server Node
    c
    • 2
    • 39
  • l

    lively-night-78214

    12/08/2022, 7:37 AM
    I run this installation
    curl -sfL <https://get.k3s.io> |  INSTALL_K3S_VERSION=v1.25.4+k3s1 sh -s - server --datastore-endpoint=\"<mysql://user:pass@tcp>(host:3306)/my_k3s_test
    STD out shows
    "[INFO]  Using v1.25.4+k3s1 as release",
            "[INFO]  Downloading hash <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt>",
            "[INFO]  Downloading binary <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s>",
            "[INFO]  Verifying binary download",
            "[INFO]  Installing k3s to /usr/local/bin/k3s",
            "[INFO]  Skipping installation of SELinux RPM",
            "[INFO]  Creating /usr/local/bin/kubectl symlink to k3s",
            "[INFO]  Creating /usr/local/bin/crictl symlink to k3s",
            "[INFO]  Creating /usr/local/bin/ctr symlink to k3s",
            "[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh",
            "[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh",
            "[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env",
            "[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service",
            "[INFO]  systemd: Enabling k3s unit",
            "[INFO]  systemd: Starting k3s"
    However in std_err
    "Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.",
            "Job for k3s.service failed because the control process exited with error code.",
            "See \"systemctl status k3s.service\" and \"journalctl -xe\" for details."
    If i go to the server and run
    ubuntu@my-k3s-server:~$ crictl version
    FATA[0000] load config file: stat /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml: no such file or directory
    From the journal
    ubuntu@my-k3s-server:~$ journalctl -xe
    Hint: You are currently not seeing messages from other users and the system.
          Users in groups 'adm', 'systemd-journal' can see all messages.
          Pass -q to turn off this notice.
    No journal files were opened due to insufficient permissions.
    ubuntu@myreviews-k3s-server:~$ sudo journalctl -xe
    -- Subject: Automatic restarting of a unit has been scheduled
    -- Defined-By: systemd
    -- Support: <http://www.ubuntu.com/support>
    -- 
    -- Automatic restarting of the unit k3s.service has been scheduled, as the result for
    -- the configured Restart= setting for the unit.
    Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
    -- Subject: Unit k3s.service has finished shutting down
    -- Defined-By: systemd
    -- Support: <http://www.ubuntu.com/support>
    -- 
    -- Unit k3s.service has finished shutting down.
    Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
    -- Subject: Unit k3s.service has begun start-up
    -- Defined-By: systemd
    -- Support: <http://www.ubuntu.com/support>
    -- 
    -- Unit k3s.service has begun starting up.
    Dec 08 07:24:52 myreviews-k3s-server sh[4543]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
    Dec 08 07:24:52 myreviews-k3s-server sh[4543]: /bin/sh: 1: /usr/bin/systemctl: not found
    Dec 08 07:24:52 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:52Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
    Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
    Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
    Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
    Dec 08 07:24:56 myreviews-k3s-server sudo[4573]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
    Dec 08 07:24:56 myreviews-k3s-server sudo[4573]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
    From system CTL
    systemctl status k3s.service
    ● k3s.service - Lightweight Kubernetes
       Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
       Active: activating (auto-restart) (Result: exit-code) since Thu 2022-12-08 07:24:18 UTC; 3s ago
         Docs: <https://k3s.io>
      Process: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
      Process: 4372 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
      Process: 4370 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
      Process: 4363 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
     Main PID: 4373 (code=exited, status=1/FAILURE)
    What am I doing wrong?
    s
    c
    • 3
    • 44
  • l

    lively-night-78214

    12/08/2022, 9:56 AM
    Is there a value in the configuration that i can provide to ensure there is no certificate mismatch?
    Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
    Unable to connect to the server: x509: certificate is valid for kubernetes, kubernetes.default, kubernetes.default.svc, kubernetes.default.svc.cluster.local, localhost, myapp-k3s-server, not <http://k3s.domain.space|k3s.domain.space>
    c
    • 2
    • 44
  • a

    aloof-dusk-52700

    12/08/2022, 6:17 PM
    hello I have been trying to install k3s on a xilinx board PYNQ Z1 and I keep getting status notready when I run Kubectl get nodes. below is the output. Can someone help me? I am new to kubernetes root@pynq:/home/xilinx# kubectl get nodes NAME STATUS ROLES AGE VERSION pynq NotReady control-plane,master 8d v1.25.4+k3s1 root@pynq:/home/xilinx# kubectl describe node pynq Name: pynq Roles: control-plane,master Labels: beta.kubernetes.io/arch=arm beta.kubernetes.io/instance-type=k3s beta.kubernetes.io/os=linux egress.k3s.io/cluster=true kubernetes.io/arch=arm kubernetes.io/hostname=pynq kubernetes.io/os=linux node-role.kubernetes.io/control-plane=true node-role.kubernetes.io/master=true node.kubernetes.io/instance-type=k3s Annotations: flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"86:e8:b1:26:60:bc"} flannel.alpha.coreos.com/backend-type: vxlan flannel.alpha.coreos.com/kube-subnet-manager: true flannel.alpha.coreos.com/public-ip: 192.168.0.115 k3s.io/hostname: pynq k3s.io/internal-ip: 192.168.0.115 k3s.io/node-args: ["server","--flannel-backend","none"] k3s.io/node-config-hash: 447SKVE725DJE7IL6YFZKSK35XQASDGUESLF6EHAW2PGVRUPL5OQ==== k3s.io/node-env: {"K3S_DATA_DIR":"/var/lib/rancher/k3s/data/bf78b680b175415f467303cca6d37bb8d7eac8bf0b6bd0a65a72cc26d01efbdb","K3S_KUBECONFIG_MODE":"644"} node.alpha.kubernetes.io/ttl: 0 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Wed, 30 Nov 2022 08:23:56 +0000 Taints: node.kubernetes.io/not-ready:NoSchedule Unschedulable: false Lease: HolderIdentity: pynq AcquireTime: <unset> RenewTime: Thu, 08 Dec 2022 18:12:57 +0000 Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- MemoryPressure False Thu, 08 Dec 2022 18:09:34 +0000 Wed, 30 Nov 2022 08:34:42 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Thu, 08 Dec 2022 18:09:34 +0000 Wed, 30 Nov 2022 08:34:42 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Thu, 08 Dec 2022 18:09:34 +0000 Wed, 30 Nov 2022 08:34:42 +0000 KubeletHasSufficientPID kubelet has sufficient PID available Ready False Thu, 08 Dec 2022 18:09:34 +0000 Thu, 01 Dec 2022 06:23:39 +0000 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized Addresses: InternalIP: 192.168.0.115 Hostname: pynq Capacity: cpu: 2 ephemeral-storage: 15089632Ki memory: 505428Ki pods: 110 Allocatable: cpu: 2 ephemeral-storage: 14679193999 memory: 505428Ki pods: 110 System Info: Machine ID: 5a5ef3f8a3034b4c8b00d42b6daec8fb System UUID: 5a5ef3f8a3034b4c8b00d42b6daec8fb Boot ID: 63304d8c-9a1e-4d60-9f29-105755b27543 Kernel Version: 5.15.19-xilinx-v2022.1 OS Image: PynqLinux, based on Ubuntu 22.04 Operating System: linux Architecture: arm Container Runtime Version: containerd://1.6.8-k3s1 Kubelet Version: v1.25.4+k3s1 Kube-Proxy Version: v1.25.4+k3s1 PodCIDR: 10.42.0.0/24 PodCIDRs: 10.42.0.0/24 ProviderID: k3s://pynq Non-terminated Pods: (5 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age --------- ---- ------------ ---------- --------------- ------------- --- kube-system helm-install-traefik-6l4hc 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system helm-install-traefik-crd-lbmbv 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system local-path-provisioner-79f67d76f8-rbtxb 0 (0%) 0 (0%) 0 (0%) 0 (0%) 8d kube-system metrics-server-5c8978b444-9z4dk 100m (5%) 0 (0%) 70Mi (14%) 0 (0%) 8d kube-system coredns-597584b69b-2sd6j 100m (5%) 0 (0%) 70Mi (14%) 170Mi (34%) 8d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 200m (10%) 0 (0%) memory 140Mi (28%) 170Mi (34%) ephemeral-storage 0 (0%) 0 (0%) Events: <none>
    c
    • 2
    • 14
  • c

    creamy-hospital-75658

    12/08/2022, 11:05 PM
    Can anyone explain to me why when using ipv6 --service-cluster-ip-range has to be a /108 or smaller? It neither makes sense to me and especially when we can specify a /16 for ipv4, we should at least be able to specify a /48 😕... I wanted to use one /48 for pods and another /48 for services 😕
    c
    • 2
    • 70
  • q

    quiet-chef-27276

    12/09/2022, 5:22 AM
    Hey all - what is the best way to get kube logs from within a kube pod? (and maybe there is a better way to do this). I want to periodically export logs from various pods as well as system metrics and send them to a location on a filesystem for eventual download (consider this an IoT-like environment where there is sporadic data access)
    m
    • 2
    • 1
  • c

    chilly-telephone-51989

    12/09/2022, 10:36 AM
    im using k3s on amazom. restarted the master node and it started giving 504 gateway timed out. here is the log
    k -n kube-system logs traefik-7cd4fcff68-49cn2 -f
    
    time="2022-12-09T09:42:28Z" level=info msg="Configuration loaded from flags."
    time="2022-12-09T09:42:59Z" level=error msg="Error watching kubernetes events: could not retrieve server version: Get \"<https://10.43.0.1:443/version?timeout=32s>\": dial tcp 10.43.0.1:443: i/o timeout" providerName=kubernetes
    I1209 09:42:59.042242       1 trace.go:205] Trace[1457098512]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.039) (total time: 30002ms):
    Trace[1457098512]: [30.002607632s] [30.002607632s] END
    E1209 09:42:59.042279       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IngressRoute: failed to list *v1alpha1.IngressRoute: Get "<https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/ingressroutes?limit=500&resourceVersion=0>": dial tcp 10.43.0.1:443: i/o timeout
    I1209 09:42:59.042377       1 trace.go:205] Trace[1530918296]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.041) (total time: 30000ms):
    Trace[1530918296]: [30.00042368s] [30.00042368s] END
    E1209 09:42:59.042389       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.MiddlewareTCP: failed to list *v1alpha1.MiddlewareTCP: Get "<https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/middlewaretcps?limit=500&resourceVersion=0>": dial tcp 10.43.0.1:443: i/o timeout
    I1209 09:42:59.042468       1 trace.go:205] Trace[649799410]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.042) (total time: 30000ms):
    Trace[649799410]: [30.000317588s] [30.000317588s] END
    [3:00 PM] please note that requests are not being passed to the gateway or any other pod. here is my ingress file:
    ingress.middleware.yaml
    
    apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
    kind: Middleware
    metadata:
      name: strip-path
      namespace: xplorie
    spec:
      stripPrefix:
        prefixes:
          - /api
    [3:00 PM] ingress.yaml
    
    apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
    kind: Ingress
    metadata:
      name: ingress
      namespace: xplorie
      annotations:
        <http://traefik.ingress.kubernetes.io/router.middlewares|traefik.ingress.kubernetes.io/router.middlewares>: xplorie-strip-path@kubernetescrd
    spec:
      rules:
        - http:
            paths:
              - path: "/api"
                pathType: Prefix
                backend:
                  service:
                    name: gateway
                    port:
                      number: 80
              - path: "/"
                pathType: Prefix
                backend:
                  service:
                    name: portal
                    port:
                      number: 80
    w
    • 2
    • 4
Powered by Linen
Title
c

chilly-telephone-51989

12/09/2022, 10:36 AM
im using k3s on amazom. restarted the master node and it started giving 504 gateway timed out. here is the log
k -n kube-system logs traefik-7cd4fcff68-49cn2 -f

time="2022-12-09T09:42:28Z" level=info msg="Configuration loaded from flags."
time="2022-12-09T09:42:59Z" level=error msg="Error watching kubernetes events: could not retrieve server version: Get \"<https://10.43.0.1:443/version?timeout=32s>\": dial tcp 10.43.0.1:443: i/o timeout" providerName=kubernetes
I1209 09:42:59.042242       1 trace.go:205] Trace[1457098512]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.039) (total time: 30002ms):
Trace[1457098512]: [30.002607632s] [30.002607632s] END
E1209 09:42:59.042279       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.IngressRoute: failed to list *v1alpha1.IngressRoute: Get "<https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/ingressroutes?limit=500&resourceVersion=0>": dial tcp 10.43.0.1:443: i/o timeout
I1209 09:42:59.042377       1 trace.go:205] Trace[1530918296]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.041) (total time: 30000ms):
Trace[1530918296]: [30.00042368s] [30.00042368s] END
E1209 09:42:59.042389       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.MiddlewareTCP: failed to list *v1alpha1.MiddlewareTCP: Get "<https://10.43.0.1:443/apis/traefik.containo.us/v1alpha1/middlewaretcps?limit=500&resourceVersion=0>": dial tcp 10.43.0.1:443: i/o timeout
I1209 09:42:59.042468       1 trace.go:205] Trace[649799410]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.22.1/tools/cache/reflector.go:167 (09-Dec-2022 09:42:29.042) (total time: 30000ms):
Trace[649799410]: [30.000317588s] [30.000317588s] END
[3:00 PM] please note that requests are not being passed to the gateway or any other pod. here is my ingress file:
ingress.middleware.yaml

apiVersion: <http://traefik.containo.us/v1alpha1|traefik.containo.us/v1alpha1>
kind: Middleware
metadata:
  name: strip-path
  namespace: xplorie
spec:
  stripPrefix:
    prefixes:
      - /api
[3:00 PM] ingress.yaml

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  name: ingress
  namespace: xplorie
  annotations:
    <http://traefik.ingress.kubernetes.io/router.middlewares|traefik.ingress.kubernetes.io/router.middlewares>: xplorie-strip-path@kubernetescrd
spec:
  rules:
    - http:
        paths:
          - path: "/api"
            pathType: Prefix
            backend:
              service:
                name: gateway
                port:
                  number: 80
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: portal
                port:
                  number: 80
so far Ithink that dns service isn't responding to the requests. When I run the busy box and try to lookup
nslookup gateway
I get this only after deleting the coredns pod and re-spawn. before that I wasn't able to get anything except timeout errors
nslookup gateway
Server:		10.43.0.10
Address:	10.43.0.10:53

Name:	gateway.xplorie.svc.cluster.local
Address: 10.43.156.140

** server can't find gateway.svc.cluster.local: NXDOMAIN

** server can't find gateway.svc.cluster.local: NXDOMAIN


** server can't find gateway.cluster.local: NXDOMAIN

** server can't find gateway.cluster.local: NXDOMAIN

** server can't find gateway.us-east-2.compute.internal: NXDOMAIN

** server can't find gateway.us-east-2.compute.internal: NXDOMAIN
i had seen this error before and then the reason was that traefik wasn't able to find the gateway.
ks get pods -o wide
NAME                                      READY   STATUS      RESTARTS      AGE    IP            NODE              NOMINATED NODE   READINESS GATES
helm-install-traefik-crd-jh2nn            0/1     Completed   0             87d    10.42.0.3     ip-172-31-46-55   <none>           <none>
helm-install-traefik-rzcq4                0/1     Completed   2             87d    10.42.0.2     ip-172-31-46-55   <none>           <none>
svclb-traefik-12c86e56-9728x              2/2     Running     2 (87d ago)   87d    10.42.1.3     ip-172-31-34-0    <none>           <none>
svclb-traefik-12c86e56-k8w7g              2/2     Running     8 (86d ago)   87d    10.42.2.19    ip-172-31-41-97   <none>           <none>
svclb-traefik-12c86e56-kz9fc              2/2     Running     4 (44h ago)   87d    10.42.0.15    ip-172-31-46-55   <none>           <none>
local-path-provisioner-7b7dc8d6f5-w7f68   1/1     Running     3 (44h ago)   87d    10.42.0.16    ip-172-31-46-55   <none>           <none>
metrics-server-668d979685-9gxfk           1/1     Running     3 (44h ago)   87d    10.42.0.19    ip-172-31-46-55   <none>           <none>
coredns-b96499967-c6827                   1/1     Running     0             30m    10.42.1.124   ip-172-31-34-0    <none>           <none>
traefik-7cd4fcff68-f8b7z                  1/1     Running     0             5m7s   10.42.0.20    ip-172-31-46-55   <none>           <none>
traefik-7cd4fcff68-pftq7                  1/1     Running     0             5m7s   10.42.2.107   ip-172-31-41-97   <none>           <none>
traefik-7cd4fcff68-l6t45                  1/1     Running     0             5m7s   10.42.1.125   ip-172-31-34-0    <none>           <none>
please note that this cluster is k3s, 1 master 2 agents. all pods are running on two agent nodes. This is on Amazon server. registry for docker is running on master 1. the problem happened when the master was rebooted. the code system is ubuntu 20.10 LTS. all we are getting is 504 gateway timed out whether you access from over the net or private IP of master 1
w

worried-judge-39444

02/11/2023, 11:55 AM
Old IPTables versions? https://docs.k3s.io/advanced#raspberry-pi https://github.com/k3s-io/k3s/issues/3117
View count: 67