https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
general
  • b

    bored-planet-1228

    01/09/2023, 2:56 AM
    I’ve just installed rancher-desktop for the first time on macos, and on the command line, I don’t seem to be able to list images in the registry - I can’t even pull a specific image from the docker registry. This is the error message I get:
    > nerdctl pull alpine
    INFO[0000] trying next host                              error="failed to do request: Head \"<https://registry-1.docker.io/v2/library/alpine/manifests/latest>\": proxyconnect tcp: dial tcp 127.0.0.1:3128: connect: connection refused" host=<http://registry-1.docker.io|registry-1.docker.io>
    ERRO[0000] server "<http://docker.io|docker.io>" does not seem to support HTTPS  error="failed to resolve reference \"<http://docker.io/library/alpine:latest\|docker.io/library/alpine:latest\>": failed to do request: Head \"<https://registry-1.docker.io/v2/library/alpine/manifests/latest>\": proxyconnect tcp: dial tcp 127.0.0.1:3128: connect: connection refused"
    INFO[0000] Hint: you may want to try --insecure-registry to allow plain HTTP (if you are in a trusted network)
    FATA[0000] failed to resolve reference "<http://docker.io/library/alpine:latest|docker.io/library/alpine:latest>": failed to do request: Head "<https://registry-1.docker.io/v2/library/alpine/manifests/latest>": proxyconnect tcp: dial tcp 127.0.0.1:3128: connect: connection refused
    Error: exit status 1
  • b

    bored-planet-1228

    01/09/2023, 2:57 AM
    (If there’s a more appropriate channel, please let me know! 😬 )
  • w

    witty-tiger-89786

    01/09/2023, 8:16 AM
    Hey there ! ICYMI - the Longhorn team release v1.4 just before the new year. If you want a quick run through of what some of the enhancements are - check out this blog https://www.suse.com/c/rancher_blog/longhorn-1-4-starting-a-new-year-with-a-new-release/ Posted in #longhorn-storage
    👍 1
  • a

    adorable-exabyte-35533

    01/09/2023, 10:02 AM
    hello folks I am new to Rancher and I want to build my small data center using rancher but I also want to share my gpu to VM so it is possible using rancher os thank you
    b
    • 2
    • 2
  • b

    billowy-apple-60989

    01/09/2023, 3:35 PM
    Maybe i’m missing something obvious, but how can you tell which version of Traefik will be deployed by it’s helm-chart that Rancher/k3s uses? https://github.com/traefik/traefik-helm-chart For ex i started up an k3s cluster on version
    1.21.7
    which uses chart version
    9.18.201
    by default which in turn deployed traefik
    2.4.8
    c
    • 2
    • 4
  • b

    bland-area-30120

    01/09/2023, 3:35 PM
    hello, I'm having issues with the single rke cluster we run, everything else is eks and working fine but the rke cluster is showing the infamous "Cluster agent is not connected". we're on version 2.6.9 and I noticed the rancher-agent docker running on the master nodes is version 2.6.7. Is it possible that that is the issue? can I update the agent manually or is that not advised?
    • 1
    • 1
  • b

    bumpy-printer-21267

    01/09/2023, 4:20 PM
    Is there a way to use the base services (<<: *base) in rancher? For instance: how would I deploy this https://raw.githubusercontent.com/chatwoot/chatwoot/develop/docker-compose.production.yaml
    a
    • 2
    • 1
  • p

    proud-shampoo-40692

    01/09/2023, 5:00 PM
    Hi I have installed Rancher Desktop for the first time, and it is failing to start Kubernetes, and Diagnostics give:
    "The application cannot reach the general internet for updated kubernetes versions and other components, but can still operate."
    My system is Windows 11 Pro.
    c
    • 2
    • 1
  • w

    wonderful-judge-81095

    01/09/2023, 9:27 PM
    Hello, I had to restore from a snapshot last week. Currently my cluster has an error in the rancher gui This cluster is currently Updating. EtcdBackup.management.cattle.io "c-psc9t/c-psc9t-rl-d6dfn" not found. I dont know how to clear this up as the cluster is now not allowing me to add new nodes for etc/control plane roles. I'm new to rancher
    • 1
    • 1
  • w

    wonderful-judge-81095

    01/09/2023, 9:47 PM
    i was able to find a copy on the other server in the control plane, should I just move it to /opt/rke/etcd-snapshots/ directory? Can I be assured this older snapshot wont be used without manual intervention?
  • m

    miniature-midnight-96801

    01/10/2023, 1:17 AM
    Hi all, is there a path to getting Rancher working on a containerd/kops cluster?
    c
    • 2
    • 9
  • b

    blue-tomato-62407

    01/10/2023, 2:59 AM
    Anyone run into an issue with Rancher (Docker install of v2.5.16) getting
    dial tcp 127.0.0.1:6443: connect: connection refused
    on Ubuntu 22.04.1? Just upgraded a server from 20.04 LTS and can't seem to find much to go on so far.
    c
    • 2
    • 6
  • k

    kind-vase-40458

    01/10/2023, 5:45 AM
    • Problem: I'm seeing a well-known problem of crash-looping core-dns, and i'm forwarding to
    /etc/resolv.conf
    which indeeds contains
    nameserver 127.0.0.1
    . If i manually removed the "loop" from config map, the core-dns pod would start fine, but after config map changes, it would crash soon after.. • Why can't i remove the localhost loopback? The localhost is actually intended, since I want to forward to localhost where my upstream DNS process is running on the machine node. • Question: Is there a way to distinguish between pod's localhost from node's localhost? • What i tried: adding
    127.0.0.2 localhost
    in it's /etc/hosts equivalent, and then i'd forward resolution to
    /etc/resolv.conf
    with
    nameserver 127.0.0.2
    seems like there's no DNS resolution happening
  • a

    adorable-exabyte-35533

    01/10/2023, 6:11 AM
    I have one query before installing the harvester ( with rancher ) if my harvester os will be corrupted by any scenario so how can I recover the os also my VMS will damage if os will pollute and how can I recover all my VMS drive if the rancher or harvester is got corrupted and are there any features wich can take auto backup of VM drive in OS thank you
    w
    • 2
    • 1
  • b

    billions-lifeguard-63461

    01/10/2023, 1:09 PM
    Hi! I have problem with rancher logging system. I use loki as a logging service + rancher logging by banzai cloud. My logs are up to 60mins behind the newest one and i can't figure out why. Have someone other similiar problem?
  • r

    red-waitress-37932

    01/10/2023, 2:56 PM
    I dont suppose it is possible to restore a rancher snapshot after accidentally shredding your cluster to bits?
    $ kubectl apply --all --namespace mynamespace --prune -f resources.yml
    secret/config created
    W0110 15:15:02.543661   25197 prune.go:71] Deprecated: kubectl apply will no longer prune non-namespaced resources by default when used with the --namespace flag in a future release. To preserve the current behaviour, list the resources you want to target explicitly in the --prune-allowlist flag.
    namespace/cattle-system pruned
    namespace/ingress-nginx pruned
    namespace/kubernetes-dashboard pruned
  • c

    careful-mouse-48712

    01/10/2023, 5:20 PM
    Hey, anyone found a workaround to the non-working charts/apps installation with 2.7.0 (except installing by cli)?
  • m

    miniature-notebook-6405

    01/10/2023, 5:40 PM
    Hello, I am seeking insight regarding the Rancher agent that is run on the remote cluster. We would like to run a cluster on an isolated network (Azure) and communicate to Rancher server over a private endpoint on one IP address. Is this not possible with the Rancher import agent? I had hoped only port 6443, and all ports for that matter, on a single IP would be enough for the agent to talk to the server, but it is not importing. Does Rancher server require unfettered access to the network of the remote cluster, such as peering with that network? I would have hoped we would be able to leverage the agent for isolated networks, but it appears to have more access requirements than I had thought. Thank you!
    • 1
    • 8
  • o

    orange-barista-66154

    01/10/2023, 5:49 PM
    Has anyone found out how to update kubelet-extra-args when using EKS clusters? I'm trying to override the max-pods setting that is passed in but user data passed by rancher is prepended to the file and therefore unused. The updated launch template version also appaers to be ignored so user_data passed to rancher never makes it to new nodes.
  • s

    silly-airline-38945

    01/10/2023, 8:03 PM
    Hi, testing a disaster recovery scenario with rancher 2.7.0 (on RKE v1.23.14) and a rke2 downstream cluster (v1.23.14+rke2r1 rancher provisioned). Deleting and rinsing rancher cluster nodes, creating a new cluster and following the "migrating rancher to a new cluster" docs. All seems OK, but the rancher-system-agent running on the RKE2 cluster is complaining:
    rancher-system-agent[7696]: time="2023-01-10T19:32:18Z" level=error msg="error syncing 'fleet-default/custom-4a5e8a9fa273-machine-plan': handler secret-watch: Unauthorized, requeuing"
    The fleet agent running on the RKE2 cluster is reporting:
    level=error msg="failed to report cluster node status: Unauthorized"
    Is there a way of fixing this issue?
    • 1
    • 1
  • b

    bland-painting-61617

    01/10/2023, 8:57 PM
    Is anyone from the Rancher team able to help on this one (asking here because it's a bit quiet in #vsphere? https://rancher-users.slack.com/archives/CBEB5A0H1/p1673358852381349?thread_ts=1669323888.250929&amp;cid=CBEB5A0H1
  • g

    gorgeous-cat-50570

    01/11/2023, 10:07 AM
    Hello, recently I encountered a problem in rancher v2.7.0. I was using the GUI to delete a project but in resulted in this error:
    Waiting on project-precan-alert-controller_c-vfm92
    Inspecting container logs :
    [ERROR] error syncing 'c-vfm92/creator-cluster-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
    [ERROR] error syncing 'c-vfm92/u-mxcjeyd7a6-admin': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
    [ERROR] error syncing 'c-vfm92/c-vfm92-fleet-default-owner': handler mgmt-auth-crtb-controller: couldn't create role cluster-owner: <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "cluster-owner" is forbidden: unable to create new content in namespace p-jwcwp because it is being terminated, requeuing
    l
    • 2
    • 84
  • m

    melodic-terabyte-2284

    01/11/2023, 11:55 AM
    I am using Rancher Desktop 1.7.0
  • c

    crooked-cat-21365

    01/11/2023, 12:28 PM
    Trying to setup a new cluster using RKE2 on-premises the control plane doesn't start. The system log shows
    Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
    Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> sh[2138]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
    Jan 11 13:02:44 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> sh[2139]: /bin/sh: 1: /usr/bin/systemctl: not found
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=warning msg="not running in CIS mode"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Starting rke2 v1.24.8+rke2r1 (b061438ce1f04518d980aac842b62f92095d793d)"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Managed etcd cluster initializing"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Starting etcd for new cluster"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Tunnel server egress proxy mode: agent"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,rke2 --authorization-mode=Node,RBAC --bin>
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind>
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/se>
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/>
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Server node token is available at /var/lib/rancher/rke2/server/token"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="To join server node to cluster: rke2 server -s <https://172.19.96.128:9345> -t ${SERVER_NODE_TOKEN}"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Agent node token is available at /var/lib/rancher/rke2/server/agent-token"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="To join agent node to cluster: rke2 agent -s <https://172.19.96.128:9345> -t ${AGENT_NODE_TOKEN}"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /run/k3s/containerd/containerd.sock: connect: no such file or>
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Run: rke2 kubectl"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="certificate CN=<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> signed by CN=rke2-server-ca@1673437584: notBefore=2023-01-11 11:46:24 +0000 UTC notAfter=2024-01-11 12:02:46 +0000 UTC"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="certificate CN=system:node:<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de>,O=system:nodes signed by CN=rke2-client-ca@1673437584: notBefore=2023-01-11 11:46:24 +0000 UTC notAfter=2024-01-11 12:02:46 +0000 UTC"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Module overlay was already loaded"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Module br_netfilter was already loaded"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 3670016"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Runtime image <http://index.docker.io/rancher/rke2-runtime:v1.24.8-rke2r1|index.docker.io/rancher/rke2-runtime:v1.24.8-rke2r1> bin and charts directories already exist; skipping extract"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Using private registry config file at /etc/rancher/rke2/registries.yaml"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"
    Jan 11 13:02:46 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:46+01:00" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Containerd is now running"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 12.972685ms"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 1.097175ms"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 946.039<C2><B5>s"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 1.136069ms"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 1.041789ms"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/li>
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Connecting to proxy" url="<wss://127.0.0.1:9345/v1-rke2/connect>"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Handling backend connection request [<http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de>]"
    Jan 11 13:02:47 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:47+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:02:48 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2281]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
    :
    :
  • c

    crooked-cat-21365

    01/11/2023, 12:29 PM
    Jan 11 13:02:48 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2281]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.
    Jan 11 13:02:51 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:51+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:02:52 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:52+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:02:56 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:56+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:02:58 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:02:58+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:01 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:01+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:03 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:03+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:06 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:06+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:06 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:06+01:00" level=info msg="Container for etcd not found (no matching container found), retrying"
    Jan 11 13:03:08 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:08+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:11 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:11+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:13 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:13+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: {"level":"warn","ts":"2023-01-11T13:03:16.760+0100","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005ff340/127.0.0.1:2379>","attempt":0,"error":">
    Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
    Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Waiting for etcd server to become available"
    Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:16 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:16+01:00" level=info msg="Waiting for API server to become available"
    Jan 11 13:03:18 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: {"level":"warn","ts":"2023-01-11T13:03:18.136+0100","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005ff340/127.0.0.1:2379>","attempt":0,"error":">
    Jan 11 13:03:18 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:18+01:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
    Jan 11 13:03:18 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:18+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:21 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:21+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:23 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:23+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:26 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:26+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:26 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:26+01:00" level=info msg="Container for etcd not found (no matching container found), retrying"
    Jan 11 13:03:28 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:28+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:31 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:31+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Jan 11 13:03:33 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: {"level":"warn","ts":"2023-01-11T13:03:33.136+0100","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005ff340/127.0.0.1:2379>","attempt":0,"error":">
    Jan 11 13:03:33 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:33+01:00" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
    Jan 11 13:03:33 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:33+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
    Jan 11 13:03:36 <http://srvl034a.ac.aixigo.de|srvl034a.ac.aixigo.de> rke2[2193]: time="2023-01-11T13:03:36+01:00" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
    Any helpful hint is highly appreciated.
    c
    • 2
    • 1
  • a

    adventurous-thailand-75978

    01/11/2023, 3:23 PM
    Hello All! Is there 508 documentation for Rancher Desktop? Or any guides on how to do so? Thanks!
    w
    • 2
    • 1
  • s

    stocky-fall-82040

    01/11/2023, 3:50 PM
    I'm running Rancher 2.5.16 and would like to upgrade to 2.6.x or 2.7.x. It there a particular minor version I should upgrade to?
    c
    m
    • 3
    • 10
  • b

    big-jordan-45387

    01/11/2023, 4:30 PM
    hi, I'm new to ranches, someone installed a rancher deployment and I am trying to deploy the rancher-agent in one node, the end goal is to have an rke2 cluster running. I created a cluster through the rancher web interface and ran the command provided to configure the node but the cluster is stuck in "updating" state, also I can see the following in the rancher-system-agent logs
    level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: no such file or directory"
    My question is, what could prevent this
    kube-scheduler.crt
    from being in the node?
    c
    • 2
    • 75
  • c

    creamy-pencil-82913

    01/11/2023, 4:33 PM
    --> #rancher-desktop
  • g

    gorgeous-cat-50570

    01/11/2023, 5:06 PM
    I have rancher v2.7.0 and I keep getting these errors from container logs
    [ERROR] failed on subscribe prometheus: NotFound 404: the server could not find the requested resource (get <http://prometheuses.meta.k8s.io|prometheuses.meta.k8s.io>)
    2023/01/11 17:04:39 [ERROR] failed on subscribe serviceMonitor: NotFound 404: the server could not find the requested resource (get <http://servicemonitors.meta.k8s.io|servicemonitors.meta.k8s.io>)
    2023/01/11 17:04:39 [ERROR] failed on subscribe prometheusRule: NotFound 404: the server could not find the requested resource (get <http://prometheusrules.meta.k8s.io|prometheusrules.meta.k8s.io>)
    2023/01/11 17:04:39 [ERROR] failed on subscribe alertmanager: NotFound 404: the server could not find the requested resource (get <http://alertmanagers.meta.k8s.io|alertmanagers.meta.k8s.io>)
    Sometimes I make api request to rancher but I get
    status code was 500 and not [201]: HTTP Error 500: Internal Server Error
    Any help?
Powered by Linen
Title
g

gorgeous-cat-50570

01/11/2023, 5:06 PM
I have rancher v2.7.0 and I keep getting these errors from container logs
[ERROR] failed on subscribe prometheus: NotFound 404: the server could not find the requested resource (get <http://prometheuses.meta.k8s.io|prometheuses.meta.k8s.io>)
2023/01/11 17:04:39 [ERROR] failed on subscribe serviceMonitor: NotFound 404: the server could not find the requested resource (get <http://servicemonitors.meta.k8s.io|servicemonitors.meta.k8s.io>)
2023/01/11 17:04:39 [ERROR] failed on subscribe prometheusRule: NotFound 404: the server could not find the requested resource (get <http://prometheusrules.meta.k8s.io|prometheusrules.meta.k8s.io>)
2023/01/11 17:04:39 [ERROR] failed on subscribe alertmanager: NotFound 404: the server could not find the requested resource (get <http://alertmanagers.meta.k8s.io|alertmanagers.meta.k8s.io>)
Sometimes I make api request to rancher but I get
status code was 500 and not [201]: HTTP Error 500: Internal Server Error
Any help?
View count: 12