It feels like the actual rancher server is the hold up, any ideas for troubleshooting?

Hi Everyone! We were running a k3s cluster on flatcar on top of a VM, during the upgrade, we seem to have messed up our first master. The other 2 masters are up and running fine it seems. When starting the k3s service, we keep getting this logs. We reset the 1st master and now are trying to add it using 2nd master as primary.. Appreciate if someone can help.. Also, if this is not the right forum, can someone please direct me towards one? Thanks!

Apr 19 16:28:55 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:28:55Z" level=info msg="Waiting for etcd server to become available"
Apr 19 16:28:55 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:28:55Z" level=info msg="Waiting for API server to become available"
Apr 19 16:28:57 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:28:57Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:00 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:00Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:02 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:02Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:05 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:05Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:07 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:07Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:10 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:10Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:10 ng-10-51-52-101.lab k3s[989]: {"level":"warn","ts":"2023-04-19T16:29:10.926Z","caller":"etcdserver/server.go:2065","msg":"failed to publish local member to cluster through raft","local-member-id":"7dff6a3be78a5ee3","local-member-attributes":"{Name:ng-10-51-52-101.lab-accc9f42 ClientURLs:[<https://10.51.52.101:2379>]}","request-path":"/0/members/7dff6a3be78a5ee3/attributes","publish-timeout":"15s","error":"etcdserver: request timed out"}
Apr 19 16:29:11 ng-10-51-52-101.lab k3s[989]: {"level":"warn","ts":"2023-04-19T16:29:11.746Z","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005428c0/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Apr 19 16:29:11 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:11Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Apr 19 16:29:12 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:12Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:15 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:15Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:17 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:17Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:20 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:20Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
Apr 19 16:29:22 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:22Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
Apr 19 16:29:25 ng-10-51-52-101.lab k3s[989]: time="2023-04-19T16:29:25Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"

Hi there! I want to create a deployment from an insecure local registry on a RKE2 cluster. I know the process for RKE 1 adding the insecure_registry to the daemon.json but what's about RKE2 /containerd?

Does anyone have terraform to work with vSphere based off the following? https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2

I want to be able to create a terraform to deploy RKEv2 cluster to vSphere

Thru rancher API

Not sure what broke. I'm on win11 using WSL2 on ubuntu 22.04 and yesterday my environment worked fine. Today I try to build k3d (it's a problem with k3s though) clusters and it hangs on:

DEBU[0002] Waiting for node k3d-dc3-server-0 to get ready (Log: 'k3s is up and running')

When I look at the docker logs for the server node, its crapping out on:

time="2023-04-18T18:11:05Z" level=info msg="Run: k3s kubectl"
time="2023-04-18T18:11:05Z" level=fatal msg="failed to find cpu cgroup (v2)"

This only just started happening after a windows update. Has anyone seen this issue? I tried a WSL2 kernel hack and I got it working:

[wsl2]
kernelCommandLine = cgroup_no_v1=all cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

But now my clusters that were working before keep crashing pods with OOM errors that didn't happen before. So this workround doesn't seem great.

Hello, someone has integrated metallb in a cluster managed by rancher.

Hey o/

Guys, i'm looking for good ressources, tutorials for Rancher v1.6. PM are welcomed. Thanks. 🙏

Hi Team, Kindly help me for below issue I am facing everytime one issue and temporary solution is that I am doing that, restart calico everytime when i am getting error . FailedCreatePodSandBox Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "67a4cd7e5d4f25113eb60273e7aad3da8647e7e6da0bf739f7fa0dfde463af23": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized

Does anyone have information about compatibility of recent versions of RKE2 in configuration with Calico/Typha as CNI and RHEL 9? That is: I see that only https://www.suse.com/suse-rke2/support-matrix/all-supported-versions/rke2-v1-26/ that RHEL 9.0/9.1 appeared in that version (based on https://github.com/rancher/rke2/issues/3860). Does it mean there were any changes in RKE2/Calico to have that support? To be specific, I am curious about compatibility of RKE2 1.24 and RKE2 1.25 with RHEL 9

Hi, Anyone else faced this issue: https://github.com/rancher/rancher/issues/36868? I'm currently having major issues with "Fatal: You must select at least one role" When adding new worker-node to the cluster. Between adding new node and initial setup was upgrade from 2.6.8 -> 2.7.1, faced the issue there, now upgraded to 2.7.2 and same thing. And it's not just 1 cluster, also another cluster has the same issue. Both clusters are identical architecture. Control-plane and ETCD are co-hosted on 3 masters. Any help with this would be appriciated!

How do I fix this. the application in the backend runs off 8080

Was looking at this article

Hi, I have an issue trying to create a VM from a backup made on another Harvester control node. The creation of the volume just hangs at pending with no apparent disk activity. The backup is on a NFS share with the following permissions. The NFS connection tests fine backupstore drwx------ root:user harvester drwx------ root:user

Not able to find much in the way of documentation to troubleshoot the issue as it pertains to Harvester. If there is please help me find it. Thanks!

Cam anyone help me with solving this error? I successfully installed rancher on a k8s cluster with external TLS termination. I could see the dashboard page but the authentication/login is not happening due to this.

xhr.js:178 Mixed Content: The page at 'https://lb-0a6dbbcb4432406688a3339d691ebfb7-1.upcloudlb.com/dashboard/' was loaded over HTTPS, but requested an insecure XMLHttpRequest endpoint 'http://lb-0a6dbbcb4432406688a3339d691ebfb7-1.upcloudlb.com/v3-public/localProviders/local?action=login'. This request has been blocked; the content must be served over HTTPS.

Need some help .. I was doing RKE command line-based K8s installation but k8s installation failed due below reasons: FATA[1980] Failed to get job complete status for job rke-coredns-addon-deploy-job in namespace kube-system RKE : 1.4.4 k8s : 1.25.6 Centos : 7.9 Oracle VM : 7.0

Need help I have installed rancher as single docker container on Ubuntu 18.04 and the container keeps retstarting every 5-6 min.

Its gettinv very frustrating as rancher Ui intermittently giving 502 bad gateway error. Attaching the rancher logs.

Newbie question: Why do some of the releases say with rc and what's the difference between those that don't say "with -rc". Thanks in advance!

follow up from my previous error message ..2nd time I ran k8s installation with rke up ..this time installation successful but I can see error in my pod status --- [rke@kube-master rke-k8s]$ kubectl get pod --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE ingress-nginx nginx-ingress-controller-mg8dx 0/1 Running 12 (4m8s ago) 48m ingress-nginx nginx-ingress-controller-tmp7f 0/1 CrashLoopBackOff 13 (43s ago) 48m ingress-nginx nginx-ingress-controller-z6nmx 0/1 CrashLoopBackOff 11 (2m5s ago) 48m kube-system calico-kube-controllers-5b5d9f577c-v8b8w 0/1 Running 47 (<invalid> ago) 163m kube-system calico-node-2s5h7 1/1 Running 1 (67m ago) 163m kube-system calico-node-njx7d 1/1 Running 1 (67m ago) 163m kube-system calico-node-psclx 1/1 Running 1 (67m ago) 163m kube-system calico-node-w8sgc 1/1 Running 0 163m kube-system coredns-autoscaler-74d474f45c-wmvx6 1/1 Running 1 (67m ago) 159m kube-system coredns-dfb7f8fd4-5ghs7 0/1 Running 1 (67m ago) 159m kube-system metrics-server-c47f7c9bb-l7vhp 0/1 CrashLoopBackOff 15 (20s ago) 49m kube-system rke-coredns-addon-deploy-job-7d5kl 0/1 Completed 0 159m kube-system rke-coredns-addon-deploy-job-vnxg7 0/1 Error 0 163m kube-system rke-ingress-controller-deploy-job-67b7l 0/1 Completed 0 48m kube-system rke-metrics-addon-deploy-job-jtkvs 0/1 Completed 0 49m kube-system rke-network-plugin-deploy-job-nq64n 0/1 Completed 0 164m [rke@kube-master rke-k8s]$

failing while executing this /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl.ventura start --tty=false 0 to start rancher desktop in mac M1

Hi I need to use update certs on my Rancher 2.7.0 to certs signed by private CA, I have copied certs, (.key, cacert.pem, cert.pem)to /etc/rancher/ssl inside my rancher docker container, but it fails to start then with error: [FATAL] failed to setup TLS listener: invalid SSL configuration found, please set cacerts when using self signed certificates or use --no-cacerts when using certificates from a recognized Certificate Authority, do not use both at the same time

Rancher 2.6.5. I'm trying to deploy Neuvector (100.0.3+up2.2.4) via Apps > Charts. I selected NodePort for the Service Configuration options and left everything else as default (I just want to be able to get to it through the Rancher UI). It deploys successfully, however when I try to access it, I get "Authentication Failed." I've seen this issue: https://github.com/rancher/rancher/issues/37434 However I'm not deploying via terraform. Has anyone run into this just using the helm chart within Rancher?

Can we import existing VMware workload cluster to rancher ? We do have an option to create vsphere but while trining to import an existing cluster we are getting Namespace fleet-default does not exist

I currently have a single node docker setup for rancher and am in the process of migrating it to an RKE2 cluster. The only problem I’m seeing so far is that Rancher is currently exposed on an ip - is there a way to deploy rancher on the new cluster via Metallb and the same IP?

I created a Host Based ingress, kubectl create ingress httping --rule=k3s.globe.com/=httpsvc:8080 and it worked fine. How can create a path based ingres with kubectl?