Hi! We are running

rke version v1.3.12

, and while attempting to monitor with prometheus+grafana k8s services (without success) noticed they didn’t get and endpoint IP:

kube-system      kube-prometheus-coredns                   ClusterIP      None            <none>            9153/TCP                                                       127m
kube-system      kube-prometheus-kube-controller-manager   ClusterIP      None            <none>            10257/TCP                                                      127m
kube-system      kube-prometheus-kube-etcd                 ClusterIP      None            <none>            2381/TCP                                                       127m
kube-system      kube-prometheus-kube-proxy                ClusterIP      None            <none>            10249/TCP                                                      127m
kube-system      kube-prometheus-kube-scheduler            ClusterIP      None            <none>            10259/TCP                                                      127m
kube-system      kube-prometheus-kubelet                   ClusterIP      None            <none>            10250/TCP,10255/TCP,4194/TCP                                   127m

Because of that, we are unable to monitor those metrics; this is an example configuration for one of our clusters:

---
nodes:
- address: node1.localdomain
  hostname_override: node01
  user: rke
  role:
  - controlplane
  - worker
  - etcd
  labels:
    role: storage-node
- address: node02.localdomain
  hostname_override: node02
  user: rke
  role:
  - controlplane
  - worker
  - etcd
  labels:
    role: storage-node
- address: node03.localdomain
  hostname_override: node03
  user: rke
  role:
  - controlplane
  - worker
  - etcd
  labels:
    role: storage-node
- address: node04.localdomain
  hostname_override: node04
  user: rke
  role:
  - worker
  labels:
    role: storage-node
- address: node05.localdomain
  hostname_override: node05
  user: rke
  role:
  - worker
  labels:
    role: storage-node
services:
  etcd:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    external_urls: []
    ca_cert: ""
    cert: ""
    key: ""
    path: ""
    uid: 0
    gid: 0
    snapshot: null
    retention: ""
    creation: ""
    backup_config: null
  kube-api:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    service_cluster_ip_range: 10.43.0.0/16
    service_node_port_range: ""
    pod_security_policy: false
    always_pull_images: false
    secrets_encryption_config: null
    audit_log: null
    admission_configuration: null
    event_rate_limit: null
  kube-controller:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_cidr: 10.42.0.0/16
    service_cluster_ip_range: 10.43.0.0/16
  scheduler:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
  kubelet:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
    cluster_domain: cluster.local
    infra_container_image: ""
    cluster_dns_server: 10.43.0.10
    fail_swap_on: false
    generate_serving_certificate: false
  kubeproxy:
    image: ""
    extra_args: {}
    extra_binds: []
    extra_env: []
network:
  plugin: canal
  options: {}
  mtu: 0
  node_selector: {}
authentication:
  strategy: x509
  sans: []
  webhook: null
addons: ""
addons_include: []
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
  mode: rbac
  options: {}
ignore_docker_version: false
kubernetes_version: "v1.23.7-rancher1-1"
..........

Is there anything I have to do to enable those metrics?

Are there any examples of cluster drivers using CAPI? Wasn't turning up anything on my own, and wondering if anyone has implemented this before using rancher cluster drivers. The existing cluster driver example looks like it predates CAPI

Owing to not being a source code guru, would anyone know why using the kubectl shell in the UI requires get management.cattle.io/clusters

Hi everybody, We have deployed Rancher on Kubernetes, and we want to enable the Monitoring feature. However, Prometheus cannot be deployed as its PVCs' names are longer than 63 characters which is not supported by the Storage Class (OpenEBS Jiva). Is there any way to specify the PVC name prior to the deployment as a workaround? Thanks

Hi, how can I set selfgned CA certifcates for agent authentication? https://forums.rancher.com/t/how-to-setup-selfsigned-ca-certificates-for-rancher-agent/39616

hi all , I am looking at https://docs.ranchermanager.rancher.io/v2.5/how-to-guides/new-user-guides/kubernetes-cluster-setup/k3s-for-rancher is there a way to terraform this ? I have found this ansible module ... https://github.com/k3s-io/k3s-ansible

any suggestion is highly appreciated

Hi Everyone, Problem: Rancher is "empty" after fresh install What did i do? I've installed Rancher 2.6.9. on a K3S cluster (3 SUSE Micro VM's hypervised by Harvester) on on-prem hardware (Dell PowerEdge servers). Everything seems to work fine (except the SSL certificate, but i'll fix that later). But when i enter Rancher, there is no local cluster, the list is empty. And when i want to import Harvester (virtualization management -> harvester cluster create), it says : "namespaces "fleet-default" not found". It's a fresh install and nothing else (except certmanager) is running in the cluster. What do i expect? The local cluster to appear in Rancher / able to import my Harvester hypervisor cluster. I would be nice if someone can point me in the right direction. Thanks in advance!

Hello! Its been a while... does anyone know what Rancher Government Solutions is? https://ranchergovernment.com/ Looking at products page there and click on the products it takes me to various Rancher products pages...

Thanks for this, I had the same issue and also was solved removing the last slash from the URL. If you are working with version 2.6.X also be sure that if you are using private CA you have the correct secret in tls-ca

how can I create a k8s clusters with rancher 2.6.x and set the networking plugin to none so I can install my own CNI? if I look at the cluster spec I see no option to do this, https://pkg.go.dev/github.com/rancher/rancher/pkg/apis/provisioning.cattle.io/v1#ClusterSpec

in 2.5.x I would set network.plugin: none

Hi All - running into a bit of a crisis here. I'm trying to migrate my Rancher 2.7.0 instance from a kubernetes cluster running 1.21.7 to a new one running 1.24.2. The backup/restore goes fine, but on the new cluster, Rancher never passes its readiness check. Any ideas as to where to look?

I am trying to import an AWS access key as cloud credential in rancher server. The authentication is failing saying Action=DescribeRegions&Version=2016-11-15 is denied, even though the key has admin access to the account. Any clue what's happening here ?

👋 Hi everyone!

Hi everyone I hope you are doing well! actually, try to play with a a lab that I made: installing rancher (k3s) using helm in a ubuntu vm "localy in my laptop", and it works good; but when I try to import a another cluster that I made using k3s I face that problem : "Unable to connect to the server: x509: certificate is valid for ..." Can someone help me that? and thanks in advance

Hi everyone. I am using rancher on mac m1 and I am getting a terrible performance taking like 10x more time to run something comparing to other Macs. Are you aware of it? Is there anything I can do to improve this? Anything in your plans?

👋 Hi everyone!

Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) -- are you aware about this error .

i'm freshly installed rancher and trying to pull an image

Is there a way to stop kube-system deploying local-path? it's deploying an outdated version which causes problems

Hello everyone. Could please help me. I have rke2 cluster which was installed with Rancher UI. Now we haven't Rancher UI. We want add master and worker nodes. I used standart config.yaml with token. But i get error on rke2-server

Dec  7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=warning msg="not running in CIS mode"

Dec  7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=info msg="Starting rke2 v1.23.14+rke2r1 (83c8eb91b96ed5a2a3beb06c905e075724305a46)"

Dec  7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=info msg="Managed etcd cluster not yet initialized"

Dec  7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=fatal msg="starting kubernetes: preparing server: failed to validate server configuration: https://<IP_ADDR>:9345/v1-rke2/config: 401 Unauthorized"

What do it want?

Hi, today we tried to move the local cluster from

fleet-local

to

fleet-defaul

by following the instructions here: https://fleet.rancher.io/troubleshooting#migrate-the-local-cluster-to-the-fleet-default-cluster After we initialized the move, I could see this in the logs:

rancher-5677f59677-shbs7 rancher 2022/12/07 12:39:42 [ERROR] error syncing 'local': handler provisioning-cluster-create: failed to create fleet-default/local <http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>, Kind=
Cluster for provisioning-cluster-create local: admission webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>" denied the request: cluster name must be 63 characters or fewer, cannot be "local" nor of the form "c
-xxxxx", requeuing

While this was happening, the local clulster had been removed from the

fleet-local

workspace but for the reason above it did not show up in

fleet-default

. We managed to get the cluster back to

fleet-local

again after editing the object, but it would have been nice to have all the "Continuous delivery" clusters in the same workspace. If you have any hints how to make that migration successful, please share 🙂

Hey so...we're still on RancherOS, planning to migrate at some point in the future, but we've run into a somewhat critical problem. One of our nodes is unable to verify the certificate of cdn3.quay.io, which means things are broken. Is there a way to upgrade the CA database are we stuck?

Looks like switching the console to Ubuntu fixes docker. Neat.

Hello everybody, The installation of the Monitoring application has failed, so I have decided to delete it, and reinstall it. However, although helm reports that the chart has been successfully uninstalled, the rancher-monitoring, and rancher-monitoring-crd applications keep being installed and uninstalled. Can anybody please explain why this is happening? It is really frustrating. May there be any stale resources that need to be cleaned up, or do I have to simply let Rancher stabilize? I have tried several times to install the Monitoring application. Thanks in advance

Hi, I m planning to provision 1.21.14 kubernetes version cluster using rke template from rancher-2.6.2. However its not listed in v2.6.2 release notes, wondering if rancher recommends only the listed k8s version or also the ones which are available in rke-template? https://github.com/rancher/rancher/releases/tag/v2.6.2 Kubernetes Versions • v1.21.5 • v1.20.11 • v1.19.15 • v1.18.20

👋 Hi everyone!

I am planning to take a backup for AKS cluster through rancher, is this possible, can anyone guide me on this??