Hi, I'm trying to add cluster CCE (Huawei Cloud Cluster) but it's timeout to download rancher driver. Can someone please help me ?

I've installed Rancher 2.7.0 on top of a fresh k3s installation which is running on CentOS 7. I'm running into a couple of issues which may be related to my proxy settings. 1. I see errors like these in the logs of the Rancher pods: 2022/12/15 164033 [ERROR] error syncing 'rancher-rke2-charts': handler helm-clusterrepo-ensure: git -C /var/lib/rancher-data/local-catalogs/v2/rancher-rke2-charts/675f1b63a0a83905972dcab2794479ed599a6f41b86cd6193d69472d0fa889c9 fetch origin 88c6633998c1b6e528f15a74428930c5df880908 error: exit status 128, detail: fatal: unable to access 'https://git.rancher.io/rke2-charts/': Received HTTP code 407 from proxy after CONNECT a. if I exec into the Rancher container and run "git config --global http.proxyAuthMethod 'basic'" these errors go away. Not sure if this should be considered a bug... 2. I'm also seeing errors downloading icons: time="2022-12-15T200537Z" level=error msg="Unknown error: Get \"https://charts.rancher.io/assets/logos/logging.svg\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" a. not sure how to fix this one yet. I can exec into the container and "curl --insecure https://charts.rancher.io/assets/logos/logging.svg" just fine Any help is appreciated 🙂

I've got a few 2.7.0 clusters with imported RKE2 Downstream Clusters. I've noticed when navigating to the "Edit/Upgrade" screen for an installed helm app, it's blank. Anyone else running into this? I've got 3 instances I've been able to repo on, but want to make sure it isn't just me before starting to dig deep and opening issues

do I need to do anything on Ubuntu's cloud-images to use them as templates for worker clusters on vSphere?

I've created a new Rancher 2.7.0 cluster with RKE2 on our air-gap environment. Now I need to install istio on it. I assume default https://git.rancher.io/partner-charts contain all necessary charts for it, but repo is not accessible directly. How I could download all necessary charts and images to have it on my personal repository? Or maybe some one had same challenges with air gaped env, will be appreciate for any help.

Hello everybody, i updated with rke kubernetes to version 1.24. when i run rke up to update i get this error`:`

INFO[0070] Finding container [service-sidekick] on host [10.155.10.113], try #1
INFO[0070] [sidekick] Sidekick container already created on host [10.155.10.113]
INFO[0070] [healthcheck] Start Healthcheck on service [kubelet] on host [10.155.10.113]
ERRO[0121] Failed to upgrade worker components on NotReady hosts, error: [Failed to verify healthcheck: Failed to check <http://localhost:10248/healthz> for service [kubelet] on host [10.155.10.113]: Get "<http://localhost:10248/healthz>": Unable to access the service on localhost:10248. The service might be still starting up. Error: ssh: rejected: connect failed (Connection refused), log: I1216 09:55:52.876292 1131786 docker_service.go:359] "Docker cri received runtime config" runtimeConfig="&RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:,},}"]
INFO[0121] [controlplane] Now checking status of node 10.155.10.113, try #1
INFO[0126] [controlplane] Now checking status of node 10.155.10.113, try #2
INFO[0131] [controlplane] Now checking status of node 10.155.10.113, try #3
INFO[0136] [controlplane] Now checking status of node 10.155.10.113, try #4
INFO[0141] [controlplane] Now checking status of node 10.155.10.113, try #5
ERRO[0146] Host 10.155.10.113 failed to report Ready status with error: host 10.155.10.113 not ready
INFO[0146] [controlplane] Processing controlplane hosts for upgrade 1 at a time
INFO[0146] Processing controlplane host 10.155.10.113
INFO[0146] [controlplane] Now checking status of node 10.155.10.113, try #1
INFO[0151] [controlplane] Now checking status of node 10.155.10.113, try #2
INFO[0156] [controlplane] Now checking status of node 10.155.10.113, try #3
INFO[0161] [controlplane] Now checking status of node 10.155.10.113, try #4
INFO[0166] [controlplane] Now checking status of node 10.155.10.113, try #5
ERRO[0171] Failed to upgrade hosts: 10.155.10.113 with error [host 10.155.10.113 not ready]
FATA[0171] [controlPlane] Failed to upgrade Control Plane: [[host 10.155.10.113 not ready]]

On master1 I checked the kubelet docker container log

[root@vit81-ldevsa00a ~]# docker logs kubelet --follow  --tail 10
      --tls-private-key-file string                              File containing x509 private key matching --tls-cert-file. (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.)
      --topology-manager-policy string                           Topology Manager policy to use. Possible values: 'none', 'best-effort', 'restricted', 'single-numa-node'. (default "none") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.)
      --topology-manager-scope string                            Scope to which topology hints applied. Topology Manager collects hints from Hint Providers and applies them to defined scope to ensure the pod admission. Possible values: 'container', 'pod'. (default "container") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.)
  -v, --v Level                                                  number for the log level verbosity (default 0)
      --version version[=true]                                   Print version information and quit
      --vmodule pattern=N,...                                    comma-separated list of pattern=N settings for file-filtered logging (only works for text log format)
      --volume-plugin-dir string                                 The full path of the directory in which to search for additional third party volume plugins (default "/usr/libexec/kubernetes/kubelet-plugins/volume/exec/") (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.)
      --volume-stats-agg-period duration                         Specifies interval for kubelet to calculate and cache the volume disk usage for all pods and volumes.  To disable volume calculations, set to a negative number. (default 1m0s) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See <https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/> for more information.)

Error: failed to run Kubelet: unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix: missing address"
[root@vit81-ldevsa00a ~]#

I'm using rke version 1 1.3.10

[root@vit81-ldevsa00a ~]# rke --version
rke version v1.3.10

in the cluster.yaml file I added this line:

authorization:
  mode: rbac
  options: {}
ignore_docker_version: null
#enable_cri_dockerd: null
enable_cri_dockerd: false
#kubernetes_version: "v1.23.14-rancher1-1"
kubernetes_version: "v1.24.8-rancher1-1" <<<----- THIS LINE ADDED
private_registries: []
ingress:
  provider: ""
  options: {}

Resolved. Using v1.4.2-rc1

Hi everyone, so I’m trying to create a kubernetes cluster that can allow be add servers from different provider(aws, azure, DO) as worker nodes, has anyone tried creating a kubernetes cluster using a custom provider and then tried to add an AWS EC2 node as a worker node using the “sudo docker …” command? I tried it but it’s not working, Digital ocean was added successfully and works but not AWS EC2

Hi everyone, I am trying to provision an infrastructure-provided RKE2 Cluster on the OpenStack cloud using Rancher. I have cloud credentials ready, when I create a cluster from Rancher's GUI with OpenStack cloud, in machine pool form a lot of fields are given, but I can't determine which fields are required and which are optional. Also, I can't find any proper documentation to create clusters on the OpenStack cloud, I tried many combinations but nothing worked. Any help is appreciated.

Hi Everyone, I'm having some issues with my downstreams clusters on RKE2. When I deploy any downstream cluster from Rancher (v2.6.8) GUI, the deploy stucks and the provisioning log tab shows this: [ERROR] Failed to set up SSH tunneling for host [10.1.76.18]: Can't retrieve Docker Info: error during connect. I accessed the cluster (In this case just for testing pupouse is a one node cluster doing control plane, worker and etcd) via ssh and do a "docker ps" and a "docker logs". This logs show: time="2022-12-16T135509Z" level=error msg="Issuer of last certificate found in chain (CN=AAA Certificate Services,O=Comodo CA Limited,L=Salford,ST=Greater Manchester,C=GB) does not match with CA certificate Issuer (CN=dynamiclistener-ca@1662989077,O=dynamiclistener-org). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)" time="2022-12-16T135509Z" level=fatal msg="Server certificate is not valid, please check if the host has the correct time configured and if the server certificate has a notAfter date and time in the future. Certificate information is displayed above. error: Get \"https://xxx.xxx.xxx.xxx\": x509: certificate has expired or is not yet valid: current time 2022-12-16T135509Z is after 2022-11-26T235959Z" I am running Rancher in HA cluster with HAProxy in front of it. The downstream cluster creates VMs in VMware architecture and they all stuck at this point. I know that is a certificate issue but I don't know where I can find o replace that certificate. I will appreciate any kind of help. I'm trying to solve this issue since long time.

can rancher cause

eks node groups

deletion or creation while they are just imported clusters? we are seeing some weird messages in eks node group section. FYI - we are just importing EKS clusters not creating by rancher. We see node groups creation in EKS that shows rancher trying to create.

Good morning - I had some type of issue overnight that caused almost all of my volumes in Longhorn to become 'missing'. The pvc's show up in Rancher, but the pods wont deploy as the volume is missing when I look for them in Longhorn. I can see the PVC in /var/lib/longhorn/replica but I have no idea how the volume detached from the pvc. I have exported the volume from a single replica using the doc online but is there a way to recreate the volume and point it to the already existing PVC in /var/lib/longhorn/replica? Or, do I need to recreate the longhorn pv/pvc and copy the data in myself? Thanks!

Hi, I've run Rancher 2.7.0 with CATTLE_SYSTEM_CATALOG=bundled option, and it works but when I push any buttons on App menu it loads and after couple min refreshes and returns back to menu without any result. In logs a lot of errors like that: [ERROR] Unknown error: Get "": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Any ideas about it?

OK - getting frustrated - who knows how to delete the old, legacy rancher monitoring/logging from the cluster. I’ve tried deleting them from rancher apps, but they keep showing up

what taints should I use to disable running workloads on a control plane? I can't find the docs

node-taint: - "CriticalAddonsOnly=true:NoExecute" is this one the only one?

@ambitious-plastic-3551 from my code snippets I see

kubectl taint nodes node-1 <http://node-role.kubernetes.io/controlplane=true:NoSchedule|node-role.kubernetes.io/controlplane=true:NoSchedule>

would this help: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

Thanks @bored-farmer-36655 will check it out

is it possible to expose hostPort with TCP and UDP at the same time

@ambitious-plastic-3551 Like this: https://jamesdefabia.github.io/docs/user-guide/kubectl/kubectl_expose/ or port forward?

well yaml version 😄 or even better helm chart...

I have to open github issue

Hello. I just rolled my first Harvester installation and created my first VM on this. I only have one interface, so I added a bridged interface on another VLAN but still on the mgmt interface. The VM booted and successfully got a ip from my dhcp. I can connect to the VM from outside, but cannot connect to anything from inside the VM. Is it possible the have a bridge on the same physical interface as the mgmt network?

Hi I am new to this app. After installing this no command(docker,kubectl etc) is working in either in cmd or wsl

is there a way to edit an ingress in a file locally and apply it to a remote? i promise i googled and tried finding a solution but can only see kubectl edit and never know if the formatting is right by manually editing on the shell

Hi, I'm trying to add new node to my Rancher k8s cluster. I have 1 rancher server(v2.6.9) running with docker container, 1 master node, 2 worker nodes, I want to add a new worker node to the cluster, but when I run registration command on new server, it show me below: # curl --insecure -fL https://192.168.0.57:8443/system-agent-install.sh | sudo sh -s - --server https://192.168.0.57:8443 --label 'cattle.io/os=linux' --token zxmdxmxdlwbgllpmf5zcwmr5rjg7bcb5nt7lv7k66rw6qz94k59tpd --ca-checksum 3a76abfb81778a09e15616d979bc298475dafab642b96daf23ff54b8c0a24b04 --worker % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 29798 0 29798 0 0 2645k 0 -- -- -- -- -- -- 2645k [INFO] Label: cattle.io/os=linux [INFO] Role requested: worker [INFO] Using default agent configuration directory /etc/rancher/agent [INFO] Using default agent var directory /var/lib/rancher/agent [INFO] Determined CA is necessary to connect to Rancher [INFO] Successfully downloaded CA certificate [INFO] Value from https://192.168.0.57:8443/cacerts is an x509 certificate [INFO] Successfully tested Rancher connection [INFO] Downloading rancher-system-agent binary from https://192.168.0.57:8443/assets/rancher-system-agent-amd64 [INFO] Successfully downloaded the rancher-system-agent binary. [INFO] Downloading rancher-system-agent-uninstall.sh script from https://192.168.0.57:8443/assets/system-agent-uninstall.sh [INFO] Successfully downloaded the rancher-system-agent-uninstall.sh script. [INFO] Generating Cattle ID curl: (28) Operation timed out after 60000 milliseconds with 0 bytes received [ERROR] 000 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received [ERROR] 000 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received [ERROR] 000 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again curl: (28) Operation timed out after 60001 milliseconds with 0 bytes received [ERROR] 000 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again

Can anyone knows the reason? thanks very much.

can someone tell what was the fix here https://github.com/rancher/rancher/issues/28967 ? we are running into similar issue with rancher with out imported EKS clusters?