rancher-users #general

bulky-eye-75833

07/03/2023, 7:39 AM

image.png

bulky-eye-75833

07/03/2023, 7:39 AM

Hi guys. Who can help me find out what was wrong with my k3s cluster? I have no idea what happens when this morning this issue appears

little-cat-5317

07/03/2023, 8:11 AM

Hi folks, I am new to Rancher, could you please help me to connect K3S cluster to the Rancher? When I run this command in the Linux instance which has installed K3S

Copy code

kubectl apply -f <https://rancher.example.com/v3/import/xfsm79gqkzz85n52tfxrhszrkfsjhkj2ksxm65r27z4q5m2gscf4gx_c-m-ttnw2tsk.yaml>

I am having these pods

Copy code

NAMESPACE       NAME                                      READY   STATUS             RESTARTS         AGE
cattle-system   cattle-cluster-agent-5ddc44d54f-wk2r9     0/1     CrashLoopBackOff   10 (4m25s ago)   30m
kube-system     coredns-7b5bbc6644-cpg7r                  1/1     Running            0                39h
kube-system     helm-install-traefik-crd-hr7z5            0/1     Completed          0                39h
kube-system     helm-install-traefik-stdv6                0/1     Completed          1                39h
kube-system     local-path-provisioner-687d6d7765-5n4lw   1/1     Running            0                39h
kube-system     metrics-server-667586758d-xfsbv           1/1     Running            0                39h
kube-system     svclb-traefik-15282abf-t65wv              2/2     Running            0                39h
kube-system     traefik-64b96ccbcd-l2fbh                  1/1     Running            0                39h

but the

cattle-cluster-agent

keeps restarting, and here is the error message

Copy code

INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local DOMAINS nameserver 10.43.0.10 options ndots:5
ERROR: <https://rancher.example.com/ping> is not accessible (OpenSSL/1.1.1l-fips: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure)

I am able to connect to the

<http://rancher.example.com|rancher.example.com>

using curl or openssl in the instance.

bulky-eye-75833

07/03/2023, 8:54 AM

7月 03 163314 c03 rancher-system-agent[1677692]: time="2023-07-03T163314+08:00" level=error msg="error syncing 'fleet-default/custom-09e3e8a1facc-machine-plan': handler secret-watch: secret received was too old, requeuing" 7月 03 164411 c03 rancher-system-agent[1677692]: time="2023-07-03T164411+08:00" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (4394395 vs 4394890)" 7月 03 164411 c03 rancher-system-agent[1677692]: time="2023-07-03T164411+08:00" level=error msg="error syncing 'fleet-default/custom-09e3e8a1facc-machine-plan': handler secret-watch: secret received was too old, requeuing" 7月 03 164501 c03 rancher-system-agent[1677692]: time="2023-07-03T164501+08:00" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (4395112 vs 4395313)" 7月 03 164501 c03 rancher-system-agent[1677692]: time="2023-07-03T164501+08:00" level=error msg="error syncing 'fleet-default/custom-09e3e8a1facc-machine-plan': handler secret-watch: secret received was too old, requeuing" 7月 03 164526 c03 rancher-system-agent[1677692]: time="2023-07-03T164526+08:00" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (4395313 vs 4395518)" 7月 03 164526 c03 rancher-system-agent[1677692]: time="2023-07-03T164526+08:00" level=error msg="error syncing 'fleet-default/custom-09e3e8a1facc-machine-plan': handler secret-watch: secret received was too old, requeuing" 7月 03 164612 c03 rancher-system-agent[1677692]: time="2023-07-03T164612+08:00" level=error msg="[K8s] received secret to process that was older than the last secret operated on. (4395921 vs 4395966)" 7月 03 164612 c03 rancher-system-agent[1677692]: time="2023-07-03T164612+08:00" level=error msg="error syncing 'fleet-default/custom-09e3e8a1facc-machine-plan': handler secret-watch: secret received was too old, requeuing" 7月 03 165111 c03 rancher-system-agent[1677692]: time="2023-07-03T165111+08:00" level=error msg="error while running probe (kube-apiserver): net/http: request canceled (Client.Timeout or context cancellation while reading body)" 7月 03 165111 c03 rancher-system-agent[1677692]: time="2023-07-03T165111+08:00" level=error msg="error running probe kube-apiserver" 7月 03 165150 c03 rancher-system-agent[1677692]: time="2023-07-03T165150+08:00" level=error msg="error while running probe (kube-apiserver): net/http: request canceled (Client.Timeout or context cancellation while reading body)" 7月 03 165150 c03 rancher-system-agent[1677692]: time="2023-07-03T165150+08:00" level=error msg="error running probe kube-apiserver"

bulky-eye-75833

07/03/2023, 8:55 AM

image.png

clean-electrician-68591

07/03/2023, 9:05 AM

Is downloading/managing charts from github a common problem where there is timeouts and the rancher app keeps throwing error 500s? I am at this stage considering using some form of mirror/proxy to get around this annoyance

nutritious-whale-24771

07/03/2023, 10:12 AM

Hi fellow k3s users, Let me ask a probably dumb question. Is it possible to run dockerd along with k3s? I have a k3s cluster consisting of 5 RPi 4 nodes. But now I want to build and deploy custom images to the cluster from either my MacBook (x64) or from a cluster node. Although I can’t do this easily. I can’t build arm64 images on Mac. And no docker stuff is installed on any of the RPi nodes. Any idea how I can overcome this dilemma?

wide-midnight-35930

07/03/2023, 10:47 AM

Hallo, is it possible to run longhorn behind a reverse proxy other than default path /. For instance accessing frontend like

IP/longhorn

little-cat-5317

07/03/2023, 2:37 PM

Can anyone help me with this?

bright-fireman-42144

07/03/2023, 4:20 PM

apologies for x-post from #rke2 I have deployed an Ubuntu 20.04.6 LTS (not 22, some weird issue with it hanging at the BIOS) using rancher on a harvester cluster and chose cilium as the CNI and would like to explore cilium more and would like to enable hubble and just mess around in general with ingress, loadbalancing, service mesh, etc. How do I enable hubble? rogue@TUMBLER:~/ciliumLBPool$ k get nodes NAME STATUS ROLES AGE VERSION kusanagi-pool1-78a45fe2-b4j2c Ready control-plane,etcd,master,worker 12h v1.25.10+rke2r1 kusanagi-pool1-78a45fe2-p9mnr Ready control-plane,etcd,master,worker 12h v1.25.10+rke2r1 kusanagi-pool1-78a45fe2-v6l2j Ready control-plane,etcd,master,worker 12h v1.25.10+rke2r1 rogue@TUMBLER:~/ciliumLBPool$ cilium status /¯¯\ /¯¯\__/¯¯\ Cilium: OK \__/¯¯\__/ Operator: OK /¯¯\__/¯¯\ Envoy DaemonSet: disabled (using embedded mode) \__/¯¯\__/ Hubble Relay: disabled \__/ ClusterMesh: disabled DaemonSet cilium Desired: 3, Ready: 3/3, Available: 3/3 Deployment cilium-operator Desired: 2, Ready: 2/2, Available: 2/2 Containers: cilium-operator Running: 2 cilium Running: 3 Cluster Pods: 20/20 managed by Cilium Helm chart version: Image versions cilium rancher/mirrored-cilium-ciliumv1.13.2 3 cilium-operator rancher/mirrored-cilium-operator-genericv1.13.2 2 rogue@TUMBLER:~/ciliumLBPool$ cilium hubble enable Error: Unable to enable Hubble: release: not found rogue@TUMBLER:~/ciliumLBPool$ cilium version cilium-cli: v0.15.0 compiled with go1.20.4 on linux/amd64 cilium image (default): v1.13.4 cilium image (stable): v1.13.4 cilium image (running): unknown. Unable to obtain cilium version, no cilium pods found in namespace "kube-system"

few-bird-29459

07/04/2023, 3:27 AM

Hi all, Is there any specific guide or performance tunning guidelines for Ubuntu 22.04 for Rancher and downstream cluster nodes? We are using ubuntu22.04 as base template for nodes in vSphere environment.. nodes are performing very low for the workloads

big-art-58598

07/04/2023, 5:10 AM

이 메시지는 상호작용 요소가 포함되어 있습니다.

few-jordan-3349

07/04/2023, 8:53 AM

Hi, is there an Rancher Helm Chart option to disable the feature flags multi-cluster-management & rke2? I cannot find it in docs, for my airgapped rancher install i do not need this features. Thanks:)

ambitious-petabyte-62704

07/04/2023, 9:38 AM

Hi, for some reason, I keep getting the wsl kernel does not appear to be installed, even I have Uninstalled wsl and rancher desktop and reinstall them again, I'm using wsl 2 and rancher desktop 1.9.1

curved-army-69172

07/04/2023, 11:13 AM

A general question regarding upgrades: I'm currently on 2.6.9 - are upgrades supported from that to latest 2.7.<latest> or should I upgrade to 2.6.<latest> first?

bumpy-sundown-2809

07/04/2023, 2:20 PM

Hi, I am facing an issue with Rancher Desktop version 1.8.1. I have a docker container running within Rancher Desktop from which I am trying to ping Elastic Search which is running on my host machine. The curl call just hangs. I have enabled the firewall that is mentioned in the Rancher Desktop FAQs. That has also not resolved the issue. Any other pointers that will help resolve this issue? Do we need any specific mapping in hosts file?

narrow-honey-55422

07/04/2023, 2:28 PM

i'm deploying 2.7.5 rancher on EKS 1.25. with those values :

Copy code

ingress:
  enabled: true
  extraAnnotations:
    <http://alb.ingress.kubernetes.io/certificate-arn|alb.ingress.kubernetes.io/certificate-arn>: "<ACM-ARN>"
    <http://alb.ingress.kubernetes.io/scheme|alb.ingress.kubernetes.io/scheme>: "internet-facing"
    <http://alb.ingress.kubernetes.io/success-codes|alb.ingress.kubernetes.io/success-codes>: '200,404,301,302'
    <http://alb.ingress.kubernetes.io/listen-ports|alb.ingress.kubernetes.io/listen-ports>: '[{"HTTP": 80}, {"HTTPS": 443}]'
    <http://alb.ingress.kubernetes.io/target-type|alb.ingress.kubernetes.io/target-type>: 'ip'
 
  ingressClassName: alb

tls: external

hostname: "<http://rancher.example.com|rancher.example.com>"
bootstrapPassword: "admin"

i'm getting 404 (even in https://rancher.example.com/dashboard/?setup=admin aswell), in the pod logs :

Copy code

2023/07/04 14:12:41 [INFO] Adding peer <wss://10.0.28.122/v3/connect>, 10.0.28.122
2023/07/04 14:12:41 [ERROR] Failed to connect to peer <wss://10.0.28.122/v3/connect> [local ID=10.0.39.99]: websocket: bad handshake
2023/07/04 14:12:42 [INFO] Adding peer <wss://10.0.5.72/v3/connect>, 10.0.5.72
2023/07/04 14:12:42 [ERROR] Failed to connect to peer <wss://10.0.5.72/v3/connect> [local ID=10.0.39.99]: websocket: bad handshake
2023/07/04 14:12:42 [INFO] Handling backend connection request [10.0.28.122]
2023/07/04 14:12:42 [INFO] Handling backend connection request [10.0.5.72]

2023/07/04 14:13:29 [ERROR] error syncing 'validating-webhook-configuration': handler need-a-cert: secrets "rancher-webhook-tls" already exists, requeuing

any suggestions?

full-keyboard-29604

07/04/2023, 5:45 PM

I need professional assistance with our setup. Is there anyone here that accepts (smaller) consulting assignments?

bitter-restaurant-32653

07/05/2023, 4:56 AM

Hi. I'm using RKE1 to init my K8s cluster. I know that RKE2 is the next step, but migrating to RKE2 in on-premise infra is pretty tough. So I have a few questions: • RKE1 stable and long-term support? • Should we upgrade to RKE2 for some kind of reasons? Is there any good solution for on-premise infra?

famous-king-68594

07/05/2023, 6:21 AM

hi i need support , we have deployed rancher as docker , now the container is getting restart every 10 mins and also i see there high memory utilization in the rancher container

melodic-pager-20185

07/05/2023, 8:16 AM

Hi, Im using dockerd with Rancher Desktop app on osx, and I would like to inject Custom CA Root Certs as per (https://ranchermanager.docs.rancher.com/v2.5/reference-guides/single-node-rancher-in-docker/advanced-options#custom-ca-certificate). Is there any way how to do that? Thx in advance

dry-gigabyte-68274

07/05/2023, 1:01 PM

Hi folks, does anyone know how to get an nfs mount running on rancherdesktop? I always get a Permission denied:

Copy code

Warning  FailedMount  8s (x6 over 25s)  kubelet            MountVolume.SetUp failed for volume "epimproxy-assets" : mount failed: exit status 255
Mounting command: mount
Mounting arguments: -t nfs 192.168.1.47:/volume1/edeka-nord /var/lib/kubelet/pods/7b900edb-19c0-4fd4-ad77-f3988d68f1ea/volumes/kubernetes.io~nfs/epimproxy-assets
Output: mount: mounting 192.168.1.47:/volume1/edeka-nord on /var/lib/kubelet/pods/7b900edb-19c0-4fd4-ad77-f3988d68f1ea/volumes/kubernetes.io~nfs/epimproxy-assets failed: Permission denied

It seems the directory /var/lib/kubelet/pods refers to a local dir on my host. But there is no subdir pods under /var/lib/kubelet I'm clueless.

wide-book-96534

07/05/2023, 6:55 PM

Hi all…does anyone know how to get logs from Rancher V2 logging over to OpenSearch WITH a timestamp? I can successfully connect and send logs but the

@timestamp

field is null.

wide-book-96534

07/05/2023, 6:56 PM

I’m using the following clusterflow…is there something wrong here?

wide-book-96534

07/05/2023, 6:56 PM

Copy code

apiVersion: <http://logging.banzaicloud.io/v1beta1|logging.banzaicloud.io/v1beta1>
kind: ClusterFlow
metadata:
  name: dev-namespaces-flow
  namespace: cattle-logging-system
spec:
  filters:
    - parser:
        key_name: message
        reserve_data: true
        parse:
          type: json
  globalOutputRefs:
    - aws-opensearch-dev-project
  match:
    - select:
        namespaces:
          - app-dev

abundant-hair-58573

07/06/2023, 1:56 AM

Is it possible to pre download the images needed for a worker node, like you can do in the air-gapped K3s install? I see the air-gapped instructions for Rancher but that looks like it's just for using a private registry in an air-gapped environment. I want to create an AMI for the worker with the necessary images for faster auto-scaling

important-petabyte-50139

07/06/2023, 5:27 AM

Stupid question: How do I change the config of the existing rke2-ingress-controller? Do I just change the rke2-ingress-controller-config.yaml in /var/lib/rancher/rke2/server/manifests? Are those changes automatically picked up or do I have to restart anything? Or is there a way to change it in the Rancher GUI?

curved-army-69172

07/06/2023, 6:45 AM

Hi, we're hit by https://github.com/rancher/rancher/issues/41465 - at the end it is mentioned that after manually assigning the permissions, to get rid of the error message, one needs to set cluster.status.Condition[].type=GlobalAdminsSynced to true. How do I do that? I'm struggling to find where to update that

breezy-umbrella-46432

07/06/2023, 6:50 AM

breezy-umbrella-46432

07/06/2023, 6:51 AM