https://rancher.com/ logo
Join Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
general
  • a

    able-wall-25846

    11/29/2022, 5:22 PM
    I'm not able to pull images from a private registry after creating an RKE2 cluster using rancher 2.6.5 here is my config 
    root@knode03:~# cat /etc/rancher/rke2/registries.yaml
mirrors:
  <http://registry01.sys.nc4.iad0.nsscloud.net|registry01.sys.nc4.iad0.nsscloud.net>:
    endpoint:
      - "<http://registry01.sys.nc4.iad0.nsscloud.net:5000>"
root@knode03:~#
  • a

    able-wall-25846

    11/29/2022, 5:22 PM
    but I cannot pull the images
  • a

    able-wall-25846

    11/29/2022, 5:22 PM
    root@knode03:~# crictl pull <http://registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest|registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest>
FATA[0000] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "<http://registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest|registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest>": failed to resolve reference "<http://registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest|registry01.sys.nc4.iad0.nsscloud.net:5000/ns-ubuntu-20-04-qemu:latest>": failed to do request: Head "<https://registry01.sys.nc4.iad0.nsscloud.net:5000/v2/ns-ubuntu-20-04-qemu/manifests/latest>": http: server gave HTTP response to HTTPS client
  • a

    able-wall-25846

    11/29/2022, 5:23 PM
    any tips on how I can get this working?
  • a

    able-wall-25846

    11/29/2022, 5:25 PM
    /var/lib/rancher/rke2/agent/etc/containerd/config.toml
  • a

    able-wall-25846

    11/29/2022, 5:25 PM
    has this
  • a

    able-wall-25846

    11/29/2022, 5:25 PM
    [plugins.cri.registry.mirrors."<http://registry01.sys.nc4.iad0.nsscloud.net|registry01.sys.nc4.iad0.nsscloud.net>"]
  endpoint = ["<http://registry01.sys.nc4.iad0.nsscloud.net:5000>"]
  • a

    able-wall-25846

    11/29/2022, 5:25 PM
    I think that's right
    d
    • 2
    • 31
  • f

    flaky-shampoo-86024

    11/29/2022, 6:33 PM
    Is there a way to prevent Rancher from adding finalizers to specific namespaces? We are trying to prevent this finalizer from being added to a specific namespace. finalizers: - controller.cattle.io/namespace-auth , since most of namespaces are in "Terminating" state when trying to get removed
  • g

    gentle-petabyte-40055

    11/30/2022, 5:22 AM
    Hello. Has anyone used Velero with an RKE cluster. How do you use the cli. Do you install it on one of the control plane nodes?
  • b

    best-address-42882

    11/30/2022, 5:34 AM
    Hi, I am creating brand new cluster and getting "Disconnected] Cluster agent is not connected" and "Waiting for API to be available" errors. I've tested cluster creating on Rancher 2.6.9 and 2.7.0 and got same error. Issue affects only Weave CNI, others works fine. Spent 3 days on it with no result.
  • s

    salmon-chef-55025

    11/30/2022, 9:18 AM
    hi , we have a legacy Rancher Cluster running Rancher version v2.4.4 Rke version v1.0.8 Kubernetes version v1.17.5 Docker version v18.9.2 Our deployments are failing with following errors 
    "unable to decode an event from the watch stream: backed up reader") has prevented the request from succeeding




[main] 2022/11/23 14:38:18 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:38:18 GRPC listening on :61000

[main] 2022/11/23 14:38:18 Probes listening on :50491

[main] 2022/11/23 14:38:18 Storage driver is ConfigMap

[main] 2022/11/23 14:38:18 Max history per release is 10

[tiller] 2022/11/23 14:38:19 getting history for release ne-campaign

[storage] 2022/11/23 14:38:19 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:38:19 preparing update for ne-campaign

[storage] 2022/11/23 14:38:19 getting deployed releases from "ne-campaign" history

[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:38:19 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:38:19 performing force update for ne-campaign

[storage] 2022/11/23 14:38:19 getting last revision of "ne-campaign"

[storage] 2022/11/23 14:38:19 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:38:19 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address





[main] 2022/11/23 14:38:19 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:38:19 GRPC listening on :50491

[main] 2022/11/23 14:38:19 Probes listening on :50491

[main] 2022/11/23 14:38:19 Storage driver is ConfigMap

[main] 2022/11/23 14:38:19 Max history per release is 10

[main] 2022/11/23 14:38:19 Probes server died: listen tcp :50491: bind: address already in use

[main] 2022/11/23 14:48:20 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:48:20 GRPC listening on :50491

[main] 2022/11/23 14:48:20 Probes listening on :61000

[main] 2022/11/23 14:48:20 Storage driver is ConfigMap

[main] 2022/11/23 14:48:20 Max history per release is 10

[tiller] 2022/11/23 14:48:21 getting history for release ne-campaign

[storage] 2022/11/23 14:48:21 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:21 preparing update for ne-campaign

[storage] 2022/11/23 14:48:21 getting deployed releases from "ne-campaign" history

[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:21 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:21 performing force update for ne-campaign

[storage] 2022/11/23 14:48:21 getting last revision of "ne-campaign"

[storage] 2022/11/23 14:48:21 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:21 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[main] 2022/11/23 14:48:22 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/23 14:48:22 GRPC listening on :50491

[main] 2022/11/23 14:48:22 Probes listening on :61000

[main] 2022/11/23 14:48:22 Storage driver is ConfigMap

[main] 2022/11/23 14:48:22 Max history per release is 10

[tiller] 2022/11/23 14:48:23 getting history for release ne-campaign

[storage] 2022/11/23 14:48:23 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:23 preparing update for ne-campaign

[storage] 2022/11/23 14:48:23 getting deployed releases from "ne-campaign" history

[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:23 failed to prepare update: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER%2CSTATUS%3DDEPLOYED>: dial tcp 127.0.0.1:443: connect: cannot assign requested address

[tiller] 2022/11/23 14:48:23 performing force update for ne-campaign

[storage] 2022/11/23 14:48:23 getting last revision of "ne-campaign"

[storage] 2022/11/23 14:48:23 getting release history for "ne-campaign"

[storage/driver] 2022/11/23 14:48:23 query: failed to query with labels: Get <https://localhost:443/k8s/clusters/c-xzmr2/api/v1/namespaces/ne-campaign/configmaps?labelSelector=NAME%3Dne-campaign%2COWNER%3DTILLER>: dial tcp 127.0.0.1:443: connect: cannot assign requested address
    We have tried restarting the master nodes (& docker service ) on downstream Kubernetes nodes .. but same has not helped. We have tried deploying latest code on other micro services hosted on the same cluster but that is also failing. This error seems to have worsened now .. PFB the recent logs where its getting blocked at even earlier stage. 
    2022/11/28 14:40:29 [ERROR] Error certs=InvalidBodyContent 422: failed to find cert that matched private key parsing cert template-repository-prod-tls. Will not display correctly in UI

2022/11/28 14:43:12 [INFO] Installing chart using helm version: rancher-helm

[main] 2022/11/28 14:43:12 Starting Tiller v2.16.5-rancher1 (tls=false)

[main] 2022/11/28 14:43:12 GRPC listening on :61000

[main] 2022/11/28 14:43:12 Probes listening on :61000

[main] 2022/11/28 14:43:12 Storage driver is ConfigMap

[main] 2022/11/28 14:43:12 Max history per release is 10

[main] 2022/11/28 14:43:12 Probes server died: listen tcp :61000: bind: address already in use

UPGRADE FAILED

Error: context deadline exceeded

2022/11/28 14:53:12 [ERROR] AppController p-746c7/ecomm-support [helm-controller] failed with : failed to install app ecomm-support. Error: UPGRADE FAILED: context deadline exceeded
    • 1
    • 1
  • b

    billowy-vegetable-58311

    11/30/2022, 10:21 AM
    HI team , after installing Rancher desktop I am getting the following error When I am trying to execute the first command ... Please help 
    brew install --cask --verbose --force docker
Error shows as  ...  "Error: It seems there is already a Binary at '/usr/local/share/zsh/site-functions/_docker'.
    Following is the terminal logs which I am sharing 
    vishwanathangadi@vishwanaths-MacBook-Pro ~ % brew install --cask --verbose --force docker
Running `brew update --auto-update`...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).

You have 18 outdated formulae installed.
You can upgrade them with brew upgrade
or list them with brew outdated.

==> Downloading <https://desktop.docker.com/mac/main/amd64/91661/Docker.dmg>
Already downloaded: /Users/vishwanathangadi/Library/Caches/Homebrew/downloads/1ffb31f522268752a888fbee3b48dcfa70666365c75c92b6d179e0dd22e82e89--Docker.dmg
==> Verifying checksum for cask 'docker'
==> Installing Cask docker
hdiutil attach -plist -nobrowse -readonly -mountrandom /private/tmp/d20221130-1928-ongdwu /Users/vishwanathangadi/Library/Caches/Homebrew/downloads/1ffb31f522268752a888fbee3b48dcfa70666365c75c92b6d179e0dd22e82e89--Docker.dmg
mkbom -s -i /private/tmp/20221130-1928-zb8c19.list -- /private/tmp/20221130-1928-1z0cblx.bom
ditto --bom /private/tmp/20221130-1928-1z0cblx.bom -- /private/tmp/d20221130-1928-ongdwu/dmg.khdXlw /private/tmp/d20221130-1928-1j5lsiq
diskutil info -plist /private/tmp/d20221130-1928-ongdwu/dmg.khdXlw
diskutil eject /private/tmp/d20221130-1928-ongdwu/dmg.khdXlw
cp -pR /private/tmp/d20221130-1928-1j5lsiq/Docker.app/. /usr/local/Caskroom/docker/4.14.1,91661/Docker.app
chmod -Rf +w /private/tmp/d20221130-1928-1j5lsiq
==> Moving App '<http://Docker.app|Docker.app>' to '/Applications/Docker.app'
==> Linking Binary 'docker-compose.bash-completion' to '/usr/local/etc/bash_completion.d/docker-compose'
==> Unlinking Binary '/usr/local/etc/bash_completion.d/docker-compose'
==> Backing App '<http://Docker.app|Docker.app>' up to '/usr/local/Caskroom/docker/4.14.1,91661/Docker.app'
==> Removing App '/Applications/Docker.app'
==> Purging files for version 4.14.1,91661 of Cask docker
Error: It seems there is already a Binary at '/usr/local/share/zsh/site-functions/_docker'.
    w
    • 2
    • 1
  • f

    future-alarm-92026

    11/30/2022, 2:05 PM
    👋 Hi everyone!
  • b

    bitter-noon-90925

    11/30/2022, 2:28 PM
    Hello everyone. I hope this channel is the correct one. We are currently evaluating Rancher Desktop at our Company and I’m coordinating this process. We have currently this issue https://github.com/rancher-sandbox/rancher-desktop/issues/905 with our company proxy. We have defined our proxies in our override.yamls and iff rancher desktop or lima VM boots, we can use docker registries through our proxy. But rancher desktop doesn’t start up every now and then, with the error message in the issue #905. Is there any workaround to this issue? Thanks
    ✅ 1
  • f

    fierce-nest-35400

    11/30/2022, 2:58 PM
    s
    • 2
    • 1
  • f

    fierce-nest-35400

    11/30/2022, 2:59 PM
    Hi restarted my computer and started rancher it stuck waiting for kube apis
  • f

    fierce-nest-35400

    11/30/2022, 2:59 PM
    any suggestions how to solve this issue
  • o

    orange-gigabyte-8918

    11/30/2022, 3:08 PM
    I'm not 100% sure, but I don't think so. Sorry.
  • l

    little-father-38603

    11/30/2022, 3:45 PM
    Hi, do anyone know how to increase the vebosity level for kubelet running in docker and deployed with RKE? I'm trying to debug a situation where the 
    /stats/summary
    endpoint takes a really long time to respond
  • l

    little-father-38603

    11/30/2022, 4:42 PM
    Actually, never mind, 
    docker stats
    hangs, I suppose kubelet calls that. If anyone knows why this could happen, please let me know :)
  • e

    early-lunch-37616

    11/30/2022, 5:15 PM
    Hi everyone, I'm hoping someone can help me figure out a docker auth issue I'm having. Using Rancher Desktop with nerdctl I'm getting 403 auth errors when trying to pull images from GCR. I'm logged in with nerdctl and was able to pull images when using dockerCLI. Any ideas?
    s
    • 2
    • 1
  • b

    breezy-ram-80329

    12/01/2022, 4:18 AM
    getting [ERROR] Error during subscribe websocket: close sent in the rancher server logs. And the cluster agent logs is stuck with level=info msg="Connecting to proxy" url="wss://rancher-poc.corp.xxxxcom/v3/connect". My rancher server is behind a aws application load balancer. Any idea why this is happening ?
  • f

    famous-grass-8099

    12/01/2022, 5:03 AM
    Hi Guys Here is how my setup is. single node docker image running behind nginx as reverse proxy (it is docker-compose.yml having both containers). That single node is behind AWS classic load balancer. following are the errors being logged by rancher node (docker-compose logs -f). 
    rancher-server_1  | 2022/12/01 04:54:09 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49716 (X-Forwarded-For: 44.230.106.196, 172.31.30.132): response 400: websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header
rancher-server_1  | 2022/12/01 04:54:09 [ERROR] Failed to handle tunnel request from remote address 172.19.0.2:49716 (X-Forwarded-For: 44.230.106.196, 172.31.30.132): response 400: Error during upgrade for host [c-sxlgx]: websocket: the client is not using the websocket protocol: 'websocket' token not found in 'Upgrade' header
    I have another AWS EKS cluster. I am trying to import in above rancher instance. It is showing 
    waiting
    status.
  • e

    early-lifeguard-63817

    12/01/2022, 8:50 AM
    Can I use an existing Prometheus instance to show metrics in the Rancher UI? I have imported an existing cluster which has a long-running prom operator on it. This must be some kind of FAQ but can't find a real answer.
    s
    • 2
    • 4
  • s

    shy-tent-66642

    12/01/2022, 10:18 AM
    👋 Hi everyone!
  • k

    kind-analyst-38351

    12/01/2022, 10:51 AM
    Hello, does anyone know how to enable prometheus metrics in nginx-ingress-controller (installed via RKE)?
  • g

    gifted-shampoo-48019

    12/01/2022, 12:16 PM
    The fleet-agent auto generates cluster-name such as 
    cluster-fb7393592079
    . is there a way for me to change that name?
  • a

    acceptable-printer-7134

    12/01/2022, 12:24 PM
    What is the minimum aws permissions required for rancher cloud credentials. Our use case is to just import EKS and rancher as UI and help us manage RBAC on top our auth provider.
    k
    • 2
    • 1
  • h

    helpful-student-22864

    12/01/2022, 1:16 PM
    Hello Everyone 👋 New to Rancher and this forum. Looking for help on Global Roles. The requirement is to setup roles for team members to be able to manage resources within a particular namespace only. I would appreciate if you can guide me or direct me to available resources that I can use to complete this setup. I have created a new custom role and added users to it so far, however I am still trying to find out ways to restrict this group to have access only on a particular namespace. Thank you!
    s
    • 2
    • 1
Title
h

helpful-student-22864

12/01/2022, 1:16 PM
Hello Everyone 👋 New to Rancher and this forum. Looking for help on Global Roles. The requirement is to setup roles for team members to be able to manage resources within a particular namespace only. I would appreciate if you can guide me or direct me to available resources that I can use to complete this setup. I have created a new custom role and added users to it so far, however I am still trying to find out ways to restrict this group to have access only on a particular namespace. Thank you!
s

sparse-fireman-14239

12/01/2022, 1:24 PM
https://learnk8s.io/rbac-kubernetes
View count: 2