https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
general
  • b

    bland-school-82008

    09/08/2022, 6:15 AM
    docker is not able to pull the images ..?
  • s

    salmon-carpenter-62625

    09/08/2022, 7:44 AM
    Hello. someone here delete a namespace cattle-system with cluster agents and now the rancher show
    Unavailable
    in his status. I found site https://rancher.com/docs/rancher/v2.6/en/installation/resources/update-rancher-cert/#method-3-recreate-rancher-agents that show some tips how to recreate it but, the point is that from what I read it says about executing something on control-plan nodes, In my case I use AKS clusters so have just the workload nodes. Any tips what I can do to restore the rancher agent on such scenario (I don't have etcd backup)?
    • 1
    • 2
  • e

    eager-vr-25932

    09/08/2022, 10:25 AM
    Found a workaround here: https://github.com/rancher/rancher/issues/38575 https://github.com/rancher/dashboard/issues/6122#issuecomment-1152394027
  • p

    purple-translator-99032

    09/08/2022, 12:06 PM
    Can you create a RKE2 Downstream Cluster via Rancher UI ?
    p
    • 2
    • 3
  • b

    brash-planet-10109

    09/08/2022, 12:50 PM
    Hi All, Longhorn volumes are in degraded state, however pods are up and PVCs got attached to it. I'm I doing something wrong? Rke version: 1.23.10+rke2r1 Longhorn version: v1.3.1
  • d

    dazzling-france-77424

    09/08/2022, 1:11 PM
    How many longhorn nodes do you have?
  • d

    dazzling-france-77424

    09/08/2022, 1:17 PM
    @brash-planet-10109, How many longhorn nodes do you have? If you have less than 3, Longhorn defaults replicas to 3, so you would need to update the replica count on the volumes, and change the default replicas count in Longhorn Settings
  • b

    brash-planet-10109

    09/08/2022, 1:33 PM
    @dazzling-france-77424 I've updated longhorn storage class configmap. Changed value from 3 to 6 and restarted all the pods in longhorn-system namespace. But still PVCs are in degraded state. It's a small 3 node cluster, with one master and two worker nodes
    d
    • 2
    • 6
  • b

    bright-fireman-42144

    09/08/2022, 3:01 PM
    ok.... need a sanity check. running rancher standalone right now which I should probably properly deploy 'in' my cluster (not best practice... but shrug) It is 2.6.6 and the local k3s is v1.23.6+k3s1 my downstream cluster is on GKE and is at v1.23.8-gke.1900. I would like to upgrade to RKE2. ignoring my HA issues with Rancher (I'll figure out that with the infamous, 'later')... what is the best upgrade path?
    • 1
    • 1
  • s

    square-dream-42632

    09/08/2022, 3:50 PM
    Hello, Is there :) Is it finally possible to deploy rke2 on hetzner through rancher? I think hetzner is a big enough cloud provider to be supported natively
  • b

    boundless-dog-9864

    09/08/2022, 5:26 PM
    Is there a document somewhere that tells us which versions of rke2 a given version of rancher supports? I mean to install rancher on… not the versions that rancher supports for management. I'm looking to grab the last meat version to install 2.6.8 on if that helps.
    c
    • 2
    • 2
  • a

    acoustic-sunset-13848

    09/08/2022, 7:15 PM
    On the Rancher Web UI. under “More Resources” there are groupings, including the groupings of CRDs. CRDs that I had myself to the cluster are put under their “group” which is domain name based, i.e.,
    <http://company.com|company.com>
    . Is there a way to provide a label to use instead the group name, so it fits better with the rest of the groupings?
    f
    c
    • 3
    • 5
  • d

    dazzling-computer-84464

    09/08/2022, 7:56 PM
    https://github.com/rancher/terraform-provider-rancher2/issues/891 are there any workarounds for this issue using Terraform Rancher provider?
  • m

    modern-dress-80156

    09/08/2022, 8:40 PM
    HI, We are trying to backup and restore our rancher (RKE2) + longhorn cluster using Velero, during restore, except rancher-server everything is coming up fine, rancher-server is always crashing with this error “”invalid memory address or nil pointer dereference” and we tried various combinations of restore but so far nothing worked, we wrote a script to un-install rancher and re-install but rancher gets bundled with tons of CRD’s and the cleanup script itself take 20 minutes to cleanup all the CRD’s. We want to avoid the necessity of plugging the cleanup script. So, Any idea how can we get rid of this error or what’s causing this ?
    2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=Cluster controller
    2022/06/20 16:15:13 [INFO] Watching metadata for <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineSet
    2022/06/20 16:15:13 [INFO] Watching metadata for <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineHealthCheck
    2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineHealthCheck controller
    2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineSet controller
    2022/06/20 16:15:14 [ERROR] error syncing 'git-webhook': handler apiservice: failed to update cattle-system/git-webhook-api-service /v1, Kind=ServiceAccount for apiservice git-webhook: ServiceAccount "git-webhook-api-service" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing
    2022/06/20 16:15:14 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label <http://field.cattle.io/projectId=p-bcdbv|field.cattle.io/projectId=p-bcdbv> to namespace=kube-system
    2022/06/20 16:15:15 [INFO] Updating global catalog library
    2022/06/20 16:15:15 [ERROR] error syncing 'git-webhook': handler apiservice: failed to update cattle-system/git-webhook-api-service /v1, Kind=ServiceAccount for apiservice git-webhook: ServiceAccount "git-webhook-api-service" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing
    2022/06/20 16:15:17 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:35675
    2022/06/20 16:15:17 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:41777
    2022/06/20 16:15:17 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:45839
    E0620 16:15:17.118103      34 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
    goroutine 3105 [running]:
    <http://k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39b9940|k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39b9940>, 0x6920620)
    	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:74 +0x95
    <http://k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0|k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0>, 0x0, 0x0)
    	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:48 +0x86
    panic(0x39b9940, 0x6920620)
    	/usr/local/go/src/runtime/panic.go:965 +0x1b9
    <http://google.golang.org/grpc.(*Server).Stop(0x0)|google.golang.org/grpc.(*Server).Stop(0x0)>
    	/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1482 +0x4a
    <http://github.com/rancher/rancher/pkg/kontainer-engine/types.(*GrpcServer).Stop(...)|github.com/rancher/rancher/pkg/kontainer-engine/types.(*GrpcServer).Stop(...)>
    	/go/src/github.com/rancher/rancher/pkg/kontainer-engine/types/rpc_server.go:157
    <http://github.com/rancher/rancher/pkg/kontainer-engine/service.(*RunningDriver).Stop(0xc00bafcce0)|github.com/rancher/rancher/pkg/kontainer-engine/service.(*RunningDriver).Stop(0xc00bafcce0)>
    	/go/src/github.com/rancher/rancher/pkg/kontainer-engine/service/service.go:491 +0x45
    <http://github.com/rancher/rancher/pkg/kontainer-engine/service.(*EngineService).GetDriverCreateOptions(0xc00bafcf90|github.com/rancher/rancher/pkg/kontainer-engine/service.(*EngineService).GetDriverCreateOptions(0xc00bafcf90>, 0x48ec250, 0xc000060060, 0xc00e1badf8, 0x16, 0xc012c5a000, 0x0, 0x0, 0x0, 0x0, ...)
    	/go/src/github.com/rancher/rancher/pkg/kontainer-engine/service/service.go:331 +0x277
    <http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).getResourceFields(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).getResourceFields(0xc00cf15e80>, 0xc012c5a000, 0x0, 0x0, 0x0)
    	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:182 +0x245
    <http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createDynamicSchema(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createDynamicSchema(0xc00cf15e80>, 0xc012c5a000, 0x418235f, 0x8)
    	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:115 +0x3f
    <http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createOrUpdateDynamicSchema(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createOrUpdateDynamicSchema(0xc00cf15e80>, 0xc012c5a000, 0x6, 0x417fb4d)
    	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:149 +0x1a5
    <http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).Updated.func2(0x40d9ee0|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).Updated.func2(0x40d9ee0>, 0xc012c5a000, 0x417cb5d, 0x6)
    	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:404 +0x3b
    <http://github.com/rancher/norman/condition.Cond.doInternal(0x417cb5d|github.com/rancher/norman/condition.Cond.doInternal(0x417cb5d>, 0x6, 0x489fe01, 0x489fee0, 0xc012c5a000, 0xc00b37b6c0, 0x197, 0x417d1db, 0x6, 0x4187626)
    	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/condition/condition.go:212 +0x9d
    <http://github.com/rancher/norman/condition.Cond.do2(0x417cb5d|github.com/rancher/norman/condition.Cond.do2(0x417cb5d>, 0x6, 0x417cb01, 0x489fee0, 0xc012c5a000, 0xc00b37b6c0, 0x378e9c0, 0xc001c08010, 0x198, 0x417d1db, ...)
    	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/condition/condition.go:181 +0x1dd
    <http://github.com/rancher/norman/condition.Cond.do(...)|github.com/rancher/norman/condition.Cond.do(...)>
    b
    c
    f
    • 4
    • 7
  • b

    brash-school-7230

    09/09/2022, 7:34 AM
    Hello, rancher is executing endless number of apps on the server after deploy to k8s. Is there any way to stop it?
    a
    • 2
    • 5
  • w

    worried-rain-56725

    09/09/2022, 9:31 AM
    Rancher version: 2.6.8
    Installation method: Helm
    Helm repo: <https://releases.rancher.com/server-charts/stable>
    After installing rancher, I tried to bootstrap it, using rancher terraform provider and next tf configuration:
    resource "rancher2_bootstrap" "admin" {
      provider         = rancher2.bootstrap
      initial_password = "Password1"
      password         = local.rancher_bootstrap_password
      telemetry        = false
    }
    For some reason after few minutes, creating of this process fails with the following error log:
    │ **Error:** **[ERROR] Updating token: Bad response statusCode [403]. Status [403 Forbidden]. Body: [baseType=error, code=Forbidden, message=<http://settings.management.cattle.io|settings.management.cattle.io> "k8s-version" is forbidden: User "user-bfkj6" cannot get resource "settings" in API group "<http://management.cattle.io|management.cattle.io>" at the cluster scope] from [<https://rancher-internal.foo.bar.com/v3/settings/k8s-version]**>
    
    │
    
    │ with rancher2_bootstrap.admin,
    
    │ on <http://main.tf|main.tf> line 48, in resource "rancher2_bootstrap" "admin":
    
    │ 48: resource "rancher2_bootstrap" "admin" {
    That user has GlobalBindingRole to admin role, so it should has access to Setting CRD. Also there are a lot of such error messages in rancher pod:
    2022/09/09 09:07:41 [ERROR] Failed to connect to peer <wss://10.0.3.184/v3/connect> [local ID=10.0.1.59]: websocket: bad handshake
    I’ve also tried to bootstrap rancher via UI, but after entering bootstrap password the only thing I see is the white screen. Any ideas?
  • s

    salmon-carpenter-62625

    09/09/2022, 12:54 PM
    Hello. Any clue why `rancher2_cluster_sync.cluster_sync: make destruction complete after 0s? I need it to wait till real cluster is removed from cloud in my case azure aks. Otherwise after sync completed the terraform remove in my case a cloud_credential then probably this lead to race condition where rancher cluster is removed from rancher ui but stay intact on aks side.
    l
    • 2
    • 1
  • s

    stale-dinner-99388

    09/09/2022, 1:44 PM
    Hey everyone is there any way i can pass rancher username at the time of installation. I am using helm chart for install rancher. Command:
    helm upgrade --install rancher rancher-latest/rancher   --namespace cattle-system   --set hostname=url   --set ingress.tls.source="letsEncrypt" --set bootstrapPassword=abcdxyz  --set letsEncrypt.email="email"   --set letsEncrypt.environment="production"
  • b

    bright-fireman-42144

    09/09/2022, 2:06 PM
    new problem while I am trying to manage my gke cluster... unfortunately many operations were performed. Initially I was trying to deploy minio but had some constraints on minimum nodes and memory. I discovered my cluster is only RKE in the process of trying to do a rolling upgrade of the node pool with higher memory nodes (didn't get that far) then I tried to at least get the cluster up to RKE2 which didn't work (didn't even get as far as it running the plan, I don't think). Now that cluster is in a pending state. When running through the troubleshooting steps I see:
    kubectl -n cattle-system get pods -l app=cattle-agent -o wide
    No resources found.
    but I do see cattle-cluster-agent
  • b

    bright-fireman-42144

    09/09/2022, 2:06 PM
    just how borked am I now?
  • c

    clever-air-65544

    09/09/2022, 6:39 PM
    Hello! k3s weekly report is ready: https://github.com/k3s-io/k3s/discussions/6116
  • f

    flat-balloon-88238

    09/10/2022, 4:42 AM
    Hi everyone, Does anyone know how to use the container exec function on the latest Rancher version?
  • a

    alert-fireman-21958

    09/10/2022, 7:47 AM
    hi team i am unable import cluster in my rancher dashboard
    👀 1
  • a

    alert-fireman-21958

    09/10/2022, 7:47 AM
    error: resource mapping not found for name: "cattle-admin-binding" namespace: "cattle-system" from "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"
    t
    • 2
    • 1
  • a

    alert-fireman-21958

    09/10/2022, 7:48 AM
    can any one help me on this issue
  • f

    faint-thailand-7471

    09/10/2022, 8:00 AM
    Hello everyone. I am using Rancher Dekstop 1.5.1 on Macosx. Trying to use testcontainers 1.17.3, I keep getting a connection error:
    [INFO] Creating container for image: testcontainers/ryuk:0.3.3
    [INFO] Credential helper/store (docker-credential-osxkeychain) does not have credentials for <https://index.docker.io/v1/>
    [INFO] Container testcontainers/ryuk:0.3.3 is starting: 66a6f578877a5625474812c4587b3910d1e3da7466ea96336516e43aacd42c64
    [INFO] Container testcontainers/ryuk:0.3.3 started in PT0.544596S
    [WARNING] Can not connect to Ryuk at localhost:49153
    java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.pollConnect (Native Method)
        at sun.nio.ch.Net.pollConnectNow (Net.java:672)
        at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:542)
        at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
        at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
        at java.net.Socket.connect (Socket.java:633)
        at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
        at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
        at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
        at java.lang.Thread.run (Thread.java:833)
    I have seen a few discussion around the same problem for people using Docker Desktop. Has someone had/solved the same problem with Rancher? Any help is highly apprechiated. Thanks so much.
  • c

    clean-butcher-81304

    09/10/2022, 11:51 PM
    👋 Hi everyone!
  • b

    best-microphone-20624

    09/11/2022, 6:13 AM
    Greetings! Is it supported to install Rancher on an existing Rancher-managed downstream cluster? If not, is there an open issue for this feature request?
    c
    • 2
    • 2
  • f

    faint-airport-18378

    09/11/2022, 9:58 AM
    hello. i'm using v1.6.30 and i'm interested in understanding the meaning of the resource limit in the Infrastructure > Host > Edit screen. are these limits for the entire host, or per container?
    f
    • 2
    • 1
  • h

    hundreds-terabyte-36933

    09/11/2022, 3:16 PM
    https://github.com/rancher/rancher/issues/24918
Powered by Linen
Title
h

hundreds-terabyte-36933

09/11/2022, 3:16 PM
https://github.com/rancher/rancher/issues/24918
View count: 17