docker is not able to pull the images ..?

Hello. someone here delete a namespace cattle-system with cluster agents and now the rancher show

Unavailable

in his status. I found site https://rancher.com/docs/rancher/v2.6/en/installation/resources/update-rancher-cert/#method-3-recreate-rancher-agents that show some tips how to recreate it but, the point is that from what I read it says about executing something on control-plan nodes, In my case I use AKS clusters so have just the workload nodes. Any tips what I can do to restore the rancher agent on such scenario (I don't have etcd backup)?

Found a workaround here: https://github.com/rancher/rancher/issues/38575 https://github.com/rancher/dashboard/issues/6122#issuecomment-1152394027

Can you create a RKE2 Downstream Cluster via Rancher UI ?

Hi All, Longhorn volumes are in degraded state, however pods are up and PVCs got attached to it. I'm I doing something wrong? Rke version: 1.23.10+rke2r1 Longhorn version: v1.3.1

How many longhorn nodes do you have?

@brash-planet-10109, How many longhorn nodes do you have? If you have less than 3, Longhorn defaults replicas to 3, so you would need to update the replica count on the volumes, and change the default replicas count in Longhorn Settings

@dazzling-france-77424 I've updated longhorn storage class configmap. Changed value from 3 to 6 and restarted all the pods in longhorn-system namespace. But still PVCs are in degraded state. It's a small 3 node cluster, with one master and two worker nodes

ok.... need a sanity check. running rancher standalone right now which I should probably properly deploy 'in' my cluster (not best practice... but shrug) It is 2.6.6 and the local k3s is v1.23.6+k3s1 my downstream cluster is on GKE and is at v1.23.8-gke.1900. I would like to upgrade to RKE2. ignoring my HA issues with Rancher (I'll figure out that with the infamous, 'later')... what is the best upgrade path?

Hello, Is there :) Is it finally possible to deploy rke2 on hetzner through rancher? I think hetzner is a big enough cloud provider to be supported natively

Is there a document somewhere that tells us which versions of rke2 a given version of rancher supports? I mean to install rancher on… not the versions that rancher supports for management. I'm looking to grab the last meat version to install 2.6.8 on if that helps.

On the Rancher Web UI. under “More Resources” there are groupings, including the groupings of CRDs. CRDs that I had myself to the cluster are put under their “group” which is domain name based, i.e.,

<http://company.com|company.com>

. Is there a way to provide a label to use instead the group name, so it fits better with the rest of the groupings?

https://github.com/rancher/terraform-provider-rancher2/issues/891 are there any workarounds for this issue using Terraform Rancher provider?

HI, We are trying to backup and restore our rancher (RKE2) + longhorn cluster using Velero, during restore, except rancher-server everything is coming up fine, rancher-server is always crashing with this error “”invalid memory address or nil pointer dereference” and we tried various combinations of restore but so far nothing worked, we wrote a script to un-install rancher and re-install but rancher gets bundled with tons of CRD’s and the cleanup script itself take 20 minutes to cleanup all the CRD’s. We want to avoid the necessity of plugging the cleanup script. So, Any idea how can we get rid of this error or what’s causing this ?

2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=Cluster controller
2022/06/20 16:15:13 [INFO] Watching metadata for <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineSet
2022/06/20 16:15:13 [INFO] Watching metadata for <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineHealthCheck
2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineHealthCheck controller
2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineSet controller
2022/06/20 16:15:14 [ERROR] error syncing 'git-webhook': handler apiservice: failed to update cattle-system/git-webhook-api-service /v1, Kind=ServiceAccount for apiservice git-webhook: ServiceAccount "git-webhook-api-service" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing
2022/06/20 16:15:14 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label <http://field.cattle.io/projectId=p-bcdbv|field.cattle.io/projectId=p-bcdbv> to namespace=kube-system
2022/06/20 16:15:15 [INFO] Updating global catalog library
2022/06/20 16:15:15 [ERROR] error syncing 'git-webhook': handler apiservice: failed to update cattle-system/git-webhook-api-service /v1, Kind=ServiceAccount for apiservice git-webhook: ServiceAccount "git-webhook-api-service" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing
2022/06/20 16:15:17 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:35675
2022/06/20 16:15:17 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:41777
2022/06/20 16:15:17 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:45839
E0620 16:15:17.118103      34 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 3105 [running]:
<http://k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39b9940|k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39b9940>, 0x6920620)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:74 +0x95
<http://k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0|k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0>, 0x0, 0x0)
	/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:48 +0x86
panic(0x39b9940, 0x6920620)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
<http://google.golang.org/grpc.(*Server).Stop(0x0)|google.golang.org/grpc.(*Server).Stop(0x0)>
	/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1482 +0x4a
<http://github.com/rancher/rancher/pkg/kontainer-engine/types.(*GrpcServer).Stop(...)|github.com/rancher/rancher/pkg/kontainer-engine/types.(*GrpcServer).Stop(...)>
	/go/src/github.com/rancher/rancher/pkg/kontainer-engine/types/rpc_server.go:157
<http://github.com/rancher/rancher/pkg/kontainer-engine/service.(*RunningDriver).Stop(0xc00bafcce0)|github.com/rancher/rancher/pkg/kontainer-engine/service.(*RunningDriver).Stop(0xc00bafcce0)>
	/go/src/github.com/rancher/rancher/pkg/kontainer-engine/service/service.go:491 +0x45
<http://github.com/rancher/rancher/pkg/kontainer-engine/service.(*EngineService).GetDriverCreateOptions(0xc00bafcf90|github.com/rancher/rancher/pkg/kontainer-engine/service.(*EngineService).GetDriverCreateOptions(0xc00bafcf90>, 0x48ec250, 0xc000060060, 0xc00e1badf8, 0x16, 0xc012c5a000, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/rancher/rancher/pkg/kontainer-engine/service/service.go:331 +0x277
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).getResourceFields(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).getResourceFields(0xc00cf15e80>, 0xc012c5a000, 0x0, 0x0, 0x0)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:182 +0x245
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createDynamicSchema(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createDynamicSchema(0xc00cf15e80>, 0xc012c5a000, 0x418235f, 0x8)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:115 +0x3f
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createOrUpdateDynamicSchema(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createOrUpdateDynamicSchema(0xc00cf15e80>, 0xc012c5a000, 0x6, 0x417fb4d)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:149 +0x1a5
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).Updated.func2(0x40d9ee0|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).Updated.func2(0x40d9ee0>, 0xc012c5a000, 0x417cb5d, 0x6)
	/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:404 +0x3b
<http://github.com/rancher/norman/condition.Cond.doInternal(0x417cb5d|github.com/rancher/norman/condition.Cond.doInternal(0x417cb5d>, 0x6, 0x489fe01, 0x489fee0, 0xc012c5a000, 0xc00b37b6c0, 0x197, 0x417d1db, 0x6, 0x4187626)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/condition/condition.go:212 +0x9d
<http://github.com/rancher/norman/condition.Cond.do2(0x417cb5d|github.com/rancher/norman/condition.Cond.do2(0x417cb5d>, 0x6, 0x417cb01, 0x489fee0, 0xc012c5a000, 0xc00b37b6c0, 0x378e9c0, 0xc001c08010, 0x198, 0x417d1db, ...)
	/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/condition/condition.go:181 +0x1dd
<http://github.com/rancher/norman/condition.Cond.do(...)|github.com/rancher/norman/condition.Cond.do(...)>

Hello, rancher is executing endless number of apps on the server after deploy to k8s. Is there any way to stop it?

Rancher version: 2.6.8
Installation method: Helm
Helm repo: <https://releases.rancher.com/server-charts/stable>

After installing rancher, I tried to bootstrap it, using rancher terraform provider and next tf configuration:

resource "rancher2_bootstrap" "admin" {
  provider         = rancher2.bootstrap
  initial_password = "Password1"
  password         = local.rancher_bootstrap_password
  telemetry        = false
}

For some reason after few minutes, creating of this process fails with the following error log:

│ **Error:** **[ERROR] Updating token: Bad response statusCode [403]. Status [403 Forbidden]. Body: [baseType=error, code=Forbidden, message=<http://settings.management.cattle.io|settings.management.cattle.io> "k8s-version" is forbidden: User "user-bfkj6" cannot get resource "settings" in API group "<http://management.cattle.io|management.cattle.io>" at the cluster scope] from [<https://rancher-internal.foo.bar.com/v3/settings/k8s-version]**>

│

│ with rancher2_bootstrap.admin,

│ on <http://main.tf|main.tf> line 48, in resource "rancher2_bootstrap" "admin":

│ 48: resource "rancher2_bootstrap" "admin" {

That user has GlobalBindingRole to admin role, so it should has access to Setting CRD. Also there are a lot of such error messages in rancher pod:

2022/09/09 09:07:41 [ERROR] Failed to connect to peer <wss://10.0.3.184/v3/connect> [local ID=10.0.1.59]: websocket: bad handshake

I’ve also tried to bootstrap rancher via UI, but after entering bootstrap password the only thing I see is the white screen. Any ideas?

Hello. Any clue why `rancher2_cluster_sync.cluster_sync: make destruction complete after 0s? I need it to wait till real cluster is removed from cloud in my case azure aks. Otherwise after sync completed the terraform remove in my case a cloud_credential then probably this lead to race condition where rancher cluster is removed from rancher ui but stay intact on aks side.

Hey everyone is there any way i can pass rancher username at the time of installation. I am using helm chart for install rancher. Command:

helm upgrade --install rancher rancher-latest/rancher   --namespace cattle-system   --set hostname=url   --set ingress.tls.source="letsEncrypt" --set bootstrapPassword=abcdxyz  --set letsEncrypt.email="email"   --set letsEncrypt.environment="production"

new problem while I am trying to manage my gke cluster... unfortunately many operations were performed. Initially I was trying to deploy minio but had some constraints on minimum nodes and memory. I discovered my cluster is only RKE in the process of trying to do a rolling upgrade of the node pool with higher memory nodes (didn't get that far) then I tried to at least get the cluster up to RKE2 which didn't work (didn't even get as far as it running the plan, I don't think). Now that cluster is in a pending state. When running through the troubleshooting steps I see:

kubectl -n cattle-system get pods -l app=cattle-agent -o wide
No resources found.

but I do see cattle-cluster-agent

just how borked am I now?

Hello! k3s weekly report is ready: https://github.com/k3s-io/k3s/discussions/6116

Hi everyone, Does anyone know how to use the container exec function on the latest Rancher version?

hi team i am unable import cluster in my rancher dashboard

error: resource mapping not found for name: "cattle-admin-binding" namespace: "cattle-system" from "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"

can any one help me on this issue

Hello everyone. I am using Rancher Dekstop 1.5.1 on Macosx. Trying to use testcontainers 1.17.3, I keep getting a connection error:

[INFO] Creating container for image: testcontainers/ryuk:0.3.3
[INFO] Credential helper/store (docker-credential-osxkeychain) does not have credentials for <https://index.docker.io/v1/>
[INFO] Container testcontainers/ryuk:0.3.3 is starting: 66a6f578877a5625474812c4587b3910d1e3da7466ea96336516e43aacd42c64
[INFO] Container testcontainers/ryuk:0.3.3 started in PT0.544596S
[WARNING] Can not connect to Ryuk at localhost:49153
java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.pollConnect (Native Method)
    at sun.nio.ch.Net.pollConnectNow (Net.java:672)
    at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:542)
    at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
    at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
    at java.net.Socket.connect (Socket.java:633)
    at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
    at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
    at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
    at java.lang.Thread.run (Thread.java:833)

I have seen a few discussion around the same problem for people using Docker Desktop. Has someone had/solved the same problem with Rancher? Any help is highly apprechiated. Thanks so much.

👋 Hi everyone!

Greetings! Is it supported to install Rancher on an existing Rancher-managed downstream cluster? If not, is there an open issue for this feature request?

hello. i'm using v1.6.30 and i'm interested in understanding the meaning of the resource limit in the Infrastructure > Host > Edit screen. are these limits for the entire host, or per container?