salmon-carpenter-62625
09/08/2022, 7:44 AMUnavailable
in his status.
I found site https://rancher.com/docs/rancher/v2.6/en/installation/resources/update-rancher-cert/#method-3-recreate-rancher-agents that show some tips how to recreate it but, the point is that from what I read it says about executing something on control-plan nodes,
In my case I use AKS clusters so have just the workload nodes. Any tips what I can do to restore the rancher agent on such scenario (I don't have etcd backup)?eager-vr-25932
09/08/2022, 10:25 AMpurple-translator-99032
09/08/2022, 12:06 PMbrash-planet-10109
09/08/2022, 12:50 PMdazzling-france-77424
09/08/2022, 1:11 PMdazzling-france-77424
09/08/2022, 1:17 PMbrash-planet-10109
09/08/2022, 1:33 PMbright-fireman-42144
09/08/2022, 3:01 PMsquare-dream-42632
09/08/2022, 3:50 PMboundless-dog-9864
09/08/2022, 5:26 PMacoustic-sunset-13848
09/08/2022, 7:15 PM<http://company.com|company.com>
. Is there a way to provide a label to use instead the group name, so it fits better with the rest of the groupings?dazzling-computer-84464
09/08/2022, 7:56 PMmodern-dress-80156
09/08/2022, 8:40 PM2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=Cluster controller
2022/06/20 16:15:13 [INFO] Watching metadata for <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineSet
2022/06/20 16:15:13 [INFO] Watching metadata for <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineHealthCheck
2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineHealthCheck controller
2022/06/20 16:15:13 [INFO] Starting <http://cluster.x-k8s.io/v1alpha3|cluster.x-k8s.io/v1alpha3>, Kind=MachineSet controller
2022/06/20 16:15:14 [ERROR] error syncing 'git-webhook': handler apiservice: failed to update cattle-system/git-webhook-api-service /v1, Kind=ServiceAccount for apiservice git-webhook: ServiceAccount "git-webhook-api-service" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing
2022/06/20 16:15:14 [INFO] namespaceHandler: addProjectIDLabelToNamespace: adding label <http://field.cattle.io/projectId=p-bcdbv|field.cattle.io/projectId=p-bcdbv> to namespace=kube-system
2022/06/20 16:15:15 [INFO] Updating global catalog library
2022/06/20 16:15:15 [ERROR] error syncing 'git-webhook': handler apiservice: failed to update cattle-system/git-webhook-api-service /v1, Kind=ServiceAccount for apiservice git-webhook: ServiceAccount "git-webhook-api-service" is invalid: [metadata.ownerReferences.apiVersion: Invalid value: "": version must not be empty, metadata.ownerReferences.kind: Invalid value: "": kind must not be empty, metadata.ownerReferences.name: Invalid value: "": name must not be empty], requeuing
2022/06/20 16:15:17 [INFO] kontainerdriver azurekubernetesservice listening on address 127.0.0.1:35675
2022/06/20 16:15:17 [INFO] kontainerdriver googlekubernetesengine listening on address 127.0.0.1:41777
2022/06/20 16:15:17 [INFO] kontainerdriver amazonelasticcontainerservice listening on address 127.0.0.1:45839
E0620 16:15:17.118103 34 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 3105 [running]:
<http://k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39b9940|k8s.io/apimachinery/pkg/util/runtime.logPanic(0x39b9940>, 0x6920620)
/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:74 +0x95
<http://k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0|k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0>, 0x0, 0x0)
/go/pkg/mod/k8s.io/apimachinery@v0.21.0/pkg/util/runtime/runtime.go:48 +0x86
panic(0x39b9940, 0x6920620)
/usr/local/go/src/runtime/panic.go:965 +0x1b9
<http://google.golang.org/grpc.(*Server).Stop(0x0)|google.golang.org/grpc.(*Server).Stop(0x0)>
/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1482 +0x4a
<http://github.com/rancher/rancher/pkg/kontainer-engine/types.(*GrpcServer).Stop(...)|github.com/rancher/rancher/pkg/kontainer-engine/types.(*GrpcServer).Stop(...)>
/go/src/github.com/rancher/rancher/pkg/kontainer-engine/types/rpc_server.go:157
<http://github.com/rancher/rancher/pkg/kontainer-engine/service.(*RunningDriver).Stop(0xc00bafcce0)|github.com/rancher/rancher/pkg/kontainer-engine/service.(*RunningDriver).Stop(0xc00bafcce0)>
/go/src/github.com/rancher/rancher/pkg/kontainer-engine/service/service.go:491 +0x45
<http://github.com/rancher/rancher/pkg/kontainer-engine/service.(*EngineService).GetDriverCreateOptions(0xc00bafcf90|github.com/rancher/rancher/pkg/kontainer-engine/service.(*EngineService).GetDriverCreateOptions(0xc00bafcf90>, 0x48ec250, 0xc000060060, 0xc00e1badf8, 0x16, 0xc012c5a000, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/rancher/rancher/pkg/kontainer-engine/service/service.go:331 +0x277
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).getResourceFields(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).getResourceFields(0xc00cf15e80>, 0xc012c5a000, 0x0, 0x0, 0x0)
/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:182 +0x245
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createDynamicSchema(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createDynamicSchema(0xc00cf15e80>, 0xc012c5a000, 0x418235f, 0x8)
/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:115 +0x3f
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createOrUpdateDynamicSchema(0xc00cf15e80|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).createOrUpdateDynamicSchema(0xc00cf15e80>, 0xc012c5a000, 0x6, 0x417fb4d)
/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:149 +0x1a5
<http://github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).Updated.func2(0x40d9ee0|github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver.(*Lifecycle).Updated.func2(0x40d9ee0>, 0xc012c5a000, 0x417cb5d, 0x6)
/go/src/github.com/rancher/rancher/pkg/controllers/management/drivers/kontainerdriver/kontainerdriver.go:404 +0x3b
<http://github.com/rancher/norman/condition.Cond.doInternal(0x417cb5d|github.com/rancher/norman/condition.Cond.doInternal(0x417cb5d>, 0x6, 0x489fe01, 0x489fee0, 0xc012c5a000, 0xc00b37b6c0, 0x197, 0x417d1db, 0x6, 0x4187626)
/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/condition/condition.go:212 +0x9d
<http://github.com/rancher/norman/condition.Cond.do2(0x417cb5d|github.com/rancher/norman/condition.Cond.do2(0x417cb5d>, 0x6, 0x417cb01, 0x489fee0, 0xc012c5a000, 0xc00b37b6c0, 0x378e9c0, 0xc001c08010, 0x198, 0x417d1db, ...)
/go/pkg/mod/github.com/rancher/norman@v0.0.0-20210608202517-59b3523c3133/condition/condition.go:181 +0x1dd
<http://github.com/rancher/norman/condition.Cond.do(...)|github.com/rancher/norman/condition.Cond.do(...)>
brash-school-7230
09/09/2022, 7:34 AMworried-rain-56725
09/09/2022, 9:31 AMRancher version: 2.6.8
Installation method: Helm
Helm repo: <https://releases.rancher.com/server-charts/stable>
After installing rancher, I tried to bootstrap it, using rancher terraform provider and next tf configuration:
resource "rancher2_bootstrap" "admin" {
provider = rancher2.bootstrap
initial_password = "Password1"
password = local.rancher_bootstrap_password
telemetry = false
}
For some reason after few minutes, creating of this process fails with the following error log:
│ **Error:** **[ERROR] Updating token: Bad response statusCode [403]. Status [403 Forbidden]. Body: [baseType=error, code=Forbidden, message=<http://settings.management.cattle.io|settings.management.cattle.io> "k8s-version" is forbidden: User "user-bfkj6" cannot get resource "settings" in API group "<http://management.cattle.io|management.cattle.io>" at the cluster scope] from [<https://rancher-internal.foo.bar.com/v3/settings/k8s-version]**>
│
│ with rancher2_bootstrap.admin,
│ on <http://main.tf|main.tf> line 48, in resource "rancher2_bootstrap" "admin":
│ 48: resource "rancher2_bootstrap" "admin" {
That user has GlobalBindingRole to admin role, so it should has access to Setting CRD.
Also there are a lot of such error messages in rancher pod:
2022/09/09 09:07:41 [ERROR] Failed to connect to peer <wss://10.0.3.184/v3/connect> [local ID=10.0.1.59]: websocket: bad handshake
I’ve also tried to bootstrap rancher via UI, but after entering bootstrap password the only thing I see is the white screen.
Any ideas?salmon-carpenter-62625
09/09/2022, 12:54 PMstale-dinner-99388
09/09/2022, 1:44 PMhelm upgrade --install rancher rancher-latest/rancher --namespace cattle-system --set hostname=url --set ingress.tls.source="letsEncrypt" --set bootstrapPassword=abcdxyz --set letsEncrypt.email="email" --set letsEncrypt.environment="production"
bright-fireman-42144
09/09/2022, 2:06 PMkubectl -n cattle-system get pods -l app=cattle-agent -o wide
No resources found.
but I do see cattle-cluster-agentbright-fireman-42144
09/09/2022, 2:06 PMclever-air-65544
09/09/2022, 6:39 PMflat-balloon-88238
09/10/2022, 4:42 AMalert-fireman-21958
09/10/2022, 7:47 AMalert-fireman-21958
09/10/2022, 7:47 AMalert-fireman-21958
09/10/2022, 7:48 AMfaint-thailand-7471
09/10/2022, 8:00 AM[INFO] Creating container for image: testcontainers/ryuk:0.3.3
[INFO] Credential helper/store (docker-credential-osxkeychain) does not have credentials for <https://index.docker.io/v1/>
[INFO] Container testcontainers/ryuk:0.3.3 is starting: 66a6f578877a5625474812c4587b3910d1e3da7466ea96336516e43aacd42c64
[INFO] Container testcontainers/ryuk:0.3.3 started in PT0.544596S
[WARNING] Can not connect to Ryuk at localhost:49153
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect (Native Method)
at sun.nio.ch.Net.pollConnectNow (Net.java:672)
at sun.nio.ch.NioSocketImpl.timedFinishConnect (NioSocketImpl.java:542)
at sun.nio.ch.NioSocketImpl.connect (NioSocketImpl.java:597)
at java.net.SocksSocketImpl.connect (SocksSocketImpl.java:327)
at java.net.Socket.connect (Socket.java:633)
at org.testcontainers.utility.RyukResourceReaper.lambda$null$0 (RyukResourceReaper.java:92)
at org.rnorth.ducttape.ratelimits.RateLimiter.doWhenReady (RateLimiter.java:27)
at org.testcontainers.utility.RyukResourceReaper.lambda$maybeStart$1 (RyukResourceReaper.java:88)
at java.lang.Thread.run (Thread.java:833)
I have seen a few discussion around the same problem for people using Docker Desktop. Has someone had/solved the same problem with Rancher?
Any help is highly apprechiated. Thanks so much.clean-butcher-81304
09/10/2022, 11:51 PMbest-microphone-20624
09/11/2022, 6:13 AMfaint-airport-18378
09/11/2022, 9:58 AMhundreds-terabyte-36933
09/11/2022, 3:16 PMhundreds-terabyte-36933
09/11/2022, 3:19 PM