rancher-users #general

ancient-energy-15842

08/30/2022, 11:31 PM

Hi, after trying to upgrade one of my Rancher clusters from 2.6.6 to 2.6.7 or 2.6.8 I get this error for only one of the two clusters that are running Both clusters were created with RKE1 and are running in EC2 with RancherOS

[ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-4p8gk: ClusterUnavailable 503: cluster not found, failed to start user controllers for cluster c-qgw7q: ClusterUnavailable 503: cluster not found, requeuing

ancient-energy-15842

08/30/2022, 11:31 PM

the other cluster doesn't throw any errors

little-accountant-66169

08/31/2022, 12:28 AM

👋 Hi everyone!

little-accountant-66169

08/31/2022, 12:29 AM

just installed rancher desktop. Tried running my compose files after installing, but I get error

time="2022-08-31T09:23:43+09:00" level=fatal msg="cannot access containerd socket \"\\\\\\\\.\\\\pipe\\\\containerd-containerd\": open \\\\.\\pipe\\containerd-containerd: The system cannot find the file specified."

big-architect-39828

08/31/2022, 1:44 AM

k3s on ubuntu 20.04 arm64 failed to use docker, anybody help me

big-architect-39828

08/31/2022, 1:44 AM

https://gist.github.com/lazywhite/52ee1643747555643c4e06519106c798

quiet-insurance-89872

08/31/2022, 2:30 AM

Hello everyone My password is changing whenever Rancher pod restart is there any way to persist the password so after restart it won't change??

quiet-football-25667

08/31/2022, 3:36 AM

Hello everyone, can someone help me Integrate Rancher 2.6 with Onelogin, as i can see under Auth Provider Okta listed but Onelogin we required

stale-dinner-99388

08/31/2022, 5:47 AM

bright-whale-83501

08/31/2022, 7:18 AM

My RKE cluster which I run Rancher inside is k8s 1.19. Is there any upgrade method using RKE CLI? Like editing the config file or something?

echoing-animal-32877

08/31/2022, 9:18 AM

👋 Hi everyone!

chilly-caravan-31853

08/31/2022, 10:30 AM

Hello Colleagues! I've created a HA cluster of 4 nodes with the PostgreSQL datastore. I'm not planning to add agent nodes, just want to run all payload on the master nodes. I've noticed however that a 4-node cluster is running only one coredns pod, one local-path-provisioner pod etc. What if a node where they are running fails? Should I 1) scale these Deployments up to 3-4 pods 2) run the kube-system pods as DaemonSets instead of Deployments somehow 3) leave everything as it is and wait 5 minutes for the system pods to be rescheduled to a healthy node?

witty-balloon-26464

08/31/2022, 10:38 AM

Hey folks, I'd like to change some option for containerd and documentation states that some config should be generated in /var/lib/rancher/k3s/agent/etc/containerd/config.toml but I have no such file. Is https://rancher.com/docs/k3s/latest/en/advanced/#configuring-containerd outdated?

witty-balloon-26464

08/31/2022, 10:44 AM

Where is containerd config stored?

thankful-area-70093

08/31/2022, 11:28 AM

Hi! I am on rancher 2.5.5 and do get the issue that can be seen in the screenshot. I tried the rotation of the certificates within the UI, but that does not work. I tried it that way: https://rancher.com/docs/rancher/v2.5/en/troubleshooting/expired-webhook-certificates/ but there is no such a secret. Rancher was installed with:

docker run -d --name=rancher --restart=unless-stopped -p 80:80 -p 443:443 -v /root/rancher:/var/lib/rancher --privileged rancher/rancher:v2.5.5 --acme-domain <http://rancher.xxx|rancher.xxx>

. I wanted to connect to the docker rancher/rancher container, but there is none. Only rancher/rancher-agent exists on my worker nodes, but on my master node there is only rancher/hyperkube:v1.19.6 etc. I have read through all of these github issues, but cannot get it solved.

curved-baker-5639

08/31/2022, 12:08 PM

Hi, I updated from rancher 2.6.4 to 2.6.8 and users are unable to deploy applications from the rancher api, they get the following message:

Copy code

Rancher faced an issue (CONFLICT) during the deployment of server: [...] info: an <http://apps.catalog.cattle.io|apps.catalog.cattle.io> already exists with ID [...] in namespace [...]"}

I think the bit up to “info” comes from their code, so it’s just expressing a 409 I guess. I checked the app.catalog.cattle.io crd and it exists. What’s interesting is when I visit

v3/project/[project]/apps

I see a list of apps, but when they try to upgrade one, they get that error and the app disappears from that list

many-area-51777

08/31/2022, 12:19 PM

Hey, just upgrade to 2.6.8 and rancher is not coming up anymore. First I had to extend the liveness settings as mentioned in https://github.com/rancher/rancher/issues/38177 but even though the logs show that something is happening, at the end the pod dies and tries to restart.

witty-balloon-26464

08/31/2022, 3:30 PM

anyone from k3s team is reading issues that show up here? or is it better to simply submit bugreport on github?

brash-monitor-41966

08/31/2022, 4:27 PM

does anyone know how to get ingress/loadbalancer working with GCP with RKE cluster?

brash-monitor-41966

08/31/2022, 4:29 PM

when i expose my deployment as a loadbalancer, it keeps saying IP PENDING

best-oil-69865

08/31/2022, 6:04 PM

I am trying to upgrade my rancher from 2.6.1 to 2.6.7 but after upgrade I keep getting below error

Copy code

current-db-size":"11 MB","current-db-size-in-use-bytes":11010048,"current-db-size-in-use":"11 MB"}
{"level":"info","ts":"2022-08-31T18:03:31.302Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes-diff":0,"current-db-size-bytes":11018240,"current-db-size":"11 MB","current-db-size-in-use-bytes-diff":0,"current-db-size-in-use-bytes":11010048,"current-db-size-in-use":"11 MB","took":"283.469846ms"}
{"level":"info","ts":"2022-08-31T18:03:31.303Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
time="2022-08-31T18:03:31Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [e79044ef3c73-b6f3123a=<http://localhost:2380>], expect: e79044ef3c73-b6f3123a=172.17.0.2"
time="2022-08-31T18:03:32Z" level=error msg="CA cert validation failed: Get \"<https://127.0.0.1:6444/cacerts>\": x509: certificate signed by unknown authority"
time="2022-08-31T18:03:33Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2022-08-31T18:03:34Z" level=error msg="CA cert validation failed: Get \"<https://127.0.0.1:6444/cacerts>\": x509: certificate signed by unknown authority"
time="2022-08-31T18:03:36Z" level=error msg="CA cert validation failed: Get \"<https://127.0.0.1:6444/cacerts>\": x509: certificate signed by unknown authority"
time="2022-08-31T18:03:36Z" level=info msg="Defragmenting etcd database"
{"level":"info","ts":"2022-08-31T18:03:36.635Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2022-08-31T18:03:36.639Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/k3s/server/db/etcd/member/snap/db","current-db-size-bytes":11018240,"current-db-size":"11 MB","current-db-size-in-use-bytes":11010048,"current-db-size-in-use":"11 MB"}

best-oil-69865

08/31/2022, 6:04 PM

in docker log I could see still as

Copy code

INFO: Running k3s server --cluster-init --cluster-reset

best-oil-69865

08/31/2022, 6:05 PM

in the k3s-cluster-reset.log I keep seeing the above error log

best-oil-69865

08/31/2022, 6:05 PM

will it come or is it kindly of failure

best-oil-69865

08/31/2022, 6:05 PM

can someone help on this I keep struggling for past two days to upgrade my production upgrade

best-oil-69865

08/31/2022, 6:07 PM

is it possible first to upgrade rancher from 2.6.1 to 2.6.7

best-oil-69865

08/31/2022, 6:46 PM

Copy code

2022/08/31 18:45:24 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:24 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update user: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:29 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:29 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update restricted-admin: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:33 [ERROR] error syncing 'fleet-default/c-g5d8r': handler rke-cluster-remove: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority, requeuing
2022/08/31 18:45:34 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:34 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update roles-manage: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:39 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:39 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update roles-manage: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:42 [ERROR] error syncing 'c-g5d8r': handler provisioning-cluster-create: failed to update fleet-default/c-g5d8r <http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>, Kind=Cluster for provisioning-cluster-create c-g5d8r: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority, requeuing
2022/08/31 18:45:44 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:44 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update user-base: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:49 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:49 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update user-base: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:54 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:54 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update restricted-admin: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority
2022/08/31 18:45:59 [INFO] Reconciling GlobalRoles
2022/08/31 18:45:59 [ERROR] failed to call leader func: failed to add management data: problem reconciling global roles: couldn't update user-base: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority

2022/08/31 18:46:01 [ERROR] error syncing 'fleet-local/local': handler rke-cluster-remove: Internal error occurred: failed calling webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": x509: certificate signed by unknown authority, requeuing

white-yacht-60472

08/31/2022, 6:54 PM

Hi there, I have completed my submission to the Red Hat certificate authority today and I would like to begin k3s or k8s, which would you suggest to begin and is there a good link to share?

powerful-kangaroo-7234

08/31/2022, 7:12 PM

What's up guys, I'm currently doing a test lab but when installing rancher server the URL doesn't work for me. does anyone know why?

busy-easter-55684

09/01/2022, 2:06 AM

#k3s, does anyone have good example to start k3s by docker? I try a lot of times, but failed. Here is my command: