nice-businessperson-14225
11/24/2025, 8:55 PMcareful-dress-15206
11/25/2025, 1:47 PMancient-dinner-76338
11/26/2025, 2:31 AMshy-exabyte-31531
11/27/2025, 11:33 AMrich-address-13891
11/27/2025, 12:40 PMimportant-rain-51381
11/28/2025, 8:31 AMbig-television-41348
12/01/2025, 7:25 AMhundreds-sugar-37524
12/02/2025, 1:15 PMrich-thailand-55018
12/03/2025, 1:11 PMadamant-kite-43734
12/04/2025, 3:22 AMhundreds-evening-84071
12/04/2025, 8:27 PMbrainy-architect-56011
12/05/2025, 8:58 AMgreen-activity-72286
12/05/2025, 9:35 AMcrooked-cat-21365
12/05/2025, 10:08 AMsteep-barista-45074
12/05/2025, 8:06 PMbillowy-agency-77430
12/08/2025, 8:41 AMbillowy-agency-77430
12/08/2025, 8:42 AMsalmon-plastic-18937
12/08/2025, 1:53 PMERRO[2025/12/08 13:32:29] Error restoring resource fleet-local of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=fleetworkspaces: restoreResource: err creating resource admission webhook "<http://rancher.cattle.io.fleetworkspaces.management.cattle.io|rancher.cattle.io.fleetworkspaces.management.cattle.io>" denied the request: namespace 'fleet-local' already exists
ERRO[2025/12/08 13:32:29] Error restoring cluster-scoped resources [error restoring fleet-local of type <http://management.cattle.io/v3|management.cattle.io/v3>, Resource=fleetworkspaces: restoreResource: err creating resource admission webhook "<http://rancher.cattle.io.fleetworkspaces.management.cattle.io|rancher.cattle.io.fleetworkspaces.management.cattle.io>" denied the request: namespace 'fleet-local' already exists]
ERRO[2025/12/08 13:32:29] error syncing 'pre-migration': handler restore: error restoring cluster-scoped resources, check logs for exact error, requeuing
I have tried to remove the fleet-local namespace by removing the validationwebhookconfig. Apparently, in the restore, the objects are recreated and then it loops with the error above. Is this the correct channel to ask this type of question? if no, please help me find the correct one, if yes, any pointers? 😉limited-pizza-33551
12/08/2025, 1:55 PMhundreds-evening-84071
12/08/2025, 4:01 PM/etc/rancher/rke2/registries.yaml?
it is definitely not using the registries.yaml for rancher images.
For RKE2 - it is...modern-dentist-18028
12/09/2025, 2:40 PM$ sudo kubectl exec -it -n default test-pod -- sh
error: Internal error occurred: error sending request: Post "<https://10.0.0.3:10250/exec/default/test-pod/test-pod?command=sh&input=1&output=1&tty=1>": proxy error from 127.0.0.1:6443 while dialing 10.0.0.3:10250, code 502: 502 Bad Gateway
$ sudo curl --cacert /var/lib/rancher/k3s/server/tls/server-ca.crt --cert /var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --key /var/lib/rancher/k3s/server/tls/client-kube-apiserver.key <https://10.0.0.3:10250/healthz>
ok
$ kubectl describe pod test-pod | grep node
Node: k3s-node/10.0.0.3
Error from logs
"Unhandled Error" err="apiserver received an error that is not an metav1.Status: &url.Error{Op:\"Get\", URL:\"<https://10.0.0.3:10250/containerLogs/default/test-pod/test-pod>\", Err:(*errors.errorString)(0xc02cb1ef40)}: Get \"<https://10.0.0.3:10250/containerLogs/default/test-pod/test-pod>\": proxy error from 127.0.0.1:6443 while dialing 10.0.0.3:10250, code 502: 502 Bad Gateway"
So network connection seems to be working fine. Both nodes are pingable from each other
Any suggestions what can be an issue? I'm out of ideas right nownutritious-insurance-81723
12/09/2025, 11:27 PMadamant-kite-43734
12/10/2025, 8:07 PMsalmon-plastic-18937
12/11/2025, 6:43 AM- lastTransitionTime: "2025-12-09T20:46:31Z"
message: |-
* Machines rancher-k3s-pool-rancher-6c2c4-6vn9h, rancher-k3s-pool-rancher-6c2c4-7dt9t, rancher-k3s-pool-rancher-6c2c4-9fpzh:
* NodeHealthy: ConnectionDown
observedGeneration: 1
reason: ReadyUnknown
status: Unknown
type: MachinesReady
The machine objects themselves are showing ready.
❯ kubectl get machines
NAME CLUSTER NODENAME PROVIDERID PHASE AGE VERSION
rancher-k3s-pool-rancher-6c2c4-6vn9h rancher rancher-k3s-pool-rancher-6c2c4-6vn9h <k3s://rancher-k3s-pool-rancher-6c2c4-6vn9h> Running 34h
rancher-k3s-pool-rancher-6c2c4-7dt9t rancher rancher-k3s-pool-rancher-6c2c4-7dt9t <k3s://rancher-k3s-pool-rancher-6c2c4-7dt9t> Running 34h
rancher-k3s-pool-rancher-6c2c4-9fpzh rancher rancher-k3s-pool-rancher-6c2c4-9fpzh <k3s://rancher-k3s-pool-rancher-6c2c4-9fpzh> Running 34h
The machineset shows 0 ready with the reason:
- lastTransitionTime: "2025-12-09T20:46:34Z"
message: |-
* Machines rancher-k3s-pool-rancher-6c2c4-6vn9h, rancher-k3s-pool-rancher-6c2c4-7dt9t, rancher-k3s-pool-rancher-6c2c4-9fpzh:
* NodeHealthy: ConnectionDown
observedGeneration: 1
reason: ReadyUnknown
status: Unknown
type: MachinesReady
I need help in the next steps on how to approach this problem. Ultimate goal is having everything up again.future-refrigerator-96508
12/11/2025, 3:08 PMancient-raincoat-46356
12/11/2025, 6:06 PMancient-raincoat-46356
12/11/2025, 6:08 PMis not a valid chart repository or cannot be reached: failed to fetch <https://github.com/rancher/vsphere-charts/tree/main/charts/rancher-vsphere-csi/index.yaml> : 404 Not Found. I checked and sure enough there is no index.yaml in that repo, so how do I install it?victorious-airport-41755
12/12/2025, 6:09 PMsquare-rose-18388
12/12/2025, 6:51 PMshy-book-69032
12/13/2025, 1:45 AM