kind-wolf-27908
03/23/2023, 9:35 PMkind-wolf-27908
03/23/2023, 9:35 PMkind-wolf-27908
03/23/2023, 9:36 PMkind-wolf-27908
03/23/2023, 9:37 PMkind-wolf-27908
03/23/2023, 9:37 PMpolite-piano-74233
03/24/2023, 1:23 AMwonderful-rain-13345
03/24/2023, 1:49 AM<http://field.cattle.io/publicEndpoints|field.cattle.io/publicEndpoints>
annotation set? i went from single node running old k3os to latest k3s by adding new node and removing old node. Now IP is incorrect and i get a 404.
oh IP is set in the load balancer in ingress, but no effect when i updatepolite-piano-74233
03/24/2023, 1:59 AMnutritious-vase-53845
03/24/2023, 3:51 AMbetter-ocean-90584
03/24/2023, 7:13 AMlittle-ice-73809
03/24/2023, 8:55 AMk3d cluster create
stuck forever on macos m1.brash-grass-87626
03/24/2023, 9:35 AMgifted-agent-35161
03/24/2023, 9:47 AMrancher/sever : 2.7.1
rke2 : v1.24.9+rke2r2 (2f4571a879954e1ea8d4560023eaf57c567df737)
go : version go1.18.7b7
k8s : v1.24.9+rke2r2
Os : Ubuntu 22.04LTS
Description :
I have changed the certificates on docker rancher container from self signed to a custom CA. I followed the documentation there : https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/resources/update-rancher-certificate
For step 4. Reconfigure Rancher agents to trust the private CA
, i have choosen method 2 by injecting the custom CA checksum inside rancher deployments/daemonsets.
For a reason that i don't understand, there's no daemonset/cattle-node-agent
on cattle-system namespace, so i didn't edited the rancher agent, just the cattle-cluster-agent
.
After i rebooted my rancher docker container with the new certificates bind to it.
What happened ?
Cluster sets itself into 'Updating' state, and master nodes are in state Reconciling
, with message "Waiting for plan to be applied".
Here's the YAML status of the kind Machine
of my stuck master node :
conditions:
- lastTransitionTime: '2022-10-14T09:53:34Z'
status: 'True'
type: Ready
- lastTransitionTime: '2022-10-14T09:53:32Z'
status: 'True'
type: BootstrapReady
- lastTransitionTime: '2022-10-14T09:56:29Z'
status: 'True'
type: InfrastructureReady
- lastTransitionTime: '2023-03-01T14:13:52Z'
status: 'True'
type: NodeHealthy
- lastTransitionTime: '2022-10-14T09:53:38Z'
status: 'True'
type: PlanApplied
- lastTransitionTime: '2023-03-24T09:14:48Z'
message: waiting for plan to be applied
reason: Waiting
status: Unknown
type: Reconciled
infrastructureReady: true
lastUpdated: '2022-10-14T09:57:38Z'
So i dive into machine logs, didn't find anything in journalctl
like empty for all rancher / rke services. So i decided to launch rancher-system-agent sentinel
by hand, and here we go :
rancher-system-agent sentinel
INFO[0000] Rancher System Agent version v0.2.13 (4fa9427) is starting
INFO[0000] Using directory /var/lib/rancher/agent/work for work
INFO[0000] Starting remote watch of plans
INFO[0000] Initial connection to Kubernetes cluster failed with error Get "<https://192.168.10.203/version>": x509: certificate signed by unknown authority, removing CA data and trying again
panic: error while connecting to Kubernetes cluster with nullified CA data: Get "<https://192.168.10.203/version>": x509: certificate signed by unknown authority
goroutine 10 [running]:
<http://github.com/rancher/system-agent/pkg/k8splan.(*watcher).start(0xc0002be280|github.com/rancher/system-agent/pkg/k8splan.(*watcher).start(0xc0002be280>, {0x18bd5c0?, 0xc0002b8740})
/go/src/github.com/rancher/system-agent/pkg/k8splan/watcher.go:99 +0x9b4
created by <http://github.com/rancher/system-agent/pkg/k8splan.Watch|github.com/rancher/system-agent/pkg/k8splan.Watch>
/go/src/github.com/rancher/system-agent/pkg/k8splan/watcher.go:63 +0x155
So this means that my custom CA certificate didn't propagate to the host, as a "workaround", I added the certificate to the node local openssl truststore and it worked fine after.
But every time i want to scale a new node it fails to install rke2. Also that it triggers a rollout of the cattle-cluster-agent
deployment with the old certificate checksum in CATTLE_CA_CHECKSUM
variable.
Do you have any advices / tips to help ? Thank you so muuuch 🙂ambitious-furniture-5481
03/24/2023, 12:12 PMFailed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "047ddd7327b53025c968db44b3cb7b11c0424a160a69cfbe2676de05ca2cbd14": plugin type="calico" failed (add): error getting ClusterInformation: connection is unauthorized: Unauthorized
best-agency-48985
03/24/2023, 3:42 PMGET
) similar to what’s in the Dashboard UI for resources such as /explorer/networking.k8s.io.ingress
.. I can’t seem to get the ingress information via the documented API endpoints for /v3/
in a way that’s similar to the ‘Service Discovery’ section of the Dashboard. Is there anyway to access this without using the method for generating the KUBECONFIG from a POST
for action=generateKubeconfig
and just querying the k8s API?chilly-translator-36463
03/24/2023, 5:22 PMchilly-translator-36463
03/24/2023, 5:23 PMroot@rkemas01:~# kubectl get nodes
E0324 22:53:27.340529 62189 memcache.go:238] couldn't get current server API group list: Get "<https://192.168.9.126/k8s/clusters/c-7xqz9/api?timeout=32s>": x509: certificate signed by unknown authority
E0324 22:53:27.393329 62189 memcache.go:238] couldn't get current server API group list: Get "<https://192.168.9.126/k8s/clusters/c-7xqz9/api?timeout=32s>": x509: certificate signed by unknown authority
E0324 22:53:27.462317 62189 memcache.go:238] couldn't get current server API group list: Get "<https://192.168.9.126/k8s/clusters/c-7xqz9/api?timeout=32s>": x509: certificate signed by unknown authority
E0324 22:53:27.496275 62189 memcache.go:238] couldn't get current server API group list: Get "<https://192.168.9.126/k8s/clusters/c-7xqz9/api?timeout=32s>": x509: certificate signed by unknown authority
E0324 22:53:27.568562 62189 memcache.go:238] couldn't get current server API group list: Get "<https://192.168.9.126/k8s/clusters/c-7xqz9/api?timeout=32s>": x509: certificate signed by unknown authority
Unable to connect to the server: x509: certificate signed by unknown authority
busy-judge-4614
03/24/2023, 5:25 PMcareful-article-61190
03/24/2023, 5:40 PMchilly-translator-36463
03/24/2023, 5:53 PMminiature-ambulance-98143
03/24/2023, 6:48 PMpolite-piano-74233
03/24/2023, 8:03 PMbusy-judge-4614
03/24/2023, 9:07 PMbest-agency-48985
03/24/2023, 10:07 PMmodifiers
are enabled for <https://RANCER_ADDRESS/v3/clusters?name_>
because I can only seem to get eq
to work, even though the schema suggests prefix
and like/notlike
should work but they do not work for me.. https://github.com/rancher/api-spec/blob/master/specification.md#filteringhelpful-beard-54962
03/25/2023, 1:22 AMmost-kite-870
03/25/2023, 6:25 AMaloof-dog-75479
03/25/2023, 9:11 AMaloof-dog-75479
03/25/2023, 9:14 AMaloof-dog-75479
03/25/2023, 9:14 AMbusy-judge-4614
03/25/2023, 11:00 AM