gorgeous-minister-309
09/13/2022, 8:04 AMretrieve_connection_info
function. The call to /v3/connect/agent
is failing with a 401 Unauthorized error.
The rancher server is accessible from the virtual machine. The script is called without any parameters but includes some references to my rancher instance : CATTLE_AGENT_BINARY_BASE_URL
, CATTLE_SERVER
, CATTLE_TOKEN
. I don't understand why this is failing.
Any idea?shy-actor-78724
09/13/2022, 9:22 AMgorgeous-minister-309
09/13/2022, 9:37 AM2022/09/13 08:49:48 [INFO] [planner] rkecluster fleet-default/rke2: waiting: waiting for viable init node
2022/09/13 08:50:51 [INFO] [planner] rkecluster fleet-default/rke2: waiting: configuring bootstrap node(s) rke2-pool1-5fb5f65fbf-mtwbm: waiting for bootstrap etcd to be available
2022/09/13 08:50:51 [ERROR] [planner] rkecluster fleet-default/rke2: error encountered during plan processing was Operation cannot be fulfilled on <http://machines.cluster.x-k8s.io|machines.cluster.x-k8s.io> "rke2-pool1-5fb5f65fbf-mtwbm": the object has been modified; please apply your changes to the latest version and try again
2022/09/13 08:50:51 [INFO] [planner] rkecluster fleet-default/rke2: waiting: configuring bootstrap node(s) rke2-pool1-5fb5f65fbf-mtwbm: waiting for agent to check in and apply initial plan
And from the node side:
error 401 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again
The command used at this step is:
curl --connect-timeout 60 --max-time 60 --write-out %{http_code}\n -sS -H "Authorization: Bearer 47DEQp{REDACTED}uFU=" -H "X-Cattle-Id: 2c47306bda0e6f8{REDACTED}d4eaf217eb" -H "X-Cattle-Role-Etcd: false" -H "X-Cattle-Role-Control-Plane: false" -H "X-Cattle-Role-Worker: false" -H "X-Cattle-Node-Name: " -H "X-Cattle-Address: " -H "X-Cattle-Internal-Address: " -H "X-Cattle-Labels: " -H "X-Cattle-Taints: " <https://rancher.REDACTED/v3/connect/agent> -o /var/lib/rancher/agent/rancher2_connection_info.json
agreeable-oil-87482
09/13/2022, 11:41 AMrancher-system-agent
service. pleasegorgeous-minister-309
09/13/2022, 11:54 AMagreeable-oil-87482
09/13/2022, 11:59 AMgorgeous-minister-309
09/13/2022, 12:04 PMagreeable-oil-87482
09/13/2022, 12:06 PMgorgeous-minister-309
09/13/2022, 12:30 PM2022/09/13 12:29:36 [DEBUG] [CAPI] Reconcile MachineSet
2022/09/13 12:29:36 [DEBUG] [CAPI] Cannot retrieve CRD with metadata only client, falling back to slower listing
2022/09/13 12:29:36 [DEBUG] [CAPI] Cannot retrieve CRD with metadata only client, falling back to slower listing
2022/09/13 12:29:36 [DEBUG] [CAPI] Unable to retrieve Node status, missing NodeRef
2022/09/13 12:29:36 [DEBUG] [CAPI] Some nodes are not ready yet, requeuing until they are ready
2022/09/13 12:29:38 [DEBUG] [rke2configserver] parsed 8698dcff893f96d39ab97dde871968ae56b13a687e1e2680cfd16c5184fd5c6 as machineID
2022/09/13 12:29:38 [DEBUG] [rke2configserver] Got / machine from provisioning SA
2022/09/13 12:29:38 [DEBUG] [rke2configserver] Got / machine from cluster token
2022/09/13 12:29:40 [DEBUG] ObjectsAreEqualResults for machine-xwqg8: statusEqual: true conditionsEqual: false specEqual: true nodeNameEqual: true labelsEqual: true annotationsEqual: true requestsEqual: true limitsEqual: true rolesEqual: true
2022/09/13 12:29:40 [DEBUG] ObjectsAreEqualResults for machine-xwqg8: statusEqual: true conditionsEqual: false specEqual: true nodeNameEqual: true labelsEqual: true annotationsEqual: true requestsEqual: true limitsEqual: true rolesEqual: true
2022/09/13 12:29:40 [DEBUG] Updating machine for node [local-node]
2022/09/13 12:29:40 [DEBUG] Updated machine for node [local-node]
2022/09/13 12:29:40 [DEBUG] DesiredSet - No change(2) <http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>, Kind=Cluster fleet-local/local for provisioning-cluster-create local
2022/09/13 12:29:40 [DEBUG] DesiredSet - No change(2) /v1, Kind=Secret fleet-local/local-kubeconfig for cluster-create fleet-local/local
2022/09/13 12:29:40 [DEBUG] DesiredSet - No change(2) <http://management.cattle.io/v3|management.cattle.io/v3>, Kind=ClusterRoleTemplateBinding local/local-fleet-local-owner for cluster-create fleet-local/local
2022/09/13 12:29:40 [DEBUG] DesiredSet - No change(2) <http://fleet.cattle.io/v1alpha1|fleet.cattle.io/v1alpha1>, Kind=Cluster fleet-local/local for fleet-cluster fleet-local/local
2022/09/13 12:29:43 [DEBUG] [rke2configserver] parsed 8698dcff893f96d39ab97dde871968ae56b13a687e1e2680cfd16c5184fd5c6 as machineID
2022/09/13 12:29:43 [DEBUG] [rke2configserver] Got / machine from provisioning SA
2022/09/13 12:29:43 [DEBUG] [rke2configserver] Got / machine from cluster token
2022/09/13 12:29:45 [DEBUG] DesiredSet - No change(2) /v1, Kind=ServiceAccount fleet-default/cl1-bootstrap-template-x52q7-machine-bootstrap for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:29:45 [DEBUG] DesiredSet - No change(2) /v1, Kind=ServiceAccount fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:29:45 [DEBUG] DesiredSet - No change(2) /v1, Kind=Secret fleet-default/cl1-bootstrap-template-x52q7-machine-bootstrap for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:29:45 [DEBUG] DesiredSet - No change(2) /v1, Kind=Secret fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:29:45 [DEBUG] DesiredSet - No change(2) <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=Role fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:29:45 [DEBUG] DesiredSet - No change(2) <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=RoleBinding fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:29:48 [DEBUG] [CAPI] Cannot retrieve CRD with metadata only client, falling back to slower listing
2022/09/13 12:29:48 [DEBUG] [CAPI] Infrastructure provider is not ready, requeuing
2022/09/13 12:29:48 [DEBUG] [CAPI] Cannot reconcile Machine's Node, no valid ProviderID yet
2022/09/13 12:29:48 [DEBUG] Searching for providerID for selector <http://rke.cattle.io/machine=c3492044-4855-44ed-8d9b-c8d4c5185a8f|rke.cattle.io/machine=c3492044-4855-44ed-8d9b-c8d4c5185a8f> in cluster fleet-default/cl1, machine cl1-pool1-769dbbd958-w5tb2: Get "<https://10.43.10.155/k8s/clusters/c-m-stx6kqzc/api/v1/nodes?labelSelector=rke.cattle.io%!F(MISSING)machine%!D(MISSING)c3492044-4855-44ed-8d9b-c8d4c5185a8f>": dial tcp 10.43.10.155:443: connect: no route to host
2022/09/13 12:29:48 [DEBUG] [rke2configserver] parsed 8698dcff893f96d39ab97dde871968ae56b13a687e1e2680cfd16c5184fd5c6 as machineID
2022/09/13 12:29:48 [DEBUG] [rke2configserver] Got / machine from provisioning SA
2022/09/13 12:29:48 [DEBUG] [rke2configserver] Got / machine from cluster token
2022/09/13 12:29:51 [DEBUG] Extras returned map[principalid:[<local://user-lrxjb>] username:[admin]]
2022/09/13 12:29:51 [DEBUG] Triggering auth refresh on user-lrxjb
2022/09/13 12:29:51 [DEBUG] Skipping refresh for user-lrxjb due to max-age
2022/09/13 12:29:51 [DEBUG] [CAPI] Reconcile MachineSet
2022/09/13 12:29:51 [DEBUG] [CAPI] Cannot retrieve CRD with metadata only client, falling back to slower listing
2022/09/13 12:29:51 [DEBUG] [CAPI] Cannot retrieve CRD with metadata only client, falling back to slower listing
2022/09/13 12:29:51 [DEBUG] [CAPI] Unable to retrieve Node status, missing NodeRef
2022/09/13 12:29:51 [DEBUG] [CAPI] Some nodes are not ready yet, requeuing until they are ready
2022/09/13 12:29:53 [DEBUG] [rke2configserver] parsed 8698dcff893f96d39ab97dde871968ae56b13a687e1e2680cfd16c5184fd5c6 as machineID
2022/09/13 12:29:53 [DEBUG] [rke2configserver] Got / machine from provisioning SA
2022/09/13 12:29:53 [DEBUG] [rke2configserver] Got / machine from cluster token
2022/09/13 12:29:59 [DEBUG] [rke2configserver] parsed 8698dcff893f96d39ab97dde871968ae56b13a687e1e2680cfd16c5184fd5c6 as machineID
2022/09/13 12:29:59 [DEBUG] [rke2configserver] Got / machine from provisioning SA
2022/09/13 12:29:59 [DEBUG] [rke2configserver] Got / machine from cluster token
2022/09/13 12:30:03 [DEBUG] DesiredSet - No change(2) /v1, Kind=ServiceAccount fleet-default/cl1-bootstrap-template-x52q7-machine-bootstrap for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:30:03 [DEBUG] DesiredSet - No change(2) /v1, Kind=ServiceAccount fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:30:03 [DEBUG] DesiredSet - No change(2) /v1, Kind=Secret fleet-default/cl1-bootstrap-template-x52q7-machine-bootstrap for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:30:03 [DEBUG] DesiredSet - No change(2) /v1, Kind=Secret fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:30:03 [DEBUG] DesiredSet - No change(2) <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=Role fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:30:03 [DEBUG] DesiredSet - No change(2) <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=RoleBinding fleet-default/cl1-bootstrap-template-x52q7-machine-plan for rke-machine fleet-default/cl1-bootstrap-template-x52q7
2022/09/13 12:30:04 [DEBUG] [rke2configserver] parsed 8698dcff893f96d39ab97dde871968ae56b13a687e1e2680cfd16c5184fd5c6 as machineID
2022/09/13 12:30:04 [DEBUG] [rke2configserver] Got / machine from provisioning SA
2022/09/13 12:30:04 [DEBUG] [rke2configserver] Got / machine from cluster token
2022/09/13 12:30:06 [DEBUG] [CAPI] Reconcile MachineSet
2022/09/16 13:24:11 [ERROR] [CAPI] Reconciler error: error creating client and cache for remote cluster: error creating dynamic rest mapper for remote cluster "fleet-default/cdr4": Get "<https://10.43.150.240/k8s/clusters/c-m-shq6vqt7/api?timeout=10s>": dial tcp 10.43.150.240:443: connect: no route to host
I don't understand it. Does rancher try to reach the cluster on an internal private IP address?