silly-solstice-24970
12/05/2022, 8:29 PMrke version v1.3.12
, and while attempting to monitor with prometheus+grafana k8s services (without success) noticed they didn’t get and endpoint IP:
kube-system kube-prometheus-coredns ClusterIP None <none> 9153/TCP 127m
kube-system kube-prometheus-kube-controller-manager ClusterIP None <none> 10257/TCP 127m
kube-system kube-prometheus-kube-etcd ClusterIP None <none> 2381/TCP 127m
kube-system kube-prometheus-kube-proxy ClusterIP None <none> 10249/TCP 127m
kube-system kube-prometheus-kube-scheduler ClusterIP None <none> 10259/TCP 127m
kube-system kube-prometheus-kubelet ClusterIP None <none> 10250/TCP,10255/TCP,4194/TCP 127m
Because of that, we are unable to monitor those metrics; this is an example configuration for one of our clusters:
---
nodes:
- address: node1.localdomain
hostname_override: node01
user: rke
role:
- controlplane
- worker
- etcd
labels:
role: storage-node
- address: node02.localdomain
hostname_override: node02
user: rke
role:
- controlplane
- worker
- etcd
labels:
role: storage-node
- address: node03.localdomain
hostname_override: node03
user: rke
role:
- controlplane
- worker
- etcd
labels:
role: storage-node
- address: node04.localdomain
hostname_override: node04
user: rke
role:
- worker
labels:
role: storage-node
- address: node05.localdomain
hostname_override: node05
user: rke
role:
- worker
labels:
role: storage-node
services:
etcd:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
external_urls: []
ca_cert: ""
cert: ""
key: ""
path: ""
uid: 0
gid: 0
snapshot: null
retention: ""
creation: ""
backup_config: null
kube-api:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
service_cluster_ip_range: 10.43.0.0/16
service_node_port_range: ""
pod_security_policy: false
always_pull_images: false
secrets_encryption_config: null
audit_log: null
admission_configuration: null
event_rate_limit: null
kube-controller:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
cluster_cidr: 10.42.0.0/16
service_cluster_ip_range: 10.43.0.0/16
scheduler:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
kubelet:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
cluster_domain: cluster.local
infra_container_image: ""
cluster_dns_server: 10.43.0.10
fail_swap_on: false
generate_serving_certificate: false
kubeproxy:
image: ""
extra_args: {}
extra_binds: []
extra_env: []
network:
plugin: canal
options: {}
mtu: 0
node_selector: {}
authentication:
strategy: x509
sans: []
webhook: null
addons: ""
addons_include: []
ssh_key_path: ~/.ssh/id_rsa
ssh_cert_path: ""
ssh_agent_auth: false
authorization:
mode: rbac
options: {}
ignore_docker_version: false
kubernetes_version: "v1.23.7-rancher1-1"
..........
Is there anything I have to do to enable those metrics?creamy-accountant-88363
12/05/2022, 9:09 PMlittle-actor-95014
12/05/2022, 10:26 PMgorgeous-iron-45755
12/06/2022, 8:51 AMlively-stone-91987
12/06/2022, 10:07 AMlively-night-78214
12/06/2022, 11:19 AMlively-night-78214
12/06/2022, 11:20 AMdry-island-5201
12/06/2022, 11:43 AMhundreds-evening-84071
12/06/2022, 3:05 PMdelightful-magazine-28676
12/06/2022, 5:18 PMable-wall-25846
12/06/2022, 5:39 PMable-wall-25846
12/06/2022, 5:40 PMprehistoric-advantage-39331
12/06/2022, 6:54 PMbreezy-ram-80329
12/07/2022, 8:25 AMbig-spoon-24382
12/07/2022, 9:39 AMmost-laptop-5033
12/07/2022, 10:14 AMgentle-laptop-43224
12/07/2022, 11:32 AMbest-room-53751
12/07/2022, 11:40 AMbest-room-53751
12/07/2022, 11:40 AMbest-room-53751
12/07/2022, 11:41 AMchilly-truck-71927
12/07/2022, 12:14 PMable-island-23791
12/07/2022, 1:08 PMDec 7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=warning msg="not running in CIS mode"
Dec 7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=info msg="Starting rke2 v1.23.14+rke2r1 (83c8eb91b96ed5a2a3beb06c905e075724305a46)"
Dec 7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=info msg="Managed etcd cluster not yet initialized"
Dec 7 18:44:09 <SERVER_NAME> rke2[91604]: time="2022-12-07T18:44:09+06:00" level=fatal msg="starting kubernetes: preparing server: failed to validate server configuration: https://<IP_ADDR>:9345/v1-rke2/config: 401 Unauthorized"
What do it want?limited-potato-16824
12/07/2022, 2:31 PMfleet-local
to fleet-defaul
by following the instructions here:
https://fleet.rancher.io/troubleshooting#migrate-the-local-cluster-to-the-fleet-default-cluster
After we initialized the move, I could see this in the logs:
rancher-5677f59677-shbs7 rancher 2022/12/07 12:39:42 [ERROR] error syncing 'local': handler provisioning-cluster-create: failed to create fleet-default/local <http://provisioning.cattle.io/v1|provisioning.cattle.io/v1>, Kind=
Cluster for provisioning-cluster-create local: admission webhook "<http://rancherauth.cattle.io|rancherauth.cattle.io>" denied the request: cluster name must be 63 characters or fewer, cannot be "local" nor of the form "c
-xxxxx", requeuing
While this was happening, the local clulster had been removed from the fleet-local
workspace but for the reason above it did not show up in fleet-default
. We managed to get the cluster back to fleet-local
again after editing the object, but it would have been nice to have all the "Continuous delivery" clusters in the same workspace. If you have any hints how to make that migration successful, please share 🙂salmon-noon-33588
12/07/2022, 5:14 PMsalmon-noon-33588
12/07/2022, 5:21 PMgorgeous-iron-45755
12/07/2022, 5:55 PMdamp-hydrogen-20107
12/07/2022, 9:37 PMgifted-lunch-48361
12/08/2022, 6:17 AMgifted-lunch-48361
12/08/2022, 6:18 AMacceptable-vr-23713
12/08/2022, 8:34 AMacceptable-vr-23713
12/08/2022, 8:34 AMwitty-jelly-95845
12/09/2022, 6:16 PM