https://rancher.com/ logo
Join SlackCommunities
Powered by
Hi. I installed k3s on rhel9 (curl -sfL <https://g...
# k3s
c
Hi. I installed k3s on rhel9 (curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.32.3+k3s1 sh -s - server --cluster-init) and metrics endpoint is broken.
Copy code
kubectl get apiservice
...
<http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io>              kube-system/metrics-server               False (MissingEndpoints)   23h
How can I reinstall the metrics server?
✅ 1
c
That's not what you need to do
Figure out why the pod isn't running and ready
✅ 1
c
Thanks @creamy-pencil-82913 ! E0429 162429.029473 1 scraper.go:149] "Failed to scrape node" err="Get \"https://192.168.160.209:10250/metrics/resource\": dial tcp 192.168.160.20910250 connect: no route to host" node="hostname"
My VM’s hostname was changed after the k3s installation. Thank you!
c
have you restarted since changing the hostname?
but note that you can’t rename nodes in kubernetes. you have to delete the node with the old name after changing the hostname.
c
From the log, even with the old hostname, it is resolving to the correct IP. I checked the services and the listening ports on the VM, and port 10250/tcp is fine. The problem might be the firewall (firewalld), which is still using the default configuration. I am checking.
Troubleshooting:
Copy code
[root@rancher-ui ~]# kubectl -n kube-system get events
LAST SEEN   TYPE      REASON             OBJECT                                 MESSAGE
2m39s       Warning   Unhealthy          pod/coredns-ff8999cc5-c2mp6            Readiness probe failed: Get "<http://10.42.0.70:8181/ready>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2m5s        Normal    Scheduled          pod/metrics-server-6f4c6675d5-777lz    Successfully assigned kube-system/metrics-server-6f4c6675d5-777lz to rancher-ui.intrajus.tjrn
2m5s        Normal    Pulled             pod/metrics-server-6f4c6675d5-777lz    Container image "rancher/mirrored-metrics-server:v0.7.2" already present on machine
2m5s        Normal    Created            pod/metrics-server-6f4c6675d5-777lz    Created container: metrics-server
2m5s        Normal    Started            pod/metrics-server-6f4c6675d5-777lz    Started container metrics-server
2m5s        Warning   Unhealthy          pod/metrics-server-6f4c6675d5-777lz    Readiness probe failed: Get "<https://10.42.0.73:10250/readyz>": dial tcp 10.42.0.73:10250: connect: connection refused
110s        Warning   Unhealthy          pod/metrics-server-6f4c6675d5-777lz    Readiness probe failed: HTTP probe failed with statuscode: 500
3m29s       Warning   Unhealthy          pod/metrics-server-6f4c6675d5-t46mf    Readiness probe failed: HTTP probe failed with statuscode: 500
2m5s        Normal    Killing            pod/metrics-server-6f4c6675d5-t46mf    Stopping container metrics-server
2m5s        Normal    SuccessfulCreate   replicaset/metrics-server-6f4c6675d5   Created pod: metrics-server-6f4c6675d5-777lz
[root@rancher-ui ~]# curl -k <https://10.42.0.73:10250/readyz>
ok[root@rancher-ui ~]#
[root@rancher-ui ~]# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens192
  sources:
  services: cockpit dhcpv6-client ssh zabbix-agent
  ports: 10250/tcp
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
[root@rancher-ui ~]#
c
it doesn’t come in on the physical interface
it comes in via the cni virtual interface
generally we recommend turning off firewalld as it doesn’t play well with Kubernetes
c
Pod is up and running, thank you. Logs show only the following alert: I0429 184212.761697 1 server.go:191] "Failed probe" probe="metric-storage-ready" err="no metrics to serve" It's normal?
c
yes, that is normal on startup
✅ 1
as long as you don’t see it repeating every probe interval
the first time it is probed for readiness it hasn’t scraped any metrics from nodes yet, so there are no metrics to serve
3 Views
Open in Slack
PreviousNext
Powered by