creamy-room-58344
11/25/2022, 10:06 PMstocky-article-82001
11/25/2022, 11:34 PMcluster [local] contains more than 1 [System] project, requeuing
- Getting this error a lot, not sure how to delete a System project as they're protected from being deleted.damp-agent-26487
11/26/2022, 4:57 AMelegant-school-20229
11/26/2022, 6:57 AMgentle-petabyte-40055
11/26/2022, 9:08 AMnutritious-analyst-54861
11/26/2022, 10:56 PMnutritious-analyst-54861
11/26/2022, 10:56 PMgentle-petabyte-40055
11/26/2022, 11:32 PMwooden-morning-99379
11/27/2022, 5:06 AMgentle-petabyte-40055
11/27/2022, 7:24 AMgentle-petabyte-40055
11/27/2022, 7:24 AMwhite-addition-57207
11/27/2022, 7:18 PMwhite-addition-57207
11/27/2022, 7:21 PMbright-fireman-42144
11/27/2022, 9:58 PMgreen-greece-89415
11/28/2022, 4:17 AMgreen-greece-89415
11/28/2022, 4:18 AMstraight-appointment-92464
11/28/2022, 7:27 AMlittle-father-38603
11/28/2022, 1:57 PMFailed to scrape node" err="Get \"https://***:10250/metrics/resource\": context deadline exceeded" node="node-that-stopped-reporting"
kubectl top node shows:
node-that-stopped-working <unknown> <unknown> <unknown> <unknown>
HPAs are not working because of this, and I'm not sure what other issues this could cause.
If I restart the node, the reporting starts working again for a few days, and then it stops.
The machines are powerful enough and this problem started when the cluster was still empty, so I doubt is resources-related.
Even now that the cluster is running some workloads, there are pleny of resources available.
I have no idea what could cause this, but I noticed something interesting:
kubectl get --raw /api/v1/nodes/$NODE_NAME/proxy/stats/summary
takes a really long time (~2 minutes) on those nodes that don't work, while it's fast (~2 seconds) on a working node.
Any kind of help is greatly appreciated!green-greece-89415
11/28/2022, 4:00 PMhelpful-beard-54962
11/28/2022, 7:56 PMagreeable-actor-89488
11/28/2022, 9:42 PMbright-daybreak-52307
11/29/2022, 9:08 AMacceptable-soccer-28720
11/29/2022, 9:30 AMGet "<https://charts.bitnami.com/bitnami/index.yaml>": dial tcp: lookup <http://charts.bitnami.com|charts.bitnami.com> on 10.43.0.10:53: no such host
breezy-ram-80329
11/29/2022, 1:50 PMsilly-solstice-24970
11/29/2022, 3:58 PMcareful-piano-35019
11/29/2022, 4:46 PMable-wall-25846
11/29/2022, 5:22 PMroot@knode03:~# cat /etc/rancher/rke2/registries.yaml
mirrors:
<http://registry01.sys.nc4.iad0.nsscloud.net|registry01.sys.nc4.iad0.nsscloud.net>:
endpoint:
- "<http://registry01.sys.nc4.iad0.nsscloud.net:5000>"
root@knode03:~#
able-wall-25846
11/29/2022, 5:22 PM