hello, it seems that the new rke2 (at least 1.35.6...
# rke2
n
hello, it seems that the new rke2 (at least 1.35.6) is broken. I cannot do server restart. The kubelet either won't run at all. Or it runs, but only logs:
Copy code
I0625 20:03:37.294027 3966573 labels.go:289] "Label not found" label="io.kubernetes.container.ports"
I0625 20:03:37.294032 3966573 labels.go:278] "Label not found" label="io.kubernetes.pod.deletionGracePeriod"
I0625 20:03:37.294037 3966573 labels.go:289] "Label not found" label="io.kubernetes.container.preStopHandler"
I0625 20:03:37.294042 3966573 labels.go:289] "Label not found" label="io.kubernetes.container.ports"
I0625 20:03:37.294061 3966573 kuberuntime_manager.go:514] "Retrieved pods from runtime" all=true
E0625 20:03:37.344526 3966573 kubelet.go:3369] "Unable to register mirror pod because node is not registered yet" err="node \"<http://kub-r1.priv.cerit-sc.cz|kub-r1.priv.cerit-sc.cz>\" not found" node="<http://kub-r1.priv.cerit-sc.cz|kub-r1.priv.cerit-sc.cz>"
I0625 20:03:37.344564 3966573 config.go:107] "Looking for sources, have seen" sources=["api","file"] seenSources={"file":{}}
I0625 20:03:37.410804 3966573 eviction_manager.go:256] "Eviction manager: synchronize housekeeping"
I0625 20:03:37.410816 3966573 eviction_manager.go:292] "FileSystem detection" DedicatedImageFs=false SplitImageFs=false
E0625 20:03:37.410910 3966573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"<http://kub-r1.priv.cerit-sc.cz|kub-r1.priv.cerit-sc.cz>\" not found"
I0625 20:03:37.444089 3966573 config.go:107] "Looking for sources, have seen" sources=["api","file"] seenSources={"file":{}}
E0625 20:03:37.444106 3966573 kubelet.go:3369] "Unable to register mirror pod because node is not registered yet" err="node \"<http://kub-r1.priv.cerit-sc.cz|kub-r1.priv.cerit-sc.cz>\" not found" node="<http://kub-r1.priv.cerit-sc.cz|kub-r1.priv.cerit-sc.cz>"
I0625 20:03:37.449266 3966573 container_log_manager.go:197] "Starting container log rotation sequence"
and this is repetitive with a lot more messages. Reverted back to 1.35.5 and everything is ok, esp. the node kub-r1 does exist.
c
You're going to need to share more information than just a few log lines. Please open an issue and attach full rke2 service logs from journald, containerd.log, as well as etcd and apiserver pod logs.
Also confirm that you experienced this on the final release, and not one of the RCs.
Mostly it just sounds like the apiserver isn't up yet; without full logs can't really say why.
n
hmm, wanted to reproduce for the logs and the second time, it started ok, at least on the first node. (1.35.5 -> 1.35.6 -> 1.35.5 -> 1.35.6). If I encounter it again somewhere, I will file the issue.
probably my bad, only extremly slow container pull. combined with local docker proxy and a fact, that traefik a bit reogranized the chart. previous
service.type
is now
service.spec.type
, so it deconfigured loadbalancer (for the proxy).