we upgraded the rancher from 2.6.6 to 2.6.9 last w...
# general
h
we upgraded the rancher from 2.6.6 to 2.6.9 last week, we run the rancher with rke cluster and have 3 node to deploy the rancher pod, it is a bit weird that after we upgraded, one node (say node1)with high CPU load, always >98%, and keeps the high level, other two nodes are normal; then after two days, we rollout restart the rancher deployment, and another node(node2) CPU start to grows with high load and >98% and keeps the high level, node1 and node3 keeps normal; and yesterday we rollout restart the rancher deployment again, then the high CPU happened on the node3, other two keeps normal. we also found kind of below message in the log: I0222 124417.493109 33 trace.go:205] Trace[1965410108]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 124333.949) (total time: 43543ms): I0222 124436.012515 33 trace.go:205] Trace[765542589]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 124358.939) (total time: 37073ms): I0222 132156.380811 33 trace.go:205] Trace[1175746524]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 132140.078) (total time: 16302ms): I0222 132224.878559 33 trace.go:205] Trace[1882912053]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 132139.950) (total time: 44927ms): I0222 134314.716513 33 trace.go:205] Trace[1266043037]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 134230.269) (total time: 44446ms): I0222 135610.856860 33 trace.go:205] Trace[84778214]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 135526.561) (total time: 44295ms): I0222 150714.510236 33 trace.go:205] Trace[18689844]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 150630.834) (total time: 43675ms): I0222 151416.370077 33 trace.go:205] Trace[371800202]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 151358.672) (total time: 17697ms): I0222 151442.331126 33 trace.go:205] Trace[182465127]: "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (22-Feb-2023 151358.320) (total time: 44010ms): anyone met such issue or advise? thanks