02/23/2023, 9:17 AM
we upgraded the rancher from 2.6.6 to 2.6.9 last week, we run the rancher with rke cluster and have 3 node to deploy the rancher pod, it is a bit weird that after we upgraded, one node (say node1)with high CPU load, always >98%, and keeps the high level, other two nodes are normal; then after two days, we rollout restart the rancher deployment, and another node(node2) CPU start to grows with high load and >98% and keeps the high level, node1 and node3 keeps normal; and yesterday we rollout restart the rancher deployment again, then the high CPU happened on the node3, other two keeps normal. we also found kind of below message in the log: I0222 12:44:17.493109 33 trace.go:205] Trace[1965410108]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 12:43:33.949) (total time: 43543ms): I0222 12:44:36.012515 33 trace.go:205] Trace[765542589]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 12:43:58.939) (total time: 37073ms): I0222 13:21:56.380811 33 trace.go:205] Trace[1175746524]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 13:21:40.078) (total time: 16302ms): I0222 13:22:24.878559 33 trace.go:205] Trace[1882912053]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 13:21:39.950) (total time: 44927ms): I0222 13:43:14.716513 33 trace.go:205] Trace[1266043037]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 13:42:30.269) (total time: 44446ms): I0222 13:56:10.856860 33 trace.go:205] Trace[84778214]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 13:55:26.561) (total time: 44295ms): I0222 15:07:14.510236 33 trace.go:205] Trace[18689844]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 15:06:30.834) (total time: 43675ms): I0222 15:14:16.370077 33 trace.go:205] Trace[371800202]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 15:13:58.672) (total time: 17697ms): I0222 15:14:42.331126 33 trace.go:205] Trace[182465127]: "Reflector ListAndWatch" name:pkg/mod/ (22-Feb-2023 15:13:58.320) (total time: 44010ms): anyone met such issue or advise? thanks