Answering it myself. After investigating bit more this looks like
intermittent NTP sync issue due to which Kubernetes errorred out while computing CPU usage based on the current and the cached time and K8s at that time transitions node into
NodeNotReady
state and reschedule all the pods.