I have noticed that there is a issue/PR related to this problem.
https://github.com/k3s-io/k3s/issues/7424
So, I rollback to 1.27.1, and the error disappeared.
"If it's a single node can you just redeploy k3s on the node?", how to do this? Do you mean to execute script k3s-agent-uninstall.sh and reinstall k3s-agent? This couldn't be done cause it's in production, contained will be destroyed.
I'm wondering why only this node encounter this.