Hello, I have problem with adding new node to clus...
# k3s
g
Hello, I have problem with adding new node to cluster. I had 3 node cluster, then I deleted one node, did k3s-uninstall and stuff, I also did kubectl delete node. But now I cannot again add/install this node, as k3s.service cannot properly start, it start looping errors:
Copy code
Jul 02 16:29:50 <mynode> k3s[4671]: E0702 16:29:50.392111    4671 cacher.go:478] cacher (secrets): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/ai/forgejo-registry": invalid padding on input; reinitializing...
Jul 02 16:29:51 <mynode> k3s[4671]: E0702 16:29:51.500764    4671 transformer.go:164] "failed to decrypt data" err="invalid padding on input"
Jul 02 16:29:51 <mynode> k3s[4671]: E0702 16:29:51.500888    4671 transformer.go:164] "failed to decrypt data" err="invalid padding on input"
Jul 02 16:44:11 <mynode> k3s[4886]: W0702 16:44:11.071869    4886 reflector.go:569] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/ai/forgejo-registry": invalid padding on input
When I delete this secret (ai/forgejo-registry) it just spams with another. I tried multiple times to reinstall and it still fails to handle those encrypted secrets. All nodes in cluster have
--secrets-encryption
. When trying to run without it the k3s fails with error that the config is not consistend among nodes. Encryption status:
Copy code
k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey
Maybe important, that the 2 existing nodes are amd64, and the one problematic is arm64. The cluster have somewhat long history, it started as single node k8s, then it was transformed to 3 node, and I think it already had some nodes reinstalled like that. k3s is in version 1.32.5, on the problematic node nor 1.32.5 nor 1.32.6 works.
okay, chatgpt suggested me to check
/var/lib/rancher/k3s/server/cred/encryption-config.json
And it was indeed generating different for the new node, while being the same for 2 existing nodes. I copied the file from one of 2 nodes to the new node after initialization, restarted, and now it works. LOL