glamorous-camera-42186
07/02/2025, 2:39 PMJul 02 16:29:50 <mynode> k3s[4671]: E0702 16:29:50.392111 4671 cacher.go:478] cacher (secrets): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/ai/forgejo-registry": invalid padding on input; reinitializing...
Jul 02 16:29:51 <mynode> k3s[4671]: E0702 16:29:51.500764 4671 transformer.go:164] "failed to decrypt data" err="invalid padding on input"
Jul 02 16:29:51 <mynode> k3s[4671]: E0702 16:29:51.500888 4671 transformer.go:164] "failed to decrypt data" err="invalid padding on input"
Jul 02 16:44:11 <mynode> k3s[4886]: W0702 16:44:11.071869 4886 reflector.go:569] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/ai/forgejo-registry": invalid padding on input
When I delete this secret (ai/forgejo-registry) it just spams with another. I tried multiple times to reinstall and it still fails to handle those encrypted secrets.
All nodes in cluster have --secrets-encryption
. When trying to run without it the k3s fails with error that the config is not consistend among nodes.
Encryption status:
k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match
Active Key Type Name
------ -------- ----
* AES-CBC aescbckey
Maybe important, that the 2 existing nodes are amd64, and the one problematic is arm64. The cluster have somewhat long history, it started as single node k8s, then it was transformed to 3 node, and I think it already had some nodes reinstalled like that. k3s is in version 1.32.5, on the problematic node nor 1.32.5 nor 1.32.6 works.glamorous-camera-42186
07/02/2025, 5:21 PM/var/lib/rancher/k3s/server/cred/encryption-config.json
And it was indeed generating different for the new node, while being the same for 2 existing nodes. I copied the file from one of 2 nodes to the new node after initialization, restarted, and now it works. LOL