https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
✅ 1
c
your secrets encryption config is messed up. The message indicates that it is expecting to find unencrypted secrets (“identity” means unencrypted) but they are in fact encrypted.
s
Hm. I wonder how it has worked through multiple restarts in that case. `k3s secrets-encrypt status`reports that encryption is enabled and is using AES-CBC suggestion there is some kind of desync in that case? Suggestions on how to proceed? Also 
Server Encryption Hashes: All hashes match
c
you could check the config file on disk to see what it says, on the off chance that someone modified it by hand
s
Found the sqlite database for etcd (or service replacing it) and it dumped it and looks like the values are encrypted. Last field is always a alphanumeric string
Looks like it is hexadecimal.
Does not look like AES-CBC though since it does not end with ==?
c
go check out the config file. the message indicates that it thinks it should be finding things in plaintext.
s
Oh. I should add that I'm currently in a reverted state that (snapshot) from where I have not attempted to update and everything is working
Are we talking about /var/lib/rancher/k3s/server/cred/encryption-config.json? It does look OK and contains aescbc and a secret. I did check it after last update and it contained the same info had not been modified
c
where in the order is the identity vs aescbc
s
In the JSON document? 
{"kind":"EncryptionConfiguration","apiVersion":"<http://apiserver.config.k8s.io/v1|apiserver.config.k8s.io/v1>","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"<hidden>="}]}},{"identity":{}}]}]}
Hm. I ran 
k3s secrets-encrypt  disable
and it reported it is disabled (but left 
--secrets-encryption
in k3s startup arguments and appears to have successfully updated to 
Server Version: v1.31.4+k3s1
. secrets-encrypt list status still reports that it is disabled. What does 
--secrets-encryption
do in that case? Or does 
k3s secrets-encrypt disable
overwrite the CLI argument?
c
That is the correct way to do that, if you do not want encryption. Once you’ve started with the --secrets-encryption flag you should not ever remove it. You can rotate back to the identity key (effectively making the encryption a no-op) but you shouldn’t remove it from the args.
s
OK. As I stated. I have not removed it. Restarted the system to make sure everything looks OK and I'm now back to broken secrets
c
what does the status currently report, and how does that compare to what you see in the encryption config file?
s
After reboot is states:
Copy code
# k3s secrets-encrypt status
Encryption Status: Disabled, no configuration file found
c
and is there in fact no config file?
And you can confirm that it was started with the secrets-encryption cli flag?
s
There is a config file. Verifying whether the contents of the file has changed
c
there is nothing else tampering with the encryption config file contents?
s
identity and keys has switch places in the config file (/mnt/vm/k3s/server/cred/encryption-config.json). Old:
Copy code
{
  "kind": "EncryptionConfiguration",
  "apiVersion": "<http://apiserver.config.k8s.io/v1|apiserver.config.k8s.io/v1>",
  "resources": [
    {
      "resources": [
        "secrets"
      ],
      "providers": [
        {
          "aescbc": {
            "keys": [
              {
                "name": "aescbckey",
                "secret": "key="
              }
            ]
          }
        },
        {
          "identity": {}
        }
      ]
    }
  ]
}
New:
Copy code
{
  "kind": "EncryptionConfiguration",
  "apiVersion": "<http://apiserver.config.k8s.io/v1|apiserver.config.k8s.io/v1>",
  "resources": [
    {
      "resources": [
        "secrets"
      ],
      "providers": [
        {
          "identity": {}
        },
        {
          "aescbc": {
            "keys": [
              {
                "name": "aescbckey",
                "secret": "key="
              }
            ]
          }
        }
      ]
    }
  ]
}
Is there any easy way to confirm the arguments to the daemon?
no changes at least in service and its environment file.
Copy code
EnvironmentFile=-/etc/systemd/system/k3s.service.env
and:
Copy code
CMD_K3S_EXEC="server --secrets-encryption --flannel-backend=none --disable-network-policy --resolv-conf=/data/kubernetes/resolv.conf --kube-controller-manager-arg terminated-pod-gc-threshold=5"
Did a rollback yesterday again since the stack was severely broken after version upgrade.. Is there any other files that might have changed that causes 
3s secrets-encrypt status
to state that there is no config file. Is it documented which files are relevant? As asked earlier, does it exist a good way to confirm that the expected commandline arguments are actually passed to k3s? Couldn't find anything about it in the systemd journal
What files are 
k3s secrets-encrypt
looking in when deciding whether encryption is enabled or not?
OK. You were right. The command line arguments was never passed to the daemon. I showed the environment file earlier and that one is the problem. Apparently 
CMD_K3S_EXEC
is not accepted any more. Added the arguments to Exec= string directly in systemd unit file and it works
c
I’m not familiar with that env var.
s
Was something I saw online. Moved all config to /etc/rancher/k3s/config.yaml as is proper and everything looks fine. Thanks for your help
c
it wasn’t chatgpt was it lol
s
No. To be honest I don't know how I came to the conclusion to use that environment variable. Because google gives no good reason to use it at all. I'm on Gentoo and found an unmaintained ebuild (used by the package manager in Gentoo) and picked it up and started fixing it for later releases. Must've been in connection to that somehow.Ah yes. It was. From here https://github.com/gentoo-mirror/myov/blob/master/sys-cluster/k3s-bin/files/k3s.service#L12
Ah.. and then I updated the installed systemd unit file with the official systemd unit from the k3s project and this line was lost and hence this problem (maybe)?
c
ah yeah that’d do it. I wonder why they added that.
s
Only thing I can think of that might have helped in this situation was if `k3s secrets-encrypt status`gave a suggestion that it was not even turned on instead of no config file existing
Plus including the command line arguments for k3s in the service logs. Couldn't see it there at all
c
I believe that is what the no config file message indicates. If the flag is enabled but encryption is disabled then there will still be a config file, but it will only use the identity provider.
22 Views
Open in Slack
PreviousNext
Powered by