https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
c
howdy! no, I haven’t heard of that… the encryption config file is stored in the bootstrap key along with the certs and such, so if the node comes up with the correct cluster CAs it should also come up with the same secrets encryption config. What symptoms are you seeing? Can you cat the encryption config file on the different nodes and compare the content?
m
so yeah, comparing the configs, they keys are different. logs look like this:
Copy code
Mar 31 20:49:29 ip-10-10-11-206 k3s[8438]: W0331 20:49:29.108204    8438 reflector.go:424] storage/cacher.go:/secrets: failed to list *core.Secret: unable to transform key "/registry/secrets/kube-system/i-01db15558336bc664.node-password.k3s": invalid padding on input
Mar 31 20:49:29 ip-10-10-11-206 k3s[8438]: E0331 20:49:29.108227    8438 cacher.go:440] cacher (*core.Secret): unexpected ListAndWatch error: failed to list *core.Secret: unable to transform key "/registry/secrets/kube-system/i-01db15558336bc664.node-password.k3s": invalid padding on input; reinitializing...
Mar 31 20:49:29 ip-10-10-11-206 k3s[8438]: E0331 20:49:29.785882    8438 available_controller.go:524] <http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io> failed with: Operation cannot be fulfilled on <http://apiservices.apiregistration.k8s.io|apiservices.apiregistration.k8s.io> "<http://v1beta1.metrics.k8s.io|v1beta1.metrics.k8s.io>": the object has been modified; please apply your changes to the latest version and try again
...i should say, we do have darren and he's going to investigate, so i dont necessarily need to burn a bunch of your time. was more just curious if it was a known issue
c
that is odd. I know there are some race conditions around multiple servers initializing the SQL DB when first creating the cluster, but I’ve not seen anything where just the secrets encryption config getting out of sync, later on when adding new server.
m
right on. we're probably doing something wrong. ill let ya know
on a cooler note: we have a pretty cool k3s setup going. in aws and using their karpenter tool for autoscaling, their webhook identity tool for injecting iam roles into pods. its a beautiful monster
c
Before I came here I used to run kiam on all our clusters to handle mapping pods to IAM roles, it was super handy. I’m always surprised that more people don’t do that. It’s nice that they have a first-party tool to do that now.
m
yeah, works well
we did repro and darren root caused:
This is a bug in k3s. I found the root issue. What happens is that if you enable secrets encryption after the fact (not first boot) the encryption key is never saved in the bootstrap data. This issue isn’t seen in etcd setups because they copy files from peers not from DB. But this is an issue even with etcd because if all nodes are lost you lose the encryption key because it never got saved.
me, follow up: so even daishan's suggested work around of scaling CP down to 1 node and back up would not have worked around? Darren:
No. Because the bootstrap data is rarely updated. You’d have to scale to one node and then run key rotate
But it’s scale to one node where the one node has to be the one working.
c
ahh yeah that’s listed as not supported in the docs, because we know it’s not handled right https://docs.k3s.io/cli/secrets-encrypt#encryption-key-rotation
Starting K3s without encryption and enabling it at a later time is currently not supported.
sorry I figured you’d ruled that out
m
right on. yeah, just missed that
c
that is probably fixable if you want to open an issue about it, I think we just didn’t want to have to worry about it. Especially since most of the folks that are using secrets encryption are doing so in rke2 where it’s hardcoded on and we can rely on it being enabled from the start.
m
gotcha. yeah, ill open an issue. thanks for the quick responses. mountain biking weather up there yet?
c
haha not quite. It’s been a cold and fairly wet last half of spring and winter. We keep getting snow down into the foothills and it’s keeping the trails covered or muddy.
c
Patches welcome Craig 😂
🫡 1
(not often I get to do that to my own former manager, can't blame me)
57 Views
Open in Slack
PreviousNext
Powered by