https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# general
a
This message was deleted.
d
Hi, can you please open a github issue in dashboard repo with clear steps to reproduce it?
b
I created a case already. I just wanted to tell people...
Another test: It wipes the cilium config even if it is reapplied after an upgrade.
It does not wipe the config for 1.33 clusters.
c
there is an existing rancher issue where addon values are lost when the cluster version is changed. I’m not sure if it was supposed to be fixed or not
I see @dry-father-91460 opened https://github.com/rancher/dashboard/issues/17498 for you but I swear we already had one
d
We had one for Harvester before
I can reproduce this on master, so I am pretty positive this is a UI bug and it is not fixed
c
https://github.com/rancher/dashboard/issues/5672 might be the one I’m thinking of
seems like something that’s broke and been fixed multiple times
d
it's a tricky area to test, but I will improve test coverage
b
The issue that I mention here was not present until 2.13.5 and RKE2 1.34. Therefore I suppose it's not the same. And if you guys tell me you're not going to fix it, our customer will go mayhem...
c
@bumpy-portugal-40754 do these other issues I’ve linked to not describe the same problem?
d
it looks like a problem we keep re-introducing
🤦 1
b
Give me a moment to read and understand them.
c
I found four other issues for this same thing in 30 seconds of searching, all of them closed. not sure if its never been properly fixed, or if we keep re-breaking it
b
5672 is not the same. There is no pop-up. It's silently wiped out. And only when the guest cluster is upgraded to 1.34.
5837 also sounds different. There is no version conflict in the Cilium addon. All the values are valid with the old and new RKE2 and Cilium versions.
n
so if one is not using rancher addon-ui, then it should work without problems? i.e., configuring (cilium) via files in 
/var/lib/rancher/rke2/server/manifests
b
12418 still happens from time to time. But the new issue is not related to tab switching.
c
5837 sounds like it exactly to me
b
@narrow-guitar-87575 Good question. The rke2-cilium values need to move to additionalManifests then? Usually this is also managed by Rancher...
c
note that if you try to use AdditionalManifests for CNI/CPI/CSI config AND AddOn config in the same cluster, they will deploy conflicting HelmChartConfig resources and things will break
👍 1
so, make sure they do not overlap at all
b
@creamy-pencil-82913 So why is it only happening with RKE2 1.34? And only with Rancher 2.13.5, but not with 2.13.4?
d
This is not related to the version change. Any edit operation (through the UI) on the cluster will do it
👍 1
n
@bumpy-portugal-40754 for legacy reasons, I configure everything using files under 
/var/lib/rancher/rke2/server/manifests
and nothing in rancher ui. I started this way long time ago and keeping it this way.
c
that’s the only place you’ve seen it. doesn’t mean thats the only place its happening. Lets let @dry-father-91460 handle triage.
b
Sure. I never said that I identified a root cause or any other occurrences. I told you about the bug and I cancelled all planned upgrades until it's fixed. And it has to be fixed in 2.13 soon because of the Harvester support matrix and lifecycle.
c
right but I wouldn’t make any claims about it “only” happening with 1.34 + 2.13.5
b
I did a couple of tests. So I can prove at least that the same cluster upgrades work with 2.13.4 and the same RKE2 versions. And the same was working with 2.12.4.
d
I know what the problem is, I am fixing it. Once I have a fix, I will update you here
For now, I will recommend not upgrading Rancher to 2.13.5 or 2.14 if you have add ons configured. If you have already upgraded, please do not edit the clusters in the UI. This might be an overreaction, but I will let you know once I know more
b
Thanks. This is basically ~50% of all managed clusters. I don't think this is an overreaction. It's reproducible and happens in 100% of the clusters with a non-default Cilium config (and no Cilium runs with a default config).
@creamy-pencil-82913 I read 5837 again. Imho it mainly deals with ways to merge values if they differ in old and new versions. This is not the case here, they are the same. The only thing which is remotely connected is Imho the scenario 4, but it describes only the removal of obsolete lines, not the whole config. But this is what's actually happening: There are no obsolete values, and all are removed. So if it's similar, then it's only the fact that scenario 4 implemention is wrong.
f
I had already upgraded to 2.14.1 and experiencing empty add-on values no matter what I do in rancher GUI. When creating a new cluster, edit existing one, add machine pool etc.
🤕 1
b
@freezing-soccer-22150 I have only tested this with 2.13.5. Please add a comment to the new GitHub issue!
f
ok done
b
I did a few more tests. Rancher 2.13.5 is also removing the addon config of other addons e.g. calico. And it also removes the config for RKE2 versions other than 1.34. I verified it with e.g. upgrading from 1.33.5 to 1.33.6. Imho this deserves a very high prio, because basically any managed guest cluster is in danger with 2.13.5 (and probably 2.14.1)!
d
Hi everyone, I've merged the fix into our master branch and I am trying to get the backports through the CI. The fix should be available for 2.13.6 and 2.14.2. The issue is that UI incorrectly merges previous add-on configs resulting to empty configs when cluster is viewed or edited. If user saves the edited cluster, the new empty config is applied. Editing or viewing as YAML is not affected.
👍 1
f
thanks @dry-father-91460 I saw it in vsphere Add-On. Only remedy was to put backup back of the rancher cluster upgrade before 2.14.1
p
This is still happening? I've been dealing with forms of this cilium rancher gui yaml bug for like, over a year now. Its gotten to the point that we've written multiple guide docs around 'before you upgrade rancher, copy the yaml, then paste it back in'. The absolute worst is if you forget about the bug and it upgrades and wipes all of your cilium stuff away.
b
Again... again.
25 Views
Open in Slack
PreviousNext
Powered by