I currently have all my clusters updating simultan...
# general
b
I currently have all my clusters updating simultaneously on Rancher 2.12.0, all seemingly stuck with the message "KDM release data is empty for v1.33.3+rke2r1" (or slight variation in release), all with one controller stating "Reconciling. Waiting for plan to be applied". Should I be worried?
c
did you hardcode a custom KDM URL in your settings, and forget to update it when upgrading rancher to 2.12?
image.png
the message indicates that the data.json in use by your rancher instance is missing data for v1.33.3+rke2r1.
Copy code
brandond@dev01:~$ curl -sL <https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json> | jq '.rke2.releases[].version' | grep v1.33
"v1.33.0+rke2r1"
"v1.33.1+rke2r1"
"v1.33.2+rke2r1"
"v1.33.3+rke2r1"
b
I have changed the rke-metadata-config entry from 2.11 to the one that you've shown. There were a number of other 2.11 urls too - ui-index and ui-dashboard-index. I definitely didn't deliberately hardcode anything, just tried following instructions to install, then to update. I might well have missed something. Is there a doc showing what I should be doing in these circumstances? Do I just wait until the next update to see if it clears up?
c
those would normally be handled automatically as part of the upgrade. if they were not, that indicates someone went in and changed them at some point so that they are not using the default that is built into the product. If that is done, you need to update them manually after upgrading
b
They definitely weren't changed manually. I must have missed something when I did the upgrade, not entered something I should have. Should I just wait for the daily update to make the warnings go away?
I changed the update period and the warnings haven't gone away. More problematically, it stops installation of a new cluster as it's paused with the same warning. Even tried downloading data.json and swapping it with the one in place on the cluster server which was waiting for the change. Is there something I can run to kick-start this again?
Still haven't got anywhere with this. I can't even delete the node from the cluster. It seems that it will continue "Waiting for the plan to be applied" forever. How can I apply the plan, or edit something so that it is ignored, or considered complete? Is there a process I should stop or restart?
c
Are you running Rancher as a standalone docker container? Or deployed via helm chart?
You need to fix the metadata issue first and foremost
b
If that's just adjusting the global variables via Settings, that was done after you first mentioned it. If there's more to do, if you could just point me towards the right part of the docs I'll run through that. ui-index https://releases.rancher.com/ui/release-2.12/index.html ui-dashboard-index https://releases.rancher.com/dashboard/release-2.12/index.html rke-metadata-config
Copy code
{
  "refresh-interval-minutes": "144",
  "url": "<https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json>"
}
{
  "refresh-interval-minutes": "144",
  "url": "<https://releases.rancher.com/kontainer-driver-metadata/release-v2.12/data.json>"
}
I cut down 1440 to 144. It's a helm installation, three Rancher servers, three MariaDB as a Galera cluster in place for etcd (though only the first was in the initial Rancher configuration).
I've seen that v2.12.1 has just been released. Would an upgrade clear this or break it even more?
s
Hi guys! I ran into the same issue after upgrading to Rancher v2.12.1 and fixed it like this: 1. First I verified the
rke-metadata-config
was set to v2.12 — it already was, so no change needed. 2. In Rancher: ☰ → Cluster Management → Drivers → Refresh Kubernetes Metadata. 3. After it refreshed, I went to the local cluster (where Rancher runs) and redeployed the Fleet deployments — both the agent and the controller. 4. Finally, I redeployed the
rancher
deployment
itself. After that, a quick refresh showed all my RKE2 clusters finishing their updates and returning to Active with no messages. Hope this helps! @best-appointment-66450
b
Thank you very much! I'll take a poke at it.
👍 1
That was it! Superb! Thank you so much for the incredibly simple, duh, I should have thought of it myself solution. Brilliant! I can get on with getting a new cluster in place now!🍻
🙌 1
s
Awesome! Glad that did the trick. Good luck with the new cluster! 🍻
🙏 1