are there any solution to the system upgrade controller cont rancher-users #rke2

Title

bright-whale-83501

09/28/2022, 6:23 PM

are there any solution to the system upgrade controller continuously trying to upgrade exe-files like kubelet.exe on and on?

creamy-pencil-82913

09/28/2022, 6:50 PM

the SUC does not replace anything other than the rke2 binary…

Can you clarify exactly what you’re seeing? It sounds like you’re talking about Windows which is even more confusing.

bright-whale-83501

09/29/2022, 10:57 AM

Hi. Yes this is a windows worker. With rke2 on.

All files inside C:\var\lib\rancher\rke2\data\v1.24.4-rke2r1-windows-amd64-908a56a4a1b9\bin are being subject for upgrade by rke2

I check the eventlogs recorded from provider rke2, and the message is "failed to extract runtime image: open C:\var\lib\rancher\rke2\data\v1.24.4-rke2r1-windows-amd64-908a56a4a1b9\bin\kubelet.exe: The process cannot access the file because it is being used by another process"

If I stop kubelet process, I'm able to delete the exe-file manually. Then the new kubelet.exe are extracted and started. After that, the same problem occur again. rke2 are trying to upgrade kubelet.exe which is in use. On and on

creamy-pencil-82913

09/29/2022, 9:39 PM

That is not done by the SUC. Every RKE2 release includes a bunch of critical host binaries, in

C:\var\lib\rancher\rke2\data\<version>

. These files, if they don’t exist, are re-extracted during startup. If they’re still in use, that indicates that another instance of RKE2 is still running. The kubelet and containerd should all exit when rke2 stops; if they don not then I suspect something else is going on.

bright-whale-83501

10/04/2022, 7:32 PM

It happens pretty often. I just now edited the cluster, and I see in the logs failed to extract runtime image: open C:\var\lib\rancher\rke2\data\v1.22.13-rke2r1-windows-amd64-273269a46386\bin\calico-node.exe: The process cannot access the file because it is being used by another process. The change I did, were to add a private registry from within the Rancher UI.

creamy-pencil-82913

10/04/2022, 7:40 PM

would you mind opening a GH issue

This sounds like a provisioned cluster, not an imported cluster. Are you sure it’s the system upgrade controller doing it and not the rancher system agent?

bright-whale-83501

10/04/2022, 9:00 PM

Looks like this: https://github.com/rancher/rke2/issues/2571 and I run rke.exe v1.22.130rke2r1

@creamy-pencil-82913 it's a provisioned cluster. Using rancher url. I removed all deployments that were named anything like "upgrade". After that the system settled down (cpu wise) but as soon as I reconfigure the cluster, I hit this problem

creamy-pencil-82913

10/04/2022, 9:06 PM

ok. then it’s not system upgrade controller that’s doing it, which was what I was confused about.

For some reason it’s re-extracting the binaries every time the node is reconfigured, which it should not be doing.

Can you open an issue describing what actions you’re taking and what happens?

23 Views

#rke2 Join Slack