hi team, I have a cluster that won't update in Rancher.

Failed to validate cluster: cluster version must be at least v1.23 to use PodSecurity in RKE

it looks like the cluster is a node-driver cluster. So I do not think you can use the external RKE CLI to upgrade the cluster. If it is acceptable to upgrade the cluster to 1.23, you can do it in rancher UI by editing the cluster and change the Kubernetes version t o1.23. Otherwise, you can consider rolling rancher back to the previous version if you have the backup file.

hm rolling back isn't really an option anymore, we've upgraded other clusters already...

well, the error happens because Rancher detects that the cluster version is < 1.23 and

spec.Services.KubeAPI.PodSecurityConfiguration

is set. Do you set the

Pod Security Admission Configuration Template

when editing the cluster in Rancher ?

hm so it's based on an rke template so I can't edit as yaml. any relevant info from the

apiVersion: <http://management.cattle.io/v3|management.cattle.io/v3>
kind: Cluster

I can share, though?

can you share the MgmtCluster object?

yes, I'll redact a few thing first though

Thanks. I see the k8s version is set to v1.23.16-rancher2-2 and

spec.Services.KubeAPI.PodSecurityConfiguration

is not set, which is good. Now I wonder if spec.Services.KubeAPI.PodSecurityConfiguration is set in the clusterTemplateRevision used for the cluster. Can you check it?

doesn't seem like it

kubeApi:
          secretsEncryptionConfig: {}
          serviceNodePortRange: 30000-32767

OK. I may know what is wrong ,and it might be a potential bug. For you cluster, I think it is still functional even the UI shows the upgrade fails, the upgrade does not pass the pre-flight validation. Can you check if you can still open the cluster in rancher UI, ie. click on the cluster name and open its cluster dashboard?

yes the cluster still works, it's just stuck in Error in the Clulster Management tab

good to know it is still working. I am going to use the following GH issue to track. https://github.com/rancher/rancher/issues/41292 please stay tuned 🙂

okay thanks!

can you try this: edit the cluster to changing the k8s version back to the previous v1.22.x, and save? Rancher UI might not show 1.22 in the dropdown list anymore, so you need to edit the cluster as YAML to change the value. ( I think it is doable even clustertemplate is used)

I'll give that a try, thanks!