https://rancher.com/ logo
Title
m

microscopic-diamond-94749

04/28/2023, 5:59 PM
hi team, I have a cluster that won't update in Rancher.
Failed to validate cluster: cluster version must be at least v1.23 to use PodSecurity in RKE
It seems like the Rancher 2.7.3 bundled RKE is of the 1.4.X kind that does not support K8s 1.22 any more. Any ideas how to proceed? I could update the cluster manually with RKE 1.3.X but how do I do that?
f

future-night-17486

04/28/2023, 6:55 PM
it looks like the cluster is a node-driver cluster. So I do not think you can use the external RKE CLI to upgrade the cluster. If it is acceptable to upgrade the cluster to 1.23, you can do it in rancher UI by editing the cluster and change the Kubernetes version t o1.23. Otherwise, you can consider rolling rancher back to the previous version if you have the backup file.
m

microscopic-diamond-94749

04/28/2023, 7:02 PM
hm rolling back isn't really an option anymore, we've upgraded other clusters already...
I thought using
full-cluster-state
ConfigMap in the
kube-system
might work ?
I got the error when trying to upgrade to 1.23 ( I just realized I tried 1.24 first, but cluster is still stuck in Error State, Maybe etcd restore?)
f

future-night-17486

04/28/2023, 7:10 PM
well, the error happens because Rancher detects that the cluster version is < 1.23 and
spec.Services.KubeAPI.PodSecurityConfiguration
is set. Do you set the
Pod Security Admission Configuration Template
when editing the cluster in Rancher ?
if possible, can you edit the cluster and choose "Edit as YAML", and share the cluster YAML ?
m

microscopic-diamond-94749

04/28/2023, 7:32 PM
hm so it's based on an rke template so I can't edit as yaml. any relevant info from the
apiVersion: <http://management.cattle.io/v3|management.cattle.io/v3>
kind: Cluster
I can share, though?
f

future-night-17486

04/28/2023, 7:32 PM
can you share the MgmtCluster object?
m

microscopic-diamond-94749

04/28/2023, 7:33 PM
yes, I'll redact a few thing first though
cluster.yaml
f

future-night-17486

04/28/2023, 7:44 PM
Thanks. I see the k8s version is set to v1.23.16-rancher2-2 and
spec.Services.KubeAPI.PodSecurityConfiguration
is not set, which is good. Now I wonder if spec.Services.KubeAPI.PodSecurityConfiguration is set in the clusterTemplateRevision used for the cluster. Can you check it?
m

microscopic-diamond-94749

04/28/2023, 7:49 PM
doesn't seem like it
kubeApi:
          secretsEncryptionConfig: {}
          serviceNodePortRange: 30000-32767
f

future-night-17486

04/28/2023, 7:58 PM
OK. I may know what is wrong ,and it might be a potential bug. For you cluster, I think it is still functional even the UI shows the upgrade fails, the upgrade does not pass the pre-flight validation. Can you check if you can still open the cluster in rancher UI, ie. click on the cluster name and open its cluster dashboard?
m

microscopic-diamond-94749

04/28/2023, 8:10 PM
yes the cluster still works, it's just stuck in Error in the Clulster Management tab
f

future-night-17486

04/28/2023, 8:15 PM
good to know it is still working. I am going to use the following GH issue to track. https://github.com/rancher/rancher/issues/41292 please stay tuned 🙂
m

microscopic-diamond-94749

04/28/2023, 8:15 PM
okay thanks!
should I attempts an etcd restore or just leave it for now?
f

future-night-17486

04/28/2023, 8:19 PM
can you try this: edit the cluster to changing the k8s version back to the previous v1.22.x, and save? Rancher UI might not show 1.22 in the dropdown list anymore, so you need to edit the cluster as YAML to change the value. ( I think it is doable even clustertemplate is used)
Screenshot 2023-04-28 at 1.20.34 PM.png
then change the version to the previous value
m

microscopic-diamond-94749

04/28/2023, 8:45 PM
I'll give that a try, thanks!
since I was using an RKETemplate edit as YAML didn't work, but I could set the
spec.kubernetesVersion
in the k8s object directly and now the cluster isn't in
Error
anymore 😛