I am trying to move my rancher install from one cluster to another with rancher-backup. I'm getting errors restoring because it can't create the CRDs. I see that apiextensions.k8s.io/v1beta1 is unavailable on EKS 1.22, but what I can't figure out is why on the old cluster the CRDs are defined with apiextensions.k8s.io/v1 but the json objects in my backup tarball is defined with the v1beta1 version. For example this is how apps.catalog.cattle.io is defined when I inspect it in the rancher UI

apiVersion: <http://apiextensions.k8s.io/v1|apiextensions.k8s.io/v1>
kind: CustomResourceDefinition
metadata:
... trimmed ...
  name: <http://apps.catalog.cattle.io|apps.catalog.cattle.io>

In my backup file I have ./customresourcedefinitions.apiextensions.k8s.io#v1beta1/apps.catalog.cattle.io.json that starts with this:

{"apiVersion":"<http://apiextensions.k8s.io/v1beta1|apiextensions.k8s.io/v1beta1>","kind":"CustomResourceDefinition","metadata":  ...trimmed...

I can't work out why it is defined in my source cluster as

v1

but recorded in the backup set as

v1beta1

.

Both clusters same k8s version? https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122

I think I found a likely cause. It wasn't until rancher-backup v2.1.0-rc1 that the CRDs were saved with apiextensions.k8s.io/v1 instead of v1beta1. You can see the commit here. We have inherited a rancher installation on k3s 1.18.20 and we're trying to migrate to EKS 1.22. Based on what I see, the newest version of rancher-backup that I can run on the k3s cluster is 2.0.1 because changes in 2.1.0 are incompatible with 1.18.x. If we had finished this even 2 weeks ago we could have created a new EKS 1.21 and we likely could have restored the backup stored with apiVersion apiextensions.k8s.io/v1beta1. It looks like we just need to commit to upgrading the k3s cluster we're trying to move out of to 1.20 and then we can attempt a backup using rancher-backup 2.1.x series which exports all of the CRDs that are currently throwing errors with apiVersion apiextensions.k8s.io/v1 which will be restorable on the EKS 1.22 cluster we're moving in to. I know this is a messy picture and that there are risks lurking that mean we might still not be out of the woods, but that's the unfortunate lay of the land we have.