Hello all, As suggested by a slack user, I'm posting my question here. I am having a bit of trouble creating PVCs with access mode "RWX" in downstream clusters of Rancher v2.7.9 . The downstream clusters' K8s versions range from 1.26.8 - 1.26.13. The clusters' cloud provider is vSphere. The PVCs are using a storage class to provision PVs. The storage class's provisioner is vSphere CSI. After creating, PVCs' status remain pending and no PV creation is triggered. From the error logs of the vsphere-csi-controller created automatically by the "rancher-vsphere-csi" chart, it looks like the csi-controller is trying to speak with vCenter to create container volumes / vSAN file shares, but ends up in errors (see below) After adjusting image tags of vsphere-csi-controller deployment to lower values on the other hand, PVC's trigger creation of PVs and are bound to them without any delay. The default image tags used by rancher-vsphere-csi are v3.0.1 and its supported sidecars containers versions. The Image tags which enable creation of bound PVCs successfully are v2.7.2 and its supported sidecars container versions. vCenter Version is 7.0 U3p Build: 22837322. What is your experience of using vSphere Provisioner to create bound PVCs and creating vSAN file share successfully using vsphere-csi-controller? I look forward to your replies. Error logs of

vsphere.csi-controller

pod:

2024-03-05T15:03:51.933610102Z {"level":"info","time":"2024-03-05T15:03:51.933519537Z","caller":"vanilla/controller.go:1805","msg":"CreateVolume: called with args {Name:pvc-92a703ce-2341-43ae-a572-af4838234308 CapacityRange:required_bytes:5368709120  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Parameters:map[storagepolicyname:K8s-clusters-storage-policy] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"5b7272e0-7229-4546-8dbc-3cb6e1318fa4"}
2024-03-05T15:03:51.934092725Z {"level":"info","time":"2024-03-05T15:03:51.933878029Z","caller":"vanilla/controller.go:1805","msg":"CreateVolume: called with args {Name:pvc-a065755c-db26-4a1b-9498-6654e1ab2551 CapacityRange:required_bytes:5368709120  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Parameters:map[] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"66a7bd8c-8047-496a-a88c-e56732f3d4e3"}
2024-03-05T15:03:51.937904672Z {"level":"info","time":"2024-03-05T15:03:51.937767865Z","caller":"vanilla/controller.go:1805","msg":"CreateVolume: called with args {Name:pvc-0680d14c-eca2-44e5-b43d-7c1818d6c41b CapacityRange:required_bytes:10737418240  VolumeCapabilities:[mount:<fs_type:\"ext4\" > access_mode:<mode:MULTI_NODE_MULTI_WRITER > ] Parameters:map[] Secrets:map[] VolumeContentSource:<nil> AccessibilityRequirements:<nil> XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}","TraceId":"fd77419b-5da9-472c-bad9-112a2780e501"}
2024-03-05T15:03:51.943866111Z panic: runtime error: invalid memory address or nil pointer dereference
2024-03-05T15:03:51.943898566Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x1b22a25]
2024-03-05T15:03:51.943901505Z 
2024-03-05T15:03:51.943904260Z goroutine 270 [running]:
2024-03-05T15:03:51.943907040Z <http://sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).createFileVolume(0xc000034000|sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).createFileVolume(0xc000034000>, {0x26af658, 0xc00093a8d0}, 0xc000486770)
2024-03-05T15:03:51.943927864Z 	/build/pkg/csi/service/vanilla/controller.go:1736 +0xd05
2024-03-05T15:03:51.943943618Z <http://sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).CreateVolume.func1()|sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).CreateVolume.func1()>
2024-03-05T15:03:51.943968699Z 	/build/pkg/csi/service/vanilla/controller.go:1848 +0x3d7
2024-03-05T15:03:51.943974291Z <http://sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).CreateVolume(0xc000034000|sigs.k8s.io/vsphere-csi-driver/v3/pkg/csi/service/vanilla.(*controller).CreateVolume(0xc000034000>, {0x26af658, 0xc00093a750}, 0xc000486770)
2024-03-05T15:03:51.943977838Z 	/build/pkg/csi/service/vanilla/controller.go:1858 +0x1bb
2024-03-05T15:03:51.944082915Z <http://github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x229bba0|github.com/container-storage-interface/spec/lib/go/csi._Controller_CreateVolume_Handler({0x229bba0>?, 0xc000034000}, {0x26af658, 0xc00093a750}, 0xc00092e420, 0x0)
2024-03-05T15:03:51.944089917Z 	/go/pkg/mod/github.com/container-storage-interface/spec@v1.7.0/lib/go/csi/csi.pb.go:5671 +0x170
2024-03-05T15:03:51.944346148Z <http://google.golang.org/grpc.(*Server).processUnaryRPC(0xc000366e00|google.golang.org/grpc.(*Server).processUnaryRPC(0xc000366e00>, {0x26b6218, 0xc000109040}, 0xc00054c120, 0xc0008bb050, 0x38db8a0, 0x0)
2024-03-05T15:03:51.944354703Z 	/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1283 +0xcfe
2024-03-05T15:03:51.944357688Z <http://google.golang.org/grpc.(*Server).handleStream(0xc000366e00|google.golang.org/grpc.(*Server).handleStream(0xc000366e00>, {0x26b6218, 0xc000109040}, 0xc00054c120, 0x0)
2024-03-05T15:03:51.944361009Z 	/go/pkg/mod/google.golang.org/grpc@v1.47.0/server.go:1620 +0xa2f
2024-03-05T15:03:51.944364862Z <http://google.golang.org/grpc.(*Server).serveStreams.func1.2()|google.golang.org/grpc.(*Server).serveStreams.func1.2()>

Regards, Vishesh