https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
😭 1
Options include: • Insufficient permissions to use multipart object upload APIs • Your storage provider doesn’t support multipart object uploads • Bug in your storage provider
I’m assuming whatever you’re using isn’t actually S3 but something that claims to be “compatible” with S3?
b
It is. It's a Netapp as far as I could find out. But isn't Minio also "just" S3 compatible? Since it's working most of the time but not always, I suspect a bug at the provider side. But when I give them this error message, they send me back to do my homework...
Do I get better logs when I set 
debug: true
in config.yaml? Or is there a way to be more specific and just debug the S3 part?
c
this is the minio golang s3 client library. it has nothing to do with their java-based server product.
b
No problem (for now). Let's talk about ways to debug this.
c
I would probably engage with your storage vendor to figure out how to get additional logs from their side.
this is just an error being returned by the storage backend
b
I'm trying...
c
I don’t have an account though so I can’t see what it actually says
looks like that whole message is coming back from your storage system verbatim
so yeah…. talk to your vendor
b
Me neither... the link might be helpful. Thanks!
FYI: The mentioned netapp url suggested to upgrade their software to the (probably) newest level 11.6 at that time (4 years ago). The minio client version at that time was 
Minio client version 2021-06-13T17-48-22Z
Just for completeness: What minio s3 go library version do you use currently (in 1.26.13)?
c
that version is from August 2022 so I guess we could take a look at updating it for the May releases, just to see if that helps
I will also note that Kubernetes 1.26 is end of life, so you’d probably want to work on upgrading that cluster anyway
b
I would love to go to >1.26 if it was possible. But I'm stuck here because of some dependencies from Rancher and Harvester.
c
well you’re not going to be able to use the updated version of minio-go then, as we stopped putting out 1.26 releases of RKE2 a couple months ago, when the Kubernetes version went EOL
b
I know, unfortunately. But I'm collecting information just in case I need to provide it to the storage vendor.
c
ok. I’ll create an issue to track updating minio-go, try it when you can.
b
Ok. It all depends on you (all Suse)... I'll update if you guys let me 🙂
c
what version of Rancher are you on that doesn’t support anything newer than 1.26? Recent releases of Rancher support up to 1.28
b
Rancher 2.7.9 because of Harvester 1.2.1 and because I can't use the prime repo at the moment.
You guys make it really hard for me. 2.7.11 would be fine, but I have no chance to get it (air gapped with limited access to images).
c
ah yeah that’s… complicated. Not planning on 2.8 yet?
b
We have guest clusters in Harvester with still 1.23 and 1.24. This prevents us from updating to Rancher 2.8. And nothing is harder to convince guest cluster owners to update...
c
ah yeah those are going to be a real albatross around your neck until they upgrade. those are VERY end of life!
😭 1
b
FYI: We also do Rancher backups to the same S3 storage. Rancher is using github.com/minio/minio-go/v7 v7.0.10, which is older than yours. But we don't see the issue there. Not asking you to downgrade, but maybe taking a look to the minio client version in general :-) Our storage people changed some "consistency level" on their side. I'm waiting for the issue to either come back or disappear.
11 Views
Open in Slack
PreviousNext
Powered by