Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Title
c
crooked-cat-21365
06/08/2022, 11:55 AMI have severe problems upgrading my rancher cluster to a new version of rke. Currently it is running rke v1.21.7 and Rancher v2.6.3-patch1. Everytime I try to upgrade the rke cluster it gets bricked somehow, see https://github.com/rancher/rke/issues/2833. This is a production system, i.e. I cannot afford a huge downtime to examine some rke --debug mode, etc. What would you suggest?
c
creamy-pencil-82913
06/08/2022, 5:01 PMHave you opened a support ticket yet?
c
crooked-cat-21365
06/09/2022, 6:20 AMc
creamy-pencil-82913
06/09/2022, 6:27 AMI meant like, an actual ticket for support. We sometimes get folks coming on here with paid support contracts and production outages that haven't created a ticket yet. Community Slack support is best-effort and folks on here are mostly devs or other free users so you're not likely to get a ton of handholding.
c
crooked-cat-21365
06/09/2022, 6:29 AMYou mean I am on my own without paid support contract?
c
creamy-pencil-82913
06/09/2022, 6:35 AMIf you have specific questions they are more likely to get answers. We're not a VC funded startup burning cash for fun, if we gave everyone hands-on production SLAd support for free there would be no money to pay anyone with and there wouldn't be a product at all.
Open source is free as in beer, so the saying goes.
If you can diagnose it to a specific problem with the project it's highly likely to get fixed. Hands-on troubleshooting isn't something we can offer gratis.
c
crooked-cat-21365
06/09/2022, 6:46 AMI understand that you have to make money, esp. after the expensive acquisition.
I had the impression that I am not the only one with this problem. I can help to examine the problem for the benefit of all, including Suse's paying customers. Its a give and take, as for other Open Source projects. IMHO.
c
creamy-pencil-82913
06/09/2022, 6:52 AMI see there was some back and forth on the ticket, with another community user, but it looks like you weren't interested in doing additional troubleshooting on a production system, which is understandable. Hard to say specifically what's going from the info that's in the ticket there.
Will probably require digging around in various pod logs and checking component versions. Older versions of cert-manager in particular aren't compatible with 1.22. cert-manager is a rancher dependency, and as far as I know is not managed by either rancher or rke when you upgrade versions.
c
crooked-cat-21365
06/09/2022, 7:08 AMWhen I started with Rancher cert-manager was not a dependency. It was sufficient to create a secret holding a Digicert certificate to hand-off on the helm install command line. See the 3rd option on https://rancher.com/docs/rancher/v2.5/en/installation/install-rancher-on-k8s/#3-choose-your-ssl-configuration. I would guess that if cert-manager is a problem here, then its because it is not installed.
Right now I am setting up a second cluster to verify Rancher's migration procedure. Using a different host name for the cluster this won't affect my production environment (hopefully).
PS: Bad link, but the current documentation for 2.6 appears to be the same wrt cert-manager.
Hi @creamy-pencil-82913, can you confirm that cert-manager is not a dependency, or is the documentation misleading?
I managed to upgrade my rke cluster running Rancher to v1.21.12-rancher1-1 and Rancher version 2.6.5. Seems to be working fine. If I try to upgrade rke to version v1.23.6-rancher1-1 then Rancher becomes unresponsive, as for version 1.22.x before. The managed clusters cannot be accessed via kubectl anymore (which is unacceptable). The web browser cannot connect to the front-end, either.
FYI: Using the stage system I guess I found out what the problem is: apparmor. For the old kubernetes v1.21 this wasn't a problem, but using v1.22 + it is. After dropping apparmor the web interface is available again.
To be sure I have to upgrade my production system, of course.