https://rancher.com/ logo
Title
r

red-boots-23091

10/08/2022, 11:50 AM
Hey all, I have a cluster that i created on a docker install within v2.5.7. I have used this for testing for around 1 year now. However at the end of August I started getting problems with it (kubectl connection refused and various crash loops). I cant seem to resolve these. I assume it is to do with updates I made to my server packages. Long story short I am trying to upgrade my Rancher server. However no matter what I do I cant reliably connect to rancher UI and when I can i still have trouble with kubectl. I am running on Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-125-generic x86_64). Docker Engine Version: 20.10.12 and I am trying to spin up rancher v2.6.8. These are my server logs https://pastebin.com/cDgYiJCb. Does anyone have any guidance
c

cool-petabyte-32540

10/08/2022, 4:17 PM
How did you try to upgrade the rancher server from v2.5.7 to v2.6.8?
c

cool-petabyte-32540

10/08/2022, 5:57 PM
First try upgrading to v2.6.3.
r

red-boots-23091

10/08/2022, 6:00 PM
I did I get the same results no matter what version I upgrade too
c

cool-petabyte-32540

10/08/2022, 6:08 PM
r

red-boots-23091

10/08/2022, 6:17 PM
Thank you for the suggestion. I have also tried this. Rancher comes up but kubectl still doesnt work and I get connection refused in the logs. The logs I posted earlier did not have the grub edits. I will post logs from after this grub update
2022/10/08 19:42:24 [INFO] Rancher version v2.6.8 (f650ce839) is starting
2022/10/08 19:42:24 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
2022/10/08 19:42:24 [INFO] Listening on /tmp/log.sock
2022/10/08 19:42:24 [INFO] Waiting for server to become available: Get "<https://127.0.0.1:6444/version?timeout=15m0s>": dial tcp 127.0.0.1:6444: connect: connection refused2022/10/08 19:46:01 [INFO] Running in single server mode, will not peer connections
2022/10/08 19:46:08 [INFO] Applying CRD <http://features.management.cattle.io|features.management.cattle.io>
2022/10/08 19:46:59 [FATAL] k3s exited with: exit status 1
this was my docker run command, for reference
docker run -d --volumes-from rancher-data --restart=unless-stopped -p 9090:80 -p 9091:443 --privileged -v /opt/rancher:/var/lib/rancher --name=rancher_docker_server rancher/rancher:v2.6.8
c

cool-petabyte-32540

10/09/2022, 1:02 AM
Following issue seems same. Try upgrading to 2.7.0-rc3. If it doesn't work, not sure how this issue can be solved. https://github.com/rancher/rancher/issues/36238
r

red-boots-23091

10/09/2022, 7:39 AM
Thank you for advice I tried 2.7- head yesterday after seeing this github issue. However got lots of etcd errors. I will try this version thank you for your suggestions
c

cool-petabyte-32540

10/09/2022, 8:40 AM
Try with rc3 version
v2.7-head is not stable
r

red-boots-23091

10/09/2022, 10:40 AM
Should I upgrade view versions or straight from 2.5.7?
c

cool-petabyte-32540

10/09/2022, 11:00 AM
Straight from 2.5.7
­čĹŹ 1
r

red-boots-23091

10/11/2022, 1:26 PM
Hi V2.7-rc3 also did not work, however I managed to get v2.5.7 working by deleting the
dynamic-cert.json
and adding
GRUB_CMDLINE_LINUX="cgroup_memory=1 cgroup_enable=memory swapaccount=1 systemd.unified_cgroup_hierarchy=0"
however its not stable
c

cool-petabyte-32540

10/11/2022, 1:27 PM
Okay
r

red-boots-23091

10/11/2022, 1:28 PM
i will pastebin logs maybe you will see something else that might help further
thank you for your help so far