Hey all I have a cluster that i created on a docker install rancher-users #k3s

Title

red-boots-23091

10/08/2022, 11:50 AM

Hey all, I have a cluster that i created on a docker install within v2.5.7. I have used this for testing for around 1 year now. However at the end of August I started getting problems with it (kubectl connection refused and various crash loops). I cant seem to resolve these. I assume it is to do with updates I made to my server packages. Long story short I am trying to upgrade my Rancher server. However no matter what I do I cant reliably connect to rancher UI and when I can i still have trouble with kubectl. I am running on Ubuntu 20.04.5 LTS (GNU/Linux 5.4.0-125-generic x86_64). Docker Engine Version: 20.10.12 and I am trying to spin up rancher v2.6.8. These are my server logs https://pastebin.com/cDgYiJCb. Does anyone have any guidance

cool-petabyte-32540

10/08/2022, 4:17 PM

How did you try to upgrade the rancher server from v2.5.7 to v2.6.8?

red-boots-23091

10/08/2022, 4:22 PM

Thank you for your reply. I followed this path http://docs.ranchermanager.rancher.io//getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher

cool-petabyte-32540

10/08/2022, 5:57 PM

First try upgrading to v2.6.3.

red-boots-23091

10/08/2022, 6:00 PM

I did I get the same results no matter what version I upgrade too

cool-petabyte-32540

10/08/2022, 6:08 PM

Try this workaround https://github.com/rancher/rancher/issues/29223#issuecomment-969901491

red-boots-23091

10/08/2022, 6:17 PM

Thank you for the suggestion. I have also tried this. Rancher comes up but kubectl still doesnt work and I get connection refused in the logs. The logs I posted earlier did not have the grub edits. I will post logs from after this grub update

2022/10/08 19:42:24 [INFO] Rancher version v2.6.8 (f650ce839) is starting

2022/10/08 19:42:24 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}

2022/10/08 19:42:24 [INFO] Listening on /tmp/log.sock

2022/10/08 19:42:24 [INFO] Waiting for server to become available: Get "<https://127.0.0.1:6444/version?timeout=15m0s>": dial tcp 127.0.0.1:6444: connect: connection refused2022/10/08 19:46:01 [INFO] Running in single server mode, will not peer connections

2022/10/08 19:46:08 [INFO] Applying CRD <http://features.management.cattle.io|features.management.cattle.io>

2022/10/08 19:46:59 [FATAL] k3s exited with: exit status 1

this was my docker run command, for reference

docker run -d --volumes-from rancher-data --restart=unless-stopped -p 9090:80 -p 9091:443 --privileged -v /opt/rancher:/var/lib/rancher --name=rancher_docker_server rancher/rancher:v2.6.8

cool-petabyte-32540

10/09/2022, 1:02 AM

Following issue seems same. Try upgrading to 2.7.0-rc3. If it doesn't work, not sure how this issue can be solved. https://github.com/rancher/rancher/issues/36238

red-boots-23091

10/09/2022, 7:39 AM

Thank you for advice I tried 2.7- head yesterday after seeing this github issue. However got lots of etcd errors. I will try this version thank you for your suggestions

cool-petabyte-32540

10/09/2022, 8:40 AM

Try with rc3 version

v2.7-head is not stable

red-boots-23091

10/09/2022, 10:40 AM

Should I upgrade view versions or straight from 2.5.7?

cool-petabyte-32540

10/09/2022, 11:00 AM

Straight from 2.5.7

👍 1

red-boots-23091

10/11/2022, 1:26 PM

Hi V2.7-rc3 also did not work, however I managed to get v2.5.7 working by deleting the

dynamic-cert.json

and adding

GRUB_CMDLINE_LINUX="cgroup_memory=1 cgroup_enable=memory swapaccount=1 systemd.unified_cgroup_hierarchy=0"

however its not stable

cool-petabyte-32540

10/11/2022, 1:27 PM

Okay

red-boots-23091

10/11/2022, 1:28 PM

i will pastebin logs maybe you will see something else that might help further

thank you for your help so far

https://pastebin.com/uqHPCHbp

2 Views

#k3s Join Slack