rough-cricket-22538
11/01/2022, 10:54 PMsalmon-noon-33588
11/02/2022, 1:00 AMcattle-cluster-agent
has lost its mind in our sandbox cluster to the point where the cluster becomes unusuable.
Whenever it's running, the CPU usage of all of the api-servers in the cluster are pegged at 100%. I'm not sure if this is requests coming in from Rancher or the agent. There don't seem to be any suspicious logs except for the persistent error syncing 'system-library': handler system-image-upgrade-catalog-controller: upgrade cluster {cluster} system service alerting failing: template system-library-rancher-monitoring incompatible with rancher version or cluster's [{cluster}] kubernetes version, requeueing
. Oddly, we get this message for the cluster whether or not its agent is running.
After a little bit, we start seeing errors that seem related to the API server just being too busy, things like:
• Unexpected error when reading response body: context canceled
• "Reflector ListAndWatch" name:pkg/mod/github.com/rancher/client-go@v1.24.0-rancher1/tools/cache/reflector.go:168 (02-Nov-2022 00:36:47.268) (total time: 29608ms)...
And others. Is it possible to determine what the agent is doing that's causing this? I'm wondering if these nodes are a tad underprovisioned at 2 cores and 8GB of RAM? That seems weird though, they've been fine for a few years now. Also:
Rancher v2.6.7 and Kubernetes v1.22.11.salmon-noon-33588
11/02/2022, 1:00 AMfreezing-activity-5466
11/02/2022, 1:49 AMcurved-daybreak-34183
11/02/2022, 6:41 AMbland-summer-47692
11/02/2022, 7:24 AMbland-summer-47692
11/02/2022, 7:24 AMclean-lawyer-76009
11/02/2022, 10:28 AMsudo docker run -d --restart=unless-stopped -p 80:80 -p 443:443 --privileged rancher/rancher
silly-jordan-81965
11/02/2022, 2:23 PMError from server (InternalError): an error on the server ("unable to create impersonator account: error getting service account token: error getting secret: Get \"<https://10.43.0.1:443/api/v1/namespaces/cattle-impersonation-system/secrets/cattle-impersonation-u-7xsnq-token>\": net/http: TLS handshake timeout") has prevented the request from succeeding (get nodes)
Anyone know why ?hundreds-mouse-3032
11/02/2022, 2:57 PMfamous-gold-48752
11/02/2022, 3:32 PMError from server: error dialing backend: proxy error from 78.47.195.214:6443 while dialing 195.201.35.245:10250, code 503: 503 Service Unavailable
famous-gold-48752
11/02/2022, 3:32 PMfamous-gold-48752
11/02/2022, 5:18 PMhundreds-state-15112
11/02/2022, 6:47 PMlimited-eye-44568
11/02/2022, 11:59 PM}
2022-11-02T23:35:39.575Z: Waited more than 60 secs for nerdctl images to succeed. Giving up.
2022-11-02T23:35:39.575Z: Running: wsl.exe --distribution rancher-desktop --exec /sbin/rc-update --update
2022-11-02T23:35:39.769Z: Running: wsl.exe --distribution rancher-desktop --exec /usr/local/bin/wsl-service k3s start
2022-11-02T23:35:46.113Z: Capturing output: wsl.exe --distribution rancher-desktop --exec cat /proc/net/route
2022-11-02T23:35:46.329Z: Capturing output: wsl.exe --distribution rancher-desktop --exec cat /proc/net/fib_trie
2022-11-02T23:36:07.582Z: Capturing output: wsl.exe --distribution rancher-desktop --exec /bin/sh -c if test -r /etc/rancher/k3s/k3s.yaml; then echo yes; else echo no; fi
2022-11-02T23:36:07.784Z: Capturing output: wsl.exe --distribution rancher-desktop --exec wslpath -a -u C:\Users\marzamor\AppData\Local\Programs\Rancher Desktop\resources\resources\linux\wsl-helper
2022-11-02T23:36:07.995Z: Capturing output: wsl.exe --distribution rancher-desktop --exec /mnt/c/Users/marzamor/AppData/Local/Programs/Rancher Desktop/resources/resources/linux/wsl-helper k3s kubeconfig
silly-jordan-81965
11/03/2022, 9:46 AMlively-night-78214
11/03/2022, 1:05 PMicy-iron-20239
11/03/2022, 4:12 PMwide-kitchen-20738
11/03/2022, 5:33 PM/wal
folder but with no success
• This is causing some issue in autoscaling i.e horizontal pod auto scaler
2. New users cannot login to rancher
• We are not able to add new users using their github account or even manually to rancher
• We see this issue - https://forums.rancher.com/t/x509-certificate-has-expired-or-is-not-yet-valid/20518
• We already tried the solution from this post and it did not work
We are new to rancher. The guy who has setup the entire thing is not in the team no more. So we are trying trial and error with zero knowledge. It would be helpful if you guys help us resolving this.
Thanks..miniature-zebra-6218
11/03/2022, 5:50 PMdocker run ... -v ...
). How can that be done in a K8S deployment ?limited-eye-27484
11/03/2022, 6:22 PMlimited-eye-27484
11/03/2022, 6:36 PMclean-ability-26001
11/03/2022, 7:06 PMclean-ability-26001
11/03/2022, 7:07 PMclean-ability-26001
11/03/2022, 7:07 PMhundreds-state-15112
11/03/2022, 10:28 PMrich-thailand-55018
11/04/2022, 1:15 AMdamp-magician-5939
11/04/2022, 1:46 PMagreeable-byte-93427
11/04/2022, 2:24 PMclean-painting-58815
11/04/2022, 2:48 PM