green-kangaroo-48114
05/06/2022, 4:15 AMgorgeous-match-58406
05/11/2022, 8:26 PMbland-painting-61617
05/13/2022, 10:46 AMWaiting for cluster agent to connect
and in the machine yaml I can see
- lastTransitionTime: "2022-05-13T10:34:16Z"
reason: WaitingForNodeRef
severity: Info
status: "False"
type: NodeHealthy
Guess I'll have to ssh into the VM to check, which component do I need to look at?strong-boots-98959
05/18/2022, 5:23 PMearly-tiger-9643
05/25/2022, 12:34 PMchilly-iron-96675
06/13/2022, 7:32 PMv1.3.2
to v1.3.11
and afterwards I found that I couldn't connect to the ingress anymore.
ingress:
provider: nginx
network_mode: hostPort
http_port: 80
https_port: 443
The pods are running, I've explicitly set the ports in the cluster.yml but nginx doesn't seem to be answering TCP requests on either 80 or 443.
Has anyone else experienced ingress unavailable after an rke up
?incalculable-painting-771
06/25/2022, 2:16 PMechoing-address-20868
06/25/2022, 3:47 PMwide-easter-7639
06/30/2022, 3:54 PMtall-memory-52585
07/08/2022, 12:47 AMelegant-rose-89654
07/14/2022, 7:44 AMCluster agent is not connected
. Can someone tell me how to debug this? Thank youbig-dawn-71012
07/15/2022, 5:02 PMrke up
fails at this point:
WARN[0304] [etcd] host [10.10.1.86] failed to check etcd health: failed to get /health for host [10.10.154.86]: Get "<https://10.10.1.86:2379/health>": net/http: TLS handshake timeout
FATA[0304] [etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts [10.10.1.86] failed to report healthy. Check etcd container logs on each host for more information
Checking the etcd logs on 10.10.1.86, I see lots of "connection refused" failure from rafthttp/probing
as follows:
{"level":"warn","ts":"2022-07-15T17:00:22.813Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"fa5a1d6f91d877ef","rtt":"0s","error":"dial tcp 10.10.1.88:2380: connect: connection refused"}
{"level":"warn","ts":"2022-07-15T17:00:22.813Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"b8ffb4898a3dabab","rtt":"0s","error":"dial tcp 10.10.1.87:2380: connect: connection refused"}
where 10.10.1.87 and 88 are the two worker nodes (non etcd, non controlplane). Why in the world is it trying to do those probes against those nodes?rich-army-97106
07/19/2022, 1:26 PMrancher-cluster.yaml
https://github.com/rancher/rke/issues/2976brave-country-95392
07/19/2022, 1:33 PMnetstat -nltp
• Found some suggestions on GitHub to use host network for the controller instead of host ports. Tried it, doesn’t change anything. However ports 80 and 443 shows on netstat.
• I have applied a workaround by manually creating a NodePort service for the controller and re-configuring our load balancer trough to use the service.
Anyone has any ideas why this might happen or any additional steps to debug?rich-army-97106
07/21/2022, 8:06 AMacceptable-evening-17718
07/22/2022, 9:18 AMbored-nest-98612
08/01/2022, 6:14 PMrich-shoe-36510
08/03/2022, 5:34 PMclean-airplane-85370
08/04/2022, 1:06 AMfreezing-wolf-83208
08/08/2022, 10:29 AMstale-fish-49559
08/12/2022, 7:14 PMresponse from daemon: path /var/lib/kubelet is mounted on /var/lib/kubelet but it is not a shared mount
; however, mount --make-rshared /
is the suggested fix, it does not solve my problem. Any ideas?tall-doctor-28108
08/17/2022, 1:45 PMrke-1.3.13 config --list-version --all
. But it successfully goes through when I run rke up
and I'm not seeing any problems yet.freezing-wolf-83208
08/20/2022, 7:21 AMmysterious-shoe-77682
08/30/2022, 4:04 PMspec.externalTrafficPolicy
set to Local
. I have the pod running on each node with a DaemonSet, and when Local
is set, it seems all packets are dropped. Things work with Cluster
, but the source IP is NATed, which we dont want.
Any ideas as to where I should look first?rich-shoe-36510
09/05/2022, 5:29 PMancient-bird-9943
09/07/2022, 2:19 AMfailed to set up SSH tunneling for host [192.168.86.82]: Can't retrieve Docker Info: error during connect: Get "<http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info>": Unable to access the service on /var/run/docker.sock. The service might be still starting up. Error: ssh: rejected: connect failed (open failed)
on all three of my nodes yet when I run ssh rke@192.168.86.82 'docker ps -a'
for any of them, it returns CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
as expected. Does anyone have any things to try? I've adjusted the sshd_config to have AllowTcpForwarding yes
. I can give more of my cluster.yml file if needed.brave-country-95392
09/09/2022, 8:11 AMdazzling-computer-84464
09/12/2022, 7:32 PM{
"baseType": "error",
"code": "ServerError",
"message": "error getting tokens for user: u-wkdcbhe4uz selector: <http://authn.management.cattle.io/token-userId=u-wkdcbhe4uz|authn.management.cattle.io/token-userId=u-wkdcbhe4uz> err: the server was unable to return a response in the time allotted, but may still be processing the request (get <http://tokens.meta.k8s.io|tokens.meta.k8s.io>)",
"status": 500,
"type": "error"
}
After upgrading to 2.6.6, the api https://rancher.domain.com/v3/tokens or user account & api key page not working anymore. Does anyone know how to fix this issue?
RKE K8 version 1.21dazzling-smartphone-16726
09/13/2022, 3:17 PMCreatecontainererror
CreateContainerError (Error response from daemon: Duplicate mount point: /var/lib/docker/containers)
freezing-wolf-83208
09/14/2022, 6:46 AM