rancher-users #rke

green-kangaroo-48114

05/06/2022, 4:15 AM

Hi everyone, I've been trying to deploy a k8s cluster (rancher/hyperkube:v1.20.15-rancher1) with rke 1.3.10. Everything appears to be functional except for metrics-server.

gorgeous-match-58406

05/11/2022, 8:26 PM

Hello all! how to add user in rke2?

bland-painting-61617

05/13/2022, 10:46 AM

Hey guys, I'm having trouble provisioning an RKE2 cluster from rancher since upgrading to 2.6.4 (and now 2.6.5), it's stuck at:

Copy code

Waiting for cluster agent to connect

and in the machine yaml I can see

Copy code

- lastTransitionTime: "2022-05-13T10:34:16Z"
    reason: WaitingForNodeRef
    severity: Info
    status: "False"
    type: NodeHealthy

Guess I'll have to ssh into the VM to check, which component do I need to look at?

strong-boots-98959

05/18/2022, 5:23 PM

https://github.com/rancher/dashboard/issues/5980

early-tiger-9643

05/25/2022, 12:34 PM

Hello all! I want to migrate a Rancher Provisioned RKE cluster on RancherOS to Flatcar Linux. However in the documentation it states that for Flatcar Linux I need to add some extra network options for canal and an extra flex plugin dir for kube-controller according to https://rancher.com/docs/rke/latest/en/os/#flatcar-container-linux. If I add those to my cluster template it will then start reconfiguring existing RancherOS nodes, will it break anything in RancherOS?

🙄 1

chilly-iron-96675

06/13/2022, 7:32 PM

Has anyone encountered ingress unavailable after an RKE upgrade? I've just upgraded a 3-node test cluster from

v1.3.2

v1.3.11

and afterwards I found that I couldn't connect to the ingress anymore.

Copy code

ingress:
    provider: nginx
    network_mode: hostPort
    http_port: 80
    https_port: 443

The pods are running, I've explicitly set the ports in the cluster.yml but nginx doesn't seem to be answering TCP requests on either 80 or 443. Has anyone else experienced ingress unavailable after an
rke up
?

incalculable-painting-771

06/25/2022, 2:16 PM

how can I restore the kube_config_cluster.yml if it gets deleted locally? I mean I still have ssh access to the cluster machine ( it is a one node cluster )

echoing-address-20868

06/25/2022, 3:47 PM

Is there any Terraform module to deploy rke newer than v1.3.0? I’m looking for 1.3.11 or 1.3.12. Terraform registry has v1.3.0 as latest.

wide-easter-7639

06/30/2022, 3:54 PM

Does any one has setup RKE with multiple NICs ?

tall-memory-52585

07/08/2022, 12:47 AM

Hello, is this the latest system images list for rke, https://github.com/rancher/kontainer-driver-metadata/blob/dev-v2.6/rke/k8s_rke_system_images.go Thanks.

elegant-rose-89654

07/14/2022, 7:44 AM

Hi everyone. I wonder if someone can help me with a little debugging. I have Rancher 2.6.3 and today my RKE cluster is unavailable (albeit the applications are working and responding). The message I see is

Cluster agent is not connected

. Can someone tell me how to debug this? Thank you

big-dawn-71012

07/15/2022, 5:02 PM

I'm going mad trying to setup a 3 node rke cluster. One node is etcd + controlplane + worker, the other two are just workers.

rke up

fails at this point:

Copy code

WARN[0304] [etcd] host [10.10.1.86] failed to check etcd health: failed to get /health for host [10.10.154.86]: Get "<https://10.10.1.86:2379/health>": net/http: TLS handshake timeout 
FATA[0304] [etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts [10.10.1.86] failed to report healthy. Check etcd container logs on each host for more information

Checking the etcd logs on 10.10.1.86, I see lots of "connection refused" failure from

rafthttp/probing

as follows:

Copy code

{"level":"warn","ts":"2022-07-15T17:00:22.813Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"fa5a1d6f91d877ef","rtt":"0s","error":"dial tcp 10.10.1.88:2380: connect: connection refused"}
{"level":"warn","ts":"2022-07-15T17:00:22.813Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"b8ffb4898a3dabab","rtt":"0s","error":"dial tcp 10.10.1.87:2380: connect: connection refused"}

where 10.10.1.87 and 88 are the two worker nodes (non etcd, non controlplane). Why in the world is it trying to do those probes against those nodes?

rich-army-97106

07/19/2022, 1:26 PM

I might need someone who points me to the correct point of documentation 😐 I opened the issue but have no idea how I can tackle it. Basically I have old user-addons and dont want to maintain them anymore with the

rancher-cluster.yaml

https://github.com/rancher/rke/issues/2976

brave-country-95392

07/19/2022, 1:33 PM

Hello everyone. Today I have upgraded a RKE cluster from 1.21 to 1.22 with RKE v1.3.12 and after the upgrade, the Nginx Ingress controller became unreachable. Here is what I have gathered so far: • Accessing the controller works from within the pod • Accessing the exposed 80 and 443 ports from the node results in a timeout (not connection refused) • The hostPorts used by the controller doesn’t show using

netstat -nltp

• Found some suggestions on GitHub to use host network for the controller instead of host ports. Tried it, doesn’t change anything. However ports 80 and 443 shows on netstat. • I have applied a workaround by manually creating a NodePort service for the controller and re-configuring our load balancer trough to use the service. Anyone has any ideas why this might happen or any additional steps to debug?

rich-army-97106

07/21/2022, 8:06 AM

looks like rancher is not interested in supporting rke users

🙄 4

😕 1

🤔 1

acceptable-evening-17718

07/22/2022, 9:18 AM

Hello. Is it possible to upgrade imported RKE cluster from Rancher? I don't see the option. Thanks a lot

bored-nest-98612

08/01/2022, 6:14 PM

Im trying to update with new RKE template the cluster but i get This cluster is currently Updating. Cannot proceed with upgrade of controlplane since 1 host(s) cannot be reached prior to upgrade any advice ? 10x 🙂

rich-shoe-36510

08/03/2022, 5:34 PM

Hey all, Is there graceful / supported way to remove/add controlplane/etcd node without tearing all cluster down? It seems, per doc, that only removal of worker nodes can be done in such fashion?

clean-airplane-85370

08/04/2022, 1:06 AM

hi, I just recreated my rke2 v1.23.9+rke2r1 cluster, and seeing all my ingress stop working. It is getting connection refused. Anyone know what is happening? I saw one of the fix is nginx-ingress updated to 4.1.x that have some hostport changes.... I was not able to find any doc on any config change needed on the new version. How can I fix this?

freezing-wolf-83208

08/08/2022, 10:29 AM

Hi I recently upgraded our non-prod rancher to 2.6.6 and when I try to upgrade the k8s from 1.19.4 to 1.23.6, I run into the following error and had to restore to my snapshot The IngressClass "nginx" is invalid: spec.controller: Invalid value: "k8s.io/ingress-nginx": field is immutable any idea what I am missing? I use rke

stale-fish-49559

08/12/2022, 7:14 PM

Hi, i am trying to use RKE CLI to install k8s on a Yocto machine. I keep getting

response from daemon: path /var/lib/kubelet is mounted on /var/lib/kubelet but it is not a shared mount

; however,

mount --make-rshared /

is the suggested fix, it does not solve my problem. Any ideas?

tall-doctor-28108

08/17/2022, 1:45 PM

Am I going to have any issues when I use rke-1.3.13 to provision v1.22.9-rancher1-1 in my cluster.yml? Only v1.22.11-rancher1-1 is listed when I run

rke-1.3.13 config --list-version --all

. But it successfully goes through when I run

rke up

and I'm not seeing any problems yet.

✅ 1

freezing-wolf-83208

08/20/2022, 7:21 AM

trying to upgrade kubernetes on my local cluster - I first downloaded rke 1.3.12, we were using 1.2.2 before. and then on running the rke up command I run into WARN[0196] [etcd] host [161.211.xx.xx] failed to check etcd health: failed to get /health for host [161.211.xx.xx]: Get "https://161.211.xx.xx:2379/health": remote error: tls: bad certificate FATA[0196] [etcd] Failed to bring up Etcd Plane: etcd cluster is unhealthy: hosts [161.211.xx.xx] failed to report healthy. Check etcd container logs on each host for more information anyone run into the same?

mysterious-shoe-77682

08/30/2022, 4:04 PM

Im having trouble in an RKE cluster with a NodePort using

spec.externalTrafficPolicy

set to

Local

. I have the pod running on each node with a DaemonSet, and when

Local

is set, it seems all packets are dropped. Things work with

Cluster

, but the source IP is NATed, which we dont want. Any ideas as to where I should look first?

rich-shoe-36510

09/05/2022, 5:29 PM

Hi people, Hope I'll got an answer -- is it supported to remove and add servers with mixed roles (control plane and worker)?

ancient-bird-9943

09/07/2022, 2:19 AM

Hi everyone, I'm having a hard time getting RKE set up on Ubuntu 22.04. I get the error

failed to set up SSH tunneling for host [192.168.86.82]: Can't retrieve Docker Info: error during connect: Get "<http://%2Fvar%2Frun%2Fdocker.sock/v1.24/info>": Unable to access the service on /var/run/docker.sock. The service might be still starting up. Error: ssh: rejected: connect failed (open failed)

on all three of my nodes yet when I run

ssh rke@192.168.86.82 'docker ps -a'

for any of them, it returns

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

as expected. Does anyone have any things to try? I've adjusted the sshd_config to have

AllowTcpForwarding yes

. I can give more of my cluster.yml file if needed.

brave-country-95392

09/09/2022, 8:11 AM

Hello everyone, I have a requirement to migrate our 3 master nodes to new vms. I plan on creating the new vms, adding them to cluster.yaml, and having a 6 master node setup. Then finally, remove the old masters. Any ideas if this could cause any issues? I couldn’t find any clear documentation on migrating master nodes.

dazzling-computer-84464

09/12/2022, 7:32 PM

Hi there,

Copy code

{
  "baseType": "error",
  "code": "ServerError",
  "message": "error getting tokens for user: u-wkdcbhe4uz selector: <http://authn.management.cattle.io/token-userId=u-wkdcbhe4uz|authn.management.cattle.io/token-userId=u-wkdcbhe4uz>  err: the server was unable to return a response in the time allotted, but may still be processing the request (get <http://tokens.meta.k8s.io|tokens.meta.k8s.io>)",
  "status": 500,
  "type": "error"
}

After upgrading to 2.6.6, the api https://rancher.domain.com/v3/tokens or user account & api key page not working anymore. Does anyone know how to fix this issue? RKE K8 version 1.21

dazzling-smartphone-16726

09/13/2022, 3:17 PM

Hi there has anyone had an issue with the Rancher Logging: Pod: rancher-logging-root-fluentbit is not starting.

Createcontainererror

Copy code

CreateContainerError (Error response from daemon: Duplicate mount point: /var/lib/docker/containers)

freezing-wolf-83208

09/14/2022, 6:46 AM

Is there any migration path/utility for switching from RKE to RKE2? I did see a link https://docs.rke2.io/migration/ mentioned in slack earlier, but get a 404 error on the link.