Hi All, While I start rke2-server service on first node of HA cluster, i see following lines in journalctl.

Sep 13 17:41:21 svmaster rke2[14824]: time="2022-09-13T17:41:21+09:00" level=info msg="Latest etcd manifest deployed"
Sep 13 17:41:22 svmaster rke2[14824]: {"level":"warn","ts":"2022-09-13T17:41:22.837+0900","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {<https://127.0.0.1:2379>  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\". Reconnecting..."}

Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Stopped tunnel to 127.0.0.1:9345"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Proxy done" err="context canceled" url="<wss://127.0.0.1:9345/v1-rke2/connect>"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Connecting to proxy" url="<wss://192.168.7.15:9345/v1-rke2/connect>"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Handling backend connection request [svmaster]"

And in agent journalctl logs following are displayed.

Sep 13 18:03:02 svagent rke2[2659960]: W0913 18:03:02.914417 2659960 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {<https://127.0.0.1:2379>  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

Sep 13 18:03:06 svagent rke2[2659960]: time="2022-09-13T18:03:06+09:00" level=debug msg="Wrote ping"

any inputs to resolve?

The first one is normal, you’ll see errors connecting to etcd on the server at

<https://127.0.0.1:2379>

until the etcd pod is up. The second one is normal as things bootstrap; you’ll see it switch the load-balancer tunnel connection over to the new endpoint.

In the Restored cluster observed those error messages in server, so thought of confirming, whether it’s good to ignore or not. -- one of the agent, i did restarted services but it failed to join and observed those error messages.

did you accidentally start rke2-server service on the agent at some point?

not very much sure about that incident, but in agent nodes if we execute

systemctl restart rke2-server

it will fail right, since because it can’t find the

rke2-server

unit file?

depends on how you install it. If you install from tarball, both units are available.

oh, yeah just now noticed that in a server,

-rw-r--r-- 1 root root  11 Jul  5 05:27 /usr/local/lib/systemd/system/rke2-agent.env
-rw-r--r-- 1 root root 870 Jul  5 05:27 /usr/local/lib/systemd/system/rke2-agent.service
-rw-r--r-- 1 root root  11 Jul  5 05:27 /usr/local/lib/systemd/system/rke2-server.env
-rw-r--r-- 1 root root 871 Jul  5 05:27 /usr/local/lib/systemd/system/rke2-server.service

That’s really weird to have it, since because we have an understanding in RKE2 either a node can be server/agent but not both. I strongly believe there is a possibility of hitting wrong service files both in server and agent nodes. Irrespective of installer it’s good to have either

server

or

agent

unit files only to avoid such mishap.

no, there is just one tarball and it contains everything including both units, unlike the RPM which is split into three (common/server/agent). The rancher provisioning framework exclusively uses the tarball and needs to be able to install a single package and then later configure the node to perform the appropriate roles.

As a part of configuring appropriate roles, its good to remove unwanted unit files. 🙂 Looks its good to have this documented in the tarball procedure to remove those unwanted unit files.

we do support switching agents to servers by running

rke2 server

instead of

rke2 agent

, but if you prefer not to have the other files around, you’re free to remove them

sure got it, for improvement and awareness for users about those unit file existence in the system it’s good to document.

no, switching back is not currently supported, there is a bunch of component cleanup that needs to be handled.

sure got it, thank you.