hey, anyone can help why the cluster agent disconnected error due to the ca cert issue ? Thx

chardmaster@chardmaster:~$ journalctl -u rancher-system-agent.service

Dec 30 02:47:30 chardmaster systemd[1]: Started Rancher System Agent.

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Rancher System Agent version v0.2.13 (4fa9427) is starting"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Starting remote watch of plans"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Starting /v1, Kind=Secret controller"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Detected first start, force-applying one-time instruction set"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-k3s:v1.24.8-k3s1 to directory /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a86>

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"

Dec 30 02:47:30 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:30Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-k3s:v1.24.8-k3s1"

Dec 30 02:47:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:33Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/installer.sh"

Dec 30 02:47:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:33Z" level=info msg="Extracting file k3s to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/k3s"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0/run.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + cp -f /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d>

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + chmod 755 /usr/local/bin/k3s"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + chown root:root /usr/local/bin/k3s"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + mkdir -p /var/lib/rancher/k3s"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + RESTART_STAMP_FILE=/var/lib/rancher/k3s/restart_stamp"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ -f /var/lib/rancher/k3s/restart_stamp ]"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ -n efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ != efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + FORCE_RESTART=true"

1c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + env INSTALL_K3S_FORCE_RESTART=true INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true installer.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Skipping k3s download and verify"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Skipping installation of SELinux RPM"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Skipping /usr/local/bin/kubectl symlink to k3s, command exists in PATH at /usr/bin/kub>

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating /usr/local/bin/crictl symlink to k3s"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating /usr/local/bin/ctr symlink to k3s"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating killall script /usr/local/bin/k3s-killall.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] systemd: Creating service file /etc/systemd/system/k3s.service"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] systemd: Enabling k3s unit"

Dec 30 02:47:38 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:38Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/>

Dec 30 02:47:39 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:39Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: [INFO] systemd: Starting k3s"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + [ -n efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4 ]"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tderr]: + echo efaa1d165db848318ae2650ab08cb74fa54429e383179fd01926d45b4d1facd4"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221230-024730/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3>

Dec 30 02:47:45 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:45Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"

Dec 30 02:47:46 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:46Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: Name Location Size Created"

Dec 30 02:47:46 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:46Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:47 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:47Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:47:52 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:52Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/k3s/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/k3s/server/tls/kube-scheduler/ku>

Dec 30 02:47:52 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:47:52Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221230-025733/f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3>

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[f29d4dbf33fa7473a1220dbddeab20c7a861c8375d7b023a5fe36a4f21bd3135_0😒tdout]: Name Location Size Created"

Dec 30 02:57:33 chardmaster rancher-system-agent[3478]: time="2022-12-30T02:57:33Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

1. please don’t spam the channel with a giant block of logs. Link to the logs on gist/pastebin/etc 2. The errors here are expected when k3s hasn’t started yet. Check the k3s journald logs to see why it’s not starting.

hey, for people who meet the same error, you have to register the nodes with all roles to activate the cluster. otherwise it blocks at inactive state. all good now. Thx. @creamy-pencil-82913 👍