This message was deleted Rancher Users #general

Join Slack

This message was deleted.

# general

adamant-kite-43734

01/18/2023, 1:12 PM

This message was deleted.

agreeable-oil-87482

01/18/2023, 2:19 PM

Log onto the node and run:

Copy code

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl get po -A

little-ram-17683

01/18/2023, 3:16 PM

/var/lib/rancher/rke

there are only: agent, server, system-agent-installer

little-ram-17683

01/18/2023, 3:16 PM

no bin folder

little-ram-17683

01/18/2023, 3:18 PM

Copy code

[INFO ] configuring bootstrap node(s) custom-2158b6d6d032: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) custom-2158b6d6d032: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet

little-ram-17683

01/18/2023, 3:18 PM

from rancher gui

little-ram-17683

01/18/2023, 3:18 PM

node was clear

agreeable-oil-87482

01/18/2023, 3:18 PM

Copy code

Ignore that

little-ram-17683

01/18/2023, 3:18 PM

looks like there is a problem with CAPI CLuster

agreeable-oil-87482

01/18/2023, 3:19 PM

does

journalctl -u rancher-system-agent

return anything?

little-ram-17683

01/18/2023, 3:21 PM

Copy code

Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Rancher System Agent version v0.2.10 (6a89e30) is starting"
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Starting remote watch of plans"
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://<my_rancher>/version\": x509: certificate signed by unknown authority, removing CA data and trying again"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Starting /v1, Kind=Secret controller"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Detected first start, force-applying one-time instruction set"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.23.14-rke2r1 to directory /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Pulling image <http://index.docker.io/rancher/system-agent-installer-rke2:v1.23.14-rke2r1|index.docker.io/rancher/system-agent-installer-rke2:v1.23.14-rke2r1>"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/installer.sh"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/rke2.linux-amd64.tar.gz"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/sha256sum-amd64.txt"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/run.sh"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"

little-ram-17683

01/18/2023, 3:21 PM

Copy code

Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ ! -d /var/lib/rancher/rke2/system-agent-installer ]"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + mkdir -p /var/lib/rancher/rke2/system-agent-installer"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + check_target_mountpoint"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + mountpoint -q "
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + check_target_ro"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + touch /usr/local/.rke2-ro-test"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + test 0 -ne 0"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ]"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa ]"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [  != 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa ]"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RESTART=true"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0 
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf64
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/rke2
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  verifying tarball"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + grep ^RKE2_"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + true"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_ENV="
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n  ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + grep -Ei ^(NO|HTTP|HTTPS)_PROXY"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + true"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + PROXY_ENV_INFO="
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n  ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + awk {print $1}"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -z  ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + INSTALL_RKE2_TYPE=server"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + echo 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + systemctl daemon-reload"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [  = true ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ server = server ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + systemctl is-active --quiet rke2-agent"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + systemctl enable rke2-server"

little-ram-17683

01/18/2023, 3:23 PM

Copy code

rke2-server.service - Rancher Kubernetes Engine v2 (server)
   Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-01-18 15:23:00 UTC; 503ms ago
     Docs: <https://github.com/rancher/rke2#readme>
  Process: 30970 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
  Process: 30973 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 30974 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 30975 ExecStart=/usr/local/bin/rke2 server (code=exited, status=1/FAILURE)
  Process: 30997 ExecStopPost=/bin/sh -c systemd-cgls /system.slice/rke2-server.service | grep -Eo '[0-9]+ (containerd|kubelet)' | awk '{print $1}' | xargs -r kill (code=exited, status=0/SUCCESS)
 Main PID: 30975 (code=exited, status=1/FAILURE)

agreeable-oil-87482

01/18/2023, 3:24 PM

journalctl -u rke2-server

- run that please

little-ram-17683

01/18/2023, 3:25 PM

Copy code

Jan 18 12:24:59 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 5.
Jan 18 12:25:04 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:25:04 sirius sh[5603]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:25:04 sirius sh[5603]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:25:04 sirius rke2[5607]: time="2023-01-18T12:25:04Z" level=warning msg="not running in CIS mode"

agreeable-oil-87482

01/18/2023, 3:29 PM

Is that the entirety of the log?

little-ram-17683

01/18/2023, 3:30 PM

Copy code

Jan 18 12:24:38 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:38 sirius sh[5367]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:38 sirius sh[5367]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:38 sirius rke2[5378]: time="2023-01-18T12:24:38Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:38 sirius rke2[5378]: time="2023-01-18T12:24:38Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:38 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:38 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:38 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:38 sirius systemd[1]: rke2-server.service: Consumed 182ms CPU time.
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 1.
Jan 18 12:24:43 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Consumed 182ms CPU time.
Jan 18 12:24:43 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:43 sirius sh[5430]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:43 sirius sh[5430]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:43 sirius rke2[5434]: time="2023-01-18T12:24:43Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:43 sirius rke2[5434]: time="2023-01-18T12:24:43Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:43 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Consumed 192ms CPU time.
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 2.
Jan 18 12:24:48 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Consumed 192ms CPU time.
Jan 18 12:24:48 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:48 sirius sh[5474]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:48 sirius sh[5474]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:48 sirius rke2[5478]: time="2023-01-18T12:24:48Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:48 sirius rke2[5478]: time="2023-01-18T12:24:48Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:48 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Consumed 189ms CPU time.
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 3.
Jan 18 12:24:54 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Consumed 189ms CPU time.
Jan 18 12:24:54 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:54 sirius sh[5519]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:54 sirius sh[5519]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:54 sirius rke2[5523]: time="2023-01-18T12:24:54Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:54 sirius rke2[5523]: time="2023-01-18T12:24:54Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:54 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Consumed 197ms CPU time.
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 4.
Jan 18 12:24:59 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Consumed 197ms CPU time.
Jan 18 12:24:59 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:59 sirius sh[5560]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:59 sirius sh[5560]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:59 sirius rke2[5564]: time="2023-01-18T12:24:59Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:59 sirius rke2[5564]: time="2023-01-18T12:24:59Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:59 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 5.
Jan 18 12:25:04 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:25:04 sirius sh[5603]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:25:04 sirius sh[5603]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:25:04 sirius rke2[5607]: time="2023-01-18T12:25:04Z" level=warning msg="not running in CIS mode"

little-ram-17683

01/18/2023, 3:31 PM

I guess

fdf3e1c5257259

is fdf3e1c52572::59

little-ram-17683

01/18/2023, 3:31 PM

ipv6

agreeable-oil-87482

01/18/2023, 3:32 PM

IPv6 only node?

little-ram-17683

01/18/2023, 3:32 PM

yes

agreeable-oil-87482

01/18/2023, 3:38 PM

I'm not 100% certain if we support deploying RKE2 clusters from Rancher to IPV6 only nodes - @creamy-pencil-82913 do you happen to know?

little-ram-17683

01/18/2023, 3:39 PM

https://docs.rke2.io/reference/linux_agent_config

little-ram-17683

01/18/2023, 3:39 PM

Copy code

--node-ip value, -i value                     (agent/networking) IPv4/IPv6 addresses to advertise for node

agreeable-oil-87482

01/18/2023, 3:40 PM

Yeah if you run the binary directly you can add those flags, but through Rancher is different

little-ram-17683

01/18/2023, 3:40 PM

little-ram-17683

01/18/2023, 3:41 PM

So would be great if you could confirm or not

agreeable-oil-87482

01/18/2023, 3:41 PM

I'll follow this up internally

❤️ 1

creamy-pencil-82913

01/18/2023, 4:38 PM

IPv6 only should work I think, but you would need to format the node-ip value properly. How are you ending up with

fdf3e1c5257259

instead of

fdf3:e1c5:2572::59

- the former is NOT a valid ipv6 address.

creamy-pencil-82913

01/18/2023, 4:39 PM

I will also note that there is a LOT of stuff out there that isn’t ipv6 native yet, if your pods need to talk to the internet or anything, not having an ipv4 address is going to cause problems.

creamy-pencil-82913

01/18/2023, 4:41 PM

I am also not sure if IPv6 only is supported by Rancher provisioning. I know dual-stack works but I am not sure about ipv6-only. I don’t know that we have any ipv6-only environments.

creamy-pencil-82913

01/18/2023, 4:41 PM

But, one thing at a time, I would start by fixing that address.

little-ram-17683

01/18/2023, 4:54 PM

@creamy-pencil-82913 when I register node I put proper ipv6 in "Node Private IP" field

little-ram-17683

01/18/2023, 4:56 PM

So you basically mean, it's not good to have LAN address with ipv6 address only on server

little-ram-17683

01/18/2023, 4:58 PM

and how can I fix this address? I suppose rancher try to use

fdf3e1c5257259

instead of

fdf3:e1c5:2572::59

, even when I type fdf3e1c52572::59 in "Node Private IP" field. What can i do now? 🙂

little-ram-17683

01/18/2023, 5:01 PM

also one more thing - provisioning using RKE1 works fine

creamy-pencil-82913

01/18/2023, 5:02 PM

If the UI is removing the : from the address then I suspect thats a bug. Would you mind opening an issue at https://github.com/rancher/dashboard/issues

little-ram-17683

01/18/2023, 5:36 PM

Sure. Is there any other way to create rke2 on prem downstream cluster? Rancher cli?

little-ram-17683

01/18/2023, 5:42 PM

Also, can I fix it manually on my node? For example change some config to good ip address?

creamy-pencil-82913

01/18/2023, 5:44 PM

no, if that parameter is coming from Rancher, the Rancher system agent is going to put it back even if you try to fix it.

creamy-pencil-82913

01/18/2023, 5:44 PM

you can manually create your RKE2 cluster and then import it into Rancher. You don’t have to use Rancher to provision it.

little-ram-17683

01/18/2023, 5:47 PM

Ok, but then I think I cant upgrade kubernetes version from gui? Can I?

creamy-pencil-82913

01/18/2023, 5:56 PM

You can’t manage the full cluster configuration or provision new nodes from the UI but you can still do upgrades via the system-upgrade-controller

little-ram-17683

01/18/2023, 5:58 PM

Ok, so last question. What about trying to create news cluster using rancher cli? Could this approach change anything?

creamy-pencil-82913

01/18/2023, 6:15 PM

little-ram-17683

01/18/2023, 8:59 PM

@creamy-pencil-82913 @agreeable-oil-87482 thank you guys 🙂

603 Views

Open in Slack

Previous Next