https://rancher.com/ logo
Powered by
Title
l

little-ram-17683

01/18/2023, 1:12 PM
Hi, I try to create RKE2 downstream cluster, but I can't provision it. RKE1 works fine. Any ideas?
a

agreeable-oil-87482

01/18/2023, 2:19 PM
Log onto the node and run: 
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
/var/lib/rancher/rke2/bin/kubectl get po -A
l

little-ram-17683

01/18/2023, 3:16 PM
in 
/var/lib/rancher/rke
there are only: agent, server, system-agent-installer
no bin folder
[INFO ] configuring bootstrap node(s) custom-2158b6d6d032: waiting for agent to check in and apply initial plan
[INFO ] configuring bootstrap node(s) custom-2158b6d6d032: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet
from rancher gui
node was clear
a

agreeable-oil-87482

01/18/2023, 3:18 PM
Ignore that
l

little-ram-17683

01/18/2023, 3:18 PM
image.png
looks like there is a problem with CAPI CLuster
a

agreeable-oil-87482

01/18/2023, 3:19 PM
does 
journalctl -u rancher-system-agent
return anything?
l

little-ram-17683

01/18/2023, 3:21 PM
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Rancher System Agent version v0.2.10 (6a89e30) is starting"
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Starting remote watch of plans"
Jan 18 12:24:34 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:34Z" level=info msg="Initial connection to Kubernetes cluster failed with error Get \"https://<my_rancher>/version\": x509: certificate signed by unknown authority, removing CA data and trying again"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Starting /v1, Kind=Secret controller"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Detected first start, force-applying one-time instruction set"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.23.14-rke2r1 to directory /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jan 18 12:24:35 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:35Z" level=info msg="Pulling image <http://index.docker.io/rancher/system-agent-installer-rke2:v1.23.14-rke2r1|index.docker.io/rancher/system-agent-installer-rke2:v1.23.14-rke2r1>"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/installer.sh"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/rke2.linux-amd64.tar.gz"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/sha256sum-amd64.txt"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/run.sh"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ ! -d /var/lib/rancher/rke2/system-agent-installer ]"
Jan 18 12:24:36 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:36Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + mkdir -p /var/lib/rancher/rke2/system-agent-installer"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + check_target_mountpoint"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + mountpoint -q "
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + check_target_ro"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + touch /usr/local/.rke2-ro-test"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + test 0 -ne 0"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_stamp ]"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa ]"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [  != 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa ]"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RESTART=true"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0 
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf64
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20230118-122435/2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0/rke2
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  verifying tarball"
Jan 18 12:24:37 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:37Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + grep ^RKE2_"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + true"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + RKE2_ENV="
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n  ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + grep -Ei ^(NO|HTTP|HTTPS)_PROXY"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + true"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + PROXY_ENV_INFO="
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n  ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + awk {print $1}"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 != e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -z  ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + INSTALL_RKE2_TYPE=server"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env /usr/local/lib/systemd/system/rke2-server.service"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ -n 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + echo 96ddcbd8d9b0b4bbb030bee8062ddf869f582669e7d3f6390857a3231cf4e3fa"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + systemctl daemon-reload"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [  = true ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + [ server = server ]"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + systemctl is-active --quiet rke2-agent"
Jan 18 12:24:38 sirius rancher-system-agent[5229]: time="2023-01-18T12:24:38Z" level=info msg="[2c4a00f4d0cc94e6ea7102e0cb90df9786edc759b308af1823f6ca1840bbf644_0:stderr]: + systemctl enable rke2-server"
rke2-server.service - Rancher Kubernetes Engine v2 (server)
   Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Wed 2023-01-18 15:23:00 UTC; 503ms ago
     Docs: <https://github.com/rancher/rke2#readme>
  Process: 30970 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
  Process: 30973 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 30974 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 30975 ExecStart=/usr/local/bin/rke2 server (code=exited, status=1/FAILURE)
  Process: 30997 ExecStopPost=/bin/sh -c systemd-cgls /system.slice/rke2-server.service | grep -Eo '[0-9]+ (containerd|kubelet)' | awk '{print $1}' | xargs -r kill (code=exited, status=0/SUCCESS)
 Main PID: 30975 (code=exited, status=1/FAILURE)
a

agreeable-oil-87482

01/18/2023, 3:24 PM
journalctl -u rke2-server
- run that please
l

little-ram-17683

01/18/2023, 3:25 PM
Jan 18 12:24:59 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 5.
Jan 18 12:25:04 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:25:04 sirius sh[5603]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:25:04 sirius sh[5603]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:25:04 sirius rke2[5607]: time="2023-01-18T12:25:04Z" level=warning msg="not running in CIS mode"
a

agreeable-oil-87482

01/18/2023, 3:29 PM
Is that the entirety of the log?
l

little-ram-17683

01/18/2023, 3:30 PM
Jan 18 12:24:38 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:38 sirius sh[5367]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:38 sirius sh[5367]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:38 sirius rke2[5378]: time="2023-01-18T12:24:38Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:38 sirius rke2[5378]: time="2023-01-18T12:24:38Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:38 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:38 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:38 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:38 sirius systemd[1]: rke2-server.service: Consumed 182ms CPU time.
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 1.
Jan 18 12:24:43 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Consumed 182ms CPU time.
Jan 18 12:24:43 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:43 sirius sh[5430]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:43 sirius sh[5430]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:43 sirius rke2[5434]: time="2023-01-18T12:24:43Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:43 sirius rke2[5434]: time="2023-01-18T12:24:43Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:43 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:43 sirius systemd[1]: rke2-server.service: Consumed 192ms CPU time.
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 2.
Jan 18 12:24:48 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Consumed 192ms CPU time.
Jan 18 12:24:48 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:48 sirius sh[5474]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:48 sirius sh[5474]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:48 sirius rke2[5478]: time="2023-01-18T12:24:48Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:48 sirius rke2[5478]: time="2023-01-18T12:24:48Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:48 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:48 sirius systemd[1]: rke2-server.service: Consumed 189ms CPU time.
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 3.
Jan 18 12:24:54 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Consumed 189ms CPU time.
Jan 18 12:24:54 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:54 sirius sh[5519]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:54 sirius sh[5519]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:54 sirius rke2[5523]: time="2023-01-18T12:24:54Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:54 sirius rke2[5523]: time="2023-01-18T12:24:54Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:54 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:54 sirius systemd[1]: rke2-server.service: Consumed 197ms CPU time.
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 4.
Jan 18 12:24:59 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Consumed 197ms CPU time.
Jan 18 12:24:59 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:24:59 sirius sh[5560]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:24:59 sirius sh[5560]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:24:59 sirius rke2[5564]: time="2023-01-18T12:24:59Z" level=warning msg="not running in CIS mode"
Jan 18 12:24:59 sirius rke2[5564]: time="2023-01-18T12:24:59Z" level=fatal msg="invalid node-ip: invalid ip format 'fdf3e1c5257259'"
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Main process exited, code=exited, status=1/FAILURE
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Failed with result 'exit-code'.
Jan 18 12:24:59 sirius systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).
Jan 18 12:24:59 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Service RestartSec=5s expired, scheduling restart.
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Scheduled restart job, restart counter is at 5.
Jan 18 12:25:04 sirius systemd[1]: Stopped Rancher Kubernetes Engine v2 (server).
Jan 18 12:25:04 sirius systemd[1]: rke2-server.service: Consumed 186ms CPU time.
Jan 18 12:25:04 sirius systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jan 18 12:25:04 sirius sh[5603]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jan 18 12:25:04 sirius sh[5603]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jan 18 12:25:04 sirius rke2[5607]: time="2023-01-18T12:25:04Z" level=warning msg="not running in CIS mode"
I guess 
fdf3e1c5257259
is fdf3:e1c5:2572::59
ipv6
a

agreeable-oil-87482

01/18/2023, 3:32 PM
IPv6 only node?
l

little-ram-17683

01/18/2023, 3:32 PM
yes
a

agreeable-oil-87482

01/18/2023, 3:38 PM
I'm not 100% certain if we support deploying RKE2 clusters from Rancher to IPV6 only nodes - @creamy-pencil-82913 do you happen to know?
l

little-ram-17683

01/18/2023, 3:39 PM
https://docs.rke2.io/reference/linux_agent_config
--node-ip value, -i value                     (agent/networking) IPv4/IPv6 addresses to advertise for node
a

agreeable-oil-87482

01/18/2023, 3:40 PM
Yeah if you run the binary directly you can add those flags, but through Rancher is different
l

little-ram-17683

01/18/2023, 3:40 PM
ok
So would be great if you could confirm or not
a

agreeable-oil-87482

01/18/2023, 3:41 PM
I'll follow this up internally
❤️ 1
c

creamy-pencil-82913

01/18/2023, 4:38 PM
IPv6 only should work I think, but you would need to format the node-ip value properly. How are you ending up with 
fdf3e1c5257259
instead of 
fdf3:e1c5:2572::59
- the former is NOT a valid ipv6 address.
I will also note that there is a LOT of stuff out there that isn’t ipv6 native yet, if your pods need to talk to the internet or anything, not having an ipv4 address is going to cause problems.
I am also not sure if IPv6 only is supported by Rancher provisioning. I know dual-stack works but I am not sure about ipv6-only. I don’t know that we have any ipv6-only environments.
But, one thing at a time, I would start by fixing that address.
l

little-ram-17683

01/18/2023, 4:54 PM
@creamy-pencil-82913 when I register node I put proper ipv6 in "Node Private IP" field
So you basically mean, it's not good to have LAN address with ipv6 address only on server
and how can I fix this address? I suppose rancher try to use 
fdf3e1c5257259
instead of 
fdf3:e1c5:2572::59
, even when I type fdf3:e1c5:2572::59 in "Node Private IP" field. What can i do now? 🙂
also one more thing - provisioning using RKE1 works fine
c

creamy-pencil-82913

01/18/2023, 5:02 PM
If the UI is removing the : from the address then I suspect thats a bug. Would you mind opening an issue at https://github.com/rancher/dashboard/issues
l

little-ram-17683

01/18/2023, 5:36 PM
Sure. Is there any other way to create rke2 on prem downstream cluster? Rancher cli?
Also, can I fix it manually on my node? For example change some config to good ip address?
c

creamy-pencil-82913

01/18/2023, 5:44 PM
no, if that parameter is coming from Rancher, the Rancher system agent is going to put it back even if you try to fix it.
you can manually create your RKE2 cluster and then import it into Rancher. You don’t have to use Rancher to provision it.
l

little-ram-17683

01/18/2023, 5:47 PM
Ok, but then I think I cant upgrade kubernetes version from gui? Can I?
c

creamy-pencil-82913

01/18/2023, 5:56 PM
You can’t manage the full cluster configuration or provision new nodes from the UI but you can still do upgrades via the system-upgrade-controller
l

little-ram-17683

01/18/2023, 5:58 PM
Ok, so last question. What about trying to create news cluster using rancher cli? Could this approach change anything?
c

creamy-pencil-82913

01/18/2023, 6:15 PM
no
l

little-ram-17683

01/18/2023, 8:59 PM
@creamy-pencil-82913 @agreeable-oil-87482 thank you guys 🙂
128 Views
#generalJoin Slack
Powered by