hi guys i am trying to deploy rancher cluster to a seperate rancher-users #vsphere

Title

steep-furniture-72588

10/24/2022, 10:42 AM

hi guys, i am trying to deploy rancher cluster to a seperate vlan. i had issues with downloading images but I solved that with a private registry mirror. now the deployment does not finish and its just sitting tthere. I am getting this in the rancher-system-agent. The bold part is most interesting, the rest are repeated. Any help to diagnose this issue? TIA i get this error: ct 24 20:07:34 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:34+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 20:07:39 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:39+13:00" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\"" Oct 24 20:07:39 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:39+13:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2> Oct 24 20:07:39 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:39+13:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler" Oct 24 20:07:39 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:39+13:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt:> Oct 24 20:07:39 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:39+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\"" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20221024-200744/be7588c5c8d8b0831b57f1afe08907455a> Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tdout]: Name Location Size Created" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt:> Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2> Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error while appending ca cert to pool for probe kube-scheduler" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt:> Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error msg="error while appending ca cert to pool for probe kube-controller-manager" Oct 24 20:07:44 internal-pool1-96c94324-btpch rancher-system-agent[1872]: time="2022-10-24T20:07:44+13:00" level=error

additionally i get this bunch of error messages in the log:

="2022-10-24T20:07:17+13:00" level=info msg="[Applyinator] Running command: sh [-c run.sh]"

="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + SA_INSTALL_PREFIX=/usr/local" ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + mkdir -p /var/lib/rancher/rke2" ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + SAI_FILE_DIR=/var/lib/rancher/r> ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + RESTART_STAMP_FILE=/var/lib/ran> ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.e> ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' '!' -d /var/lib/rancher/rke> ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + mkdir -p /var/lib/rancher/rke2/> ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + check_target_mountpoint" ="2022-10-24T20:07:17+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + mountpoint -q ''" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + check_target_ro" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + touch /usr/local/.rke2-ro-test" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + rm -rf /usr/local/.rke2-ro-test" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + test 0 -ne 0" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + SYSTEMD_BASE_PATH=/usr/local/li> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + RKE2_SA_ENV_SRV_REF=Environment> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -f /var/lib/rancher/rke2/sy> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -n 1f2f2de03a48dc44853d2d52> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' '' '!=' 1f2f2de03a48dc44853> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + RESTART=true" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + env INSTALL_RKE2_ARTIFACT_PATH=> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tdout]: [INFO] staging local checksums f> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tdout]: [INFO] staging tarball from /var> ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tdout]: [INFO] verifying tarball" ="2022-10-24T20:07:18+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tdout]: [INFO] unpacking tarball file to> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -f /var/lib/rancher/rke2/sy> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c442> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + install -m 600 /dev/null /var/l> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ env" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ grep '^RKE2_'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ true" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + RKE2_ENV=" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -n '' ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ env" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PRO> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ true" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + PROXY_ENV_INFO=" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -n '' ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ sha256sum /var/lib/rancher/rke> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: ++ awk '{print $1}'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c442> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' e3b0c44298fc1c149afbf4c8996> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -z '' ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + INSTALL_RKE2_TYPE=server" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + grep -q EnvironmentFile=-/var/l> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + echo EnvironmentFile=-/var/lib/> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' -n 1f2f2de03a48dc44853d2d52> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + echo 1f2f2de03a48dc44853d2d522c> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + systemctl daemon-reload" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' '' = true ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' server = server ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + systemctl is-active --quiet rke> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + systemctl enable rke2-server" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: Created symlink /etc/systemd/syst> ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' '' = true ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + '[' true = true ']'" ="2022-10-24T20:07:23+13:00" level=info msg="[be7588c5c8d8b0831b57f1afe08907455a5176798b2508d726b4bcc0f3a85164_0😒tderr]: + systemctl --no-block restart rk> ="2022-10-24T20:07:23+13:00" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"

modern-television-79263

10/24/2022, 3:48 PM

Can the network your Rancher server is on communicate to the network your cluster deployment is on? And vice versa? Can both networks connect to the internet for their images? Ir connect thru a proxy or private/air-gapped registry for their system images?

steep-furniture-72588

10/24/2022, 5:12 PM

Thank you @modern-television-79263 for replying. I did add sonatype nexus and that fixed the initial problem of downloading initial containers . In regards to the first question of network connectivity I would presume so as I can ssh into it. I also did a test and that is I stopped the rancher system-agent on that node and just installed rke2 using the normal curl command...

Installing with the initial curl command worked for straight generic rke2 although it had a bit of a delay

👍 1

I am still having issues installing from vsphere provisioning which is what i am trying to achieve on the secondary network

modern-television-79263

10/25/2022, 3:04 PM

Gotcha. The next bit I would check are your node templates to be sure the drops downs for the networking, datastore and what not are correct.

steep-furniture-72588

10/26/2022, 1:03 PM

Thank you @modern-television-79263. I got the correct values your cloud-init and set up a container registry mirror and it is now working using rancher

🎉 1

3 Views

#vsphere Join Slack