lively-night-78214
12/08/2022, 7:37 AMcurl -sfL <https://get.k3s.io> | INSTALL_K3S_VERSION=v1.25.4+k3s1 sh -s - server --datastore-endpoint=\"<mysql://user:pass@tcp>(host:3306)/my_k3s_test
STD out shows
"[INFO] Using v1.25.4+k3s1 as release",
"[INFO] Downloading hash <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt>",
"[INFO] Downloading binary <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s>",
"[INFO] Verifying binary download",
"[INFO] Installing k3s to /usr/local/bin/k3s",
"[INFO] Skipping installation of SELinux RPM",
"[INFO] Creating /usr/local/bin/kubectl symlink to k3s",
"[INFO] Creating /usr/local/bin/crictl symlink to k3s",
"[INFO] Creating /usr/local/bin/ctr symlink to k3s",
"[INFO] Creating killall script /usr/local/bin/k3s-killall.sh",
"[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh",
"[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env",
"[INFO] systemd: Creating service file /etc/systemd/system/k3s.service",
"[INFO] systemd: Enabling k3s unit",
"[INFO] systemd: Starting k3s"
However in std_err
"Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.",
"Job for k3s.service failed because the control process exited with error code.",
"See \"systemctl status k3s.service\" and \"journalctl -xe\" for details."
If i go to the server and run
ubuntu@my-k3s-server:~$ crictl version
FATA[0000] load config file: stat /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml: no such file or directory
From the journal
ubuntu@my-k3s-server:~$ journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
Users in groups 'adm', 'systemd-journal' can see all messages.
Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.
ubuntu@myreviews-k3s-server:~$ sudo journalctl -xe
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
--
-- Automatic restarting of the unit k3s.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
-- Subject: Unit k3s.service has finished shutting down
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
--
-- Unit k3s.service has finished shutting down.
Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
-- Subject: Unit k3s.service has begun start-up
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
--
-- Unit k3s.service has begun starting up.
Dec 08 07:24:52 myreviews-k3s-server sh[4543]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 08 07:24:52 myreviews-k3s-server sh[4543]: /bin/sh: 1: /usr/bin/systemctl: not found
Dec 08 07:24:52 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:52Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
Dec 08 07:24:56 myreviews-k3s-server sudo[4573]: ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 08 07:24:56 myreviews-k3s-server sudo[4573]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
From system CTL
systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2022-12-08 07:24:18 UTC; 3s ago
Docs: <https://k3s.io>
Process: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
Process: 4372 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 4370 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 4363 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Main PID: 4373 (code=exited, status=1/FAILURE)
What am I doing wrong?sparse-fireman-14239
12/08/2022, 7:49 AMlively-night-78214
12/08/2022, 7:52 AM/var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml
does not existsparse-fireman-14239
12/08/2022, 7:52 AMlively-night-78214
12/08/2022, 7:53 AMsparse-fireman-14239
12/08/2022, 7:53 AMProcess: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
lively-night-78214
12/08/2022, 7:53 AMDec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
creamy-pencil-82913
12/08/2022, 8:06 AM--datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test
does not look like a properly formatted endpoint. It should be something like user:pass@tcp:(host:3306)/database
as shown in the example. Can you edit the systemd unit and confirm that the value is formatted correctly?lively-night-78214
12/08/2022, 8:06 AMcreamy-pencil-82913
12/08/2022, 8:06 AMlively-night-78214
12/08/2022, 8:06 AMcreamy-pencil-82913
12/08/2022, 8:07 AMrm -rf /var/lib/rancher/k3s/data/
and then restarting itlively-night-78214
12/08/2022, 8:07 AMcreamy-pencil-82913
12/08/2022, 8:07 AMlively-night-78214
12/08/2022, 8:09 AMDec 08 08:09:47 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:47Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Database tables and indexes are up to date"
Dec 08 08:09:50 myreviews-k3s-server sudo[17540]: ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 08 08:09:50 myreviews-k3s-server sudo[17540]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
creamy-pencil-82913
12/08/2022, 8:10 AMjournalctl -n 50 -l -u k3s
lively-night-78214
12/08/2022, 8:11 AM-- Logs begin at Thu 2022-12-08 07:10:28 UTC, end at Thu 2022-12-08 08:11:25 UTC. --
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: k3s.service: Scheduled restart job, restart counter is at 12.
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
Dec 08 08:10:57 myreviews-k3s-server sh[17804]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 08 08:10:57 myreviews-k3s-server sh[17804]: /bin/sh: 1: /usr/bin/systemctl: not found
Dec 08 08:10:57 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:57Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Database tables and indexes are up to date"
Dec 08 08:11:00 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:00Z" level=info msg="Slow SQL (started: 2022-12-08 08:10:59.034109225 +0000 UTC m=+1.763292914) (total time: 1.464733848s): SELECT ( SELECT MAX(rkv.id) AS id F
Dec 08 08:11:00 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:00Z" level=info msg="Kine available at <unix://kine.sock>"
Dec 08 08:11:01 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:01Z" level=info msg="Slow SQL (started: 2022-12-08 08:11:00.771949377 +0000 UTC m=+3.501133034) (total time: 1.093364032s): SELECT * FROM ( SELECT ( SELECT MAX
Dec 08 08:11:01 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:01Z" level=info msg="Slow SQL (started: 2022-12-08 08:11:00.792156733 +0000 UTC m=+3.521340379) (total time: 1.146873127s): SELECT * FROM ( SELECT ( SELECT MAX
Dec 08 08:11:02 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:02Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: k3s.service: Failed with result 'exit-code'.
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: Failed to start Lightweight Kubernetes.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: k3s.service: Service hold-off time over, scheduling restart.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: k3s.service: Scheduled restart job, restart counter is at 13.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
creamy-pencil-82913
12/08/2022, 8:12 AMDec 08 08:11:02 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:02Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
lively-night-78214
12/08/2022, 8:13 AMcreamy-pencil-82913
12/08/2022, 8:13 AMlively-night-78214
12/08/2022, 8:13 AMcreamy-pencil-82913
12/08/2022, 8:13 AMlively-night-78214
12/08/2022, 8:13 AMcreamy-pencil-82913
12/08/2022, 8:14 AMlively-night-78214
12/08/2022, 8:14 AMcreamy-pencil-82913
12/08/2022, 8:14 AMEnsure that you retain a copy of this token as it is required when restoring from backup and adding nodes. Previously, K3s did not enforce the use of a token when using external SQL datastores.If you reuse the same datastore, you are technically adding nodes to an existing cluster. It’s just that there aren’t currently any nodes in it.
lively-night-78214
12/08/2022, 8:18 AMcreamy-pencil-82913
12/08/2022, 8:20 AMlively-night-78214
12/08/2022, 8:29 AMcreamy-pencil-82913
12/08/2022, 8:35 AMlively-night-78214
12/08/2022, 8:36 AM