https://rancher.com/ logo
Powered by
#k3s
Title
# k3s
l

lively-night-78214

12/08/2022, 7:37 AM
I run this installation
Copy code
curl -sfL <https://get.k3s.io> |  INSTALL_K3S_VERSION=v1.25.4+k3s1 sh -s - server --datastore-endpoint=\"<mysql://user:pass@tcp>(host:3306)/my_k3s_test
STD out shows
Copy code
"[INFO]  Using v1.25.4+k3s1 as release",
        "[INFO]  Downloading hash <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/sha256sum-amd64.txt>",
        "[INFO]  Downloading binary <https://github.com/k3s-io/k3s/releases/download/v1.25.4+k3s1/k3s>",
        "[INFO]  Verifying binary download",
        "[INFO]  Installing k3s to /usr/local/bin/k3s",
        "[INFO]  Skipping installation of SELinux RPM",
        "[INFO]  Creating /usr/local/bin/kubectl symlink to k3s",
        "[INFO]  Creating /usr/local/bin/crictl symlink to k3s",
        "[INFO]  Creating /usr/local/bin/ctr symlink to k3s",
        "[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh",
        "[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh",
        "[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env",
        "[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service",
        "[INFO]  systemd: Enabling k3s unit",
        "[INFO]  systemd: Starting k3s"
However in std_err
Copy code
"Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.",
        "Job for k3s.service failed because the control process exited with error code.",
        "See \"systemctl status k3s.service\" and \"journalctl -xe\" for details."
If i go to the server and run
Copy code
ubuntu@my-k3s-server:~$ crictl version
FATA[0000] load config file: stat /var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml: no such file or directory
From the journal
Copy code
ubuntu@my-k3s-server:~$ journalctl -xe
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
No journal files were opened due to insufficient permissions.
ubuntu@myreviews-k3s-server:~$ sudo journalctl -xe
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
-- 
-- Automatic restarting of the unit k3s.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
-- Subject: Unit k3s.service has finished shutting down
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
-- 
-- Unit k3s.service has finished shutting down.
Dec 08 07:24:52 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
-- Subject: Unit k3s.service has begun start-up
-- Defined-By: systemd
-- Support: <http://www.ubuntu.com/support>
-- 
-- Unit k3s.service has begun starting up.
Dec 08 07:24:52 myreviews-k3s-server sh[4543]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 08 07:24:52 myreviews-k3s-server sh[4543]: /bin/sh: 1: /usr/bin/systemctl: not found
Dec 08 07:24:52 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:52Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
Dec 08 07:24:56 myreviews-k3s-server sudo[4573]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 08 07:24:56 myreviews-k3s-server sudo[4573]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
From system CTL
Copy code
systemctl status k3s.service
● k3s.service - Lightweight Kubernetes
   Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Thu 2022-12-08 07:24:18 UTC; 3s ago
     Docs: <https://k3s.io>
  Process: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
  Process: 4372 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Process: 4370 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 4363 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
 Main PID: 4373 (code=exited, status=1/FAILURE)
What am I doing wrong?
s

sparse-fireman-14239

12/08/2022, 7:49 AM
https://docs.k3s.io/reference/binary-tools Check that the crictl.yaml file actually exists crictl has to be run as root, afaik.
l

lively-night-78214

12/08/2022, 7:52 AM
running crictl with root/sudo yields no different result and file 
/var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml
does not exist
s

sparse-fireman-14239

12/08/2022, 7:52 AM
And verify that the user can actually connect to mysql and have the correct privileges.
l

lively-night-78214

12/08/2022, 7:53 AM
in the logs it shows it connect to mysql and checks its schema correctly
s

sparse-fireman-14239

12/08/2022, 7:53 AM
Copy code
Process: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
l

lively-night-78214

12/08/2022, 7:53 AM
Copy code
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
i saw that msg but i dont get why it is failing if it is actually able to check the db
the user has all grants
i can login just fine on mysql if i try manually from the server
c

creamy-pencil-82913

12/08/2022, 8:06 AM
--datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test
does not look like a properly formatted endpoint. It should be something like 
user:pass@tcp:(host:3306)/database
as shown in the example. Can you edit the systemd unit and confirm that the value is formatted correctly?
l

lively-night-78214

12/08/2022, 8:06 AM
i just removed the details
c

creamy-pencil-82913

12/08/2022, 8:06 AM
If you have weird non-alphanumeric characters in the password, sometimes those get mangled by the install script.
l

lively-night-78214

12/08/2022, 8:06 AM
but they are correct
i know but indeed they are just letters and numbers
underscores at most
c

creamy-pencil-82913

12/08/2022, 8:07 AM
try doing 
rm -rf /var/lib/rancher/k3s/data/
and then restarting it
l

lively-night-78214

12/08/2022, 8:07 AM
by restarting do you mean try the installation again?
c

creamy-pencil-82913

12/08/2022, 8:07 AM
no, just restarting the service
l

lively-night-78214

12/08/2022, 8:09 AM
ubuntu@myreviews-k3s-server:~$ sudo systemctl restart k3s.service Job for k3s.service failed because the control process exited with error code.
Copy code
Dec 08 08:09:47 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:47Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Database tables and indexes are up to date"
Dec 08 08:09:50 myreviews-k3s-server sudo[17540]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 08 08:09:50 myreviews-k3s-server sudo[17540]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
c

creamy-pencil-82913

12/08/2022, 8:10 AM
what is the output of 
journalctl -n 50 -l -u k3s
l

lively-night-78214

12/08/2022, 8:11 AM
Copy code
-- Logs begin at Thu 2022-12-08 07:10:28 UTC, end at Thu 2022-12-08 08:11:25 UTC. --
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: k3s.service: Scheduled restart job, restart counter is at 12.
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
Dec 08 08:10:57 myreviews-k3s-server sh[17804]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 08 08:10:57 myreviews-k3s-server sh[17804]: /bin/sh: 1: /usr/bin/systemctl: not found
Dec 08 08:10:57 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:57Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Database tables and indexes are up to date"
Dec 08 08:11:00 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:00Z" level=info msg="Slow SQL (started: 2022-12-08 08:10:59.034109225 +0000 UTC m=+1.763292914) (total time: 1.464733848s):  SELECT ( SELECT MAX(rkv.id) AS id F
Dec 08 08:11:00 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:00Z" level=info msg="Kine available at <unix://kine.sock>"
Dec 08 08:11:01 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:01Z" level=info msg="Slow SQL (started: 2022-12-08 08:11:00.771949377 +0000 UTC m=+3.501133034) (total time: 1.093364032s):  SELECT * FROM ( SELECT ( SELECT MAX
Dec 08 08:11:01 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:01Z" level=info msg="Slow SQL (started: 2022-12-08 08:11:00.792156733 +0000 UTC m=+3.521340379) (total time: 1.146873127s):  SELECT * FROM ( SELECT ( SELECT MAX
Dec 08 08:11:02 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:02Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: k3s.service: Failed with result 'exit-code'.
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: Failed to start Lightweight Kubernetes.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: k3s.service: Service hold-off time over, scheduling restart.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: k3s.service: Scheduled restart job, restart counter is at 13.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
is it possible that the connection with mysql is too slow and k3s has a time out and dies ?
i can connect instantly but just wondering
c

creamy-pencil-82913

12/08/2022, 8:12 AM
no
Copy code
Dec 08 08:11:02 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:02Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
You are reusing a database from a previous install and have not supplied the same --token value
l

lively-night-78214

12/08/2022, 8:13 AM
ohhh interesting so i cant do that
didnt know
c

creamy-pencil-82913

12/08/2022, 8:13 AM
wipe the database, or supply the same --token
l

lively-night-78214

12/08/2022, 8:13 AM
where do i get the token if i wanted to reuse it?
c

creamy-pencil-82913

12/08/2022, 8:13 AM
from the other server that you connected to that database
l

lively-night-78214

12/08/2022, 8:13 AM
destroyed
c

creamy-pencil-82913

12/08/2022, 8:14 AM
if you didn’t specify it, it was randomly generated
l

lively-night-78214

12/08/2022, 8:14 AM
i mean is it somewhere on the db?
i guess it will be encrypted on db so no use to look it up
c

creamy-pencil-82913

12/08/2022, 8:14 AM
no. the token is used as an encryption key to protect the RSA keys for the cluster CAs. You cannot access them without the token, and the token cannot be derived from the encrypted data
The importance of the token is called out in the docs at https://docs.k3s.io/installation/ha - we could probably put the warning a little higher up I guess?
Ensure that you retain a copy of this token as it is required when restoring from backup and adding nodes. Previously, K3s did not enforce the use of a token when using external SQL datastores.
If you reuse the same datastore, you are technically adding nodes to an existing cluster. It’s just that there aren’t currently any nodes in it.
l

lively-night-78214

12/08/2022, 8:18 AM
maybe moving it in the docs might help but I am just looking at it from a disaster recovery point of view
in my case it probably makes more sense to start clean all the times
c

creamy-pencil-82913

12/08/2022, 8:20 AM
yeah for DR purposes, you need to either specify your own token in the config on the servers, or copy the token off the first node if you let it create a random one for you, and then reuse that token on all other servers.
l

lively-night-78214

12/08/2022, 8:29 AM
i just checked the script and i can see K3S_TOKEN . Is this token the same one and is it also used by nodes to join servers?
c

creamy-pencil-82913

12/08/2022, 8:35 AM
that’s the token, yes
👍 1
l

lively-night-78214

12/08/2022, 8:36 AM
i will add that to my secrets to ensure i can recover the db too
376 Views
Powered by