https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
s
https://docs.k3s.io/reference/binary-tools Check that the crictl.yaml file actually exists crictl has to be run as root, afaik.
l
running crictl with root/sudo yields no different result and file 
/var/lib/rancher/k3s/data/7c994f47fd344e1637da337b92c51433c255b387d207b30b3e0262779457afe4/bin/crictl.yaml
does not exist
s
And verify that the user can actually connect to mysql and have the correct privileges.
l
in the logs it shows it connect to mysql and checks its schema correctly
s
Copy code
Process: 4373 ExecStart=/usr/local/bin/k3s server --datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test (code=exited, status=1/FAILURE)
l
Copy code
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 07:24:54 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 07:24:55 myreviews-k3s-server k3s[4555]: time="2022-12-08T07:24:55Z" level=info msg="Database tables and indexes are up to date"
i saw that msg but i dont get why it is failing if it is actually able to check the db
the user has all grants
i can login just fine on mysql if i try manually from the server
c
--datastore-endpoint=<mysql://OMISSIS:3306>)/my_k3s_test
does not look like a properly formatted endpoint. It should be something like 
user:pass@tcp:(host:3306)/database
as shown in the example. Can you edit the systemd unit and confirm that the value is formatted correctly?
l
i just removed the details
c
If you have weird non-alphanumeric characters in the password, sometimes those get mangled by the install script.
l
but they are correct
i know but indeed they are just letters and numbers
underscores at most
c
try doing 
rm -rf /var/lib/rancher/k3s/data/
and then restarting it
l
by restarting do you mean try the installation again?
c
no, just restarting the service
l
ubuntu@myreviews-k3s-server:~$ sudo systemctl restart k3s.service Job for k3s.service failed because the control process exited with error code.
Copy code
Dec 08 08:09:47 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:47Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 08:09:49 myreviews-k3s-server k3s[17516]: time="2022-12-08T08:09:49Z" level=info msg="Database tables and indexes are up to date"
Dec 08 08:09:50 myreviews-k3s-server sudo[17540]:   ubuntu : TTY=pts/0 ; PWD=/home/ubuntu ; USER=root ; COMMAND=/bin/journalctl -xe
Dec 08 08:09:50 myreviews-k3s-server sudo[17540]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
c
what is the output of 
journalctl -n 50 -l -u k3s
l
Copy code
-- Logs begin at Thu 2022-12-08 07:10:28 UTC, end at Thu 2022-12-08 08:11:25 UTC. --
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: k3s.service: Scheduled restart job, restart counter is at 12.
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
Dec 08 08:10:57 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
Dec 08 08:10:57 myreviews-k3s-server sh[17804]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Dec 08 08:10:57 myreviews-k3s-server sh[17804]: /bin/sh: 1: /usr/bin/systemctl: not found
Dec 08 08:10:57 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:57Z" level=info msg="Starting k3s v1.25.4+k3s1 (0dc63334)"
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Configuring mysql database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
Dec 08 08:10:58 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:10:58Z" level=info msg="Database tables and indexes are up to date"
Dec 08 08:11:00 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:00Z" level=info msg="Slow SQL (started: 2022-12-08 08:10:59.034109225 +0000 UTC m=+1.763292914) (total time: 1.464733848s):  SELECT ( SELECT MAX(rkv.id) AS id F
Dec 08 08:11:00 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:00Z" level=info msg="Kine available at <unix://kine.sock>"
Dec 08 08:11:01 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:01Z" level=info msg="Slow SQL (started: 2022-12-08 08:11:00.771949377 +0000 UTC m=+3.501133034) (total time: 1.093364032s):  SELECT * FROM ( SELECT ( SELECT MAX
Dec 08 08:11:01 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:01Z" level=info msg="Slow SQL (started: 2022-12-08 08:11:00.792156733 +0000 UTC m=+3.521340379) (total time: 1.146873127s):  SELECT * FROM ( SELECT ( SELECT MAX
Dec 08 08:11:02 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:02Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: k3s.service: Failed with result 'exit-code'.
Dec 08 08:11:02 myreviews-k3s-server systemd[1]: Failed to start Lightweight Kubernetes.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: k3s.service: Service hold-off time over, scheduling restart.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: k3s.service: Scheduled restart job, restart counter is at 13.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: Stopped Lightweight Kubernetes.
Dec 08 08:11:07 myreviews-k3s-server systemd[1]: Starting Lightweight Kubernetes...
is it possible that the connection with mysql is too slow and k3s has a time out and dies ?
i can connect instantly but just wondering
c
no
Copy code
Dec 08 08:11:02 myreviews-k3s-server k3s[17815]: time="2022-12-08T08:11:02Z" level=fatal msg="starting kubernetes: preparing server: bootstrap data already found and encrypted with different token"
You are reusing a database from a previous install and have not supplied the same --token value
l
ohhh interesting so i cant do that
didnt know
c
wipe the database, or supply the same --token
l
where do i get the token if i wanted to reuse it?
c
from the other server that you connected to that database
l
destroyed
c
if you didn’t specify it, it was randomly generated
l
i mean is it somewhere on the db?
i guess it will be encrypted on db so no use to look it up
c
no. the token is used as an encryption key to protect the RSA keys for the cluster CAs. You cannot access them without the token, and the token cannot be derived from the encrypted data
The importance of the token is called out in the docs at https://docs.k3s.io/installation/ha - we could probably put the warning a little higher up I guess?
Ensure that you retain a copy of this token as it is required when restoring from backup and adding nodes. Previously, K3s did not enforce the use of a token when using external SQL datastores.
If you reuse the same datastore, you are technically adding nodes to an existing cluster. It’s just that there aren’t currently any nodes in it.
l
maybe moving it in the docs might help but I am just looking at it from a disaster recovery point of view
in my case it probably makes more sense to start clean all the times
c
yeah for DR purposes, you need to either specify your own token in the config on the servers, or copy the token off the first node if you let it create a random one for you, and then reuse that token on all other servers.
l
i just checked the script and i can see K3S_TOKEN . Is this token the same one and is it also used by nodes to join servers?
c
that’s the token, yes
👍 1
l
i will add that to my secrets to ensure i can recover the db too
517 Views
Open in Slack
PreviousNext
Powered by