https://rancher.com/ logo
Join SlackCommunities
Powered by
This message was deleted.
# rke2
a
This message was deleted.
j
Hello, I'm trying to start a Downstream rke2 cluster, each node with their specific roles. The etcd nodes and Control Plane nodes stated successfuly, but the worker nodes does not start. After executing the install command provided by Rancher, the system agent starts, but the proccess os rke2-server setup does not start. This is the rancher-system-agent.service journalctl logs after executing the install command as follow: Install command: 
curl --insecure -fL <https://rancher.mycompany.com.br/system-agent-install.sh> | sudo  sh -s - --server <https://rancher.mycompany.com.br> --label '<http://cattle.io/os=linux|cattle.io/os=linux>' --token hvwlv52tkjz7f76s9kxsq5747gv9rrvth8lsmz5jq5ccqsqgb..... --worker
rancher-system-agent.service logs
Copy code
Jun 05 11:15:49 FNCWBSLX577 systemd[1]: Started Rancher System Agent.
Jun 05 11:15:49 FNCWBSLX577 rancher-system-agent[2085]: time="2023-06-05T11:15:49-03:00" level=info msg="Rancher System Agent version v0.3.2 (afbc4aa) is starting"
Jun 05 11:15:49 FNCWBSLX577 rancher-system-agent[2085]: time="2023-06-05T11:15:49-03:00" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Jun 05 11:15:49 FNCWBSLX577 rancher-system-agent[2085]: time="2023-06-05T11:15:49-03:00" level=info msg="Starting remote watch of plans"
Jun 05 11:15:50 FNCWBSLX577 rancher-system-agent[2085]: E0605 11:15:50.067940    2085 memcache.go:206] couldn't get resource list for <http://management.cattle.io/v3|management.cattle.io/v3>:
Jun 05 11:15:50 FNCWBSLX577 rancher-system-agent[2085]: time="2023-06-05T11:15:50-03:00" level=info msg="Starting /v1, Kind=Secret controller"
It does no go ahead with the process of instalation of rke2-server service. This is the log from the same process, but using the etcd role.
Copy code
Jun 05 09:59:12 FNCWBSLX572 systemd[1]: Started Rancher System Agent.
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Rancher System Agent version v0.3.2 (afbc4aa) is starting"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Starting remote watch of plans"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: E0605 09:59:13.163899  682854 memcache.go:206] couldn't get resource list for <http://management.cattle.io/v3|management.cattle.io/v3>:
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Starting /v1, Kind=Secret controller"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Detected first start, force-applying one-time instruction set"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 2f483f7c0bd6065bb48dbb513441d13d34a113d0e38df1daabfc246916041b1b"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.25.9-rke2r1 to directory /var/lib/rancher/agent/work/20230605-095913/2f483f7c0bd6065bb48dbb513441d13d34a113d0e38df1daabfc246916041b1b_0"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jun 05 09:59:13 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:13-03:00" level=info msg="Pulling image <http://index.docker.io/rancher/system-agent-installer-rke2:v1.25.9-rke2r1|index.docker.io/rancher/system-agent-installer-rke2:v1.25.9-rke2r1>"
Jun 05 09:59:14 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:14-03:00" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20230605-095913/2f483f7c0bd6065bb48dbb513441d13d34a113d0e38df1daabfc246916041b1b_0/installer.sh"
Jun 05 09:59:14 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:14-03:00" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20230605-095913/2f483f7c0bd6065bb48dbb513441d13d34a113d0e38df1daabfc246916041b1b_0/rke2.linux-amd64.tar.gz"
Jun 05 09:59:17 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:17-03:00" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20230605-095913/2f483f7c0bd6065bb48dbb513441d13d34a113d0e38df1daabfc246916041b1b_0/sha256sum-amd64.txt"
Jun 05 09:59:17 FNCWBSLX572 rancher-system-agent[682854]: time="2023-06-05T09:59:17-03:00" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20230605-095913/2f483f7c0bd6065bb48dbb513441d13d34a113d0e38df1daabfc246916041b1b_0/run.sh"
...
Rancher Version: 2.7.2 Rancher RKE2 Cluster Version: v1.24.12 +rke2r1 Downstream RKE2 Cluster version: v1.25.9+rke2r1 Actual state fo Downstream Cluster in Rancher UI: Updating (Configuring control plane node(s) custom-b0de6dd768ec,custom-f3c8988c82da) Nodes state of the Downstream Cluster:
Copy code
[root@FNCWBSLX572 ~]# k get nodes
NAME          STATUS   ROLES                  AGE     VERSION
fncwbslx572   Ready    etcd                   2d23h   v1.25.9+rke2r1
fncwbslx573   Ready    etcd                   2d23h   v1.25.9+rke2r1
fncwbslx574   Ready    etcd                   2d23h   v1.25.9+rke2r1
fncwbslx575   Ready    control-plane,master   2d23h   v1.25.9+rke2r1
fncwbslx576   Ready    control-plane,master   2d23h   v1.25.9+rke2r1
Any help will be appreciated. Thank's in advance.
a
you are using the rke2-agent service right?
j
Hi Simon, I'm just installing a downstream cluster using the command generated by Rancher UI. I noticed a certificate error in the cattle-cluster-agent:
Copy code
$ k logs cattle-cluster-agent-5dfb5dc455-p4bb7 -n cattle-system
...
...x509: certificate is valid for ingress.local, not <http://rancher.mycompany.com.br|rancher.mycompany.com.br>
...
I'm trying to install a valid certificate on NGinx Load Balancer to see if it solve the certificate issue.
👍 1
Hi, I've tried to registrer a worker node on a Rancher 2.7.2 with valid certificate but the start the worker installation proccess does not starts. It shows that it's not a certificate problem. Steps to reproduce: Install a single RKE2 Node (v1.25.9+rke2r1) with all roles (etcd, controlplane and worker) Install Rancher 2.7.2 using Helm and Rancher managed certificate Try to create a Downstream cluster starting from the worker role only.
Hi, I updated the Rancher letsencrypt certificate to a valid certificate with Public CA. In the Browser, the site shows the certificate is valid and the connection is secure, but the cattle-cluster-agent pod log shows the error:
Copy code
time="2023-06-07T20:20:44Z" level=error msg="Certficate's Subject (CN=GlobalSign RSA OV SSL CA 2018,O=GlobalSign nv-sa,C=BE) does not match with previous certificate Issuer (CN=Valid Certificadora Digital AlphaSSL CA 2018,O=VALID CERTIFICADORA DIGITAL,C=BR). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)"
Any help will be appreciated.
a
You need the chain or registered intermediate somewhere or if you concat them in the same tls secret
j
Hi Simon, What I did was: 1 - renamed the certificate to tls.crt 2 - renamed the key to tls.key 3 - Generate the tls-rancher-ingress secter with the command: 
k -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key --dry-run=client -o yaml > finaxis-tls-rancher-ingress.yaml
and applyed the manifest to create the secret in cattle-system namespace. After that, I redeploy the rancher pods and the certificate on Browser was shown as Valid. I followed this guide: Updating from a Private CA Certificate to a Public CA Certificate I don't know what would be the intermediate certificate or how to get the chain.
a
yeah, can you join tls.crt and intermediate ca into one file and then generate tls
you can check in Browser the whole chain if it works 🙂 and you can export from there or if you know the CA has it for download
j
Please, could you provide some example commands about how to join tls.crt and intermediate CA that I can start this process? When you say "Intermediate CA", you are refering to the tls.key?
a
cat file1 file2 > file3
no key is a key the most important this is always standalone
but you can put more certs into one tls.crt
here's the example
when you scroll down and you have a chain of certificates
--- begin --- --- end --- --- begin ---
you can have all the certs
j
Thank's for the example. Let me see if I understand... I have to copy the respoective CA bundle to a file, for example, bundle.crt. After that, I can concatenate my certificate.crt with the bundle.crt to a tls.crt file: 
cat mycertificate.crt bundle.crt > tls.crt
And finally execute the steps to create the 
tls-rancher-ingress
secret?
a
Thats the general idea yes
👍 1
j
I'll try it right now...
a
There’s also separate ca.crt I think you can put into secret
Into tls secret if you want to organize it better otherwise doesn’t matter much
j
The first try didn't works. Propbably I'm missing something. I'll continuing trying until it works... This is the cattle-clsuter-agent pod logs:
Copy code
k logs cattle-cluster-agent-66bcc79d4d-fzcph -n cattle-system

INFO: Environment: CATTLE_ADDRESS=10.42.73.194 CATTLE_CA_CHECKSUM= CATTLE_CLUSTER=true CATTLE_CLUSTER_AGENT_PORT=<tcp://10.43.166.209:80> CATTLE_CLUSTER_AGENT_PORT_443_TCP=<tcp://10.43.166.209:443> CATTLE_CLUSTER_AGENT_PORT_443_TCP_ADDR=10.43.166.209 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PORT=443 CATTLE_CLUSTER_AGENT_PORT_443_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_PORT_80_TCP=<tcp://10.43.166.209:80> CATTLE_CLUSTER_AGENT_PORT_80_TCP_ADDR=10.43.166.209 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PORT=80 CATTLE_CLUSTER_AGENT_PORT_80_TCP_PROTO=tcp CATTLE_CLUSTER_AGENT_SERVICE_HOST=10.43.166.209 CATTLE_CLUSTER_AGENT_SERVICE_PORT=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTP=80 CATTLE_CLUSTER_AGENT_SERVICE_PORT_HTTPS_INTERNAL=443 CATTLE_CLUSTER_REGISTRY= CATTLE_FEATURES=embedded-cluster-api=false,fleet=false,monitoringv1=false,multi-cluster-management=false,multi-cluster-management-agent=true,provisioningv2=false,rke2=false CATTLE_INGRESS_IP_DOMAIN=<http://sslip.io|sslip.io> CATTLE_INSTALL_UUID=4fc5293b-6d65-4623-a3db-0872f427be71 CATTLE_INTERNAL_ADDRESS= CATTLE_IS_RKE=false CATTLE_K8S_MANAGED=true CATTLE_NODE_NAME=cattle-cluster-agent-66bcc79d4d-fzcph CATTLE_SERVER=<https://rancher.mycompany.com.br> CATTLE_SERVER_VERSION=v2.7.2
INFO: Using resolv.conf: search cattle-system.svc.cluster.local svc.cluster.local cluster.local <http://intra.grupopetra.com.br|intra.grupopetra.com.br> nameserver 10.43.0.10 options ndots:5
INFO: <https://rancher.mycompany.com.br/ping> is accessible
INFO: <http://rancher.mycompany.com.br|rancher.mycompany.com.br> resolves to 172.16.1.177
time="2023-06-07T21:42:09Z" level=info msg="Listening on /tmp/log.sock"
time="2023-06-07T21:42:09Z" level=info msg="Rancher agent version v2.7.2 is starting"
time="2023-06-07T21:42:09Z" level=info msg="Certificate details from <https://rancher.mycompany.com.br>"
time="2023-06-07T21:42:09Z" level=info msg="Certificate #0 (<https://rancher.mycompany.com.br>)"
time="2023-06-07T21:42:09Z" level=info msg="Subject: CN=*.<http://mycompany.com.br|mycompany.com.br>"
time="2023-06-07T21:42:09Z" level=info msg="Issuer: CN=Valid Certificadora Digital AlphaSSL CA 2018,O=VALID CERTIFICADORA DIGITAL,C=BR"
time="2023-06-07T21:42:09Z" level=info msg="IsCA: false"
time="2023-06-07T21:42:09Z" level=info msg="DNS Names: [*.<http://mycompany.com.br|mycompany.com.br> <http://mycompany.com.br|mycompany.com.br>]"
time="2023-06-07T21:42:09Z" level=info msg="IPAddresses: <none>"
time="2023-06-07T21:42:09Z" level=info msg="NotBefore: 2022-11-03 16:46:14 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="NotAfter: 2023-09-19 00:00:00 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2023-06-07T21:42:09Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-06-07T21:42:09Z" level=info msg="Certificate #1 (<https://rancher.mycompany.com.br>)"
time="2023-06-07T21:42:09Z" level=info msg="Subject: CN=GlobalSign RSA OV SSL CA 2018,O=GlobalSign nv-sa,C=BE"
time="2023-06-07T21:42:09Z" level=info msg="Issuer: CN=GlobalSign,OU=GlobalSign Root CA - R3,O=GlobalSign"
time="2023-06-07T21:42:09Z" level=info msg="IsCA: true"
time="2023-06-07T21:42:09Z" level=info msg="DNS Names: <none>"
time="2023-06-07T21:42:09Z" level=info msg="IPAddresses: <none>"
time="2023-06-07T21:42:09Z" level=info msg="NotBefore: 2018-11-21 00:00:00 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="NotAfter: 2028-11-21 00:00:00 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2023-06-07T21:42:09Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-06-07T21:42:09Z" level=error msg="Certficate's Subject (CN=GlobalSign RSA OV SSL CA 2018,O=GlobalSign nv-sa,C=BE) does not match with previous certificate Issuer (CN=Valid Certificadora Digital AlphaSSL CA 2018,O=VALID CERTIFICADORA DIGITAL,C=BR). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)"
time="2023-06-07T21:42:09Z" level=info msg="Certificate #2 (<https://rancher.mycompany.com.br>)"
time="2023-06-07T21:42:09Z" level=info msg="Subject: CN=GlobalSign,OU=GlobalSign Root CA - R3,O=GlobalSign"
time="2023-06-07T21:42:09Z" level=info msg="Issuer: CN=GlobalSign,OU=GlobalSign Root CA - R3,O=GlobalSign"
time="2023-06-07T21:42:09Z" level=info msg="IsCA: true"
time="2023-06-07T21:42:09Z" level=info msg="DNS Names: <none>"
time="2023-06-07T21:42:09Z" level=info msg="IPAddresses: <none>"
time="2023-06-07T21:42:09Z" level=info msg="NotBefore: 2009-03-18 10:00:00 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="NotAfter: 2029-03-18 10:00:00 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="SignatureAlgorithm: SHA256-RSA"
time="2023-06-07T21:42:09Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-06-07T21:42:09Z" level=info msg="Certificate #3 (<https://rancher.mycompany.com.br>)"
time="2023-06-07T21:42:09Z" level=info msg="Subject: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE"
time="2023-06-07T21:42:09Z" level=info msg="Issuer: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE"
time="2023-06-07T21:42:09Z" level=info msg="IsCA: true"
time="2023-06-07T21:42:09Z" level=info msg="DNS Names: <none>"
time="2023-06-07T21:42:09Z" level=info msg="IPAddresses: <none>"
time="2023-06-07T21:42:09Z" level=info msg="NotBefore: 2000-05-30 10:48:38 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="NotAfter: 2020-05-30 10:48:38 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="SignatureAlgorithm: SHA1-RSA"
time="2023-06-07T21:42:09Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-06-07T21:42:09Z" level=error msg="Certficate's Subject (CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE) does not match with previous certificate Issuer (CN=GlobalSign,OU=GlobalSign Root CA - R3,O=GlobalSign). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)"
time="2023-06-07T21:42:09Z" level=info msg="Certificate #4 (<https://rancher.mycompany.com.br>)"
time="2023-06-07T21:42:09Z" level=info msg="Subject: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US"
time="2023-06-07T21:42:09Z" level=info msg="Issuer: CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE"
time="2023-06-07T21:42:09Z" level=info msg="IsCA: true"
time="2023-06-07T21:42:09Z" level=info msg="DNS Names: <none>"
time="2023-06-07T21:42:09Z" level=info msg="IPAddresses: <none>"
time="2023-06-07T21:42:09Z" level=info msg="NotBefore: 2000-05-30 10:48:38 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="NotAfter: 2020-05-30 10:48:38 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="SignatureAlgorithm: SHA384-RSA"
time="2023-06-07T21:42:09Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-06-07T21:42:09Z" level=error msg="Certficate's Subject (CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US) does not match with previous certificate Issuer (CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)"
time="2023-06-07T21:42:09Z" level=info msg="Certificate #5 (<https://rancher.mycompany.com.br>)"
time="2023-06-07T21:42:09Z" level=info msg="Subject: CN=Sectigo RSA Organization Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB"
time="2023-06-07T21:42:09Z" level=info msg="Issuer: CN=USERTrust RSA Certification Authority,O=The USERTRUST Network,L=Jersey City,ST=New Jersey,C=US"
time="2023-06-07T21:42:09Z" level=info msg="IsCA: true"
time="2023-06-07T21:42:09Z" level=info msg="DNS Names: <none>"
time="2023-06-07T21:42:09Z" level=info msg="IPAddresses: <none>"
time="2023-06-07T21:42:09Z" level=info msg="NotBefore: 2018-11-02 00:00:00 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="NotAfter: 2030-12-31 23:59:59 +0000 UTC"
time="2023-06-07T21:42:09Z" level=info msg="SignatureAlgorithm: SHA384-RSA"
time="2023-06-07T21:42:09Z" level=info msg="PublicKeyAlgorithm: RSA"
time="2023-06-07T21:42:09Z" level=error msg="Certficate's Subject (CN=Sectigo RSA Organization Validation Secure Server CA,O=Sectigo Limited,L=Salford,ST=Greater Manchester,C=GB) does not match with previous certificate Issuer (CN=AddTrust External CA Root,OU=AddTrust External TTP Network,O=AddTrust AB,C=SE). Please check if the configured server certificate contains all needed intermediate certificates and make sure they are in the correct order (server certificate first, intermediates after)"
time="2023-06-07T21:42:09Z" level=fatal msg="Certificate chain is not complete, please check if all needed intermediate certificates are included in the server certificate (in the correct order) and if the cacerts setting in Rancher either contains the correct CA certificate (in the case of using self signed certificates) or is empty (in the case of using a certificate signed by a recognized CA). Certificate information is displayed above. error: Get \"<https://rancher.mycompany.com.br>\": x509: certificate signed by unknown authority"
a
It detected more certs that’s okay, the main cert should be on top of file though
I hope you didn’t copy examples I showed you ;) you got the ca bundles when you received certs
j
rsrsrs. I found the page to download the chain. I'll download and retry...
a
I guess it worked 😄
j
Hi Simon, yesterday was hollyday here in Brazil, so I didn't work. Today I'll continuing following your tips to install the certificate with the full CA certificate chain in Rancher. I talk you about the evolution.
👍 1
Hi Simon good news!!! It Finally worked!!! I did what you taold me to do. 1 - I exported and download all CA certificates chain (one by one - 3 in total) via Browser (Chrome) 2 - I concatenated mycompany.crt with the 3 CA exported certificates in a file names tls.crt 3 - Finally I created the tls-rancher-ingress secret providing the tls.key and tls.crt. Very thanks for the help my friend!!!
✅ 1
🎉 1
976 Views
Open in Slack
PreviousNext
Powered by