dazzling-twilight-41550
01/27/2023, 5:50 PMconfig.yaml
file in the /etc/rancher/k3s
directory to configure the k3s kube-apiserver service,
• copies the RSA public/private key files to the /etc/rancher/k3s/pki
directory,
• change the mode of the RSA key files to 644.
When I start Rancher Desktop and review the k3s.log file, the kube-apiserver service tries to start with the updated service-account-* command line arguments. However, the kube-apiserver service writes this error to the log, "`Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory`". I have opened the rancher-desktop WSL distro and verified that I can read both key files at their specified locations. I have also tried placing the keys files in a /var/lib/rancher/k3s/server/pki
directory and updated the config.yaml to point to the new directory. However, the kube-apiserver service throws the same error. Unfortunately, I have not found much on the internet for a go-forward path. Should I write the RSA keys to a different directory in order for k3s to find the RSA key files? Is there another setting that I need to configure for k3s to have?
Rancher Desktop 1.7.0
Kubernetes 1.25.6
provisioning script:
#!/bin/sh
mkdir -p /etc/rancher/k3s
cat > /etc/rancher/k3s/config.yaml <<EOF
kube-apiserver-arg:
- service-account-signing-key-file="/etc/rancher/k3s/pki/sa.key"
- service-account-key-file="/etc/rancher/k3s/pki/sa.pub"
- service-account-issuer="<https://www.example.net/oidc-test/>"
EOF
mkdir -p /etc/rancher/k3s/pki
cat > /etc/rancher/k3s/pki/sa.key <<EOF2
-----BEGIN PRIVATE KEY-----
MII...
-----END PRIVATE KEY-----
EOF2
chmod 644 /etc/rancher/k3s/pki/sa.key
cat > /etc/rancher/k3s/pki/sa.pub <<EOF3
-----BEGIN PUBLIC KEY-----
MII...
-----END PUBLIC KEY-----
EOF3
chmod 644 /etc/rancher/k3s/pki/sa.pub
k3s.log
time="2023-01-27T16:40:54Z" level=info msg="Starting k3s v1.25.6+k3s1 (9176e03c)"
time="2023-01-27T16:40:54Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2023-01-27T16:40:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2023-01-27T16:40:55Z" level=info msg="Database tables and indexes are up to date"
time="2023-01-27T16:40:55Z" level=info msg="Kine available at <unix://kine.sock>"
time="2023-01-27T16:40:55Z" level=info msg="Reconciling bootstrap data between datastore and disk"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy mode: agent"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=<unix://kine.sock> --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=\"<https://csgcorpitaaioidcissuer01.blob.core.windows.net/oidc-test/>\" --service-account-key-file=\"/etc/rancher/k3s/pki/sa.pub\" --service-account-signing-key-file=\"/etc/rancher/k3s/pki/sa.key\" --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-01-27T16:40:55Z" level=info msg="Waiting for API server to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
I0127 16:40:55.053432 436 server.go:581] external host was not specified, using 192.168.67.3
Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory
time="2023-01-27T16:40:55Z" level=fatal msg="apiserver exited: failed to parse service-account-issuer-key-file: open \"/etc/rancher/k3s/pki/sa.key\": no such file or directory"
time="2023-01-27T16:40:55Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
creamy-pencil-82913
01/27/2023, 7:29 PMdazzling-twilight-41550
01/27/2023, 8:01 PMcreamy-pencil-82913
01/27/2023, 8:02 PM/var/lib/rancher/k3s/server/tls/
instead of in /etc/rancher/k3s/pki/
?dazzling-twilight-41550
01/27/2023, 9:10 PMsa.key
file. I tried placing the RSA key files the following directories and updating the config.yaml
to point to the directory:
• /etc/kubernetes/pki
• /etc/rancher/k3s/pki
• /etc/rancher/k3s
• /var/lib/rancher/k3s/server/tls
• /var/lib/rancher/k3s/server/pki
• /var/lib/rancher/k3s/server
When I log into the rancher-desktop
distro, I can see that the files are located in the specified folder in the configuration with a mode of 644. However, k3s logs the error stating that the file does not exist. Based on the conversation on the Rancher Desktop thread and the Rancher Desktop documentation, the provisioning scripts execute prior to the start of k3s. I was able to visually verify this is the case. Also, I just retried using the /var/lib/rancher/k3s/server/tls
directory. Here is the log entry from the k3s.log:
Error: failed to parse service-account-issuer-key-file: open "/var/lib/rancher/k3s/server/tls/sa.key": no such file or directory
creamy-pencil-82913
01/27/2023, 9:34 PMdazzling-twilight-41550
01/27/2023, 10:19 PM