I apologize if this questions has already been answered. I just recently joined this slack channel. I am using Windows Rancher Desktop (v1.7.0) with WSL to develop a local Kubernetes cluster (v1.25.5) with pods the need to federate a projected service account to a managed identitiy in Azure. I have generated RSA public/private key files, which will be used by the Kubernetes API server to sign the projected service account token and by Azure to verify the token's signature. For this, I created a provisioning script that 1. generates a config.yaml file in the /etc/rancher/k3s directory to configure the k3s kube-apiserver service, 2. copies the RSA public/private key files to the /etc/rancher/k3s/pki directory, 3. change the mode of the RSA key files to 644. When I start Rancher Desktop and review the k3s.log file, the kube-apiserver service tries to start with the updated service-account-* command line arguments. However, the kube-apiserver service writes this error to the log, "Error: failed to parse service-account-issuer-key-file: open "/etc/kubernetes/pki/sa.key": no such file or directory". I have opened the rancher-desktop WSL distro and verified that I can read both key files at their specified locations. I have also tried placing the keys files in a /var/lib/rancher/k3s/server/pki directory. However, the kube-apiserver service throws the same error. Unfortunately, I have not found much on the internet for a go-forward path. Am I trying to do something that Rancher Desktop cannot do at this time? Should I write the RSA keys to a different directory?

Hi @dazzling-twilight-41550 you should check with the k3s folks about what to do here since we haven't run into this before. Let me know if you have already and if they pointed back to us with where to look. I didn't find any references about locations in https://docs.k3s.io/ but maybe #k3s can offer some guidance if you haven't heard from them yet.

hi @dazzling-twilight-41550, based on the error message, my guess is that you're not targeting the right WSL distro. While you have access to Rancher Desktop Kubernetes cluster from your WSL distro, please note that

k3s

binary is running only on the Rancher Desktop distribution

rancher-desktop

. With that said, can I kindly ask you to create the pki files as follow:

wsl -d rancher-desktop
# run your provisioning script

ATTENTION: please note that Rancher Desktop distro is "ephemeral" , all your changes will be lost.

Hi @magnificent-napkin-96586 and @thankful-hospital-82027,

@dazzling-twilight-41550 thanks for the further details, it helps as a

provisioning script

could exist for many layers. So, it's fair to assume that you created a

$env:USERPROFILE\AppData\Roaming\rancher-desktop\provisioning\[script name].start

with the content you shared above? I tried it, and at first I had also the same issue, the files didn't appear, however as stated in the docs, the line endings should be Unix format and not DOS. Example: if you use VSCode, you'll need to change the line endings from CRLF to LF. Once I've done that change, the directories and files correctly appeared in Rancher Desktop distro

@thankful-hospital-82027 - Thank you for the quick response. You are correct. Per the documentation, I created the

k3s_overrides.start

file in the

$env:USERPROFILE\AppData\Roaming\rancher-desktop\provisioning

windows folder and added the contents above to the file. I did verify that the line endings were LF in the provisioning script file. When Rancher Desktop runs, it executes the provisioning script and creates the files as expected in the rancher-desktop distro. However, when Rancher Desktop starts k3s in the rancher-desktop distro, the kube-apiserver begins throwing error messages stating that the sa.key file or directory could not be found. Once this happens, authentication fails for other components and causes Rancher Desktop to hang, while configuring kubectl. Unfortunately, when this happens, the only option left is to perform a factory reset and start over. I am wondering if this is related to k3s and not Rancher Desktop. I will post a message to the #k3s channel to see if this is a k3s configuration issue.

@dazzling-twilight-41550 thanks for the extra details and yes, let's see what K3s team answers you. Still, one last question/request: which container runtime are your running? and have you tried running it with the other (if you're using Docker, switch to containerd and vice-versa)

@thankful-hospital-82027 - Per your suggestion, I switched container engine from dockerd (moby) to containerd. Unfortunately, I am still seeing the same errors in the k3s.log file.

time="2023-01-27T16:41:00Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=<unix://kine.sock> --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=\"<https://csgcorpitaaioidcissuer01.blob.core.windows.net/oidc-test/>\" --service-account-key-file=\"/etc/rancher/k3s/pki/sa.pub\" --service-account-signing-key-file=\"/etc/rancher/k3s/pki/sa.key\" --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-01-27T16:41:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-01-27T16:41:00Z" level=info msg="Waiting for API server to become available"
I0127 16:41:00.631039     467 server.go:581] external host was not specified, using 192.168.67.3
Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory