I apologize if this question has already been answered. I just recently joined this slack channel. I am using Windows Rancher Desktop (v1.7.0) with WSL to develop a local Kubernetes cluster (v1.25.6) with pods the need to federate a projected service account to a managed identity in Azure. I have generated RSA public/private key files, which will be used by the Kubernetes API server to sign the projected service account token and by Azure to verify the token's signature. For this, I use a provisioning script that • generates a

config.yaml

file in the

/etc/rancher/k3s

directory to configure the k3s kube-apiserver service, • copies the RSA public/private key files to the

/etc/rancher/k3s/pki

directory, • change the mode of the RSA key files to 644. When I start Rancher Desktop and review the k3s.log file, the kube-apiserver service tries to start with the updated service-account-* command line arguments. However, the kube-apiserver service writes this error to the log, "`Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory`". I have opened the rancher-desktop WSL distro and verified that I can read both key files at their specified locations. I have also tried placing the keys files in a

/var/lib/rancher/k3s/server/pki

directory and updated the config.yaml to point to the new directory. However, the kube-apiserver service throws the same error. Unfortunately, I have not found much on the internet for a go-forward path. Should I write the RSA keys to a different directory in order for k3s to find the RSA key files? Is there another setting that I need to configure for k3s to have?

Rancher Desktop 1.7.0
Kubernetes 1.25.6

provisioning script:

#!/bin/sh

mkdir -p /etc/rancher/k3s
cat > /etc/rancher/k3s/config.yaml <<EOF
kube-apiserver-arg:
- service-account-signing-key-file="/etc/rancher/k3s/pki/sa.key"
- service-account-key-file="/etc/rancher/k3s/pki/sa.pub"
- service-account-issuer="<https://www.example.net/oidc-test/>"
EOF

mkdir -p /etc/rancher/k3s/pki
cat > /etc/rancher/k3s/pki/sa.key <<EOF2
-----BEGIN PRIVATE KEY-----
MII...
-----END PRIVATE KEY-----
EOF2
chmod 644 /etc/rancher/k3s/pki/sa.key

cat > /etc/rancher/k3s/pki/sa.pub <<EOF3
-----BEGIN PUBLIC KEY-----
MII...
-----END PUBLIC KEY-----
EOF3
chmod 644 /etc/rancher/k3s/pki/sa.pub

k3s.log

time="2023-01-27T16:40:54Z" level=info msg="Starting k3s v1.25.6+k3s1 (9176e03c)"
time="2023-01-27T16:40:54Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2023-01-27T16:40:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2023-01-27T16:40:55Z" level=info msg="Database tables and indexes are up to date"
time="2023-01-27T16:40:55Z" level=info msg="Kine available at <unix://kine.sock>"
time="2023-01-27T16:40:55Z" level=info msg="Reconciling bootstrap data between datastore and disk"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy mode: agent"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=<unix://kine.sock> --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=\"<https://csgcorpitaaioidcissuer01.blob.core.windows.net/oidc-test/>\" --service-account-key-file=\"/etc/rancher/k3s/pki/sa.pub\" --service-account-signing-key-file=\"/etc/rancher/k3s/pki/sa.key\" --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-01-27T16:40:55Z" level=info msg="Waiting for API server to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
I0127 16:40:55.053432     436 server.go:581] external host was not specified, using 192.168.67.3
Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory
time="2023-01-27T16:40:55Z" level=fatal msg="apiserver exited: failed to parse service-account-issuer-key-file: open \"/etc/rancher/k3s/pki/sa.key\": no such file or directory"
time="2023-01-27T16:40:55Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"

I would probably take this to #rancher-desktop

Hi @creamy-pencil-82913. Thank you for the quick response. Actually, I started with a thread on the rancher-desktop channel. @magnificent-napkin-96586 suggested that I post this question to the #k3s channel, since it appeared to a possible issue with they way that I was configuring k3s Here is a link to the thread on the #rancher-desktop channel. https://rancher-users.slack.com/archives/C0200L1N1MM/p1674771678215759

It doesn’t appear to me to be an issue with k3s, but rather how rancher-desktop sets up the VM paths.

Actually, I have tried dropping the RSA files into several different directories for k3s to pick the

sa.key

file. I tried placing the RSA key files the following directories and updating the

config.yaml

to point to the directory: •

/etc/kubernetes/pki

•

/etc/rancher/k3s/pki

•

/etc/rancher/k3s

•

/var/lib/rancher/k3s/server/tls

•

/var/lib/rancher/k3s/server/pki

•

/var/lib/rancher/k3s/server

When I log into the

rancher-desktop

distro, I can see that the files are located in the specified folder in the configuration with a mode of 644. However, k3s logs the error stating that the file does not exist. Based on the conversation on the Rancher Desktop thread and the Rancher Desktop documentation, the provisioning scripts execute prior to the start of k3s. I was able to visually verify this is the case. Also, I just retried using the

/var/lib/rancher/k3s/server/tls

directory. Here is the log entry from the k3s.log:

Error: failed to parse service-account-issuer-key-file: open "/var/lib/rancher/k3s/server/tls/sa.key": no such file or directory

The error is pretty cut and dried from the k3s side. It’s not there. Are you sure that you’re in the correct VM?

It is my understanding that Rancher Desktop installs and runs k3s in their rancher-desktop WSL distro on Windows. I have also verified that the RSA key files are created in the rancher-desktop distro. I will go back to my original thread in the #rancher-desktop and verify this.