https://rancher.com/ logo
Title
d

dazzling-twilight-41550

01/27/2023, 5:50 PM
I apologize if this question has already been answered. I just recently joined this slack channel. I am using Windows Rancher Desktop (v1.7.0) with WSL to develop a local Kubernetes cluster (v1.25.6) with pods the need to federate a projected service account to a managed identity in Azure. I have generated RSA public/private key files, which will be used by the Kubernetes API server to sign the projected service account token and by Azure to verify the token's signature. For this, I use a provisioning script that • generates a
config.yaml
file in the
/etc/rancher/k3s
directory to configure the k3s kube-apiserver service, • copies the RSA public/private key files to the
/etc/rancher/k3s/pki
directory, • change the mode of the RSA key files to 644. When I start Rancher Desktop and review the k3s.log file, the kube-apiserver service tries to start with the updated service-account-* command line arguments. However, the kube-apiserver service writes this error to the log, "`Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory`". I have opened the rancher-desktop WSL distro and verified that I can read both key files at their specified locations. I have also tried placing the keys files in a
/var/lib/rancher/k3s/server/pki
directory and updated the config.yaml to point to the new directory. However, the kube-apiserver service throws the same error. Unfortunately, I have not found much on the internet for a go-forward path. Should I write the RSA keys to a different directory in order for k3s to find the RSA key files? Is there another setting that I need to configure for k3s to have?
Rancher Desktop 1.7.0
Kubernetes 1.25.6
provisioning script:
#!/bin/sh

mkdir -p /etc/rancher/k3s
cat > /etc/rancher/k3s/config.yaml <<EOF
kube-apiserver-arg:
- service-account-signing-key-file="/etc/rancher/k3s/pki/sa.key"
- service-account-key-file="/etc/rancher/k3s/pki/sa.pub"
- service-account-issuer="<https://www.example.net/oidc-test/>"
EOF

mkdir -p /etc/rancher/k3s/pki
cat > /etc/rancher/k3s/pki/sa.key <<EOF2
-----BEGIN PRIVATE KEY-----
MII...
-----END PRIVATE KEY-----
EOF2
chmod 644 /etc/rancher/k3s/pki/sa.key

cat > /etc/rancher/k3s/pki/sa.pub <<EOF3
-----BEGIN PUBLIC KEY-----
MII...
-----END PUBLIC KEY-----
EOF3
chmod 644 /etc/rancher/k3s/pki/sa.pub
k3s.log
time="2023-01-27T16:40:54Z" level=info msg="Starting k3s v1.25.6+k3s1 (9176e03c)"
time="2023-01-27T16:40:54Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
time="2023-01-27T16:40:54Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
time="2023-01-27T16:40:55Z" level=info msg="Database tables and indexes are up to date"
time="2023-01-27T16:40:55Z" level=info msg="Kine available at <unix://kine.sock>"
time="2023-01-27T16:40:55Z" level=info msg="Reconciling bootstrap data between datastore and disk"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy mode: agent"
time="2023-01-27T16:40:55Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=<https://kubernetes.default.svc.cluster.local>,k3s --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --etcd-servers=<unix://kine.sock> --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=\"<https://csgcorpitaaioidcissuer01.blob.core.windows.net/oidc-test/>\" --service-account-key-file=\"/etc/rancher/k3s/pki/sa.pub\" --service-account-signing-key-file=\"/etc/rancher/k3s/pki/sa.key\" --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --leader-elect=false --profiling=false --secure-port=10259"
time="2023-01-27T16:40:55Z" level=info msg="Waiting for API server to become available"
time="2023-01-27T16:40:55Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --leader-elect=false --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true"
I0127 16:40:55.053432     436 server.go:581] external host was not specified, using 192.168.67.3
Error: failed to parse service-account-issuer-key-file: open "/etc/rancher/k3s/pki/sa.key": no such file or directory
time="2023-01-27T16:40:55Z" level=fatal msg="apiserver exited: failed to parse service-account-issuer-key-file: open \"/etc/rancher/k3s/pki/sa.key\": no such file or directory"
time="2023-01-27T16:40:55Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/k3s/server/etc/cloud-config.yaml --cloud-provider=k3s --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route --kubeconfig=/var/lib/rancher/k3s/server/cred/cloud-controller.kubeconfig --leader-elect=false --leader-elect-resource-name=k3s-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false"
c

creamy-pencil-82913

01/27/2023, 7:29 PM
I would probably take this to #rancher-desktop
d

dazzling-twilight-41550

01/27/2023, 8:01 PM
Hi @creamy-pencil-82913. Thank you for the quick response. Actually, I started with a thread on the rancher-desktop channel. @magnificent-napkin-96586 suggested that I post this question to the #k3s channel, since it appeared to a possible issue with they way that I was configuring k3s Here is a link to the thread on the #rancher-desktop channel. https://rancher-users.slack.com/archives/C0200L1N1MM/p1674771678215759
c

creamy-pencil-82913

01/27/2023, 8:02 PM
It doesn’t appear to me to be an issue with k3s, but rather how rancher-desktop sets up the VM paths.
the file you’re creating isn’t available in the VM in the location where it needs to be
have you tried just creating the sa.key file at the default location of
/var/lib/rancher/k3s/server/tls/
instead of in
/etc/rancher/k3s/pki/
?
d

dazzling-twilight-41550

01/27/2023, 9:10 PM
Actually, I have tried dropping the RSA files into several different directories for k3s to pick the
sa.key
file. I tried placing the RSA key files the following directories and updating the
config.yaml
to point to the directory: •
/etc/kubernetes/pki
/etc/rancher/k3s/pki
/etc/rancher/k3s
/var/lib/rancher/k3s/server/tls
/var/lib/rancher/k3s/server/pki
/var/lib/rancher/k3s/server
When I log into the
rancher-desktop
distro, I can see that the files are located in the specified folder in the configuration with a mode of 644. However, k3s logs the error stating that the file does not exist. Based on the conversation on the Rancher Desktop thread and the Rancher Desktop documentation, the provisioning scripts execute prior to the start of k3s. I was able to visually verify this is the case. Also, I just retried using the
/var/lib/rancher/k3s/server/tls
directory. Here is the log entry from the k3s.log:
Error: failed to parse service-account-issuer-key-file: open "/var/lib/rancher/k3s/server/tls/sa.key": no such file or directory
c

creamy-pencil-82913

01/27/2023, 9:34 PM
The error is pretty cut and dried from the k3s side. It’s not there. Are you sure that you’re in the correct VM?
I was under the impression that there are multiple layers of nesting on some platforms
but again I don’t know much about Rancher Desktop
d

dazzling-twilight-41550

01/27/2023, 10:19 PM
It is my understanding that Rancher Desktop installs and runs k3s in their rancher-desktop WSL distro on Windows. I have also verified that the RSA key files are created in the rancher-desktop distro. I will go back to my original thread in the #rancher-desktop and verify this.