I've created my own self signed ca cert (cacert.pem) -- and then created a client cert and signed it. The closest I've gotten so far is starting k3d like so:

k3d cluster create \
  --volume "$(pwd)/certs:/my-certs@server:0" \
  --k3s-arg "--kube-apiserver-arg=client-ca-file=/my-certs/cacert.pem@server:0" \
[...]

That successfully mounted my cacert on the node... but the way I knew it worked was that I looked at the logs and saw all intra-cluster communication was failing b/c the certs intra-cluster clients were using were no longer valid (because they were presumably signed by the cert k3d creates as part of its provisioning process)

snippet from the log when cluster created as above:

E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
time="2022-10-03T21:02:48Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:6443/v1-k3s/readyz>: 500 Internal Server Error"
E1003 21:02:49.182867       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=883684690406892750, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:49.720672       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:49.720887       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:49.720888       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:50.182981       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=883684690406892750, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:50.719374       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:50.720575       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:50.721399       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:51.190505       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=883684690406892750, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:51.721076       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:51.721133       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:51.722118       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=7508131499274857855, SKID=, AKID=56:E3:81:B1:6A:6C:E6:A6:2B:4E:74:18:7A:D3:E9:9C:44:9F:CC:D9 failed: x509: certificate signed by unknown authority]"
E1003 21:02:52.183632       7 authentication.go:63] "Unable to authenticate the request" err="[x509: certificate signed by unknown authority, verifying certificate SN=883684690406892750, SKID=, AKID=56:E

it hangs waiting to come up... and messages like that just keep showing up in the log. I'm not too familiar with the internals of kubernetes... but I'm guessing those are failed attempts of the kublet to talk to the api?

because the kublet was provisioned a certificate at provisioning time that I stomped all over by replacing the CA that signed it? 🙂

I shelled into the node via:

docker exec -it k3d-k3s-default-server-0 /bin/sh

hoping to find the default cacert in one of the locations that a kubeadmin provisioned cluster would have it (https://kubernetes.io/docs/setup/best-practices/certificates/) but couldn't find it. Where does the cacert (and key) live on the container node? I'd prefer to use my own cert to sign client certs... but, if I had the ones on the container, I could at least use those to sign client certs, right?

You’d probably want to use this signer:

`kubernetes.io/kube-apiserver-client`: signs certificates that will be honored as client certificates by the API server.

There’s an example of how to use that to issue a cert and use it in a kubeconfig at the bottom of the page