hi, i was able to find track down the issue with read-only mounts when you enable audit logs, here is cluster configuration:
kube-apiserver-arg:
- audit-policy-file=/var/lib/rancher/rke2/agent/policy-test.yaml
- audit-log-path=/var/log/audit-k8s/policy-log.log
kube-apiserver-extra-mount:
- /var/log/audit-k8s/:/var/log/audit-k8s/
Sure, you need to create some policy file before on all master nodes. Everything looks fine:
-rw------- 1 root root 100582521 May 11 13:43 policy-log-2023-05-11T10-43-26.885.log
-rw------- 1 root root 103313216 May 11 13:43 policy-log-2023-05-11T10-43-28.441.log
-rw------- 1 root root 79832869 May 11 13:54 policy-log-2023-05-11T10-54-15.567.log
-rw------- 1 root root 52660013 May 11 13:54 policy-log.log
Here is mounts from pod manifest for kube-api:
- hostPath:
path: /var/log/audit-k8s/
type: DirectoryOrCreate
name: extra-mount-0
- mountPath: /var/log/audit-k8s/
name: extra-mount-0
Everything works until you restart rke2-server, the log file does exist and rke2 generates the following mounts:
- hostPath:
path: /var/log/audit-k8s/policy-log.log
type: File
name: file18
- hostPath:
path: /var/log/audit-k8s/
type: DirectoryOrCreate
name: extra-mount-0
- mountPath: /var/log/audit-k8s/policy-log.log
name: file18
readOnly: true
- mountPath: /var/log/audit-k8s/
name: extra-mount-0
And of course, kube-api starts to crash:
crictl logs cbbfd26e1514d
I0511 10:57:57.404782 1 server.go:558] external host was not specified, using 172.24.150.133
I0511 10:57:57.405249 1 server.go:158] Version: v1.24.2+rke2r1
I0511 10:57:57.405290 1 server.go:160] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0511 10:57:57.914086 1 shared_informer.go:255] Waiting for caches to sync for node_authorizer
E0511 10:57:57.914548 1 run.go:74] "command failed" err="ensureLogFile: open /var/log/audit-k8s/policy-log.log: read-only file system"
Removing /var/log/audit-k8s/policy-log.log fixes the issue till the next restart.