https://rancher.com/ logo
Join SlackCommunities
Powered by
Hi Folks, I've got an issue with NFS mounting v4 e...
# k3s
q
Hi Folks, I've got an issue with NFS mounting v4 exports and I'm not sure if this is a k3s issue or a kubernetes issue. What I'm seeing is that on the host, looking at the output of 
mount
shows me that the options used in the mount are not the same that I put in the 
pv
. Is this expected?
c
this isn’t a k3s thing, as we don’t modify upstream behavior at all with regards to mounts
m
If possible give a bit more information, but I have had a similar experience with RKE2
c
the kubelet does have some default mount options though. it doesn’t ONLY use the ones you set in the PV.
were you expecting to see only the options you specify in your volume mountOptions?
q
Copy code
mountOptions:
  - timeo=600
  - retrans=3
  - proto=tcp6
  - nfsvers=4.2
  - rsize=32768
  - wsize=32768
  - noacl
  - noatime
  - nodiratime
  - nolock
  - nosuid
  - nofail
olaf.wire.znet:/data01/stuff on /var/lib/kubelet/pods/5115dfbc-7427-4e63-9a4e-914a8396e43f/volumes/kubernetes.io~nfs/stuff type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp6,timeo=600,retrans=2,sec=sys,mountaddr=fc15::11,mountvers=3,mountport=893,mountproto=udp6,local_lock=none,addr=fc15::11)
At least I was expecting nfsv4 and tcp v6.
c
can you show the whole pv yaml?
m
What is your error exactly? In our case, the error messages complaining about the NFS options were a Red Herring, ended up being the East-West firewall blocking the NFS port.
Answer Brandond first, I'm more noise, he's more constructive. (Edit: clarified my sentence.)
c
the code that translates the options from the spec to mount command flags is here: https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/nfs/nfs.go
👀 1
q
Copy code
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    <http://kubectl.kubernetes.io/last-applied-configuration|kubectl.kubernetes.io/last-applied-configuration>: |
      {"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"tanka.dev/environment":"6b75f2470136c3ddee55479b95978d0fe68d7dc8220fd00f"},"name":"media"},"spec":{"accessModes":["ReadWriteMany"],"capacity":{"storage":"1Gi"},"mountOptions":["timeo=600","retrans=3","proto=tcp6","nfsvers=4.2","rsize=32768","wsize=32768","noacl","noatime","nodiratime","nolock","nosuid","nofail"],"nfs":{"path":"/data01/stuff","readOnly":false,"server":"olaf.wire.znet"}}}
    <http://pv.kubernetes.io/bound-by-controller|pv.kubernetes.io/bound-by-controller>: "yes"
  creationTimestamp: "2025-04-17T02:16:46Z"
  finalizers:
  - <http://kubernetes.io/pv-protection|kubernetes.io/pv-protection>
  labels:
    tanka.dev/environment: 6b75f2470136c3ddee55479b95978d0fe68d7dc8220fd00f
  name: media
  resourceVersion: "634678158"
  uid: ca249f20-ffe2-4f93-a625-3ca20e9a98e5
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: media
    namespace: media
    resourceVersion: "634674471"
    uid: 0bcb4bed-4695-4f77-818c-734bb400fdbe
  mountOptions:
  - timeo=600
  - retrans=3
  - proto=tcp6
  - nfsvers=4.2
  - rsize=32768
  - wsize=32768
  - noacl
  - noatime
  - nodiratime
  - nolock
  - nosuid
  - nofail
  nfs:
    path: /data01/stuff
    server: olaf.wire.znet
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem
status:
  lastPhaseTransitionTime: "2025-04-17T02:16:56Z"
  phase: Bound
There isn't an error currently, its just mounting with the wrong options. I switched to 
prefer-bundled-bin
for the recent bug in linux-util and then ran into https://github.com/k3s-io/k3s/issues/10206 but now I'm patched and so removed prefer-bundled-bin and seeing that the mount is not as requested.
I'll read that code a little closer in a bit.
c
keep in mind that the mount command itself does a fair bit of mangling of the options before passing them to the kernel, I suspect that’s probably what you’re running into.
q
That's fair, but check this out. 
mount -t nfs -o timeo=600,retrans=3,proto=tcp6,nfsvers=4.2,rsize=32768,wsize=32768,noacl,noatime,nodiratime,nolock,nosuid,nofail ...
Results in these options showing up in 
mount
ouput.
Copy code
type nfs4 (rw,nosuid,noatime,nodiratime,vers=4.2,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp6,timeo=600,retrans=3,sec=sys,clientaddr=fc15::60,local_lock=none,addr=fc15::11)
Compared to what is mouned in kubernetes.
Copy code
type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp6,timeo=600,retrans=2,sec=sys,mountaddr=fc15::11,mountvers=3,mountport=893,mountproto=udp6,local_lock=none,addr=fc15::11)
Many of the options don't appear to be respected at all.
c
if you’re still using the bundled binaries its possible they don’t understand some of those options
but I don’t think it’s related to the kubelet or k3s
q
I dropped that argument from the binary now.
With that argument in place, I was getting "connection refused" which was wrong, and likely just a miinterpretatin of the "exit 255" from the mount command, which was related to some lock module.
c
if you add 
--v=4
to the k3s args kubernetes will dump out a ton of logs, including the mount options
👀 1
q
Like a billion of these.
Copy code
[31122.091771] svc: failed to register lockdv1 RPC service (errno 111).
Nice.
I wasn't running lockd on the server since nfsv4 doesn't require it and the 
nolock
option was specified.
c
nfs v4 requires locking, are you perhaps not running the right services to support that and it is falling back to v3?
q
Maybe, but the manual mount command above works.
c
nfs v4 doesn’t support nolock. locking is mandatory in v4
q
Ah right, more what I mean is that 
rpcinfo -p
doesn't require that rpc.lockd is in place, or whatever it was.
True of statd too iirc.
And maybe one other.
All that to say, I have the ports and services necessary exposed and configured correctly as long as the client side options are respected.
My issue with the 
prefer-bundled-bin
was slightly different that what is reported in https://github.com/k3s-io/k3s/issues/10206
In their closing remark on that issue, they say that nfsvers=4 works, but I wonder if they checked the resutls.
I see that nfs_test.go in kubernetes doesn't mention mount options.
Maybe they come in from outside. Hmm. 👀
I filed https://github.com/kubernetes/kubernetes/issues/131449 for this. Thanks for the pointers.
2 Views
Open in Slack
PreviousNext
Powered by