This message was deleted.

What OS are you running, and where did you increase the nofile limits?

Nix -- I set it in both login and in the docker daemon settings. When I spin up generic bash container in docker it works correctly, but running:

kubectl run busybox -i --tty --image=busybox --restart=Never --rm -- sh -c 'ulimit -a'

I still get 1024

Not to familiar with nix myself, but there seem to be some issues with systemd & nix combination. Give this a read: https://github.com/NixOS/nixpkgs/issues/159964

I've been able to narrow this down to k3d because when running k3s locally it works; my system ulimit -a returns correct value as well

When we run K3D with GPU support we have to rebuild the K3S container and point K3D at it, have you tried tweaking the K3S container?

how are you rebuilding the container? ive tried to prune docker before launching to force a rebuild

You'd need to do an explicit docker build with a new Dockerfile. For us we use a completely different base container and copy the K3S binaries into it since GPU support is a bit more involved. For you I'd think you'd just use the K3S container as a base container and toss in a few commands and redo the entrypoint.

My thought is that it should just assume the limits from the host like any other container, so something within k3d is explicitly setting/resetting the limits

Then use

-i $REBUILD_K3S_CONTAINER_NAME

when creating your cluster.

yes, I follow; but when checking the k3d container itself I see the lower limit. so it makes sense to me that the pods inherit from it docker ps

58705cfb4038   rancher/k3s:v1.29.0-k3s1         "/bin/k3d-entrypoint…"   20 minutes ago   Up 20 minutes                                                                       k3d-local-server-0

❯

docker exec -it 58705cfb4038 sh

/ # ulimit -n 1024

Yeah, so what I'm saying is how to change the container that's running K3S. You already said you tried some other containers and it was using from the host. If this one isn't, then you need to change the container.

I guess I don't understand what needs changed in the container, if I inspect it I can see that docker is setting the right limits, so it should be inherited. I can change the entrypoint to include a call to ulimit but that shouldn't be necessary unless its being changed before that somehow

@adorable-orange-51420 do you just get

too many open files

in pods running in your k3d cluster?

yes

https://stackoverflow.com/questions/34588/how-do-i-change-the-number-of-open-files-limit-in-linux ? Containers have file systems, did you check if the K3S container has /etc/security/limits.conf setting it to 1024? Or maybe a different place (since Linux can be irritating with different distros putting configs in different places)?

Some hints: 1. K3s containers created by k3d run in

privileged

mode, so they're not constrained in any way to modify their system settings 2. k3d's

--runtime-ulimit

flag is basically the equivalent to Docker's

--ulimit

flag (see blog) 3. The

too many open files

issue is often solved on the host system by increasing the limits with

sysctl

(see some other issues)

As a side note, if you want to trace and verify where it changes, since containers run on the host kernel, you can see your container and the processes in it from the host and you can check parent PIDs to trace upwards. You can use

cat /proc/${PID}/limits

to check the limits of the different processes and that'd show you where it changes if you're trying to narrow down where. Should be that process' parent or maybe the process setting it for itself.