:wave: I'm using k3d for local development/e2e tes...
# k3d
h
๐Ÿ‘‹ I'm using k3d for local development/e2e testing. I startup a local cluster with 10+ agent nodes. Everything is working great on my mac using docker for mac. However I can't bring clusters up with more than 4-5 nodes on my linux workstation which has WAY more memory and CPU that my macbook. It just hangs during the starting agent node phase. I've tried slow adding nodes one at a time. It still hangs after adding a couple of nodes. I checked my ulimits etc, everything is fine as far as I can tell. Not sure how to proceed... Any ideas?
Ok, I'm getting some where, I see this inside the agent logs:
Copy code
E1002 23:51:28.878328      87 dynamic_cafile_content.go:169] "Failed to watch CA file, will retry later" err="error creating fsnotify watcher: too many open files"
But this baffling because the max is very high inside the container
Copy code
~ # cat /proc/sys/fs/inotify/max_user_watches
1048576
Yet the number watches seems to be 7:
Copy code
~ # find /proc/*/fd -lname 'anon_inode:inotify' -print 2>/dev/null | wc -l
7
I figured it out. Turns out it was
/proc/sys/fs/inotify/max_user_instances
... this was set to 128 which is too low apaprently, I set it to 8192 and now it works. ๐Ÿ‘‹ ๐Ÿ™‚
๐Ÿ‘ 2