https://rancher.com/ logo
Title
s

shy-tent-66642

01/12/2023, 6:12 AM
Steps I followed to increase pod limit To update your existing installation with an increased max-pods, add a kubelet config file into a k3s associated location such as `/etc/rancher/k3s/kubelet.config`:
apiVersion: <http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>
kind: KubeletConfiguration
maxPods: 250
edit
/etc/systemd/system/k3s.service
to change the k3s server args:
ExecStart=/usr/local/bin/k3s \
    server \
        '--disable' \
        'servicelb' \
        '--disable' \
        'traefik' \
        '--kubelet-arg=config=/etc/rancher/k3s/kubelet.config'
reload systemctl to pick up the service change:
sudo systemctl daemon-reload
restart k3s:
sudo systemctl restart k3s
c

creamy-pencil-82913

01/12/2023, 6:23 AM
Don't do that. If you use a config file it will ignore ALL the other cli args
Just start the nodes with --kubelet-arg=max-pods=150 or whatever
You will also need to change the node cidr mask, but that needs to be done before starting the cluster the first time. If you don't do that you will run out of IP address for pods before you hit the increased max pods.
s

shy-tent-66642

01/12/2023, 6:44 AM
Thanks for reply. I have not used k3s before. I have initiated the master with this argument. So I believe /16 is enough IPs.
INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16 server --disable traefik,servicelb"
Coming to the point of --kubelet-arg=max-pods how do I do that exactly on already running nodes? What if I want to change pod limit to another value few times? We are just experimenting with pod we can run. These are AMD EPYC metal nodes.
c

creamy-pencil-82913

01/12/2023, 7:20 AM
no, that’s not what you need to change. you need to change the node cidr mask. By default each node only gets a /24
--kube-controller-manager-arg=node-cidr-mask-size=23
youll need to rebuild the cluster
you can change the args in the k3s systemd unit, or by re-running the installer
s

shy-tent-66642

01/12/2023, 8:46 AM
Thanks. Let me check your new suggestions. Is there any need to reset iptables after I uninstall k3s (agent or server) using uninstall script? I used calico before and after I run uninstallation script, I could see lods of entries in Iptables.
c

creamy-pencil-82913

01/12/2023, 8:53 AM
if you’re using Calico you might need to change things a bit, it has its own IPAM if I remember correctly so the node cidr mask is not used.
its helpful to mention anything you’ve customized on the cluster up front, the recommendations on next steps may be different if you’re not using the default k3s components
if you’re using flannel no, just run the uninstall script and you’ll be fine
s

shy-tent-66642

01/12/2023, 9:37 AM
Noted. This is working for me
On master 
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16 server --disable traefik,servicelb --kubelet-arg=max-pods=350" sh - 
  
on workers  
curl -sfL <https://get.k3s.io> | K3S_URL=<https://MASTER-IP:6443> K3S_TOKEN=<token> INSTALL_K3S_EXEC="--kubelet-arg=max-pods=350" sh -
BTW, if I have to re run installed w/o the script....how can I do that? Some thing like below?
k3s server/agent INSTALL_K3S_EXEC="--flannel-backend=none --disable-network-policy --cluster-cidr=192.168.0.0/16 server --disable traefik,servicelb --kubelet-arg=max-pods=350"
ok...so I tried running
k3s agent --kubelet-arg=max-pods=300 --token=<RKN> --server=<https://IP:6443>
c

creamy-pencil-82913

01/12/2023, 10:28 AM
You still need to fix the cidr mask or you will get errors once you get to around 200 pods and the nodes run out of IPs.
And it looks like you're trying to run another copy of the agent while the service is still running?
Thread was not updating for me :/
s

shy-tent-66642

01/12/2023, 10:43 AM
Well.....Looks like If I run
k3s agent K3S_URL=<https://MASTER-IP:6443> K3S_TOKEN=<token> INSTALL_K3S_EXEC="--kubelet-arg=max-pods=350"
multiple times (with different value of kubelet-arg) on agent node, the k3s agent services just hangs in second attempt and nodes goes into unready state. I could only got script installation to rerun
c

creamy-pencil-82913

01/12/2023, 5:12 PM
I think you’re mixing up running the k3s command to run k3s, and running the installer command to install it
INSTALL_K3S_EXEC is something you would set for the installer, not for
k3s agent
itself
s

shy-tent-66642

01/13/2023, 6:55 AM
Plz ignore command posted in my last message (which I can not edit now). I used
k3s agent --kubelet-arg=max-pods=300 --token=<RKN> --server=<https://IP:6443>
with max pod values multiple times and that kind of hanged the k3s agent on node which caused node to become notready state