I want to publicly thank
@wide-garage-9465 for all the info he provided regarding the k3s/k3d with GPU support!
It wasn't easy since it's never easy to find the right NVIDIA driver version & etc but I've managed to create a working k3s container which runs good in a k3d cluster so I'm sharing it in case someone needs it too:
https://gitlab.com/vainkop1/k3s-gpu
Bonus: how to Gitlab Runner with Docker TLS on K3d