melodic-market-4209206/24/2022, 10:00 AM
to expose a few things from my k3d cluster. The first three of those work, but the last one just.. doesn't. I'm struggling to figure out why. If I do a port-forward to the service, then things work as expected. The ingress and service for the fourth entry there (8087:2746) looks exactly similar to the ingress'es for the other three 🤔 The application behind 2746 is https://argoproj.github.io/argo-workflows/argo-server/. Any hints anybody could give on this? Hopefully someone has seen this before.
k3d cluster create $CLUSTER_NAME \ -p "8081:3000@loadbalancer" \ -p "8082:8000@loadbalancer" \ -p "8086:8086@loadbalancer" \ -p "8087:2746@loadbalancer"
apiVersion: v1 kind: Service metadata: name: argo-server spec: ports: - name: web port: 2746 targetPort: 2746 selector: app: argo-server type: LoadBalancer
apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1> kind: Ingress metadata: name: argo-server-ingress spec: rules: - http: paths: - backend: service: name: argo-server port: number: 2746 path: / pathType: Prefix
wide-garage-946506/24/2022, 10:05 AM
and see if there are any svclb pods pending?
kubectl get pods -A
melodic-market-4209206/24/2022, 10:34 AM
╰─ k -n myns get po NAME READY STATUS RESTARTS AGE svclb-argo-server-pnf69 1/1 Running 0 38m
So which do you want to use? The LoadBalancer type service or the ingress?I get confused. I want to expose these things externally in my live clusters (k3s), and I wish to expose them when using k3d locally. And preferably I wish the configuration for the live clusters and the localhost cluster to look as similar as possible. In my live clusters, I think that I do need a service of type loadbalancer OR type nodeport to be able to get the ingress working. Answers like this one (https://stackoverflow.com/a/60076900) tells me that.
“map portthat I'm misunderstandingfrom the host to port
8081on the container which matches the nodefilter `loadbalancer`“
will map port 8081 on my system to port 3000 in the loadbalancer. I think that the load balancer is the second one here:
I also noticed this:
╰─ k3d node list NAME ROLE CLUSTER STATUS k3d-localcluster-server-0 server localcluster running k3d-localcluster-serverlb loadbalancer localcluster running
Where I see those port mappings in the docker container. What I don't understand yet is why localhost:8081 in my browser now doesn't work. What I've changed now is to make all of services be of type clusterip. I still have the ingress'es.
╰─ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 66828b350766 <http://ghcr.io/k3d-io/k3d-proxy:5.4.1|ghcr.io/k3d-io/k3d-proxy:5.4.1> "/bin/sh -c nginx-pr…" 8 minutes ago Up 8 minutes 80/tcp, 0.0.0.0:8086->8086/tcp, :::8086->8086/tcp, 0.0.0.0:8087->2746/tcp, :::8087->2746/tcp, 0.0.0.0:8081->3000/tcp, :::8081->3000/tcp, 0.0.0.0:37503->6443/tcp, 0.0.0.0:8082->8000/tcp, :::8082->8000/tcp k3d-localcluster-serverlb 0ef97c8d1ace rancher/k3s:v1.22.7-k3s1 "/bin/k3d-entrypoint…" 8 minutes ago Up 8 minutes k3d-localcluster-server-0
╰─ docker logs k3d-localcluster-serverlb -f 2022/06/24 12:49:28 [error] 58#58: *187 connect() failed (111: Connection refused) while connecting to upstream, client: 172.29.0.1, server: 0.0.0.0:3000, upstream: "172.29.0.2:3000", bytes from/to client:0/0, bytes from/to upstream:0/0
Since you're forwarding to ports that are not 80 or 443, I guess you're not using Ingress at all.Can I not use ingress to expose something on ports different than those? https://stackoverflow.com/a/56243253 tells me no 🤔
wide-garage-946506/24/2022, 3:40 PM
, which (in K3s) means, that some
pods will be spawned in the nodes (k3d containers) that forward traffic from the nodes' (k3d containers) ports 80 and 443 to the Ingress Controller (simplified). • Now as per the k3d documentation, you map some port of your host, say
(which is the one container ending in
always spawned by k3d as you noticed already) -> this makes the serverlb forward all coming in on port 80 to port 80 on all server nodes in your k3d cluster, where it will then be routed to the Ingress controller • The Ingress controller configures the cluster (simplified) in a way, that traffic is then routed to the Kubernetes service that you defined in the Ingress object -> Here, it doesn't matter, if it's of type
, as the traffic is already inside the cluster through the Ingress • You use Ingress to access your services by domain name, e.g. myservice.mydomain.com
What I don't understand yet is why localhost:8081 in my browser now doesn't work. What I've changed now is to make all of services be of type clusterip. I still have the ingress'es.Port 8081 from your machine to 3000 on the loadbalancer, which forwards that to port 3000 on the server0 container. But since you don't have anything exposed on that port on the server0 container (
makes the service only available inside the cluster), there is nothing it can talk to, hence the error also in the proxy logs. You can use many ways to expose your service externally, e.g. `type: NodePort`: opens a port on every node in your cluster and forwards traffic from there to your service;
talks to the API to get some loadbalancer with an external IP provisioned (in cloud providers that's a VM with a public IP, in K3s that's the IP of one of your nodes, since it cannot provision something);
in your PodSpec maps directly to a port on the node the pod is on, without using a service at all.
melodic-market-4209206/27/2022, 7:46 AM
To check my understanding: When I go to localhost:8081 after this, the request goes through (at least) these steps: • (outside the cluster) To the k3d docker container: k3d-localcluster-serverlb. This is the k3d load balancer. • (inside the cluster) To/through the traefik (loadbalancer) service • (inside the cluster) To/through the svclb-traefik-xxxxx pod. This pod should've picked up on any
k3d cluster create -p "8081:80@loadbalancer"
that's defined. • (inside the cluster) To/through the nginx clusterip service • (inside the cluster) To/through the nginx pod Does that look correct?
wide-garage-946506/27/2022, 8:38 AM
is an instance of Rancher's
(https://github.com/k3s-io/klipper-lb), which checks for
type services and binds to the specified service ports on one of the nodes to "imitate" a real
. On k3d level:
In K3s (simplified flow):
melodic-market-4209206/27/2022, 8:43 AM
) when developing locally. Then they'll get in each other's way
) to the traefik ingress controller. But I can have additional to services of type loadbalancer
wide-garage-946506/27/2022, 9:06 AM
to resolve all
addresses to 127.0.0.1, so you won't have to touch your hosts file.
melodic-market-4209206/27/2022, 9:07 AM