<https www jeffgeerling com blog 2022 quick hello world http rancher-users #k3s

Title

billowy-smartphone-2833

05/18/2023, 7:56 AM

https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik#comment-32065 I am folowing this article to test the pod networking of my cluster. It creates a simple hello world application using deployment , service and ingress and it used traefik for ingress . Somehow I am not able to do curl localhost:80 ( ingress is configured to work on :80 ) can anyone please suggest me what has gone wrong here ( service , pods , endpoints all are responding on port 80 )

bland-account-99790

05/18/2023, 1:42 PM

can you show

kubectl get service -A -o wide

billowy-smartphone-2833

05/18/2023, 5:33 PM

NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE   SELECTOR
default       hello-world          ClusterIP      10.43.186.203   <none>           80/TCP                       12h   app=hello-world
default       kubernetes           ClusterIP      10.43.0.1       <none>           443/TCP                      31h   <none>
kube-system   kube-dns             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       31h   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      10.43.178.48    <none>           443/TCP                      31h   k8s-app=metrics-server
kube-system   traefik              LoadBalancer   10.43.90.30     10.192.177.101   80:32097/TCP,443:31916/TCP   31h   app=traefik,release=traefik
kube-system   traefik-prometheus   ClusterIP      10.43.95.70     <none>           9100/TCP                     31h   app=traefik,release=traefik

thank you @bland-account-99790

bland-account-99790

05/19/2023, 5:45 AM

Does it work when you curl: • 10.43.186.203:80? • 10.43.90.30:80? • 10.192.177.101:80?

billowy-smartphone-2833

05/19/2023, 9:28 AM

This works curl 10.43.186.203:80 <html> <head> <title>Hello World!</title> </head> <body>Hello World!</body> </html>

AA-ANSIBLE001@9289946cvk3s01:~$ curl 10.43.90.30:80 404 page not found AA-ANSIBLE001@9289946cvk3s01:~$ curl 10.192.177.101:80 404 page not found

bland-account-99790

05/19/2023, 10:59 AM

so the problem is in ingress

probably in the config

can you show me

kubectl get ingress -A -o yaml

And

kubectl get endpoints -A

And the logs of traefik pod ==>

kubectl logs traefik.... -n kube-system

billowy-smartphone-2833

05/19/2023, 12:02 PM

AA-ANSIBLE001@9289946cvk3s01:~$ kubectl get endpoints -A NAMESPACE NAME ENDPOINTS AGE default hello-world 10.42.0.219:80,10.42.0.221:80,10.42.0.222:80 31h default kubernetes 10.192.177.101:6443,10.192.177.102:6443,10.192.177.103:6443 2d2h kube-system kube-dns 2d2h kube-system metrics-server 10.42.0.225:443 2d2h kube-system rancher.io-local-path <none> 2d2h kube-system traefik 10.42.0.223:443,10.42.0.223:80 2d2h kube-system traefik-prometheus 10.42.0.223:9100 2d2h

kubectl get ingress -A -o yaml

ingress.txt

Hello @bland-account-99790 I continously get these statements E0519 12:01:23.503152 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:24.503507 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:24.504410 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:25.504745 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:25.505626 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:26.505978 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:26.506784 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:27.507214 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:27.508033 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:28.508430 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:28.509213 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:29.509614 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:29.510589 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:30.510823 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:30.511773 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:31.512021 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:31.512915 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:32.513271 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:32.514169 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:33.514531 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:33.515341 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:34.515905 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:34.516794 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:35.517152 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:35.517990 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:36.518384 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:36.519223 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:37.519589 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:37.520507 E0519 12:02:23.576915 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:02:24.577273 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:02:24.578098 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:02:25.578495 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:02:25.579321 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:02:26.579729 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:02:26.580562 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized

bland-account-99790

05/19/2023, 12:26 PM

how did you deploy k3s?

Why your ingress definition includes

ingressClassName: traefik

, that is not part of https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik#comment-32065? Are you adding extra stuff yourself?

billowy-smartphone-2833

05/19/2023, 1:02 PM

yes that is the extra stuff

I am deploying etcd and then k3s using ansible

k3s version 1.20.4

bland-account-99790

05/19/2023, 1:03 PM

that is very old

can you show me

kubectl get ClusterRole traefik-kube-system  -o yaml

I think what is happening is that the ClusterRole assigned to the serviceAccount of traeffik is wrong. That role should provide traeffik access to things like Endpoints or Services

Therefore, you should not see the errors you are seeing in the logs

But that should come with k3s correctly. How are you installing k3s in ansible?

billowy-smartphone-2833

05/19/2023, 1:54 PM

I am using get.k3s.io.sh script to install k3s . I mean I execute this script using ansible. My environment is Air-gapped , hence I download all the images , k3s binary on the VM , Push the images to local registry , and then create k3s service

clusterroles_and_bindinds.txt

bland-account-99790

05/19/2023, 1:57 PM

what about

kubectl get ClusterRole traefik-kube-system -o yaml

billowy-smartphone-2833

05/19/2023, 1:58 PM

it does not exists in the cluster kubectl get ClusterRole traefik-kube-system -o yaml Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "traefik-kube-system" not found

right now this setup is done on 275 edge environments , hence upgrading k3s and etcd is difficult since the servies are already running on few environments . Hence we are looking for a fixes in existing environment which we can keep running until all the workloads are live and later aftr 2 months we are planning to upgrade

bland-account-99790

05/19/2023, 2:01 PM

ok, I think how traefik was deployed changed a bit in the latest versions, that's probably why

ClusterRole traefik-kube-system

does not exist

what about

kubectl get ClusterRole traefik -o yaml

billowy-smartphone-2833

05/19/2023, 2:02 PM

AA-ANSIBLE001@k3s01:~$ kubectl get ClusterRole traefik -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: meta.helm.sh/release-name: traefik meta.helm.sh/release-namespace: kube-system creationTimestamp: "2023-05-17T09:41:58Z" labels: app.kubernetes.io/managed-by: Helm managedFields: - apiVersion: rbac.authorization.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:meta.helm.sh/release-name: {} f:meta.helm.sh/release-namespace: {} f:labels: .: {} f:app.kubernetes.io/managed-by: {} f:rules: {} manager: Go-http-client operation: Update time: "2023-05-17T09:41:58Z" name: traefik resourceVersion: "1850" uid: 13ad53cb-ecf5-43be-8373-7fbeee754d0a rules: - apiGroups: - "" resources: - pods - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update

bland-account-99790

05/19/2023, 2:08 PM

That looks correct, can you check the serviceAccount you get when executing:

kubectl get deployment/traefik -n kube-system -o yaml

? And then verify there is a

clusterRoleBinding

mapping that serviceAccount with the ClusterRole you just showed me?

As you can observe, your problem is that traefik ingress is unable to work properly because it is unable to access kube-api and fetch Service and Endpoints as it is unauthorized

E0519 12:01:23.503152       1 reflector.go:205] <http://github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86|github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86>: Failed to list *v1.Service: Unauthorized
E0519 12:01:24.503507       1 reflector.go:205] <http://github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86|github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86>: Failed to list *v1.Endpoints: Unauthorized

To get authorization, traefik serviceAccount should have the proper permissions

billowy-smartphone-2833

05/19/2023, 2:12 PM

just fyi.. traefik version is image: rancher/library-traefik:1.7.19

serviceAccount: traefik serviceAccountName: traefik

traefik.yaml

bland-account-99790

05/19/2023, 2:17 PM

there you go:

serviceAccount: traefik
      serviceAccountName: traefik

billowy-smartphone-2833

05/19/2023, 2:17 PM

yes.. it looks correct

my hello-world application works fine till service

but at ingress it doesnt

my deployment is on Ubuntu 20.04 , any system level changes / configurations i am missing

bland-account-99790

05/19/2023, 2:20 PM

ok, I'm currently busy with a customer issue but I could try and reproduce it using the versions you gave me next week

in any case, there must be something in the roles. It does not look obvious though but it shouldn't be hard to find out 😛

billowy-smartphone-2833

05/19/2023, 2:24 PM

./k3s --version k3s version v1.20.4+k3s1 (838a906a) go version go1.15.8 cat k3s-images.txt rancher/coredns-coredns:1.8.0 rancher/klipper-helm:v0.4.3 rancher/klipper-lb:v0.1.2 rancher/library-busybox:1.32.1 rancher/library-traefik:1.7.19 rancher/local-path-provisioner:v0.0.19 rancher/metrics-server:v0.3.6 rancher/pause:3.1

many thanks @bland-account-99790 , let me know if you can do it over a google/zoom call , as per your convenience

bland-account-99790

05/19/2023, 2:25 PM

I tried with k3s 1.26 and it worked well though

🙌 1

2 Views

#k3s Join Slack