https://rancher.com/ logo
Title
b

billowy-smartphone-2833

05/18/2023, 7:56 AM
https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik#comment-32065 I am folowing this article to test the pod networking of my cluster. It creates a simple hello world application using deployment , service and ingress and it used traefik for ingress . Somehow I am not able to do curl localhost:80 ( ingress is configured to work on :80 ) can anyone please suggest me what has gone wrong here ( service , pods , endpoints all are responding on port 80 )
b

bland-account-99790

05/18/2023, 1:42 PM
can you show
kubectl get service -A -o wide
?
b

billowy-smartphone-2833

05/18/2023, 5:33 PM
NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE   SELECTOR
default       hello-world          ClusterIP      10.43.186.203   <none>           80/TCP                       12h   app=hello-world
default       kubernetes           ClusterIP      10.43.0.1       <none>           443/TCP                      31h   <none>
kube-system   kube-dns             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       31h   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      10.43.178.48    <none>           443/TCP                      31h   k8s-app=metrics-server
kube-system   traefik              LoadBalancer   10.43.90.30     10.192.177.101   80:32097/TCP,443:31916/TCP   31h   app=traefik,release=traefik
kube-system   traefik-prometheus   ClusterIP      10.43.95.70     <none>           9100/TCP                     31h   app=traefik,release=traefik
thank you @bland-account-99790
b

bland-account-99790

05/19/2023, 5:45 AM
Does it work when you curl: • 10.43.186.203:80? • 10.43.90.30:80? • 10.192.177.101:80?
b

billowy-smartphone-2833

05/19/2023, 9:28 AM
This works curl 10.43.186.203:80 <html> <head> <title>Hello World!</title> </head> <body>Hello World!</body> </html>
AA-ANSIBLE001@9289946cvk3s01:~$ curl 10.43.90.30:80 404 page not found AA-ANSIBLE001@9289946cvk3s01:~$ curl 10.192.177.101:80 404 page not found
b

bland-account-99790

05/19/2023, 10:59 AM
so the problem is in ingress
probably in the config
can you show me
kubectl get ingress -A -o yaml
And
kubectl get endpoints -A
And the logs of traefik pod ==>
kubectl logs traefik.... -n kube-system
b

billowy-smartphone-2833

05/19/2023, 12:02 PM
AA-ANSIBLE001@9289946cvk3s01:~$ kubectl get endpoints -A NAMESPACE NAME ENDPOINTS AGE default hello-world 10.42.0.219:80,10.42.0.221:80,10.42.0.222:80 31h default kubernetes 10.192.177.101:6443,10.192.177.102:6443,10.192.177.103:6443 2d2h kube-system kube-dns 2d2h kube-system metrics-server 10.42.0.225:443 2d2h kube-system rancher.io-local-path <none> 2d2h kube-system traefik 10.42.0.223:443,10.42.0.223:80 2d2h kube-system traefik-prometheus 10.42.0.223:9100 2d2h
kubectl get ingress -A -o yaml
Hello @bland-account-99790 I continously get these statements E0519 12:01:23.503152 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:24.503507 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:24.504410 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:25.504745 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:25.505626 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:26.505978 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:26.506784 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:27.507214 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:27.508033 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:28.508430 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:28.509213 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:29.509614 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:29.510589 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:30.510823 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:30.511773 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:31.512021 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:31.512915 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:32.513271 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:32.514169 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:33.514531 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:33.515341 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:34.515905 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:34.516794 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:35.517152 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:35.517990 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:36.518384 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:36.519223 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:01:37.519589 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:01:37.520507 E0519 12:02:23.576915 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:02:24.577273 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:02:24.578098 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:02:25.578495 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:02:25.579321 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 12:02:26.579729 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 12:02:26.580562 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized
b

bland-account-99790

05/19/2023, 12:26 PM
how did you deploy k3s?
Why your ingress definition includes
ingressClassName: traefik
, that is not part of https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik#comment-32065? Are you adding extra stuff yourself?
b

billowy-smartphone-2833

05/19/2023, 1:02 PM
yes that is the extra stuff
I am deploying etcd and then k3s using ansible
k3s version 1.20.4
b

bland-account-99790

05/19/2023, 1:03 PM
that is very old
can you show me
kubectl get ClusterRole traefik-kube-system  -o yaml
?
I think what is happening is that the ClusterRole assigned to the serviceAccount of traeffik is wrong. That role should provide traeffik access to things like Endpoints or Services
Therefore, you should not see the errors you are seeing in the logs
But that should come with k3s correctly. How are you installing k3s in ansible?
b

billowy-smartphone-2833

05/19/2023, 1:54 PM
I am using get.k3s.io.sh script to install k3s . I mean I execute this script using ansible. My environment is Air-gapped , hence I download all the images , k3s binary on the VM , Push the images to local registry , and then create k3s service
clusterroles_and_bindinds.txt
b

bland-account-99790

05/19/2023, 1:57 PM
what about
kubectl get ClusterRole traefik-kube-system -o yaml
?
b

billowy-smartphone-2833

05/19/2023, 1:58 PM
it does not exists in the cluster kubectl get ClusterRole traefik-kube-system -o yaml Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "traefik-kube-system" not found
right now this setup is done on 275 edge environments , hence upgrading k3s and etcd is difficult since the servies are already running on few environments . Hence we are looking for a fixes in existing environment which we can keep running until all the workloads are live and later aftr 2 months we are planning to upgrade
b

bland-account-99790

05/19/2023, 2:01 PM
ok, I think how traefik was deployed changed a bit in the latest versions, that's probably why
ClusterRole traefik-kube-system
does not exist
what about
kubectl get ClusterRole traefik -o yaml
?
b

billowy-smartphone-2833

05/19/2023, 2:02 PM
AA-ANSIBLE001@k3s01:~$ kubectl get ClusterRole traefik -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: meta.helm.sh/release-name: traefik meta.helm.sh/release-namespace: kube-system creationTimestamp: "2023-05-17T09:41:58Z" labels: app.kubernetes.io/managed-by: Helm managedFields: - apiVersion: rbac.authorization.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: .: {} f:meta.helm.sh/release-name: {} f:meta.helm.sh/release-namespace: {} f:labels: .: {} f:app.kubernetes.io/managed-by: {} f:rules: {} manager: Go-http-client operation: Update time: "2023-05-17T09:41:58Z" name: traefik resourceVersion: "1850" uid: 13ad53cb-ecf5-43be-8373-7fbeee754d0a rules: - apiGroups: - "" resources: - pods - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update
b

bland-account-99790

05/19/2023, 2:08 PM
That looks correct, can you check the serviceAccount you get when executing:
kubectl get deployment/traefik -n kube-system -o yaml
? And then verify there is a
clusterRoleBinding
mapping that serviceAccount with the ClusterRole you just showed me?
As you can observe, your problem is that traefik ingress is unable to work properly because it is unable to access kube-api and fetch Service and Endpoints as it is unauthorized
E0519 12:01:23.503152       1 reflector.go:205] <http://github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86|github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86>: Failed to list *v1.Service: Unauthorized
E0519 12:01:24.503507       1 reflector.go:205] <http://github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86|github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86>: Failed to list *v1.Endpoints: Unauthorized
To get authorization, traefik serviceAccount should have the proper permissions
b

billowy-smartphone-2833

05/19/2023, 2:12 PM
just fyi.. traefik version is image: rancher/library-traefik:1.7.19
serviceAccount: traefik serviceAccountName: traefik
traefik.yaml
b

bland-account-99790

05/19/2023, 2:17 PM
there you go:
serviceAccount: traefik
      serviceAccountName: traefik
b

billowy-smartphone-2833

05/19/2023, 2:17 PM
yes.. it looks correct
my hello-world application works fine till service
but at ingress it doesnt
my deployment is on Ubuntu 20.04 , any system level changes / configurations i am missing
b

bland-account-99790

05/19/2023, 2:20 PM
ok, I'm currently busy with a customer issue but I could try and reproduce it using the versions you gave me next week
in any case, there must be something in the roles. It does not look obvious though but it shouldn't be hard to find out 😛
b

billowy-smartphone-2833

05/19/2023, 2:24 PM
./k3s --version k3s version v1.20.4+k3s1 (838a906a) go version go1.15.8 cat k3s-images.txt rancher/coredns-coredns:1.8.0 rancher/klipper-helm:v0.4.3 rancher/klipper-lb:v0.1.2 rancher/library-busybox:1.32.1 rancher/library-traefik:1.7.19 rancher/local-path-provisioner:v0.0.19 rancher/metrics-server:v0.3.6 rancher/pause:3.1
many thanks @bland-account-99790 , let me know if you can do it over a google/zoom call , as per your convenience
b

bland-account-99790

05/19/2023, 2:25 PM
I tried with k3s 1.26 and it worked well though
🙌 1