https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
b
can you show 
kubectl get service -A -o wide
?
b
Copy code
NAMESPACE     NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE   SELECTOR
default       hello-world          ClusterIP      10.43.186.203   <none>           80/TCP                       12h   app=hello-world
default       kubernetes           ClusterIP      10.43.0.1       <none>           443/TCP                      31h   <none>
kube-system   kube-dns             ClusterIP      10.43.0.10      <none>           53/UDP,53/TCP,9153/TCP       31h   k8s-app=kube-dns
kube-system   metrics-server       ClusterIP      10.43.178.48    <none>           443/TCP                      31h   k8s-app=metrics-server
kube-system   traefik              LoadBalancer   10.43.90.30     10.192.177.101   80:32097/TCP,443:31916/TCP   31h   app=traefik,release=traefik
kube-system   traefik-prometheus   ClusterIP      10.43.95.70     <none>           9100/TCP                     31h   app=traefik,release=traefik
thank you @bland-account-99790
b
Does it work when you curl: • 10.43.186.203:80? • 10.43.90.30:80? • 10.192.177.101:80?
b
This works curl 10.43.186.203:80 <html> <head> <title>Hello World!</title> </head> <body>Hello World!</body> </html>
AA-ANSIBLE001@9289946cvk3s01:~$ curl 10.43.90.30:80 404 page not found AA-ANSIBLE001@9289946cvk3s01:~$ curl 10.192.177.101:80 404 page not found
b
so the problem is in ingress
probably in the config
can you show me 
kubectl get ingress -A -o yaml
And 
kubectl get endpoints -A
And the logs of traefik pod ==> 
kubectl logs traefik.... -n kube-system
b
AA-ANSIBLE001@9289946cvk3s01:~$ kubectl get endpoints -A NAMESPACE NAME ENDPOINTS AGE default hello-world 10.42.0.21980,10.42.0.22180,10.42.0.222:80 31h default kubernetes 10.192.177.1016443,10.192.177.1026443,10.192.177.103:6443 2d2h kube-system kube-dns 2d2h kube-system metrics-server 10.42.0.225:443 2d2h kube-system rancher.io-local-path <none> 2d2h kube-system traefik 10.42.0.223443,10.42.0.22380 2d2h kube-system traefik-prometheus 10.42.0.223:9100 2d2h
kubectl get ingress -A -o yaml
ingress.txt
Hello @bland-account-99790 I continously get these statements E0519 120123.503152 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120124.503507 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120124.504410 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120125.504745 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120125.505626 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120126.505978 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120126.506784 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120127.507214 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120127.508033 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120128.508430 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120128.509213 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120129.509614 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120129.510589 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120130.510823 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120130.511773 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120131.512021 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120131.512915 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120132.513271 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120132.514169 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120133.514531 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120133.515341 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120134.515905 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120134.516794 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120135.517152 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120135.517990 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120136.518384 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120136.519223 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120137.519589 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120137.520507 E0519 120223.576915 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120224.577273 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120224.578098 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120225.578495 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120225.579321 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized E0519 120226.579729 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Endpoints: Unauthorized E0519 120226.580562 1 reflector.go:205] github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86: Failed to list *v1.Service: Unauthorized
b
how did you deploy k3s?
Why your ingress definition includes 
ingressClassName: traefik
, that is not part of https://www.jeffgeerling.com/blog/2022/quick-hello-world-http-deployment-testing-k3s-and-traefik#comment-32065? Are you adding extra stuff yourself?
b
yes that is the extra stuff
I am deploying etcd and then k3s using ansible
k3s version 1.20.4
b
that is very old
can you show me 
kubectl get ClusterRole traefik-kube-system  -o yaml
?
I think what is happening is that the ClusterRole assigned to the serviceAccount of traeffik is wrong. That role should provide traeffik access to things like Endpoints or Services
Therefore, you should not see the errors you are seeing in the logs
But that should come with k3s correctly. How are you installing k3s in ansible?
b
I am using get.k3s.io.sh script to install k3s . I mean I execute this script using ansible. My environment is Air-gapped , hence I download all the images , k3s binary on the VM , Push the images to local registry , and then create k3s service
b
what about 
kubectl get ClusterRole traefik-kube-system -o yaml
?
b
it does not exists in the cluster kubectl get ClusterRole traefik-kube-system -o yaml Error from server (NotFound): clusterroles.rbac.authorization.k8s.io "traefik-kube-system" not found
right now this setup is done on 275 edge environments , hence upgrading k3s and etcd is difficult since the servies are already running on few environments . Hence we are looking for a fixes in existing environment which we can keep running until all the workloads are live and later aftr 2 months we are planning to upgrade
b
ok, I think how traefik was deployed changed a bit in the latest versions, that's probably why 
ClusterRole traefik-kube-system
does not exist
what about 
kubectl get ClusterRole traefik -o yaml
?
b
AA-ANSIBLE001@k3s01:~$ kubectl get ClusterRole traefik -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: annotations: meta.helm.sh/release-name: traefik meta.helm.sh/release-namespace: kube-system creationTimestamp: "2023-05-17T094158Z" labels: app.kubernetes.io/managed-by: Helm managedFields: - apiVersion: rbac.authorization.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: fmetadata fannotations .: {} f:meta.helm.sh/release-name: {} f:meta.helm.sh/release-namespace: {} flabels .: {} f:app.kubernetes.io/managed-by: {} frules {} manager: Go-http-client operation: Update time: "2023-05-17T094158Z" name: traefik resourceVersion: "1850" uid: 13ad53cb-ecf5-43be-8373-7fbeee754d0a rules: - apiGroups: - "" resources: - pods - services - endpoints - secrets verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update
b
That looks correct, can you check the serviceAccount you get when executing: 
kubectl get deployment/traefik -n kube-system -o yaml
? And then verify there is a 
clusterRoleBinding
mapping that serviceAccount with the ClusterRole you just showed me?
As you can observe, your problem is that traefik ingress is unable to work properly because it is unable to access kube-api and fetch Service and Endpoints as it is unauthorized
Copy code
E0519 12:01:23.503152       1 reflector.go:205] <http://github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86|github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86>: Failed to list *v1.Service: Unauthorized
E0519 12:01:24.503507       1 reflector.go:205] <http://github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86|github.com/containous/traefik/vendor/k8s.io/client-go/informers/factory.go:86>: Failed to list *v1.Endpoints: Unauthorized
To get authorization, traefik serviceAccount should have the proper permissions
b
just fyi.. traefik version is image: rancher/library-traefik:1.7.19
serviceAccount: traefik serviceAccountName: traefik
b
there you go:
Copy code
serviceAccount: traefik
      serviceAccountName: traefik
b
yes.. it looks correct
my hello-world application works fine till service
but at ingress it doesnt
my deployment is on Ubuntu 20.04 , any system level changes / configurations i am missing
b
ok, I'm currently busy with a customer issue but I could try and reproduce it using the versions you gave me next week
in any case, there must be something in the roles. It does not look obvious though but it shouldn't be hard to find out 😛
b
./k3s --version k3s version v1.20.4+k3s1 (838a906a) go version go1.15.8 cat k3s-images.txt rancher/coredns-coredns:1.8.0 rancher/klipper-helm:v0.4.3 rancher/klipper-lb:v0.1.2 rancher/library-busybox:1.32.1 rancher/library-traefik:1.7.19 rancher/local-path-provisioner:v0.0.19 rancher/metrics-server:v0.3.6 rancher/pause:3.1
many thanks @bland-account-99790 , let me know if you can do it over a google/zoom call , as per your convenience
b
I tried with k3s 1.26 and it worked well though
🙌 1
29 Views
Open in Slack
PreviousNext
Powered by