https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
w
This is in reference to the above message: https://rancher-users.slack.com/archives/CGGQEHPPW/p1690096022092449 Meant to reply to that thread but it went to the main channel and I cannot delete this message…
c
Where are you looking in KubeVirt? This is coming from the apiserver, not anything in K3s itself. https://github.com/kubernetes/kubernetes/blob/v1.27.3/staging/src/k8s.io/apiserver/pkg/endpoints/discovery/aggregated/handler.go#L232
w
Yea I looked at that and noticed it was added in 
kubernetes v1.27
and then decided to install 
k3s
on another fresh node, similar messages are being repeated but not with 
kubevirt
since I didn’t install that there:
Copy code
Jul 24 13:55:42 csbuild k3s[816135]: I0724 13:55:42.239616  816135 handler.go:232] Adding GroupVersion <http://helm.cattle.io|helm.cattle.io> v1 to ResourceManager
Jul 24 13:55:42 csbuild k3s[816135]: I0724 13:55:42.239692  816135 handler.go:232] Adding GroupVersion <http://k3s.cattle.io|k3s.cattle.io> v1 to ResourceManager
Jul 24 13:55:42 csbuild k3s[816135]: I0724 13:55:42.239818  816135 handler.go:232] Adding GroupVersion <http://traefik.containo.us|traefik.containo.us> v1alpha1 to ResourceManager
Jul 24 13:55:42 csbuild k3s[816135]: I0724 13:55:42.239940  816135 handler.go:232] Adding GroupVersion <http://helm.cattle.io|helm.cattle.io> v1 to ResourceManager
Jul 24 13:55:42 csbuild k3s[816135]: I0724 13:55:42.285663  816135 handler.go:232] Adding GroupVersion <http://metrics.k8s.io|metrics.k8s.io> v1beta1 to ResourceManager
Jul 24 13:56:42 csbuild k3s[816135]: I0724 13:56:42.228518  816135 handler.go:232] Adding GroupVersion <http://metrics.k8s.io|metrics.k8s.io> v1beta1 to ResourceManager
Jul 24 13:57:42 csbuild k3s[816135]: I0724 13:57:42.229540  816135 handler.go:232] Adding GroupVersion <http://metrics.k8s.io|metrics.k8s.io> v1beta1 to ResourceManager
Jul 24 13:58:42 csbuild k3s[816135]: I0724 13:58:42.229354  816135 handler.go:232] Adding GroupVersion <http://metrics.k8s.io|metrics.k8s.io> v1beta1 to ResourceManager
c
yes as far as I can tell, the apiserver just logs that periodically now. it’s not anything specific to K3s.
w
Wonder why minikube isn’t logging it? Maybe a config or?
c
different apiserver log level perhaps?
I don’t know what minikube defaults to
w
It’s a bit of a nuisance if innocuous because the log gets pretty big and eventually we log rotate it. Perhaps I should report this to the kubernetes project? Seems this should be more debug than info or at least not repeat it.
c
Does minikube perhaps have the 
AggregatedDiscoveryEndpoint
featuregate disabled? It looks to me like the controller that’s syncing those regularly is gated behind that. It’s beta in 1.27 so on by default.
w
I’ve tired looking for 
k3s
featuregates and wasn’t quite sure how I see those listed and I’m not sure how to see them on minikube either. I ran a few commands but it didn’t show me much:
Copy code
kubectl describe pod/kube-apiserver-minikube -n kube-system
... 
  Command:
      kube-apiserver
      --advertise-address=10.0.2.15
      --allow-privileged=true
      --authorization-mode=Node,RBAC
      --client-ca-file=/var/lib/minikube/certs/ca.crt
      --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/var/lib/minikube/certs/etcd/ca.crt
      --etcd-certfile=/var/lib/minikube/certs/apiserver-etcd-client.crt
      --etcd-keyfile=/var/lib/minikube/certs/apiserver-etcd-client.key
      --etcd-servers=<https://127.0.0.1:2379>
      --kubelet-client-certificate=/var/lib/minikube/certs/apiserver-kubelet-client.crt
      --kubelet-client-key=/var/lib/minikube/certs/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/var/lib/minikube/certs/front-proxy-client.crt
      --proxy-client-key-file=/var/lib/minikube/certs/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/var/lib/minikube/certs/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=8443
      --service-account-issuer=<https://kubernetes.default.svc.cluster.local>
      --service-account-key-file=/var/lib/minikube/certs/sa.pub
      --service-account-signing-key-file=/var/lib/minikube/certs/sa.key
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/var/lib/minikube/certs/apiserver.crt
      --tls-private-key-file=/var/lib/minikube/certs/apiserver.key
How does one check for the featuregates configured for k3s? Maybe that can help me find it in minikube. I know how to configure them for k3s and minikube via the config yaml or command line.
c
normally you’d see them listed in the --feature-gates flag
You’ve looked at the kube-apiserver pod logs on your minikube system and don’t see any similar messages?
I believe you’d need to have something deployed that uses apiserver aggregation
w
Just to try it out I enabled 
AggregatedDiscoveryEndpoint
and still don’t see the repeated log messages: 
minikube start --feature-gates=AggregatedDiscoveryEndpoint=true
Output of `kubectl logs kube-apiserver-minikube -n kube-system`: https://hastebin.com/share/ozisojahel.bash
Ah ok good point, perhaps nothing in minikube is using that feature
I can try to disable it in my k3s setup, though my guess is that isn’t recommended for production?
c
I’m not sure exactly what uses it
w
Ok confirmed, deploying kubevirt which uses the apiserver aggregation exhibits the same behavior:
Copy code
kubectl logs kube-apiserver-minikube -n kube-system
...
I0724 21:51:01.671458       1 handler.go:232] Adding GroupVersion <http://kubevirt.io|kubevirt.io> v1 to ResourceManager
I0724 21:51:01.671659       1 handler.go:232] Adding GroupVersion <http://kubevirt.io|kubevirt.io> v1alpha3 to ResourceManager
I0724 21:51:39.013565       1 trace.go:219] Trace[910383161]: "Get" accept:application/json, */*,audit-id:0045bcdd-e1ab-44bb-b221-2eef51b64a28,client:127.0.0.1,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-minikube/log,user-agent:k9s/v0.0.0 (linux/amd64) kubernetes/$Format,verb:CONNECT (24-Jul-2023 21:51:36.867) (total time: 2146ms):
Trace[910383161]: ---"Writing http response done" 2141ms (21:51:39.013)
Trace[910383161]: [2.146247435s] [2.146247435s] END
I0724 21:55:30.640176       1 handler.go:232] Adding GroupVersion <http://kubevirt.io|kubevirt.io> v1 to ResourceManager
I0724 21:55:30.640284       1 handler.go:232] Adding GroupVersion <http://kubevirt.io|kubevirt.io> v1alpha3 to ResourceManager
Seems like this should be reported though to kubernetes.
c
yeah I’d probably take it upstream
w
Awesome thanks you’ve been super helpful as always!
c
👍
101 Views
Open in Slack
PreviousNext
Powered by