Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Title
f
fancy-river-86403
02/27/2023, 1:50 PMHi, I am trying to run a k3d cluster for the first time, and like other users, all the pods are in pending state.
However, this doesn't seem to be the same problem as https://github.com/k3d-io/k3d/issues/1229, I don't see the same error.
Any hints on how to debug this ?
I'm on Ubuntu 22.10 with a ZFS filesystem, and after reading k3d and k3s issues with ZFS, I'm not sure whether ZFS is supported now or not ?
~ 14:17:19
❯ k3d cluster create mycluster
INFO[0000] Prep: Network
INFO[0000] Created network 'k3d-mycluster'
INFO[0000] Created image volume k3d-mycluster-images
INFO[0000] Starting new tools node...
INFO[0000] Starting Node 'k3d-mycluster-tools'
INFO[0001] Creating node 'k3d-mycluster-server-0'
INFO[0001] Creating LoadBalancer 'k3d-mycluster-serverlb'
INFO[0001] Using the k3d-tools node to gather environment information
INFO[0001] HostIP: using network gateway 172.21.0.1 address
INFO[0001] Starting cluster 'mycluster'
INFO[0001] Starting servers...
INFO[0001] Starting Node 'k3d-mycluster-server-0'
INFO[0006] All agents already running.
INFO[0006] Starting helpers...
INFO[0006] Starting Node 'k3d-mycluster-serverlb'
INFO[0012] Injecting records for hostAliases (incl. host.k3d.internal) and for 2 network members into CoreDNS configmap...
INFO[0014] Cluster 'mycluster' created successfully!
INFO[0014] You can now use it like this:
kubectl cluster-info
~ 15s 14:17:39
❯ kubectl cluster-info
E0227 14:17:51.700887 1004432 memcache.go:255] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:17:51.705815 1004432 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:17:51.708650 1004432 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:17:51.709917 1004432 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
Kubernetes control plane is running at <https://0.0.0.0:39541>
CoreDNS is running at <https://0.0.0.0:39541/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy>
Metrics-server is running at <https://0.0.0.0:39541/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy>
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
~ 14:18:07
❯ kubectl get pods -n kube-system
E0227 14:18:22.176214 1005153 memcache.go:255] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:18:22.197499 1005153 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:18:22.198935 1005153 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:18:22.200326 1005153 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
NAME READY STATUS RESTARTS AGE
helm-install-traefik-p4kzq 0/1 Pending 0 38s
helm-install-traefik-crd-m5jgv 0/1 Pending 0 38s
local-path-provisioner-79f67d76f8-lrbh2 0/1 Pending 0 38s
coredns-597584b69b-thvd9 0/1 Pending 0 38s
metrics-server-5f9f776df5-vf52f 0/1 Pending 0 38s
~ 14:47:23
❯ kubectl describe node
E0227 14:47:40.154262 1042003 memcache.go:255] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:47:40.163356 1042003 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:47:40.165430 1042003 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0227 14:47:40.167727 1042003 memcache.go:106] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
No resources found in default namespace.I'm not sure what to look for in logs, but this might be relevant :
Feb 27 14:40:03 sylvain-thinkpad kernel: [43848.161346] overlayfs: upper fs does not support RENAME_WHITEOUT.
Feb 27 14:40:03 sylvain-thinkpad kernel: [43848.161365] overlayfs: upper fs missing required features.Well I found this log in docker logs :
time="2023-02-27T14:07:43Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: invalid argument"b
big-librarian-12753
02/27/2023, 2:30 PMJust copying pasting response for a similar message https://rancher-users.slack.com/archives/CHM1EB3A7/p1677045656078989?thread_ts=1677037029.968989&cid=CHM1EB3A7
f
fancy-river-86403
02/27/2023, 2:55 PMOK I solved my problem, it was the ZFS issue, despite me creating a single (local) server setup and not having the same errors as in the FAQ, the workaround from https://k3d.io/v5.2.2/faq/faq/#issues-with-zfs solved my issue
👍 1