Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
c
creamy-room-58344
05/24/2023, 9:09 PMhello there
I added a new disk to my server, it is using ZFS
I won't be using this disk on kubernetes but k3s is trying to use it somehow and won't start
c
creamy-pencil-82913
05/24/2023, 9:16 PMwhat makes you think that?
where did you mount it?
c
creamy-room-58344
05/24/2023, 9:17 PM//mnt/bigdata
c
creamy-pencil-82913
05/24/2023, 9:21 PMthat’s not a path k3s would be looking at by default; what makes you think it’s trying to use it?
c
creamy-room-58344
05/24/2023, 9:53 PMthis error in /var/lib/rancher/k3s/agent/containerd/containerd.log
time="2023-05-24T18:52:13.612877359-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1this is the full log:
time="2023-05-24T18:52:13.585368316-03:00" level=info msg="starting containerd" revision= version=v1.6.15-k3s1
time="2023-05-24T18:52:13.609975892-03:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2023-05-24T18:52:13.610022406-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.611800070-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/6.3.3-arch1-1\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.611877720-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612077465-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.btrfs (xfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612101932-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612118678-03:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-05-24T18:52:13.612131643-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612165261-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612281124-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612316273-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612709528-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612877359-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612906185-03:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2023-05-24T18:52:13.612930493-03:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-05-24T18:52:13.612945193-03:00" level=info msg="metadata content store policy set" policy=shared
panic: invalid freelist page: 1320, page type is leaf
goroutine 68 [running]:
<http://go.etcd.io/bbolt.(*freelist).read(0x2383575|go.etcd.io/bbolt.(*freelist).read(0x2383575>?, 0x7f7723928000)
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/freelist.go:266 +0x22e
<http://go.etcd.io/bbolt.(*DB).loadFreelist.func1()|go.etcd.io/bbolt.(*DB).loadFreelist.func1()>
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc0001d8ac8?, 0x10?)
/usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
/usr/local/go/src/sync/once.go:65
<http://go.etcd.io/bbolt.(*DB).loadFreelist(0xc0001d8900?)|go.etcd.io/bbolt.(*DB).loadFreelist(0xc0001d8900?)>
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:316 +0x47
<http://go.etcd.io/bbolt.Open({0xc0002dea50|go.etcd.io/bbolt.Open({0xc0002dea50>, 0x4c}, 0x23829c7?, 0xc0006f0e60)
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
<http://github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc00041a700)|github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc00041a700)>
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/services/server/server.go:466 +0x94e
<http://github.com/containerd/containerd/plugin.(*Registration).Init(0xc0001152c0|github.com/containerd/containerd/plugin.(*Registration).Init(0xc0001152c0>, 0xc00041a700)
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/plugin/plugin.go:113 +0x2e
<http://github.com/containerd/containerd/services/server.New({0x27048b8|github.com/containerd/containerd/services/server.New({0x27048b8>, 0xc000060f80}, 0xc000146380)
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/services/server/server.go:243 +0x13d9
<http://github.com/containerd/containerd/cmd/containerd/command.App.func1.1()|github.com/containerd/containerd/cmd/containerd/command.App.func1.1()>
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/cmd/containerd/command/main.go:195 +0x87
created by <http://github.com/containerd/containerd/cmd/containerd/command.App.func1|github.com/containerd/containerd/cmd/containerd/command.App.func1>
/go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/cmd/containerd/command/main.go:192 +0x93bany idea?
c
creamy-pencil-82913
05/24/2023, 10:40 PMthat zfs error has nothing to do with anything, its just telling you that it’s not on zfs so it can’t use the zfs snapshotter.
the actual error is that one of containerd’s databases has been corrupted; did you shut it down uncleanly?
c
creamy-room-58344
05/24/2023, 10:41 PMyeap
i had a power failure :-(
c
creamy-pencil-82913
05/24/2023, 10:41 PMyeah you’re going to have to run k3s-killall.sh and then delete all the containerd directories and let it start over
c
creamy-room-58344
05/24/2023, 10:42 PMok
i'll try that
rm -rf /var/lib/rancher/k3s/agent/containerd/io.containerd*/up and running 🙂
thanks @creamy-pencil-82913