https://rancher.com/ logo
Title
c

creamy-room-58344

05/24/2023, 9:09 PM
hello there I added a new disk to my server, it is using ZFS I won't be using this disk on kubernetes but k3s is trying to use it somehow and won't start
c

creamy-pencil-82913

05/24/2023, 9:16 PM
what makes you think that?
where did you mount it?
c

creamy-room-58344

05/24/2023, 9:17 PM
//mnt/bigdata
c

creamy-pencil-82913

05/24/2023, 9:21 PM
that’s not a path k3s would be looking at by default; what makes you think it’s trying to use it?
c

creamy-room-58344

05/24/2023, 9:53 PM
this error in /var/lib/rancher/k3s/agent/containerd/containerd.log
time="2023-05-24T18:52:13.612877359-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
this is the full log:
time="2023-05-24T18:52:13.585368316-03:00" level=info msg="starting containerd" revision= version=v1.6.15-k3s1
time="2023-05-24T18:52:13.609975892-03:00" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
time="2023-05-24T18:52:13.610022406-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.611800070-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/6.3.3-arch1-1\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.611877720-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612077465-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.btrfs (xfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612101932-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612118678-03:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-05-24T18:52:13.612131643-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612165261-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612281124-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612316273-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612709528-03:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612877359-03:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-05-24T18:52:13.612906185-03:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
time="2023-05-24T18:52:13.612930493-03:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
time="2023-05-24T18:52:13.612945193-03:00" level=info msg="metadata content store policy set" policy=shared
panic: invalid freelist page: 1320, page type is leaf

goroutine 68 [running]:
<http://go.etcd.io/bbolt.(*freelist).read(0x2383575|go.etcd.io/bbolt.(*freelist).read(0x2383575>?, 0x7f7723928000)
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/freelist.go:266 +0x22e
<http://go.etcd.io/bbolt.(*DB).loadFreelist.func1()|go.etcd.io/bbolt.(*DB).loadFreelist.func1()>
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:323 +0xb8
sync.(*Once).doSlow(0xc0001d8ac8?, 0x10?)
        /usr/local/go/src/sync/once.go:74 +0xc2
sync.(*Once).Do(...)
        /usr/local/go/src/sync/once.go:65
<http://go.etcd.io/bbolt.(*DB).loadFreelist(0xc0001d8900?)|go.etcd.io/bbolt.(*DB).loadFreelist(0xc0001d8900?)>
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:316 +0x47
<http://go.etcd.io/bbolt.Open({0xc0002dea50|go.etcd.io/bbolt.Open({0xc0002dea50>, 0x4c}, 0x23829c7?, 0xc0006f0e60)
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/vendor/go.etcd.io/bbolt/db.go:293 +0x48b
<http://github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc00041a700)|github.com/containerd/containerd/services/server.LoadPlugins.func2(0xc00041a700)>
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/services/server/server.go:466 +0x94e
<http://github.com/containerd/containerd/plugin.(*Registration).Init(0xc0001152c0|github.com/containerd/containerd/plugin.(*Registration).Init(0xc0001152c0>, 0xc00041a700)
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/plugin/plugin.go:113 +0x2e
<http://github.com/containerd/containerd/services/server.New({0x27048b8|github.com/containerd/containerd/services/server.New({0x27048b8>, 0xc000060f80}, 0xc000146380)
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/services/server/server.go:243 +0x13d9
<http://github.com/containerd/containerd/cmd/containerd/command.App.func1.1()|github.com/containerd/containerd/cmd/containerd/command.App.func1.1()>
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/cmd/containerd/command/main.go:195 +0x87
created by <http://github.com/containerd/containerd/cmd/containerd/command.App.func1|github.com/containerd/containerd/cmd/containerd/command.App.func1>
        /go/src/github.com/k3s-io/k3s/build/src/github.com/containerd/containerd/cmd/containerd/command/main.go:192 +0x93b
It all started after I added the disk and ZFS DKMS module
zfs kernel 6.3
any idea?
c

creamy-pencil-82913

05/24/2023, 10:40 PM
that zfs error has nothing to do with anything, its just telling you that it’s not on zfs so it can’t use the zfs snapshotter.
the actual error is that one of containerd’s databases has been corrupted; did you shut it down uncleanly?
c

creamy-room-58344

05/24/2023, 10:41 PM
yeap i had a power failure :-(
c

creamy-pencil-82913

05/24/2023, 10:41 PM
yeah you’re going to have to run k3s-killall.sh and then delete all the containerd directories and let it start over
c

creamy-room-58344

05/24/2023, 10:42 PM
ok i'll try that
rm -rf /var/lib/rancher/k3s/agent/containerd/io.containerd*/
?
up and running 🙂 thanks @creamy-pencil-82913