https://rancher.com/ logo
Join Slack
Powered by
is the containerd binary used by rke2 altered from...
# rke2
b
is the containerd binary used by rke2 altered from upstream (specifically the windows binary)?
c
yes, we build from the k3s fork of containerd so that it supports registry rewrites. Upstream has not accepted that feature.
that is true for both linux and windows
b
does it simply add features or are any bits removed?
I'm trying to get a situation where the 'scratch' disk size on windows actually takes effect and for the life of me I cannot get rke2 to actually give me a larger disk size.
I have a patched (a bug in 
ctr
which doesn't actually set the 
label
on the snapshot info from the cli args) 
ctr
binary running against upstream 
v2.0.4
(what our clusters are currently reporting) and that combo seems to work..but when I run it against the windows node using the rke2 binary/service it doesn't expand the volume
c
You can go look at the k3s fork
b
was just doing that, code seems to be there...or at least the higher level stuff is
c
b
let me build this real fast and run against that
hmm, that seems to work..maybe need to run against a different version of windows
currently testing against 2019 (launching containerd manually) but the rke2 nodes are running 22 I think
not sure if cimfs is working there or not so 2022 is getting a different snapshotter than 2019
c
I'm not super familiar with how the snapshotter works on Windows, i just know it's a huge pain to clean up manually - you need some specific tool that you can build from the hcs repo
👍 1
b
We have had lots of clean up issues when containers stop etc for sure.
If you happen to know the tool send it over so I can share with the team as it may be better than whatever process they are doing now
c
👍 1
b
I have now confirmed this works with both upstream and rke containerd after hacking ctr binary etc
c
What are you patching in ctr?
b
I’m specifically patching the 
image mount
sub command to actually do something with the —label parameter and pass it in as the snapshot labels
It has a documented —label parameter that currently does nothing. I need it to specifically be the label on the snapshot and not the image or any other stuff…must be the snapshot
With the patch where the labels are actually sent into snapshot.Prepare method I am getting windows to actually adjust the disk scratch size.
The end goal here is create a csi driver that is 
inline ephemeral
which creates a rw scratch dir on the node with a oci image as the source content. Essentially we have large git repos that we want to cache on the node level and inherit all the layer goodness for ci jobs. We want to let each build create a ‘cow’ fork of the ‘fs’ that is rw so we don’t use up so much space and don’t need to keep recloning for every build incurring the network delay etc
Each build will mount the oci image with the git repos, then pull any delta and checkout the correct commit and also have a rw space to diff from the current image content (as opposed to the new oci volume type which is ro).
Open in Slack
PreviousNext
Powered by