https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
b
@creamy-pencil-82913 any info on this one?
@creamy-pencil-82913 bump, this is killing me :( not sure if I should be exploring more the rke2 side or hit up the calico team.
c
What version are you using?
b
latest 1.34..we've upgraded for several but I think this has been a chronic problem for a long time
c
Upstream project is the correct place to look for fixes. If they fix it we can pull that in to a future release.
b
are they launched separately from the rke2 service? can I replace them in-place if I start hacking on new binaries
c
we just include calico, calico-node, and calico-ipam. I do not honestly know off the top of my head where exactly felix comes from
I don’t touch the windows stuff often
b
ok, I can't seem to find the felix binary in the rke stuff really..you don't know where that file is so that I can place hacked binaries with more debugging etc?
amen to not touching windows..can't avoid it atm unfortunately 😞
c
it looks like felix is 
calico-node.exe -felix
b
ah, ok..and the rke2 service kicks that off as part of it's startup?
c
if you’re going to be poking at this you should probably familiarize yourself with this codebase a bit. https://github.com/rancher/rke2/blob/acf308a5247d10704b1d81bb192c8d9a4c0a2fc2/pkg/windows/calico.go#L444-L472
b
from what I've been told it seems to be die'ing and only restarting the rke service brings it back up..like possibly the process isn't being monitored and automatically restarting it
c
yeah it looks like we just error there instead of restarting it
b
yeah, I've been in this file before digging into the node ip detection stuff..
c
I am working on some of the CNI startup stuff as part of a larger refactoring effort, I can make a note to see if restarting it automatically makes sense
👍 1
b
Yeah, please do. This info has been very helpful. Thanks! Hypothetically if I replace the calico-node.exe file in place on the fs rke2 will happily start it up yeah?
c
yes if you replace the binaries they won’t be overwritten
Feel free to add any info you have on the crash here, as it might help our QA team repro: https://github.com/rancher/rke2/issues/9324
b
Glorious! I will dig in and probably follow-up about the restarts on this thread after I have dug in a bit.
Yeah, I will add some notes there..the crashes are pretty strange honestly.
Copy code
2025-11-20 21:37:43.571 [INFO][10028] felix/win_dataplane.go 252: Received *proto.ServiceAccountUpdate update from calculation graph msg=id:{namespace:"build-agents-github-actions" name:"dotnet-adc-software-poc-runner-scale-set-gha-rs-no-permission"} labels:{key:"<http://actions.github.com/scale-set-name|actions.github.com/scale-set-name>" value:"dotnet-adc-software-poc-runner-scale-set"} labels:{key:"<http://actions.github.com/scale-set-namespace|actions.github.com/scale-set-namespace>" value:"build-agents-github-actions"} labels:{key:"<http://app.kubernetes.io/instance|app.kubernetes.io/instance>" value:"dotnet-adc-software-poc-runner-scale-set"} labels:{key:"<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>" value:"Helm"} labels:{key:"<http://app.kubernetes.io/name|app.kubernetes.io/name>" value:"dotnet-adc-software-poc-runner-scale-set"} labels:{key:"<http://app.kubernetes.io/part-of|app.kubernetes.io/part-of>" value:"gha-rs"} labels:{key:"<http://app.kubernetes.io/version|app.kubernetes.io/version>" value:"0.12.1"} labels:{key:"<http://argocd.argoproj.io/instance|argocd.argoproj.io/instance>" value:"dev-na01-workload-00-build-agents-github-actions"} labels:{key:"<http://helm.sh/chart|helm.sh/chart>" value:"gha-rs-0.12.1"} labels:{key:"<http://projectcalico.org/name|projectcalico.org/name>" value:"dotnet-adc-software-poc-runner-scale-set-gha-rs-no-permission"}
runtime: g 276: unexpected return pc for <http://github.com/projectcalico/calico/libcalico-go/lib/backend/k8s/resources.pagedList|github.com/projectcalico/calico/libcalico-go/lib/backend/k8s/resources.pagedList> called from 0xc000aeef14
Copy code
2025-11-20 15:31:27.435 [INFO][116416] felix/win_dataplane.go 254: Received *proto.NamespaceUpdate update from calculation graph msg=id:<name:"platform-dogfood" > labels:<key:"app" value:"raw" > labels:<key:"argocd.argoproj.io/instance" value:"dev-na01-workload-00-service-onboarding-namespaces" > labels:<key:"chart" value:"raw-0.2.5" > labels:<key:"field.cattle.io/projectId" value:"platform" > labels:<key:"heritage" value:"Helm" > labels:<key:"istio-injection" value:"disabled" > labels:<key:"kubernetes.io/metadata.name" value:"platform-dogfood" > labels:<key:"projectcalico.org/name" value:"platform-dogfood" > labels:<key:"rancher-to-argocd/projectName" value:"platform" > labels:<key:"release" value:"namespaces" > 
runtime: frame sigtramp untyped locals 0xc000882e58+0xf8
fatal error: missing stackmap
almost like they are compiler issues or something..I grep the calico codebase for anything and nothing shows up
c
unexpected return pc indicates something really wrong in core golang, or some sort of underlying memory issue on the node itself
yeah
those are all memory/stack corruption crashes
b
not a go guy..but glad my initial thoughts were not too far off
is there a quick/dirty hack I can do to force it restart at least?
probably just put that 
.Run
in some kind of loop?
c
not really no…
not without rebuilding rke2
b
I can probably get that done..
if felix is restarted would the networking restore for the respective running pod? or it's useless if felix happens to crash and will never come back?
c
I don’t know
👍 1
most of our logic was cribbed from upstream’s sketchy powershell scripts - we just reimplimented it in go because running a bunch of powershell scripts seemed lazy. https://github.com/projectcalico/calico/blob/master/node/windows-packaging/CalicoWindows/felix/felix-service.ps1
but yeah it looks like we missed the bit where it exits intentionally after a config change
b
yeah, we're seeing a bunch of those too
so it should restart after a config change and come up happy and healthy?
that actually could be the largest portion of our problem!
c
apparently? idk what happens when you restart it when it exits unexpectedly. It seems like this script just leaves it down if it exits with any other code. I couldn’t say why though.
oh wait no it restarts it either way
b
well, either way the thing should seemingly be in a loop
c
it just logs a different message lol
b
'shat myself' instead of 'config changed'? lol
c
I guess the restart logic was added more recently, after we converted it to go. https://github.com/projectcalico/calico/commit/f2ea5bd356d79d6265667c1a7a32ad4c473144ed
b
yeah, that probably should have been there all along I am guessing
c
yeah. windows is kinda unloved everywhere
you could always use flannel instead ;)
b
don't the rke2 docs say calico is the only cni that works with windows?
I mean, that shipped has sailed for us essentially, but still..pretty sure that's what the docs say.
c
no. we added flannel like, a year ago?
almost 2 now
b
ah..well I can look into it..but we've got a lot going on that I would be pretty cautious about that at this point
but certainly noted
c
MS has/had an issue with their network firewall implimentation (which is the backend for the calico netpol on windows) where it resets all the connection tracking info every time any of the endpoints change - so any time a source OR destination pod comes or goes it drops all the connections
b
good hell
how does flannel handle netpols?
c
Flannel has no netpol support so we were suggesting folks work around that by just using flannel instead
b
lol ah
is the dropped connection situation still the case?
c
as far as I know. it is covered in the Tigera docs for Windows: https://docs.tigera.io/calico/latest/getting-started/kubernetes/windows-calico/limitations#pod-to-pod-connections-are-dropped-with-tcp-reset-packets
Restarting Felix or changes to policy (including changes to endpoints referred to in policy), can cause pod-to-pod connections to be dropped with TCP reset packets. When one of the following occurs:
• The policy that applies to a pod is updated
• Some ingress or egress policy that applies to a pod contains selectors and the set of endpoints that those selectors match changes
Felix must reprogram the HNS ACL policy attached to the pod. This reprogramming can cause TCP resets. Microsoft has confirmed this is a HNS issue, and they are investigating.
I am not convinced that even MS actually uses containers on windows. it is full of fun bugs like this
b
yeah...right now we're 'just' trying to get CI to run on the damn things for various reasons
I had a hell of a time trying to build my own csi drivers so we could use cache drives and other stuff...learned a bunch of crazy things about windows containers along the way
like..the 20G 'scratch' space limit on a running container image
c
the way MS does filesystems for containers is literally insane. You need special tools that are almost impossible to find if you want to clean up the mount points without the container runtime up
all of it feels like a terrible hack
b
at first I built a csi that replicates the newly created 'image' volume type in k8s...but add a writable layer
so the builds could have a massive cached git repo and actually do something with it...prototyped fine but then when running under rke2 there was some kind of permission issue
then I switched gears and created a 'vhdx' csi where we literally distribute a vhdx to all build nodes and then when the ci container launches the csi mounts the vhdx as a rw snapshot (so each build has a clean start and writable area)
that's actually working quite well, but the containerd oci image approach would be preferrable to solve the distribution issue along with more gracefully handling deltas
@creamy-pencil-82913 what go build command would I use to build the rke2.exe binary if I start hacking this loop
c
You need to run a full build including producing the rke2-runtime image. We cross-build windows on Linux.
You will also need to push the rke2-runtime image to a registry somewhere, or use an airgap tarball, or override the rke2-runtime image name to point at the GA tag.
b
nah, I'm good
make build-windows-binary
gets me an 
rke2.exe
just need to plop that on the fs and restart the service no?
c
No, thatll want an rke2-runtime image that doesn't exist - something with a -dev tag corresponding to whatever state you left the repo in
You can use that if you override the runtime image name to point at an existing tag
b
well, I can just checkout the git tag that matches the version we are currently running?
c
No, it looks at the tree state. If you're not building on clean repo with a tagged commit it will want to use a -dev image.
b
any way around that?
well, easily around that lol
c
Yes set the runtime image name in the config like I said
Put it in config.yaml or pass it on the cli
b
it's just 1 datapoint?
how can I discover what the current binary is using?
c
The default rke2-runtime image tag is determined from the running version string. You can hardcode that if you want to use something else.
b
well, can I run rke2 --foo to get the values?
seems like rke1 had something along those lines
c
Just point it at the rke2-runtime image listed in the image list for the release you're building off of
b
I'm an ignorant..where would I find that?
c
There are image lists published in the GH release artifacts for every version
b
ok, gimme a sec to see if I can connect the dots
I think we're on the latest 1.34.2
c
It is basically just 
rancher/rke2-runtime:v1.34.2-rke2r1
b
k, let me see if I can find the correct place in the yaml to place that
c
But for non release builds it is -devXxx
b
this yeah? 
runtime-image
c
Yep
b
@creamy-pencil-82913 any way around this? 
time="2025-12-04T20:24:05-05:00" level=fatal msg="Failed to validate golang version: incorrect golang build version - kubernetes v1.34.2 should be built with go1.24.9, runtime version is go1.25.4"
c
Build with the correct golang version.
Just run 
make in-docker-build-windows-binary
👍 1
b
@creamy-pencil-82913 by looking at the code shouldn't I have multiple processes of 
calico-node.exe
running?
looks like it starts at least 3..calico, felix, and confd
c
calico doesn’t stay running. it just does setup.
b
ok, so confd would stay running but may not be active in my setup or something?
in short, it's plausible that I would only have 1 calico-node.exe process running long-term on the node
c
confd is only used if you’re using bgp
👍 1
b
Unless that's a default I doubt we are.
Probably would conflict on the linux nodes with metallb or something anyway.
we're testing a custom build now and will put it through the paces over the weekend..if things look happy (at least for the relative issue) I will attach a diff to that gh issue
c
b
ok great! Mine is a small subset of that and pretty similar so I think we should be good. Results so far seem quite positive.
how long is the sleep in there?
actually, I see the sleep is only in confd so less worried about it
Will your PR be in 1.34.3?
@creamy-pencil-82913 ^ forgot to tag you on that question
c
yep
b
perfect, thanks for hearing me out on this one! I will probably have another windows thing or 2 to share in the near-ish future
16 Views
Open in Slack
PreviousNext
Powered by