https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rancher-desktop
a
This message was deleted.
f
It is a long-standing issue that connections sometimes seem to start hanging (randomly). I suspect it is related to them being tunnelled over ssh. Do you have a large number of simultaneous connections? Since the port forwarding uses a the ControlMaster/ControlPath feature of ssh to avoid auth overhead on each single connection, it is conceivable that you could hit the max connection limit. I would just think if that were the case, that problem would be more predictable.
m
It's a single connection
f
One thing to try would be to use a routable port instead. If you enable administrative access (Release Rancher Desktop 1.12.1), then you should get a bridged network from VMNET with an IP address from your local network. That connection doesn't go through libslirp and ssh. So it would be interesting to see if that interface has the same issues.
m
Interesting, I saw that pop up and gave it permissions, how would I go about finding the IP assigned to the LB for that? Is it the External IP for the LB or is found elsewhere?
f
Yes, the external IP for your LB should be updated.
It is 
192.168.5.15
when using the internal networking, but should be in you local LAN address range when you get a bridged interface
Could you also write up your test case as a Github issue, with more detail about how somebody else could reproduce it? That way somebody could investigate further where this issue is introduced (I suspect libslirp or ssh).
Another thing to narrow this down is to switch emulation engines from QEMU to VZ or vice versa, depending on what you are currently using (Emulation (macOS) | Rancher Desktop Docs).
Because in the non-admin configuration VZ uses a different networking stack (vzNAT) than QEMU
m
Currently reboot the cluster to make sure there I actually had the admin perms enabled. I'm currently using 
socket-vmnet
and 
VZ
emulation. I'll try flipping them around and see if it makes a difference in a moment here.
Hm, trying to netcat to the external IP seems to not work, but netcat to localhost works fine:
Copy code
% nc -vz localhost 31919
Connection to localhost port 31919 [tcp/*] succeeded!
% nc -vz 192.168.50.100 31919
... never finishes
f
Ok, thanks! Note that VZ in admin mode will always use 
socket_vmnet
, regardless of the setting because the alternative (
vde_vmnet
) is not supported with VZ.
m
Oh, good to know
f
But 
vde_vmnet
and 
socket_vmnet
are extremely similar, so I would not bother with toggling this setting to see if it makes a difference. The plan is to drop 
vde_vmnet
in the near future anyways
m
I do see two bridges adapters, one with 
inet 192.168.205.1
f
Note that automatic port-forwarding is always to localhost, not to the bridged IP. Only your LB etc should be visible on the bridged network.
m
Yeah my lb shows:
Copy code
streamer-service-tcp    LoadBalancer   10.43.73.54     192.168.50.100   31919:31446/TCP
Oh interesting, so I may have not been using the LB?
f
192.168.205.0/24
is a "shared" network, that is local to the host, but also provided by VMNET. It is only used if the "bridged" network doesn't get an IP address, which sometimes happens for mostly unknown reasons (e.g. Wifi does not really support multiple IP addresses on a single MAC, but most of the time it somehow does anyways)
m
Ah, I am on WIFI.
f
Yeah, but 
192.168.50.100
looks like a local network IP, so the bridged network seems to be working
Just testing with the default LB:
Copy code
kube-system   traefik          LoadBalancer   10.43.78.101    192.168.18.119   80:31025/TCP,443:32191/TCP   6h24m
Copy code
$ curl <http://192.168.18.119>
404 page not found
$ curl -k <https://192.168.18.119>
404 page not found
m
hmm, strange, localhost works fine for me, but external ip still isn't.
!
I dropped off our VPN and it connected
One moment, let me see if I can still replicate the connections dropping
Still seeing 
FIN/ACK
spam after connecting on both 127.0.0.1 and the external ip
Interesting though that our VPN was preventing me from connecting to the external IP
f
I was expecting that using the external IP would avoid the issue; interesting that it doesn't. Can you do the connection test again localhost from inside the VM? Does it show the same issue?
m
How do I go about getting a shell in the VM?
f
rdctl shell
The VM is running Alpine/OpenRC, so you don't have systemd...
You can install additional packages with 
apk add ...
, but that has to be redone after each VM restart because it boots from an ISO image (the package will be cached locally; they just have to be reinstalled)
E.g. if you need the 
nc
for your tests, then you probably want to install 
apk add netcat-openbsd
instead of using the 
busybox
version of 
nc
, which is a completely different implementation.
m
Through investigating some differences between those two layers I think I've observed something that may point to it being not a fault of the underlying networking layers. I'm not certain on it yet, but have run out of time to debug it for today, so will get back to you on Tuesday with some positive results hopefully. Have a great weekend @fast-garage-66093 appreciate your help 馃檪
f
Thank you, have a great weekend! Looking forward to what you find out!
b
@fast-garage-66093 what about Lima VM disk corruption under VZ?
m
We ended up solving this one. Was going to post here in a bit. I'm not entirely sure why the rst KA side was coming from the wrong direction, but we had some application logic to only hold one connection open at a time. And it seems something was probing the port by trying to connect to it, then immediately closing the connection. Thus kicking off the previous real connection due to our code. After making some validation changes there were back in action. But still unsure where the probes are coming from.
f
@broad-train-31975 This should be fixed in Rancher Desktop 1.12.0+
b
Thank you, Jan
f
b
Love the 
rumored
part 馃槂
f
Yeah, it is unsatisfying, but only so much you can do with closed source software... But then, there are even rumours about disk corruption with QEMU, but I haven't seen any reports about this with Rancher Desktop.
@most-bear-54693 Thanks for reporting back! If you discover an actual bug, please make sure to file an issue!
b
long thread... probably hard to chase the root cause. Kudos to selfless people!
22 Views
Open in Slack
PreviousNext
Powered by