https://rancher.com/ logo
Powered by
Title
i

icy-garage-31078

09/15/2022, 5:42 PM
Has anyone seen this before. RD 1.5.1 w/docker engine: I'm running a basic 
docker build
on a M1. The command runs, getting a couple steps in before it dies with a message of 
unexpected EOF
. If I immediately try to run the same command again, I instantly get the dread 
...docker.sock. Is the docker daemon running
. I tried using 
DOCKER_HOST=<tcp://ip:2375> docker build ...
but this fails with the same error above after the build failed
FYI, Just restarting RD does 'fix' the issue in that docker commands run again, but I immediately get the same error.
w

wide-mechanic-33041

09/15/2022, 5:58 PM
and the context is set to rancher-desktop?
i

icy-garage-31078

09/15/2022, 6:04 PM
Yes, I have it set to rancher-desktop.
👍 1
w

wide-mechanic-33041

09/15/2022, 6:08 PM
is this a big build? could it be an out of memory issue?
i

icy-garage-31078

09/15/2022, 6:12 PM
In my opinion it isn't a big build. It's the final step of 9 running 
yarn run build
. Additionally, I've got the VM info in RD set to 24GB of Ram and 6 vCPUs
w

wide-mechanic-33041

09/15/2022, 6:13 PM
heh i have to get a desktop upgrade! 🤣
🤣 1
i

icy-garage-31078

09/15/2022, 6:14 PM
I don't think it matters, but I am using DOCKER_BUILDKIT=1
w

wide-mechanic-33041

09/15/2022, 6:14 PM
if you rdctl shell into the VM anything odd inside the VM that you can see during the build?
i

icy-garage-31078

09/15/2022, 6:18 PM
I've not previously done anything like that. Do you have a quick how-to?
w

wide-mechanic-33041

09/15/2022, 6:20 PM
just type 
rdctl shell
and you will be in the Alpine VM where the containers are running. challenge will be limited tooling in there by default. top is in there though
and you can 
tail -f /var/log/messages
to see what flies by as you do the build
it seems like some resource issue leading to instability in the VM which is why a restart (reboot of the VM) fixes it temporarily
i

icy-garage-31078

09/15/2022, 6:30 PM
Nothing jumping out at me ATM, but I am seeing this scroll through often 
process '/sbin/getty -L 0 ttyS0 vt100' (pid 2047) exited. Scheduling for restart.
. I don't think it's related. I need to reboot RD and try a build again.
So I ran the build and noticed a couple things. First, I am regurally getting the message 
alpine auth.err getty[24624]: tcgetattr: I/O error^M
. This is regardless of if the build is running or not and comes up every second or so. After running the build, I noticed a few different messages. When I first started, I saw 
/usr/local/bin/rancher-desktop-guestagent, pid 23857, exited with return code 0
, but after the build ran for a little it became 
/usr/local/bin/rancher-desktop-guestagent, pid 24620, terminated by signal 9
. While the build was running I saw the message 
Child command line: /usr/local/bin/cri-dockerd --container-runtime-endpoint unix:///run/cri-dockerd.sock --network-plugin=cni --cni-bin-dir=/usr/libexec/cni --cni-conf-dir=/etc/cni/net.d --cni-cache-dir=/var/lib/cni/cache
come up twice. Finally, right at the same time the docker build failed with the 
unexpected EOF
, I got 
Error: exit status 255
and went back to my normal shell. I ran 
rdctl shell
near immediately and it went right back in.
w

wide-mechanic-33041

09/15/2022, 7:08 PM
you seeing anything in 
top
i

icy-garage-31078

09/15/2022, 7:11 PM
I'm working on that view now.
Looks like it may be memory related, which is odd. I went ahead and installed htop directly then watched it while the build ran. Nothing out of the ordinary happened with the processes that I saw. I did see a spike where all 6 vCPUs went to 100% for 10 seconds or so, but it dropped off and stayed stable. Right before it crashed and kicked me out again, it was at 50% on all processors. Memory usage is a bit odd though in that it is only showing 2-3 GB of ram. Right before it died, the memory did spike to basically 100% if I'm reading it correctly. Need to double check the color-coding on that, but 90% of the line was green while the least 10% was purple. First I've seen that.
w

wide-mechanic-33041

09/15/2022, 7:28 PM
i mean 90% of a 24GB vm is a big heap. is it like an ai model or something with a huge working set?
i

icy-garage-31078

09/15/2022, 7:32 PM
I'm not sure I understand the question? Additionally, I'm not convienced it is using the full 24G, as htop reports 2.19G
w

wide-mechanic-33041

09/15/2022, 7:35 PM
just trying to figure out where all the memory was going. some process is trying to allocate more than it’s fair share, but again hard ot know from this side of the chat
i

icy-garage-31078

09/16/2022, 1:30 PM
Keep looking into this and found out that my VM didn't actually have the 24 GB of RAM, even though it was configured for it. Found this: https://rancher-users.slack.com/archives/C0200L1N1MM/p1661873444221709 which basically talks about a limit and needed to upgrade OSX on a M1. I can't do this right now as it's a work laptop, so I'm going to try and add a Swap Space to the image and go from there.
Swap Space Worked
8 Views
#rancher-desktopJoin Slack
Powered by