This message was deleted.

Things come back and operate normally for a bit, and then it pauses again.

Is there a reason you aren't using a native image for your workload? (qemu is expensive...)

Yes

qemu-system-aarch64 runs at over 100% while it’s hung

What do you mean by this? 100% CPU from the point of view of the host system? Have you tried giving the VM more CPUs and memory? I wonder if there's anything in the logs for envoy, redis etc that would say what is happening during a pause

Hi Adam

What do you mean by this? 100% CPU from the point of view of the host system?

Yes. When things are operating smoothly, qemu-system-aarch64 uses around 15% of System CPU as reported by the MacOS Activity Monitor.

Have you tried giving the VM more CPUs and memory?

I upping to 8GB RAM and 4 CPUs and the behavior didn’t change.

I wonder if there’s anything in the logs for envoy, redis etc that would say what is happening during a pause

container logs don’t say anything interesting. A thing I did find is that there are bursts of DNS queries getting logged in lima.ha.stderr.log. I think I’m reproducing a similar problem by doing the following. 1. Run a loop querying dns for host.docker.internal.

docker run --rm --name crashy-crashy -ti ubuntu:20.04 bash -c 'apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y dnsutils psmisc && while   true ; do dig host.docker.internal ; done'

2. Once that’s started, run another loop that kills

dig

over and over within the same container.

docker exec -ti crashy-crashy bash -c "while true ; do killall dig ; sleep .1 ; done"

If I let this sit for a bit, it seems to get qemu into the same state as the redis/envoy/ratelimit combo. I’m going to try this on rancher desktop 1.4 to see if it seems more stable. If 1.4 is more stable, I’ll create a github issue.

By any chance, did this problem first start happening after you upgraded to macOS 12.5.1? It seems that this update is messing with a bunch of different parts of RD

No, it’s been going on for a while. I tried RD 1.5.0 on macos 12.5.0 and it had this problem, reverted to RD 1.4.1, upgraded my mac, tried RD 1.5.1 and continue to see the issue.

Probably worth mentioning