moby / moby/hyperkit

virtio-vsock can't handle a high connection rate

Open
#105 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
3.7k
Forks
336
PR merge metrics
No merged PRs in 30d

Description

This Docker for Mac bug report: https://github.com/docker/for-mac/issues/1417 contains an interesting set of repro steps:

dd if=/dev/zero of=zero.txt bs=1000 count=1
docker run -dit -p 8888:80 --name apache -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4
for i in {1..100}; do curl -s -S -0 --no-keepalive "http://127.0.0.1:8888/zero.txt?[1-100]" > /dev/null & done; wait

I believe this runs 100 concurrent instances of curl which each perform 100 TCP opens and closes in series. If I run this program inside Moby (where 127.0.0.1:8888 is also bound by the user space proxy) then it works fine.

If I run it on the Mac, I get a bunch of errors about dropped connections.

The difference between the 2 configurations is on the Mac, vpnkit is listening on the TCP port and calling connect to the Unix domain socket used to establish virtio-vsock connections. It then forwards to this Unix domain socket.

I don't see any sign of hitting the vpnkit maximum connection limit (although if the fds weren't closed promptly then the host could be temporarily running out of fds). Ideally I'd like to run a test on the Mac which bypassed the virtio-vsock interface but I don't have anything set up atm (perhaps using vmnet.framework would do the trick?)

I suspect that the virtio-vsock Unix domain socket interface is unable to cope with the high connect/close rate. If this is a fundamental (or hard to fix) problem then we could rewrite the proxy to multiplex over one virtio-vsock connection.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reproduction commands from the linked Docker for Mac report, comparing the Moby path with the Mac path through vpnkit and the Unix domain socket. Investigate the virtio-vsock connect/close rate and determine whether the dropped connections come from that interface; done means a confirmed cause and a scoped fix or a justified multiplexing design.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
networking, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.