moby / moby/hyperkit

Assertion failed: (!REPLY_RING_EMPTY(sc)), function send_response_common, file src/lib/pci_virtio_sock.c, line 934.

Open
#213 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
3.7k
Forks
336
PR merge metrics
No merged PRs in 30d

Description

In https://github.com/docker/for-mac/issues/2872#issuecomment-386560236 a Docker container is shutdown and a set of AF_VSOCK connections are simultaneously closed:

2018-05-02 15:21:58.519471-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.37/containers/ce3d053af106a4f13790f82e721f88ece
8e85aa907f465afbc59d7f78cbc3676/wait?condition=next-exit
2018-05-02 15:21:58.520536-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.37/containers/ce3d053af106a4f13790f82e721f88ece8e85aa907f465afbc59d7f78cbc3676/attach?stderr=1&stdin=1&stdout=1&stream=1
2018-05-02 15:21:58.559605-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/777ba957b4f05c6b359a64c4dfe54383e8d3e3e0374c23d82dfd69284fd1333b/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.560708-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.560819-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.560880-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/7374b83bcea635423f15aaaebe0e68c52de4f783393f3f03c5d4c04bfe792f58/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.560971-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.561036-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.561133-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/6b12ec8ec282ed688811f66dd5dfc06fb1baa8ab0de37ead0bb4aa401dc285c6/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.561180-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.561217-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.561276-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/1d3c0a5d337ac23ff97ae6b561cf40bde538181e348800e89f7dc4fa3edc6877/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.561370-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.561438-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.561588-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/764caeb2552f7eeda49f6292fc104e0173fa8952376f17c99395d86d59aa1ef5/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.561650-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/989e12d766b90e09fb66cf34c2761737eb8dee757530dcd8eccb78d890f8443e/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.561720-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.561750-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.561910-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/7367d897b4239a174ef2d7bc5df9e3a58cfddbb7780de092ae779286515a35d6/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.561919-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.561974-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.562102-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/b7c958118e20cd6aebbe20081dd0bdbec34f6b1844bb10e607635d64caf8c935/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.562167-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.562189-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.562222-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/b43381a52e98a6f057e3eb3ddb22b11d7ba7df43ad0a3eaac30b2358da8e7fb5/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.562407-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.562431-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.562466-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/2734b5133d06521b91cbf12ec933d4e6710388dc58ca8224f91256c2b26b3066/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.565070-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/0ebb0d164dcb574a1c9293f4b89f6c5980b28ea6b7afbbfa02f61c48aafc5d62/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.565306-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/1a10c59d4fd2ff59249fd950bac9c6e98cda4bd2e63e8418cbd9121c95e1e649/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.565350-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/23febad602e8110ae73f53887e8c5c8547cbcc41c788705216426403a4cf57f8/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.565941-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/25549937a3743c5771ef8c7354be68090ad1e057302f723ed7a1decd3fe25391/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.566170-0700  localhost com.docker.driver.amd64-linux[19791]: Error forwarding raw stream from container: write unix s60->: write: broken pipe
2018-05-02 15:21:58.566198-0700  localhost com.docker.driver.amd64-linux[19791]: Error closing raw stream from container: close unix s60->: shutdown: socket is not connected
2018-05-02 15:21:58.566319-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/f4288843126045dc510da935b41319d9c261ed9b4c2636885d0b240c5224204e/attach?logs=0&stdout=1&stderr=1&stream=1
2018-05-02 15:21:58.566712-0700  localhost com.docker.driver.amd64-linux[19791]: proxy << POST /v1.25/containers/0d618ab342e4225c4ddd8aeebc45dd1497d9775aa2a30b35ed9588e7af30ddfd/attach?logs=0&stdout=1&stderr=1&stream=1

This triggers

2018-05-02 15:22:03.400455-0700  localhost com.docker.driver.amd64-linux[19791]: Assertion failed: (!REPLY_RING_EMPTY(sc)), function send_response_common, file src/lib/pci_virtio_sock.c, line 934.

the code has:

        /*
         * We have just incremented reply_prod above but we hold the
         * lock so the consumer cannot have caught us up. Hence for
         * the ring to appear empty it must actually have just overflowed.
         */
        assert(!REPLY_RING_EMPTY(sc));

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at src/lib/pci_virtio_sock.c around send_response_common line 934 and trace reply-ring handling during simultaneous AF_VSOCK connection closes. Reproduce the container shutdown sequence described in the report, then verify that the assertion no longer occurs during concurrent connection teardown.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.