lablup / lablup/backend.ai

DockerError network not found when deleting multi-kernel session

Open
#1,220 0 comments 0 reactions 1 assignee Claimed by @fregataa View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

### What Operating System(s) are you seeing this problem on?

macOS (Apple Silicon)

### Backend.AI version

main

### Describe the bug

DockerError(404, 'network not found') occurs when deleting multi kernel sessions.

```Java
2023-04-11 16:52:25.683 DEBUG ai.backend.agent.server [32323] rpc::destroy_local_network(name:bai-singlenode-HASH)
2023-04-11 16:52:25.684 ERROR ai.backend.agent.server [32323] unexpected error
Traceback (most recent call last):
File "/Users/sh/bai-dev/src/ai/backend/agent/server.py", line 157, in _inner
return await meth(
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/src/ai/backend/agent/server.py", line 133, in _inner
return await meth(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/src/ai/backend/agent/server.py", line 623, in destroy_local_network
return await self.agent.destroy_local_network(network_name)
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/src/ai/backend/agent/docker/agent.py", line 1439, in destroy_local_network
network = await docker.networks.get(network_name)
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/dist/export/python/virtualenvs/python-default/3.11.2/lib/python3.11/site-packages/aiodocker/networks.py", line 21, in get
data = await self.docker._query_json(
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/dist/export/python/virtualenvs/python-default/3.11.2/lib/python3.11/site-packages/aiodocker/docker.py", line 300, in _query_json
async with self._query(
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/dist/export/python/virtualenvs/python-default/3.11.2/lib/python3.11/site-packages/aiodocker/utils.py", line 309, in __aenter__
resp = await self._coro
^^^^^^^^^^^^^^^^^
File "/Users/sh/bai-dev/dist/export/python/virtualenvs/python-default/3.11.2/lib/python3.11/site-packages/aiodocker/docker.py", line 275, in _do_query
raise DockerError(response.status, json.loads(what.decode("utf8")))
^^^^^^^^^^^^^^^^^
aiodocker.exceptions.DockerError: DockerError(404, 'network bai-singlenode-HASH not found')

```

### To Reproduce

Just simply create a multi-kernel session and destroy it.

### Expected Behavior

the number of attempt to delete the docker network should be one, not multiple times.And although such error occurs, we should handle the error.

### Anything else?

_No response_

JIRA Issue: BA-212

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.