lablup / lablup/backend.ai

No container ID in DB when krunner is failed to start

Open
#322 0 comments 0 reactions 1 assignee Claimed by @kmkwon94 View on GitHub
comp:agent
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

On a customer site, we found a bug that a specific user's session is always stuck at the PREPARING state for a specific group of images. It turns out that the problem was caused by failing to start kernel runner inside a container, in this case, the location of python binary is mis-indicated. And, the container ID is not recorded in the DB even though the container is actually created (but not kernel runner), so we cannot destroy the container since no container ID is found from the DB.

I suggest that we have to fix this issue:

Once a container is created, its container ID should be recorded in the DB, regardless of the kernel runner's status.
If kernel runner is failed to start, we can mark the container's status as ERROR or just automatically destroy it.

To easily reproduce this phenomenon, just add `raise` in the `_init_jupyter_kernel` method in `kernel/base.py` (image below).

![image](https://user-images.githubusercontent.com/7539358/145742552-0e8b3905-bff3-4fe0-b77c-412d59f62e06.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.