lablup / lablup/backend.ai

Kernel get-or-create error when agent is temporarily lost

Open
#151 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

```Java
2018-10-05 05:28:57 WARNING ai.backend.manager.registry [Process-0] agent i-033... heartbeat timeout detected.
2018-10-05 05:28:58 INFO ai.backend.gateway.kernel [Process-1] GET_OR_CREATE (u:AKIA..., lang:python-tensorflow, token:a4d36...)
2018-10-05 05:28:58 ERROR ai.backend.gateway.kernel [Process-1] GET_OR_CREATE: unexpected error!
Traceback (most recent call last):
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 330, in get_or_create_session
kern = await self.get_session(sess_id, access_key)
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 274, in get_session
raise KernelNotFound
ai.backend.gateway.exceptions.KernelNotFound: (404, 'Not Found', 'https://api.backend.ai/probs/kernel-not-found')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/devops/backend.ai-manager/ai/backend/gateway/kernel.py", line 106, in create
conn=conn)
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 340, in get_or_create_session
conn=conn)
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 446, in create_session
result = await conn.execute(query)
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/utils.py", line 70, in __await__
resp = yield from self._coro
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/sa/connection.py", line 110, in _execute
yield from cursor.execute(str(compiled), post_processed_params[0])
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/cursor.py", line 114, in execute
yield from self._conn._poll(waiter, timeout)
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/connection.py", line 238, in _poll
yield from asyncio.wait_for(self._waiter, timeout, loop=self._loop)
File "/home/devops/.pyenv/versions/3.6.4/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/connection.py", line 135, in _ready
state = self._conn.poll()
psycopg2.IntegrityError: duplicate key value violates unique constraint "ix_kernels_unique_sess_token"
DETAIL: Key (access_key, sess_id)=(AKIA..., a4d36...) already exists.
```

This happens when a user tries to reuse a kernel on a temporarily lost agent after previous kernel launches on that agent has succeeded.

JIRA Issue: BA-331

Contributor guide

Open the contributing guide

Research direction

Start in ai/backend/manager/registry.py at get_or_create_session, get_session, and create_session, then trace the call from ai/backend/gateway/kernel.py create. Inspect how the ix_kernels_unique_sess_token constraint is reached after an agent heartbeat timeout. Done means the reported get-or-create scenario no longer ends with the duplicate-key IntegrityError.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.