Kernel get-or-create error when agent is temporarily lost
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 15h 13m
- Merged PRs (30d)
- 368
Description
```Java
2018-10-05 05:28:57 WARNING ai.backend.manager.registry [Process-0] agent i-033... heartbeat timeout detected.
2018-10-05 05:28:58 INFO ai.backend.gateway.kernel [Process-1] GET_OR_CREATE (u:AKIA..., lang:python-tensorflow, token:a4d36...)
2018-10-05 05:28:58 ERROR ai.backend.gateway.kernel [Process-1] GET_OR_CREATE: unexpected error!
Traceback (most recent call last):
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 330, in get_or_create_session
kern = await self.get_session(sess_id, access_key)
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 274, in get_session
raise KernelNotFound
ai.backend.gateway.exceptions.KernelNotFound: (404, 'Not Found', 'https://api.backend.ai/probs/kernel-not-found')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/devops/backend.ai-manager/ai/backend/gateway/kernel.py", line 106, in create
conn=conn)
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 340, in get_or_create_session
conn=conn)
File "/home/devops/backend.ai-manager/ai/backend/manager/registry.py", line 446, in create_session
result = await conn.execute(query)
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/utils.py", line 70, in __await__
resp = yield from self._coro
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/sa/connection.py", line 110, in _execute
yield from cursor.execute(str(compiled), post_processed_params[0])
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/cursor.py", line 114, in execute
yield from self._conn._poll(waiter, timeout)
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/connection.py", line 238, in _poll
yield from asyncio.wait_for(self._waiter, timeout, loop=self._loop)
File "/home/devops/.pyenv/versions/3.6.4/lib/python3.6/asyncio/tasks.py", line 358, in wait_for
return fut.result()
File "/home/devops/venv/lib/python3.6/site-packages/aiopg/connection.py", line 135, in _ready
state = self._conn.poll()
psycopg2.IntegrityError: duplicate key value violates unique constraint "ix_kernels_unique_sess_token"
DETAIL: Key (access_key, sess_id)=(AKIA..., a4d36...) already exists.
```
This happens when a user tries to reuse a kernel on a temporarily lost agent after previous kernel launches on that agent has succeeded.
JIRA Issue: BA-331
Contributor guide
Research direction
Start in ai/backend/manager/registry.py at get_or_create_session, get_session, and create_session, then trace the call from ai/backend/gateway/kernel.py create. Inspect how the ix_kernels_unique_sess_token constraint is reached after an agent heartbeat timeout. Done means the reported get-or-create scenario no longer ends with the duplicate-key IntegrityError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100