mars-project / mars-project/mars
[BUG] ref_count < 0 in get_remove_chunk_keys
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
To help us reproducing this bug, please provide information below:
1. Your Python version: python3.7.9
2. The version of Mars you use: mars 0.8.1
3. Versions of crucial packages, such as numpy, scipy and pandas: numpy==1.17.3 scipy==1.5.0 pands==1.3.0
4. Full stack of the error.
```
Traceback (most recent call last):
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
return await self._handle_actor_result(result)
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
task_result = await coros[0]
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
async with self._lock:
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
with debug_async_timeout('actor_lock_timeout',
File "mars/oscar/core.pyx", line 297, in mars.oscar.core._BaseActor._run_actor_async_generator
res = await gen.asend(res)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 89, in decref_chunks
to_remove_chunk_keys = self._get_remove_chunk_keys(chunk_keys)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 81, in _get_remove_chunk_keys
assert ref_count >= 0
AssertionError
2022-03-01 11:08:57,466 ERROR processor.py:74 -- Unexpected error happens in
Traceback (most recent call last):
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 71, in inner
return await func(processor, *args, **kwargs)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 358, in decref_stage
await self._lifecycle_api.decref_chunks(decref_chunk_keys)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/api/oscar.py", line 134, in decref_chunks
return await self._lifecycle_tracker_ref.decref_chunks(chunk_keys)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 186, in send
return self._process_result_message(result)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 70, in _process_result_message
raise message.as_instanceof_cause()
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 590, in send
result = await self._run_coro(message.message_id, coro)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
return await coro
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 115, in __on_receive__
return await super().__on_receive__(message)
File "mars/oscar/core.pyx", line 371, in __on_receive__
raise ex
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
return await self._handle_actor_result(result)
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
task_result = await coros[0]
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
async with self._lock:
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
with debug_async_timeout('actor_lock_timeout',
File "mars/oscar/core.pyx", line 297, in mars.oscar.core._BaseActor._run_actor_async_generator
res = await gen.asend(res)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 89, in decref_chunks
to_remove_chunk_keys = self._get_remove_chunk_keys(chunk_keys)
File "/home/admin/ray-pack/tmp/job/85010080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 81, in _get_remove_chunk_keys
assert ref_count >= 0
mars.oscar.backends.message.ErrorMessage.as_instanceof_cause.._MarsError: [address=ray://mars_cluster_1646102907/0/0, pid=257926]
```
Contributor guide
Research direction
Start with mars/services/lifecycle/supervisor/tracker.py, especially _get_remove_chunk_keys and decref_chunks, using the reported assertion and stack trace as the entry point. Trace how chunk reference counts are updated during decref operations; done means the reported flow no longer reaches ref_count < 0 without triggering the assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100