mars-project / mars-project/mars
[BUG] ref count error
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
To help us reproducing this bug, please provide information below:
- Your Python version
- The version of Mars you use
- Versions of crucial packages, such as numpy, scipy and pandas
- Full stack of the error.
2022-03-25 12:31:18,361 ERROR processor.py:74 -- Unexpected error happens in <function TaskProcessor.decref_stage at 0x7fcfc7ebd0e0>
Traceback (most recent call last):
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 71, in inner
return await func(processor, *args, **kwargs)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 348, in decref_stage
await self._lifecycle_api.decref_chunks(decref_chunk_keys)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/api/oscar.py", line 134, in decref_chunks
return await self._lifecycle_tracker_ref.decref_chunks(chunk_keys)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 186, in send
return self._process_result_message(result)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 70, in _process_result_message
raise message.as_instanceof_cause()
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 590, in send
result = await self._run_coro(message.message_id, coro)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
return await coro
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 115, in __on_receive__
return await super().__on_receive__(message)
File "mars/oscar/core.pyx", line 371, in __on_receive__
raise ex
File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
return await self._handle_actor_result(result)
File "mars/oscar/core.pyx", line 250, in _handle_actor_result
task_result = await coros[0]
File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
async with self._lock:
File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
with debug_async_timeout('actor_lock_timeout',
File "mars/oscar/core.pyx", line 297, in mars.oscar.core._BaseActor._run_actor_async_generator
res = await gen.asend(res)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 89, in decref_chunks
to_remove_chunk_keys = self._get_remove_chunk_keys(chunk_keys)
File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 81, in _get_remove_chunk_keys
assert ref_count >= 0
mars.oscar.backends.message.ErrorMessage.as_instanceof_cause.<locals>._MarsError: [address=ray://mars_cluster_1648179994/0/0, pid=100894]
2022-03-25 12:31:18,365 INFO processor.py:839 -- Processor OgL9bjFjOwr7JIJVrelZS1gA finished.
- Minimized code to reproduce the error.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with services/task/supervisor/processor.py at decref_stage and services/lifecycle/supervisor/tracker.py at decref_chunks and _get_remove_chunk_keys, where the reported assertion occurs. Reproduce the failure using the provided stack trace context, then determine what input or lifecycle sequence produces the negative reference count. Done means the ref-count error is reproducible with a minimized case and the relevant test or behavior no longer triggers the assertion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100