mars-project / mars-project/mars

[BUG] ref count error

Open
#2,867 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
To help us reproducing this bug, please provide information below:

  1. Your Python version
  2. The version of Mars you use
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
2022-03-25 12:31:18,361	ERROR processor.py:74 -- Unexpected error happens in <function TaskProcessor.decref_stage at 0x7fcfc7ebd0e0>
Traceback (most recent call last):
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 71, in inner
    return await func(processor, *args, **kwargs)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/task/supervisor/processor.py", line 348, in decref_stage
    await self._lifecycle_api.decref_chunks(decref_chunk_keys)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/api/oscar.py", line 134, in decref_chunks
    return await self._lifecycle_tracker_ref.decref_chunks(chunk_keys)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 186, in send
    return self._process_result_message(result)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/context.py", line 70, in _process_result_message
    raise message.as_instanceof_cause()
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 590, in send
    result = await self._run_coro(message.message_id, coro)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/backends/pool.py", line 343, in _run_coro
    return await coro
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/oscar/api.py", line 115, in __on_receive__
    return await super().__on_receive__(message)
  File "mars/oscar/core.pyx", line 371, in __on_receive__
    raise ex
  File "mars/oscar/core.pyx", line 365, in mars.oscar.core._BaseActor.__on_receive__
    return await self._handle_actor_result(result)
  File "mars/oscar/core.pyx", line 250, in _handle_actor_result
    task_result = await coros[0]
  File "mars/oscar/core.pyx", line 293, in mars.oscar.core._BaseActor._run_actor_async_generator
    async with self._lock:
  File "mars/oscar/core.pyx", line 294, in mars.oscar.core._BaseActor._run_actor_async_generator
    with debug_async_timeout('actor_lock_timeout',
  File "mars/oscar/core.pyx", line 297, in mars.oscar.core._BaseActor._run_actor_async_generator
    res = await gen.asend(res)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 89, in decref_chunks
    to_remove_chunk_keys = self._get_remove_chunk_keys(chunk_keys)
  File "/home/admin/ray-pack/tmp/job/b8050080/pyenv/lib/python3.7/site-packages/mars/services/lifecycle/supervisor/tracker.py", line 81, in _get_remove_chunk_keys
    assert ref_count >= 0
mars.oscar.backends.message.ErrorMessage.as_instanceof_cause.<locals>._MarsError: [address=ray://mars_cluster_1648179994/0/0, pid=100894] 
2022-03-25 12:31:18,365	INFO processor.py:839 -- Processor OgL9bjFjOwr7JIJVrelZS1gA finished.
  1. Minimized code to reproduce the error.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with services/task/supervisor/processor.py at decref_stage and services/lifecycle/supervisor/tracker.py at decref_chunks and _get_remove_chunk_keys, where the reported assertion occurs. Reproduce the failure using the provided stack trace context, then determine what input or lifecycle sequence produces the negative reference count. Done means the ref-count error is reproducible with a minimized case and the relevant test or behavior no longer triggers the assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.