ydb-platform / ydb-platform/ydb-python-sdk

Flaky unit test: TestReaderReconnector.test_reconnect_on_repeatable_error (timeout on 3.12)

Open
#850 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
102
Forks
75
Avg merge
1d 23h
Merged PRs (30d)
13

Description

Summary

ydb/_topic_reader/topic_reader_asyncio_test.py::TestReaderReconnector::test_reconnect_on_repeatable_error fails intermittently in CI with a timeout. Recurring flake — has been seen on several unrelated PRs.

Failure

ydb._topic_common.test_helpers.WaitConditionError: Bad condition in test
ydb/_topic_common/test_helpers.py:67: WaitConditionError

The test drives ReaderReconnector through one repeatable error (Overloaded) and then a healthy stream, and waits via:

await wait_for_fast(reconnector.wait_message())   # topic_reader_asyncio_test.py:1561

wait_for_fastwait_condition budgets 1s wall / 1000 loop iterations (test_helpers.py:46-67). When reconnector.wait_message() does not resolve within that budget the helper raises WaitConditionError. So this is a timing-sensitive timeout, not an assertion about reconnect behavior.

A teardown side-effect also shows up after the failure (likely secondary, from the aborted coroutine):

PytestUnraisableExceptionWarning: Exception ignored in: <coroutine object Queue.get ...>
RuntimeError: Event loop is closed

Environment / occurrence

Hypothesis / directions to investigate

  • The reconnect path (error stream → recreate → first message) occasionally needs more than the 1s / 1000-iteration budget under CI load, especially on 3.12's event-loop scheduling. Bumping the budget would only mask it.
  • Worth checking whether ReaderReconnector does extra await asyncio.sleep(0) hops on the reconnect path on 3.12, or whether a backoff/retry delay leaks into the loop and eats the budget.
  • Confirm the Event loop is closed teardown warning is purely secondary (GC of the parked wait_forever/Queue.get coroutine) and not contributing to the hang.

Workaround for now

Re-run the job; failure is intermittent and not a regression from the triggering PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ydb/_topic_reader/topic_reader_asyncio_test.py::TestReaderReconnector::test_reconnect_on_repeatable_error and the wait_for_fast/wait_condition implementation in ydb/_topic_common/test_helpers.py:46-67. Reproduce the timeout on Python 3.12, then trace ReaderReconnector through the repeatable-error and healthy-stream path, including the teardown warning. Done means the test reliably completes under CI load without masking the behavior by only increasing the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.