ydb-platform / ydb-platform/ydb-python-sdk
Flaky unit test: TestReaderReconnector.test_reconnect_on_repeatable_error (timeout on 3.12)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 102
- Forks
- 75
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
Summary
ydb/_topic_reader/topic_reader_asyncio_test.py::TestReaderReconnector::test_reconnect_on_repeatable_error fails intermittently in CI with a timeout. Recurring flake — has been seen on several unrelated PRs.
Failure
ydb._topic_common.test_helpers.WaitConditionError: Bad condition in test
ydb/_topic_common/test_helpers.py:67: WaitConditionError
The test drives ReaderReconnector through one repeatable error (Overloaded) and then a healthy stream, and waits via:
await wait_for_fast(reconnector.wait_message()) # topic_reader_asyncio_test.py:1561
wait_for_fast → wait_condition budgets 1s wall / 1000 loop iterations (test_helpers.py:46-67). When reconnector.wait_message() does not resolve within that budget the helper raises WaitConditionError. So this is a timing-sensitive timeout, not an assertion about reconnect behavior.
A teardown side-effect also shows up after the failure (likely secondary, from the aborted coroutine):
PytestUnraisableExceptionWarning: Exception ignored in: <coroutine object Queue.get ...>
RuntimeError: Event loop is closed
Environment / occurrence
- Observed on unit (3.12) only; unit 3.8–3.11, 3.13, 3.14 passed in the same run.
- Run: https://github.com/ydb-platform/ydb-python-sdk/actions/runs/28467229726/job/84370035632 (PR #849, whose diff is unrelated — touches
ydb/convert.pyresult-set rows). - Reproduces locally only occasionally; ~7s suite, fails at ~54% progress.
Hypothesis / directions to investigate
- The reconnect path (error stream → recreate → first message) occasionally needs more than the 1s / 1000-iteration budget under CI load, especially on 3.12's event-loop scheduling. Bumping the budget would only mask it.
- Worth checking whether
ReaderReconnectordoes extraawait asyncio.sleep(0)hops on the reconnect path on 3.12, or whether a backoff/retry delay leaks into the loop and eats the budget. - Confirm the
Event loop is closedteardown warning is purely secondary (GC of the parkedwait_forever/Queue.getcoroutine) and not contributing to the hang.
Workaround for now
Re-run the job; failure is intermittent and not a regression from the triggering PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ydb/_topic_reader/topic_reader_asyncio_test.py::TestReaderReconnector::test_reconnect_on_repeatable_error and the wait_for_fast/wait_condition implementation in ydb/_topic_common/test_helpers.py:46-67. Reproduce the timeout on Python 3.12, then trace ReaderReconnector through the repeatable-error and healthy-stream path, including the teardown warning. Done means the test reliably completes under CI load without masking the behavior by only increasing the timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100