ydb-platform / ydb-platform/ydb-python-sdk
Async topic writer wedges under reconnect: 'cannot schedule new futures after shutdown' in _encode_data_inplace (compressing codec)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 102
- Forks
- 75
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
Summary
Under connection loss + reconnect (e.g. a node killed during chaos testing), the async topic writer using a compressing codec (GZIP) can get permanently wedged: write_with_ack never returns because the encode ThreadPoolExecutor was shut down while the writer is still live and re-encoding, so submitting the encode job raises RuntimeError('cannot schedule new futures after shutdown'). The error surfaces only as an unretrieved background future, and the awaiting write_with_ack hangs indefinitely (no timeout of its own).
Traceback (observed)
grpc/aio/_call.py ... raise asyncio.CancelledError() # stream cancelled by the node kill
asyncio.exceptions.CancelledError
ERROR Future exception was never retrieved
future: <Future finished exception=RuntimeError('cannot schedule new futures after shutdown')>
Traceback (most recent call last):
File ".../ydb/_topic_writer/topic_writer_asyncio.py", line 128, in write_with_ack
results = [f.result() for f in futures]
File ".../ydb/_topic_writer/topic_writer_asyncio.py", line 578, in _encode_loop
await self._encode_data_inplace(batch_codec, messages)
File ".../ydb/_topic_writer/topic_writer_asyncio.py", line 599, in _encode_data_inplace
encoded_data_futures = eventloop.run_in_executor(self._encode_executor, encoder_function, ...)
File ".../concurrent/futures/thread.py", line 167, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Conditions
ydb.aiotopic writer (topic_client.writer(...)), compressing codec (TopicCodec.GZIP).- Connection drop + reconnect (chaos: a
CancelledErrorpropagates from the gRPC stream). - After that,
_encode_data_inplacesubmits toself._encode_executor, which has already been shut down →RuntimeError.
Impact
write_with_ack never completes → the writer silently hangs. In an application that awaits write_with_ack without its own timeout, the whole writer path stalls. Same class of async-topic-writer reconnect-lifecycle issue as the earlier WriterAsyncIOStream.create() thread leak (PR #845).
Root cause (likely)
self._encode_executor is shut down while the writer is still alive and re-encoding after a reconnect (executor lifecycle not tied correctly to the writer's live/reconnect state).
Workaround
Use TopicCodec.RAW — _encode_data_inplace returns early for RAW and never touches the executor.
How it was found
YDB Python SLO chaos testing (PR #851): the async topic workload silently stalled at ~half the run; the baseline container's delivery metrics went N/A mid-run. The SLO harness was hardened (timeouts + writer/reader recreate + RAW) to not hang, but the underlying SDK writer bug remains.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in ydb/_topic_writer/topic_writer_asyncio.py, especially write_with_ack, _encode_loop, and _encode_data_inplace, and trace how self._encode_executor is created and shut down around the gRPC reconnect path. Reproduce the connection-loss scenario described in the issue or the chaos workload from PR #851. Done means a compressing-codec write_with_ack completes or reports an error after reconnect instead of hanging or leaving an unretrieved executor exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100