questdb / questdb/py-questdb-client
Prevent Arrow batch leaks when stream callbacks fail
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 71
- Forks
- 14
- Avg merge
- 1h 7m
- Merged PRs (30d)
- 1
Description
Summary
An Arrow stream callback can report an error after placing a batch in its output parameter. The Python client currently raises the stream error immediately and skips cleanup of that batch. If the batch has a release callback, its memory and other producer-owned resources can leak.
This affects the Arrow-native DataFrame ingestion path and is most likely to appear with custom or faulty Arrow stream producers.
Why it happens
The client follows this sequence in _capsule_consume_stream:
- Create an empty
ArrowArray. - Pass it to
ArrowArrayStream.get_next(). - If
get_next()returns a nonzero status, raise an exception.
Normal batches are released in a later try/finally block. The error branch runs before that block, so a failing callback that populated batch.release leaves the batch unreleased.
Expected behavior
- Define the ownership rule for a failed
get_next()call that leaves a non-NULLbatch.releasecallback. - Release any producer-owned batch exactly once before reporting the stream error.
- Preserve the current behavior when the callback leaves the output array untouched.
- Keep the original stream error available to the caller.
Regression coverage
Add a small forged Arrow stream with a counting release callback and cover:
get_next()fails without touching the output array;get_next()fails after populating the output array;- the populated batch is released once, with no leak or double release.
This is separate from Arrow schema and array validation in c-questdb-client #195: the failure occurs while reading the stream, before native preflight validation begins.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at _capsule_consume_stream, following how ArrowArrayStream.get_next() fills an ArrowArray and how normal batches are released. Add regression coverage with a forged Arrow stream for untouched and populated output arrays. Done means a populated batch is released exactly once before the original stream error is reported, while untouched output preserves current behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100