questdb / questdb/py-questdb-client

Prevent Arrow batch leaks when stream callbacks fail

Open
#147 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
71
Forks
14
Avg merge
1h 7m
Merged PRs (30d)
1

Description

Summary

An Arrow stream callback can report an error after placing a batch in its output parameter. The Python client currently raises the stream error immediately and skips cleanup of that batch. If the batch has a release callback, its memory and other producer-owned resources can leak.

This affects the Arrow-native DataFrame ingestion path and is most likely to appear with custom or faulty Arrow stream producers.

Why it happens

The client follows this sequence in _capsule_consume_stream:

  1. Create an empty ArrowArray.
  2. Pass it to ArrowArrayStream.get_next().
  3. If get_next() returns a nonzero status, raise an exception.

Normal batches are released in a later try/finally block. The error branch runs before that block, so a failing callback that populated batch.release leaves the batch unreleased.

Expected behavior

  • Define the ownership rule for a failed get_next() call that leaves a non-NULL batch.release callback.
  • Release any producer-owned batch exactly once before reporting the stream error.
  • Preserve the current behavior when the callback leaves the output array untouched.
  • Keep the original stream error available to the caller.

Regression coverage

Add a small forged Arrow stream with a counting release callback and cover:

  • get_next() fails without touching the output array;
  • get_next() fails after populating the output array;
  • the populated batch is released once, with no leak or double release.

This is separate from Arrow schema and array validation in c-questdb-client #195: the failure occurs while reading the stream, before native preflight validation begins.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at _capsule_consume_stream, following how ArrowArrayStream.get_next() fills an ArrowArray and how normal batches are released. Add regression coverage with a forged Arrow stream for untouched and populated output arrays. Done means a populated batch is released exactly once before the original stream error is reported, while untouched output preserves current behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.