questdb / questdb/py-questdb-client
Prevent unbounded reads of Arrow stream error messages
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 71
- Forks
- 14
- Avg merge
- 1h 7m
- Merged PRs (30d)
- 1
Description
Summary
When an Arrow stream callback fails, the Python client asks the stream for an error message through ArrowArrayStream.get_last_error(). It currently treats the returned pointer as an unlimited C string.
A valid Arrow producer returns a stable, NUL-terminated message. A buggy or hand-written in-process producer may not. In that case, looking for the terminator can read beyond the message allocation and may crash the Python process while it is trying to report the original error.
This path is used after both get_schema() and get_next() failures.
Why it happens
The message is decoded directly:
stream_err.decode("utf-8", errors="replace")
Because the Arrow API returns a pointer without a length, decoding first scans memory until it finds a NUL byte. There is currently no upper bound on that scan.
Expected behavior
- Set and document a maximum accepted Arrow stream error length.
- Search or copy only within that limit.
- Return a safe diagnostic when the pointer is NULL or no terminator appears within the limit.
- Continue replacing invalid UTF-8 rather than failing while formatting the error.
- Copy the message before another stream callback runs or the stream is released, as required by the Arrow lifetime rules.
A bounded scan cannot make an arbitrary invalid pointer safe. The producer must still return readable, stable memory; the limit prevents a missing terminator from causing an unrestricted read.
Regression coverage
Use subprocess tests so a memory-safety regression cannot take down the main test process. Cover:
- a NULL error pointer;
- valid UTF-8;
- invalid UTF-8;
- a terminator at the configured boundary;
- no terminator within the configured limit.
This is separate from Arrow schema and array validation in c-questdb-client #195, which does not inspect callback error strings.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the Python/Cython path that calls ArrowArrayStream.get_last_error() after get_schema() and get_next() failures, including the stream_err.decode("utf-8", errors="replace") handling. Review the Arrow stream lifetime requirements and existing subprocess-test patterns first. Done means bounded handling for NULL, valid and invalid UTF-8, a boundary terminator, and a missing terminator, with the limit documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100