questdb / questdb/py-questdb-client
Use each Arrow stream's own schema when ingesting DataFrame slices
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 71
- Forks
- 14
- Avg merge
- 1h 7m
- Merged PRs (30d)
- 1
Description
Summary
When a large DataFrame is split into smaller row slices, each slice is exported as its own Arrow stream. The Python client fetches the schema from the first stream and then reuses it for every later slice.
Arrow guarantees that batches from one stream share that stream's schema. It does not guarantee that two separately exported streams have identical schemas. Common pyarrow, pandas, and Polars slices normally do, but a custom producer can return a different field type or metadata for a later slice. The client may then interpret that batch using the wrong schema.
Why it happens
The DataFrame ingestion path keeps one ArrowSchema for the whole operation. _capsule_consume_stream calls get_schema() only while that schema has no release callback. Once the first slice fills it, later slice streams are consumed without asking them for their own schema.
This can cause a later batch to be rejected for a confusing reason or, worse, to be classified and imported according to stale type or metadata information.
Expected behavior
For each separately exported Arrow stream, either:
- fetch and use that stream's own schema; or
- fetch it, verify that it is equivalent to the cached schema, and reject a mismatch before importing any batch.
Keep each schema alive until all batches from its stream have been released, and release every schema exactly once. Equivalent schemas may still share derived planning state so the normal multi-slice path does not become unnecessarily expensive.
Regression coverage
Add a custom sliceable producer and cover:
- two slices with equivalent schemas continue to ingest normally;
- the second slice changes a field format;
- the second slice changes relevant field metadata;
- mismatches are rejected deterministically instead of using the first slice's schema;
- schema and batch release callbacks run exactly once.
This issue is separate from the structural Arrow validation in c-questdb-client #195. Native validation checks whether one schema and array are structurally safe to import; it cannot know that the schema came from a different exported stream.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at _capsule_consume_stream, where the issue says the first stream's schema is cached, and trace schema and batch lifetime handling for each slice. Add regression coverage with a custom sliceable producer for equivalent schemas, field-format and metadata mismatches, and exactly-once release callbacks; done means mismatches are rejected deterministically without stale-schema ingestion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100