cytomining / cytomining/CytoTable
Consider use of `pyarrow.lib.RecordBatchReader` for internal streamable data handling
Open
- Dominant language
- Python
- Stars
- 21
- Forks
- 6
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 6
Description
DuckDB enables the use of `pyarrow.lib.RecordBatchReader` returns which could help us speed up or optimize certain internal processing for CytoTable. We could consider this as time goes on to help with performance improvements.
References #369
Contributor guide
Research direction
Start by reviewing issue #369 and the current internal data-processing paths in CytoTable. Determine where DuckDB returns could be exposed as pyarrow.lib.RecordBatchReader objects, then define a representative performance comparison. Done requires a scoped decision or implementation with evidence, since this issue does not name files, tests, or a specific expected change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100