Feature: Add Arrow IPC Feather as a writable file format
- Dominant language
- Python
- Stars
- 5.9k
- Forks
- 600
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 38
Description
### Feature description
As per chat on Slack, I'd like to complement the excellent `arrow_stream` feature by implementing the [Arrow IPC Feather format (v2)](https://arrow.apache.org/docs/python/feather.html) as a target file format for Arrow processes to write directly to via stream.
### Are you a dlt user?
Yes, I'm already a dlt user.
### Use case
Parquet is a great format for long-term storage and analytical use cases, but my use of dlt is at the start of longer data pipelines and I need to be able to write to a file format more suited for speed and zero-copy memory mapping.
### Proposed solution
Implement the [Arrow IPC Feather v2 file format](https://arrow.apache.org/docs/python/feather.html) (`*.arrow`) as a new DataWriter and LoaderFileFormat implementation alongside its Parquet sibling.
### Related issues
_No response_
Contributor guide
Research direction
Start by locating the existing Parquet DataWriter and LoaderFileFormat implementations, then compare them with the arrow_stream feature. Implement the Arrow IPC Feather v2 (*.arrow) format as a writable target, and verify that Arrow processes can write to it directly via stream.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100