dlt-hub / dlt-hub/dlt

Feature: Add Arrow IPC Feather as a writable file format

Open
#3,452 1 comment 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
5.9k
Forks
600
Avg merge
1d 14h
Merged PRs (30d)
38

Description

### Feature description

As per chat on Slack, I'd like to complement the excellent `arrow_stream` feature by implementing the [Arrow IPC Feather format (v2)](https://arrow.apache.org/docs/python/feather.html) as a target file format for Arrow processes to write directly to via stream.

### Are you a dlt user?

Yes, I'm already a dlt user.

### Use case

Parquet is a great format for long-term storage and analytical use cases, but my use of dlt is at the start of longer data pipelines and I need to be able to write to a file format more suited for speed and zero-copy memory mapping.

### Proposed solution

Implement the [Arrow IPC Feather v2 file format](https://arrow.apache.org/docs/python/feather.html) (`*.arrow`) as a new DataWriter and LoaderFileFormat implementation alongside its Parquet sibling.

### Related issues

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Parquet DataWriter and LoaderFileFormat implementations, then compare them with the arrow_stream feature. Implement the Arrow IPC Feather v2 (*.arrow) format as a writable target, and verify that Arrow processes can write to it directly via stream.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.