apache / apache/arrow

Best way to add columns to an existing Feather file (>10GB) in terms of performance in python

Open
#35,689 3 comments 0 reactions 0 assignees View on GitHub
Component: Other Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the usage question you have. Please include as many useful details as possible.

The current recommendation is: "Now that Feather supports zero-copy reads into Arrow, the preferred approach would be to zero-copy read the file, add a column, then write a new Feather file" from https://issues.apache.org/jira/browse/ARROW-1518.

This approach is quite costly. I'm very interested to know a more performant way. Thank you.

P.S. I assume that I don't need to write usage details, as this problem is quite common if large data is stored in this format and needs to be updated.

### Component(s)

Other

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the current zero-copy read, add-column, and rewrite recommendation cited from ARROW-1518, along with the Python Feather usage described in the issue. Investigate whether a supported, more performant approach exists for files over 10GB; done would be a documented recommendation or a clearly scoped implementation change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.