Best way to add columns to an existing Feather file (>10GB) in terms of performance in python
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the usage question you have. Please include as many useful details as possible.
The current recommendation is: "Now that Feather supports zero-copy reads into Arrow, the preferred approach would be to zero-copy read the file, add a column, then write a new Feather file" from https://issues.apache.org/jira/browse/ARROW-1518.
This approach is quite costly. I'm very interested to know a more performant way. Thank you.
P.S. I assume that I don't need to write usage details, as this problem is quite common if large data is stored in this format and needs to be updated.
### Component(s)
Other
Contributor guide
Research direction
Start by reviewing the current zero-copy read, add-column, and rewrite recommendation cited from ARROW-1518, along with the Python Feather usage described in the issue. Investigate whether a supported, more performant approach exists for files over 10GB; done would be a documented recommendation or a clearly scoped implementation change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100