Feather file with compression and larger than RAM
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
Last time I checked, `mmap` breaks down for files with compression. This is understandable because the compressed buffers clearly can't be re-interpreted without inflation.
But the larger the file is more likely to be compressed, can we decompress only a single "row group" (and only the relevant columns, of course) on the fly yet? -- this is for the case when a user is doing per-row iteration
if user access the table by `tbl[range, range]`, then clearly, we might need to read more than one row-group and chop off head/tails depending on where the overlap is
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how compressed Feather files are handled through mmap and how per-row iteration accesses a table. Then examine the tbl[range, range] path and the row-group and column boundaries described in the issue. Done means compressed files larger than RAM can decompress only the needed row groups and columns for iteration and ranged table access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100