apache / apache/arrow-julia

write fails on large dataframes

Open
#326 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

hi :)

as indicated in the Tables.jl docs, for large dataframes (>10k columns), Tables.Schema stores its values in
.storenames and .storedtypes instead of the type parameters. This needs to be taken into account in,
eg. makeschema in write.jl and possibly other places.

EDIT: I just added if isnothing(names) names_ = sch.names else names_ = names end
(and replace names with names_ below) in my checkout and that fixed writing for me.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with makeschema in write.jl and the Tables.jl documentation on Tables.Schema for large dataframes. Check how .storenames and .storedtypes are handled when schema values are not in the type parameters, then verify that writing dataframes with more than 10k columns succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.