apache / apache/arrow-julia

Non-reproducible ReadOnlyMemoryError

Open
#218 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

This is not the kind of issue I like to write, because I have no minimal example with a reproducible error.
Still, perhaps the information I have tells you something:
```
data = DataFrame(Arrow.Table(input))
Matrix{Float64}(data[:,cols])
```
has in some cases lead to:
```
ERROR: TaskFailedException
nested task error: ReadOnlyMemoryError()
Stacktrace:
[1] copy
@ ./array.jl:349 [inlined]
[2] copy
@ ~/.julia/packages/Arrow/k23fl/src/arraytypes/primitive.jl:37 [inlined]
[3] _preprocess_column(col::Arrow.Primitive{Float64, Vector{Float64}}, len::Int64, copycols::Bool)
@ DataFrames ~/.julia/packages/DataFrames/nxjiD/src/dataframe/dataframe.jl:221
[4] (::DataFrames.var"#150#152"{Bool, Vector{AbstractVector{T} where T}, Int64})()
@ DataFrames ./threadingconstructs.jl:169
Stacktrace:
[1] sync_end(c::Channel{Any})
@ Base ./task.jl:369
[2] macro expansion
@ ./task.jl:388 [inlined]
[3] DataFrames.DataFrame(columns::Vector{AbstractVector{T} where T}, colindex::DataFrames.Index; copycols::Bool)
@ DataFrames ~/.julia/packages/DataFrames/nxjiD/src/dataframe/dataframe.jl:195
[4] manipulate(df::DataFrames.DataFrame, args::Vector{Int64}; copycols::Bool, keeprows::Bool, renamecols::Bool)
@ DataFrames ~/.julia/packages/DataFrames/nxjiD/src/abstractdataframe/selection.jl:1418
[5] manipulate(df::DataFrames.DataFrame, c::Vector{String}; copycols::Bool, keeprows::Bool, renamecols::Bool)
@ DataFrames ~/.julia/packages/DataFrames/nxjiD/src/abstractdataframe/selection.jl:1427
[6] #select#473
@ ~/.julia/packages/DataFrames/nxjiD/src/abstractdataframe/selection.jl:926 [inlined]
[7] getindex(df::DataFrames.DataFrame, row_ind::Colon, col_inds::Vector{String})
@ DataFrames ~/.julia/packages/DataFrames/nxjiD/src/dataframe/dataframe.jl:583
[8] infer(output::String; input::String, params::String)
@ App /app/App/src/infer.jl:6
```
I currently have no access to the arrow file or situation causing this problem, but I will investigate further.

Contributor guide

No contributing guide indexed for this repository

Research direction

The reported path enters App/src/infer.jl:6, then DataFrames selection and Arrow primitive copying; start there and inspect the exact Arrow.Table input and selected columns. Reproduce the ReadOnlyMemoryError with a minimal input before deciding whether the fault is in Arrow, DataFrames, or the caller. Done means a reproducible case, a localized cause, and a verified fix or documented upstream issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.