apache / apache/arrow-julia

Error with CSV.Chunks and InlineStrings

Open
#275 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Julia
Stars
312
Forks
78
PR merge metrics
No merged PRs in 30d

Description

`Arrow.write("testcsv.arrow", CSV.Chunks("data.csv"), ntasks=2)`

with data.csv:

```
heading1,heading2,heading3
test1, test2, 3
test1, test2, 3
test1, test2, 3
test1, test2, 3
...
```
throws this error:

```
Arrow.write("testcsv.arrow", CSV.Chunks("data/test/data.csv", ntasks=2))
┌ Error: error writing arrow data on partition = 2
│ exception =
│ MethodError: no method matching pointer(::String7)
│ Closest candidates are:
│ pointer(::Random.UnsafeView) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:510
│ pointer(::WeakRefString) at XXX/.julia/packages/WeakRefStrings/AN9bY/src/WeakRefStrings.jl:85
│ pointer(::WeakRefString, ::Integer) at XXX/.julia/packages/WeakRefStrings/AN9bY/src/WeakRefStrings.jl:86
│ ...
│ Stacktrace:
│ [1] pointer(A::Arrow.ToList{UInt8, true, String7, Int32}, i::Int64)
│ @ Arrow ~/.julia/packages/Arrow/x6smw/src/arraytypes/list.jl:113
│ [2] getindex
│ @ ~/.julia/packages/Arrow/x6smw/src/arraytypes/list.jl:56 [inlined]
│ [3] getindex
│ @ ~/.julia/packages/Arrow/x6smw/src/arraytypes/dictencoding.jl:48 [inlined]
│ [4] iterate
│ @ ./abstractarray.jl:1096 [inlined]
│ [5] iterate
│ @ ./abstractarray.jl:1094 [inlined]
│ [6] iterate
│ @ ./iterators.jl:159 [inlined]
│ [7] iterate
│ @ ./iterators.jl:158 [inlined]
│ [8] iterate
│ @ ./generator.jl:44 [inlined]
│ [9] Dict{String7, Int8}(kv::Base.Generator{Base.Iterators.Enumerate{Arrow.DictEncoding{String7, Int8, Arrow.List{String7, Int32, Arrow.ToList{UInt8, true, String7, Int32}}}}, Arrow.var"#52#54"})
│ @ Base ./dict.jl:103
...
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with Arrow.write and CSV.Chunks using ntasks=2, then start with arraytypes/list.jl and arraytypes/dictencoding.jl, the files named in the stack trace. Trace how String7 values reach pointer and determine the smallest correction supported by a regression test; done means the sample CSV writes successfully without the partition error.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.