Error with CSV.Chunks and InlineStrings
- Dominant language
- Julia
- Stars
- 312
- Forks
- 78
- PR merge metrics
- No merged PRs in 30d
Description
`Arrow.write("testcsv.arrow", CSV.Chunks("data.csv"), ntasks=2)`
with data.csv:
```
heading1,heading2,heading3
test1, test2, 3
test1, test2, 3
test1, test2, 3
test1, test2, 3
...
```
throws this error:
```
Arrow.write("testcsv.arrow", CSV.Chunks("data/test/data.csv", ntasks=2))
┌ Error: error writing arrow data on partition = 2
│ exception =
│ MethodError: no method matching pointer(::String7)
│ Closest candidates are:
│ pointer(::Random.UnsafeView) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:510
│ pointer(::WeakRefString) at XXX/.julia/packages/WeakRefStrings/AN9bY/src/WeakRefStrings.jl:85
│ pointer(::WeakRefString, ::Integer) at XXX/.julia/packages/WeakRefStrings/AN9bY/src/WeakRefStrings.jl:86
│ ...
│ Stacktrace:
│ [1] pointer(A::Arrow.ToList{UInt8, true, String7, Int32}, i::Int64)
│ @ Arrow ~/.julia/packages/Arrow/x6smw/src/arraytypes/list.jl:113
│ [2] getindex
│ @ ~/.julia/packages/Arrow/x6smw/src/arraytypes/list.jl:56 [inlined]
│ [3] getindex
│ @ ~/.julia/packages/Arrow/x6smw/src/arraytypes/dictencoding.jl:48 [inlined]
│ [4] iterate
│ @ ./abstractarray.jl:1096 [inlined]
│ [5] iterate
│ @ ./abstractarray.jl:1094 [inlined]
│ [6] iterate
│ @ ./iterators.jl:159 [inlined]
│ [7] iterate
│ @ ./iterators.jl:158 [inlined]
│ [8] iterate
│ @ ./generator.jl:44 [inlined]
│ [9] Dict{String7, Int8}(kv::Base.Generator{Base.Iterators.Enumerate{Arrow.DictEncoding{String7, Int8, Arrow.List{String7, Int32, Arrow.ToList{UInt8, true, String7, Int32}}}}, Arrow.var"#52#54"})
│ @ Base ./dict.jl:103
...
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the failure with Arrow.write and CSV.Chunks using ntasks=2, then start with arraytypes/list.jl and arraytypes/dictencoding.jl, the files named in the stack trace. Trace how String7 values reach pointer and determine the smallest correction supported by a regression test; done means the sample CSV writes successfully without the partition error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100