apache / apache/arrow

can't export to csv when arrow have array or json

Open
#14,152 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

my arrow data:
`pyarrow.Table
a: int64
b: list
child 0, item: int64
c: struct
child 0, a: string
child 1, b: int64
d: string
e: timestamp[us]
----
a: [[1,null]]
b: [[[1,2],[3,2]]]
c: [
-- is_valid: all not null
-- child 0 type: string
["x",null]
-- child 1 type: int64
[2,2]]
d: [["a"b","a,b"]]
e: [[2022-02-02 00:00:00.000000,9999-12-31 00:00:00.000000]]`

when `pa.csv.write_csv(dd.query("select a,b,c,d,e from ar").arrow(),'df_ar.csv',write_options=write_options)` raise error:

ArrowInvalid: Unsupported Type:list

ArrowInvalid: Unsupported Type:struct

and if value has ", for example a"b , then in csv will be "a""b", and no options to handle this problem

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the shown pyarrow.Table and pa.csv.write_csv call, including the list, struct, timestamp, and quoted-string values. Read the CSV writer entry point and its tests, then define completion as handling the reported nested types and producing correctly escaped CSV output with an available way to configure quoting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.