can't export to csv when arrow have array or json
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
my arrow data:
`pyarrow.Table
a: int64
b: list
child 0, item: int64
c: struct
child 0, a: string
child 1, b: int64
d: string
e: timestamp[us]
----
a: [[1,null]]
b: [[[1,2],[3,2]]]
c: [
-- is_valid: all not null
-- child 0 type: string
["x",null]
-- child 1 type: int64
[2,2]]
d: [["a"b","a,b"]]
e: [[2022-02-02 00:00:00.000000,9999-12-31 00:00:00.000000]]`
when `pa.csv.write_csv(dd.query("select a,b,c,d,e from ar").arrow(),'df_ar.csv',write_options=write_options)` raise error:
ArrowInvalid: Unsupported Type:list
ArrowInvalid: Unsupported Type:struct
and if value has ", for example a"b , then in csv will be "a""b", and no options to handle this problem
Contributor guide
Research direction
Reproduce the failure with the shown pyarrow.Table and pa.csv.write_csv call, including the list, struct, timestamp, and quoted-string values. Read the CSV writer entry point and its tests, then define completion as handling the reported nested types and producing correctly escaped CSV output with an available way to configure quoting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100