apache / apache/arrow

[C++] Can't use union in read_json

Open
#36,509 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: usage
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the usage question you have. Please include as many useful details as possible.

I'm trying to use pyarrow to read a json file into a pyarrow data set.

I am having trouble with the union type though.

applicants.large.jsonl
```
{"Type":"insert","Key":[2],"SeqNo":2,"Timestamp":2,"Fields":[{"Name":"added_id","Value":3},{"Name":"row_key","Value":"/XX"},{"Name":"column_key","Value":"syz"},{"Name":"ref_key","Value":2},{"Name":"updated","Value":"2010-12-08T02:15:07.664543Z"}]}
```

For reference, this is the formatted JSON:
```
{
"Type": "insert",
"Key": [
2
],
"SeqNo": 2,
"Timestamp": 2,
"Fields": [
{
"Name": "added_id",
"Value": 3
},
{
"Name": "row_key",
"Value": "/XX"
},
{
"Name": "column_key",
"Value": "syz"
},
{
"Name": "ref_key",
"Value": 2
},
{
"Name": "updated",
"Value": "2010-12-08T02:15:07.664543Z"
}
]
}
```

Code:
```
data = "applicants.large.jsonl"

schema = pa.schema([
('Type', pa.string()),
('Fields', pa.list_(pa.struct([
('Name', pa.string()),
('Value', pa.union([
pa.field("int64", pa.int64()),
pa.field("string", pa.string()),
], mode='sparse'))
])))
])

table = pa.json.read_json(data, parse_options=pa.json.ParseOptions(explicit_schema=schema))
```

Error:
```

ArrowNotImplementedError: JSON conversion to sparse_union is not supported
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.