[C++][Python] Metadata from C data interface is not valid utf8
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
While trying to roundtrip an extension from schema.metadata (see ARROW-13855 for details), I got invalid utf8, which imo goes against
> A binary string describing the type’s metadata [1]
Specifically, a field
field = pyarrow.field("aa", UuidType())
contains the following:
```
key len: 20
key: "ARROW:extension:name"
value len: 23
value: "arrow.py_extension_type"
key len: 24
key: "ARROW:extension:metadata"
value len: 28
```
with the value's data for this key being:
```
[128, 3, 99, 116, 101, 115, 116, 95, 115, 113, 108, 10, 85, 117, 105, 100, 84, 121, 112, 101, 10, 113, 0, 41, 82, 113, 1, 46]
```
This is not a valid utf8 (see e.g. https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=02b67658b3cddf8dc095bc9750fa7032).
Maybe I am reading the values incorrectly? (null point?)
[1] https://arrow.apache.org/docs/format/CDataInterface.html#c.ArrowSchema.metadata
**Reporter**: [Jorge Leitão](https://issues.apache.org/jira/browse/ARROW-15613) / @jorgecarleitao
**Note**: *This issue was originally created as [ARROW-15613](https://issues.apache.org/jira/browse/ARROW-15613). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the ArrowSchema.metadata entry point in the C Data Interface and the pyarrow.field(..., UuidType()) example; compare the documented metadata contract with the reported byte sequence. Done means the metadata representation is confirmed and the round-trip behavior is corrected or the reading clarified under that contract.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- api, data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 38/100