apache / apache/arrow

[C++][Python] Metadata from C data interface is not valid utf8

Open
#20,107 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

While trying to roundtrip an extension from schema.metadata (see ARROW-13855 for details), I got invalid utf8, which imo goes against

> A binary string describing the type’s metadata [1]

Specifically, a field

field = pyarrow.field("aa", UuidType())

contains the following:

```
key len: 20
key: "ARROW:extension:name"
value len: 23
value: "arrow.py_extension_type"
key len: 24
key: "ARROW:extension:metadata"
value len: 28
```

with the value's data for this key being:

```
[128, 3, 99, 116, 101, 115, 116, 95, 115, 113, 108, 10, 85, 117, 105, 100, 84, 121, 112, 101, 10, 113, 0, 41, 82, 113, 1, 46]
```

This is not a valid utf8 (see e.g. https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=02b67658b3cddf8dc095bc9750fa7032).

Maybe I am reading the values incorrectly? (null point?)

[1] https://arrow.apache.org/docs/format/CDataInterface.html#c.ArrowSchema.metadata

**Reporter**: [Jorge Leitão](https://issues.apache.org/jira/browse/ARROW-15613) / @jorgecarleitao

**Note**: *This issue was originally created as [ARROW-15613](https://issues.apache.org/jira/browse/ARROW-15613). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the ArrowSchema.metadata entry point in the C Data Interface and the pyarrow.field(..., UuidType()) example; compare the documented metadata contract with the reported byte sequence. Done means the metadata representation is confirmed and the round-trip behavior is corrected or the reading clarified under that contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
api, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.