apache / apache/arrow

[Format] Encapsulated message format metadata_size ambiguity.

Open
#47,824 6 comments 0 reactions 0 assignees View on GitHub
Component: Documentation Component: Python Component: Swift Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

This is a follow-up to a discussion started in https://github.com/apache/arrow-swift/pull/95#discussion_r2426443074 where we were fixing some issues in `ArrowWriter`.

@kou was asking how the fixes lined up with the spec, and that was a difficult question to answer. Looking at the [encapsulated-message-format](https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format):

"The metadata_size includes the size of the Message plus padding. The metadata_flatbuffer contains a serialized Message Flatbuffer value"

It seems reasonable to assume that the `Message` probably refers to a FlatBuffers `Message` therefore metadata_size should be exactly the length of this message plus padding. However, according to PyArrow this includes the `Message` size plus 8 bytes.

Doing a small experiment with the testFileWriter_bool example in arrow-swift, writing the block metadata without the 8 byte prefix in the metadataLength:

offset: 120
metadataLength: 208
bodyLength: 296

PyArrow won't open the file, throwing an error:
pyarrow.lib.ArrowInvalid: flatbuffer size 8 invalid. File offset: 128, metadata length: 208
(Note that the `Message` size is confirmed to be 208 bytes).

However if the metadataLength includes the 8 byte prefix, i.e.:

offset: 120
metadataLength: 216
bodyLength: 296

The file is valid according to PyArrow.

### Component(s)

Documentation, Python, Swift

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.