[Format] Encapsulated message format metadata_size ambiguity.
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
This is a follow-up to a discussion started in https://github.com/apache/arrow-swift/pull/95#discussion_r2426443074 where we were fixing some issues in `ArrowWriter`.
@kou was asking how the fixes lined up with the spec, and that was a difficult question to answer. Looking at the [encapsulated-message-format](https://arrow.apache.org/docs/format/Columnar.html#encapsulated-message-format):
"The metadata_size includes the size of the Message plus padding. The metadata_flatbuffer contains a serialized Message Flatbuffer value"
It seems reasonable to assume that the `Message` probably refers to a FlatBuffers `Message` therefore metadata_size should be exactly the length of this message plus padding. However, according to PyArrow this includes the `Message` size plus 8 bytes.
Doing a small experiment with the testFileWriter_bool example in arrow-swift, writing the block metadata without the 8 byte prefix in the metadataLength:
offset: 120
metadataLength: 208
bodyLength: 296
PyArrow won't open the file, throwing an error:
pyarrow.lib.ArrowInvalid: flatbuffer size 8 invalid. File offset: 128, metadata length: 208
(Note that the `Message` size is confirmed to be 208 bytes).
However if the metadataLength includes the 8 byte prefix, i.e.:
offset: 120
metadataLength: 216
bodyLength: 296
The file is valid according to PyArrow.
### Component(s)
Documentation, Python, Swift
Contributor guide
Assessment
This issue has not been assessed yet.