Allow for custom_metadata for Tensor and SparseTensor IPC messages
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
The `Schema` IPC message allows `custom_metadata`. [See here:](https://github.com/apache/arrow/blob/69d436dff83ce33e43acf3e2402e72317fa82aa7/format/Schema.fbs#L401)
It would be great if one could specify similar `custom_metadata` for `Tensor` and `SparseTensor` messages. Currently if one wants to pass the "context/metadata" of a tensor, one has to find out-of-band solutions.
It would be nice, if this could be incorporated into the IPC flow. E.g. one might want to pass a timestamp associated with a tensor indicating when the data was obtained or geographical coordinates indicating the edges of a 2D tensor to locate the data geographically.
**Reporter**: [ARF](https://issues.apache.org/jira/browse/ARROW-11902) / @ARF1
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-11902) / @rok
**Note**: *This issue was originally created as [ARROW-11902](https://issues.apache.org/jira/browse/ARROW-11902). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with format/Schema.fbs and compare the existing Schema IPC message custom_metadata with the Tensor and SparseTensor message definitions. Trace the IPC flow for both message types and confirm that custom metadata can be specified and preserved for each without relying on out-of-band context.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100