apache / apache/arrow

Allow for custom_metadata for Tensor and SparseTensor IPC messages

Open
#27,744 1 comment 0 reactions 0 assignees View on GitHub
Component: Format Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The `Schema` IPC message allows `custom_metadata`. [See here:](https://github.com/apache/arrow/blob/69d436dff83ce33e43acf3e2402e72317fa82aa7/format/Schema.fbs#L401)

It would be great if one could specify similar `custom_metadata` for `Tensor` and `SparseTensor` messages. Currently if one wants to pass the "context/metadata" of a tensor, one has to find out-of-band solutions.

It would be nice, if this could be incorporated into the IPC flow. E.g. one might want to pass a timestamp associated with a tensor indicating when the data was obtained or geographical coordinates indicating the edges of a 2D tensor to locate the data geographically.

**Reporter**: [ARF](https://issues.apache.org/jira/browse/ARROW-11902) / @ARF1
**Watchers**: [Rok Mihevc](https://issues.apache.org/jira/browse/ARROW-11902) / @rok

**Note**: *This issue was originally created as [ARROW-11902](https://issues.apache.org/jira/browse/ARROW-11902). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with format/Schema.fbs and compare the existing Schema IPC message custom_metadata with the Tensor and SparseTensor message definitions. Trace the IPC flow for both message types and confirm that custom metadata can be specified and preserved for each without relying on out-of-band context.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.