apache / apache/arrow

[C++] Support for textual, JSON schema representation

Open
#25,078 19 comments 0 reactions 0 assignees View on GitHub
Component: C++ Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Currently, Arrow has no textual representation for its schema that could serve the same purposes as JSON-Schema for JSON, the .proto files for Protobuf, etc. This issue is about adding such a text representation for an Arrow schema, to fill the same use cases that these textual representations fill for other data serialization formats.

The requirements for a text schema representation:
- Data, not code (can be used without being run directly, unlike e.g. calls to the Python API to create a Schema object)
- Readable by people who are experts in their field (e.g. data scientists, etc.) and are however _not_ Arrow experts, without needing the doc side by side
- Small modifications possible with no or light usage of the doc (e.g. changing a field from int32 to int64)
- Writing new schemas from scratch possible with the doc for non-Arrow experts
- Not tied to a particular version of Arrow & compatible between Arrow versions

And from a software engineering point of view, it would be very desirable for the implementation to not add another library dependency for Arrow (which already has many).

After discussion on the mailing list, the JSON representation for Flatbuffers data seemed the best candidate. It is a format supported by the Flatbuffers projects for serializing Flatbuffers assets in a human-readable format, for inclusion under source-control. And there is already functionality in Arrow to convert Schema objects to a Flatbuffers representation. This would meet all the requirements above, while requiring only a small amount of new Arrow code to implement.

This issue will add functions Arrow to load and save a textual, JSON representation of an Arrow schema, by first converting it to a FlatBuffers object, and then using the Flatbuffers functionality to save/load such objects as JSON.

**Reporter**: [Christian Hudon](https://issues.apache.org/jira/browse/ARROW-8952) / @chrish42
#### Related issues:
- [[Java] Support for textual JSON schema representation (was: JSON representation of pojo.Schema is incompatible with flatbuffers JSON generated via C++ API)](https://github.com/apache/arrow/issues/33203) (supercedes)
#### PRs and other links:
- [GitHub Pull Request #7110](https://github.com/apache/arrow/pull/7110)
- [WIP Pull Request](https://github.com/apache/arrow/pull/7110)

**Note**: *This issue was originally created as [ARROW-8952](https://issues.apache.org/jira/browse/ARROW-8952). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Arrow's existing conversion from Schema objects to FlatBuffers and the FlatBuffers functionality for loading and saving JSON representations. Done means Arrow can load and save a version-compatible, human-readable textual JSON schema without adding another library dependency; note that the issue links a superseding related issue and an earlier WIP pull request.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.