[Go] Support external schema in ipc readers
- Dominant language
- Assembly
- Stars
- 404
- Forks
- 145
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 87
Description
(Apologies if I'm imprecise here, I'm still coming up to speed on the arrow details.)
The IPC message format describes how data and metadata messages are encapsulated, but it is not a requirement that each message must include the schema.
In Go, github.com/apache/arrow/go/arrow/ipc contains NewReader() for setting up reading of IPC messages, and accepts the option WithSchema to pass the schema into said reader. However, the implementation merely uses that information to compare that the schema it reads from the IPC stream matches the passed in reader. This request is to allow WithSchema to behave as expected, and use the option-provided Schema for performing reads.
The one gotcha here appears to be the dictionary type map, which is currently retained independently of the schema but is part of the internal readSchema() setup. Completeness may warrant another option for communicating those externally as well? Or perhaps option-passed Schema should be documented to not support dictionary types?
**Reporter**: [Seth Hollyman](https://issues.apache.org/jira/browse/ARROW-13191)
#### PRs and other links:
- [GitHub Pull Request apache/arrow#10603](https://github.com/apache/arrow/pull/10603)
**Note**: *This issue was originally created as [ARROW-13191](https://issues.apache.org/jira/browse/ARROW-13191). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start in github.com/apache/arrow/go/arrow/ipc, focusing on NewReader, WithSchema, and the internal readSchema setup. Trace how the option-provided schema is currently compared with the IPC schema, including the independently retained dictionary type map. Done means external schemas drive reads, with dictionary-type behavior addressed or documented; review PR apache/arrow#10603 for existing work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100