apache / apache/arrow-go

[Go] Support external schema in ipc readers

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
Type: enhancement
Dominant language
Assembly
Stars
404
Forks
145
Avg merge
2d 4h
Merged PRs (30d)
87

Description

(Apologies if I'm imprecise here, I'm still coming up to speed on the arrow details.)

 

The IPC message format describes how data and metadata messages are encapsulated, but it is not a requirement that each message must include the schema.

 

In Go, github.com/apache/arrow/go/arrow/ipc contains NewReader() for setting up reading of IPC messages, and accepts the option WithSchema to pass the schema into said reader.  However, the implementation merely uses that information to compare that the schema it reads from the IPC stream matches the passed in reader.  This request is to allow WithSchema to behave as expected, and use the option-provided Schema for performing reads.

 

The one gotcha here appears to be the dictionary type map, which is currently retained independently of the schema but is part of the internal readSchema() setup. Completeness may warrant another option for communicating those externally as well?  Or perhaps option-passed Schema should be documented to not support dictionary types?

**Reporter**: [Seth Hollyman](https://issues.apache.org/jira/browse/ARROW-13191)
#### PRs and other links:
- [GitHub Pull Request apache/arrow#10603](https://github.com/apache/arrow/pull/10603)

**Note**: *This issue was originally created as [ARROW-13191](https://issues.apache.org/jira/browse/ARROW-13191). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start in github.com/apache/arrow/go/arrow/ipc, focusing on NewReader, WithSchema, and the internal readSchema setup. Trace how the option-provided schema is currently compared with the IPC schema, including the independently retained dictionary type map. Done means external schemas drive reads, with dictionary-type behavior addressed or documented; review PR apache/arrow#10603 for existing work.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.