apache / apache/iceberg-go

tracking issue: Bump arrow-go / geoarrow-go versions to add logic type resolution for geo extension types

Open
#1,569 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 12h
Merged PRs (30d)
118

Description

_Creating this mainly as an issue to track progress / dependencies in external repos._

Currently in iceberg-go geo data can be read / written, but the parquet logical type labeling is wrong leading to issues like this for query engines: https://github.com/duckdb/duckdb-iceberg/issues/1175 The iceberg schema shows geometry and the physical byte array type is correct but the parquet logical type isn't resolving the logical type mapping from the geo extension type.

I have added logic in arrow-go to fix this. It created a new interface for resolving logical type <-> physical type for an arrow extension type.
- [x] https://github.com/apache/arrow-go/pull/1028
- [x] https://github.com/apache/arrow-go/pull/969
- [x] https://github.com/apache/arrow-go/pull/960

And then in geoarrow-go I have implemented the necessary associated read/write interfaces:
- https://github.com/geoarrow/geoarrow-go/pull/5
- https://github.com/geoarrow/geoarrow-go/pull/6

Outstanding tasks:
- Once arrow-go has its version bumped, I will create a PR to bump the arrow-go version in geoarrow-go and then the arrow-go version in iceberg-go will need to be bumped.

Contributor guide

Open the contributing guide

Research direction

Start by checking the completed arrow-go and geoarrow-go pull requests and the dependency declarations in those repositories. After the arrow-go version is available, update the geoarrow-go and iceberg-go versions in sequence, then verify that geo data preserves the correct Parquet logical type labeling.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
build-system, data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.