apache / apache/iceberg-go

Iceberg Extensions

Open
#183 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Feature Request / Improvement

There are some various table formats that extend Iceberg by allowing additional metadata to be added to various components, for example, [Havasu](https://github.com/wherobots/havasu/tree/main). I (and I'm sure others) would like the ability to to extend metadata such as the table metadata, manifest lists, and manifest entries to read additional metadata. I can't speak for all iceberg extensions, but havasu extends iceberg by adding some additional metadata fields in:
1. Table Metadata: https://github.com/wherobots/havasu/blob/main/spec.md#table-metadata
2. Manifests: https://github.com/wherobots/havasu/blob/main/spec.md#manifests

The addition of the geometry bounds on the manifests is primarily for scan planning. It seems like at this point, until Iceberg support for geospatial types is added (which seems like it could be in the [fairly distant future](https://github.com/apache/iceberg/issues/10260)), augmenting scan planning may be a bit far away, but I'd propose that allowing metatadata extensions to table metadata and manifests maybe be worth consideration.

There are probably many ways this could be implemented, and I'm happy to take a stab at it since it's something I personally need on my project. First I wanted to ask if this is within the scope of this project or if you have a recommended path to something like this?

Contributor guide

Open the contributing guide

Research direction

Review the linked Havasu specifications for table metadata and manifests, then inspect iceberg-go's existing handling of table metadata, manifest lists, and manifest entries. The issue does not identify repository files, tests, or a settled implementation path, so the scope and completion criteria need maintainer clarification before work can begin.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.