apache / apache/paimon

[Bug] Iceberg metadata unreadable by Snowflake

Open
#9,012 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

2.0.0-rc4

### Compute Engine

Flink 2.2

### Minimal reproduce step

Create a Paimon table with Iceberg compatibility enabled, try to read the table from Snowflake.

### What doesn't meet your expectations?

Snowflake cannot read the Iceberg metadata, returning this error: `Unsupported: external field ID in metadata file cannot be zero.`
It looks like Snowflake enforces all field IDs to be > 0 (which is how the Iceberg implementation works), but Paimon generates metadata with field IDs starting from 0, so Snowflake treats that as invalid metadata.

### Anything else?

I'm not really sure what the best option is here. Looks like field mappings are encoded in Parquet footers, so maybe an option could be to specify an option at table creation to start all field IDs from 1 rather than 0 in order to produce more universally Iceberg-compatible layout?

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the failure with Paimon 2.0.0-rc4, Flink 2.2, Iceberg compatibility enabled, and Snowflake. Trace the Iceberg metadata generation and field-ID handling, then verify the resulting metadata is accepted by Snowflake without the zero-ID error. No source files or tests are named, so locating the relevant compatibility code and regression coverage is part of the work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.