[Bug] Iceberg manifest file does not conform to spec due to missing Avro metadata
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
1.4.1
### Compute Engine
Flink
### Minimal reproduce step
1. Create any table with `'metadata.iceberg.storage'` set to any option.
2. Write any data.
3. Find the resultant `metadata` directory.
4. Choose any manifest file (of the format `{manifest-uuid}.avro`) from that directory.
5. Dump the Avro metadata of that manifest file (e.g. `avro-tools getmeta path/to/file`).
6. Observe the missing fields:
```
% avro-tools getmeta database/table/metadata/bc25ca15-c856-4800-83bd-a38d88ea6faa-m1.avro
avro.schema {"type":"record","name":"manifest_entry","fields":[{"name":"status","type":"int","field-id":0},{"name":"snapshot_id","type":["null","long"],"default":null,"field-id":1},{"name":"sequence_number","type":["null","long"],"default":null,"field-id":3},{"name":"file_sequence_number","type":["null","long"],"default":null,"field-id":4},{"name":"data_file","type":{"type":"record","name":"r2","fields":[{"name":"content","type":"int","field-id":134},{"name":"file_path","type":"string","field-id":100},{"name":"file_format","type":"string","field-id":101},{"name":"partition","type":{"type":"record","name":"r102","fields":[]},"field-id":102},{"name":"record_count","type":"long","field-id":103},{"name":"file_size_in_bytes","type":"long","field-id":104},{"name":"null_value_counts","type":["null",{"type":"array","items":{"type":"record","name":"k121_v122","fields":[{"name":"key","type":"int","field-id":121},{"name":"value","type":"long","field-id":122}]},"logicalType":"map"}],"default":null,"field-id":110},{"name":"lower_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k126_v127","fields":[{"name":"key","type":"int","field-id":126},{"name":"value","type":"bytes","field-id":127}]},"logicalType":"map"}],"default":null,"field-id":125},{"name":"upper_bounds","type":["null",{"type":"array","items":{"type":"record","name":"k129_v130","fields":[{"name":"key","type":"int","field-id":129},{"name":"value","type":"bytes","field-id":130}]},"logicalType":"map"}],"default":null,"field-id":128},{"name":"referenced_data_file","type":["null","string"],"default":null,"field-id":143},{"name":"content_offset","type":["null","long"],"default":null,"field-id":144},{"name":"content_size_in_bytes","type":["null","long"],"default":null,"field-id":145}]},"field-id":2}]}
avro.codec snappy
```
Only `avro.schema` and `avro.codec` are present. `schema` and `partition-spec` are absent (Iceberg V1), as are `schema-id`, `partition-spec-id`, `format-version`, and `content` (Iceberg V2/V3).
### What doesn't meet your expectations?
According to the [Iceberg Manifests specification](https://iceberg.apache.org/spec/#manifests):
> A manifest file must store the partition spec and other metadata as properties in the Avro file's key-value metadata:
> v1 | v2 and v3 | Key | Value
> -- | -- | -- | --
> required | required | schema | JSON representation of the table schema at the time the manifest was written
> optional | required | schema-id | ID of the schema used to write the manifest as a string
> required | required | partition-spec | JSON representation of only the partition fields array of the partition spec used to write the manifest. See [Appendix C](https://iceberg.apache.org/spec/#partition-specs)
> optional | required | partition-spec-id | ID of the partition spec used to write the manifest as a string
> optional | required | format-version | Table format version number of the manifest as a string
> | required | content | Type of content files tracked by the manifest: "data" or "deletes"
Paimon does not store **any** Avro metadata in the manifest file. It treats the file as a generic data file. This is out of compliance with the Iceberg table specification.
### Anything else?
This issue was discovered while trying to read a Paimon table with Iceberg metadata via the Clickhouse Iceberg table engine. Because the Clickhouse Iceberg engine enforces strict Iceberg schema compliance, it detected that the manifest file was noncompliant, and failed to read the table.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the issue by creating a table with metadata.iceberg.storage, writing data, and inspecting a {manifest-uuid}.avro file in the metadata directory with avro-tools getmeta. Compare the emitted properties with the Iceberg Manifests specification; done means the required Avro metadata is present so strict Iceberg readers such as ClickHouse can read the table.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100