apache / apache/iceberg-rust

Manifest reader hard-depends on the manifest `schema`/`partition-spec` keys; should derive from table metadata (cf. iceberg-java `specsById`)

Open
#2,682 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

## The iceberg-rust issue

When reading a manifest, `ManifestMetadata::parse` (`spec/manifest/metadata.rs`) parses the **table schema** and **partition spec** from that manifest's *own* `schema` / `partition-spec` Avro key-value metadata, and `Manifest::try_from_avro_bytes` then uses them to derive the partition type for decoding entries:

```rust
serde_json::from_slice::(meta.get("schema"))? // hard error if not a valid Iceberg schema
let partition_type = metadata.partition_spec.partition_type(&metadata.schema)?;
```

This has two problems, independent of any particular writer:

1. **Redundant dependency.** The scan already holds the authoritative `TableMetadata` (`ObjectCache::get_manifest_list` takes it, and `TableMetadata::{schema_by_id, partition_spec_by_id}` exist). A manifest's embedded `schema`/`partition-spec` is a redundant copy of what table metadata already provides by id.
2. **Out of step with the ecosystem.** Other implementations don't read the manifest's `schema` key on the scan path:
- **iceberg-java** `ManifestReader` takes `specsById` (specs from table metadata); reading the schema from manifest file metadata is **deprecated and slated for removal (1.12.0)** — the warning is literally *"Pass specsById to avoid reading from file metadata."*
- **pyiceberg** decodes via a fixed `MANIFEST_ENTRY_SCHEMAS` + the table-metadata schema.
- **iceberg-go** decodes via the Avro writer schema.

So iceberg-rust is the only implementation that hard-depends on the manifest's self-described schema, which is both unnecessary and brittle.

## Observable impact (the symptom)

Because of this, manifests whose `schema` key holds anything other than a valid Iceberg table schema are unreadable in iceberg-rust **only** — pyiceberg, Apache Doris, and Spark (iceberg-java) all read the same tables. For example, **duckdb-iceberg** serializes the manifest_entry Avro record schema into the `schema` key (using Avro type names like `array`/`record`), so iceberg-rust fails with:

```
Fail to parse schema in manifest metadata
→ data did not match any variant of untagged enum SchemaEnum
```

This is the symptom; the root concern is the redundant, ecosystem-divergent dependency above.

## Proposed fix

Derive the schema + partition spec from the **table metadata** (by the manifest's `schema-id` / `partition-spec-id`) rather than the manifest's own keys, falling back to the manifest metadata when no table metadata is available — mirroring iceberg-java's `ManifestReader(specsById)`. The scan already has the `TableMetadataRef` to thread down. PR opening shortly.

Contributor guide

Open the contributing guide

Research direction

Start in spec/manifest/metadata.rs at ManifestMetadata::parse and trace Manifest::try_from_avro_bytes into ObjectCache::get_manifest_list and the TableMetadata schema_by_id and partition_spec_by_id APIs. Check how table metadata can be threaded through the scan path, preserving manifest-metadata fallback when unavailable. Done means manifests with non-Iceberg values in their schema key can be read using table metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.