apache / apache/iceberg-python

Avro schema is re-converted to an Iceberg schema on every manifest read during scan planning

Open Beginner friendly
#3,662 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.1k
Forks
581
Avg merge
1d 17h
Merged PRs (30d)
78

Description

### Feature Request / Improvement

`AvroFileHeader.get_schema()` (`pyiceberg/avro/file.py`) runs the full `avro_to_iceberg` conversion every time an
Avro file is opened.

**The inefficiency**

- Every manifest under a spec embeds an *identical* Avro schema string.
- So during scan planning, that same conversion is repeated **once per manifest**.
- The cost grows with the manifest count, even though only a couple of distinct schemas are ever involved.

**Proposed fix**

- The conversion depends only on the schema string, so the result can be cached (keyed on that string).

**Measured impact**

- `scan().plan_files()` on an unpartitioned 150-manifest table: **~86 ms → ~48 ms (~1.8× faster)**.
- The saving grows as the number of manifests increases.

I am willing to contribute for this improvement.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in pyiceberg/avro/file.py at AvroFileHeader.get_schema() and trace the avro_to_iceberg conversion. Use scan().plan_files() with repeated manifests to observe the repeated work and confirm that conversion results are reused by schema string, with planning performance improved as manifest count grows.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.