avro schema reader ignores iceberg-field-name property — returns sanitized names from java manifests
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
## problem
when java writes manifests with partition field names that violate avro naming rules, it sanitizes them (e.g. `815d3b...` → `_815d3b...`) and stores the original in an `iceberg-field-name` avro field property. iceberg-rust's avro reader uses the avro field name directly without checking this property, so it returns the sanitized name instead of the original.
this causes field name mismatches when reading java-written manifests — the partition field name in the manifest entry won't match the partition field name in the table's partition spec.
## relevant code
`crates/iceberg/src/avro/schema.rs` — `avro_schema_to_schema` reads `&avro_field.name` without checking for `iceberg-field-name` custom property.
java's read path resolves fields by `field-id` (integer), but also provides the original name via `ICEBERG_FIELD_NAME_PROP`:
```java
public static final String ICEBERG_FIELD_NAME_PROP = "iceberg-field-name";
```
## reproduction
1. create a table in java with a partition field name starting with a digit
2. write data (java produces manifests with sanitized avro field names + `iceberg-field-name` property)
3. read the manifest in iceberg-rust — field names will be the sanitized form, not the original
## expected behavior
when reading avro schemas from manifests, check each field for the `iceberg-field-name` property. if present, use that as the iceberg field name instead of the avro field name.
## notes
- iceberg resolves fields by `field-id` in most paths, so the impact is limited to code that uses field names from manifest entries
- this is a well-established convention in the java implementation (since 2019)
- pyiceberg implements its own avro reader that bypasses name validation entirely
Contributor guide
Research direction
Start in crates/iceberg/src/avro/schema.rs, focusing on avro_schema_to_schema and how each Avro field's name is read. Compare the documented iceberg-field-name property with Java's ICEBERG_FIELD_NAME_PROP convention, then verify that Java-written manifests expose the original partition field name rather than the sanitized Avro name.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100