apache / apache/iceberg-rust

avro schema reader ignores iceberg-field-name property — returns sanitized names from java manifests

Open Beginner friendly
#2,536 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

## problem

when java writes manifests with partition field names that violate avro naming rules, it sanitizes them (e.g. `815d3b...` → `_815d3b...`) and stores the original in an `iceberg-field-name` avro field property. iceberg-rust's avro reader uses the avro field name directly without checking this property, so it returns the sanitized name instead of the original.

this causes field name mismatches when reading java-written manifests — the partition field name in the manifest entry won't match the partition field name in the table's partition spec.

## relevant code

`crates/iceberg/src/avro/schema.rs` — `avro_schema_to_schema` reads `&avro_field.name` without checking for `iceberg-field-name` custom property.

java's read path resolves fields by `field-id` (integer), but also provides the original name via `ICEBERG_FIELD_NAME_PROP`:
```java
public static final String ICEBERG_FIELD_NAME_PROP = "iceberg-field-name";
```

## reproduction

1. create a table in java with a partition field name starting with a digit
2. write data (java produces manifests with sanitized avro field names + `iceberg-field-name` property)
3. read the manifest in iceberg-rust — field names will be the sanitized form, not the original

## expected behavior

when reading avro schemas from manifests, check each field for the `iceberg-field-name` property. if present, use that as the iceberg field name instead of the avro field name.

## notes

- iceberg resolves fields by `field-id` in most paths, so the impact is limited to code that uses field names from manifest entries
- this is a well-established convention in the java implementation (since 2019)
- pyiceberg implements its own avro reader that bypasses name validation entirely

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/avro/schema.rs, focusing on avro_schema_to_schema and how each Avro field's name is read. Compare the documented iceberg-field-name property with Java's ICEBERG_FIELD_NAME_PROP convention, then verify that Java-written manifests expose the original partition field name rather than the sanitized Avro name.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.