Manifest decoding does not support lower-case values for `file_format`, making data files from some writers unreadable
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
## Version
Observed on `main` at `87789102577c` (2026-09-01) using DuckDB `v1.5.5` as writer.
## Summary
Reading a table whose Avro manifests specify a lowercase `file_format` for data files (e.g. `parquet`) fails on the first data file with the following error:
```
not implemented: only parquet format is implemented, got parquet
```
This happens via the ordinary read path: `LoadTable` followed by `tbl.Scan()`.
The Iceberg spec [uses lowercase strings as illustrative example values](https://iceberg.apache.org/spec/#data-file-fields) for `file_format`, so so I don't think this is reasonably characterized as a DuckDB quirk/bug.
Meanwhile, unlike `iceberg-go`, both `pyiceberg` and the Java reference implementation appear to be case-agnostic with respect to `file_format` (Pyiceberg has a [test case](https://github.com/apache/iceberg-python/blob/d12f68042a459e5d4732bd6ee5a188e891b0dd44/tests/catalog/test_scan_planning_models.py#L208) specifically intended to ensure case-insensitivity when scanning). `iceberg-go` ought to behave similarly to ensure portability of tables written by other writers.
Steps to reproduce
**1. Start a REST catalog over a local directory.**
```bash
mkdir -p /tmp/icewh
docker run -d --name ice-rest -p 8181:8181 \
-v /tmp/icewh:/tmp/icewh \
-e CATALOG_WAREHOUSE=file:///tmp/icewh \
apache/iceberg-rest-fixture:latest
```
**2. Create and populate a table with DuckDB.**
```sql
INSTALL iceberg; LOAD iceberg;
ATTACH 'file:///tmp/icewh' AS cat (
TYPE ICEBERG, ENDPOINT 'http://localhost:8181', AUTHORIZATION_TYPE 'none'
);
CREATE SCHEMA cat.ns;
CREATE TABLE cat.ns.t (id BIGINT);
INSERT INTO cat.ns.t VALUES (1);
```
**3. Confirm the manifest records a lowercase `file_format`**
```sql
SELECT file_format FROM iceberg_metadata(cat.ns.t); -- "parquet" expected
```
**4. Scan the table with iceberg-go**:
```go
cat, _ := rest.NewCatalog(ctx, "cat", "http://localhost:8181",
rest.WithWarehouseLocation("file:///tmp/icewh"))
tbl, err := cat.LoadTable(ctx, table.Identifier{"ns", "t"})
// succeeds: table metadata and manifest lists parse fine
_, recs, err := tbl.Scan().ToArrowRecords(ctx)
for rec, err := range recs {
if err != nil {
log.Fatal(err) // not implemented: only parquet format is implemented, got parquet
}
_ = rec
}
```
Details on Java/Python behavior
Both the Java and Python reference implementations avoid problems here by normalizing `file_format` casing on decode. `pyiceberg` has a test case specifically to ensure case-insensitivity.
**Java** — `BaseFile.internalSet` routes the raw Avro value through `FileFormat.fromString`:
```java
// core/src/main/java/org/apache/iceberg/BaseFile.java
case 2:
this.format = FileFormat.fromString(value.toString());
return;
```
```java
// api/src/main/java/org/apache/iceberg/FileFormat.java
public static FileFormat fromString(String fileFormat) {
Preconditions.checkArgument(null != fileFormat, "Invalid file format: null");
try {
return FileFormat.valueOf(fileFormat.toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException(
String.format("Invalid file format: %s", fileFormat), e);
}
}
```
**Python** — `FileFormat` is a `str` enum with a `_missing_` hook that uppercases before matching:
```python
# pyiceberg/manifest.py
class FileFormat(str, Enum):
AVRO = "AVRO"
PARQUET = "PARQUET"
ORC = "ORC"
PUFFIN = "PUFFIN"
@classmethod
def _missing_(cls, value: object) -> None | str:
for member in cls:
if member.value == str(value).upper():
return member
return None
```
Contributor guide
Research direction
Start at the manifest decoding path used by LoadTable, then follow tbl.Scan() through ToArrowRecords(ctx) to where file_format is checked. Run the DuckDB and REST-catalog reproduction from the issue, and verify that scanning a manifest containing lowercase "parquet" succeeds without the reported error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100