apache / apache/iceberg-go

Manifest decoding does not support lower-case values for `file_format`, making data files from some writers unreadable

Open
#1,983 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Apache Iceberg version

main (development)

### Please describe the bug 🐞

## Version

Observed on `main` at `87789102577c` (2026-09-01) using DuckDB `v1.5.5` as writer.

## Summary

Reading a table whose Avro manifests specify a lowercase `file_format` for data files (e.g. `parquet`) fails on the first data file with the following error:
```
not implemented: only parquet format is implemented, got parquet
```

This happens via the ordinary read path: `LoadTable` followed by `tbl.Scan()`.

The Iceberg spec [uses lowercase strings as illustrative example values](https://iceberg.apache.org/spec/#data-file-fields) for `file_format`, so so I don't think this is reasonably characterized as a DuckDB quirk/bug.

Meanwhile, unlike `iceberg-go`, both `pyiceberg` and the Java reference implementation appear to be case-agnostic with respect to `file_format` (Pyiceberg has a [test case](https://github.com/apache/iceberg-python/blob/d12f68042a459e5d4732bd6ee5a188e891b0dd44/tests/catalog/test_scan_planning_models.py#L208) specifically intended to ensure case-insensitivity when scanning). `iceberg-go` ought to behave similarly to ensure portability of tables written by other writers.

Steps to reproduce

**1. Start a REST catalog over a local directory.**

```bash
mkdir -p /tmp/icewh
docker run -d --name ice-rest -p 8181:8181 \
-v /tmp/icewh:/tmp/icewh \
-e CATALOG_WAREHOUSE=file:///tmp/icewh \
apache/iceberg-rest-fixture:latest
```

**2. Create and populate a table with DuckDB.**

```sql
INSTALL iceberg; LOAD iceberg;
ATTACH 'file:///tmp/icewh' AS cat (
TYPE ICEBERG, ENDPOINT 'http://localhost:8181', AUTHORIZATION_TYPE 'none'
);
CREATE SCHEMA cat.ns;
CREATE TABLE cat.ns.t (id BIGINT);
INSERT INTO cat.ns.t VALUES (1);
```

**3. Confirm the manifest records a lowercase `file_format`**

```sql
SELECT file_format FROM iceberg_metadata(cat.ns.t); -- "parquet" expected
```

**4. Scan the table with iceberg-go**:

```go
cat, _ := rest.NewCatalog(ctx, "cat", "http://localhost:8181",
rest.WithWarehouseLocation("file:///tmp/icewh"))

tbl, err := cat.LoadTable(ctx, table.Identifier{"ns", "t"})
// succeeds: table metadata and manifest lists parse fine

_, recs, err := tbl.Scan().ToArrowRecords(ctx)
for rec, err := range recs {
if err != nil {
log.Fatal(err) // not implemented: only parquet format is implemented, got parquet
}
_ = rec
}
```

Details on Java/Python behavior


Both the Java and Python reference implementations avoid problems here by normalizing `file_format` casing on decode. `pyiceberg` has a test case specifically to ensure case-insensitivity.

**Java** — `BaseFile.internalSet` routes the raw Avro value through `FileFormat.fromString`:

```java
// core/src/main/java/org/apache/iceberg/BaseFile.java
case 2:
this.format = FileFormat.fromString(value.toString());
return;
```

```java
// api/src/main/java/org/apache/iceberg/FileFormat.java
public static FileFormat fromString(String fileFormat) {
Preconditions.checkArgument(null != fileFormat, "Invalid file format: null");
try {
return FileFormat.valueOf(fileFormat.toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException(
String.format("Invalid file format: %s", fileFormat), e);
}
}
```

**Python** — `FileFormat` is a `str` enum with a `_missing_` hook that uppercases before matching:

```python
# pyiceberg/manifest.py
class FileFormat(str, Enum):
AVRO = "AVRO"
PARQUET = "PARQUET"
ORC = "ORC"
PUFFIN = "PUFFIN"

@classmethod
def _missing_(cls, value: object) -> None | str:
for member in cls:
if member.value == str(value).upper():
return member
return None
```

Contributor guide

Open the contributing guide

Research direction

Start at the manifest decoding path used by LoadTable, then follow tbl.Scan() through ToArrowRecords(ctx) to where file_format is checked. Run the DuckDB and REST-catalog reproduction from the issue, and verify that scanning a manifest containing lowercase "parquet" succeeds without the reported error.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.