apache / apache/iceberg-go

Manifest: Partition logical-type decoding panics and mis-scales milliseconds

Open
#1,847 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
463
Forks
232
Avg merge
2d 11h
Merged PRs (30d)
121

Description

### Apache Iceberg version

main (development)

### Please describe the bug 🐞

Right now, `dataFile.convertAvroValueToIcebergType` asserts `v.(int64)` unchecked on every time and timestamp arm. Per the Avro spec, `twmb/avro` ignores a logical type invalid for its underlying Avro type and decodes the raw primitive - but the parsed schema still reports the `logicalType`.

Reading such a manifest panics. `Partition()` returns no error, so callers cannot recover.
```bash
panic: interface conversion: interface {} is int32, not int64
iceberg-go.(*dataFile).convertAvroValueToIcebergType(manifest.go:2316)
iceberg-go.(*dataFile).Partition(manifest.go:2370)
```

Separately, `time-millis` and `timestamp-millis` are returned as milliseconds, though `iceberg.Time` and `iceberg.Timestamp` count microseconds : 1000× too small, silently corrupting partition filtering.

Reachable only via manifests from foreign writers.

### Status
Currently short on bandwidth to raise a PR for this fix. Contributions are welcome.

Contributor guide

Open the contributing guide

Research direction

Start in manifest.go at dataFile.convertAvroValueToIcebergType and follow its callers into dataFile.Partition. Check how raw Avro values are handled for time and timestamp logical types, including invalid underlying types and millisecond units. Done means foreign-writer manifests no longer panic and time-millis/timestamp-millis values are scaled to the microsecond units expected by iceberg.Time and iceberg.Timestamp.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.