Manifest: Partition logical-type decoding panics and mis-scales milliseconds
- Dominant language
- Go
- Stars
- 463
- Forks
- 232
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 121
Description
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
Right now, `dataFile.convertAvroValueToIcebergType` asserts `v.(int64)` unchecked on every time and timestamp arm. Per the Avro spec, `twmb/avro` ignores a logical type invalid for its underlying Avro type and decodes the raw primitive - but the parsed schema still reports the `logicalType`.
Reading such a manifest panics. `Partition()` returns no error, so callers cannot recover.
```bash
panic: interface conversion: interface {} is int32, not int64
iceberg-go.(*dataFile).convertAvroValueToIcebergType(manifest.go:2316)
iceberg-go.(*dataFile).Partition(manifest.go:2370)
```
Separately, `time-millis` and `timestamp-millis` are returned as milliseconds, though `iceberg.Time` and `iceberg.Timestamp` count microseconds : 1000× too small, silently corrupting partition filtering.
Reachable only via manifests from foreign writers.
### Status
Currently short on bandwidth to raise a PR for this fix. Contributions are welcome.
Contributor guide
Research direction
Start in manifest.go at dataFile.convertAvroValueToIcebergType and follow its callers into dataFile.Partition. Check how raw Avro values are handled for time and timestamp logical types, including invalid underlying types and millisecond units. Done means foreign-writer manifests no longer panic and time-millis/timestamp-millis values are scaled to the microsecond units expected by iceberg.Time and iceberg.Timestamp.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100