bug: OOM error inserting parquet with partition
Open
@xieandrew is already working on this.
Since Mar 19, 2026.
bug
p2
- Dominant language
- Java
- Stars
- 35
- Forks
- 7
- Avg merge
- 5d 12h
- Merged PRs (30d)
- 2
Description
../../ice/ice/target/ice-jar insert ontime.ontime_by_month -p ../../../datasets/ontime/parquet/ontime.parquet --partition='[{"column":"FlightDate","transform":"month"}]' --insecure
2026-02-26 11:32:56 [main] WARN com.altinity.ice.cli.Main > SSL certificate verification is DISABLED. This is insecure and should only be used for development.
2026-02-26 11:32:57 [main] INFO c.a.i.c.internal.cmd.Insert > Updating write.distribution-mode to "range"
2026-02-26 11:32:57 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > ../../../datasets/ontime/parquet/ontime.parquet: processing
2026-02-26 11:32:57 [-5-thread-1] WARN c.a.i.c.internal.cmd.Insert > ../../../datasets/ontime/parquet/ontime.parquet: File contains multiple partition values for 'FlightDate' (min: 213, max: 643). Falling back to full scan (slow)
2026-02-26 11:32:57 [-5-thread-1] INFO c.a.i.c.internal.cmd.Insert > ../../../datasets/ontime/parquet/ontime.parquet: partitioning
2026-02-26 11:35:58 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:231)
at com.altinity.ice.cli.Main.insert(Main.java:512)
at com.altinity.ice.cli.Main.lambda$main$1(Main.java:779) [9 skipped]
at com.altinity.ice.cli.Main.main(Main.java:786) [1 skipped]
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.iceberg.data.GenericRecord.<init>(GenericRecord.java:71) [1 skipped]
at org.apache.iceberg.data.GenericRecord.copy(GenericRecord.java:134)
at org.apache.iceberg.parquet.ParquetValueReaders$RecordReader.newStructData(ParquetValueReaders.java:1005)
at org.apache.iceberg.parquet.ParquetValueReaders$RecordReader.newStructData(ParquetValueReaders.java:990)
at org.apache.iceberg.parquet.ParquetValueReaders$StructReader.read(ParquetValueReaders.java:913)
at org.apache.iceberg.parquet.ParquetReader$FileIterator.next(ParquetReader.java:130)
at com.altinity.ice.cli.internal.iceberg.Partitioning.partition(Partitioning.java:266)
at com.altinity.ice.cli.internal.cmd.Insert.copyPartitionedAndSorted(Insert.java:597)
at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:505)
at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$2(Insert.java:194)
at com.altinity.ice.cli.internal.cmd.Insert$$Lambda/0x00007b02f830df28.call(Unknown Source)
[5 skipped]
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.