Optimize parquet_files_to_data_files by reusing schema index
- Dominant language
- Rust
- Stars
- 1.4k
- Forks
- 567
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 93
Description
In the current implementation of [parquet_files_to_data_files](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L312) in [parquet_writer.rs](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs), we iterate over a list of files to convert them into Iceberg
[DataFile](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/base_writer/data_file_writer.rs#L66-L69) structs.
However, for every single file in the loop, we invoke
[parquet_to_data_file_builder](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L350)
, which internally rebuilds the schema index from scratch:
```
// crates/iceberg/src/writer/file_writer/parquet_writer.rs
pub(crate) fn parquet_to_data_file_builder(...) -> Result {
// This runs for every file!
let index_by_parquet_path = {
let mut visitor = IndexByParquetPathName::new();
visit_schema(&schema, &mut visitor)?;
visitor
};
// ...
}
```
**Problem**: When importing a large number of files (e.g., thousands of files in a bulk import), we are traversing the entire schema and allocating a new name_to_id HashMap thousands of times, even though the schema is constant for the entire operation.
**Proposed Solution**:
- Extract the IndexByParquetPathName creation logic out of parquet_to_data_file_builder
https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L357-L361
- Compute this index once at the beginning of parquet_files_to_data_files (outside the loop).
https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L320-L344
- Update parquet_to_data_file_builder to accept the index as a reference argument.
- Reuse the same index for every file iteration.
Contributor guide
Research direction
Start in crates/iceberg/src/writer/file_writer/parquet_writer.rs at parquet_files_to_data_files and parquet_to_data_file_builder, then inspect IndexByParquetPathName creation and its use. The work is done when the schema index is built once before the file loop, passed by reference to the builder, and reused for every file without changing the resulting DataFile values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100