apache / apache/iceberg-rust

Optimize parquet_files_to_data_files by reusing schema index

Open Beginner friendly
#2,032 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.4k
Forks
567
Avg merge
2d 2h
Merged PRs (30d)
93

Description

In the current implementation of [parquet_files_to_data_files](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L312) in [parquet_writer.rs](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs), we iterate over a list of files to convert them into Iceberg
[DataFile](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/base_writer/data_file_writer.rs#L66-L69) structs.

However, for every single file in the loop, we invoke
[parquet_to_data_file_builder](https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L350)
, which internally rebuilds the schema index from scratch:

```

// crates/iceberg/src/writer/file_writer/parquet_writer.rs

pub(crate) fn parquet_to_data_file_builder(...) -> Result {
// This runs for every file!
let index_by_parquet_path = {
let mut visitor = IndexByParquetPathName::new();
visit_schema(&schema, &mut visitor)?;
visitor
};

// ...
}
```

**Problem**: When importing a large number of files (e.g., thousands of files in a bulk import), we are traversing the entire schema and allocating a new name_to_id HashMap thousands of times, even though the schema is constant for the entire operation.

**Proposed Solution**:

- Extract the IndexByParquetPathName creation logic out of parquet_to_data_file_builder
https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L357-L361
- Compute this index once at the beginning of parquet_files_to_data_files (outside the loop).
https://github.com/apache/iceberg-rust/blob/b05a675db44645becc60422b596f16cca8816a89/crates/iceberg/src/writer/file_writer/parquet_writer.rs#L320-L344
- Update parquet_to_data_file_builder to accept the index as a reference argument.
- Reuse the same index for every file iteration.

Contributor guide

Open the contributing guide

Research direction

Start in crates/iceberg/src/writer/file_writer/parquet_writer.rs at parquet_files_to_data_files and parquet_to_data_file_builder, then inspect IndexByParquetPathName creation and its use. The work is done when the schema index is built once before the file loop, passed by reference to the builder, and reused for every file without changing the resulting DataFile values.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Refactor
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.