apache / apache/datafusion

Allow deriving a Parquet file Arrow schema from metadata during lazy opening

Closed
#25,251 1 comment 0 reactions 1 assignee Claimed by @peterxcli View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Comet's projected Variant scans need to infer each file's Arrow schema from its physical Parquet schema, ignore advisory `ARROW:schema`, and map ENUM leaves to Utf8 while retaining raw BINARY. The shredded layout can differ between files and is only available after loading that file's footer.

#22360 already supports an explicit `PartitionedFile.arrow_schema`. That solves this when the complete physical file schema is known before opening. Arrow's `ArrowReaderOptions::with_schema` also supports nested Binary-to-Utf8 hints, so a new Arrow ENUM conversion API is unnecessary for this case.

The remaining gap is configuring that schema during lazy file opening. On DataFusion `9082d6b10c29b72d56bede3d8e353d9d61fde542`, [the opener](https://github.com/apache/datafusion/blob/9082d6b10c29b72d56bede3d8e353d9d61fde542/datafusion/datasource-parquet/src/opener/mod.rs#L922-L1022) creates options from the already populated `PartitionedFile`, then calls `ArrowReaderMetadata::load_async`. `ParquetFileReaderFactory` returns an `AsyncFileReader`, whose `get_metadata` receives borrowed options and returns `ParquetMetaData`; it cannot return a newly derived Arrow schema or revised reader options to the opener.

Comet currently rewrites the returned footer's Arrow schema hint. Rebuilding public `FileMetaData` cannot retain its private decryptor state, so this path requires an encrypted Variant fallback.

### Describe the solution you'd like

Allow a ParquetSource reader policy to derive a supplied Arrow schema from the loaded physical `SchemaDescriptor` during file opening, before advisory Arrow schema parsing and filter/schema preparation. Reuse Arrow's existing `with_schema` support and the original `Arc`, preserving decryption properties, page indexes and existing metadata-cache behavior.

The policy must be optional per source, leave ordinary scans unchanged, and support files with different nested schemas without fetching their footers during planning. A specific trait or callback design is not prescribed here.

### Describe alternatives you've considered

- Populate `PartitionedFile.arrow_schema` through #22360 when the full per-file schema is already available. For Comet's current path, this requires moving footer loading ahead of lazy opening.
- Rewrite the footer hint: works for unencrypted files but rebuilds metadata and couples schema policy to storage metadata.
- `with_skip_arrow_metadata(true)` handles hint suppression alone; the ENUM policy still needs the physical annotation.
- #25192 addresses recursion in existing table-schema coercions. The table's logical Variant schema does not contain the per-file shredded leaves or distinguish physical ENUM from BINARY, so recursion alone does not solve this case.

### Additional context

Tracked in apache/datafusion-comet#5477, implemented as a workaround in apache/datafusion-comet#5868. This requests a reader integration point, not a change to DataFusion's default ENUM interpretation or an expansion of the canonical Variant format.

A local check against arrow-rs `4cd8be954f6bc6b6dd265140207365b59a9900ec` passed a supplied schema for `nested { e: ENUM, b: BINARY }`: `e` became Utf8, `b` stayed Binary, an invalid advisory `ARROW:schema` was ignored, and `Arc::ptr_eq(reader_metadata.metadata(), original_metadata)` remained true. No encrypted scan was run; preserving decryption behavior needs coverage when implementing this integration.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.