apache / apache/datafusion

Allow providing file schema for parquet files

Open
#7,067 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently, [manually specifying the schema for a Parquet file will error](https://github.com/apache/arrow-datafusion/blob/49c91b563ad894b2f368690d85402895bdeaa73a/datafusion/sql/src/statement.rs#L636). There is no particular reason for this, we may allow schema providing for parquet files as well.

This is a blocker for the https://github.com/apache/arrow-datafusion/issues/7036.

### Describe the solution you'd like

- Add schema field in `ParquetReadOptions`
- Remove the error message.
- Add tests when a schema is provided.

### Describe alternatives you've considered

NA

### Additional context

NA

Contributor guide

Open the contributing guide

Research direction

Start at the linked location in datafusion/sql/src/statement.rs and inspect ParquetReadOptions and the existing error path. Find the relevant Parquet SQL tests, add coverage for a provided schema, and verify that schema-provided Parquet reads no longer produce the current error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.