Add schema mapper factory to `ListingOptions`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
I think using the default schema mapper makes sense for now / in this PR, but in general I think it would make sense to allow the user to provide their own schema mapping rules here (so a default value that is not `NULL` can be used, for example) via their own mapper.
However, we woudl have to add a schema mapper factory to `ListingOptions`
https://github.com/apache/datafusion/blob/f1bbb1d636650c7f28f52dc507f36e64d71e1aa8/datafusion/core/src/datasource/listing/table.rs#L256-L255
(this is not a change needed for this PR, I just noticed it while reviewing this PR)
_Originally posted by @alamb in https://github.com/apache/datafusion/pull/15865#discussion_r2065156402_
Contributor guide
Research direction
Start in datafusion/core/src/datasource/listing/table.rs at ListingOptions, then read the linked PR discussion for the schema-mapper context. Confirm how the default schema mapper is currently selected; done means ListingOptions can accept a user-provided schema mapper factory for custom mapping rules.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100