apache / apache/datafusion

Add schema mapper factory to `ListingOptions`

Open
#15,889 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

I think using the default schema mapper makes sense for now / in this PR, but in general I think it would make sense to allow the user to provide their own schema mapping rules here (so a default value that is not `NULL` can be used, for example) via their own mapper.

However, we woudl have to add a schema mapper factory to `ListingOptions`

https://github.com/apache/datafusion/blob/f1bbb1d636650c7f28f52dc507f36e64d71e1aa8/datafusion/core/src/datasource/listing/table.rs#L256-L255

(this is not a change needed for this PR, I just noticed it while reviewing this PR)

_Originally posted by @alamb in https://github.com/apache/datafusion/pull/15865#discussion_r2065156402_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/src/datasource/listing/table.rs at ListingOptions, then read the linked PR discussion for the schema-mapper context. Confirm how the default schema mapper is currently selected; done means ListingOptions can accept a user-provided schema mapper factory for custom mapping rules.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.