apache / apache/datafusion

make some datasource listing helper functions public?

Open
#10,449 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

I'm trying to implement a custom variant of `ListingTable`, and I'm running into an issue that the following three helper functions are currently not public:

https://github.com/apache/datafusion/blob/cf0cba72204e54c8eb05156d372dc5bab248e4ea/datafusion/core/src/datasource/listing/helpers.rs#L313

https://github.com/apache/datafusion/blob/cf0cba72204e54c8eb05156d372dc5bab248e4ea/datafusion/core/src/datasource/listing/helpers.rs#L125

https://github.com/apache/datafusion/blob/cf0cba72204e54c8eb05156d372dc5bab248e4ea/datafusion/core/src/datasource/listing/helpers.rs#L51

That alone prevents me from implementing my own `ListingTable` with an otherwise reasonably level of complexity.

### Describe the solution you'd like

I'd like either:
* to make those three functions public
* or, to provide a way to customise `ListingTable` so I can:
* apply extra pruning based on column equivalence, see #10399
* and, use file lists and file statistics that are stored somewhere other than the object store

### Describe alternatives you've considered

I'd rather not copy and paste the whole of `/datafusion/core/src/datasource/listing/helpers.rs`, but that seems to be the only way around this.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in datafusion/core/src/datasource/listing/helpers.rs at the three linked helper functions and review how ListingTable uses them. Determine whether exposing those helpers or adding ListingTable customization supports extra pruning and externally stored file lists and statistics; done means a viable public API is defined and the relevant behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.