apache / apache/datafusion

Add way to specify custom prefix for partitioned file outputs

Open
#19,261 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

Currently there is no way to control file name for generated files when you write partitioned outputs

### Describe the solution you'd like

I would like to have option to allow user to specify prefix which would allow user to have predictable part of file name instead of just randomised or `part-X`

### Describe alternatives you've considered

A more complex solution would be to introduce function callback that would be called to generate such prefix which would enable user to customise behaviour of prefix generation or replace randomized generation with his own implementation

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the partitioned file output implementation and the tests covering generated file names. Trace how the current randomized or `part-X` names are produced, then define where a user-specified prefix belongs in the public interface. Done means partitioned outputs use the requested predictable prefix without breaking existing naming behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.