apache / apache/datafusion

Support extended partition cols for listing table.

Open
#18,482 5 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When scanning partitioned files, there are scenarios where runtime-generated values (not persisted in the files) need to be attached to each RecordBatch.

Currently, when a partition contains multiple files, loadNextBatch has no context about which file it is returning rows from.
This makes it impossible to append per-file runtime data to the resulting RecordBatch.
We’d like a way to extend the file schema and stream with additional columns—similar to how table_partition_cols are added from directory structure.

Example
```
Partition directory: /data1/
Files:
/data1/file1
/data1/file2
/data1/file3

File schema: { row_id: Int32, b: Int32 }

Runtime metadata:
file1 -> cumulative_total_rows = 5
file2 -> cumulative_total_rows = 7
file3 -> cumulative_total_rows = 17

Derived schema:
{ row_id: Int32, b: Int32, cumulative_total_rows }

Example expression:
row_id + cumulative_total_rows
```

### Describe the solution you'd like

Extend the ListingTable and ListingOptions to support user-provided extended columns (extended_cols), which are appended to each file’s stream and schema—analogous to table_partition_cols.

- Add extended_cols to ListingOptions, defined as:

```
extended_cols: HashMap>
```
where:
- outer key = column name
- inner key = file name
- value = runtime constant for that file

- These values should be made available in the scan output (similar to partition columns), allowing expressions to reference them.

### Describe alternatives you've considered

- Expose ObjectMeta to PhysicalExprAdapter, allowing it to append file metadata (e.g., file name) to the stream.
- Then a MemTable with file_name → extended_col mappings could be joined to enrich data.

Any alternative mechanism that makes per-file runtime context accessible during scan would work.

@alamb @timsaucer Any thoughts?

Contributor guide

Open the contributing guide

Research direction

Start by tracing ListingTable and ListingOptions through the scan path, especially loadNextBatch and the existing table_partition_cols handling. Define how extended_cols values are matched to files and appended to the schema and stream, then verify that scan expressions can reference them and that per-file values remain correct across multiple files.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.