apache / apache/doris

[Enhancement] Prune trailing fields in Hive TEXTFILE split for column-pruned queries

Open
#66,121 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues.

### Description

A column-pruned Hive `TEXTFILE` query reads only the columns listed in `_col_idxs`, but the field splitter still scans every separator of each line — including trailing fields that are never used. For wide tables with many unused trailing columns this is wasted work proportional to the unused tail length of every row.

### Proposal

On the query path, after the reader resolves `_col_idxs`, compute `max(_col_idxs) + 1` and pass it to the splitter as a split limit. The single-char, non-escape path of `HiveTextFieldSplitter` then stops as soon as it has emitted that many fields. Each emitted field is still bounded by its own separator, so the produced prefix is byte-identical to a full split, and the reader only needs those prefix columns — query results are unchanged.

The escape-configured path, the multi-char (KMP) path, and `EncloseCsvTextFieldSplitter` ignore the limit and split fully, so escaped separators and enclose quoting are unaffected. The load path is unaffected.

### Use case

Faster scans of wide Hive `TEXTFILE` tables when queries select only a prefix of the columns.

### Related issues

_No response_

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start by locating HiveTextFieldSplitter and the reader path that resolves _col_idxs. Compare the single-character, non-escape path with the escape, multi-character, and enclose paths, then run the relevant existing tests or add coverage to verify prefix output and unchanged query results.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.