[Enhancement] Prune trailing fields in Hive TEXTFILE split for column-pruned queries
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues.
### Description
A column-pruned Hive `TEXTFILE` query reads only the columns listed in `_col_idxs`, but the field splitter still scans every separator of each line — including trailing fields that are never used. For wide tables with many unused trailing columns this is wasted work proportional to the unused tail length of every row.
### Proposal
On the query path, after the reader resolves `_col_idxs`, compute `max(_col_idxs) + 1` and pass it to the splitter as a split limit. The single-char, non-escape path of `HiveTextFieldSplitter` then stops as soon as it has emitted that many fields. Each emitted field is still bounded by its own separator, so the produced prefix is byte-identical to a full split, and the reader only needs those prefix columns — query results are unchanged.
The escape-configured path, the multi-char (KMP) path, and `EncloseCsvTextFieldSplitter` ignore the limit and split fully, so escaped separators and enclose quoting are unaffected. The load path is unaffected.
### Use case
Faster scans of wide Hive `TEXTFILE` tables when queries select only a prefix of the columns.
### Related issues
_No response_
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start by locating HiveTextFieldSplitter and the reader path that resolves _col_idxs. Compare the single-character, non-escape path with the escape, multi-character, and enclose paths, then run the relevant existing tests or add coverage to verify prefix output and unchanged query results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100