Refactor hardcoded file-format conditions into a pluggable file format adapter
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Motivation
In [PR #18678 review](https://github.com/apache/hudi/pull/18678#discussion_r3178430212), @yihua noted that format-specific conditions are hardcoded throughout the codebase (e.g., `isSplitable`, `supportBatch`, `buildBaseFileReader` in `HoodieFileGroupReaderBasedFileFormat`). Each time a new file format is added (like Lance), every such branch must be updated — this is error-prone and violates the open/closed principle.
### Proposal
Introduce a pluggable **file format adapter** interface so that adding a new base file format only requires implementing the adapter rather than modifying every conditional in the read/write path.
Hardcoded conditions to consolidate (non-exhaustive, scoped to `HoodieFileGroupReaderBasedFileFormat`):
| Location | Current pattern |
|---|---|
| `isSplitable` (line 222) | `!isLance && superSplitable` |
| `supportBatch` (line 161-170) | `if PARQUET/ORC ... else if LANCE ...` |
| `buildBaseFileReader` (line 336-353) | `if PARQUET ... else if LANCE ...` |
| `withVectorRewrite` (line 447) | `if (hoodieFileFormat != HoodieFileFormat.PARQUET)` |
Similar format-branching exists in:
- `HoodieSparkFileReaderFactory`
- `HoodieSparkFileWriterFactory`
- `HoodieInternalRowFileWriterFactory`
### Suggested approach
Define a trait/interface (e.g., `HoodieBaseFileFormatAdapter`) with methods like:
- `isSplitable(): Boolean`
- `supportsBatchRead(): Boolean`
- `createReader(...): SparkColumnarFileReader`
- `needsVectorRewrite(): Boolean`
Each format (Parquet, ORC, Lance) implements the adapter. The format object is resolved once from `HoodieFileFormat` and threaded through — no more `if/else` chains on the enum.
### Context
This was identified during the Lance duplicate-read fix (#18677 / PR #18678), where `isSplitable` inherited Parquet's `true` because no Lance-specific branch existed.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading HoodieFileGroupReaderBasedFileFormat, especially isSplitable, supportBatch, buildBaseFileReader, and withVectorRewrite, then inspect the related Spark reader and writer factories. Trace how HoodieFileFormat is resolved and how Parquet, ORC, and Lance branches are used. Done means format behavior is provided through an adapter and the listed conditional branches no longer need per-format updates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100