apache / apache/datafusion-comet
Comet throws RuntimeException instead of SparkException for invalid row index column type
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Description
The Spark SQL test \`ParquetRowIndexSuite: invalid row index column type - vectorized reader\` expects a \`SparkException\` with condition \`FAILED_READ_FILE\` when the row index temporary column is declared with the wrong type (e.g., \`StringType\` instead of \`LongType\`).
Spark's vectorized reader wraps the \`RuntimeException\` from \`findRowIndexColumnIndexInSchema\` in a \`SparkException\` via \`QueryExecutionErrors.cannotReadFilesError()\`. Comet's \`NativeBatchReader\` lets the \`RuntimeException\` propagate unwrapped.
## Steps to reproduce
Run the Spark SQL test suite \`ParquetRowIndexSuite\` with Comet enabled against Spark 4.0.1.
## Expected behavior
\`SparkException\` with condition starting with \`FAILED_READ_FILE\` should be thrown, matching Spark's native behavior.
## Actual behavior
\`RuntimeException\` is thrown directly.
Contributor guide
Research direction
Start with NativeBatchReader and the ParquetRowIndexSuite test named `invalid row index column type - vectorized reader`. Compare Comet's handling of the RuntimeException from `findRowIndexColumnIndexInSchema` with Spark's vectorized reader, then run the Spark SQL test suite with Comet enabled. Done means the test receives a SparkException whose condition starts with `FAILED_READ_FILE`.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100