apache / apache/datafusion-comet

Comet throws RuntimeException instead of SparkException for invalid row index column type

Open Beginner friendly
#3,886 0 comments 0 reactions 0 assignees View on GitHub
spark sql tests test
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Description

The Spark SQL test \`ParquetRowIndexSuite: invalid row index column type - vectorized reader\` expects a \`SparkException\` with condition \`FAILED_READ_FILE\` when the row index temporary column is declared with the wrong type (e.g., \`StringType\` instead of \`LongType\`).

Spark's vectorized reader wraps the \`RuntimeException\` from \`findRowIndexColumnIndexInSchema\` in a \`SparkException\` via \`QueryExecutionErrors.cannotReadFilesError()\`. Comet's \`NativeBatchReader\` lets the \`RuntimeException\` propagate unwrapped.

## Steps to reproduce

Run the Spark SQL test suite \`ParquetRowIndexSuite\` with Comet enabled against Spark 4.0.1.

## Expected behavior

\`SparkException\` with condition starting with \`FAILED_READ_FILE\` should be thrown, matching Spark's native behavior.

## Actual behavior

\`RuntimeException\` is thrown directly.

Contributor guide

Open the contributing guide

Research direction

Start with NativeBatchReader and the ParquetRowIndexSuite test named `invalid row index column type - vectorized reader`. Compare Comet's handling of the RuntimeException from `findRowIndexColumnIndexInSchema` with Spark's vectorized reader, then run the Spark SQL test suite with Comet enabled. Done means the test receives a SparkException whose condition starts with `FAILED_READ_FILE`.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.