apache / apache/auron

Silent fallback to empty partitions may cause data loss

Open
#2,229 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

The current implementation silently falls back to Seq.empty when both DataSource V2 and reflective partition retrieval fail. This may lead to empty input partitions without any failure signal, potentially causing silent data loss or empty query results.

**Problem Description**

In inputPartitions(exec: BatchScanExec), when partition planning fails:

1. V2 API failure → caught and ignored
2. Reflection failure → caught and ignored
3. Final fallback → Seq.empty

`
catch {
case t: Throwable =>
logWarning(...)
Seq.empty
}
`

and

`
catch {
case NonFatal(t) =>
logWarning(...)
Seq.empty
}
`

**Risk**

Returning an empty partition list is dangerous because:

1. Silent data loss

Downstream execution may interpret empty partitions as:

no data available
valid empty scan result

→ leads to incorrect empty query results without failure

2. Hard to detect in production

Only warning logs are emitted
No exception or job failure
Easy to miss in large pipelines

3. Misleading semantics

A scan failure (cannot plan partitions) is semantically different from:

dataset is actually empty

But both are currently indistinguishable.

**Expected Behavior**

When both:

V2 planInputPartitions
reflection-based partition retrieval

fail, the system should:

Fail fast with a clear exception:

`
throw new IllegalStateException(
s"Failed to plan input partitions for ${exec.getClass.getName}")
`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.