apache / apache/paimon

[Bug] [Hive] IndexOutOfBoundsException when converting an unavailable dynamic BETWEEN predicate

Open Beginner friendly
#9,440 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

Paimon 1.4, using the Hive 3.1 connector.

### Compute Engine

- Hive 3.1.2
- Tez
- Java 8

### Minimal reproduce step

A Hive SQL query joining multiple Paimon tables occasionally fails while Tez is initializing input splits.

The failure happens before any data is read:

```text
ROOT_INPUT_INIT_FAILURE
Vertex Input: nws_fleet_van_proof_parcel_detail_p initializer failed

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.get(ArrayList.java:435)
at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertLeaf(
SearchArgumentToPredicateConverter.java:163)
at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertTree(
SearchArgumentToPredicateConverter.java:124)
at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convert(
SearchArgumentToPredicateConverter.java:91)
at org.apache.paimon.hive.utils.HiveUtils.createPredicate(HiveUtils.java:96)
at org.apache.paimon.hive.utils.HiveSplitGenerator.generateSplits(
HiveSplitGenerator.java:68)
at org.apache.paimon.hive.mapred.PaimonInputFormat.getSplits(
PaimonInputFormat.java:45)
```

The failing code is in the BETWEEN branch:
```java
List literalList = leaf.getLiteralList();
return builder.between(
idx,
toLiteral(columnType, literalList.get(0)),
toLiteral(columnType, literalList.get(1)));
```
Hive Tez dynamic semijoin reduction can inject a runtime predicate similar to:
```text
proof_id BETWEEN DynamicValue(min) AND DynamicValue(max)
```
When Paimon generates splits, these dynamic values may not be available yet. In this case, PredicateLeaf.getLiteralList() returns an empty list, but Paimon accesses elements 0 and 1 without checking the list size.
The issue is intermittent because whether Hive generates the runtime filter depends on the execution plan, table statistics, data size, and runtime timing.

### Minimal reproduce steps
1. Enable the Tez dynamic semijoin optimization:
```sql
SET hive.execution.engine=tez;
SET hive.tez.dynamic.semijoin.reduction=true;
```
2. Run a join where Hive pushes a dynamic min/max predicate into a Paimon table scan:
```sql
SELECT COUNT(*)
FROM paimon_db.proof_table p
JOIN paimon_db.proof_detail_table d
ON p.id = d.proof_id
WHERE p.p_date >= date_sub(current_date, 7)
AND d.p_date >= date_sub(current_date, 8)
AND d.relation_category IN (1, 3);
```
3. When the dynamic values are unavailable during split initialization, the Paimon input initializer fails with IndexOutOfBoundsException.
Disabling dynamic semijoin reduction prevents the failure:
```sql
SET hive.tez.dynamic.semijoin.reduction=false;
```

### What doesn't meet your expectations?

### Expected behavior
An unavailable or incomplete dynamic predicate should not cause the query to fail.
Because predicate pushdown is a best-effort optimization, Paimon should skip this predicate and allow Hive to apply the normal join/filter logic.
For example:
```java
List literalList = leaf.getLiteralList();
if (literalList == null || literalList.size() < 2) {
throw new UnsupportedOperationException(
"BETWEEN literals are not available");
}
```
convert() already catches UnsupportedOperationException, so the unavailable predicate can be ignored safely.
### Actual behavior
Paimon directly calls literalList.get(0) and literalList.get(1), causing the entire Tez DAG to fail during root input initialization.

### Anything else?

The Hive project documents that dynamic BETWEEN values may be unavailable while predicate pushdown is being evaluated:
- https://issues.apache.org/jira/browse/HIVE-18827
- https://issues.apache.org/jira/browse/HIVE-23295
Paimon source:
- https://github.com/apache/paimon/blob/release-1.4/paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java#L159-L164
The current workaround is:
```sql
SET hive.tez.dynamic.semijoin.reduction=false;
```
However, this disables a useful Hive runtime optimization and may increase the amount of data read and shuffled.

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java, especially convertLeaf and convert, and trace how BETWEEN literals are handled. Confirm that an unavailable or incomplete dynamic predicate is skipped during split initialization without causing IndexOutOfBoundsException, while normal predicate conversion remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.