[Bug] [Hive] IndexOutOfBoundsException when converting an unavailable dynamic BETWEEN predicate
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Paimon version
Paimon 1.4, using the Hive 3.1 connector.
### Compute Engine
- Hive 3.1.2
- Tez
- Java 8
### Minimal reproduce step
A Hive SQL query joining multiple Paimon tables occasionally fails while Tez is initializing input splits.
The failure happens before any data is read:
```text
ROOT_INPUT_INIT_FAILURE
Vertex Input: nws_fleet_van_proof_parcel_detail_p initializer failed
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.get(ArrayList.java:435)
at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertLeaf(
SearchArgumentToPredicateConverter.java:163)
at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convertTree(
SearchArgumentToPredicateConverter.java:124)
at org.apache.paimon.hive.SearchArgumentToPredicateConverter.convert(
SearchArgumentToPredicateConverter.java:91)
at org.apache.paimon.hive.utils.HiveUtils.createPredicate(HiveUtils.java:96)
at org.apache.paimon.hive.utils.HiveSplitGenerator.generateSplits(
HiveSplitGenerator.java:68)
at org.apache.paimon.hive.mapred.PaimonInputFormat.getSplits(
PaimonInputFormat.java:45)
```
The failing code is in the BETWEEN branch:
```java
List literalList = leaf.getLiteralList();
return builder.between(
idx,
toLiteral(columnType, literalList.get(0)),
toLiteral(columnType, literalList.get(1)));
```
Hive Tez dynamic semijoin reduction can inject a runtime predicate similar to:
```text
proof_id BETWEEN DynamicValue(min) AND DynamicValue(max)
```
When Paimon generates splits, these dynamic values may not be available yet. In this case, PredicateLeaf.getLiteralList() returns an empty list, but Paimon accesses elements 0 and 1 without checking the list size.
The issue is intermittent because whether Hive generates the runtime filter depends on the execution plan, table statistics, data size, and runtime timing.
### Minimal reproduce steps
1. Enable the Tez dynamic semijoin optimization:
```sql
SET hive.execution.engine=tez;
SET hive.tez.dynamic.semijoin.reduction=true;
```
2. Run a join where Hive pushes a dynamic min/max predicate into a Paimon table scan:
```sql
SELECT COUNT(*)
FROM paimon_db.proof_table p
JOIN paimon_db.proof_detail_table d
ON p.id = d.proof_id
WHERE p.p_date >= date_sub(current_date, 7)
AND d.p_date >= date_sub(current_date, 8)
AND d.relation_category IN (1, 3);
```
3. When the dynamic values are unavailable during split initialization, the Paimon input initializer fails with IndexOutOfBoundsException.
Disabling dynamic semijoin reduction prevents the failure:
```sql
SET hive.tez.dynamic.semijoin.reduction=false;
```
### What doesn't meet your expectations?
### Expected behavior
An unavailable or incomplete dynamic predicate should not cause the query to fail.
Because predicate pushdown is a best-effort optimization, Paimon should skip this predicate and allow Hive to apply the normal join/filter logic.
For example:
```java
List literalList = leaf.getLiteralList();
if (literalList == null || literalList.size() < 2) {
throw new UnsupportedOperationException(
"BETWEEN literals are not available");
}
```
convert() already catches UnsupportedOperationException, so the unavailable predicate can be ignored safely.
### Actual behavior
Paimon directly calls literalList.get(0) and literalList.get(1), causing the entire Tez DAG to fail during root input initialization.
### Anything else?
The Hive project documents that dynamic BETWEEN values may be unavailable while predicate pushdown is being evaluated:
- https://issues.apache.org/jira/browse/HIVE-18827
- https://issues.apache.org/jira/browse/HIVE-23295
Paimon source:
- https://github.com/apache/paimon/blob/release-1.4/paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java#L159-L164
The current workaround is:
```sql
SET hive.tez.dynamic.semijoin.reduction=false;
```
However, this disables a useful Hive runtime optimization and may increase the amount of data read and shuffled.
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in paimon-hive/paimon-hive-connector-common/src/main/java/org/apache/paimon/hive/SearchArgumentToPredicateConverter.java, especially convertLeaf and convert, and trace how BETWEEN literals are handled. Confirm that an unavailable or incomplete dynamic predicate is skipped during split initialization without causing IndexOutOfBoundsException, while normal predicate conversion remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100