apache / apache/parquet-format

Parquet Filter predicate storing nested string causing OOM's

Open
#403 3 comments 0 reactions 0 assignees View on GitHub
Priority: Critical Type: bug
Dominant language
Thrift
Stars
2.6k
Forks
508
Avg merge
2d 8h
Merged PRs (30d)
4

Description

Each Instance of ColumnFilterPredicate stores the filter values in toString variable eagerly. Which is not useful
```java

static abstract class ColumnFilterPredicate> implements FilterPredicate, Serializable {
private final Column column;
private final T value;
private final String toString;

protected ColumnFilterPredicate(Column column, T value) {
this.column = Objects.requireNonNull(column, "column cannot be null");

// Eq and NotEq allow value to be null, Lt, Gt, LtEq, GtEq however do not, so they guard against
// null in their own constructors.
this.value = value;

String name = getClass().getSimpleName().toLowerCase(Locale.ENGLISH);
this.toString = name + "(" + column.getColumnPath().toDotString() + ", " + value + ")";
}
```
 

 

If your filter predicate is too long/nested this can take a lot of memory while creating Filter.
We have seen in our productions this can go upto 4gbs of space while opening multiple parquet readers

Same thing is replicated in BinaryLogicalFilterPredicate. Where toString is eagerly calculated and stored in string and lot of duplication is happening while making And/or filter.

I did not find use case of storing it so eagerly

**Reporter**: [Abhishek Jain](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abhiSumo304)

**Note**: *This issue was originally created as [PARQUET-2220](https://issues.apache.org/jira/browse/PARQUET-2220). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reading the ColumnFilterPredicate constructor shown in the issue, then inspect BinaryLogicalFilterPredicate for the same eager toString storage. The change is complete when nested filter predicates no longer retain duplicated eagerly built strings while their string representation remains available when needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
performance
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.