apache / apache/parquet-format
Parquet Filter predicate storing nested string causing OOM's
- Dominant language
- Thrift
- Stars
- 2.6k
- Forks
- 508
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 4
Description
Each Instance of ColumnFilterPredicate stores the filter values in toString variable eagerly. Which is not useful
```java
static abstract class ColumnFilterPredicate> implements FilterPredicate, Serializable {
private final Column column;
private final T value;
private final String toString;
protected ColumnFilterPredicate(Column column, T value) {
this.column = Objects.requireNonNull(column, "column cannot be null");
// Eq and NotEq allow value to be null, Lt, Gt, LtEq, GtEq however do not, so they guard against
// null in their own constructors.
this.value = value;
String name = getClass().getSimpleName().toLowerCase(Locale.ENGLISH);
this.toString = name + "(" + column.getColumnPath().toDotString() + ", " + value + ")";
}
```
If your filter predicate is too long/nested this can take a lot of memory while creating Filter.
We have seen in our productions this can go upto 4gbs of space while opening multiple parquet readers
Same thing is replicated in BinaryLogicalFilterPredicate. Where toString is eagerly calculated and stored in string and lot of duplication is happening while making And/or filter.
I did not find use case of storing it so eagerly
**Reporter**: [Abhishek Jain](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=abhiSumo304)
**Note**: *This issue was originally created as [PARQUET-2220](https://issues.apache.org/jira/browse/PARQUET-2220). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
Research direction
Start by reading the ColumnFilterPredicate constructor shown in the issue, then inspect BinaryLogicalFilterPredicate for the same eager toString storage. The change is complete when nested filter predicates no longer retain duplicated eagerly built strings while their string representation remains available when needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100