filter2 API performance regression
- Dominant language
- Java
- Stars
- 3.1k
- Forks
- 1.6k
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 33
Description
The new filter API seems to be much slower (or perhaps I'm using it wrong \:)
Code using an UnboundRecordFilter:
```java
ColumnRecordFilter.column(column,
ColumnPredicates.applyFunctionToBinary(
input -> Binary.fromString(value).equals(input)));
```
vs. code using FilterPredicate:
```java
eq(binaryColumn(column), Binary.fromString(value));
```
The latter performs twice as slow on the same Parquet file (built using 1.6.0rc2).
Note: the reader is constructed using
```java
ParquetReader.builder(new ProtoReadSupport().withFilter(filter).build()
```
The new filter API based approach seems to create a whole lot more garbage (perhaps due to reconstructing all the rows?).
**Reporter**: [Viktor Szathmáry](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=phraktle) / @phraktle
#### Related issues:
- [FilteredRecordReader skips rows it shouldn't for schema with optional columns](https://github.com/apache/parquet-java/issues/1730) (is related to)
**Note**: *This issue was originally created as [PARQUET-98](https://issues.apache.org/jira/browse/PARQUET-98). Please see the [migration documentation](https://issues.apache.org/jira/browse/PARQUET-2502) for further details.*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the UnboundRecordFilter path using ColumnRecordFilter.column and ColumnPredicates.applyFunctionToBinary with the FilterPredicate path using eq and binaryColumn. Reproduce both cases on the same Parquet file with ProtoReadSupport and measure runtime and garbage creation; done means explaining and correcting the reported performance regression without changing filtering results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100